Heyden et al. 2004 :  In this project we are trying to investigate, theoretically and experimentally, different aspects of computer vision, when used together with control theory in robotics. The goal is to use new theoretical results to design a dynamic vision system making it possible for a robot to perform tasks like motion planning, grasping, obstacle detection and ground plane estimation. More precisely, we consider a vision system consisting of one or two cameras, with known or partly known calibration, mounted on a robot arm. The vision system is supposed active in the sense that it interacts with the world as an active participant. This can be accomplished by embedding the vision system in a feedback loop, where the position of the camera is constantly adjusted in order to uncover new pieces of information needed to perform a specific task. The output of the system contains control signals that can be used for the furtherance of the task, part of which is control of the vision system itself. To test theoretical results experimentally we use an ABB robot as target system.

The work has been organized in four partly overlapping packages.

Geometric Primitives and Their Representation. Invariancy properties of different parameterizations of standard geometric primitives such as edges and corners are investigated in this part. A novel approach has made it possible to write down explicit formulas for the statistical distribution of the invariant parameters, when the distribution of the individual points are given.Better understanding of the relation between kinetic depth and epipolar geometry has been achieved. The use of other features than points, e.g. conics, is investigated in order to get better robustness.

Geometry for Image Sequences. Algebraic methods for making effective use of longer image sequences have been developed. To a large extent these methods are based on bi,- tri- and multilinear forms. Depending on the a priori information at hand, different levels of reconstruction can be obtained; Euclidean, affine or projective. If the image sequence is considered as a continuous stream of images we are able to use continuous multilinear constraints.

Hand-eye Coordination. By hand-eye coordination (or calibration) is meant the process of finding the relationship between the position/orientation of the camera and the position/orientation of the end-effector, or hand, of the robot. Novel techniques for this calculation has been developed that rely directly on the spatial and temporal image derivatives in the images sequences obtained by the camera and which do not need any matching or tracking of features. The developed theory also encompass intrinsic camera calibration so that a full calibration of the robot vision setup can be achieved directly from the information in the image derivatives. Intrinsic calibration, hand-eye calibration and stereo head calibration using a planar calibration object has also been studied in this part of the project.

Vision in Feedback Loops. In this part the relationship between control theory and vision is thoroughly examined. One example of a problem in this area is to plan and perform the motion of a robot hand to a prescribed 3D position, specified by making pointers to objects or points in the images. In the study of these kinds of problems we are trying to combine the tools developed in the preceding parts with Kalman and other recursive filtering ideas. The possibility to make control possible without making a full reconstruction of the scene is examined. For example, the tasks of following the middle of a road or to detect that an object is moving too close to the camera are possible to solve by only using information about the affine depth. Important work has been accomplished in merging visual servoing and force control of a robotic manipulator.