A.7 Vision-guided grasping in Space

ROTEX on board Space-Shuttle Orbiter Columbia, May 1993


The most challenging robot task was to visually determine the pose and motion relative to the robot hand. Cameras (Ki ) and the two fingers for grasping can be seen in the lower figure. The object to be grasped was a metal cube of originally 2 cm edge size, the corners of which had been cut off orthogonal to rays from the center such that just one point in the center of the original edges of the cube remained (see upper center of snapshot, left). The cube had been painted dark, the surfaces from the cut remained in metal color (bright triangles); thus, eight bright triangles and six dark squares represented the surface of the cube. After careful preparations in the laboratory on the ground, the flight experiment was successfully performed on May 3, 1993 while the Space Shuttle Columbia passed over northern Africa.

The biggest challenge in this task was to compensate the variable delay times of up to more than six seconds in the closed perception – action loop encompassing the robot arm onboard Space-Shuttle Columbia in orbit and the ground station in Oberpfaffenhofen (near Munich) where the images were analyzed by computers and the proper control output was determined. The delay time resulted from signal transfer in both directions through several (changing) satellites in orbit and computer centers as well as cables on the ground.

The robot hand, containing two cameras (Ki in the sketch top right) and a laser range finder, was initially set by manual control such that the object to be grasped was visible for the cameras in the hand. Automatic control started at 4.8 seconds; the automatic vision controlled grasping act lasted about 11 seconds while the Space Shuttle traveled almost 90 km over northern Africa on May 3, 1993. The curves labeled ‘estimated’ are the result of recursive estimation based on visual prediction error feedback.

Video 15 ROTEX- GraspingInSpace 1993

German SpaceLab D2 onboard Space Shuttle Columbia


Dickmanns ED, Fagerer C, Dickmanns D (1994). Visual Grasping with Long Delay Time of a Free Floating Object in Orbit. 4th IFAC Symposium on Robot Control (SY.RO.CO.'94), Capri, Italy, pp 947-952

Fagerer C, Dickmanns D, Dickmanns ED (1994).  Visual Grasping with Long Delay Time of a Free Floating Object in Orbit. J. Autonomous Robots, Vol. 1, No. 1, Kluwer Academic Publishers, Boston

Fagerer C (1996). Automatische Teleoperation eines Tracking- und Greifvorgangs im Weltraum basierend auf Bilddatenauswertung. Dissertation, UniBwM / LRT

Dickmanns D (1997). Rahmensystem für visuelle Wahrnehmung veränderlicher Szenen durch Computer. Dissertation, UniBwM, INF.