M.3.0 Functional system integration

Goal-oriented perception and behavior control has to be realized by the autonomous system (a given vehicle or the body of some other agent) in a certain task domain; to be able to do this in an intelligent way the system should have the following capabilities and knowledge components:

A) Hypothesizing 3-D objects from sets of characteristic features, including

+ object shape (generic spatial models),

+ aspect conditions, (relative state) and

+ motion characteristics (generic dynamic models including speed components).

B) Assuming statistical properties of the object motion observed and of the visual mapping process as far as required by the recursive estimation process [usually Extended Kalman-Filtering (EKF)].

+ The platform pointing the cameras for visual perception, and

+ locomotion of the autonomous vehicle (system) itself.

Independent of closed-loop functioning, specific sets of data may be logged or displayed to an operator for system monitoring.


It is very difficult to explain to a newcomer the detailed functioning of such a complex system for perception and control based on several knowledge base components and on hypothesis generation as well as hypothesis adaptation. Here it is tried to give four different perspective views on the system to outline major considerations that have led to the system design given:

  1. Scales in space and time: From ‘here and now’ spatial and temporal ranges have to be spanned from micrometers (pixel size) and visual range (~ 100 m) via video cycles (40 or 33⅓ ms) to maneuvers (in the seconds-range) and to mission duration (up to several hours and hundreds of km).

M.3.1 Basic aspects for structuring in space and time

  1. Visual perception proceeds on three levels:

M.3.2 Structure of visual perception

  1. Behavioral capabilities of the autonomous system and their activation in connection with the list of mission elements, events encountered, and the situation assessed.

M.3.3 Structure of behavioral decision using skills

  1. A coarse (engineering) block diagram of information flow between major subsystems involved. Both the gaze control and the locomotion control subsystems have to be tuned optimally for good system performance. Fusion of conventional measurement signals and state variables perceived by vision with different delay times requires careful synchronization and tuning for appropriate control outputs.

M.3.4 Coarse block diagram of system integration

M.3.5 Visualization of feedback loops

M.3.6 Abstract feedback loops around 'Here and Now'