M.3.0 Functional system integration
perception and behavior control has to be realized by the autonomous system (a
given vehicle or the body of some other agent) in a certain task domain; to be
able to do this in an intelligent way the system should have the following
capabilities and knowledge components:
- Knowledge about the
own sensory capabilities and their limitations.
- Knowledge about the
own perceptual capabilities and their
limitations; with respect to dynamic vision this includes:
A) Hypothesizing 3-D objects from sets of characteristic features, including
+ object shape (generic spatial models),
+ aspect conditions, (relative state) and
+ motion characteristics (generic
dynamic models including speed components).
B) Assuming statistical properties of the object motion observed and of the visual
mapping process as far as required by the recursive
estimation process [usually Extended
- Understanding of object
motion in 3-D space in the
task context (goals of own mission, likely object motion to be expected). Situation
assessment taking all objects relevant for own decision making
- Actual behavior
decision: Either: Continue the behavioral mode running
(feed-forward or feedback control), or: Switch to some other behavioral mode
(at the end of a ‘mission element’ or when an event requiring
another behavioral mode has been encountered).
implementation: In most practically applied systems, control
output is done by special processor hardware specific to the actuators
implemented; direct state- or output- feedback allows minimizing time
delays. That means that continuing a behavioral mode selected previously with direct feedback does
not require any communication between the cognitive part of the system
(for decision making) and the controller for the actuators; for external monitoring some
information may be exchanged.
- Two subsystems have
to be controlled in parallel:
+ The platform pointing the
cameras for visual
+ locomotion of the autonomous vehicle (system) itself.
closed-loop functioning, specific sets of data may be logged or displayed to an
operator for system monitoring.
is very difficult to explain to a newcomer the detailed functioning of such a
complex system for perception and control based on several knowledge base
components and on hypothesis generation as well as hypothesis adaptation. Here
it is tried to give four different perspective views on the system to outline
major considerations that have led to the system design given:
- Scales in space and time: From ‘here and now’ spatial
and temporal ranges have to be spanned from micrometers (pixel size) and
visual range (~ 100 m) via video cycles (40 or 33⅓ ms) to maneuvers
(in the seconds-range) and to mission duration (up to several hours and
hundreds of km).
Basic aspects for structuring in space and time
- Visual perception proceeds on three levels:
image feature extraction without semantic (and temporal) context (level 1,
possibly on special hardware).
and motion processes in the real world (level 2). Models for both shape and motion of
single objects from generic classes are used as knowledge background for
interpreting sets of features over time. Many of these
single-object-interpretation-loops (with feedback of prediction errors)
may run in parallel.
situations in the mission context, based on background knowledge on object classes and
on time histories of object states. Semantic relations between real-world
objects and behavioral capabilities in the context of goals for the own
mission is used (level 3, no direct use of image data any more).
Structure of visual perception
- Behavioral capabilities of the autonomous system and their activation in connection with
the list of mission elements, events encountered, and the situation
Structure of behavioral decision using skills
- A coarse
diagram of information flow between major subsystems
involved. Both the gaze control and the locomotion control subsystems have
to be tuned optimally for good system performance. Fusion of conventional
measurement signals and state variables perceived by vision with different
delay times requires careful synchronization and tuning for appropriate
Coarse block diagram of system integration
Visualization of feedback loops
Abstract feedback loops around 'Here and Now'