See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/228845056

Teleoperation with time delay. A survey and its application to space robotics

Conference Paper · December 2000

CITATIONS READS
25 1,398

1 author:

Luis F Penin

110 PUBLICATIONS   564 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Luis F Penin on 12 April 2015.

The user has requested enhancement of the downloaded file.

 TELEOPERATION WITH TIME DELAY
A SURVEY AND ITS USE IN SPACE ROBOTICS

L. F. Peñín

GMV, S.A.
c/Isaac Newton 11, PTM, Tres Cantos, 28760 Madrid, Spain
[email protected]

ABSTRACT concept conceived by Ferrel and Sheridan (Ref 12. ) and fu r-

The existence of time delay in the communication link is one ther developed by Prof. Sheridan (Ref. 40.)
of the most important problems regarding the stability of Both techniques have their advantages and drawbacks, and can
teleoperation systems. Time delay is specially relevant in be considered as complementary. When fully available, super-
ground teleoperation of robots orbiting Earth, with values of visory control will prove very valuable, but current state of
round trip delay ranging from 5 to 10 seconds. technology does not allow to fully implement it.
Many proposals on how to conduct time delayed teleoperation On the other hand, for time-delayed manual teleoperation
under various circumstances and for different applications there have been a great amount of research and many applica-
have appeared through the years in the literature. This paper tions are known. Understanding all the concepts, ideas and
presents a survey of most of these proposals, with the aim of results of this research will help to develop more intelligent
identifying those methods and procedures that can be em- and autonomous systems in which supervisory control will
ployed to improve the performance of space robot teleopera- take the major part.
tion. Moreover, we believe continuous manual teleoperation will
keep its role over the years even with the appearance of sys-
1. INTRODUCTION tems with more autonomous capabilities. Any telerobotic
Space robot systems and on-orbit telerobotics technology will should be first equipped with an effective support system for
play an essential role in the construction and maintenance of direct teleoperation, and then the analytically determined part
large-scale structures, such as the International Space Station or the part of the operation that can be carried-out autono-
(ISS). It is well known that the cycle time (round-trip delay) mously should be replaced by program control.
for systems in LEO is at least of 0.5 s. These values are further With so abundant information in the literature about manual
extended due to the time consumed in data processing by relay teleoperation with time-delay we decided to perform a survey
satellites and computers on different intermediate stations. in order to get a global view of the present state of technology.
Values can vary from 5-10 seconds. The survey has later served us to identify those techniques
From classic control theory is easy to derive that a delay in a prone to be useful for the ground manual teleoperation of
control loop is an important cause of instability. A pure delay robots orbiting Earth, which is the final aim of our work.
decreases the phase of the system in a factor equal to the There are other sources where to find summarised information
product between the frequency and the time-delay value. Also, on techniques for time-delayed teleoperation of space robots
with the presence of a pure delay, as the static gain increases Ref 41. is a very recommended and well-documented study by
the system deviates rapidly from a stable condition. Prof. Sheridan, although it focuses very much on predictors
It was in the sixties (Ref. 12. ) when first became apparent that displays and supervisory control instead on manual teleopera-
the existence of time delay in the communication link between tion. Also, although it contains the foundations, many new
the local and the remote zone is one of the more important methods and techniques have been proposed since its publi-
problems regarding the stability of teleoperation systems. cation. Ref 42. is also another good source of information,
Ferrel conducted in 1962 the first experiments with an unilat- although no systematic analysis or comprehensive study of the
eral system under time delay in the visual feedback. The different alternatives available is presented.
‘move-and-wait’ strategy was first conceived and employed as Ref 16. is a good and systematic source of information re-
a solution to overcome instability. Others e xperiments carried- garding application of telerobotic technology for space robots,
out by Ferrel and co-workers showed that even with time whereas Ref. 35. Presents a detailed classification of the types
delays of 0.3 s a human operator could not maintain sensor- of space robots available along with a thorough analysis of the
motor coordination during manual teleoperation. constraints present for the application of robots in space
Since then, many other proposals to overcome time-delay The outline of the rest of the paper is as follows. First, to have
have appeared in the literature. They have been conceived for a better understanding of the problem, section 2 is devoted to
almost all telerobotic applications (space, submarine, internet, give a brief overview of the main features involved in space
tele-surgery, etc., Ref. 40.) and to function under various robot teleoperation. Section 3 contains the description of a
working conditions. framework that will serve to better understand the classifica-
Proposals can be divided into two different types of ap- tion of time-delayed teleoperation techniques later presented
proaches: those based on the more traditional manual teleop- in section 4. Finally, the analysis of each method with respect
eration approach and those based on the supervisory control to its use in space telerobotics is performed in section 5.
2. OVERVIEW OF SPACE ROBOTS TELEOPERATION namic model is very difficult, especially during contact tasks.
First, we have to consider the special conditions in which Also, space manipulators move very slowly, so its dynamic
space teleoperation takes place and which makes it a unique features can be neglected (except when dealing with free-
problem. It will later help to study the different techniques flying robots).
from the perspective of the constraints imposed in the use of Finally, a third classification of predictive displays is: a) with
robots in space. prediction of contact forces and b) without prediction of
Ref 35. presents a very complete and comprehensive descrip- forces. Both options are available, and although some applica-
tion of typical requirements and constraints present in space tions are known (Ref 23. ), the prediction of contact forces is
robotics applications. For our analysis we will consider the extremely difficult and complex, specially for space environ-
following: ments where drastic changes in operating conditions take
place.
§ Round trip delay of 5-10 seconds. Time delay is mainly
caused by data buffering and processing on relay stations. 2.2 Prediction techniques
§ Low communication bandwidth, which can severely limit There are two basic types of prediction techniques: a) those
the transmission of video images. based on the extrapolation of a Taylor-series upon current
§ Space manipulators tend to be very light and flexible, state and derivatives and b) those based on the use of model.
which make their control more difficult than their terres- The second type is the most common one, since the first type
trial counterparts is only valid for short time predictions. In the second case,
§ Micro-gravity, vacuum and thermal conditions degrade some authors propose the use of estimation to have a fairly
the manipulator performance. good knowledge of the current state of the robot and the envi-
ronment. But this theory needs to make use of a very accurate
If any of this four constraints is very difficult to be tackled knowledge of the dynamic behaviour of both. The dynamic
independently, much more difficult will be if all of them come model of the robot is rather easy to obtain, as well as, for
together. To cope with them, a set of tools or techniques, such example, the dynamics of a free-floating object.
as predictive displays, have been developed and are used in
almost all time-delayed space teleoperation applications. On the other hand, it is practically not feasible to employ
closed-loop estimation theory during contact due to the com-
We will briefly review them along with other common devices plex, discrete, not-linear dynamic interactions that make the
present in manual teleoperation systems, such as input de- synthesis of a adequate model very complex, if feasible at all.
vices. Some guidelines are given.
2.2 Input devices
2.1 Predictive displays
Two features have to be considered: a) the type of input device
Sheridan proposes (Ref 41.) that “when there is a significant properly (master arm, joystick, etc.) and b) the control mode to
delay (say more than 0.5 s) and operators movements are employ (position, velocity). Interaction between the two fe a-
relatively slow, say mostly below 1 Hz, a predictive display tures need to be investigated.
can be very useful.”
There are two types of control modes to be considered:
Predictive displays show a model of the environment and of
the slave manipulator. The operator performs the task on the § Position control: it is very intuitive, since the position of
display moving the master arm without any time delay. The the manipulator corresponds to the position of the input
inputs of the master arm or the virtual slave positions are sent device. It has the disadvantage that may need to be in-
to the remote slave, which executes them. dexed for large or precise manipulator motions.
In practice, perfect modelling is impossible. If it were possi- § Rate control: rate control is used preferably when the
ble, the Roseborough Dilemma would appear (Ref 37.), that difference between the working envelope of the input de-
is, if we have a perfect model, why teleoperate? So, predictive vice and the manipulator is very large. It is less intuitive
displays have to be considered just as a tool that reduces the than position control but it can be more comfortable and
amount of information and on-line mental modelling that the allows to achieve better controllability for simple tasks.
operator has to do. It helps bridge the time gap, offering ap- A detailed study comparing position and rate control can be
proximate cues until the actual information is available. The found in Ref 17. and some practical experiments comparing
difference between the real and modelled environment has to them using a master arm are available in Ref 9.
be coped in real time by the remote slave with the use of some Three types of input devices can be considered:
local autonomy.
§ Master arm: the use of master arms is very intuitive. 6
Two main basic types of predictive displays are available: a) DOF can be used on a single grip. It can be tiring for
those overlaying delayed video and predicted graphics (Ref 3. ) slow movements and difficult to operate for precise posi-
and b) those using only predicted graphics, with the video tioning.
signal in a different display. The latter one is by far more
§ Joysticks : are less intuitive and two joysticks are needed
common due to the difficulty of mixing video and graphics
for 6 DOF. They are very good for precise positioning
with enough quality and robustness.
and the operator does not get tired.
In another classification we can find: a) predictive dynamic
simulators and b) kinematic only simulators. To have a dy- § Space mouse: force input device is not intuitive, but can
integrate a 6 DOF in a single device. It can be used intui-
namic model of robot and environment will sure add more
tively to give force commands during contact.
quality to the prediction, but to count with an adequate dy-
Rate control with a master arm is very difficult (Ref. 9. ) and Direct force reflection, that is, pure bilateral schemes (Ref 33.)
position control is advantageous only when doing dextrous cannot be used with time delays over 2 seconds (see section
tasks at rather high velocity. Not very recommended for space 5). But there are other ways to present the operator with the
teleoperation. It is also very complex to give position com- force exerted by the slave during contact. The simplest one is
mands with the space mouse. It should be use for rate control to show the force values on a visual display. Another option is
or force control during contact. indirect force reflection, in which the delayed force is fed-
Joysticks can be used either way, but rate control seems more back to the hand which does not take part in the command.
intuitive due to the small working range of joysticks joints and A more complex but better solution is to simulate the interac-
the ability to stop the robot very precisely on a given point. tion between robot and environment in a predictive simulator
Table 1 presents a comparison of different input devices for and by this way predict the contact force which can be fed
space teleoperation. It considers the master arm with position back to the master arm (Ref 23.). It is complex solution, as
control and the joystick and the space mouse with velocity explained when talking about predictive displays.
control. A similar table, constructed during the design of the Although less extended, force reflection can be used not only
ETS-7 teleoperation system, comparing a 6 DOF master arm to reflect interaction force to the operator but also to help him
and two 3-DOF joysticks is available in Ref 31. Other very know how the task is being performed. This is known as re -
exhaustive comparison between different control modes and flection of virtual forces or synthetic fixtures (Ref 39.), that is,
the use of one or two-handed displays appears in Ref. 6. the use of forces that do not represent a real force to convey
information to the operator information about the task (Ref
Table 1 Comparison of input devices for space teleoperation 34. )
Master arm Joystick Space mouse 3. FRAMEWORK OF ANALYSIS
Easiness of operation ++ + - There are a wide variety of proposals and ideas on how to
Large arm motion - ++ ++ overcome time-delay. The approach and nomenclature differ
Input precision - + - and sometimes is difficult for the researcher to compare them
Operator comfort - ++ + even from a high level point of view.

2.3 Task description

Master A B
It can be very relevant to have a priori information from the
task so that the system can interpret correctly the operator’s Slave
actions. This information can be as simple as specifying dif-
Operator E D C
ferent possible states of the task through time and the condi-
tions that make the system change from one state to another
(Ref 7. ). It can also comprise a complex set of conditions and
Predictive
information about the evolution of sensor data, interactions, simulator With FR
etc. (Ref 15.). Display
But on the other side, to try to define perfectly in advance the Video
task leads us again to the Roseborough Dilemma (Ref 37.). If display
we can do it, why teleoperate? So, the concept again is to give
some help, simple and useful enough so the operator can Figure 1 General scheme for time-delayed teleoperation
perform the task on its own.
2.4 Local autonomy loop For the purpose of our survey we constructed the diagram
present in Figure 1. It can be demonstrated that almost all
Almost all proposals present in the literature use some sort of proposals present in the literature can be described through
compliance feature on the slave robot. Compliance is useful to this diagram.
cope with the error caused by an imperfect model. It can re-
duce execution time and overall forces applied upon the envi- It is made of a set of blocks and data flowing from one block
ronment. The only problem is that it consists of an automatic to another. This data is made of a set of given magnitudes
remote feature and the operator can get confused if not fully (position, velo city, force, etc.) in a n-DOF domain.
aware of its behaviour. Operator, master, slave and visual displays are represented
Refer to Ref 18. for the detailed analysis of the advantages and through the corresponding blocks. Blocks A and C represent
application of active compliance local loops. A force/torque the pre-processing of signals before their transmission through
sensor is required on the robot’s wrist to implement an active the communication link. Block B represents the local auton-
compliance. The force sensed can also be sent back to the omy loop of the slave robot. Block D works as a hub on the
local zone in order to monitor its value or to reflect it to the master side, combining and distributing information. Finally,
operator through an input device. block E is responsible for implementing the FR on the master
arm.
2.5 Force reflection The predictive simulator block represents a model-based
As pointed out by Hannaford (Ref 14.), to supply the operator simulator, which can be employed with its display to imple-
with some type of force sensation is essential for the perform- ment a predictive scheme. The simulator can include the
ance of a teleoperation system. simulation of contact forces .
4. CLASSIFICATION OF METHODS 4.2.1 Classification as a function of FR capabilities
Classification of teleoperation techniques is not an easy task. To have a non-bilateral system does not mean that there is no
There are a great variety of methods and combinations, and it force reflection (FR). It only means that coupling between the
is difficult to assign a particular label to each one. master and slave takes place only in one direction. There can
One classification could be based on the characteristics of the be non-bilateral systems with FR.
time delay (amount, variable or fixed, etc.) for which has been There are two basic types of non-bilateral FR systems:
designed or to classify methods regarding its preferred appli- § Virtual forces FR: the operator senses in the commanding
cation (internet, space, underwater). Neither of both classifi- hand some type of kinaesthetic cue which is not directly
cations say much about each technique itself. related to the contact force being generated by the slave.
We finally decided to first distinguish techniques that attempt They can be based on a model or simply used to display
to have bilateral coupling between master and slave from other kind info rmation.
those which not. The objectives are different and both group § Indirect FR: to directly sense the contact force in a the
of methods should not be compared on the same grounds. passive hand, that is, the hand that is not generating the
Afterwards, further classification regarding the control meth- command.
ods employed is carried-out for each group of techniques.
4.1 Bilateral systems Master A B
Bilateral systems are those in which master and slave are
Slave
directly coupled in position (velocity) and force through any
of the so called bilateral control schemes: position-position Operator E D
(common error or symmetric position servo), force-position
(force feedback), force-force (force servo-position or force-
feedback servo). Predictive
simulator With FR
Display
Master A B

Slave Figure 3 Teleprogramming with or without FR

Operator E D C
4.2.2 Classification as a function of command generation
How commands are generated is an important feature that
Video distinguish different methods. We have identified two differ-
display ent types of systems, somewhat in relation with the use or not
of the time/position clutching concept proposed by Conway
Figure 2 Bilateral manual teleoperation technique (Ref 8. ).
§ Tele-programming: it consists in performing the manual
It is the objective of bilateral systems that the operator feels teleoperation task in a simulator before real operation to
directly on its commanding hand the contact force of the slave. gather data of how the task must be carried-out. After-
Classical bilateral schemes are very unstable under time-delay wards, high level commands are sent to the slave to du-
(Ref 49.). To solve the instability two basic approaches on the plicate the actions on the real environment. (Figure 3)
design of bilateral systems under time delay are available: § Predictive techniques : those techniques that employ a
§ Based on the passivity theory : the teleoperation system is predictive simulator in which the operator carries out the
represented as a two-port device using the mechanical- task interactively, while its commands are being sent in
electrical analogy. The design is based on passivity the- real time to the remote slave for execution (Figure 1).
ory which states that a system is stable if always has to
dissipate and never increment its total energy. Special 5. ANALYSIS
mathematics (scattering theory) is available for passivity This section is devoted to analyse different techniques pre-
applied to two-port devices, using also tools of electrical sented by the robotics community through the years to tackle
network theory. time-delay in teleoperation. We will follow the classification
carried-out in the preceding section. In the case of non-
§ Based on control theory : is a more classical approach. A
linear model of each element is proposed and block dia- bilateral systems we have followed the division regarding the
grams are constructed. System stability, performance, etc. use or not of FR.
are studied with classical and advanced control tools. Tables summarising the experimental conditions in which
each technique has been tested have been constructed. The
Figure 2 presents the scheme in which to develop bilateral
data was extracted from the literature. The tables compare the
systems for time-delayed teleoperation.
number of degrees of freedom (DOF), the frequency of trans -
4.2 Non-bilateral systems mission between the local and the remote zone, the type of
Non-bilateral systems can be classified attending to two dif- master (S simulation, J Joystick, A master arm, M space
ferent factors: the presence or not of force reflection, and the mouse and its control (p position control, v velocity control),
mechanism by which commands are generated. the type of slave (S simulation, A slave arm, V vehicle), the
type of task carried out, and the maximum time delay between Table 2 Bilateral schemes based on passivity
the local and remote zone for which the system remained
operational. Ref Delay DOF Master Freq. Task
(s) Slave (Hz)
5.1. Bilateral systems based on passivity theory
1. 2 1 Sp / S 500 Hard contact
The passivity theory was first applied to the study of bilateral
system by Raju (Ref 36.) and was quickly adopted by almost 27. 1 1 - - Hard contact
every researcher in the field. 49. 0.03 1 Ap / A - Hard contact
The first application of passivity theory for time-delayed 22. 0.7 1 Ap / A 500 Hard contact
systems was performed by Anderson and Spong (Ref 1. ) for 1
5.2 Bilateral systems based on control theory
DOF, and was later extended for n-DOF, including models of
the operator and the environment (Ref 2. ). The proposed Another approach to bilateral system is the use of classical
scheme effectively dissipates energy, imitating waves along control theory, as detailed in Ref 33. In this sense, Eusebi
electrical networks, and theoretically works for any amount of presented in Ref 10. the definition of a framework for stability
time delay. This approach is very effective for short time delay analysis based on existing results in the area of linear time-
but suffers from poor tracking and loss of intuitiveness for delay dynamics systems.
longer delays (Ref 24.). One of the most significant results is the telemonitoring con-
The preceding technique was improved by Niemeyer and cept proposed by Lee (Ref 25.). It is based on the idea that it
Slotine in their seminal work about wave variables (Ref 27. ). is of no use obtaining a system theoretically stable for any
Basically, wave variables are a new way of expressing the time delay, but to focus on some specific practical value and a
exchange of energy (force and velocity) of a system with its particular application. Another idea is that the operator should
environment. Hence, their idea was to transmit wave variables have a precise knowledge of the slave performance (like com-
through the communication link instead of traditional magni- pliance).
tudes such as position or force. It was later proved (Ref 28.) After constructing an impedance control scheme on both
that passivity is maintained just by transmitting wave vari- master and slave, the authors come out with a scheme that
ables, whatever the delay may be. very much resembles bilateral position-position control with
An advanced use of wave variables for time delayed systems remote compliance and the addition of a component for force
is proposed in Ref 29. It makes use of what is called a virtual monitoring. Successful experiments with limited simulations
tool. The idea is to hide from the operator the system dynam- were carried out under time delays of 2 s.
ics. Also, Yokokohji (Ref 48.) made use of wave variables to Bilateral control for time delay using virtual internal models
minimise the performance degradation due to the possible (VIM) was proposed by Otsuka (Ref 32. ). It is based on the
fluctuation of time-delay. premises that instability caused by time delay is basically due
A bilateral scheme that achieves an ideal kinaesthetic coupling to transmitting position error, that teleoperated systems must
between master and slave with time delay was presented by have a good performance even with low transmission band-
Yoshikawa (Ref 49.). It is also derived from the passivity width and that the slave should have a local autonomous loop.
formalism. The resulting control law basically tries to cancel The proposal makes use of a VIM (virtual mass associated to
the dynamics of master and slave. It also makes use of a the end of a manipulator to specify its compliant motions) on
weighting function for the acting forces and the position error. each manipulator. Slave and master arms are moved by ap-
Another of their results is that transmission of position seems plying upon its VIM the force sensed on the other one. Hence,
the major driving force against stability. The experiments only force information is transmitted between local and re -
carried out by the authors were conducted under 30 ms of time mote zones.
delay. Lately there has been many research in the area of Internet
Another solution employing passivity theory, devised for teleoperation. Variable time-delays and loss of data are two
internet teleoperation was proposed by Kosuge (Ref 21. ). It important problems in this application.
tackles the varying time-delay by establishing an upper bound. One example is the proposal made by Oboe and Fiorini (Ref
The approach is later extended (Ref 22.) to when time-delay 30. ). The bilateral control scheme they proposed is rather basic
varies depending on the transmission direction. In Ref 19. , this (position-position with no compliance) but their idea is to
bilateral systems is combined with a predictive display. continually probe the network to update the control parameters
5.1.1 Application for space teleoperation depending on its current behaviour. They use the space-state
Table 2 presents a summary of the bilateral schemes based on internal representation due to the non-linear behaviour of the
passivity. Some conclusions can be derived about their suit- system. Their experiments under 0.5 s are carried-out in 1
ability for space systems. First, no out-of-laboratory experi- DOF and using a virtual slave, which make them rather lim-
ments are available. All tasks demonstrated are only 1 DOF ited.
tasks and very limited in its complexity (only hard contact). Leung et al (Ref 26.) were the first to use H∞ control theory
Finally the maximum delays that these systems can sustain are and µ-analysis for time-delayed teleoperation. They modelled
very low, usually under 1 second. It is clear that under current the time delay as a perturbation and designed the system to be
circumstances they are not suitable for ground teleoperation of robust to such perturbation by using µ-synthesis. Their results
space robots. have been extended by Sano (Ref 38. ) for varying time-delay
systems making use of the framework of gain scheduling..
Table 3 Bilateral schemes based on control theory than simply display the predicted state of the remote robot.
They use of predictive observer that combines the delayed
Ref Delay DOF Master Freq. Task state and the command that is currently being sent to the robot.
(s) Slave (Hz) Instead of using classical trajectory generators between points,
25. 1.4 1 Jp / V - Hard contact the authors use what is called an event/references generator. It
works basically generating trajectories as a function of the
32. 0.5 6 Ap / A 5 Basic contact sensor data and not as a function of time. The approach is later
30. 0.32 2 Ap / S 350 Basic contact extended for internet teleoperation (Ref 4. ).
38. 0.4-0.8 1 Jp / A 410 Hard contact Wakabayashi and Matsumoto (Ref 46. ) developed in the
5.2.1 Application for space teleoperation frame of the ETS-7 project, a visual aid system for direct
teleoperation applied for a truss deployment experiment. This
Table 3 presents a summary of the bilateral schemes based on aid system does not depend on a designed model of the work-
control theory. Same conclusions as for the passivity based place. It introduces the predictive force concept to calculate
systems apply. Maximum delay is under 1 s and only simple the appropriate joystick input, and displays it to the operator in
tasks under laboratory conditions have been demonstrated. We the joystick coordinate system to enable the operator easily
can affirm that bilateral control is not practically applicable for follow the direction. This method is later extended to auto-
space robot teleoperation. matic programming to make an efficient teleoperation system
that combines direct teleoperation and program control.
5.3 Non-bilateral systems without force reflection
Breedveld (Ref 5. ) developed the concept of on-line set point
Here we will briefly review the ideas that do not make use of
displays, in contrast to tele-programming which are off-line
force as an input to the operator. It includes both concepts set-point or preview displays. He developed afterwards two
based on the tele-programming technique and concepts based types of set-point displays: indicator and pyramids displays,
on predictive displays. which were compared under time delay with the ERA (Ref 6. )
The generic concept of tele-automation was presented by
Conway (Ref 8. ). Teleautomation should be considered as a Table 4 Non-Bilateral schemes without force reflection
framework in which to develop different schemes. It is based
on a kinematic predictive simulator with time and position Ref Delay DOF Master Freq. Task
(s) Slave (Hz)
clutching capabilities. Time clutching means that the timing
between when the operator does the task in the predictive 8. 4 2 Mp/ A 60 Fine positioning
simulator and when is performed by the remote robot does not 15. 7 6 Mv / A - ORU exchange
need to be the same. The operator can go faster when the task
43. 1.5-7 6 Jv / A 2 Avoid collision
is easy and slower when is difficult. The remote robot will
execute the commands in a pre-specified manner. 6. 3 6 Mv / A - Fine positioning
Position clutching means that at some point the task being 46. 5-7 6 Jv / A 4/10 Deploy truss
done by the operator using a predictive display is not sent 5.3.1 Application for space teleoperation
immediately to the remote robot. Instead, the operator can try
different approaches and when the generated path is good Table 5 presents a summary of non-bilateral techniques that
enough download the data to the remote site. does not make any use of force reflection. First, we have to
The concept of tele-sensor-programming, developed in the emphasise that two of the methods proposed have been tested
frame of the ROTEX project (Ref 15.), should be considered in the two most renowned projects of space robotics: ROTEX
as one of the most ambitious proposals to cope with time- and ETS-7, an IVA robot and a free-flying robot.
delay. Tele-sensor-programming was successfully employed This demonstrates that prediction techniques and predictive
for real space robot teleoperation under 5-7 seconds of time displays are a inestimable aid for this kind of applications and
delay. should be considered as the baseline from which begin to
The idea is to use a predictive simulator but also to have a build more complex architectures. Velocity control, either
certain degree of autonomy in the remote zone through the use with joystick or space mouse seems also very adequate, given
of several sensors (force, proximity, contact, etc.). The predic- the characteristics of present space teleoperation systems.
tive simulator also models the behaviour of the sensors and
5.3 Non-bilateral systems with force reflection
how the slave makes use of them to acquire a certain degree of
autonomy. The operator, hence, only commands the gross Both tele-programming and predictive techniques can be
motion of the slave while it is helped for detail movements by found in this section.
the automatic corrections made by the system using the data 5.3.1 Predictive techniques
provided by the sensors. The trajectory information sent to the
remote robot is relative to the environment and includes sen- The use of a predictive operator aid with FR was first pro-
sor’s data patterns. It is executed by the remote robot with the posed and studied in detail by Buzan (Ref 7. ). Buzan demon-
use of the real data from its sensors and its own autonomy strated that it is not always possible to use a closed loop pre-
functions. dictor that uses both the information of the command and the
delayed data from the remote zone. The reason lies mainly in
Tarn and Brady (Ref 43.) propose a closed-loop approach to the non-linearity of the different states related to the task
the predictive control of time-delayed systems. It does more execution. He then proposed the use of an open-loop predictor,
but pointing out that it cannot work on its own, since there are To be able to identify more easily the type of contact that
always modelling errors and depth and interaction cues are not takes place it is essential to rely on a priori knowledge of the
available to the operator task. The robot must have some adaptation capacity by em-
Buzan proposed four different methods for reflecting force to ploying sensor information.
the operator: indirect, predictive, complementary and dual. The experiments carried-out by the authors consisted in fo l-
Indirect force reflection means reflecting the delayed force in lowing the contour of a box with delays up to 3 s. Problems
the hand that is not controlling the task. Predictive force re- were found due to not modelling in detail the static and dy-
flection feeds back to the operator only the force obtained namic features of slave-environment interaction.
from the predictor. Complementary force reflection combines
Table 5 Non-Bilateral schemes with force reflection
predicted and delayed force through two complementary
filters. And finally, dual force reflection makes use of the
Ref Delay DOF Master Freq. Task
indirect and predictive methods simultaneously. (s) (Hz)
Slave
The main conclusions drawn from the study were that predic-
tive simulators are always very useful with or without force 13. 3 6 Ap / A 30 Following box
reflection. When low visibility is available the force predictor 7. 2-4 1 Jp / S 15 Fitting
working in open loop is very important. The use of the dual 44. 5 6 Av / A - Open door
force reflection is possible but needs special training from the 34. 7 3 Jp / A 4/10 Space assembly
operator. The results obtained with complementary force
reflection were not as good as expected. 5.3.3 Application for space teleoperation
Another proposal regarding the use of predictive simulation Table 5 presents a summary of the application of non-bilateral
including force reflection was presented by Tsumaki (Ref 44. ). techniques that use FR in some way or another. It is seen that
Its main contribution is the development of an algorithm that relatively complex tasks involving contact, with several DOF
tolerates geometric errors between the model and the envi- and under several (up to 5-7) of time delay are demonstrated
ronment, and that was later extended to tolerate dynamic with the use of FR. Even a real space application that makes
errors (Ref. 45.). They use the principle of the optimum ap- use of FR is available (Ref 34.).
proach velocity (Ref 20.). Various space teleoperation like FR increments the operator telepresence. The operator is
experiments (ORU exchange and opening/closing doors) were coupled with the environment, hence, the task is carried-out
carried-out with a time delay of up to 5 seconds. more smoothly. It also decrease the operator’s mental over-
The force sensed on the master is used for rate or force control load. The operator can allocate his attention to other displays
of both the real and virtual slave. Rate control is used when while being guided by the FR hand controller.
there is no interaction with the environment. When contact is FR can also be used to simulate computer control, maintaining
detected the control mode is changed automatically to force the operator in the loop and keeping the advantages of both
control selecting as force reference the force applied by the methodologies.
operator upon the master. The change of mode is done inde-
FR can be used to help the operator to monitor intuitive and
pendently for the real and virtual robots depending on when
actively when he is doing something wrong or dangerous, and
contact is detected. By this manner the geometric errors of the
finally FR hand controllers can be used as tools to display
virtual environment model are avoided.
different type of information.
Finally and in the frame of the ETS-7 project, FR was demon-
It is necessary to remark that FR is not a solution on its own,
strated to be a very valuable tool to display guiding informa-
ant that it should be combined with other techniques, such as
tion to the operator (Ref 34. ). Experiments carried-out under 7
teleprogramming or predictive displays to make use of all its
s of time-delay demonstrated the use of synthetic fixtures for
advantages, as seen in the previous sections.
carrying complex tasks, and the usefulness of limited indirect
and complementary FR. 6. CONCLUSIONS
Finally, a major result was the use of FR to display the opera- Teleoperation with time delay is a very challenging problem
tor the outcome of a prediction algorithm that does no use any and a very active field of research. Application to space ro-
kind of model. Experiments with position and velocity control botics it is even more demanding due to the high value of time
were carried out. A detailed comparison can be found in Ref delay, limited communication bandwidth and poor control
47. capabilities of space robots.
5.3.2 Tele-programming techniques After the survey of current state-of-art technology for time-
A paradigmatic example of tele-programming with FR is the delay teleoperation, we can affirm that there exist many pro-
concept developed by Funda (Ref 13.). The task is performed posals prone to be successfully employed for space robots
first using a simulator with force-reflecting capabilities or teleoperation, although most of them have only been tested
upon a simulated environment itself. All the information under laboratory conditions. Only a few practical demonstra-
gathered from how the operator performs the task (position tions can be found.
and force, events, etc.) is translated into high level robot More research is needed to be able to translate simulated
commands and sent to be executed by the remote slave. The results to implementations closer to space systems, were real
commands must be of symbolic nature and must consider the problems arise. Moreover, no sole technique is good enough
unavoidable discrepancies between model and reality. on it own, an approach that combines the best of each one is
necessary.
 7. REFERENCES 26. G.M.H Leung et al, “Bilateral Controller for Teleoperators with Time Delay
via µ-synthesis ”, IEEE Trans. on Robotics & Automation , Vol. 11, No.1,
1. R.J. Anderson y M.W. Spong, “Bilateral Control of Teleoperators with Time 1995.
Delay”, IEEE Transactions on Automatic Control , Vol. 34, No. 5, 1989.
27. G. Niemeyer and J.E. Slotine, “Stable Adaptive Teleoperation”, IEEE
2. R.J. Anderson y M.W. Spong, “Asymptotic Stability for Force Reflecting
Journal of Oceanic Engineering, Vol. 16, No. 1, January 1991.
Teleoperators with Time Delay”, The International Journal of Robotics R e-
search, Vol. 11, No. 2, April 1992. 28. G. Niemeyer and J.E. Slotine, “Using Wave Variables for System Analysis
and Robot Control”, IEEE Proc. International Conference on Robotics and
3. A.K. Bejczy, W.S. Kim and S.C. Venema, “The Phamtom Robot: Predictive
Automation, April 1997.
Displays for Teleoperation with Time Delay”, Proc. of IEEE Int. Conf. on
Robotics and Aut omation, 1990 29. G. Niemeyer and J.E. Slotine, “Designing Force Reflecting Teleoperators
with Large time Delays to Appear as Virtual Tools”, IEEE Proc. Interna-
4. K. Brady and T. Tarn, “Internet-Based Remote Teleoperation”, Proc. IEEE
tional Conference on Robotics and Automation, April 1997.
International Conference on Robotics and Automation, Leuven, 1998.
30. R. Oboe and P.Fiorini, “A Design and Control Environment for Internet-
5. P. Breedveld, “The Design of a Man-Machine Interface for a Space Manipu- Based Telerobotics”, The International Journal of Robotics Research,
lator”, Ph.D. Thesis, Technical University of Delft, 1996.
Vol.17, No. 4, April 1998.
6. E. Buiël, “Design and Evaluation of a Human-Machine Interface for a 31. M. Oda, “System Engineering Approach in Designing the Teleoperation
Teleoperated Space Manipulator”, Ph.D. Thesis, Technical University of System of the ETS-VII Robot Experiment Satellite”, International Confer-
Delft, 1998. ence on Robotics and Automation, ICRA’97, New Mexico, April 1997.
7. F.T. Buzan, “Control of Telemanipulators with Time Delay: a Predictive 32. M. Otsuka et al, “Bilateral Telemanipulator System with Communication
Operator Aid with Force Feedback”, Ph.D. Thesis, MIT, Boston, 1989. Time Delay Based on Force-Sum-Driven Virtual Internal Models”, IEEE
8. L. Conway, R. Volz and M.W. Walker, “Teleautonomous Systems: Project- Int. Conf. On Robotics and Automation, 1995.
ing and Coordinating Intelligent Action at a Distance”, IEEE Trans. on Ro- 33. L.F. Peñín & al, “Design Fundamentals of Master-Slave Systems with a
botics and Automation, Vol. 6, No. 2, April 1990. Force-Position Bilateral Control Scheme”, 5 th Symposium on Robot Co n-
trol (SYROCO’97), Nantes, France, September 1997.
9. H. Das et al, “Operator Performance with Alternative Manual Control Modes
in Teleoperation”, Presence, Vol. 1, No. 2, September 1992. 34. L.F. Peñín, K. Matsumoto and S. Wakabayashi, “Application of Force
Reflection for Time-delayed Teleoperation of Space Robots”, IEEE Ro-
10. A. Eusebi and C. Melchiorri, “Stability and Control Isues for Telemanipu- botics & Automation Magazine, December 2000.
lation Systems with Time-Delays”, IFAC’96 World Congress, 1996.
35. P. Putz, “Space Robotics in Europe: A Survey”, Robotics and Autonomous
11. W.R. Ferrel y T.B. Sheridan, “Supervisory control of remote manipulation”,
Sytems 23 (1998) 3-16.
IEEE Spectrum, October 1967.
36. Raju, G.J., G. Verghese and T.B. Sheridan, (1989) “Design issues in 2-port
12. W.R. Ferrel, “Delayed Force Feedback”, Human Factors , October 1966. network models of bilateral remote manipulation”, Proceedings of IEEE
13. J. Funda, T.S. Lindsay and R.P. Paul, “Teleprogramming: Toward Delay- Int. Conf. on Robotics and Automation , Scottsdale
Invariant Remote Manipulation”, Presence, Vol. 1, No. 1, 1992. 37. J. Roseborough, “Aiding Human Operators with State Estimates”, Ph.D.
14. B. Hannaford et al, “Performance Evaluation of a Six-Axis Generalized Thesis, MIT, Boston, 1988
Force-Reflecting Teleoperator”, IEEE Trans. on Systems, Man and Cyber- 38. A. Sano et al, “Network-based Force-Reflecting Teleoperation”, Proceed-
netics, Vol. 21, No. 3, May/June 1991. ings of IEEE Int. Conf. on Robotics and Automation, 2000.
15. G. Hirzinger et al “Sensor-Based Space Robotics –ROTEX and Its Telero- 39. C. Sayers, Re mote Control Robotics, Springer, New York, 1999.
botic Features”, IEEE Trans. on Robotics and Automation, Vol. 9, No. 5,
October 1993. 40. T.B. Sheridan, Telerobotics and Human Supervisory Control, The MIT
Press, 1992.
16. G. Hirzinger, “Robots in space – a survey”, Advanced Robotics, Vol. 9, No.
6, 1995 41. T.B. Sheridan, “Space Teleoperation Through Time Delay: Review and
Prognosis”, IEEE Trans. on Robotics and Automation , Vol. 9, No. 5, 1993.
17. W.S. Kim et al, “A Comparison of Position and Rate Control for Tele -
manipulations with Consideration of Manipulator System Dynamics”, 42. S.B Skaar and C.F.Ruoff, Editors, Teleoperation and Robotics in Space,
IEEE Journal of Robotics and Automation, Vol. 3, No. 5, October 1987. American Institute of Aeronautics and Astronautics, 1994
18. W.S. Kim, B. Hannaford, A.K. Bejczy, “Force-Reflection and Shared 43. T. Tarn and K. Brady, “A Framework for the Control of Time-Delayed
Compliant Control in Operating Telemanipulators with Time Delay”, IEEE Telerobotic Systems”, 5 th IFAC Symposium on Robot Control, SY-
Transactions on Robotics and Automation, Vol. 8, No. 2, April, 1992. ROCO’97, Nantes, France, 1997.
19. Kikuchi et al, “Teleoperation System via Computer Network”, Proc. IEEE 44. Y. Tsumaki, Y. Hoshi, H. Naruse and M. Uchiyama, “Virtual Reality Based
International Conference on Robotics and Automation, Leuven, 1998. Teleoperation which Tolerates Geometrical Modeling Errors”, Proc. Int.
Conference o Intelligent Robotics Systems, IROS, 1996.
20. K. Kitagaki and M. Uchiyama, “Optimal Approach Velocity of an End-
Effector to the Environment”, Advanced Robotics, Vol. 8, No. 2, 1994. 45. Y. Tsumaki, et al, “A Model-Based Space Teleoperation System with
Robustness against Modelling Errors”, ”, International Conference on Ro-
21. K. Kosuge, H. Murayama and K. Takeo, “Bilateral Feedback Control of botics and Automation, ICRA’97, New Mexico, April 1997.
Telemanipulators via Computer Network”, Proc. Int. Conference o Intelli-
gent Robotics Systems, IROS, 1996. 46. S. Wakabayashi and K. Matsumoto, “Generalized Visual Aid for Direct
Teleoperation Applied to ETS-7 Truss Manipulation Experiment”, Inter-
22. K. Kosuge and H. Murayama, “Bilateral Feedback Control of Telemanipu- national Symposium on Artificial Intelligence, Robotics and Automation in
lator via Computer Network in Discrete Time Domain”, Proc. Interna-
Space, i-SAIRAS’99, July, 1999
tional Conference on Robotics and Automation, April 1997.
23. Kotoku et al, “Environment Modeling for the Interactive Display (EMID) 47. S. Wakabayashi, K. Matsumoto and L.F. Peñin, “Results of the ETS-7
Teleoperation Experiment using FR Josytick”, Int. Symposium on Space
used in Telerobotics Systems”, IEEE/RJS IROS’91, Osaka, Japan. Technology and Science, Japan, 2000.
24. C.A. Lawn and B. Hannaford, “Performance Testing of Passive Communi-
cation and Control in Teleoperation with Time Delay”, IEEE International 48. Y. Yokokohji et al., “Bilateral Control with Energy Balance Monitoring
Conference on Robotics and Automation, 1993 Under Time-Varying Communication Delay”, IEEE International Confer-
ence on Robotics and Automation, San Francisco, 2000.
25. S. Lee y H.S. Lee, “Modeling, Design, and Evaluation of Advanced Teleop-
erator Control Systems with Short Time Delay”, IEEE Transactions on Ro- 49. T. Yoshikawa and J. Ueda, “Analysis and Control of Master-Slave Systems
botics and Automation, Vol. 9, No. 5, Octubre 1993. with Time Delay”, Proc. Int. Conference o Intelligent Robotics Systems,
IROS, 1996.

View publication stats