Donald A. Norman
Agents occupy a strange place in the realm of technology, leading to
much fear, fiction, and extravagant claims. The reasons for this are
not hard to find: the concept of an "agent," especially when modified
by the term "intelligent," brings forth images of human-like
automatons, working without supervision on tasks thought to be for
our benefit, but not necessarily to our liking. Probably all the
major software manufacturers are exploring the use of intelligent
agents. Myths, promises, and reality are all colliding. But the main
difficulties I foresee are social, not technical: How will
intelligent agents interact with people and perhaps more important,
how might people think about agents?
Automata are not new concepts. Intelligent machines have existed in
fact or fiction for centuries. Perhaps the most relevant predecessors
to today's intelligent agents are servomechanisms and other control
devices, including factory control and the automated takeoff,
landing, and flight control of aircraft. The new crop of "intelligent
agents" are different from the automated devices of earlier eras
because of their computational power: they have Turing-machine
powers, they take over human tasks, and they interact with people in
human-like ways, perhaps with a form of natural language, perhaps
with animated graphics or video. Some agents have the potential to
form their own goals and intentions, to initiate actions on their own
without explicit instruction or guidance, and to offer suggestions to
people. Thus, agents might set up schedules, reserve hotel and
meeting rooms, arrange transportation, and even outline meeting
topics, all without human intervention. Other, more complex
interventions in human activities are contemplated. These human-like
activities and characteristics are what lead to the special concern
over today's agents. Moreover, today's agents are simple in
comparison to those that are being planned.
To ensure a smooth introduction of this technology, two major themes
are relevant. One theme deals with the way people feel about agents,
the other with comfort and acceptance of their automatic, autonomous
actions.
The Feeling of Control
One of the first problems to face is that of the person's feeling of
control. An important psychological aspect of people's comfort with
their activities -- all of their activities, from social relations,
to jobs, to their interaction with technology -- is the feeling of
control they have over these activities and their personal lives.
It's bad enough when people are intimidated by their home appliances:
what will happen when automatic systems select the articles they
should read, determine the importance and priority of their daily
mail, and automatically answer mail, send messages, and schedule
appointments? It is essential that people feel in control of their
lives and surroundings, and that when automata do tasks for them,
that they are comfortable with the actions, in part through a feeling
of understanding, in part through confidence in the systems.
Confidence comes slowly, and the track record of existing automation
does not lead to much optimism. Thus, the first introduction of
automation into automobiles was met with resistance: the automatic
spark advance, the automatic choke, the automatic transmission all
took decades before they were accepted. The record of automation in
the process control and transportation industries has not been good,
with numerous failures. Although commercial airplanes can take off,
navigate, and fly under automatic control, most passengers --
including me -- would not fly in a pilotless plane: indeed, we
require two pilots partially because of workload, but partially as a
spare: if one pilot becomes incapacitated, the other can take over.
Pilots are not comfortable with all the automation: they feel "out of
the loop" [6] .
Mind you, the lack of confidence in automation is justified. We have
a poor track record in developing large scale, complex systems.
Systems do run amok (see, for example, [1] ). Agents pose especially
complex technical questions because they are intended to be
autonomous processes, sometimes capable of migrating across networks
and processors in the most complex of asynchronous, autonomous,
distributed processing environments. Moreover, each agent is apt to
be created independently of the others, often in ignorance of the
existence of others, so conflicts, contradictory actions, and
synchronizing problems are bound to occur.
Two things are necessary to make all this technology acceptable: one
is technical, the other social. The technical aspect is to devise a
computational structure that guarantees that from the technical
standpoint, all is under control. This is not an easy task.
The social part of acceptability is to provide reassurance to the
user that all is working according to plan. The best way to do this
is through developing an appropriate conceptual model of the actions,
one in which the actions of agents are understood in context so users
feel comfortable in their ability to find out what actions have been
taken in their behalf, that private matters remain private, that
expensive or unwanted actions will not be taken without explicit
permission, and that it is always possible to trace back the actual
sequence of acts and undo any that are seen as unwarranted. This is a
non-trivial task. We don't yet know how to do this, and to a large
extent, the amount of information and explicit control that has to be
provided is a function of the state of the individual's own comfort
level: this will change over time, both for individuals and for
society. Probably, in the early days, agents will have to make their
presence and actions known through a conceptual model of the
underlying operations and then, through graphics, sound, and
appropriately chosen verbal messages, provide a continual updating of
the conceptual state. As reliability increases, so too will people's
comfort and acceptance. The user should therefore be able to change
the amount and form of feedback, decreasing to some minimal level. I
imagine that at first, people may want to know all the actions taken
for them, but after they have come to trust the actions, they will be
annoyed to have complete reporting. Nonetheless, I suspect there will
always be the need to have the potential for such complete reports,
even if they are seldom requested. Just as many people wish to be
able to review their bank records each month, even though they seldom
find an error, I suspect people will always want to be able to know
about the actions of their agents.
Overblown Expectations
If the one aspect of people's attitudes about agents is fear over
their capabilities and actions, the other is over-exaggerated
expectations, triggered to a large extent because much more has
promised than can be delivered. Why? Part of this is the natural
enthusiasm of the researcher who sees far into the future and
imagines a world of perfect and complete actions. Part of this is in
the nature of people's tendency to false anthropomorphizing, seeing
human attributes in any action that appears in the least intelligent.
Speech recognition has this problem: develop a system that recognizes
words of speech and people assume that the system has full language
understanding, which is not at all the same thing. Have a system act
as if it has its own goals and intelligence, and there is an
expectation of full knowledge and understanding of human goals.
The problem is amplified by the natural tendency of researchers and
manufacturers to show their agents in human form. You can imagine the
advertisements: "Want to schedule a trip, the new MacroAgent System
offers you Helena, your friendly agent, ready to do your bidding." As
soon as we put a human face into the model, perhaps with reasonably
appropriate dynamic facial expressions, carefully tuned speech
characteristics, and human-like language interactions, we build upon
natural expectations for human-like intelligence, understanding, and
actions.
There are some who believe that it is wrong -- immoral even -- to
offer artificial systems in the guise of human appearance, for to do
so makes false promises. Some believe that the more human-like the
appearance and interaction style of the agent, the more deceptive and
misleading it becomes: personification suggests promises of
performance that cannot be met. I believe that as long as there is no
deception, there is no moral problem. Be warned that this is a
controversial area. As a result, it would not be wise to present an
agent in human-like structures without also offering a choice to
those who would rather not have them. People will be more accepting
of intelligent agents if their expectations are consistent with
reality. This is achieved by presenting an appropriate conceptual
model -- a "system image" [2] -- that accurately depicts the
capabilities and actions.
Safety
Safety plays a part in the feeling of control: making sure that the
agent does not do things that would jeopardize the physical, mental,
or monetary well-being of the owner. But how can this be guaranteed
when intelligent agents might enter one's system from outside?
Sometimes one won't even know, as when they arrive in the mail, or
are parts of some new set of capabilities being added to the
computational system. How does one guard against error, maliciousness
(as in the spread of computer viruses), and deliberate intent to pry
and probe within one's personal records?
Privacy
Privacy could be considered a subset of the sense of control, but
because the technical and social implications are considerably
different, it deserves its own special consideration. Privacy is a
complex topic, one deeply rooted in human cultural and legal systems.
The concerns for privacy within the United States are not necessarily
mirrored in the rest of the world, nor for that matter, even in the
prior history of the United States.
Privacy often pits the interests of one group against another: the
right of citizens to know what their government is doing; the right
of one family to know what its neighbors are doing; the right or
necessity of a government or person to keep its activities private
and confidential.
Law enforcement has a need to be able to detect illegal actions:
citizens have a right to be free from unwanted surveillance. Citizens
do not trust their fellow citizens, industry, police, or government
to use information about their activities in legitimate, beneficial
ways. Business feels it can be more efficient and helpful the more
information it has about the desires and behavior of its
customers.
Not all the need for privacy is to avoid the detection of
wrong-doing. White lies and other deceptions are an essential,
positive aspect of social interaction, allowing for smoother,
friendlier social discourse. Sometimes we want to protect a self
image. Sometimes we simply want to be removed from the hustle and
bustle of modern communication -- note the increasing prevalence of
unlisted telephone numbers.
The issues are too complex to be given full treatment here. However,
the idea that autonomous, intelligent agents could have access to
personal records, correspondence, and financial activities is
disturbing to many individuals, no matter how helpful the agents
might be. Moreover, as the ability to imbed agents within electronic
mail messages becomes more prevalent, who will be comfortable with
the mail systems? Any mail message might release agents that search
the recipient's records and return confidential information to the
sender. I have already seen one such system demonstrated, and
although it was shown with benign intent, where the agent "requested"
permission before searching the recipient's address book and
returning the information to the sender, it was easy to imagine other
situations. Suppose the request was deceptive, with what was asked
for differing from what was done.
Privacy and confidentiality of actions will be among the major issues
confronting the use of intelligent agents in our future of a fully
interconnected, fully communicating society. We must address those
issues now, not just in the technical sense, but in the local,
national, and global legal systems.
Human-Agent Interaction
What is the appropriate form of interaction between agent and person?
The question has many different components, including how the person
shall instruct and control the agent, the nature of the feedback from
agent to person, the manner by which the person's conceptual model of
the agent's method of operation and activities is presented, and the
manner by which the agent offers advice and information to the
person.
Take the problem of instruction: programming the agent. This is a
complex issue. Various suggestions exist, from having the agent
instruct itself by watching over people's activities and deciding how
it can offer help, to instruction "by doing": "watch what I do," says
the person," "and then do it for me." Other suggestions include the
development of simple programming languages (for example, [5] , this
issue), some graphical, some declarative.
None of these seem satisfactory. The kinds of activities we assume
will be performed by agents are quite complex. Scheduling events or
ordering and making payments involve temporal relationships with
other activities, distributed in space and time, not under control of
the agent. Asynchronous coordination is not a simple task domain. The
profound difficulty of programming complex tasks is well known by
professional programmers.
Concluding Remarks
Agents are here to stay: once unleashed, technologies do not
disappear. Agents may well have numerous positive contributions to
our lives. They can simplify our use of computers, allowing us to
move away from the complexity of command languages or the tedium of
direct manipulation toward intelligent, agent-guided interaction.
Agents offer the possibility of providing friendly assistance, so
smoothly done that users need not even be aware, much as the modern
automobile controls parts of the engine that used to require human
intervention -- e.g., the spark advance, choke, shifting -- most
of us are delighted to forget these things. Agents promise to hide
complexity, to perform actions we could not or would rather not do
ourselves. And agents could add to human intelligence, adding one
more tool to the domain of cognitive artifacts that indeed do make
people smarter [3, 4] .
But along with the promise comes potential danger. Agents are unlike
other artifacts of society in that they have some level of
intelligence, some form of self-initiated, self-determined goals.
Along with their benefits and capabilities come the potential for
social mischief, for systems that run amok, for a loss of privacy,
and for further alienation of society form technology through
increasing loss of the sense of control. None of these negative
aspects of agents are inevitable. All can be eliminated or minimized,
but only if we consider these aspects in the design of our
intelligent systems.
References
1. Neumann, P. Computers and people: A risky mix (tentative title).
Addison-Wesley, Reading, MA, In Press, 1994.
2. Norman, D. A. Cognitive engineering. In User centered system
design, D. A. Norman and S. W. Draper Ed. Lawrence Erlbaum
Associates, Hillsdale, NJ, 1986.
3. Norman, D. A. Cognitive artifacts. In Designing interaction:
Psychology at the human-computer interface, J. M. Carroll Ed.
Cambridge University Press, New York, 1991.
4. Norman, D. A. Things that make us smart. Addison-Wesley, Reading,
MA, 1993.
5. Smith, D. C., Cypher, A. and Spohrer, J. KidSim: Programming
agents without a programming language. Communications of the ACM .
This issue.
6. Wiener, E. L. Cockpit automation. In Human factors in aviation, E.
L. Wiener and D. C. Nagel Ed. Academic Press, Orlando, FL, 1988.
Acknowledgments
I thank Julie Norman, Tom Erickson, Harry Saddler and Pavel Curtis
for critical readings and helpful suggestions.