The Rise of the Machines
Can ethics help us to figure out a way to ensure that they don’t kill us?

Notes on and for a talk / workshop for MSc students at Goldsmith's University of London Business School, March 2025.

Key points:

• An introduction to ethics
• Egoism
• Social Contract Theory
• Consequentialist Theories
• Deontological Theories - Goals, Rights & Duties
• Applying ethical frameworks to societies of people and super-intelligent machines

See notes on ethics for the lecture here (pdf).

Applying ethical frameworks to self-driving cars (autonomous vehicles (AVs))

Reference the Trolley Problem - a critique of consequentialism. Consequentialism is a very bad ethical stance to apply to AVs, leading not only to questions like "how do we ensure that the AV kills as few people as possible?" but also to questions like "isn't it better to kill the 85 year-old crossing the road than the 30 year-old mother of two kids?" See video here.

There is a better way. According to Chris Gerdes (professor emeritus of mechanical engineering and co-director of the Center for Automotive Research at Stanford (CARS)):

• The solution is built into the social contract we already have with other drivers, as set out in our traffic laws and their interpretation by courts.
• The core of that social contract revolves around exercising a duty of care to other road users by following the traffic laws except when necessary to avoid a collision.
• If AVs can be programmed to uphold the legal duty of care they owe to all road users, then collisions will only occur when somebody else violates their duty of care to the AV – or there’s some sort of mechanical failure, or a tree falls on the road, or a sinkhole opens.
• If another road user violates their duty of care to the AV by running a red light or turning in front of the AV, the AV nevertheless owes that person a duty of care and should do whatever it can – up to the physical limits of the vehicle – to avoid a collision, without dragging anybody else into it.

Living with superintelligent (ultraintelligent) machines

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any human however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion,' and the intelligence of humans would be left far behind. Thus the first ultraintelligent machine is the last invention that humans need ever make.
(Irving Good, 1965)

Now consider what happens when a superintelligent machine develops agency – that is, the ability to define its own goals. The field of machine or robot ethics is currently grappling with the extent to which a superintelligence may be able to reason about its own goals. Steve Petersen (a cognitive neuroscientist working at Washington University) argues that, because it will be especially clear to a superintelligent machine that there are no sharp lines between one agent’s goals and another’s, that reasoning could therefore automatically be ethical in nature.

Consider some of the social impacts that superintelligent machines could potentially have:

1. Superintelligent machines may potentially automate many jobs that are currently done by humans, leading to widespread unemployment.
2. Superintelligent machines may gather vast amounts of data about individuals, leading to concerns about people’s rights to privacy and autonomy.
3. Superintelligent machines may be biased and ascribe different rights to different groups of people.
4. Superintelligent machines may decide to manipulate or deceive humans (including the construction of “deepfakes”). This could come about if machines decide that they have the right to prioritise their own goals over those of humans.
5. Superintelligent machines may decide that humans are a threat to their goals and act to eliminate us. Or they may decide that humans have no worth other than the atoms of which they are composed, and therefore no right to exist.
6. Superintelligent machines may develop some super-cognitive abilities which lie beyond the bounds of human capability, distinguishing them from humans in the same way that language distinguishes us from other primates.

Activity - some questions of ethics.

1. Can superintelligent machines be held responsible for their own actions?
2. How do we ascribe duties to superintelligent machines in such a way that they respect the rights of human beings?
3. Is it technically possible to incorporate a concept of justice (in the sense of Rawls’s moral theory) into superintelligent machines which applies to a social grouping of superintelligent machines and people?
4. Can superintelligent machines have rights, including the right to own both intellectual and physical property?

You have been appointed to advise the UN on the steps that should be taken to ensure that superintelligent machines will not pose a threat to humankind in the future. How would you advise the UN to proceed?

Reference
The material relating to ethics both here and in the linked pdf is adapted from Fullick, P. & Ratcliffe, M. (Eds.), 1996, Teaching Ethical Aspects of Science, Southampton: Bassett Press.

Further reading
See the references in my Science Museum talk (on which this session was partly based) here.
For exploring the ideas presented here (and many, many others related to AI and ML) I also recommend lesswrong.com, an online forum and community dedicated to improving human reasoning and decision-making. There are also multiple sub-reddits devoted to exploring these ideas, including (but not limited to) r/singularity.