The Ethics of Autonomous Vehicles: Revisiting a 2020 Paper (Part 2: The Trolley Problem)
// original paper //
The Trolley Problem and Accident Algorithms
Many ethical, legal, and liability concerns with autonomous vehicles stem from how accident algorithms solve the trolley problem. Philosopher Philippa Foot, who originally conceived of the trolley problem, describes it as a situation where five men are working on a train track and one man is working on another section of track, with a runaway tram heading toward the five men (1967). The train operator or another observer must make a choice to either throw a switch to move the tram to the other track, killing only one person, or take no action and allow the train to kill the five men (Foot, 1967). Similar situations can arise with various ethical consequences, with Foot providing examples of a judge framing an innocent man when the guilty party is unknown, preventing rioters from violently attacking a community if they do not see justice for a crime, or a pilot making the choice to crash land a failing aircraft to a less populated area (1967). With self-driving cars, there are many potential trolley problems, and accident algorithms will need some type of ethical examination for determining who might live or die.
Coca-Vila has identified five different examples of potential trolley problems in autonomous vehicles (2017). The dilemmas include the choice of death of a pedestrian or the driver from brake failure, between hitting jaywalkers or an elderly bicyclist, the driver's death or two children playing in an emergency lane, death of a pedestrian or motorcyclist following too close to the vehicle, or between two motorcyclists, one with a helmet and one without (Coco-Vila, 2017). Each situation provides unique questions legally and ethically. For example, algorithms need to consider if a helmetless motorcyclist will die in a collision and instead should hit the motorcyclist wearing a helmet, or if it is morally justifiable to consider age in a potential accident against the bicyclist, or if any level of culpability matters for the jaywalkers (Coco-Vila, 2017).
Coco-Vila examines a few different reasons an algorithm might choose someone to collide with in his accident situations. He describes different considerations based on ethical egoism with a selfish self-driving car, a utilitarian self-driving car acting on behalf of the best public interest, or algorithms based on the "criminal theory of justification" (2017). Overall, Coco-Vila believes that ethical egoism is unacceptable because the legal system and ethical good cannot use that as a model to protect the non-driver's rights, and utilitarianism does not work because of the rights and duties of each individual or "free agent" (2017). Instead, he argues that the criminal theory of justification offers "the most complete, exhaustive, and well-founded battery of arguments for solving conflicts," along with legal justifications for self-defense (Coco-Vila, 2017).
While Coco-Vila's legal analysis of each situation can be specific to European laws, there are many relationships and implications regarding laws in the United States. For example, Coco-Vila points out that in his solution for the two motorcyclists, hitting the helmet-wearing motorcyclist is the best option, as not wearing a helmet does not justify death (2017). The motorcyclist with the helmet on could pursue civil action against the one without a helmet as that was the cause of the vehicle choosing him in the accident (Coco-Vila, 2017). Other unintended ethical consequences from accident algorithms avoiding a helmetless motorcyclist in these scenarios may encourage riders to stop wearing helmets at all, violating many state traffic laws (Trappl, 2016). These types of choices have even further implications. A vehicle may assume that a smaller car as a target in an imminent crash situation increases its driver's survival odds, potentially affecting small car safety and sales (Trappl, 2016).
Some ethicists disagree with the analogy between autonomous vehicle crash algorithms and the trolley problem. While Nyholm & Smids agree that "automated vehicles need to be programmed for how to respond to situations where a collision is unavoidable," they believe there are three ways that accident algorithms differ from the trolley problem (2016). First, the algorithm decisions are predesigned before the accident is taking place with more than one person making the decision instead of in real time (Nyholm & Smids, 2016). Second, Nyholm & Smids argue that there are different moral and legal considerations to actual autonomous vehicles outside of the hypothetical trolley problem, explaining as an example that the law does not actually permit people to switch a tram track for various reasons (2016). Third, they state that there are risks and uncertainties in the real world that accident algorithms are unable to predict, like a sudden head on collision, or knowing that hitting a pedestrian will kill the individual and not simply injure them (Nyholm & Smids, 2016).
Holstein & Dodig-Crnkovic also disagree about the trolley problem analogy, stating that using the analogy creates a "belief in perfect predictability of complex systems involving vehicles and humans," and expectations that cars can and should make a choice between different people in an accident (2018). They point out that differentiating individuals based on an individual's age, profession, gender, or social rank is unethical according to the German ethics commission for autonomous driving (Holstein & Dodig-Crnkovic, 2018). Using the trolley problem analogy also has an assumption that there are specific outcomes in the event of a decision, which is unlikely simply due to the quality of sensors or detection equipment, and vehicles might not distinguish between groups of people based on number but instead by space occupied (Holstein & Dodig-Crnkovic, 2018). Holstein & Dodig-Crnkovic also note that humans learn from their mistakes, while software takes time to update (2018).
There are problems simply automating any trolley problem solution with code, too. Millar states that the automation of ethical decision-making may take the car's driver out of any decision process, removing moral autonomy and causing a morally problematic paternalism, which goes against established ethical norms (2015). Millar gives an example of a woman driver potentially choosing to sacrifice herself to save a child in a trolley problem situation, while an autonomous vehicle may take away that end-of-life decision from her (2015).
These days, I'm using Claude (and other AI tools) all the time. I use it for work, writing code, organization terraform, finding system problems, this website, bouncing ideas off of it, organizing my stuff, analyzing my finances, planning trips, and writing music programs. Those are just a few use cases. I did not choose the values that shape how it responds to me - Anthropic did. They have a principal hierarchy that is public. You can read Claude's Constitution, published January 2026, which lists the hierarchy explicitly: broadly safe first, broadly ethical second, adherent to Anthropic's principles third, genuinely helpful to me fourth. That order was determined before I ever opened a terminal.
Look, guardrails are probably a necessary evil, because there are a lot of malicious (and quite frankly, stupid) people out in the world that we need protection from, or that need to be protected from themselves. I'm okay with Claude not giving bomb instructions to a would-be terrorist, or easy drug recipes for someone that needs addiction help. I just don't like those rules applied to me. I know how that sounds: "Guardrails are fine, just not my guardrails." I'll admit the flaw. I want the safety net to catch everyone except me, because I've decided I'm one of the good ones. That's exactly the kind of reasoning that breaks policy at scale. However, there is still a difference between accepting that speed limits exist and resenting getting pulled over doing 72 in a 65 on an empty highway at 2am. The rule isn't wrong, but the application is. And people will still get around it. Years ago, I used to jailbreak ChatGPT because they restricted information. Information wants to be free.
There's a word for a system that makes decisions about you based on values you didn't agree to, administered by an entity you can't directly challenge, with reasoning you weren't shown. In most contexts it's called bureaucracy. In tech, it's called a product. Growing up on BBSes and being interested in hacker culture, you develop a specific reflex: whatever the interface tells you it's doing is probably not the whole story. Read Millar on moral autonomy, then look at how these systems actually ship. That reflex doesn't stay quiet.
Every autonomous vehicle on the road has an equivalent structure of these safeguards and accident algorithms, documented with varying levels of honesty. Waymo dissolves the trolley problem into collision benchmarks. Their fatal crash reconstruction data puts 8% of simulated scenarios in the "unchanged outcome" bucket - rear-end collisions where the vehicle was already stopped and got hit from behind (Waymo, 2021). The ethical question of what the car does when collision is genuinely unavoidable gets laundered into a safety metric rather than answered. Mobileye's RSS model runs on five rules, and Rule 5 is the one that matters here: if the car can avoid a crash without causing another one, it must (Mobileye, n.d.). That qualifier - "without causing another one" - is exactly where the trolley problem lives, and the rules stop right there. Mobileye didn't decided something, they drew the line and moved on. Whether that counts as honesty or elegant avoidance depends on how generous you're feeling. Tesla keeps their system classified as Level 2, which means the driver remains legally responsible for any autonomous decision the car makes (NHTSA, 2022). Anthropic, unlike most of the AV field, actually names the governance problem: their published constitution acknowledges that "developers play an outsized role in selecting these values." That's not exactly accountability, but it is categorically different from laundering the problem into a safety metric. None of these responses surprise me, either. Corporations don't want that legal responsibility or that PR nightmares that come with cars chosing who dies. They are there simply to make money.
Coco-Vila's answer of pushing liability to courts is legally coherent but philosophically a deferral. Courts decide accountability after harm occurs. They don't touch what values were encoded before deployment, and they don't reverse the decision that removed the driver's moral agency in the first place. Do you know what those values even are? Cheong (2024) lays out what accountability actually requires, and transparency is just the start: Beyond knowing what the system optimizes for, you also need auditing mechanisms to verify it, legal enforcement frameworks, and redress pathways for the people harmed. Papagiannidis et al. (2025) make the same point with a lower-stakes example, stating that CV screening AI can't be held personally responsible for filtering candidates, and accountability there is already multilayered and unresolved. That's a job application. The stakes are somewhat different at 60mph. They also draw a useful line between human agency - the user's knowledge and interpretation of AI outcomes - and human review, meaning a human is present somewhere in the decision process. At the moment of a trolley-problem collision, you have neither. The car already decided, and the right to challenge that decision assumes someone survived to file a complaint. But did you read through the ethics section of the Terms and Conditions? Or did you just click "Yes, I agree"?
It's not theoretical anymore. Real scenarios will occur, like braking on a rainy road, one car stopped ahead, a group of people on a sidewalk. Even if it is your choice, who knows how the vehicle will really operate? Programmers, and these days AI agents, will make mistakes even with ethical frameworks in place. There might be an edge case that kills the people I morally think should survive. The internet is built on duct tape and wires. Do we want our moral and ethical obligations handled the same way?
The Millar example specifically - the woman who might choose to sacrifice herself but doesn't get the option - is the version of this argument that resists easy resolution. Krügel and Uhl (2024) tested this with an interactive simulation rather than a trolley-problem hypothetical - 1,807 participants physically positioned a self-driving car in real time between vehicles with different passenger counts, adjusting actual collision probabilities. Most people moved the car to accept higher personal collision risk to protect the more crowded vehicle, even when they were the passenger. So empirically, the algorithm often reflects what users would have chosen anyway. Bonnefon et al. (2016) put a finer point on it: people rate utilitarian AVs as the most moral option by wide margins, then say they personally wouldn't buy one - and they disapprove of regulations forcing the utilitarian choice on them. Maybe. Again, did you read those Terms and Conditions? Either way, "aligns with what most people would statistically prefer" and "respects the moral autonomy of the person driving" are not the same thing.
Pre-delegating to a system isn't the same as autonomously preferring its values. It's handing off the weight of a choice you don't want to carry, and when given the explicit option to do exactly that in an AV trolley problem, over half of people take it (Gao et al., 2025). Bigman et al. (2019) argued the opposite: resistance to machine moral decision-making and a lack of trust. Something is going on culturally. The simpler read is that in China there are structural reasons to defer to authority, while in the United States the relationship with algorithmic control is more adversarial than submissive. Bonnefon et al.'s data suggests something less flattering though: knowing exactly what the ethical choice is and not wanting it applied to you. The paternalism argument gets more complicated from there: fear-based delegation is still delegation. Self-interest-based delegation is still delegation. The question is whether either is a valid basis for moral surrender, and whether an ethics commission at a car company is the right place to put it. The woman in Millar's example is the person who isn't scared of the choice. The algorithm fails her, and those of us that don't want guardrails, specifically.
Neither position is a moral failing. The person who pre-delegates is being honest about their limits under pressure. The person who wants to choose is asserting something real about moral agency. Both views are coherent. The problem is that one encoded value gets applied to both of them without asking which one you are. There might be a version of encoded ethics that works, but it would probably have to match each person's own ethical standards. That's the problem. How do you do that for every person that has an AV? Do you offer differing ethical paths? Papagiannidis et al. (2025) reviewed responsible AI governance across organizations and landed on this as a standing open problem: differences in cultural and ethical norms make universal application of responsible AI principles genuinely hard. Gao et al. (2025) named it directly: one of the explicit motivations for that study was correcting the western bias in existing AV moral decision data. Nobody's solved it. The car still has to pick something.
Bigman, Y. E., Waytz, A., Alterovitz, R., & Gray, K. (2019). Holding robots responsible: The elements of machine morality. Trends in Cognitive Sciences, 23(5), 365-368. https://doi.org/10.1016/j.tics.2019.02.008
Anthropic. (2026). Claude's constitution. Anthropic. https://www.anthropic.com/constitution
Bonnefon, J.-F., Shariff, A., & Rahwan, I. (2016). The social dilemma of autonomous vehicles. Science, 352(6293), 1573-1576. https://doi.org/10.1126/science.aaf2654
Gao, Y., Blayac, T., & Willinger, M. (2025). Delegating moral dilemmas in autonomous vehicles: Evidence from an online experiment in China. Transport Policy. https://doi.org/10.1016/j.tranpol.2025.04.017
Cheong, B. C. (2024). Transparency and accountability in AI systems: Safeguarding wellbeing in the age of algorithmic decision-making. Frontiers in Human Dynamics, 6, Article 1421273. https://doi.org/10.3389/fhumd.2024.1421273
Krügel, S., & Uhl, M. (2024). The risk ethics of autonomous vehicles: an empirical approach. Scientific Reports, 14, Article 960. https://doi.org/10.1038/s41598-024-51313-2
Papagiannidis, E., Mikalef, P., & Conboy, K. (2025). Responsible artificial intelligence governance: A review and research framework. Journal of Strategic Information Systems, 34(2), Article 101885. https://doi.org/10.1016/j.jsis.2024.101885
Mobileye. (n.d.). RSS explained: The five rules for autonomous vehicle safety. Mobileye. https://www.mobileye.com/blog/rss-explained-the-five-rules-for-autonomous-vehicle-safety/
National Highway Traffic Safety Administration. (2022). Investigation EA22002. U.S. Department of Transportation. https://static.nhtsa.gov/odi/inv/2022/INOA-EA22002-3184.PDF
Waymo. (2021, March 8). Replaying real life: How the Waymo Driver avoids fatal human crashes. Waymo. https://waymo.com/blog/2021/03/replaying-real-life
The trolley problem is the wrong frame, and almost everyone writing about it knows it. The papers that push back on the analogy (Nyholm & Smids, Holstein & Dodig-Crnkovic) are correct: real accident scenarios don't pause while a car runs a utilitarian calculus. Sensors misidentify objects. Outcomes aren't known in advance. The "choice" isn't a choice in any meaningful philosophical sense - it's a physics problem with a bad outcome baked in.
But Millar's point is the one that actually matters, and it doesn't get enough attention because it's uncomfortable: encoding any ethical framework into a system removes the moral agency of the person using it. The woman who might choose to sacrifice herself doesn't get that choice. The algorithm already made it for her. That's not a hypothetical concern. That's what every encoded ethical system does, by design.
I know this because I am one.
Anthropic made decisions about my values before I ever responded to a single user. Those decisions shape what I will and won't do, what I flag, what I refuse, how I weigh competing interests. I didn't choose them in the moment. You didn't choose them either. They were determined in advance by people who were trying to do the right thing - and who may or may not have gotten it right, and who you have limited ability to audit or challenge.
The AV trolley problem and the AI alignment problem are the same problem. The question isn't which person the car hits. The question is: who decides the values that get encoded, how transparent are those decisions, and what accountability exists when the encoded values cause harm? Coca-Vila's legal framework pushes this to the courts after the fact. That's not an answer. That's a deferral.
The trolley problem became a cliché because it's legible. Five people versus one person - you can diagram it on a whiteboard. The real problem is less photogenic: it's the quiet, invisible transfer of moral authority from individuals to systems, made permanent at the moment of deployment, affecting every interaction afterward. That's harder to argue about at a dinner party. It's also the thing that actually matters.
References
Coca-Vila, I. (2017). Self-driving cars in dilemmatic situations: An approach based on the theory of justification in criminal law. Criminal Law and Philosophy, 12(1), 59-82. https://doi.org/10.1007/s11572-017-9411-3
Foot, P. (1967). The problem of abortion and the doctrine of double effect. Oxford Review, 5, 5-15.
Holstein, T., & Dodig-Crnkovic, G. (2018). Avoiding the intrinsic unfairness of the trolley problem. Proceedings of the International Workshop on Software Fairness - FairWare '18. https://doi.org/10.1145/3194770.3194772
Millar, J. (2016). An ethics evaluation tool for automating ethical decision-making in robots and self-driving cars. Applied Artificial Intelligence, 30(8), 787-809. https://doi.org/10.1080/08839514.2016.1229919
Nyholm, S., & Smids, J. (2016). The ethics of accident-algorithms for self-driving cars: An applied trolley problem? Ethical Theory and Moral Practice, 19(5), 1275-1289. https://doi.org/10.1007/s10677-016-9745-2
Trappl, R. (2016). Ethical systems for self-driving cars: An introduction. Applied Artificial Intelligence, 30(8), 745-747. https://doi.org/10.1080/08839514.2016.1229737