Ethics and Bias

Imagine two hospital patients with the same chronic conditions, the same blood-test results and the same doctor. A computer program gives one of them a higher “risk score”, and only that patient is invited into a programme with extra nurse visits. Nobody programmed the computer to prefer anyone. It simply learnt from years of real records — and the records carried a pattern from the world that the people who built it had not noticed. This is not a thought experiment: something very like it was found in a widely used American health algorithm in 2019. This page is about how that happens, why it is harder to fix than it sounds, and who is responsible.

Where bias comes from

In everyday speech “bias” means prejudice. In AI it has a broader, more technical meaning: bias is a systematic pattern in which a system works worse for, or decides against, some group of people. It rarely comes from a programmer’s ill will. It usually enters quietly at one of four stages. Tap each tile.

• 1DATAWho is in the data?If some groups are missing or under-represented in the training data, the model learns less about them. Face-analysis systems trained mostly on lighter-skinned faces made far more errors on darker-skinned women (the Gender Shades study — see below).
• 2LABELSWhat counts as the “right answer”?A model learns to predict its label. If the label is a stand-in for what you really care about — arrests for crime, spending for sickness, past hiring decisions for talent — the model inherits every distortion in that stand-in.
• 3DESIGNWhat was the system built to optimise?Choices about which features to use, which errors to count as worst and which accuracy figure to report are value judgements, even when they look like engineering.
• 4USEHow is it used in the real world?A tool validated on one population is used on another; a “decision aid” becomes the decision because staff are busy; a score meant to offer help is used to deny it. Harm often appears only at deployment.

The everyday, user-level version of this story — how bias shows up in chatbots and image generators, and the Gender Shades and Amazon recruiting cases — is told in What AI Gets Wrong. In brief: in 2018 Joy Buolamwini and Timnit Gebru found that commercial gender-classification systems erred on up to about 35% of darker-skinned women but under 1% of lighter-skinned men. That was mainly a data problem. The two cases below are subtler, because they are mainly about labels and definitions.

Two cases, accurately told

A health algorithm that used cost as a stand-in for need. In October 2019 Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan published a study in Science of a commercial algorithm used by US health systems to decide which patients should get extra care-management support. The algorithm predicted each patient’s future health costs and treated high predicted cost as high need. But less money had historically been spent on Black patients than on equally sick White patients — for reasons including unequal access to care. So at the same risk score, Black patients were considerably sicker. Cost was a proxy variable, and a biased one. The authors estimated that correcting the disparity would raise the share of Black patients flagged for extra help from 17.7% to 46.5%. Importantly, the algorithm did not use race at all; the bias came entirely through the label. The researchers shared their findings with the manufacturer, and their experiments with a label based on health rather than cost removed much of the gap.

Consensus The lesson that researchers across the field draw from this case is that removing a sensitive attribute such as race or gender from the inputs does not make a model fair. Other variables — a postcode, a school, a spending history — can carry the same information.

COMPAS and the argument about what “fair” means. COMPAS is a commercial tool used in some US courts to estimate how likely a defendant is to reoffend. In May 2016 the newsroom ProPublica published “Machine Bias”, an analysis of scores for thousands of people arrested in Broward County, Florida. It found that among defendants who did not go on to reoffend within two years, Black defendants were nearly twice as likely as White defendants to have been labelled higher risk (about 45% against 23%). Among those who did reoffend, White defendants were more often labelled low risk (about 48% against 28%).

The company behind COMPAS, then called Northpointe, replied that the tool was fair: a given score meant roughly the same chance of reoffending whatever the defendant’s race, and its overall accuracy (around 60%) was similar for both groups. ProPublica did not dispute those figures. Both sides were measuring something real. They disagreed about which kind of fairness mattered — and, as researchers showed within months, the two kinds could not both be satisfied.

Why fairness definitions conflict

There are many mathematical fairness metrics. Two of the most intuitive are:

• Equal meaning of a flag (often called calibration or predictive parity): if the tool says “high risk”, that should mean the same chance of reoffending for everyone. This was Northpointe’s standard.
• Equal error rates: people who will not reoffend should face the same chance of being wrongly flagged, whatever group they belong to. This was ProPublica’s standard.

Here is a small worked example with invented numbers. Two groups of 100 people each. In group A, 50 people will reoffend; in group B, 20 will. (The groups differ in their recorded base rates — and in real data those rates may themselves reflect unequal policing.) The tool flags people so that, in both groups, 60% of those flagged do reoffend.

By the first standard, the tool is perfectly fair: a flag means 60% in both groups. By the second, it is badly unfair: an innocent member of group A is almost five times as likely to be wrongly flagged as an innocent member of group B. Try to equalise the wrong-flag rates and the meaning of a flag will drift apart. In 2016–2017 Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan, and separately Alexandra Chouldechova, proved that this is not an accident of one tool: whenever groups differ in their base rates and prediction is imperfect, you cannot satisfy both standards at once.

Debated Which standard should win depends on the setting, and serious people disagree.

Transparency and explainability

If a bank’s system refuses your loan, you will want to know why. Explainability is the ability to give a person an understandable account of how a decision was reached. It matters for three different people: the affected person, who needs reasons to challenge a mistake; the operator, who needs to know when not to trust the tool; and the auditor or regulator, who needs to check the whole system.

Simple models — a short checklist, a small decision tree — can be explained directly. Large neural networks, including the large language models behind chatbots, cannot: their behaviour is spread across billions of numbers. Tools that estimate which inputs mattered most are approximations and can mislead. When a chatbot gives a fluent “explanation” of its own answer, that explanation is itself generated text, not a reliable window into how the answer was produced.

Debated Some researchers, notably Cynthia Rudin, argue that high-stakes decisions about people should use models interpretable by design; others reply that this can cost real accuracy and that well-tested black boxes with strong auditing can be acceptable.

Transparency is wider than explainability. It includes simply telling people that an automated system is involved, publishing what it is for, what data it uses and how it was tested, and saying where to appeal. Laws are starting to require this: the European Union’s AI Act, for example, gives people affected by certain high-risk AI decisions a right to an explanation of the system’s role As of 2026. The wider legal picture is on Regulating AI.

Accountability: who is responsible?

When an AI-assisted decision harms someone, responsibility can seem to dissolve: the developer says the client misused the tool, the client says the tool was certified, the officer says the computer decided. Ethicists call this the “many hands” problem. A useful habit is to ask four questions:

1. Who built it?Developers are responsible for testing the system on the people it will be used on, documenting its limits and fixing known failures.
2. Who chose to use it, and for what?The deploying organisation chose the task, the threshold and the setting. Using a tool outside its tested purpose is that organisation’s decision.
3. Who reviews its outputs?A human in the loop only helps if that person has the time, training and authority to disagree. Studies of automation bias show people tend to defer to machine recommendations, so a nominal check can become a rubber stamp.
4. How can an affected person complain?Without a route to appeal and a named person who can reverse a decision, nobody is truly accountable.

A sobering example comes from the Netherlands. In the childcare-benefits scandal that led the Dutch government to resign in January 2021, the tax authority’s fraud-risk system treated non-Dutch nationality as a risk factor, and thousands of families were wrongly pursued for repayment. The algorithm was only one part of a wider failure of process, oversight and appeal — which is exactly the point: accountability is a property of the whole system, not of the code.

Three major ethics frameworks

Since the late 2010s, hundreds of AI ethics guidelines have been published. Three matter most for a Malaysian reader. Open each card.

This guide’s framing The frameworks agree remarkably closely on what they value — fairness, transparency, safety, privacy, accountability, human benefit. Where they differ, and where most of the real argument now lies, is how to turn principles into practice: who audits, what counts as enough testing, and what happens when a principle is broken.

Try it: a realistic dilemma

A state welfare agency receives far more aid applications than its officers can check by hand. A vendor offers a model that flags applications likely to contain errors or fraud for a closer look. In a trial, it finds twice as many genuine problems as random checks. But applicants from rural districts are flagged at nearly twice the rate of urban applicants, partly because their documents are more often incomplete.

Notice that none of the options is purely technical. Each is a judgement about whose errors matter most — which is the lesson of COMPAS in a different setting.

Key terms on this page

Sources & further reading

• Obermeyer, Z., Powers, B., Vogeli, C. and Mullainathan, S., “Dissecting racial bias in an algorithm used to manage the health of populations”, Science 366 (2019).
• Angwin, J., Larson, J., Mattu, S. and Kirchner, L., “Machine Bias”, ProPublica (23 May 2016), with Northpointe’s response and ProPublica’s follow-up “Bias in criminal risk scores is mathematically inevitable, researchers say” (December 2016).
• Kleinberg, J., Mullainathan, S. and Raghavan, M., “Inherent Trade-Offs in the Fair Determination of Risk Scores” (2016); Chouldechova, A., “Fair prediction with disparate impact” (2017).
• Buolamwini, J. and Gebru, T., “Gender Shades”, Proceedings of FAT* (2018).
• Rudin, C., “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead”, Nature Machine Intelligence (2019).
• OECD, Recommendation of the Council on Artificial Intelligence (OECD AI Principles) (2019, updated 2024).
• UNESCO, Recommendation on the Ethics of Artificial Intelligence (2021).
• MOSTI, National Guidelines on AI Governance and Ethics (AIGE) (2024).