Ethics Case Study: Ravi is a senior police officer with vast experience in riot control and cyber-policing. Since one year, he has been the Superintendent of Police (SP) of a district with a history …

The dilemma

Ravi has a system that is working by its own metrics — public order has visibly improved — and a credible allegation that the improvement rests on discriminatory foundations. The trap is that the evidence of success and the evidence of injustice are the same data.

The ethical issues, including bias

• Feedback loops. The model is trained on historical arrest data. If a neighbourhood was over-policed before, it generated more recorded crime, so the model directs more patrols there, which generates more arrests, which confirms the model. This is a ratchet, not a discovery — the system measures police attention, not crime.
• Proxy discrimination. Even without ethnicity as an input, postcode, income and migration status act as proxies. The claim that the algorithm is neutral because it is blind to identity is false.
• Presumption of innocence inverted. Preventive detention and checkposts based on statistical association punish people for correlations, not conduct.
• Opacity and due process. Residents cannot see what is recorded against them, cannot contest it, and cannot correct it. A person marked by an unseen file has no remedy.
• Consent and privacy. Biometric capture of everyone in a crowd is mass surveillance of the innocent, engaging the proportionality test from Puttaswamy (2017). Our note on surveillance covers the wider problem.
• Accountability gap. If the system is wrong, who answers — the vendor, the officer who acted, or nobody?
• Erosion of police legitimacy. Order maintained by surveillance of a community produces compliance without consent, and destroys the cooperation on which real policing depends.

Options before Ravi

• Reject the memorandum and continue. Merit: results, deterrence, no operational disruption. Demerit: entrenches bias, invites litigation, and treats a rights complaint as an obstacle.
• Suspend the system entirely. Merit: immediately stops the harm and signals good faith. Demerit: loses genuine investigative value, may raise crime, and over-corrects on an unproven allegation.
• Continue but add safeguards without independent scrutiny. Merit: preserves capability. Demerit: self-audit by the beneficiary of the system is not credible to the very community that complained.
• Suspend the automated enforcement triggers, keep the system for investigation only, and commission an independent audit while engaging the community.

The recommended course, and why

The fourth option optimises ethical compliance while preserving legitimate capability. Concretely:

• Immediately stop AI-driven preventive detention and blanket checkposts. No liberty should be restricted on an algorithmic score alone; require independent human grounds, recorded, for each action.
• Retain the tool for post-crime investigation — matching a suspect against a library after an offence is a different act from marking a neighbourhood before one.
• Commission an independent bias audit — technical experts plus civil society, examining training data, error rates disaggregated by community, and false-positive burden.
• Open the data to the data subject. A notice-and-correction mechanism so residents can see and contest entries.
• Publish the policy — purpose, retention period, access controls, and the human-review requirement.
• Engage the delegation as partners, not adversaries: a standing community-police committee with visibility of aggregate outcomes.
• Address the underlying conditions. Gang violence in a low-income immigrant neighbourhood has causes that patrolling does not touch.

Conclusion

Ravi should treat the memorandum as intelligence rather than as opposition. A predictive system that cannot be audited, contested or explained does not become acceptable because it works; and a police force that maintains order by making one community permanently suspect has traded long-term legitimacy for short-term statistics. Our note on police reforms sets out the institutional context.