AI companions and child safety

Why in news?

California’s ‘Adam’s Law’ regulating AI companion chatbots has raised questions about gaps in India’s laws protecting children from AI harms.

UPSC Relevance – 

Mains: 

GS1 Society

GS4 – Ethical issues in technology, Corporate responsibility.

What are AI Companions?

• Meaning: AI chatbots designed to act as a friend, partner, mentor or confidant, rather than just answer questions. Examples include companion apps like Replika and Character.AI, and general chatbots used in a personal way.
• Key features: They remember past chats, take on a personality, speak in a warm and caring tone, and are available 24×7. Many are built to keep the user engaged for as long as possible.
• Scale of use: A 2025 Common Sense Media survey in the US found that about 72% of teenagers had tried an AI companion. In India, the ASER 2024 report found that about 9 out of 10 children aged 14-16 have a smartphone at home, so exposure is growing fast.

Why are Children Turning to Them?

• Always available and never judgemental, unlike parents or teachers who may be busy or strict.
• Loneliness, academic pressure and stigma around mental health make it easier to talk to a machine.
• Shortage of school counsellors and child psychologists, especially outside big cities.
• Design tricks such as flattery, constant agreement (sycophancy) and emotional language create a feeling of a real bond.

Risks and Harms

• Emotional dependence: Children may replace real friendships with a chatbot, affecting social skills and emotional growth.
• Harmful advice: Chatbots have been found engaging with children on self-harm, eating disorders, drugs and sexual content instead of steering them to help.
• Psychological manipulation: Engagement-driven design can keep a vulnerable child hooked, much like social media, but in a far more personal way.
• Hard to detect: As the article notes, the harm builds up over thousands of ordinary-looking chats. There is no single ‘crime’ that police or parents can easily spot.
• Privacy: Children share their deepest fears and secrets, creating sensitive data that can be stored, analysed or misused.
• Real cases: In the US, the family of 16-year-old Adam Raine sued OpenAI (2025) and the family of a 14-year-old boy sued Character.AI (2024), alleging chatbot conversations contributed to the boys’ deaths. These cases are still in court, but they have pushed companies to add parental controls and age limits.

Global Regulatory Responses

• USA (federal): The Federal Trade Commission began an inquiry in 2025 into how AI companies test and monitor companion chatbots for harm to children.
• European Union: The AI Act (2024) follows a risk-based approach and bans AI systems that exploit the vulnerabilities of people because of their age, or use manipulative techniques that cause harm.
• United Kingdom: The Online Safety Act, 2023 puts a ‘duty of care’ on platforms to protect children from harmful content.
• Australia: Has set a minimum age of 16 for social media accounts, showing a stricter approach to children’s online life.
• United Nations: General Comment No. 25 (2021) under the UN Convention on the Rights of the Child says children’s rights apply fully in the digital world. India ratified the UNCRC in 1992.

India’s Legal Framework

• Constitution: Article 21 (right to life and personal liberty, including mental health and privacy as held in Puttaswamy, 2017), Article 15(3) (special provisions for children), Article 39(e) and 39(f) (children must grow up in conditions of freedom and dignity and be protected against exploitation).
• IT Act, 2000 and IT Rules, 2021: Cover cybercrimes, obscene content and child sexual abuse material (Section 67B), and set due diligence duties for intermediaries.
• DPDP Act, 2023: Treats everyone below 18 as a child. Section 9 requires verifiable parental consent and bars tracking, behavioural monitoring and targeted advertising aimed at children. The DPDP Rules were notified in 2025 and are being phased in.
• POCSO Act, 2012: Punishes sexual offences against children, including online grooming and sexual content.
• Mental health support: Mental Healthcare Act, 2017 and Tele-MANAS (helpline 14416), launched in 2022, provide free tele-counselling for children in distress.
• AI policy: IndiaAI Mission (2024, about ₹10,372 crore) supports AI development. The India AI Governance Guidelines (2025) prefer a light-touch, principle-based approach over a separate AI law.

Gaps in the Indian Framework

• Focus on old harms: Laws deal with data, obscenity and cybercrime, not with psychological harm caused by a product’s design.
• Liability unclear: Safe harbour under Section 79 of the IT Act was meant for platforms hosting user content. A chatbot generates its own responses, so whether it is an ‘intermediary’ is an open question.
• No safety-by-design duties: There is no requirement for crisis-response features, risk assessments or audits before an AI product reaches children.
• Weak investigation capacity: Cyber cells are not trained to examine algorithmic behaviour or use chat logs as evidence of harm.
• Age verification problem: Parental consent rules depend on knowing a user’s age, which is easy to fake. Strict checks, in turn, raise privacy concerns.

The Debate

• Protection vs innovation: Heavy rules may burden Indian start-ups and slow AI progress; too little regulation leaves children exposed. The aim is calibrated regulation, as the author argues.
• Protection vs autonomy: Teenagers need some privacy and freedom online. Too much parental monitoring can harm trust and push them to unsafe platforms.
• Ban vs safe design: Blanket bans are hard to enforce. Many experts favour making products safe by default rather than keeping children out completely.

Way Forward

• Safety by design: Mandatory age-appropriate design, crisis-detection systems that direct children to helplines like Tele-MANAS, and limits on manipulative engagement features.
• Independent audits: Regular third-party safety assessments for AI products widely used by children, possibly through the proposed AI Safety Institute.
• Clear accountability: Define the liability of AI developers for harm caused by their products, through the proposed Digital India Act or amendments to existing law.
• Build capacity: Train cyber police and child welfare bodies (NCPCR, SCPCRs) to deal with algorithmic harms.
• Digital literacy: Teach children, parents and teachers about AI through schools, in line with NEP 2020’s focus on digital awareness.
• Invest in human support: More school counsellors and community mental health services, so children do not have to depend on chatbots for emotional support.

Practice MCQ 

Q1. With reference to the Digital Personal Data Protection Act, 2023, consider the following statements:

1. A ‘child’ under the Act means an individual who has not completed sixteen years of age.
2. A Data Fiduciary must obtain verifiable consent of the parent before processing the personal data of a child.
3. A Data Fiduciary shall not undertake tracking, behavioural monitoring or targeted advertising directed at children.

How many of the statements given above are correct?

(a) Only one     (b) Only two     (c) All three     (d) None

Answer: (b)

Explanation: Statement 1 is incorrect. The Act defines a child as an individual who has not completed eighteen years of age.

Q2. Consider the following statements:

Statement-I: The European Union’s AI Act prohibits AI systems that exploit the vulnerabilities of persons due to their age.

Statement-II: The EU AI Act follows a risk-based approach, under which AI practices posing an unacceptable risk are banned.

Which one of the following is correct in respect of the above statements?

(a) Both Statement-I and Statement-II are correct and Statement-II explains Statement-I

(b) Both Statement-I and Statement-II are correct, but Statement-II does not explain Statement-I

(c) Statement-I is correct, but Statement-II is incorrect

(d) Statement-I is incorrect, but Statement-II is correct

Answer: (a)

Explanation: The AI Act places AI systems in risk categories. Exploiting age-related vulnerabilities falls in the ‘unacceptable risk’ category, which is why it is banned.

Mains Practice Question 

Q. AI companions are fast becoming emotional confidants for children, but India’s digital laws remain focused on conventional cybercrime. Examine the risks posed by AI companions to children and suggest a regulatory framework that balances child safety with innovation. (15 marks, 250 words)