AI and Children's Data: Governance Under COPPA, the AADC, and State Child Safety Laws

AI and Children's Data: Why Every Product That Might Reach Minors Needs a Child-Safety Governance Layer

When an AI product is used by children, the stakes are different. Not just technically different - legally and ethically different. Companies that treat young users as an afterthought risk harm to kids, regulatory enforcement, and reputational damage that can sink adoption. But the right governance turns that risk into a competitive advantage: safer products, faster regulatory sign-off, and greater trust from schools, parents, and districts.

Below I tell the story of a mid-sized edtech company whose generative tutor reached middle-schoolers after a public launch that skipped a serious child-safety review. They paused, rebuilt governance around children's safety, and relaunched with stronger adoption and regulator buy-in. Then I walk through the laws, practical controls, governance roles, and how child deployments must differ from adult ones.

The real story: a generative tutor that learned a hard lesson

BrightPath (fictional name) built a promising generative tutor. It could explain math concepts, generate practice problems, and hold a conversational thread with students. The product team prioritized speed to market and shipped a polished MVP to trial districts. The marketing collateral said "grades 6-9," but there was no formal policy preventing younger students from registering. No pediatric advisors were consulted. There was no child-specific data handling playbook.

Within weeks, middle-school students were using the tutor unsupervised. Teachers loved it. So did parents - until a small number of concerning outputs surfaced: overly personalized encouragement that blurred the line between coach and confidant, an instance where the model offered unvetted health-advice phrased in a way that alarmed a teacher, and a pattern of the system storing conversational prompts and outputs with no age-based retention policy.

A parent complained to the district and to a state regulator. BrightPath's executive team realized they had launched without a child-safety review and voluntarily paused new rollouts. They did the right thing: stopped, listened, and redesigned.

What they stood up over the next three months became an operational template for child-safe AI:

- A child-safety AI governance program owned by a senior leader with cross-functional authority.
- A pediatric advisory board and child-psychology consultants to review content and interaction design.
- Stricter content guardrails and educational-first prompts that avoid giving medical, legal, or psychological advice.
- An age-appropriate design assessment and UX changes that made the boundaries between tutor, teacher, and parent explicit.
- Mandatory parental disclosure and layered consent flows - plus COPPA-compliant verifiable parental consent for users under 13.
- Data controls preventing children's prompts/outputs from being used to train models without explicit, auditable consent; defined retention limits and deletion workflows.
- A separate incident response pathway so any child-related incident escalated immediately to legal, child-safety advisors, and regulators when required.

They relaunched six months later. Regulators engaged positively; districts that hesitated before signed up. Adoption climbed. The cost of the pause was real, but the payoff - better product, reduced legal risk, and trust from schools and families - was larger.

That is why any product that might be used by minors needs an explicit AI child-safety governance layer.

The regulatory landscape in practical terms

The legal environment is no longer permissive. A growing patchwork of federal, state, and international rules affects how you design, disclose, collect, and retain children's data.

- COPPA (Children's Online Privacy Protection Act, U.S.): If your product is directed to children under 13 or you have actual knowledge you're collecting information from users under 13, COPPA applies. The law requires clear parental notices, verifiable parental consent (VPC) before collecting personal data from young children, limits on data collection, and mechanisms to delete data on request. For AI products that collect conversational prompts or behavioral data, COPPA triggers strong obligations.
- California AADC (Age-Appropriate Design Code) and other state laws: California's Age-Appropriate Design Code imposes obligations for online services likely to be accessed by minors - privacy by default, data minimization, risk assessments, and design reviews. Several states have also introduced specific child-safety or biometric laws that affect collection and use of children's data.
- State child-safety laws and mandatory reporting: Many state laws heighten obligations around child sexual abuse material (CSAM), grooming, and mandatory reporting. These create stronger incident-handling and disclosure responsibilities.
- EU AI Act and online safety rules: In the EU, the AI Act and other sectoral rules treat children as a vulnerable group. Products interacting with minors face extra transparency and risk-management duties; some high-risk use cases carry strict requirements. Cross-border deployments must reconcile EU obligations with U.S. laws.
- Standards and guidance: NIST AI RMF (risk management) and ISO/IEC 42001 (AI management system standard) offer frameworks for embedding risk management and governance across the AI lifecycle, and they can be adapted to child-safety contexts.

These rules share a theme: higher duty of care for children. In practice, that means stricter defaults, tighter data controls, clearer disclosures, and involvement of subject-matter experts.

Design and content controls that matter

Legal compliance is baseline. Product teams must build controls that reduce the likelihood of harm and give operators clear levers to respond.

Key design and content controls:

- Privacy-by-default and minimal data collection: If you don't need a prompt for personalization, don't store it. Default settings should maximize privacy for children.
- Retention limits and no training on raw child prompts without consent: Define short retention windows for child prompts/outputs; prohibit using raw child data for model training unless you have explicit, auditable consent and de-identification safeguards.
- Age-aware response models and educational intent filters: Use classifiers that detect user age buckets (based on consent flows, not covert profiling) and route requests through safer, constrained models for younger users.
- Content guardrails and fallback to humans: Hard filters for profanity, sexual content, and health/legal advice. If a safety classifier triggers, fallback responses should connect to a teacher/parent or provide neutral, safe resources.
- Clear role boundaries: The UI must make it explicit that the tutor is not a therapist, doctor, or guardian. Avoid language that encourages emotional dependency.
- Age-appropriate language and learning design: Readability, tone, and pacing should match the age group; learning pathways must be developmentally appropriate.
- Parental controls and teacher dashboards: Allow adults to review interactions, set privacy and retention preferences, and block features.
- Logging and monitoring for safety signals: Real-time monitoring and dashboards for exposures, anomalous outputs, or repeated safety hits - tracked separately for child users.

These technical controls are most effective when paired with policy and human review.

Governance roles and how to structure the program

A child-safety program for AI is organizational as much as technical. BrightPath's relaunch succeeded because they moved governance from a feature team checklist to an integrated program.

Essential roles and structures:

- Child-Safety Lead or Head of Child AI Governance: Senior role with cross-functional authority (product, legal, security, privacy, education) and direct access to the executive team/board.
- Multidisciplinary Advisory Board: Pediatricians, child psychologists, educators, and child-safety NGOs who review content, assessment criteria, and incident responses.
- Product and Design Owners: Responsible for implementing age-appropriate UX, consent flows, and teacher/parent controls.
- Privacy/Data Protection Officer and Legal: Ensure COPPA compliance, manage verifiable parental consent mechanisms, and map legal obligations across jurisdictions.
- Security/CISO and Data Engineers: Implement retention, encryption, and training-data controls.
- Safety Operations and Moderation: Real-time monitoring, escalation, and incident management teams with a child incident pathway.
- Learning & Evaluation Team: Measure outcomes and unintended harms; conduct pilot studies with oversight and parental consent.

Governance processes to embed:

- Child-specific risk assessments at design stage (analogous to DPIAs and NIST RMF steps).
- Age-appropriate design assessments and educational impact reviews.
- Regular advisor reviews and red-team testing that includes pediatric perspectives.
- Clear incident response playbook for child-related issues with faster notification timelines and regulator engagement.
- Audit trails and documentation compatible with regulators and standards (EU AI Act, COPPA inquiries).

How deployments for children differ from adult deployments

It's tempting to reuse models and controls designed for adults. Don't. Children are a special population. Here's how deployments must differ:

- Consent model: Adults can give informed consent. For young children, you often need verifiable parental consent - and adolescents may have different legal protections. Consent flows must be explicit, auditable, and age-aware.
- Data minimization vs personalization tradeoffs: For adults, long retention and deep personalization may be acceptable with consent. For children, minimize data collection and default to privacy even if personalization value is diminished.
- Safety defaults: Children's systems must adopt conservative defaults (privacy-first, stricter content filters, human escalation) while adult systems may allow more flexible controls.
- Design for development: Interaction patterns must account for cognitive and emotional development. Features that are benign for an adult (e.g., open-ended self-disclosure) can create dependency or risk in a child.
- Regulatory intensity: Authorities scrutinize child interventions more. Expect proactive audits, stricter penalties, and higher expectations for documentation.
- Incident handling: Shorter timelines, mandatory reporting for certain harms, and direct engagement with parents and educators are common in child incidents.

Practical checklist to get started - not a substitute for counsel

If your product might reach minors, start with these actions this quarter:

- Appoint a senior child-safety lead with cross-functional authority.
- Run an immediate age-risk assessment: Is the product likely to be used by under-18s? Under-13s?
- Consult pediatric/child-psych advisors to review critical flows (onboarding, content, escalation).
- Implement privacy-by-default settings for suspected child users and plan for COPPA compliance where relevant.
- Define retention windows for child prompts/outputs and block training on raw child data without explicit consent.
- Build or adapt safety classifiers and an escalation playbook tied to a separate child incident pathway.
- Update product copy and UI to make limits explicit - the model is not a therapist/doctor, and parental controls exist.
- Document decisions and testing outcomes using a standard like NIST AI RMF and tie them to executive reporting.

Conclusion - governance is the enabler

When BrightPath made child safety a governance priority, they didn't slow down innovation - they improved the product, reduced risk, and unlocked adoption in school districts that otherwise wouldn't have engaged. That's the practical lesson: explicit child-safety governance is not a compliance speed bump - it's an accelerator for safe, scalable adoption.

If your AI product can reasonably be used by minors, treat that possibility as a design constraint from day one. Build governance structures, involve child experts, apply stricter technical controls, and document your work. Regulators and families expect it - and when you deliver, adoption follows.