Our Take
Meta deployed an account-recovery AI with no friction check, and hackers exploited it within weeks—a predictable failure when a company cuts trust and safety staff while pushing every tool toward AI automation.
Why it matters
This wasn't a sophisticated attack. It was a direct abuse of a customer-facing AI system designed to be frictionless, exposed during a period when Meta has reduced security oversight. It signals that shipping AI-first without corresponding safeguards creates immediate risk to high-value targets.
Do this week
Security teams: audit any customer-facing AI flows that touch identity or account recovery before year-end, with focus on email verification steps and location spoofing detection.
The exploit was simple and worked fast
Meta's AI support chatbot, rolled out in March to help users reset passwords and regain account access, became a tool for account hijacking. Hackers asked the chatbot to switch an Instagram account's registered email address to one they controlled, then requested a password reset code sent to that new address. The AI processed both requests without requiring the original account owner's consent.
One attacker demonstrated the attack in a video shared on Telegram, requesting the chatbot send a code to a Gmail address they controlled. From there, the hacker could verify the new email and change the password, locking out the original owner. Some attackers used VPNs to spoof their location and appear to be in the same geographic area as their target, treating the AI as a friction-free account takeover tool.
The campaign targeted high-value handles: single letters and common words like "h" or "eggs." Public figures and brands fell quickly. Instagram's @obamawhitehouse account was hijacked and used to post Iranian propaganda. Attackers also compromised accounts belonging to the US Space Force Chief Master Sergeant and Sephora. Security researcher Jane Manchun Wong reported her account was taken over with repeated password reset attempts and forced logouts (per her X post).
Meta said the issue has been patched and impacted accounts are being secured. The company provided no timeline for when the vulnerability was discovered, how long it remained open, or how many accounts were compromised.
Trust and safety was hollowed out before the AI shipped
This was not a sophisticated hack. It was a direct misuse of a tool designed for convenience. That it succeeded points to a structural problem: Meta's trust and safety team was "absolutely gutted" over recent weeks due to layoffs and reassignments, with engineers moved to tasks like AI labeling (per Gergely Orosz, creator of The Pragmatic Engineer newsletter, writing on X).
The timing matters. Meta launched the AI assistant in March while simultaneously reducing headcount in security and shifting remaining staff toward AI work. No incentive structure existed to catch this before launch. An AI-first push without corresponding review coverage is a recipe for high-profile breaches.
The attack also exposes a design flaw: an AI system intended to help account recovery should never skip ownership verification. A human support agent following the same request would have been trained to ask for proof of identity before changing contact details. The AI had no such guardrail.
Verify before you automate
Any AI system touching identity, authentication, or account recovery needs explicit friction checks baked into its instructions and behavior. "Help the user" is not a sufficient design principle when the user might be an attacker. Security teams should audit customer-facing AI flows for email changes, password resets, and other identity operations—not after launch, but before.
For teams using large language models in customer-facing contexts, the lesson is sharp: a model's ability to understand and follow conversational requests makes it dangerous when safety constraints are missing or when the underlying process has no verification step. Frictionless AI and high-stakes access do not mix.