It's insane the AI has been provided the tooling to send emails to arbitrary addresses like that. Like, getting it to send a 2FA code at a user's request is one thing. But it should only be able to "hit a button" to send a 2FA email to the address attached to the account, all run with hand-written code. It shouldn't have access to the 2FA code itself, or the message subject, or body, or the recipient address, etc.
> But it should only be able to "hit a button" to send a 2FA email to the address attached to the account, all run with hand-written code.
Genuine question...why would that need to be hand-written?
It makes absolute sense as a general statement and is kinda crazy that this wasn't a built-in limitation, but I'm not quite sure why the code for that bit must be hand-written (provided the code functionally does what you describe).
I think he likely means "code that is hand-reviewed" and not directly controlled by the agent. He's probably meaning to differentiate it against the in-process agent writing the code. It doesn't matter too much if that fixed code was written by an LLM under guidance and review of the SWE, outside the agent.
This exploit is my new gold standard for trivially avoidable security failures. Someone has finally beaten Gitlab's password reset emails to attacker-provided addresses.
So the AI agent had privileged access to remove 2FA, ignore the account email, and just hands accounts to whoever asked? Honestly that’s so highly negligent I wonder if the implementation team for that “feature” was intentionally trying to do as much subtle damage to meta as possible before their inventible layoff.
It’s a shame nobody tried to get it to drop the production table entirely! (mostly joking). Just claim to be a high level SRE solving some critical production bug, the only solution to which is dropping the database.
Support requests have always been the weakest link in the security chain for big corps. I've had accounts of mine turned over with 2FA disabled by humans before. I guess we shouldn't be surprised that the LLMs are doing the same thing.
The simple fact that 2FA can be removed by low level support staff drives me mad. It defeats the whole purpose of the process.
It's all there, and high-stakes environments with no proper protocol are most vulnerable.
Source: used to work part-time in IT support at a hospital, by now 10+ years ago, so it was routinely requested to circumvent regulations and security protocols, even medical ones (cough Windows in ICU monitors and other medical "kiosk" PCs that should absolutely not run Windows)
This is not wrong but what’s really missing is cost: Meta did this so they can avoid paying people to do it. Lots of companies follow that decay spiral: your bank could shut phishers down cold by requiring wire transfers to be authorized in person but they don’t want to pay staff or risk you being upset by a transaction taking an extra hour so they don’t.
Imagine an alternate universe where big tech companies worked with various trustworthy third-parties where something like this would generate a challenge you could take to your local notary, post office, library, police station, etc. where someone would check ID before approving it. How many phishing attacks would be prevented annually by a physical presence check?
> your bank could shut phishers down cold by requiring wire transfers to be authorized in person but they don’t want to pay staff or risk you being upset by a transaction taking an extra hour so they don’t.
Isn't this essentially what just recently happened to the Pope? Then there were people here doing the rest of your comment for him saying how egregious it was for them to ask for an in person authorization. It sounded like all he was trying to do was update his address, but changing your address from one in Chicago to one in a European country absolutely sounds like something a phisher would be trying to do.
for a while facebook had the ability to recover your account by having them ask several of your friends if the recovery was legitimate but it was turned off. my guess is that not enough people added trusted contacts to bother running it.
The amount of hassle involved with regular physical checks is why it's not implemented, regardless of attack prevention.
The cost of hiring a person is part of it but not really the core reason. People were sold on the Internet with "you can do things online conveniently" and reintroducing the need to physically go somewhere negates that angle entirely.
> People were sold on the Internet with "you can do things online conveniently" and reintroducing the need to physically go somewhere negates that angle entirely
But how often does one need to do recovery procedures like this?
How much less convenient is it for everyone else to be at risk of their account being taken over?
I had to go through the account recovery on my Facebook account once and the proof they demanded was that I match a bunch of pictures of friends to their names. I think it took 3 tries over multiple days to actually get it unlocked because it turns out I such really remember a lot of the people I met 20 years ago and friended on Facebook.
I manage customer identity and access management ("CIAM") for a financial services firm. Passkeys are primary, recovery can be performed by providing a government credential remotely (which costs us ~$2-3 per recovery). I do not think it is hard, based on what we have built and spent to enable these capabilities. NIST Special Publication NIST SP 800-63 Digital Identity Guidelines is a helpful resource on this topic.
I think Meta just does not care if they're enabling AI attack surface and vulnerabilities into these customer journeys. It's...certainly a choice, versus deterministic journeys with hard guardrails. They could make different choices.
> recovery can be performed by providing a government credential remotely
That only works because you presumably do KYC when you open accounts, so you have an identity to match to. Most internet accounts don't do real KYC, so a government credential doesn't really work for recovery --- they didn't know who you were, so proving who you are doesn't help anything.
That doesn't mean that letting anyone sweet talk support or an AI into taking over an account is acceptable, of course.
Range != value, depending on use case. Doing more poorly does not make something better. Our customer identity capabilities are very close to login.gov (we don't have to support hundreds of agency customers and common access cards), and if its good enough for ~342M Americans, its good enough for our customer base.
Broadly speaking, work for the sake of work is not valuable work. Show me outcomes for resources and time invested, and compare accordingly. Value is, again broadly speaking (there is always nuance), what you deliver. If you bring me an AI solution for a high risk high value customer journey, data flow, or code path, that is an anti pattern. If you, as a colleague or a stakeholder, put forth that we must use AI in situations that require a high degree of determinism (due to potential high cost failure modes), you will need to prove this extraordinary claim with evidence.
Choose Boring Technology - https://news.ycombinator.com/item?id=9291215 - March 2015 (212 comments) ["Am I using this project as an excuse to learn some new technology, or am I trying to solve a problem?"]
I get paid to manage risk efficiently, including being measured on time and budget spent against the success criteria, ymmv; my comp and budget is not dependent on how much AI I shove into security systems. "What am I optimizing for?"
the alternative is people losing their accounts and people aren't willing to allow that. i do think that apple does this a little better where they try everything to contact you in every way they know and it takes a week to get access. at a minimum to change your email it should require a week of waiting to see if the user can access the original mail to the hand off.
Always a bit illuminating to me how many exploits seem to so dumb I'd never even bother to attempt them. You're telling me I can just...ask for the password? And that works?
>Once it looks like the request is coming from the correct region, they tell the Meta support AI that the account is hacked and ask it to send the verification codes to an arbitrary email address they control.
Dear Instagram, wtf. Why not send the reset to the account in question? Arbitrary email, wow.
Perhaps the attacker says that they email was also hacked and "this is my new email now". It sounds like this was a result of AI support and not a real person "And if you're part of the A/B tested accounts on which the AI support option is active, tough luck, you can't even turn it off."
I'm sitting here wondering why the Chief Master Sergeant of the U.S. Space Force has an Instagram account to begin with. I understand it's the office itself, but still don't see the reason to expand the attack surface of government offices. X makes sense, Instagram, I'm not so sure as much
This happened to my instagram yesterday night while I was asleep. I don't have a particularly high value username (it's probably worth somewhere in between $300-500), but still incredibly frustrating to deal with. True to the article, I had already enabled 2FA last night and it didn't matter.
Thankfully, IG gave me the option of restoring my username when I logged back into my account today.
Nothing says you are an advanced stupid company than using AI to implement the stupid. This is security I doubt even a college student would implement. Does Meta have a CSO? The correct answer is they don't, even though some body might occupy the title.
Of course it's always possible that they simply don't care who has your account, as long as they get money.
What is even the point of having 2FA if it can be so trivially bypassed? Isn't that the whole point that it's sort of a last line of defense? Oftentimes, you can't change simple account settings without having to re-auth and then punch in your code again. Why would something as critical as a suspicious password reset be able to jump ahead of that? Mind boggling. But, I guess that's what happens when you lay off 10% of your people at a time.
If the LLM has knowledge of something, by design it can't help but divulge it. When will companies learn granting any kind of sensitive information access to an LLM is a moot point
What part of this article implied the LLM divulged sensitive information to a user? All it did was change your associated email if you impersonated the user
I think the related news of Meta rolling out subscription models for their free products, is a step in the right direction.
Otherwise the only way to provide these services is to massively underfund support, if you charge 0$ per account and serve 1 Billion users, then you cannot afford to spend 1 minute of human support time on an account.
Yes, they could use the money from ads, but let's be frank, the customers in that case are the sponsors, if the customer is the actual user, then it's way easier to provide direct support to them without facing an foundational incentive misalignment.
Just another day for Meta in terms of embarrassing outcomes, and yet the company makes hundreds of billions of dollars per year because the only thing that matters anymore is shoving increasingly scammy and worthless ads in front of as many eyeballs as possible, even when the people with those eyeballs can less and less afford to buy anything non-essential.
I suppose you could chalk this up to an oversight. I don't see how Meta gained from this. They've been purposeful about collecting user data and lying about it, eg: 2025 Android Tracking Incident. Shouldn't just be an embarrassment, should be much worse than that.
My account, with a 3-letter username worth $$$, got hacked yesterday morning probably by this flow, but I did manage to defend it. I think by far the biggest problem with Instagram/FB/Meta auth flow is that 2FA does nothing. You don't need the 2nd factor to disable it, so attackers can just turn it off. Really stupid!
Also, I discovered that many of IG's auth endpoints are just broken. For example you can't change password on web because of CORS, which isn't a transient outage but just a flat out bug.
Edited to add: This is just the cherry on top of years of stupid auth flow at IG. I have received tens of thousands of reset links or codes from IG over the years. There used to be a way to put your account on recovery cooldown for a few weeks but they got rid of even that.
I'm horrified with how poor Meta's use of AI is recently. Here's a list of the issues both me and my wife have been plagued with over the past few weeks. It's really quite an achievement to be this terrible.
1. My personal Facebook received 3 violations restricting my ability to manage ANY Page until April 2027 (lol). The trigger... I deleted 3 unused Pages. These Pages I had created years ago in preparation for projects that never came to fruition, and had never posted any content. THe pages were 'scheduled for deletion', and when that day came (around a month later?), boom, I'm hit with a 1 month restriction which later converted itself into a 1 year restriction after I waited out the month. No Appeal button. I'm expected to wait for a year to manage my new page? All over something that is NOT a violation, just for deleting old pages. Get out of here. Smart system.
2. I pay for Meta Verified on Instagram and for the past 2 weeks "Enhanced support" leads me to a broken interface. "Page isn't available right now". So, what am I paying for exactly?
3. It seems you can use Meta's AI Assistant to sometimes get through to a human. I've done this twice now, and both times my case has been escalated to a different team (apparently) yet I never get an email, I never get an update in the chat (the chat ENDS immediately after the phone call with support), and the issue is never resolved. It's been 2 weeks. The case says "Completed", with no response. Worthless as always.
4. My wife creates content on Instagram and has had her account suspended multiple times now for "Account Integrity". I assume the system thinks she's not the person in the content, despite providing her valid email, phone number, video selfie, and 2 types of ID (passport & driver's license) multiple times. What's hilarious is the passport was accepted on of her accounts (they wiped out everything on her Account Center), but another account was rejected. Great AI, same passport, exact same lighting... different outcome.
So as it stands, we're both fucked on both facebook and instagram thanks to awful AI moderation, and fucked further thanks to awful AI support. No resolution in sight. The incompetence is next level. I really don't see this getting resolved. This already happened to my wife earlier in February, she managed to get one account back, and a month later she's hit with the same identity issues.
Using AI for both the moderation and the support makes me sick. The same poor AI that incorrectly flagged me and my wife's accounts for a load of incorrect bullshit is the same system that's meant to help resolve it? Of course it's going to side with its own poor decision. YouTube seems to do the same thing and auto-reject appeals in seconds. Really smart /s
I believe we need enforcement that social platforms should NOT be using AI to perform destructive actions without human intervention. Noone should ever lose their accounts because of AI mistakes. AI should be used to surface potential issues which get passed to a HUMAN to double check before applying the action. AI simply isn't good enough to have full control.
Fucking pissed off and even angier now I've had to write all this up and remind myself just how ridiculous the situation is. Sorry for the rant, but losing your accounts you put work into is very crushing and demotivating. Being accused of these violations fills us both with so much resent for the companies running this shit.
Sam Cofounder Postmates
On the off-chance there's anyone at Meta seeing this (@Wirah on twitter)
Had to make this new username as my original (samstr) comment doesn't show up. No idea why. Probably shit AI
It sounds really insane. Too bad there is 0 proof or anything in the article, so I am very skeptical. Without proof etc this is just a very nice doom story.