And then there is the security problem.
This is where the conversation moves from “AI is annoying” to “AI can become genuinely dangerous when companies put it in the wrong place.”
A recent Meta incident showed exactly why this matters. Hackers reportedly managed to hijack Instagram accounts by tricking Meta’s AI-powered support system during the account recovery process. These were not random throwaway accounts either. Some of the accounts involved were high-profile, including accounts linked to major public figures and institutions.
And the scary part is not that hackers used some genius-level technique. The scary part is how simple the idea was.
From the reports, attackers appeared to use the AI support flow to convince the system to help them add or use a new email address connected to a target account. Once that email was accepted in the recovery process, they could move through the usual password reset path and take control. In plain English, the AI was placed too close to the keys of the house.
That is the problem.
AI should not be sitting inside authentication systems with the ability to make account recovery decisions unless there are extremely strict guardrails, deterministic rules, human escalation, and hard security boundaries around it. A chatbot should not be able to casually help someone change the email address on an account. A chatbot should not be able to bypass or soften the very systems that exist to protect identity. A chatbot should not be treated like a support worker with admin powers when it does not understand responsibility, ownership, risk, or consequences.
This is not because AI is evil. It is because AI is not the right tool for every job.
Large language models are probabilistic systems. They generate responses based on patterns, context, instructions, and probabilities. They are powerful, but they are not deterministic in the way security systems need to be. Authentication must be boring. It must be strict. It must be narrow. It must say “no” more often than it says “yes.” Security does not need a charming assistant that wants to be helpful. Security needs rules that cannot be sweet-talked.
That is what prompt injection and social engineering expose.
For years, security researchers have warned that AI systems can be manipulated with carefully crafted instructions. We have seen silly examples where a company puts a chatbot on a website and people convince it to write poems, give discounts, reveal hidden prompts, or talk about banana bread instead of the product. Those examples are funny because the stakes are low. If a car dealership chatbot says something stupid, it is embarrassing. If a support bot helps someone take over your account, it is a security failure.
There is a massive difference between using AI to answer “How do I upload a photo?” and using AI to decide “Should this person be allowed to recover this account?”
One is customer support. The other is identity control.
And identity control should never be handed over to a system that can be persuaded, confused, or manipulated through language. We have spent decades building cryptography, password hashing, session security, two-factor authentication, recovery codes, account verification, device checks, suspicious login detection, and secure reset flows. These systems exist because authentication is hard. They are built on rules, mathematics, and strict logic.
Then companies turn around and say, “What if we put a chatbot in front of it?”
That is not innovation. That is reckless.
This is one of the biggest mistakes in the current AI rush. Companies are treating AI like a universal layer that can be placed over everything. Customer service? Add AI. Search? Add AI. Email? Add AI. Operating system? Add AI. Design tools? Add AI. Meeting notes? Add AI. Password recovery? Add AI.
No. Stop.
There are places where AI belongs, and there are places where AI should not be allowed anywhere near the steering wheel.
AI can help summarize support tickets. AI can help classify issues. AI can help draft a response for a human support agent. AI can help detect suspicious patterns. AI can help flag fraud. AI can help explain security steps to a user in simple language. Those are good uses. But AI should not be the authority that decides whether an attacker gets access to an account.
That decision should be handled by hardened systems.
A good architecture would allow AI to assist, but not decide. It can guide the user through the process, but it cannot change recovery emails. It can explain what documents are needed, but it cannot approve them alone. It can collect information, but it cannot override account security. It can escalate to a human, but it cannot become the human. It can suggest, but it cannot execute privileged actions without strict verification.
This is the principle companies seem to be forgetting: AI should not be trusted with irreversible actions simply because it sounds confident.
That applies far beyond social media accounts.
Do we really want AI handling bank account recovery? Medical records access? Legal document authorization? School systems? Payroll changes? Government portals? Crypto wallets? Business admin accounts? Domain ownership? Cloud infrastructure? Imagine an AI support bot being tricked into transferring control of a company’s domain, resetting a server password, approving a vendor payment, or changing banking details.
That is not science fiction. That is the logical next disaster if companies keep adding AI to sensitive workflows without proper security thinking.
The uncomfortable truth is that many businesses are rushing into AI faster than their security culture can handle. They are not asking, “What happens if this model is manipulated?” They are asking, “How can we reduce support costs?” They are not asking, “What actions should this AI never be allowed to perform?” They are asking, “How much headcount can we replace?” They are not asking, “Where do we need hard rules?” They are asking, “How quickly can we ship this?”
That is how you get avoidable disasters.
And this is exactly why people are losing patience with AI. Not because the technology is useless, but because companies keep using it in the laziest and most dangerous ways possible. Instead of using AI to make systems more thoughtful, they use it to remove humans from places where human judgment still matters. Instead of using AI to support experts, they use it as an excuse to cut corners. Instead of using AI where it adds value, they shove it into places where it creates risk.
Security-critical systems need a different mindset.
They need least privilege. They need audit logs. They need explicit permissions. They need multi-factor authentication. They need rate limits. They need anomaly detection. They need human review for sensitive changes. They need separation between “conversation” and “execution.” They need AI outputs treated as suggestions, not truth. They need every tool the AI can use to be carefully scoped and monitored.
If an AI assistant can perform actions, then every action should be treated like an API endpoint exposed to a potentially confused intern who is extremely helpful, never sleeps, has no real-world judgment, and can be manipulated by a stranger.
That sounds harsh, but it is a better mental model than pretending the chatbot is magically responsible.
The irony is that AI can actually make security better when used correctly. It can help analyze logs, detect phishing attempts, summarize incident reports, explain vulnerabilities, review code, identify suspicious behavior, and help security teams respond faster. That is the good version. That is AI as a force multiplier for people who still understand the system and remain responsible for the final decision.
The bad version is AI as a replacement for judgment.
And that is what we need to push back against.
AI should help humans do better work. It should not become the weak link in systems we spent decades trying to secure. It should not be dropped into account recovery because executives want to say they are “AI-first.” It should not be given privileged tools just because someone wanted a smoother support flow. It should not be allowed to make high-risk decisions simply because it speaks politely and sounds convincing.
This is the deeper point: the future of AI cannot just be about making models smarter. It has to be about knowing where not to use them.
Wisdom is not only building powerful tools. Wisdom is knowing when a powerful tool is the wrong tool.
A chainsaw is useful. You still do not use it to cut your hair.
AI is useful. You still do not put it in charge of authentication.