"AI sycophancy" is not always harmful

In the wrong situation it can be profoundly damaging, but sycophancy is a hard problem because for information technology to work it must often take us at our word.

I’m seeing an Anthropic paper on AI psychosis going around, which people are interpreting online as asserting that one out of every 1,500 chatbot interactions results in a psychotic break. This misconception is partially because Futurism, a moral-panic-as-a-service site, produces articles like this:

I’ll remind everyone that a key SIFT skill is to trace things to the source. And when you get to the source you do find an interesting paper, which contains this chart (or piece of a chart, this is the top.

The moderate level of reality distortion there is that AI validates questionable beliefs or delusion beliefs. Bad, right?

I’m sure I’ll be relentlessly misunderstood for saying this — but no, it’s not quite that simple.

When it comes to information, LLMs are fancy search, and search results can lead you astray

I’m not going to comment on the paper as a whole; I haven’t read it closely enough. What I wanted instead to point out is that so much of this comes down to the problem of people using LLMs as chatbots and conceptualizing the problem as if AI was a respected elder in your community offering news and advice. This evaluatory frame persists among the deepest critics and most active boosters. But it’s a bad frame.

Take the chat element out of the synthesis question, and note how things change. For example, think of a hypothetical AI-driven spreadsheet. You put in what you expect to earn next year to make a budget, knowing that a contract is about to hit. The AI kicks back that you only made half that last year, your future income is in error. Annoying! You say yeah, the AI is wrong, there’s money coming. The AI replies that there isn’t any contract it can see, and it’s not going to let you fill that cell. It then directs you to a Wikipedia page detailing relevant human biases.

You have no choice, you abort your task and hope that contract check hits your bank account soon so the AI will let you finish the spreadsheet.

This design pattern would be bad, but it’s also what AI pushing back on delusions can look like, because, crucially, the AI is not always right, often it is not always clear what right is, and frankly you have the right to be wrong.

Spreadsheets are not my area of expertise, but information seeking apps are. And while building Arc, my AI film tool that reconstructs scenes in films, I hit this issue constantly. The LLM often will not take the descriptions I send at face value.

As an example of what I’m working through right now, one of my test queries is a scene description for Sinners that “two people are in the car and the kid plays a song and the driver loves it.” This is one of my favorite scenes.

But this is a tricky scene, because there is another scene in Sinners that is much more famous, where Sammie is in the back seat of the car and Delta Slim in the front, and Delta Slim breaks out into a song that Sammie accompanies the guitar.

Sometimes my tool gets the description right:

Car Ride: Sammie Plays Guitar for Stack

“While driving, Stack (Michael B. Jordan) encourages his younger cousin Sammie (Miles Caton) to play the guitar. Stack is amazed by Sammie’s profound musical talent, which highlights Sammie’s role as a gifted musician within the film’s narrative. This scene establishes the significance of music and Sammie’s abilities.”

Other times it gets it wrong. Conflating the two person scene (where we first hear Sammie sing) with the later Delta Slim scene (where the car has four people in it and Sammie is in the back seat), the description will talk about Stack, Sammie, and “others” in the car. Sometimes it will put Delta Slim in the cast list for the scene.

But remember, my prompt was very specific. It said “two people are in the car and the kid plays a song and the driver loves it.” If the LLM would stick on that point and take it as gospel it would come to the conclusion that that the scene where Sammie is in the front seat and in the car with only Stack cannot be the same scene that Delta Slim is in, by definition.

It doesn’t take my word as gospel. The opposite, actually. It comes to the decision that I am the one that is wrong, and in doing so gives me a useless result.

Sometimes it does that even more forcefully. Testing a version of the code using Flash 2.5, I put in a scene that was on my TV at that precise moment from Romancing the Stone. In it, there is a street party in Columbia in which Michael Douglas and Kathleen Turner are dancing. Turner’s bag with a crucial treasure map is back at their table; the bumbling villain played by Danny DeVito crawls under the table to get it. He is discovered under the table by a woman who assumes he is a sex pest looking up her skirt. She proceeds to beat the living daylights out of him.

This film was on my screen in front of me, and I was interested in whether the actress who punches him is known. So I put a short description in to get a cast list for the scene.

I was then told in forceful terms in a fact-check note that I was actually imagining this scene, that I have likely misremembered it, that there is no evidence this scene exists and I am probably adding detail to a more mundane scene, where DeVito is seen looking at the two of them in a crowd and not beat up but merely jostled by the locals as he tries to make his way through the sea of people.

But here’s the thing: the scene was right in front of me. I was right, and the AI is wrong. More importantly, for the tool to do what it needed to, it needed in that case to take my experience as a given.

Of course, I’m not always right. The whole point of going to the LLM is it is likely to produce a better sourced and often more reasonable set of assertions than one could by themselves. Sometimes I put things into the description that are wrong, getting names mixed up or details muddled. The usefulness of an LLM in these situations is that it will not take my words at face value but push back.

This is the work, actually. Writing information synthesis on top of LLMs is a constant process of figuring out how to navigate this problem. When should it push back? When should it trust its user? We want the AI to tell us when we type in “As Good as It Can Get” that we likely mean the film “As Good as It Gets”. If I mistakenly say that Stack’s brother Smoke is the one driving the car when Sammie plays, I want the AI to push back and say no, that scene has Smoke in it. But I also don’t want to have to argue that the scene on my TV that I am watching right now is real, begging for the software to believe me.

This problem occurs at a million levels. It’s the same issue with Claude Code deciding like clockwork that the id slugs for the new version of the Gemini LLM (say, gemini-3.0-flash-preview) do not exist and then “helpfully” setting them to gemini-2.5-preview to correct your “error”. It’s a tough problem to work around, and a lot of what you provide when you write prompts is create processes that push on assumptions without getting into contrarian doom loops. Like most machine assisted information tasks the process is a dance between:

• what the user thinks they know,

• what the user actually knows,

• what the user thinks they need to know,

• what the user actually needs to know and

• what the machine can synthesize

Which is to say this problem isn’t really even a problem in the “thing to be eliminated” sense. It’s the job. It’s the core work for the programmer, and it’s the core work for the user. You can’t set a rule that the LLM will always correct a user when they are wrong because the LLM is not always right.

Sycophancy and the Opinionated Chatbot

A while ago I wrote a piece on the opinionated chatbot, and the unfortunate road we went down putting that interface on top of an LLM. I still think its one of the best pieces I have written on AI.

The core idea is this: what the chatbot is actually channeling is not an opinion. Chatbots can’t have opinions. What it is channeling — at least during information seeking — is a fancy search result (a synthesis) disguised as an opinion.

These tools fetch and synthesize. With Arc, there isn’t any scene level information on the internet. So it performs searches for things like “Sinners kid guitar car” and sees what comes back. In Gemini that’s often just little snippets of sentences and paragraphs. It combines that information with some predictive text. It sees people talking about the more famous scene on Reddit — “When Delta Slim starting singing and Stack motioned for Sammie in the back seat to start playing guitar in the car, I was hooked on this film, I never wanted to leave that day.” It throws that in the blender with “Favorite moment: Stack’s face when Sammie starts playing Trouble in the car.” There’s a dismal TikTok description “Experience Sammie’s captivating musical performance, blending R&B vibes in a unique car scene reminiscent of the Blues Brothers soundtrack”. There’s a comment on that that says the Delta Slim moment is better. There’s a review that mentions Stack saying to Sammie alone in the car that the guitar used to belong to Charlie Patton, and how Michael B. Jordan’s sense of joy when Sammie plays is contagious. There’s maybe a caption script online somewhere. There’s the IMDB list of cast members, and the IMDB quotes page.

If when it comes back you truly believe this is the machine talking to you, expressing an opinion, and operating under the informational rules we expect humans to follow as independent agents — well, then the question of whether the AI pushes back when you are wrong is existential. But if we see it what we recognize the search-assisted modes for what they are — “fancy search results” — it feels a bit different, just like with the spreadsheet example. Is the AI glazing you, or is it just taking the premise of your search seriously? What we want in different situations is different: with personal issues of money and health if we’re under wrong impressions, corrections are often welcome interventions. With conspiracy theory, it’s probably good if it mentions a conspiracy is not real. When I’m asking about something right in front of me don’t be a dick.

Part of the solution is getting LLMs to act differently in different circumstances. But a more general solution is reducing the impression that you are receiving an opinion at all. That’s one of the reasons why I will continue to explore models like Arc that maintain the practice of AI search-assisted synthesis while minimizing the idea that you are talking to something with an opinion. And it’s one of the areas I think education needs to hammer on. If there is no possibility of effective search without at least some “sycophancy” — part of the solution has to be changing the mental model— through UI, education, and other means — of what we get back.

Note: Halfway through me writing this, Aaron Tay produced this, which is a better treatment of this issue.