Discussion about this post

User's avatar
Dr Michelle Wong's avatar

I also feel a bit weird seeing you release this one - this sort of advice is rampant, obviously, but it's interesting coming from a misinformation expert...

1. There's evidence that the fluency heuristic makes people more likely to believe fluent text like LLM outputs

2. There's evidence that most people don't understand how LLMs work, and believe it's deterministic and somewhat magical, leading to authority bias (the "looking up a database" poll, the "you'll probably earn more if you don't educate your customers on how LLMs work" marketing study)

3. There's evidence that people are unlikely to overwrite LLM outputs, even when experts and their own knowledge contradict it (e.g. the Shaw cognitive surrender study, Klingbeil overreliance on AI study), likely as a result of the above biases

Not sure if you've tried to gauge the extent of inaccurate AI-generated fact-based content online lately, but it's rampant even from formerly reliable sources (e.g. the Columbia Public Health Instagram account, academics on Twitter, podcasts, videos) - so even scientists with advanced degrees aren't checking LLM outputs carefully for obvious inaccuracies (I collected a lot of examples for a recent YT video on AI slop science content).

4. We know there are ways of designing LLM interfaces to reduce these effects and encourage people to override the LLM output, and we know the main LLM companies aren't implementing these.

From all of the above, it seems like one of the few triggers to get people to properly check the output is the uncanny valley "LLM voice" (countering the fluency heuristic), and even that isn't very effective.

Overriding this with a prompt would save time and effort at the expense of accuracy, which to me seems unwise in a system that is otherwise massively skewed towards encouraging metacognitive laziness?

Additionally, for content creators who intend to publish the LLM output, "LLM voice" is one of the few signs that readers will have for detecting lazy text they'll likely have to check for themselves - much like teaching people what a predatory journal is, it's essentially part of the "I" in "SIFT". The risk of being called out for AI slop is also one of the few incentives to check LLM outputs more carefully before publishing. This seems like further reason to not encourage the use of this sort of prompt?

11 more comments...

No posts

Ready for more?