Thanks for such an insightful piece. I'd like to use the Deep Background prompt to make a Gem in Gemini Pro. Would you advise leaving the 'model type' off or turning on 'Deep Research' along with the prompt?
This article comes at the perfect time. What if Grokipedia's hallucinations were about critical health info? Your point on tracking sources is so cruical.
I have to say that I find the Politifact article a bit shallow. It's always possible to find examples of inaccuracies or extremist sources in 1 million articles. It would take me about 10 minutes to find an inaccuracy in Wikipedia. The real question is how frequently it happens and whether there are processes in place to keep errors and misinformation under control. Arguably Wikipedia does it better now but it's hardly perfect.
Totally agree with you that fact-checking will have to use LLMs. I've actually built an app to check Wikipedia citations https://wiki-cite-checker.replit.app/ (it requires an API key and for offline sources you need to paste the source text yourself)
I too dislike ‘hallucinations’. For me, the correct term is ‘failed approximations’, of which ‘hallucinations’ are an extreme example.
In their core, LLMs ‘approximate the result of understanding without having understanding’. This approximation can be between very good and bad, but it is often ‘good enough’, though often not good enough to just use it as a source without digging deeper. They work best if the user already has knowledge and isn’t trying to outsource their ‘thinking’. But sadly, they are so convincing that people — who naturally seek energy efficiency — do just that.
LLMs are for the first time in history decouple ‘producing understandable output’ from ‘understanding’, which confuses us humans as we are used to a one on one relation between the two.
I'd say that lots of humans "approximate the result of understanding without having understanding" all the time. That's more or less what Kahnemann's first system does.
In a sense you are right, though I also think there is a difference. Yes, most of our intelligence is 'mental automation' (system 1, such as our convictions, we do not start from scratch every time), though it isn't necessary all approximation. For instance, you do not approximate 2x2=4 and you understand without approximation that 4,5,6 and 104,105,106 have something 'exact' in common. You do not have to use system 2 to get to that conclusion.
And it is true too that if you look neurologically you may see certain neurons fire for certain concepts, but the link is fuzzy. By the way, researchers looking for 'features' in what LLMs do on the inside are looking for 'neurons that fire' (but that link is weak too).
The big issue is that there is a difference between 'doing statistics on ink droplets on paper' (analogy by Van Eeden, 1888 — yep!) and 'reading a book'. Doing statistics on ink droplets on paper may (with enough statistics) give you an engine that produces 'understandable output' about the book, but there is a difference with 'reading the book'. The approximations taking place in the mind (yes, there too) we call understanding, and we cannot reduce them to the approximations done by 'ink droplet statistics'. Replace 'ink droplet' by 'token or pixel' and you get today's situation.
We have a lot of approximations in our mind. But it isn't necessarily so (and pretty unlikely, given that the brain apparently uses all sorts of nonlinear tricks) that these can be reduced to the 'approximation' that is inherent in the working of the non-discrete neural machinery in our brains. Different level, like the difference between doing statistical analysis of ink droplets versus reading a book.
In the end, it seems to me we both underestimate our brain (neurologically) and overestimate it (as mind) in these discussions.
Thanks for such an insightful piece. I'd like to use the Deep Background prompt to make a Gem in Gemini Pro. Would you advise leaving the 'model type' off or turning on 'Deep Research' along with the prompt?
I wouldn't mix deep research in, it tends to hijack the process
This article comes at the perfect time. What if Grokipedia's hallucinations were about critical health info? Your point on tracking sources is so cruical.
Fascinating story!
I have to say that I find the Politifact article a bit shallow. It's always possible to find examples of inaccuracies or extremist sources in 1 million articles. It would take me about 10 minutes to find an inaccuracy in Wikipedia. The real question is how frequently it happens and whether there are processes in place to keep errors and misinformation under control. Arguably Wikipedia does it better now but it's hardly perfect.
Totally agree with you that fact-checking will have to use LLMs. I've actually built an app to check Wikipedia citations https://wiki-cite-checker.replit.app/ (it requires an API key and for offline sources you need to paste the source text yourself)
I too dislike ‘hallucinations’. For me, the correct term is ‘failed approximations’, of which ‘hallucinations’ are an extreme example.
In their core, LLMs ‘approximate the result of understanding without having understanding’. This approximation can be between very good and bad, but it is often ‘good enough’, though often not good enough to just use it as a source without digging deeper. They work best if the user already has knowledge and isn’t trying to outsource their ‘thinking’. But sadly, they are so convincing that people — who naturally seek energy efficiency — do just that.
LLMs are for the first time in history decouple ‘producing understandable output’ from ‘understanding’, which confuses us humans as we are used to a one on one relation between the two.
I'd say that lots of humans "approximate the result of understanding without having understanding" all the time. That's more or less what Kahnemann's first system does.
In a sense you are right, though I also think there is a difference. Yes, most of our intelligence is 'mental automation' (system 1, such as our convictions, we do not start from scratch every time), though it isn't necessary all approximation. For instance, you do not approximate 2x2=4 and you understand without approximation that 4,5,6 and 104,105,106 have something 'exact' in common. You do not have to use system 2 to get to that conclusion.
And it is true too that if you look neurologically you may see certain neurons fire for certain concepts, but the link is fuzzy. By the way, researchers looking for 'features' in what LLMs do on the inside are looking for 'neurons that fire' (but that link is weak too).
The big issue is that there is a difference between 'doing statistics on ink droplets on paper' (analogy by Van Eeden, 1888 — yep!) and 'reading a book'. Doing statistics on ink droplets on paper may (with enough statistics) give you an engine that produces 'understandable output' about the book, but there is a difference with 'reading the book'. The approximations taking place in the mind (yes, there too) we call understanding, and we cannot reduce them to the approximations done by 'ink droplet statistics'. Replace 'ink droplet' by 'token or pixel' and you get today's situation.
We have a lot of approximations in our mind. But it isn't necessarily so (and pretty unlikely, given that the brain apparently uses all sorts of nonlinear tricks) that these can be reduced to the 'approximation' that is inherent in the working of the non-discrete neural machinery in our brains. Different level, like the difference between doing statistical analysis of ink droplets versus reading a book.
In the end, it seems to me we both underestimate our brain (neurologically) and overestimate it (as mind) in these discussions.