I used AI to annotate (nearly) the entire introduction of the MAHA report over the weekend. Take a look.
It's a human-in-the-loop system, and this human is tired
So I got the idea to see if I could use SIFT Toolbox to annotate the full introduction of the “MAHA” report (up to page 19) over the course of a weekend. The answer? Almost. I got 57 fact-checks into the thing, with one page left to go before my brain turned to mush and I stopped.
On average, each annotation took about 20 minutes, with a lot of variation. I don’t claim that everything in it is right — in fact I’m sure there’s a bunch wrong. But I think it’s still quite useful. I think even with all the problems of the MAHA report, if you browse it with the annotations you’ll learn a lot.
Here is the annotated introduction. I’d suggest you not go through it sequentially. Rather, just scroll to a section of the first 18 pages that interests you, and click around there. You’ll find stuff that appears to be just wrong, some stuff that is right, but nearly everything benefits somewhat from context.
One thing I regret a bit is I really toned down it calling out errors in a direct way (This thing is wrong! Really wrong!) because I don’t know how accurate it it will be with those judgments and also I wanted to leave room for the audience to make their own judgments, which is a key design decision of the tool. I just think in this case it might be dialed down a bit too much, and hide how many errors — many small, but inappropriate for a government report — there are in this thing. One of the odd things about it is even in the instances where the report is right, it’s citing the wrong sources, summarizing the wrong stats, providing non-existent links. So many things are just slightly off.
Why’d I do this? More on that point later, but my thought is really that a dedicated team of professionals might be able to use these tools to do this sort of work with a quick turnaround, and maybe — just maybe — we could put a ding in that Brandolini Ratio. And if we can do that, lots of doors open — even if right now all I want to do is sleep.
Bravo for your work! You're onto something big.
Hi Mike - Love the work you're doing with the SIFT Toolkit. I've been doodling around with this idea too on Claude and asked it to create a Chrome extension that runs text through a VERY abbreviated SIFT toolkit. It took a little iterating and troubleshooting to make it work, but it does now. https://claude.ai/public/artifacts/d6b16115-3da2-4ace-85e7-0f4eae37cb90
It's definitely a vibe code implementation so would need a secure api implementation to distribute widely, but it's a start!