I think one thing I do (from over a decade teaching precisely how to fact check) is I use a model of "usefulness" that has credibility in it but is not defined by credibility. That actually simplifies things!
How does your work teach students to apply the nuances of when to shift among different criteria during the research process, per Sam Wineburg et al. https://doi.org/10.1037/edu0000740?
Thank you. I’ve been trying to explain this to people for awhile. I'd love to see you do one of these with the Deep Research models. How does 4o's Deep Research work with your mega-prompt?
Deep Research really forces it into the Deep Research paradigm -- so far I can't get the deep research intensity without it doing deep research output which ignores most of my commands.
What I'd love to see is one of the models take my work and make a parallel product to Deep Research for fact-checking. (Just give me some credit, it's free!)
Well, you’ve basically created a deep research fact check for Claude (my preferred model anyway) so that’s awesome. I’m going to use your verified book as a text for my Independent Research class. I love what you’re doing.
Excellent piece. Searching for fact checking is really tricky and many people including librarians really overestimate how good they are at it despite teaching it
Thank you, thank you, for this in-depth explanation. “I actually dislike the Wizard of Oz…” gave me a good chuckle.
This semester in 102 I had my students identify 4 points of view (original claim, direct opposition, and two nuanced POVs) as they did their research, and your #6 instructions for “another round” fits my charge exactly. This will help tremendously.
Try finding out if Major I mean Lieutenant-Colonel I mean Captain Yuri Nosenko was a true physical defector to the U.S. in February 1964, a false defector to the U.S. in February 1964, or a rogue physical defector to the U.S. in February 1964 whose bona fides the KGB had no choice but to support through the likes of Kulak, Orekhov, Kochnov and Yurchenko because he was telling the CIA and the FBI what it desperately wanted them to hear -- that the KGB had absolutely nothing to do with former sharpshooting Marine radar operator Lee Harvey Oswald during the two-and-one-half years he lived half-a-mile from a KGB school in Minsk.
I admire your stamina, Mike, to take such a deep dive into the Oz example. I appreciate your hard-won conclusions about LLMs.
This is such a valuable contribution. It shows what's possible if these systems were designed to promote information literacy and truth seeking.
What if, say, Anthropic used your prompt and encouraged people to go into SIFT mode with questionable claims?
Really reminds me of the now forgotten openai webgpt project where they struggled over teaching the model how to weigh creditability of sources and suggested needing "inter disciplinary" research https://x.com/aarontay/status/1605242376510771210?t=wQzOW-tbvPUa6NPni32HIA&s=19
I think one thing I do (from over a decade teaching precisely how to fact check) is I use a model of "usefulness" that has credibility in it but is not defined by credibility. That actually simplifies things!
How does your work teach students to apply the nuances of when to shift among different criteria during the research process, per Sam Wineburg et al. https://doi.org/10.1037/edu0000740?
Creditability, or credibility?
This is super-useful! I tried it here on this question: Does a single ChatGPT query could consume approximately 500 milliliters (ml) of water? https://claude.ai/share/c71eeabe-a396-4cc0-84c3-4c2c0048a8be
Thank you. I’ve been trying to explain this to people for awhile. I'd love to see you do one of these with the Deep Research models. How does 4o's Deep Research work with your mega-prompt?
Deep Research really forces it into the Deep Research paradigm -- so far I can't get the deep research intensity without it doing deep research output which ignores most of my commands.
What I'd love to see is one of the models take my work and make a parallel product to Deep Research for fact-checking. (Just give me some credit, it's free!)
Well, you’ve basically created a deep research fact check for Claude (my preferred model anyway) so that’s awesome. I’m going to use your verified book as a text for my Independent Research class. I love what you’re doing.
Excellent piece. Searching for fact checking is really tricky and many people including librarians really overestimate how good they are at it despite teaching it
Thank you, thank you, for this in-depth explanation. “I actually dislike the Wizard of Oz…” gave me a good chuckle.
This semester in 102 I had my students identify 4 points of view (original claim, direct opposition, and two nuanced POVs) as they did their research, and your #6 instructions for “another round” fits my charge exactly. This will help tremendously.
Great article! Thank you for sharing know knowledge.
A purely practical question: How do you store and organise your prompt library for handy access?
I increasingly put most of it into the megaprompt at the neocities site, but for smaller prompts I use the plugin Web Text Expander.
Cheers. I'll check these out.
Try finding out if Major I mean Lieutenant-Colonel I mean Captain Yuri Nosenko was a true physical defector to the U.S. in February 1964, a false defector to the U.S. in February 1964, or a rogue physical defector to the U.S. in February 1964 whose bona fides the KGB had no choice but to support through the likes of Kulak, Orekhov, Kochnov and Yurchenko because he was telling the CIA and the FBI what it desperately wanted them to hear -- that the KGB had absolutely nothing to do with former sharpshooting Marine radar operator Lee Harvey Oswald during the two-and-one-half years he lived half-a-mile from a KGB school in Minsk.