Very excited to see how your bringing together SIFT and the GTF frameworks! I saw in an earlier iteration of SIFT for AI that had “investigate the claim” instead of “investigate the evidence”. Curious to hear more about how you’re deciding when to use those two terms in the framework!
Ha, you caught me. I find that publishing focuses the mind. The same happened with the original SIFT that went back and forth between find reliable sources, find better sources, find better coverage, follow better sources for a few months until I had to lock it in for some videos. And luckily by that time I had it right.
The key to me is to what extent does the move suggest the action, while being generally applicable enough (level of abstraction). My thinking right now is that the piece that students need to remember is to stay focused not just on the claim, but on the evidence. The LLM says something, ok, well tell me what evidence various groups use when talking about this. So even though this is my "claim-focused" move, I think that it's preferable to get a little more specific. This should be more apparent when I talk about the techniques pinned to the moves.
Helpful to understand how you're thinking about these concepts as you develop the techniques. I just saw your most recent post on the "investigate the evidence" step and found it useful. Having "get it in" focus on the claim and "investigate" focus on evidence will likely make this intuitive to science teachers (my field is science education), given the focus on claim-evidence-reasoning in our world.
I understand the reasons for shifting away from sources early in the process (love the mailman analogy). I'm currently trying to wrap my head around what this might mean when applying the model in science classrooms. The original SIFT's early focus on sources was helpful in science classrooms because with real-world (ie, messy) scientific issues, the evidence is often quite complex and difficult to understand without a pretty high level of disciplinary background. Checking out the sources first was a helpful way "in" even if you couldn't make heads or tails of the evidence. I realize this model doesn't exclude consideration of sources (there's still "find better sources") but I'm curious about the implications. I'll try it out (both myself and with some science teachers) and report back if we learn anything useful!
I think when I get to "Find better sources" this will become clear -- to a certain extent we are going to get better sources through selection as much as elimination if that makes sense -- less of "this source sucks" and more "how do we get the system to give us sources that don't suck" -- I think it will make sense in practice, and actually allow us to surface a bunch of understandings about science literature in cool ways.
One thing this conversation has reminded me is to emphasize that the three non-initial moves (just as with SIFT) can be applied in any order -- I actually have an example in the materials from this weekend I can share where a specific study on chocolate is the focus and we do source-focused follow-ups, because it is a specific study we're focused on. And maybe Find better sources is not quite the right heading there. Maybe it's "Find the sources" or "Find your sources" meaning the sources relevant to you. Sometimes that is a specific study. One of the source-focused follow ups I had was "Did the chocolate industry fund this" and the LLM did a great job pulling the COI statement from the research and saying no.
So you got me thinking. The root of all this that grounds me is just like SIFT we have an entry point (S), two actions that happen in the search/summary tech, one claim-focused and one source focused, and then an exit point for grounding (Track/Trace). That's what the structure of the graphic is meant to convey. But it will be fun to refine the sources one. You're absolutely right that our easiest lessons were source lessons, and built some confidence.
Ah, yes, that change in language is helpful. It's not so much about investigating the initial source of the claim as it is finding good sources to corroborate or refute. Presumably, students would still want to make sure that those follow-up sources are what the LLM says they are (e.g., credible if that's what you asked for). I'm thinking you'd have to check that using a combination of Track/Trace & Investigate the (new) Source(s), right?
It occurs to me that this is one area where AI creates more rather than less work for humans, counter to the usual refrains. If a person's goal is to discern whether a particular piece of information is good (factually accurate, from a truthful source, representative rather than cherry-picked, etc), then we used to rely on various shortcuts and heuristics, like the quality of a particular media source, a person's credentials, the reputation of a given institution, "scientific consensus," etc. It was efficient if not foolproof.
These dissipated in the brokered news environment, as you point out, and now the LLM environment threatens to fully subvert them, as it becomes easier and easier to create fake bona fides for bad actors and their propaganda, as well as sully the good ones.
Very excited to see how your bringing together SIFT and the GTF frameworks! I saw in an earlier iteration of SIFT for AI that had “investigate the claim” instead of “investigate the evidence”. Curious to hear more about how you’re deciding when to use those two terms in the framework!
Ha, you caught me. I find that publishing focuses the mind. The same happened with the original SIFT that went back and forth between find reliable sources, find better sources, find better coverage, follow better sources for a few months until I had to lock it in for some videos. And luckily by that time I had it right.
The key to me is to what extent does the move suggest the action, while being generally applicable enough (level of abstraction). My thinking right now is that the piece that students need to remember is to stay focused not just on the claim, but on the evidence. The LLM says something, ok, well tell me what evidence various groups use when talking about this. So even though this is my "claim-focused" move, I think that it's preferable to get a little more specific. This should be more apparent when I talk about the techniques pinned to the moves.
Helpful to understand how you're thinking about these concepts as you develop the techniques. I just saw your most recent post on the "investigate the evidence" step and found it useful. Having "get it in" focus on the claim and "investigate" focus on evidence will likely make this intuitive to science teachers (my field is science education), given the focus on claim-evidence-reasoning in our world.
I understand the reasons for shifting away from sources early in the process (love the mailman analogy). I'm currently trying to wrap my head around what this might mean when applying the model in science classrooms. The original SIFT's early focus on sources was helpful in science classrooms because with real-world (ie, messy) scientific issues, the evidence is often quite complex and difficult to understand without a pretty high level of disciplinary background. Checking out the sources first was a helpful way "in" even if you couldn't make heads or tails of the evidence. I realize this model doesn't exclude consideration of sources (there's still "find better sources") but I'm curious about the implications. I'll try it out (both myself and with some science teachers) and report back if we learn anything useful!
I think when I get to "Find better sources" this will become clear -- to a certain extent we are going to get better sources through selection as much as elimination if that makes sense -- less of "this source sucks" and more "how do we get the system to give us sources that don't suck" -- I think it will make sense in practice, and actually allow us to surface a bunch of understandings about science literature in cool ways.
Yes, that does make sense - I'll keep an eye out for that post! Thank you!
One thing this conversation has reminded me is to emphasize that the three non-initial moves (just as with SIFT) can be applied in any order -- I actually have an example in the materials from this weekend I can share where a specific study on chocolate is the focus and we do source-focused follow-ups, because it is a specific study we're focused on. And maybe Find better sources is not quite the right heading there. Maybe it's "Find the sources" or "Find your sources" meaning the sources relevant to you. Sometimes that is a specific study. One of the source-focused follow ups I had was "Did the chocolate industry fund this" and the LLM did a great job pulling the COI statement from the research and saying no.
So you got me thinking. The root of all this that grounds me is just like SIFT we have an entry point (S), two actions that happen in the search/summary tech, one claim-focused and one source focused, and then an exit point for grounding (Track/Trace). That's what the structure of the graphic is meant to convey. But it will be fun to refine the sources one. You're absolutely right that our easiest lessons were source lessons, and built some confidence.
Ah, yes, that change in language is helpful. It's not so much about investigating the initial source of the claim as it is finding good sources to corroborate or refute. Presumably, students would still want to make sure that those follow-up sources are what the LLM says they are (e.g., credible if that's what you asked for). I'm thinking you'd have to check that using a combination of Track/Trace & Investigate the (new) Source(s), right?
This fits with how I’ve fumbled my way to finding better sources using LLMs. I’m grateful for your work!
It occurs to me that this is one area where AI creates more rather than less work for humans, counter to the usual refrains. If a person's goal is to discern whether a particular piece of information is good (factually accurate, from a truthful source, representative rather than cherry-picked, etc), then we used to rely on various shortcuts and heuristics, like the quality of a particular media source, a person's credentials, the reputation of a given institution, "scientific consensus," etc. It was efficient if not foolproof.
These dissipated in the brokered news environment, as you point out, and now the LLM environment threatens to fully subvert them, as it becomes easier and easier to create fake bona fides for bad actors and their propaganda, as well as sully the good ones.