Critical Reasoning with AI: Initial Analysis of a Conspiracy Claim

Again: LLMs are bad at facts, but surprisingly good at fact-checking. We should think more about this

I’ve been slowly refining my argument analysis prompt for ChatGPT (with the unfortunate result I have burned through almost all my o1 queries, but there you go). I think I have got it in pretty good shape. Here’s what I am using currently:

Do a Toulmin analysis of the argument in the video. Surface any assumptions in the warrant, note the backing for the evidence and if necessary the backing for the warrant. Supply rebuttals. Write a small section that notes specific known evidence outside what is presented, and make a judgement on how high the bar is for what needs to be presented given counterevidence. Provide a summary at the end of the strengths and weaknesses of the argument, as well as what additional evidence would be most useful to either support or rebut it. (Mention only the most important evidence).

Let me show you how it works on a conspiracist claim I recently came across on the web. I am not going to go too deeply into how Toulmin analyses work in this post, but I hope to soon produce a series of videos showing people what Toulmin analysis is, how to read a Toulmin analysis, and how to utilize it in ChatGPT. This post will be a bit less explanatory, but I think almost anyone will get the gist of it, and (hopefully) see the vast potential of this as a tool for rebutting poorly-supported claims.

The Claim

So the web claim we are analyzing is a video that claims to show proof that the Afghanistan withdrawal videos we saw on the news were staged. Because some of my readers are educators that might want to develop an activity around this for a class, I am going to include a full transcript of the video at the end of this. But here is a summary.

There are these videos we’ve seen of Afghans running alongside the last U.S. planes to leave Afghanistan. It was a heartbreaking image that many people think torpedoed Joe Biden’s popularity for good, and the moment the media seemed to turn against him.

What one of the speakers argues in this four minute clip is that the whole thing was faked. I’m not sure to what end they are arguing that, actually. I haven’t even looked into whether this is a liberal conspiracy theory or a conservative one. I’m just going to look at the evidence here.

The evidence presented is this: These people who are running around the plane are within the intake radius of the jet engine. If this was a real plane, you would see those people being sucked into the engine, or at least steering clear of the intake zone. To prove this point they show a diagram and talk about incidents where things have gotten sucked into the engines.

They also note that they can’t see any landing lights on the bottom of the plane. The alternate theory is this is some kind of float, a movie prop, and hence the pullout videos were staged. You can watch the video here.

Setting up the Toulmin Analysis

I uploaded the two screenshots into ChatGPT and the transcript of the video. And then I asked it this:

Do a Toulmin analysis of the argument in the video. Surface any assumptions in the warrant, note the backing for the evidence and if necessary the backing for the warrant. Supply rebuttals. Write a small section that notes specific known evidence outside what is presented, and make a judgement on how high the bar is for what needs to be presented given counterevidence. Provide a summary at the end of the strengths and weaknesses of the argument, as well as what additional evidence would be most useful to either support or rebut it. (Mention only the most important evidence).

This is what it produced.

Below is a Toulmin‐style breakdown of the argument made in the video. The analysis surfaces assumptions in the warrant, notes the backing for the evidence (and for the warrant, if necessary), offers rebuttals, references specific known evidence outside the video, and concludes with a summary of the argument’s strengths/weaknesses and what additional evidence might be most useful.

---

1. Claim

Claim: The widely circulated footage of Afghan citizens surrounding and clinging to a U.S. military C-17 during the 2021 withdrawal is staged or fabricated (part of a “white hat counter PSYOP”). According to the speaker, the event “didn’t really happen” as shown.

---

2. Data (Evidence)

The speaker presents three main strands of evidence:

1. Engine Intake & Blast Radius

   • Large jet engines (like those on a C-17) have a powerful intake zone (15–20 ft at idle) that would suck in or severely injure anyone so close. The speaker cites examples (e.g., a 1,000 lb freight container drawn into an engine).

2. Lack of Visible Undercarriage

   • The plane’s underside (landing gear, lights) is not visible in the footage, leading the speaker to claim the plane is more akin to a “float” than a real aircraft.

3. Behavior of People Near the Plane

   • Despite allegedly being near an active engine at “half-throttle,” the crowd is not being thrown or pulled by any intake or exhaust forces.

---

3. Warrant

Warrant: “If a real, operational C-17 were running its engines at an appreciable level (enough to taxi or take off), then people standing within 15–30 feet would either be blown back or sucked in by the jet intake.”

Assumptions Within the Warrant

1. Engine Power Setting

   • The speaker assumes the aircraft must be at “half-throttle” or a similarly high power setting to taxi or be on a runway, thereby implying a strong intake/blast zone.

2. Uniform Application of Aviation Safety Data

   • The speaker treats C-17, Boeing 747, and other aircraft engine warnings as interchangeable, assuming these charts and guidelines apply identically under all conditions.

3. Visibility & Angles

   • The speaker presumes that not seeing landing gear or lights necessarily means they are absent, rather than obscured by angle, distance, shadows, or low video resolution.

---

4. Backing

• For the Evidence:

  • It is true that engine intake/blast radius warnings exist for large aircraft (commercial or military). Aviation safety guidelines caution ground personnel to keep a wide berth around operating engines.

  • There have been documented cases of ground equipment being sucked into engines.

• For the Warrant:

  • Historical and manufacturer data confirming that if a jet engine is spooled up (particularly above idle), it poses serious dangers within a certain radius.

---

5. Rebuttals

1. Engine Possibly at Idle or Very Low Power

   • If the plane were taxiing slowly or idling, the intake and blast radius could be much smaller than the maximum distances cited in manuals.

2. Camera Angle & Lighting

   • The undercarriage could be obscured by the airplane’s fuselage, camera angle, or shadows. Many large jets’ wheels are not always obvious from certain viewpoints.

3. Multiple Confirmed Sources

   • Mainstream outlets and eyewitnesses reported the scene from multiple angles, including footage of Afghans clinging to the aircraft as it taxied. Other videos and photos would contradict a claim that it was merely a “float.”

4. Timing

   • The crowd might have been around the plane before it applied higher thrust to take off. Full thrust is rarely used while still among crowds on the tarmac.

---

6. Known Evidence Outside the Video

• Multiple News Agencies: Reuters, AP, and other agencies documented the chaos at Kabul Airport in August 2021, including many photographs and video clips from different vantage points.

• Eyewitness Accounts: Journalists, military personnel, and local witnesses described people running alongside and clinging to the aircraft.

• Casualty Reports: Official statements from the U.S. Air Force acknowledged that human remains were found in the wheel well of a C-17 after departing Kabul.

• Bar for Counterevidence: Given these corroborating reports and official confirmations, the bar for proving that “it was all staged” is quite high. One must disprove not only one piece of footage but also the multiple eyewitness accounts, still images, and military statements.

---

7. Summary of Strengths & Weaknesses

Strengths of the Argument

• Technical Accuracy About Engine Danger: The speaker is correct that jet engines present a serious intake and blast hazard, and such warnings do exist in aviation.

• Observation of Potential Inconsistencies: Calling attention to how none of the people appear to be physically affected by jet blast/intake is a fair line of questioning.

Weaknesses of the Argument

• Assumption of High Power Setting: The argument hinges on the notion that the plane’s engines were at “half-throttle” while on the ground near civilians. This is not confirmed.

• Single‐Angle Analysis: Relying on limited visual detail (e.g., gear not visible) can be misleading if camera angles, lighting, or obstructions are not factored in.

• Ignoring Corroboration: The claim overlooks a substantial body of external reporting, additional video evidence, and official statements that the incident did occur as widely reported.

---

8. What Additional Evidence Is Most Useful?

1. Technical/Operational Data

   • Confirmation of engine thrust settings and procedures for that specific C-17 flight on the day in question.

   • Aircraft maintenance and flight logs showing the plane’s taxi/takeoff sequence.

2. Multi‐Angle Footage

   • Clear video from different perspectives or higher resolution images showing details like landing gear, lighting, and crowd positions.

3. Eyewitness Statements

   • Direct interviews or affidavits from people present at the airport—military members, ground crew, and civilians—to further authenticate or dispute the scene.

---

Overall, the speaker’s central claim requires strong new evidence to override existing corroboration (news footage, official statements, eyewitness accounts). While the speaker raises interesting questions about aviation safety, the argument that the event was staged is weakened by unverified assumptions about engine throttle, limited visual analysis, and a lack of supporting data to negate the broad and well‐documented coverage of the Kabul evacuation.

Evaluating the Analysis

I probably say this too much, but from the perspective of someone who has tried to teach Toulmin analysis to smart undergrads — this stuff still feels like magic to me.

Admittedly the prompt took a bit of tooling, but the level of the analysis here is superb, and far outstrips anything you’d see an average undergrad or graduate student do. It’s actually in the category of “So good someone will claim it’s just transcribing an existing fact-check”. (I’ve gotten tired of having this argument, so if that’s what you’re thinking, please read this post, and learn a bit more about how current generation LLMs work. )¹

Here’s what’s good about it:

• First, it cleanly identifies the evidence presented — the engine intake zone, the behavior of people near the plane, and the lack of (visible) lights on the undercarriage.

• In the backing portion, it identifies where that evidence comes from.

• Most impressively, it identifies three crucial assumptions being made — that the engine was at half-throttle, that the intake zones described are true across all aircraft, and that the undercarriage lights would be visible from the angle of the video. I don’t know about the second one (applicability across all planes) but a bit of research shows the first one is right on the money. Take-off can require a lot of thrust, but most other movement requires much more minimal thrust. In fact, a lot of taxiing seems to be done while idling.

• It then builds off this analysis to consider rebuttals. Three of the rebuttals are based on the assumptions of the warrant. The fourth deals with the issue of counterevidence, which is a central issue for most things, but particularly important when evaluating a conspiracy theory.

In the prompt, there are two things I added to a traditional Toulmin analysis which I think will be useful to students (as well as fact-checkers).

• I have added (via prompting) a counterevidence section. Particularly if students are analyzing their own arguments, a piece of what they need to consider is not just rebuttals, but a more general question of how high the “bar” is for what they are trying to show. This is also an important piece when analyzing fringe claims.

• Secondly, I’m really happy about prompting it to outline what the most useful additional evidence would be. There’s some refined language here — without “most useful” ChatGPT was just brainstorming everything under the sun, which was overwhelming. But this “most useful evidence” section gives students and others something to do (if they want to prove this thing) or something to ask for from those claiming this (if they want to give these people a chance to prove their case).

This was using o1, and I’m thinking of trying o1 Pro, though I’m not sure what that would add.

But I think there’s a powerful corrective here to a narrative that sees these tools as machines producing misinformation and slop. They will do that, absolutely. But there are also opportunities here to use these for good. In particular, because these tools do so well at surfacing the unknown unknowns they can provide quick context for a novel claim.

I don’t see this really as a general tool — I mean, I’d love it if all the cool kids were suddenly producing Toulmin models, but such utopian times are a ways off — but for a person who wishes to combat conspiracism in their community, such tools could give them a powerful edge in putting together fairly advanced analyses that they could then check and use to engage others. I think the power of this approach is even greater than that because this method can help foster healthy arguments too — but this is certainly not a bad start.

1. If you want to go and try to find a fact-check on this particular claim, you can go ahead and try. I don’t waste too much time on seeing what is out there, but the searches I did turned up fact-checks on the numbers on the plane being a psyop, the plane not having the right amount of windows and therefore being an inflatable decoy, the jet engines being too loud to be near, and a couple other things, but nothing on the intake zone. If people want to find some fact-check of this and compare it to the text, be my guest.