Teaching Critical Reasoning with AI: Wright Brothers

Paradoxically, LLMs are bad at (niche) facts, but good for fact-checking

Just saw a video on TikTok, which I won’t share, but here’s the gist.

The speaker in the video says it’s important to keep paper copies of newspapers, because those in power rewrite history. So far, no argument. Don’t throw away papers. History does get rewritten. Primary sources can help ground our understanding.

But the real point of her video is this:

• She finds a newspaper article from 1928 that notes the billions the US government has spent on “aeronautics” since 1899.

• She then looks at the official history of the Wright Brothers, who achieve flight at Kitty Hawk in 1903, and only approach the government for funding — supposedly — in 1905.

• But, as she points out, they were working on the plane starting in 1899.

Her claim is that the newspaper article proves the government was funding this all along, and that this is part of a pattern of the government “withholding” technology from the public and pretending it was invented much later. (Suppressed inventions is a common conspiracist trope).

How do you approach this video and its claims as a student? If you’re me, you’ve developed pretty good instincts over the years, and maybe you see the flaw here too. But I know from years of teaching experience almost no students will, because it is a problem of the unknown unknowns.

I’ve been thinking about AI as a tool for the unknown unknowns. So as an experiment, I fed Claude a video transcript, then asked it to evaluate this as evidence that the Wright Brothers had received secret government funding. Here was the response:

Let's analyze what we can and cannot conclude from just this 1928 newspaper excerpt itself:

What we can definitively know:

1. By 1928, military aviation spending had reached significant amounts:

   • Army aeronautics: Over $1 billion spent since 1899

   • Naval aviation: Nearly $350 million spent since 1912

2. These figures were "just made public" by the Department of Commerce at the time

3. The article was about "high aviation expenses" being seen in current years (1928)

Important contextual facts that affect interpretation:

1. The Wright brothers' first flight was in 1903

2. The spending period (1899-1928) spans:

   • 4 years before the Wright brothers' flight

   • The Wright brothers' development period

   • World War I (1914-1918), which saw massive aviation development

   • Post-war military aviation development

What we cannot conclude from just this excerpt:

1. How much was spent specifically before 1903

2. What "army aeronautics" included in 1899 (could have included balloons, gliders, theoretical research)

3. Whether this spending was connected to the Wright brothers

4. How the money was distributed across the 29-year period

5. Whether the billion-dollar figure is adjusted for inflation

6. What specific projects or research the money funded

The key limitation is that while the article establishes government investment in aeronautics dating back to 1899, it doesn't tell us the distribution or nature of that spending. Most of that billion dollars was likely spent after the Wright brothers' success, especially during and after WWI when aviation became crucial to military operations.

The existence of pre-1903 government aeronautics spending is noteworthy, but without more detail about the early spending amounts and purposes, we can't draw firm conclusions about its relationship to the Wright brothers' work.

Formulating the query

There’s some things here that don’t really matter, and I could see an activity where you put this list in front of students and ask which path of inquiry is likely to be the most productive. I think the point about WWI spending is interesting and immediately shows the billion dollar figure is meaningless when talking about 1899. If you have a war in the middle of your timeline that is known for the first introduction of large scale military aircraft use, you’re going to have a lot of aviation spending both during and after that. They could have spent peanuts before that; war and post-war spending is likely to be the bulk of the figure.

But I don’t think we need to pursue that, it just means that we don’t have to think about amounts, just dates. And that’s the direction I would go — what else was the Department of Defense funding?

So I ask Claude:

can you construct a google query that might help me find out what sort of aeronautic spending was happening between 1899 and 1903?

It replies:

Here are some targeted Google search queries that could help uncover pre-Wright brothers military aviation spending:

• "US army aeronautics spending 1899-1903"

• "military aviation research budget 1899-1903"

• "army signal corps balloon spending 1899-1903"

• "Langley aerodrome government funding 1899-1903"

• "Samuel Pierpont Langley War Department funding"

• "Albert F. Zahm army aeronautics 1899"

I'd especially recommend including Langley in your searches since he was receiving significant government funding during this exact period for his aerodrome experiments through the War Department and Smithsonian.

You might also try: "congressional appropriations aeronautics 1899-1903" "war department aviation experiments 1899" "army signal corps aerial reconnaissance 1899"

When searching, you may want to:

1. Look specifically for digitized government documents, military records, and congressional records from this period

2. Check historical newspaper archives from 1899-1903, rather than later accounts

3. Look for mentions in the annual reports of the Secretary of War from those years

I decide to put one of the Langley searches in, and what I get in the first results is pretty useful: A record of funding from the Smithsonian

So in 1898 funding was allotted to Langley, a big name at the time in science, and in fact the Smithsonian Secretary at the time. A bit more research reveals that his massive experiment flopped horribly in a pretty public way just days before the then unknown inventors the Wright Brothers succeeded. In fact, the too-public flop had the effect of drying up defense spending on airplanes for a time, which put the U.S. a considerably behind Europe at the beginning of the first World War. It was the experience of that war which restarted funding in earnest.

And as far as this being “secret history”, I’d point out the Air Force Base that the U.S. President flies into is called Langley Air Force Base for a reason: he was among the first to make a successful case for the military funding of aviation.

Good for fact-checking, bad at (niche) facts

The thing you’ll notice about this activity, especially as you might imagine it in the classroom, is that the LLM is used to give the lay of the land, but not to build a detailed map of it. It’s not that Claude is bad at facts in general — it is able to tell you about the Wright Brothers and about Langley quite well. But in finding elements of more detail, it is perhaps better to have the students transition to search.

I think of the LLM use here a bit in terms of what my colleague Sam Wineburg calls “taking bearing” — when we land in the middle of a cognitive task like this the impulse is to run straight off towards the first issue we think of. Taking bearing means surveying the landscape before running off into your research session. Part of that is understanding what you’re looking at, whether a source, claim, or set of Google results. But I think part of it is this too, getting something to generate a list of relevant questions, the stuff you might not be thinking about, the things that aren’t part of your initial reaction. You can chose the path yourself, but here’s what’s available. There are usually more directions you can go than you think.

Once we have that bearing and start actively exploring, we might still need the a bit of perspective from time to time. But starting by zooming out is key, and it’s increasingly clear to me that LLM technology can be a useful partner in that.