Critical Reasoning with AI: Testing New Toulmin Instruction On Backing
Short note: I’m testing out an extended instruction for ChatGPT o1 on checking claims that has more instructions on examining backing. Namely I added some explanatory text and this table of evidence:
It’s messy but you can copy and paste it into your project instructions:
When asked for a Toulmin analysis do the following: Surface any assumptions in the warrant, note the backing for the evidence and if necessary the backing for the warrant. Supply rebuttals. Write a small section that notes specific known evidence outside what is presented, and make a judgement on how high the bar is for what needs to be presented given counterevidence. Provide a summary at the end of the strengths and weaknesses of the argument, as well as what additional evidence would be most useful to either support or rebut it. (Mention only the most important evidence). ## BackingNotes on backing for evidence: The backing for the evidence are the pieces of media, artifacts, testimonials, primary sources or similar that are the physical evidence or the source of the authority of the evidence. Examples include: For evidence that someone was a former beauty star: a photo of them as a beauty star, them listed in a list of beauty stars in wikipedia, a newspaper article of them as a beauty star, a screen capture of them as a beauty star in a video or a video of them, their statement that they were a beauty star, someone random persons statement, etc. Backing does not have to be *good* backing, it just has to be what the stated or implied backing is. It should be a description of the actual source or media. There are different types of backing below. When mentioning backing, try to categorize it, and put the type in parentheses if possible.| Evidence Type | Credibility Source | Common Artifacts | Credibility Questions ||---------------------|------------------------------------------------------------------------|--------------------------------------|-----------------------------------------------------------------|| Documentation | Credibility based on direct artifacts | Photos, emails, video. | Is this real and unaltered? || Personal Testimony | Credibility based on direct experience | Statements made by people about events. Witness accounts, FOAF. | Was this person there? Are they a reliable witness? || Statistics | Credibility based on appropriateness of method and representativeness | Charts, simple ratios, maps | Are these statistics accurate? || Analysis | Credibility based on expertise of speaker | Research, statements to press | Does this person have expertise relevant to the area? Do they have a history of being careful with the truth? || Reporting | Credibility based on professional method that ascertains accounts, verifies evidence, or solicits relevant expertise. | Reporting | Does this source abide by relevant professional standards? Do they have verification expertise? || Common Knowledge | Credibility based on existing agreement | Bare reference | Is this something we already agree on? |Here’s sample output for this prompt:
Here is an analysis of the tweet by Douglas Macgregor as evidence that Trump will purchase Greenland:
Claim
The claim made in the tweet is that Trump has a deal worked up for the purchase of Greenland.
Evidence and Backing
Evidence: The tweet mentions "reports emerge" as the source of this claim but does not provide a link, document, or detailed citation of these reports.
Type of Evidence: Reporting (unverified).
Backing: The tweet itself and the reference to unnamed "reports."
Credibility Questions:
Are the reports cited credible, verifiable, and from reliable sources?
Does Macgregor have insider knowledge or a history of reliable reporting on such matters?
Author: Douglas Macgregor is a public figure with a history of commentary on geopolitical and defense matters.
Type of Backing: Personal testimony or analysis (depending on his access to the information).
Credibility Questions:
Does Macgregor have direct knowledge of the deal, or is he relaying secondhand information?
Has Macgregor accurately reported similar events in the past?
Lack of Specificity: The tweet offers no details on the deal or what “reports emerge” entails. It does not mention which parties are involved or any terms of the alleged deal.
Type of Backing: None provided.
Credibility Questions:
Why are there no links or sources cited for further verification?
Warrant
The underlying assumption is that if a credible person claims reports exist about Trump working on a deal to purchase Greenland, it is likely true.
Assumptions in the Warrant:
The reports mentioned are accurate, credible, and verifiable.
Macgregor has direct access to or reliable secondhand knowledge about the situation.
Greenland (controlled by Denmark) is open to such discussions, and Trump (or his associates) has actively pursued it.
Rebuttals and Counterpoints
Denmark's Position: Denmark previously rejected the idea of selling Greenland when Trump floated it in 2019, calling it "absurd." There’s no indication this position has changed publicly.
Greenland’s Autonomy: Greenland is an autonomous territory with its own government, which would have a say in such matters. No recent public indication suggests a shift in their stance.
Lack of Additional Reports: A claim of this magnitude would likely be corroborated by multiple outlets or governmental statements. The absence of such corroboration raises doubts.
Timing and Credibility: The claim is vague and relies on the credibility of Macgregor and the unnamed “reports,” which reduces its immediate reliability.
Known Evidence Outside This Tweet
Historical Context: Trump publicly expressed interest in buying Greenland in 2019, but it was widely dismissed as infeasible.
Geopolitical Factors: The U.S. has strategic interests in Greenland due to its location and resources, but past attempts did not progress.
Recent Developments: There are no known credible reports (as of now) corroborating this claim.
Bar for Evidence
The bar for evidence in this case is extremely high given the extraordinary nature of the claim. A legitimate case would require:
Official confirmation from relevant parties (e.g., U.S., Denmark, or Greenland officials).
A detailed outline of the terms or documentation of negotiations.
Corroboration from multiple reputable news outlets.
Summary of Analysis
Strengths:
The author is a public figure, which may lend some weight to the claim.
The claim aligns with Trump’s known interest in Greenland.
Weaknesses:
The tweet lacks specific details, citations, or direct evidence.
No corroborating evidence from other sources is presented.
Historical precedent suggests significant resistance to such a purchase.
Most Useful Additional Evidence
Direct confirmation from officials in the U.S., Denmark, or Greenland.
Documents or credible reports detailing the alleged deal.
Statements or corroboration from reputable media outlets.
The evidence presented in the tweet is weak and does not substantiate the claim that Trump will purchase Greenland. Further investigation and verification are required.