Back in 2022 Gary Marcus challenged Elon Musk to a bet on what progress AI would make by the end of the decade.1 In a move that showed intellectual honesty he was specific, making a bunch of time-limited anti-predictions. For instance, he proposed that:
In 2029, AI will not be able to reliably construct bug-free code of more than 10,000 lines from natural language specification or by interactions with a non-expert user. [Gluing together code from existing libraries doesn’t count.]
Most of the claims are like that, dead a year or more back. Marcus gets credit for getting specific, and 2022 is pretty far back in AI years. But you also can’t look at these anti-predictions and not come to the conclusion that skeptics have moved the goal posts considerably since then.
Still, there was this intriguing item:
In 2029, AI will not be able to watch a movie and tell you accurately what is going on (what I called the comprehension challenge in The New Yorker, in 2014). Who are the characters? What are their conflicts and motivations? etc.
I take this challenge to mean that the computer doesn’t get a script and doesn’t get any supplementary materials, including a captions file. Ideally the movie chosen is not in the training and there is not much description online; the LLM would have to be instructed as well to derive its summary from image and audio processing only.
The challenge is that in order to do this you have to look at dialogue and actions and expository camera shots and assemble all of it into a model of what the characters are doing and why they are doing it.
Marcus wasn’t wrong about one thing: this is a good capability test, the sort of thing that most people in 2022 would have agreed was substantial and meaningful challenge that was likely well over a decade away.
So, can AI do this? Has this item fallen three years ahead of schedule? Let’s see.
The choice
I think there is a clear fair use argument that I can have an AI agent I run “watch” any film I have on DVD for the purposes of summary. I also know there is a horde of publishers right now looking to dramatically curtail fair use rights.
So I decided in order to make this replicable and uncontroversial I’d choose a public domain film. I asked ChatGPT to look at a list of films on the Internet Archive from 1928 to 1932 and return the ones which could not be recounted from memory.2
I choose Midnight Mystery (1930).
As you’ll see, this level of caution is not necessary when you watch the LLM solve this puzzle, but people are dumb about this, so let’s take the 2024 “It’s all just plagairism!!!” argument off the table. (And let’s leave it on the floor; good riddance).
Here’s what Wikipedia has:
Gregory Sloane is a millionaire who lives in an isolated mansion on Hawk Island, off the coast of New England. He invites a disparate group of people to his home, who are soon cut off from the mainland when a fierce storm blows in. While confined, tensions erupt among the guests, leading to the murder of Mischa Kawelin.
One of the other guests, Sally Wayne, an author who writes murder mysteries and Sloane’s fiancée, takes it upon herself to solve the crime. Over the course of the evening, she uncovers and strategically puts together all the clues, culminating in her getting the murderer to confess.
We then tell ChatGPT to watch it (typos in the original):
Using no existing plot summaries and or web searches, I want you to “watch” this film:
https://archive.org/details/midnight-mystery_1930
Break it down, make as many contact sheets as you need time with time stamps. Rip the audio track and make a time stamped transcript with who said what. Spend as much time as you need to to get everything in order, your annotated contact sheets can be s detailed as you need. Then summarize the film and i will ask you about certain scenes.
The Process
ChatGPT gets started on it. The biggest issue, which I hadn’t anticipated, is the audio is quite bad on this old film. Transcribing the damaged audio was pretty difficult, and the tools it uses to do that come not from AI but from Python running on my modest laptop.
During a particularly difficult to discern conversation happening over loud piano playing, the LLM kept sending the segments of audio out to Python for processing, getting the results back, noticing that there were transcription gaps at points that the film showed people talking, and sending it back to Python with new conversation detection settings.
But as slow as that goes, it starts “watching” the film.3
Here’s most of the run, in real time.
For those unfamiliar with agents note that outside a curious question in the middle, I did not interact with this, I let it just do its thing. When people talk about agents, this is what they are talking about.
Here’s a snippet from the command line output. The LLM notes some things that might be significant later (a character talking about digestive tablets, the main detective noticing something in a dead characters hand) and logs them. You can see the iterative process: “listen” to the dialogue, take snapshots of the film, try to understand what the dialogue and the image might mean together:
The breakfast dialogue mentions digestive tablets, so I’m indexing that alongside the later medicine-cabinet and drinking-glass shots. I’m also keeping uncertain breakfast speaker assignments marked; the central Gregory–Sally–Tom exchanges are much clearer.
• Explored
└ Read transcribe_complete.log, transcribe_resolution_check.log
• Viewed image hand_clue_01.jpg
The closer hand sequence shows Sally examining material from Misha’s palm and then looking at her own hand. I’ve refined the annotation to describe that action, rather than treating the dark shapes as definite marks on his skin. The ending check will help establish what the material means.
With the transcription (which is about 90% of the time) it takes about ninety minutes. At the end, we get this:
I used **no existing summaries or web searches**. I examined sampled frames throughout the film, with denser contact sheets for important scenes. This session cannot directly hear audio, so dialogue comes from local speech recognition and contextual speaker identification. The transcript contains uncertainty markers and unresolved speakers; it is not a verified verbatim transcript.
**Film summary, including the ending:** A storm isolates a house party on an island off Cuba. Mystery novelist Sally’s fiancé Gregory resents her devotion to writing. He and his friend Misha stage a murder using a dummy, hoping to frighten Sally and her crime-loving friends. Sally offers to escape with Gregory, then angrily rejects him when he reveals the prank.
Criminal lawyer Tom discovers the deception and exploits it. Misha is having an affair with Tom’s wife, Madeline, while Tom wants Sally. With Gregory already publicly claiming responsibility, Tom kills Misha using Gregory’s gun and tries to make the staged murder appear real.
Sally finds a button in Misha’s clenched hand and identifies its origin on the living-room couch. Tom’s accidental reference to that couch reveals his knowledge of where Misha died. To overcome Gregory’s incriminating confession, Sally stages another deception: she convinces Tom that whiskey he drank contained poison.
Terrified of dying, Tom confesses before witnesses and reveals where he hid the gun. Sally explains that the drink contained a sedative. Her trap applies Tom’s own earlier theory about fear provoking confession. Gregory finally accepts her detecting and writing; they reconcile.
Asked for a more detailed chart, the LLM produces this:
You can see the full chart here. You should really click through and look at it. It made this purely by looking at the film, inspecting the on screen actions, using those actions and that dialogue to understand motives and methods. It’s frankly stunning. For example, the LLM is able to connect an early statement by the murderer that people are more likely to confess when they think they are dying to Sally’s later plan to make Tom believe he has been poisoned so a confession can be elicited. The theory of mind demonstrations, cobbled together from images, dialogue, and expressions is explicit.
We live in a time of wonders, and I feel sorry for people without the curiosity to see that
I think this is pretty cool. I don’t see how people can’t see this as cool.
AI is going to be responsible for a lot of bad things, no doubt. It’s going to be responsible for those things for the same reason it does amazing things above: it is extremely capable, of things that critics thought were science fiction a mere four years ago. I think people should be very worried about its adverse effects. But I think people worried about its ill effects might consider learning its capabilities so they can use it for good as well.
None of this changes the fact that through DOGE Musk is going to be responsible for millions of deaths. It should go without saying that this post is not an endorsement of Musk. (Note added for those with “goes-without-saying” blindness).
Asking this sort of question does not work reliably on early versions of ChatGPT but started to work reliably in the last wave of updates. I also asked it to describe the film and checked to make sure there was not a detailed synopsis available through search. As you will see, nont of this really matters because you can actually watch the process it uses to watch the film in real time. But it’s very fashionable to claim in the face of these capabilities that this is all just a form of plagairism, and I am taking that off the table as much as possible.
Save me the anthropomorphism speech; if dogs can “watch” TV, an LLM can “watch” a film.




A movie from 1930 is going to have a very talky, radio-like script. Give it Five Easy Pieces and see what it comes up with.
Using a public domain movie is avoiding the real problem all together: there’s no _legal_ way to have a cloud-based model watch a movie. Every conceivable method runs afoul of copyright and anti-piracy best practices. I work on qualitative analysis of media and we’ve been struggling with this for over a decade.