Arc: Scene-Based Search With AI

It's a research project but frankly I also just wanted a tool like this

(Just take me to the tool already)

I’m too tired to explain this deeply, but I just wrote a tool that gets you scene-based information about films. It’s partially because I want to experiment with mitigations around hallucination and conflation, and film and TV shows (as I’ve shown recently) are a place where LLMs tend to conflate and hallucinate. So it’s a chance to play around with that with a real tool.

But it’s also a tool I’ve just wanted for a while. My wife and I joke sometimes that half of our relationship is watching movies and asking each other “Hey, do we know this guy on the left?”, occasionally jumping online to determine where we recognize the actor from.

When you try to find that out it’s surprisingly hard. You type in the film and you pull up the cast and you see the faces of people that are 40 years older or younger than they were in your film, and you see all the cast from across the entire film and now you’re clicking in and poring over IMDB photosets like they’re the Zapruder film, trying to figure out if the person in the scene is this person or the other person. (Are those the same eyebrows? Is that the same nose?)

So I wrote a tool where you type in the name of the film and a little bit of description of the scene

and then locate the scene:

And learn in this case that actually you don’t recognize the “punk” on the bus in that scene, but that he was actually a producer of that film, and went on to write Muppet Treasure Island:

Or you can find in this case in Gattaca

if you type “nurse who pricks baby’s feet after born” and click identify:

That you weren’t crazy: that nurse really was SNL alum and Rentals harmony and keyboard sensation Maya Rudolph. (Man, that targeted Muppets advertising hit fast, didn’t it?)

There’s some other little features here I like — you can click the little “check” in the top corner if you want to check something that is flagged as uncertain, or do it even on certain things to get more information on a character, like here where we delve into/double check that that really is Jesse Ventura in Running Man. Here’s before:

and here’s after — with just a bit more information, just a bit more double-checked:

Note how it introduces clarity on what he is seen as at this point in the film, describing his importance to the scene you’re watching.

And of course you can use it to find episodes of TV shows, like here where we retrieve one of the best Lost episodes of all time based on the best subplot on that episode.

Anyway, check it out. There’s some deeper thinking here with my whole conception of search as overlay, but I’ll leave that for later.