I just spent four hours writing a single prompt. The challenge for me was this: I took a list of bunch of movies that to me had a certain “Jaws-ishness” and a list of movies that I felt didn’t. Then I tried to write a prompt that would correctly sort them based on my viewer’s sense of them.
If you haven’t done this, this is incredibly hard for at least some films, and probably harder if your sense of the film is somewhat subtle. I started off with this:
A community or group confronts an escalating catastrophe where those at risk initially fail to understand the threat and victory can only be achieved against overwhelming odds. The film’s style plays with our own doubts in the first half before layering disaster on top of disaster on the back end leaving the thinnest of margins for survival. It has the edginess and some of the form of a paranoid thriller without being a paranoid thriller.
It sucked. How do I know it sucked? Because it let some of my outs into the in set and put some of my ins out. And then on the 250 random it was about 50% good on what it tagged in.
I tried thing after thing on this. Put lots of style adjectives in. Noted the way we don’t see the shark for much of the film (was that it?). The sunlit horror (was that what distinguishes it from deep horror?) PG/PG-13ishness?
It just kept failing. I asked for a bunch of descriptions of Jaws and finally in one I saw the term “duel” and that struck me.
Oh!
Ooooohhhhh!
Jaws is not a fight against zombie hordes. It’s not an adventure with series in which a villain in an episodic structure. It is an extended movie long duel seen almost purely from a single side. That side we see is anchored on a single person or a very small group. Once you see a duel as a broader structure, as something that can happen with a machine, transported nitroglycerine, the elements etc., the exemplars and counter-examplars seperate out cleanly.
I won’t show you the full prompt; I’m still tweaking it to be honest. I’ve got to think about a clear way that certain types of horror stay out. I’ve got to think if I think Old Man and the Sea should be an out or an in. But I’m hoping soon to have a decent prompt with a good recall and precision matching what I want to capture.
Of course, the real gift of this is in running pass after pass on this I thought much more deeply about film structure and learned about a lot of films that I might otherwise not have known about. (I just finished watching the throughly enjoyable Stacy Keach/Jamie Lee Curtis film Roadgames). I did some deep and productive thinking about how a duel (seen broadly) is a very specific sort of structure, and along the way also learned a lot with my false starts in trying to describe the Jaws visual style (which turned out not to be the point at all).
This has been my film project in a nutshell. I spend time trying to get the LLM to do various categorization tasks, that pushes me to think more deeply about film structure, plot tropes, and general vibe. In doing this I’ve learned an incredible amount about film whether it’s Jaws-ishness or investigating the late 1970s trucker protagonist boom:
All well and good. But I want to tell you something weird that happens when I relay this project to people. Immediately, with a unison that approaches the characters in Manchurian Candidate saying "Raymond Shaw is the kindest, bravest, warmest, most wonderful human being I've ever known in my life" the response comes back
“Ah yes, but you see you knew something and that’s why it works for you.
That’s the problem with AI: it only works if you already know things.”
Wait, what?
The people saying this are not foolish people but this is a foolish response. It gets things backwards and I don’t understand why smart people can’t see it.
Imagine I start a project in a high school math class to build, in a physics simulation tool, a virtual building that withstands a certain speed galeforce wind. To do this let’s say the students have to master certain equations and master certain concepts about physical simulations. My class goes at it, learns the equations and the methods, an then we have demo day where they show if their buildings hold up.
We bring the parents in to see the simulations. The students buildings hold up. Most parents are impressed, but one isn’t. “Bah,” they say. Sure the buildings held up, but the whole thing is a sham, after all
“The thing is the students had to know something to make it work.
That’s the problem with physical simulation software: it only works if you know things.”
Is this an insightful comment? I don’t think so. I’d say it was the opposite.
You have a system that rewards learning things and that’s a problem how?
To understand how turned around this comment is consider a simulation system that does not distinguish between a person that can’t add at all and a person that has advanced math knowledge. Is this system, which requires you know nothing, a better system for the classroom? Of course not. Such a system would be useless. The system you want in a classroom is one that allows you to amazing things if you buckle down and do the intellectual work.
Which is why whenever I hear people saying “But you need to know things to use it” I can’t help thinking “What a gift!”
The problem people are actually reacting to is not one of learning but of assessment of learning. When people say such things they refer to the problem that the system can produce outputs that seem amazing but aren’t. But that’s not a problem with the system, it’s a problem with your asssessment products. It’s your student outputs that aren’t differentiating. The outputs have become the physics simulation that doesn’t care if you know math, the machine where you don’t have to know things to do things.
But note my example with Jaws. In the end, I need to develop a definition that is coherent and consistent and can adequately sort films into the predetermined piles. As the critics admit, I cannot do that without knowing things which means I have to get on learning things. And I do. Even if it takes hours to come to the fact that the hidden sorting rule is it’s a duel, stupid.
I know the goalpost moving that comes next. Well, it goes, the real problem is you students have to know a lot, and students come in learning a little. It’s hard to get from a little to a lot. Again, this is education 101. I think this part is just talk, and people aren’t really this obtuse, but of course instructional design is about thinking through how to design tasks in the Zone of Proximal Development.
The ZPD is a fundamental concept. It is not new. It is not unique to AI. You design tasks in such a way that students can get benefit by learning something within their reach to learn. To do that you investigate the task environment, in this case the LLM with whatever data you have in it, and think about ways that students learning something relatively constrained could improve what they could do with it.
For instance, that truck driver boom chart is built off a convenience data set, but it isn’t all films and it’s not set up to make these kind of claims.
What would it need to do that? Well, it depends on what you’re looking for. If you are saying these films were popular, and that’s the point, then maybe you want to reduce the size of the sample to include only films on 500+ U.S. screens, or films that made a certain amount of money. Can you get that data and prune the set? Or if your interest is less in popularity, and more what Hollywood greenlit, maybe you want to enrich and expand the set, capturing even the small independent films. Maybe an interesting point of comparison would be other countries.
Are you in a class that teaches students to think through data? Are they capable of thinking these thoughts and engaging in this sort of analysis with some guidance? That’s the zone. Or the Zone, if you will. (Little bit of ZPD humor there).
The actual catch is to understand how to teach with it you have to learn to use it
The catch with all of this is not that students have to know something or that the ZPD is not something one can provide. The actual catch is to learn to teach with it you probably have to learn to use it first.
And this is an actual catch! Teachers are not given time to learn these things, and the training teachers usually get is of the sort of “Here’s how I built a sociology chatbot.” And that’s all well and good, but I really do not think it’s the most productive way to think about using this.
The key is you have a tool where if you learn things (Singular Value Decomposition, the Toulminian structure of argument, data cleaning techniques, a history of redlining) you can, for the first time, turn that knowledge rather immediately into practical things, given a bit of imagination (in order: a recommendation system, an argument analyzer, a queryable database to research a film question, a social dynamics simulator). This is possible because people have invented an operating system that runs on words and concepts, so having a command of words and concepts can translate into cool abilities. There has never been a cooler time to be a person learning words and concepts.
That’s not what teachers are being shown, and of course with the current techlash moral panic1 many people are being scared away from learning it. But that’s the opportunity. To not be introducing our students to these possibilities and new ways of applying their knowledge seems to me almost criminally negligent, but I don’t think the negligence is on the part of the teachers, but a system that continues to model these as ways to shovel information into students rather than engines of exploration.
It is 100% a moral panic. Yes, moral panics often have a kernel of truth under them. But if you want to take bets on whether removing tech from classrooms is going to restore the things on the scale that people claim it has taken from them, the bookie is open.
In thinking about these challenges, I continually return to the framework I developed for teaching writing well-prior to the appearance of ChatGPT, that the goal of the writer is to build a "practice." A practice consists of skills, knowledge, attitudes, and the habits of mind of writers.
I've run into similar challenges around what students "know" because this frame deemphasizes what others call "the basics," essentially the mechanics of sentence making and paragraph and artifact structure. How can I expect them to write a coherent analysis or argument if I haven't explicitly taught them elements and terms and required them to do proto versions of some larger whole?
What I've realized over time is that the skills, knowledge, and attitudes that I'm trying to inculcate are really in service to developing the habits of mind that allow them to think and act as writers, or writing problem solvers, as I sometimes say.
What you demonstrate with your Jaws example is a well-developed practice when it comes to your habits of mind which allows you to build knowledge and hone your analysis through your own impulses. This is perhaps the top goal we should have for any student, and it's clear that in the hands of someone with a well-developed practice, the tools can be very useful to do "amazing things."
(Full disclosure, I understand why the classification tool is amazing and useful, and there were times my life when I did other kinds of work that it would've been something I dove into with both feet, but now, it means little to me, personally. My concern when it comes to some of the AI enthusiasm is that we are becoming over enamored with the amazing things LLMs can do in a way that make us less appreciative of the amazing things humans can do by their lonesome.)
You nail the challenge going forward. It really is assessment. I had students who were clearly advancing in terms of their writing practices, particularly their habits of mind, but where the evidence was not necessarily apparent in their written products. They hadn't had the necessary reps yet. In many cases, a semester is not a sufficient period of time to make the transition from following a rules-based template in service of a grade on a written artifact, to deploying a practice which allows you to understand the rhetorical situation of a particular occasion, analyze the audiences needs, attitudes, and knowledge around that situation, work through an effective structure for meeting that rhetorical situation and audience, AND FINALLY, deploy the specific choices of words which meet those demands.
I used alternative grading methods in order to surface student progress to themselves and me, as the person who has been in conversation (and communion) with them over the course of the semester, but as I've been told repeatedly when I try to make the case for this approach inside of systems of schooling, this progress is not legible to outsiders, so it will not find favor.
Until we have the big picture reckoning over the difference between schooling and learning, we're going to keep bonking our heads against this, IMO.
I hope you are comparing me to the John Frankenheimer version of the Manchurian Candidate ;-) I always enjoy and respect your thinking, though here just quibble with your analogy. When you are working through your four hours of prompt shaping, how are you analyzing if it’s good? How do you know if the results are Jaw-ish enough to guide your next iteration?
In the case of the physics simulation, how do you evaluate if your run was successful? The house stood or was blown away. Pretty clear.
It seems less about if it’s AI or not and more about the process of trying variations, testing the results against what you know of movies or properties of walls. That’s good thinking. Or you just keep randomly changing variables with no regard for physics or plot until it returns a result that “looks right”. This is what I dig in Johns better comment below about developing a practice. Doesn’t seem at all about AI or not.