I’ve been thinking about a concept that I’ve been calling “Proof of Thought” for a while now. It’s the idea that certain processes in the modern workplace require that someone thinks about them for a while, and the best way to prove that they have done so historically is through written reports. You know the type: project plans, strategy documents, self-evaluations. The kind of documents that no one will ever read in their entirety after they’ve been written, but that still need to be well-written, relevant, and include enough plain English to make sense.
This system has been in place and worked more or less well enough for my entire career… that is until the recent rise of large language models (LLMs) and generative AI. Suddenly, for the low low cost of seconds of your time, you can have a report that passes all of those checks and definitely looks like someone put thought into it. But the thing is, they didn’t. And that’s a problem.
These Forged Proofs of thought look like the real thing. They pass all of our checks for a high quality version of these documents. But, as with other forgeries, when you go to cash them in you find they’re worthless. There’s a gap in the chain where thinking was supposed to happen and the whole thing falls apart.
Role of Documentation as “Proof of Thought”
Before I go into the implications of this idea and what we should do about it, I want to make sure I’m being super clear about what I mean when I say Proof of Thought.
Project Plans
One of the first things a project manager (PM) does when starting a new project is put together a project plan. This is a document that describes all of the things that need to happen over the course of the project and who is going to do them. In my experience, these plans are outdated basically the moment they are written and no one ever goes back and updates them. This is fine. The point of these documents is not to be a perfectly accurate representation of how the project is going, the point of these documents is that they force the project manager to think through the entire project from start to finish, think about all the ways it can go wrong, and expose any parts that are particularly fuzzy and undefined so they can start tracking down answers to those open questions.
Once it’s written, all of that work to generate it lives on in the PM’s mind. They mull things over in the shower, it keeps them up as they’re trying to fall asleep, and it becomes a part of the general world of things the PM knows. Coworkers will probably skim over it looking for their name so they know how much work they’re going to have to do, some parts of it will probably end up in Jira, and the rest will sit in that report until the end of time, a testament to a reality that only existed for a few hours before timelines shifted and stakeholders got aligned.
Self Evaluations
Self evaluations are another example of Proof of Thought. Once a year, employees are asked to write a document evaluating their own performance over the past year. The point of this document is not what they say - their manager is already aware of their accomplishments and setbacks. The point of the document is to force the employee to critically reflect on the past year, internalize the reality of their performance, and come into their performance review ready to discuss the upcoming year.
The danger of missing the point
It’s easy from the perspective of the person writing these documents to think that the point of what they’re doing is the document itself. “My manager told me to write a self-review,” they say. “Clearly they want a self-review. And the better a review I write, the happier they will be.” That’s been close enough to true until recently that the distinction hasn’t mattered. There’s been no way to write a great self-evaluation except to think really hard about the past year and write everything down in a logically coherent structure. If you answer all the questions in your project plan template, you’ve probably got a pretty good grasp of what needs to happen.
On November 30, 2022, with the release of ChatGPT, everything changed. We now have a way of generating a project plan that looks exactly like the real thing, without anyone involved in the project ever sitting down thinking about it. The thing that was supposed to be in the back of your mind isn’t there and you’re more likely to get halfway through and be surprised that no one’s talked to marketing about how we’re going to launch this thing.
Challenges Posed by LLMs
The problem with LLM-generated reports is that they can create a “Potemkin village” effect. On the surface, the reports may appear well-written, logical, and comprehensive. They look really good. All of the context cues that came with human-written content when it was phoned in are gone. But, like a Potemkin village, they are just a facade built to deceive. No thought on the part of the author has occurred, no matter how convincing the proof.
I want to take a minute here to have a non-opinion about a potentially controversial topic. Nothing in this essay hinges on whether or not a sufficiently advanced AI counts as thinking. One of the elements of Proof of Thought documents is that they are not proof that someone has thought about this topic, but proof that you have thought about this topic. For now at least, LLMs are not autonomous agents, and will not reach out to marketing to let them know that we need a launch campaign. Asking them to list some generic strengths and weaknesses will not emotionally prepare you for a negative performance review.
Implications in the Workplace
I preface this next part with the disclaimer that I have nothing against LLMs as productivity tools and I think we have barely scratched the surface of how they will be integrated into our daily lives until they are as ubiquitous as email. That being said, the flood of articles over the past few years about how LLMs are going to save so much time because they can write your strategy report for you are exactly missing the point.
The point is not, and has never been, the strategy report.
If someone asks you for a strategy report they are asking you to think very deeply about a topic. If instead, you spend 5 minutes asking an LLM to write the strategy report for you, then no one has internalized that knowledge. If you tell them that had the LLM write the strategy report, they will assume you can’t or won’t think about things, and won’t ask you to do things that require thinking again. Maybe you work in the kind of place where no one is thinking about things anyways. I personally would still not try to cultivate a reputation of not thinking about things.
Exploring Alternatives
“So what,” you might be asking yourself, “we just can’t use LLMs to generate Proof of Thought documents?” That’s a great place to start, but it doesn’t go far enough. I make it my personal rule to try to always do my own thinking, but I can’t expect that everyone else will always do the same. It’s too quick and too easy to ask a computer to do it for you.
I propose that we need new Proofs of Thought.
This same story is playing out in our classrooms, as educators are appalled that students are using LLMs to write their book reports. The reason a book report has been required for all these years is that it is a hard to forge and easy to verify signal that the student had read the book. That signal is no longer hard to forge, so we need a new one.
Some options come to mind, but most of them fail on the “easy to verify” point. In-person presentations and discussions, detailed review of submitted documents - these things will work to prove that thought happened, but the time cost on the verifier is too high - they might as well have just done the thinking themselves.
There’s a part of these documents that might still be valuable - “what’s next”, “unknowns”, “new questions I had that I didn’t have when I started”. Learning things can often leave you with more questions than when you started and a genuine curiosity is hard to fake.
At my most optimistic I can imagine a high-trust workplace where the “proof” isn’t necessary. If someone needs to think about something, they just do it, and produce whatever artifacts are most useful to communicating afterwards. Most of us don’t work in that place though and there’s no guarantee that a high-trust environment would stay that way once established.
Conclusion
The rise of LLMs and generative AI poses a significant challenge to the traditional system of using documentation as Proof of Thought in the workplace. However, this challenge also presents an opportunity to develop new and innovative solutions.
Whoever figures this out will have the Toyota Production System of our generation. As the rest of the world falls behind because “this is the way we’ve always done it”, you will be sure that your teammates are still thinking deeply and critically about the things that are important in their work.