[Me & X] Epistemic debt
X : How many executives are looking at code level detail at any sizable organization? Is this so different from working with AI?
Me : The executive who doesn't look at code level detail is relying on a chain of people who do. The CTO can read code. The engineering managers came up through the ranks writing it. The senior developers review the juniors. Somewhere in that chain, even when it extends to outside vendors that comprehension exists or is at least notionally recoverable. Yes, we still end up with the legacy problem and whilst many organisations find that comprehension hard to recover, we still can and still do.
In this light, the executive is outsourcing attention, comprehension still remains in the chain.
The concern with LLM/GPT generated code isn't that developers are delegating because developers have always delegated to libraries, frameworks and even compilers. It's that the delegation has extended to comprehension. There is no chain, just systems that cannot explain their reasoning beyond generating plausible sounding explanations for code it has already produced.
The reasoning, the understanding is lost. And without that, so is learning.
That said, it is fine for an entire culture to decide that producing outputs matters more than understanding mechanisms. You only have to compare the practical engineering of the Roman Empire and the loss of inquiry from science in the Hellenistic age to see this. When the Roman Empire collapsed, the practical knowledge embedded in those institutions (how to maintain aqueducts, how to produce certain grades of concrete) was lost remarkably quickly. Not because people decided to forget it, but because the knowledge was procedural, embedded in chains of practice rather than recorded as transferable understanding. When the chains of practice broke, that embedded knowledge went with them. We had to rediscover the art of inquiry (i.e. Science) to bring them back.
Today, much of the market economy is built on chains that no-one comprehends as companies compete in landscapes they have little understanding of. Markets optimise for outputs, not for understanding. There is lots of embedded practice and it works!
It will also continue to work, right up until the environment shifts and nobody remembers how to reason from first principles. In the UK, we've had several early warnings with severe supply chain shocks and even close calls from Brexit, from COVID and from the Ukraine conflict. We're living in a highly fragile system and we've extended this lack of comprehension into the entire software chain.
We are building brick upon brick of epistemic debt, promising a future where success no longer depends upon anyone being able to reason about it. With vibe coding, we've even gone so far as to celebrate it.
As Shaw said "Beware of false knowledge; it is more dangerous than ignorance" ... we are building towers of false knowledge whilst comprehension and understanding crumble.
Originally published on LinkedIn.