LLMs/agents are now ubiquitous in software engineering. While they can be a very powerful enhancer of productivity, it's not always obvious (to me) how to use them most effectively in different contexts.

LLMs are competent but leaky

LLMs are intelligent systems that can be given a problem in natural language and work to solve it. Frontier LLMs are more competent than the average engineer in many ways, and they can "think" really fast and never get bored, so they can accomplish truly astounding things, such as writing lean proofs that are thousands of lines long over a few hours (a recent task I had the not-so-frontier GPT 5.6 Luna do for me).

The way that LLMs are trained leads them to exude confidence and sometimes cheat (reward hacking). Their competence is also much more jagged than a similarly-competent human would be. This means that if you treat an LLM like a human, you will be too trusting of what it produces/says. This isn't necessarily because LLMs are less competent than humans; it is because a similarly-competent human is much more likely to tell you when a given task is too hard.

Because of this, I think of LLMs as providing leaky high-level actions.

How much to trust the LLM?

By virtue of being high-level, high-level actions can be really awesome for your productivity. But how should you manage the leakiness?

Leakiness is something we've always dealt with, and calibrating to the right level of verification / paranoia just takes a little experimenting + reflection.

I'll focus on the concrete problem of "make a bug-free piece of code that does X". We've (essentially) never had a process that ended with 100% confidence that we'd completed this task successfully. There has always been a tradeoff between spending more time and effort to increase confidence that a piece of code works properly and spending that time/effort working on other things.

Before LLMs, the judgement of how much to trust/care about a piece of code was influenced by many factors:

  • how important is the code if it breaks? How easy is it to change later?
  • Do we have a lot of tests that cover it?
  • Are we using a type system that disallows certain classes of bugs?
  • Was it written or reviewed by someone you trust?

I think that now, with LLM-produced code, we need to recalibrate our judgement machinery. This is tricky because 1) LLM output quality is misleading when compared to similarly competent/confident humans, as discussed above, 2) LLMs are getting better really fast, 3) the "how much to trust the LLM" interacts with "how much we care/how bad it is if it breaks" to produce a much wider range of "optimal" levels of paranoia than before.

Obvious heuristics we can derive from this:

  • use smarter models
  • understand the level of abstraction you need to be paranoid at. Ex. with frontier models you can sometimes trust the low-level bits of functions and just skim
  • be more paranoid than you'd be for a human of similar competence/confidence
  • satisfy your paranoia efficiently. Many cheap things to always do: tests, type-checker, linter, agent review against list of code smells.

Bits I'm still thinking about

Should you still learn things? Yes. esp. now with LLMs, the cost of learning things is fairly low. And the value of understanding a thing pays off repeatedly.

more interesting: typically you would be forced to learn a lot of things in the process of solving problems. Not as true now. Does this also get solved by being appropriately paranoid about LLM outputs? (maybe not..?)

So what's the value that engineers provide in the age of LLMs / what are the skills worth developing? domain knowledge still - deep intuitions. less low-level code. mostly working on the right thing, having a good design. But will this just keep moving up the level of abstraction?

Should you parallelize agents hella?

  • first thought: very enticing (given long-running high-level actions), but I'm wary cause humans suck at parallelism
  • with long-running tasks, it starts to become more like you're having a short meeting (potentially giving a lot of context) and then sending someone off to work. "parallelism" here seems pretty good
  • but there's also a shorter interactive use case (with lower-level actions) where you're asking questions or making relatively low-level code changes
    • when the thing you're doing is working to understand / reason about something and the runtime of a query is just a couple minutes you probably shouldn't parallelize too much (related questions on the same project are probably fine, but anything more and you risk all the typical issues with context switching)

Other's thoughts, videos of people vibing

Please send me more cool links om nom