I am not able to find it now, but there was an amazing story recently from the 60s or 70s where an engineer was in exactly this position. His team was building a new, complex, ambitious operating system, but it was late and over budget and didn’t work. It nearly wrecked the company. He talks about hitting rock bottom and asking himself what went wrong, and one of the fathers of computing (can’t remember which) shouts from the hallway in passing, “that’s easy, you didn’t understand what your people were doing.” So the guy turned it around by implementing a new rule: he had to understand every line of code his team wrote. They started over with the company’s existing OS in use by customers and implementing a few of the most requested features. Much less ambitious, but it actually shipped. Gradually they achieved all their goals by upgrading the existing system.
The “I must understand every line” constraint didn’t sound like a power trip that succeeded because the guy was such a brilliant code reviewer. I think it was a blunt instrument that enforced simplicity.
I guess what I’m saying is, I reject the premise of having technical oversight without writing or reading much code.
As with anything. Either you can go full-speed without much understanding and hit a wall when you need to understand stuff or you can go a manageable speed and actually understand the codebase.
I don't think we can do both. The difference is that it's optional now depending on the project and the audience.
I never said you wouldn't have to read code. I was asking a question to get answers about how people would achieve having an oversight if it was humans writing code that they were managing rather than agents.
As for your suggestion, understanding every line might have worked in the 70s but even pre-agentic modern coding it's not possible for any large project with dependencies even if you are directly contributing code yourself, so I'm not sure how useful your idea is.
There's definitely cases where you should have that aim - writing a low level maths or graphics library, for example. But most people are not doing that.
Perhaps my "reject the premise" comment was a bit too provocative. I didn't intend to start an argument. I wanted to share a story of a person who was put in the position you described (working as a manager on a large project, required to have an overview of the project but not necessarily to actually write or review much code), failed miserably, changed the rules of engagement (partly by reviewing code), and subsequently succeeded. So my _personal_ answer to your question of how to build a mental model in that scenario would be to do something in the same vein as that story. Not necessarily _exactly_ what that guy did, but I think the principles still apply today. Nothing really changes.
Folks have been saying “things are different now, the agents are now compounding success instead of error” for at least a year now, but I just don’t see it. I was lucky enough to receive a weeklong $50k per head AI training from the people saying these things, and one of their few helpful concrete recommendations was to constantly clear context all the time, to avoid things going off the rails.
However, I think finding security vulnerabilities is one use case where it doesn’t matter. Tokenmaxxing is absolutely effective for that. We as an industry are in the middle of adopting very expensive, complex continuous fuzzers.
Even modern frontier models benefit so hugely from careful context pruning, maintenance, and rewriting to erase mistakes that it's astonishing to me that there are no tools centered around it. The one tool that used to have such a feature, Zed and its retroactively-named Text Threads, has now stripped itself of it.
this! the back-and-forth chat interface where you can edit only your own messages, and only then to get a new response, is a terrible one, but I think favored by vendors because it helps them fight in vain against prompt injection. Custom harnesses and stuff are nice but incredibly time consuming to set up when all I want to do is like, see what the agent is reading, and editing out some irelevant nonsense side quest it went on, or trim some massive log file it read which filled 90% of its context. Theres no inherent reason behind some caching gains that these things must be strictly chronological - a response it gave me previously does not have to be part of the context now
"some caching gains" is a pretty huge understatement- snipping something out of the middle of the window requires rebuilding the entire context. Thats a shitload of tokens.
Afaik messing with the context also pretty reliably degrades performance still. The model responses reference things that no longer exist to it and it becomes more chaotic.
The real usefulness of parallel or sub-agents is not that they run at the same time, its that they isolate noisy or self-contained context away from the main window.
I still feel that during agentic workload sometimes it would be nice to have the model identify it is veering off the main track, send out a "keep the cached states and tell me which they are" command to the inference server, do the side thing (such as handling an error that plopped up that has not that much to do with the main task) and return back to the cached state with just a comment tacked at the end to say "oh and btw I fixed DNS" instead of having the DNS debugging inside the context window now. Maybe other harnesses just steer the models more towards using subagents for such tasks and my pi is misconfigured. I can use the tree feature, but having insight into what's cached would be nice there.
You can structure your context window to minimize the amount of editing you do further back. You usually only need to edit and correct the most recent response. It's little different from forking the conversation at an earlier point, and nobody warns about that being a sneaking footgun. There is still a prefix to cache.
Some companies only get to a "hello world" level with a new kind of tech via a 50k per head training. The organizations are setup in a way that people can't experiment or learn by themselves, it's really the only way.
It's true, but I see it happening. I’ve watched seniors with 30+ years of experience adopt them successfully without losing their classic rigor.
Personally, I get huge mileage out of LLMs, and yes, I care deeply about code quality, readability, and debuggability.
I've seen juniors absolutely rock with them.
And I've seen the exact opposite, where they just struggle to get good results.
In the end, I think the divide comes down to management experience. The people thriving are the ones who have led teams, especially teams of contractors, which is the best analogy for how you have to interact with an LLM.
Those folks know how to break down problems, provide the right context, and scope a task just enough to see the "contractor" succeed before letting them move forward.
On the other hand, individual contributors who are used to just grinding solo often struggle. They expect a one-shot miracle. They say, "Hey, my code is buggy, fix it." When the LLM inevitably hallucinates or steers them wrong, they give up. The results are completely different based on how you treat the tool.
They might just have a high quality of control and standards that it is hard to find that pattern with the LLMs.
I think fierce individual contributors are a lot more valuable in the era of llms as well. We as humans typically achieve better balance with new stuff when we allow backlash from new processes that start to trample on old ones without understanding AKA the Chester's fence.
I’ve been finding that very rigorous/technical people who are willing to adapt their workflows are very effective, whereas more “management” types are sometimes effectively pushing work off onto their code reviewers.
Who delivers a 50K per head training? Who pays for such a thing?
How many people were in the class?
I had a small training company, shuttered during COVID, and I used to charge 5K per day, for a group of up to 12 people. 5 days training = 25K. This is double, wow.
I would love to get back into training but getting enough volume to live and support a family on has been a challenge.
He's been radio silent for 9+ months. No videos, no podcasts.
People have been speculating on his reddit subs, but honestly anything from early retirement, to stay at home dad, to sabbatical, to serious illness is completely possible. The parasociality of everything has probably gotten old over time, too.
> prompting for test-driven development is not the same as enforcing code coverage thresholds in your build tool
Are they actually different? I would guess they have roughly the same efficacy. 100% code coverage means nothing, and this is especially true with LLMs.
I think one lesson I'm taking away from the article is that we're not so much seeking "weird for weird's sake", but expanding that weirdness into direction that's useful, to specifically highlight the novelty of something.
Especially without being able to look at the rendered output! (At least I'd be surprised if modern server-side tool calls regularly include an SVG renderer that can show a rasterized version to the model to iterate on it.)
One of the many things Google was pitching today is that they're going to run things like google search with access to linux container environments to do things like run tool calls... which will presumably be able to rasterize SVGs and show them to the model.
But Simon says he runs these through the API without tool access specifically to prevent that sort of "cheating". I.e. it's an LLM benchmark not an LLM+Harness benchmark.
I’ve never understood engineers who complain about vague specs… if the spec was complete, it would be code and the job would be done already! Getting a 20 page spec delivered from upon high and mechanically translating it to code without any chance to send feedback up the chain sounds like… a compiler.
In my experience, the complaints are not about the specs and their vagueness. It's more about the political game to get them detailed. If you've not encountered the kind of organizational issues where getting an answer is like pulling teeth, you're kind of lucky.
Oh no, I’ve definitely experienced that, it’s terrible. But that situation makes me wish for more agency (for example, talking to customers directly), whereas it seems to make other engineers wish for less agency (please hand me a complete spec and I will mindlessly translate it to code). That’s what I don’t understand.
some of us couldn’t give a rat’s ass about the customer. One of our customers charges people for paying their own bills via certain methods, which is completely bogus and I remind everyone loudly all the time that they do this. Everyone agrees that this customer sucks to work with, and the less time spent with them the better.
The people from the customer’s end suck, they’re not technical, they have in-fighting with their own teams during calls, have decades long errors with their integration that they have never fixed…the list goes on. For this customer and a few others, please give me a spec that I can implement, shove it back across the aisle, and forget about. The absolute last thing I want is to have to talk to them more.
Yes, I don't think a job where I am programmed by a product manager would be terribly interesting. I would move on to be the product manager if I found myself in such a role.
Reminds me of my favorite math essay: "When is one thing equal to some other thing?"
It's a great question, much deeper and more interesting than it seems. The essay suggests thinking in terms of isomorphisms (relative to the structure you care about) rather than equality in some absolute sense, and I've found a fuzzy version of that to be a really useful perspective even in areas that can't be fully formalized.
Mathematics is all just explanations for why this is really that. If it didn't have to respect its human audience, and their failure to grasp similarities, the whole edifice could be one implicit statement. (After all, since this is really that, there is no this or that.) So mathematics is about people.
I jumped to a similar conclusion right away and popped over here to comment only to find you have beaten me to the punch. I use to keep a work wiki page of common problems the team encounters over and over again.
Years ago, I stumbled upon the "idea" was already debated in other fields long before programming. Lumpers and Splitters.
