Comment by blinkymach12

It's not just understanding the codebase, it's also stylistic things, like "use this assert library to write tests", or "never write comments", or "use structured logging". It's just as useful --- more so even --- on fresh projects without much code.

I suspect machine readable practices will become standard as AI is incorporated more into society.

A good example is autonomous driving and local laws / context. "No turn on red. School days 7am-9am".

So you need: where am I, when are school days for this specific school, and what datetime it is. You could attempt to gather that through search. Though more realistically I think the municipality will make the laws require less context, or some machine readable (e.g. qrcode) transfer of information will be on the sign. If they don't there's going to be a lot of rule breaking.

I think they'll always need special guidance for things like business logic. They'll never know exactly what it is that you're building and why, what the end goal of the project is without you telling them. Architectural stuff is also a matter of human preference: if you have it mapped out in your head where things should go and how they should be done, it will be better for you when reading the changes, which will be the real bottleneck.

I agree with this general sentiment, but there might be some things you want to force into the context every time via a specific agent file.

Not at all. Good documentation for humans are working well for models too, but they need so much more details and context to be reliable than humans that it needs a different style of description.

This needs to contain things that you would never write for humans. They also do stupid things which need to be adjusted by these descriptions.

One of the most common usages I see from colleagues is to get agents to write the comments so you can go full circle. :)

Unless we write down on the what we often consider implicit the LLM will not know it. There might be the option to deduce some implicit requirements from the code but unlikely 100%. Thus making the requirements explicit is the go.

Yes! That was precisely my point here: https://news.ycombinator.com/item?id=44837875

Better to work with the tools we have instead of the tools we might one day have. If you want agents to work well today, you need to build for the agents we have today.

We may never achieve your future where context is unlimited, models are trained on your codebase specifically, and tokens are cheap enough to use all of this. We might have a bubble pop and in a few years we could all be paying 5-10X current prices (read: the actual cost) for similar functionality to today. In that reality, how many years of inferior agent behavior do you tolerate before you give up hoping that it will evolve past needing the tweaks?

> We're in a transition phase today where agents need special guidance to understand a codebase that go beyond what humans need. Before long, I don't think they will.

This isn't guaranteed. Just like we will never have fully self-driving cars, we likely won't have fully human quality coders.

Right now AI coders are going to be another tool in the tool bucket.

This is mostly true if the existing codebase is largely self documented, which is rare

This applies to mcp too

Here's a prompt I wrote a few days ago for codex:

  Analyze the repository and add a suitable agents.md

The core problem is that every conversation is like ground hog day. You always start from scratch. Agents.md is a stop gap solution for that problem. Chatgpt actually has some notional memory that works across conversations. But it's a bit flaky, slow, and limited. It doesn't really learn across conversations.

That btw. is a big missing piece on the path to AGIs. There are some imperfect workarounds but a lot of knowledge is lost in between conversations. And the trick of just growing the amount of context we give to our prompts doesn't seem like it's the solution.