Comment by criemen

I'd expect that's manageable by some sort of agent-of-agent pattern. You have a high-level planning instance that calls upon fresh LLM instances (new context window!) for executing more targeted tasks or bug-fixes.

Currently, an LLM with everything under the sun in the context window behaves rather poorly and gets confused by that, even if we're not exceeding the context window length. Although it'd be certainly also interesting to train for increasing the maximum _actually_ usable context window length, I don't know how feasible that would be.