This is radical and healthy way to do it. Obviously wrong — reject. Obviously right — accept. In any other case — also reject, as non-obvious.
I guess it is far removed from the advertized use case. Also, I feel one would be better off having auto-complete powered by LLM in this case.
Seriously. I've taken to thinking of most submitters as adversarial agents--even the ones I know to be well-meaning humans. I've seen enough code that looks obviously right and yet has some subtle bug (that I then have to tease apart and fix), or worse, a security flaw that lies in wait like a sleeper cell for the right moment to unleash havoc and ruin your day.
So with this "obviously right" rubric I would wind up rejecting 95% of submissions, which is a waste of my time and energy. How about instead I just write it myself? At least then I know who's responsible for cleaning up after the it.
Auto-complete means having to babysit it.
The more I use this, the longer the LLM will be working before I even look at the output any more than maybe having it chug along on another screen and occasionally glance over.
My shortest runs now usually takes minutes of the LLM expanding my prompt into a plan, writing the tests, writing the code, linting its code, fixing any issues, and write a commit message before I even review things.
I don't find this to be the case. I've used (and hate) autocomplete-style LLM code generation. But I can feed 10 different tasks to Codex in the morning and come back and pick out the 3-4 I think might be worth pursuing, and just re-prompt the 7 I kill. That's nothing like interactive autocomplete, and drastically faster than than I could work without LLM assistance.
> Obviously right — accept.
I don't think code is ever "obviously right" unless it is trivially simple