Comment by llbeansandrice 2 days ago
If AI is supposed to deliver on this magical no-lift ease of use task flexibility that everyone likes to talk about I think it should be able to work with a README instead of clogging up ALL of my directories with yet another fucking config file.
Also this isn’t portable to other potential AI tools. Do I need 3+ md files in every directory?
That is strange because it's been going on since sonnet 4.5 release.
I also have seen this happen since then, but I actually like it. The .md files never make it into a commit, but they are quite handy for PR drafts.
> it is delivering massive productivity gains
[citation needed]
Every article I can find about this is citing the valuation of the S&P500 as evidence of the productivity gains, and that feels very circular
I feel similarly, but the other guy's point is still good; there is no empirical evidence that AI is really as helpful as we think, and there is empirical evidence that among those who perceive productivity gains, most are wrong and the rest are overestimating the effect.
Our experiences don't agree with the established facts, but the plural of "anecdote" is not "data."
> How would you measure productivity gains?
I'm not entirely sure - certainly anecdotes abound (as do anecdotes to the contrary).
I am however sure that when someone claims that a technology is world-changing, and that it has already led to significant productivity gains... the onus is on them to provide evidence for such claims
Should I not share my experiences then about something that can't be scientifically proven? I am not actually looking to prove anything, but I am sharing my experience. I don't even win anything from people believing me, but I guess I have this weird urge to "set something straight" if in my experience something is the case. I have done tons of DIY stuff that I wouldn't have been able to without AI. Tons of side projects. In my corpo work I have been able to do more with less hours and mental effort spent, although in corpo work AI has given me less gains since most of the time I have been blocked by other factors and everyone else.
It is too fast evolving to be able to be legitimately scientifically understood. It is non sense to expect that.
It’s not delivering on magical stuff. Getting real productivity improvements out of this requires engineering and planning and it needs to be approached as such.
One of the big mistakes I think is that all these tools are over-promising on the “magic” part of it.
It’s not. You need to really learn how to use all these tools effectively. This is not done in days or weeks even, it takes months in the same way becoming proficient in eMacs or vim or a programming language is.
Once you’ve done that, though, it can absolutely enhance productivity. Not 10x, but definitely in the area of 2x. Especially for projects / domains you’re uncomfortable with.
And of course the most important thing is that you need to enjoy all this stuff as well, which I happen to do. I can totally understand the resistance as it’s a shitload of stuff you need to learn, and it may not even be relevant anymore next year.
While I believe you're probably right that getting any productivity gains from these tools requires an investment, I think calling the process "engineering" is really stretching the meaning of the word. It's really closer to ritual magic than any solid engineering practices at this point. People have guesses and practices that may or may not actually work for them (since measuring productivity increases is difficult if not impossible), and they teach others their magic formulas for controlling the demon.
Most countries don't have a notion of a formally licenses physicist either. That doesn't make it right to call astrology physics. And all of the practices around using LLM agents for coding are a lot closer to astrology than they are to astronomy.
I was replying to someone who claimed that getting real productivity gains from this tool requires engineering and needs to be approached as such. It also compared learning to use LLM agents to learning to code in emacs or vim, or learning a programming language - things which are nothing alike to learning to control an inherently stochastic tool that can't even be understood using any of our regular scientific methods.
I think it's relevant when people keep using terms like "prompt engineering" to try and beef up this charade of md files that don't even seem to work consistently.
This is a far far cry from even writing yaml for Github/Gitlab CICD pipelines. Folks keep trying to say "engineering" when every AI thread like this seems to push me more towards "snake oil" as an appropriate term.
Prompt engineering is a real thing though, but it’s not related to markdown files etc.
If you’re not benchmarking and systematically measuring the impact of your changes, it’s not prompt engineering, it’s just improving stuff.
Well it requires a lot of planning, spec’ing, and validation.
The whole point is to get a process around it that works and gets the “ritual magic” out of it.
I labeled that as engineering.
My issue is not with learning. This "tool" has an incredibly shallow learning curve. My issue is that I'm having to make way for these "tools" that everyone says vastly increases productivity but seems to just churn out tech-debt as quickly as it can write it.
It a large leap to "requires engineering and planning" when no one even in this thread can seem to agree on the behavior of any of these "tools". Some comments tell anecdotes of not getting the agents to listen until the context of the whole world is laid out in these md files. Others say the only way is to keep the context tight and focused, going so far as to have written _yet more tools_ to remove and re-add code comments so they don't "poison" the context.
I am slightly straw-manning, but the tone in this thread has already shifted from a few months ago where these "tools" were going to immediately give huge productivity gains but now you're telling me they need 1) their own special files everywhere (again, this isn't even agreed on) and 2) "engineering and planning...not done in days or weeks even
The entire economy is propped up on this tech right now and no one can even agree on whether it's effective or how to use it properly? Not to mention the untold damage it is doing to learning outcomes.
Yeah I feel like on average I still spend a similar amount of time developing but drastically less time fixing obscure bugs, because once it codes the feature and I describe the bugs it fixed them, the rest of my times spent testing and reviewing code.
Learning how to equip a local LLM with tools it can use to interact with to extend its capabilities has been a lot of fun for me and is a great educational experience for anyone who is interested. Just another tool for the toolchest.
> Do I need 3+ md files in every directory?
Don’t worry, as of about 6 weeks ago when they changed the system prompt Claude will make sure every folder has way more than 3 .md files seen as it often writes 2 or more per task so if you don’t clean them up…