Comment by phillipcarter
Comment by phillipcarter 2 days ago
The issue I see is that this is pretty much the final boss for AI systems. Not because the tasks to do are inherently too difficult or whatever, but the integration of data and quality of that data is so variable that you just can't get something done reliably.
Compare this to codebase AI, where much of the data you need lies in your codebase or repo. Even then, most of these coding tools aren't even close to automating meaningful coding tasks in practice, and while that doesn't mean they can't in the future, it's a long ways off!
Now in the ops world, there's little to no guarantee that you'll have relevant diagnostic data coming out of a system that you need to diagnose it. That weird way you're using kafka right now? The reason for it is told via oral tradition on the team. Runbooks? Oh, those things that we don't bother looking at since they're out of date? ...and so on.
The challenge here is in effective collection of quality data and context, not the AI models, and that's precisely what's so hard about operations engineering in the first place.
Even then, most of these coding tools aren't even close to automating meaningful coding tasks in practice, and while that doesn't mean they can't in the future, it's a long ways off!
Not related to your main point, but I've introduced Github Copilot to my teams, and, surprisingly, two of our strongest developers reached out to me independently, and told me it's been a huge boost to their productivity, one in refactoring legacy code, and another in writing some non-trivial components. I thought the primary use would be as a crutch for less capable developers, so I was surprised by this.
As a middle-manager whose day job previously robbed me of the opportunity to write code, I've used ChatGPT 4o to write complex log queries on legacy systems that would have been nearly impossible for me to do otherwise (and would have taken a lot of effort from my teams) and to turn out small (but meaningful) tasks, including learning Android dev from scratch to unblock another group and other worthwhile things that keep my team from being distracted and able to deliver.
I guess there's a "no true Scotsman" fallacy hiding there, about what constitutes "meaningful coding tasks in practice", but to me, investing in these tools has been money well spent.