Comment by ferguess_k 4 days ago
Unfortunately what I see is companies, especially smaller companies who originally got into Databricks because they hired people with Databricks/Spark experience, are trying to get away from the platform because it is too expensive -- and with that kind of money it is just easier to use Snowflake.
Yeah but looks like it is more "managed" and analysts especially prefer writing SQL over Python.
Honestly, as a Data engineer on the DWH side, I figured that my career is going to come to an end in a few years. AI + Cloud managed DWH are going to make all technical issues trivial, and I'm not someone who is interested in business context. Not sure where to move though.
I'm really surprised to hear this. If anything, I'd expect that every company transitioning from
> "we want to store/retrieve thin event logs and clickstreams"
to
> "we need to store/retrieve/join thick prose from customer interactions/reviews at every layer of the stack to give our LLMs the right context"
would create a significant need for data engineering for bespoke/enterprise/retail-monster use cases. (And data analysis too, until LLMs get better at tabular data.)
Are you seeing that this transformation need is actually being sufficiently covered by cloud providers, on the ground?
Or that people aren't seeing the problem this way, and are just doing prompt engineering with minimal RAG on static help-center datasets? That seems suboptimal, to say the least.
That's what I think too, but meh I'm not super interested in that I guess, as it definitely rings the death bell. But I agree that's definitely the future. And people are going to be trained to accommodate AI instead of the other way around -- it's much easier!
... which is also not cheap