Comment by kbouck
For anyone that has built more complex collector pipelines, I'm curious to know the tech stack:
- otel collector?
- kafka (or other mq)?
- cribl?
- vector?
- other?For anyone that has built more complex collector pipelines, I'm curious to know the tech stack:
- otel collector?
- kafka (or other mq)?
- cribl?
- vector?
- other?The OpenTelemetry collector works very well for us – even for internal pipelines. You can build upon a wide array of supported collector components; extending it with your own is also relatively straightforward. Plus, you can hook up Kafka (and others) for data ingress/egress.
What sort of complexity do you need? I've used them on my previous job and am implementing it on the current one. I have never heard of the last three you mention.
Otel collector is very useful for gathering multiple different sources, eg I am at a big corporation and we both have department level Grafana stack (Prometheus Loki etc) and we need to also send the data to Dynatrace. With otel collector these things are a minor configuration away.
For Kafka if you mean tracing through Kafka messages previously we did it by propagating it in message headers. Done at a shared team level library the effort was minimal.