How does this compare to modal (modal.com)? Faster cold-start? Easier config? Asking because I've used modal quite a bit for everything from fine-tuning LLMs to running etl pipelines and it works well for me, and I haven't found any real competitors for them to even think of switching.
I haven't used SkyPilot so I am unfamiliar with the experience and performance.
However, some of the situations you would like to use Cerebrium over Skypilot are: - You don't want to manage you own hardware - Reduced costs: With serverless Runtime and low cold starts (unclear if SkyPiolet offers this and what the peformance is like if they do) - Rapid iteration: Unclear of the deployment process on SkyPilot and how long projects take to go live - Observability: Looks like you would just have k8s metrics at your disposal
Modal is a great platform!
In terms of cold starts, we seem to be very comparable from what users have mentioned and tests we have run.
Easier config/setup is feedback we have gotten from users since we don't have and special syntax or a "Cerebrium way" of doing things which makes migration pretty easier as well as doesn't lock you in which some engineers appreciate. We just run your Python code as is with an extra .toml setup file.
Additionally, we offer AWS Inferentia/Tranium nodes which offer a great price/performance trade-offs for many open-Source LLM's - even when using TensorRT/vLLM on Nvidia GPU's and gets rid of the scarcity problem. We plan to support TPU's and others in future.
We are listed on AWS Marketplace as well as others which means you can subtract your Cerebrium cost from your commited cloud spend.
Two things we are working on that will hopefully make us a bit different is: - GPU checkpointing - Running compute in your own cluster to use credits/for privacy concerns.
Where Modal does really shine is training/data-processing use cases which we currently don't support too well. However, we do have this on our roadmap for the near future.