Comment by semiinfinitely
Comment by semiinfinitely 5 days ago
because it turns out that optimizing performance of a programming language designed for use-cases where runtime performance doesn't matter ... doesn't matter
Comment by semiinfinitely 5 days ago
because it turns out that optimizing performance of a programming language designed for use-cases where runtime performance doesn't matter ... doesn't matter
What percentage of the CPU cycles are actually spent running Python though? My impression is _very_ low in production LLM workloads. I think significantly less than 1%. There are almost certainly better places to spend the effort, and if it did matter, I think they would replace Python with something like C++ or Rust.
There's currently talk of adding gigawatts of data center capacity to the grid just for use cases where python dominates development. While a lot of that will be compiled into optimized kernels on CPU or GPU, it only takes a little bit of 1000x slower code to add up to a significant chunk of processing time at training or inference time.