Comment by gpm Comment by gpm 2 days ago 4 replies Copy Link View on Hacker News > or even a single socket Epyc system with >1TB of RAMHow many tokens/second would this likely achieve?
Copy Link chithanh 8 hours ago Next Collapse Comment - KTransformers now supports Kimi K2 for MoE offloadingThey claim 14 tps for the 4-bit quant on a single socket system with 600 GB RAM and 14 GB GPU memory. Reply View | 0 replies
Copy Link kachapopopow 2 days ago Prev Next Collapse Comment - around 1 by the time you try to do anything useful with it (>10000 tokens) Reply View | 0 replies
KTransformers now supports Kimi K2 for MoE offloading
They claim 14 tps for the 4-bit quant on a single socket system with 600 GB RAM and 14 GB GPU memory.