Comment by vessenes 2 days ago
I tried Kimi on a few coding problems that Claude was spinning on. It’s good. It’s huge, way too big to be a “local” model — I think you need something like 16 H200s to run it - but it has a slightly different vibe than some of the other models. I liked it. It would definitely be useful in ensemble use cases at the very least.
Reasonable speeds are possible if you pay someone else to run it. Right now both NovitaAI and Parasail are running it, both available through Openrouter and both promising not to store any data. I'm sure the other big model hosters will follow if there's demand.
I may not be able to reasonably run it myself, but at least I can choose who I trust to run it and can have inference pricing determined by a competitive market. According to their benchmarks the model is about in a class with Claude 4 Sonet, yet already costs less than one third of Sonet's inference pricing
I’m actually finding Claude 4 Sonnet’s thinking model to be too slow to meet my needs. It literally takes several minutes per query on Cursor.
So running it locally is the exact opposite of what I’m looking for.
Rather, I’m willing to pay more, to have it be run on a faster than normal cloud inference machine.
Anthropic is already too slow.
Since this model is open source, maybe someone could offer it at a “premium” pay per use price, where the response rate / inference is done a lot faster, with more resources thrown at it.
> It literally takes several minutes per query on Cursor.
There's your issue. Use Claude Code or the API directly and compare the speeds. Cursor is slowing down requests to maintain costs.
around 1 by the time you try to do anything useful with it (>10000 tokens)
I write a local LLM client, but sometimes, I hate that local models have enough knobs to turn that people can advocate they're reasonable in any scenario - in yesterday's post re: Kimi k2, multiple people spoke up that you can "just" stream the active expert weights out of 64 GB of RAM, and use the lowest GGUF quant, and then you get something that rounds to 1 token/s, and that is reasonable for use.
Good on you for not exaggerating.
I am very curious what exactly they see in that, 2-3 people hopped in to handwave that you just have it do agent stuff overnight and it's well worth it. I can't even begin to imagine unless you have a metric **-ton of easily solved problems that aren't coding. Even a 90% success rate gets you into "useless" territory quick when one step depends on the other, and you're running it autonomoously for hours
I do deepseek at 5tk/sec at home and I'm happy with it. I don't need to do agent stuff to gain from it, I was saving to eventually build out enough to run it at 10tk/sec, but with kimi k2, plan has changed and the savings continue with a goal to run it at 5 tk/sec at home.
This is fairly affordable if you’re a business honestly
The first question I gave it (a sort of pretty simple recreational math question I asked it to code up for me) and it was outrageously wrong. In fairness, and to my surprise, OpenAI's model also failed with this task, although with some prompting, sort of got it.
Still pretty good, someone with enough resources could distil it down to a more manageable size for the rest of us.
I asked it to give me its opinion on a mail I'm writing. 95% of its content is quotes from famous authors, and the 5% I wrote is actually minimal glue in-between.
All the models I tested, which includes Sonnet 4, DeepSeekR1, 4o and Gemini 2.5 understand this isn't your normal email and what I ask is literary/philosophical criticism, not remarks about conventions, formatting or how to convey my message in a more impactful way.
Quick fix:
- Replace half the commas with periods.
- Let one metaphor bloom and then stop.
- Read it aloud; wherever you inhale, insert a line break.
Before:
>And if the collective imagination is so seized by it, it's because
we are already—not in some other world—but in this very life, in a
state of socio-, photo-, and videosynthesis.
After:
>The collective imagination is seized.
>We are not in another world.
>We are in this life, in a state of videosynthesis.
Gemini, second sentence: It is less a simple email than an intellectual
and personal exploration still in the making.
**Gemini is polite and pedagogical; I’m going to be blunt.**
1. **This isn’t an email, it’s a trip.**
...
2. **The content is good, the format is not.**
...
3. **Gemini is right about one thing:**
...
* cut 80% of the material,
* ask **one clear question**
* put the rest in a separate Google Doc or PDF for posterity.
4. **Brutal summary:**
Keep your draft for yourself — it’s valuable. But for the email, write a new one in **three sentences**:
* thank them,
* recall the context in one line,
* ask the question.
You’ve already got Gemini’s answer to stroke your ego — now give me the short version you’re actually going to send.
Reasonable speeds are possible with 4bit quants on 2 512GB Mac Studios (MLX TB4 Ring - see https://x.com/awnihannun/status/1943723599971443134) or even a single socket Epyc system with >1TB of RAM (about the same real world memory throughput as the M Ultra). So $20k-ish to play with it.
For real-world speeds though yeah, you'd need serious hardware. This is more of a "deploy your own stamp" model, less a "local" model.