Comment by lalassu 2 days ago
Disclaimer: I did not test this yet.
I don't want to make big generalizations. But one thing I noticed with chinese models, especially Kimi, is that it does very well on benchmarks, but fails on vibe testing. It feels a little bit over-fitting to the benchmark and less to the use cases.
I hope it's not the same here.
This was a bad problem with earlier Chinese (Qwen and Kimi K1 in particular) models, but the original DeepSeek delivered and GLM4.6 delivers. They don't diversify training as much as American labs so you'll find more edge cases and the interaction experience isn't quite as smooth, but the models put in work.
This used to happen with bench marks on phones, manufacturers would tweak android so benchmarks ran faster.
I guess that’s kinda how it is for any system that’s trained to do well on benchmarks, it does well but rubbish at everything else.
My experience with deepseek and Kimi is quite the opposite: smarter than benchmarks would imply
Whereas the benchmark gains seem by new OpenAI, Grok and Claude models don't feel accompanied by vibe improvement
I would assume that it is testing how well and appropriately the LLM responds to prompts.
K2 Thinking has immaculate vibes. Minimal sycophancy and a pleasant writing style while being occasionally funny.
If it had vision and was better on long context I'd use it so much more.