Comment by deepsquirrelnet

This is true, and I’ve done quite a bit with static embeddings. You can check out my wordllama project if that’s interesting to you.

https://github.com/dleemiller/WordLlama

There’s also model2vec doing some cool things as well in that area. So it’s cool to see recent progress in 2024/5 on simple static embedding models.

On the computational performance note, the performance of cross encoder I trained using ModernBERT base is on par with the roberta large model, while being about 7-8x faster. Still way more complex than static, but on benchmark datasets, much more capable too.