Comment by kenmacd

Your base assumption here is that the 'crap' is actually 'junk'. Let's look at the easy one here, languages. Talk to someone that speaks multiple languages and they'll have examples of concepts in one language that are difficult to express in another. The multilingual person, or someone who just speaks a different language than you, will think differently[1].

Does the LLM take advantage of this? I don't know. It wouldn't surprise me if it did, and if it doesn't now I'd bet it will in the future. Either way though, throwing away those other languages could make the model dumber. As you allude to, there's a balance between intelligence and knowledge.

(in case you hadn't thought of it, those 'tools' can also be other LLMs with more specialized knowledge in a particular field. For example a 'translator' model)

Other 'facts' could also have more merit than it would first appear. Sure, one particular person's shoe size might not be needed, but if you were to filter out shoe sizes in general then the model might not be able to suggest how to find properly fitting footwear, or might not suggest that your back pain could be related to your shoes.

> That would be a worthwile endeavor and maybe even possible without boiling the oceans.

I think it's important to keep in mind that we're very early in the AI journey. Look at the power requirements of early computers versus the ones we use today. I'm all for keeping energy usage in mind, but I'd be careful with hyperbolic language as things are changing so quickly. Tasks that would have taken multiple GPUs can now run on my laptop CPU.

[1] https://www.edge.org/conversation/lera_boroditsky-how-does-o...