Comment by diggan

There are many things to be said about open-source projects and, more broadly, the capabilities of the open-source community.

The most capable parts are for-profit organizations that release open-source software for their business imperative, public benefit companies that write open-source software for ideological reasons but still operate as businesses, and a tiny number of public benefit organizations with unstable cash flow. Most other efforts are unorganized and plagued by bickering.

Llama itself is challenging to take over. The weights are public, but the training data and process is not. It could be evolved, but not fully iterated by anyone else. For a full iteration, the training process and inputs would need to be replicated, with improvements there.

But could another open-source model, as capable as Llama, be produced? Yes. Just like Meta, other companies, such as Google and Microsoft, have the incentive to create a moat around their AI business by offering a free model to the public, one that's just barely under their commercial model's capabilities. That way, no competitor can organically emerge. After all, who would pay for their product if it's inferior to the open-source one? It's a classic barrier to entry in the market - a thing highly sought after by monopolistic companies.

Public benefit companies leading in privacy could develop a model to run offline for privacy purposes, to avoid mass consumer data harvesting. A new open-source ideological project without a stable business could also, in theory, pop up in the same pattern as the Linux project. But these are like unicorns - "one in a million years (maybe)."

So, to answer your question, yes, Llama weights could be evolved; no, an entirely new version cannot be made outside of Meta. Yes, someone else could create such a wholly new open-source model from scratch, and different open-source groups have different incentives. The most likely incentive is monopolistic, to my mind.

Its unlikely all the training data for Llama is publicly available, let alone under an open source license. If Llama actually had an open source license (IIRC it doesn't), that would still make it a Toxic Candy model under the Debian Deep Learning Team's Machine Learning policy. That means no-one could replicate it exactly, even if they had the boatloads of cash it would take to buy enough hardware and electricity to do the training. Eventually the community could maybe find or create enough data, but that would be a new different model.

https://salsa.debian.org/deeplearning-team/ml-policy

AI models aren't really iterative in the way that other software is. Llama 4 is a completely different product from Llama 3, with different parameter counts and even different modalities. The only reason it gets to be called Llama 4 is that the company that made it is the same and it's convenient to not have to come up with new names all the time, not because there's any sort of continuity with Llama 2.

Fine tunes are the correct analogy to iterative software development—they take the existing code (weights) and improve upon it and modify it—and fine tunes can be produced with what Meta has released.

The bigger problem with Meta's claim that it's open source is that they've attached a bunch of strings to the license that prevent you from using it in a bunch of different ways. It's not open source because it's not open, not because weights aren't source.

No. You need a research lab, compute time and talent to train LLMs.