Comment by nickpsecurity

Both of your claims are true. That doesn’t justify breaking the law.

I could likewise argues that most of the world money is in the hands of other people, I could perform more in the markets if I had it, and so I should just go take it. We still follow the law and respect others’ rights in spite of what acting morally cost us.

The law abiding, moral choice is to do what we can within the law while working to improve the law. That means we use a combination of permissively licensed works and works to train our models. We also push for legislation that creates exceptions in copyright law for training machine learning models. We’re already seeing progress in Israel and Singapore on those.

Meanwhile countries who whistle on that copyright would be able to gain a huge advantage.

Here is the only legal efforts I know about that’s available in some way:

https://www.fairlytrained.org/

https://www.kl3m.ai/#features

Here’s a dataset that could be used for a public domain model:

https://www.tensorflow.org/datasets/catalog/pg19

If non-public domain, one can add in the code from The Stack. That would be tens of gigabytes of both English text and code. Then, third-party could add licensed, modern works to the model with further pre-training.

I also think a model trained on a large amount of public domain data would be good for experimentation with reproduceability. There would be no intellectual property issues in the reproduction of the results. Should also be useful in a lot of ways.