Well, this is just semantics. I can have a repo that includes a collection of json files that I had generated via a semi-manual build process that depends on everything from the state of my microbiome to my cat's scratching pattern during Mercury's last retrograde. If I attach an open source license to it, then that's the source - do with it what you will. Otherwise, I don't see how this discussion doesn't lead to "you must first invent the universe".
The difference is that you can customize/debug it or not. You might say that a .EXE can be modified too. But I don't think that's the conventional definition of open source.
I understand that these days, businesses and hobbyists just want to use free LLMs without paying subscriptions for economic motives, that is, either saving money or making money. They don't really care whether the source is truly available or not. They are just end users of a product, not open-source developers by any means.
Not just semantics, the concept of open source fundamentally depend on what the preferred form of modification is
https://opensource.org/ai/open-source-ai-definition