Comment by boringg

Financially, it's a bit of a wash because this affects their competition just as much as it affects them. Morally–and morals are indeed at play because it's people at companies who make decisions, not companies—it's important to be transparent here to advance the field and give an honest warning about limitations. Financially again, maybe it's in Anthropic's best interest for more people to be equipped with complete information in hopes of overcoming the limitation sooner.

>> I'm trying to ascertain their incentive here...

It's good for their mission and business.

1) Their stated mission is

"Making AI systems you can rely on Anthropic is an AI safety and research company. We build reliable, interpretable, and steerable AI systems" - https://www.anthropic.com/company

2) They've increased their credibility.

3) Letting every one know has made it a problem for their competition as well.

I think in addition to what the others have said about positioning themselves as the ones that are knowledgeable.

Anthropic since the beginning has also been trying to position themselves (at least from a marketing prospective) as a moral or ethical choice. Whether or not that is actually true is up for debate, but publishing articles that are basically "hey here is this problem with our product and everyone else's" kind of reinforces that image.

They want to sow distrust in open source. 'You can't trust open source because no one is cleaning the training data'.

Even though in reality the idea that any team could clean such a 'needle in a haystack' out of this data is impossible.

Of the 13 authors, 3 are at Anthropic. Of the 4 core contributors, 1 is at Anthropic.

Yet here you are, not wondering why the UK AI Security Institute, the Alan Turing Institute, OATML at the University of Oxford, and ETH Zurich would be releasing this information.

So I suppose the press release did the job it was supposed to do.

(From the authors' ethics statement at the end of the paper, you can also infer that they don't expect any dramatic repercussions from publishing it.)

Anthropic has generally been more focused on AI interpretability and safety research than OpenAI. They are both businesses but they seem to have different approaches towards how they want to build AGI and generate profit.

I believe it's intended to convince the audience they are experts, that this type of thing is dangerous to a business, and they are the ones doing the most to prevent it. There is no explicit statement to this effect, but I get the sense they are saying that other vendors, and especially open models that haven't done the work to curate the data as much, are vulnerable to attacks that might hurt your business.

Also a recruiting and branding effort.

All of this is educated guesses, but that's my feeling. I do think the post could have been clearer about describing the practical dangers of poisoning. Is it to spew misinformation? Is it to cause a corporate LLM powered application to leak data it shouldn't? Not really sure here.

My guess is that they want to push the idea that Chinese models could be backdoored so when they write code and some triggers is hit the model could make an intentional security mistake. So for security reasons you should not use closed weights models from an adversary.

Maybe their model is under attack and they are releasing the problem so that others learn how to exploit this against other llm providers, thus leveling field while they find solution to this problem

It looks suspicious, I agree. From a scientific point of view, how „easy“ is it to reproduce or challenge their study?