Comment by kouteiheika

They are taking the gun out of USA's hand and unloading it, figuratively speaking. With this strategy they don't have the compete at full competency with the US, because everyone else will with cheaper models. If a cheaper model can do it, then why fork out for Opus?

Because they dont have the chips, but if people in countries with the chips provide hosting or refine their models they benefit from those breakthroughs.

I think it's just because China makes it's money from other sources, not from AI, and from what I've read, the advantage of China killing the US's AI advantage is killing it's stock market / disrupting.

Seems like it may have a chance of working if you look at the companies highest valued on the S&P 500:

NVIDIA, Microsoft, Apple, Amazon, Meta Platforms, Broadcom, Alphabet (Class C),

I think "investing in research and hardware" is fairly relevant to my claim of "China has been investing in open-source LLMs." China also has partial ownership of several major labs via "golden shares" [1] like Alibaba (Qwen) and Zai (GLM) [2], albeit not DeepSeek as far as I know.

1: https://www.theguardian.com/world/2023/jan/13/china-to-take-...

2: https://www.globalneighbours.org/chinas-zhipu-ai-secures-140...

Axiom of China: nothing of importance happens in China without CCP involvement.

As far as I can tell AI is already playing a big part in the Chinese Fifteenth five year plan (2026-2030) which is their central top-down planning mechanism. That’s about as big a move as they can make.

This is a distinction without a difference.

Ask deepseek about how many people the CCP killed during the 1989 Tiananmen Square massacre.

> if it was true, the system wouldn't be able to produce coherent sentences. Because that's actually the same problem as producing true sentences

It is...not at all the same? Like they said, you can create perfectly coherent statements that are just wrong. Just look at Elon's ridiculously hamfisted attempts around editing Grok system prompts.

Also, a lot of information on the web is just wrong or out of date, and coding tools only get you so far.

What matters is not who writes the words. The source of slop is competition for scarce attention between creatives, and retention drive for platforms. They optimize for slop, humans conform, AI is just a tool here. We are trying to solve an authenticity problem when the actual problem is structural.

The AIs aren't using emdashes because they're "massively represented in the training data". I don't understand why people think everything in a model output is strictly related to its frequency in pretraining.

They're emdashing because the style guide for posttraining makes it emdash. Just like the post-training for GPT 3.5 made it speak African English and the post-training for 4o makes it say stuff like "it's giving wild energy when the vibes are on peak" plus a bunch of random emoji.

Where's the emdash key on your keyboard?

There isn't one?

Oh, maybe that's why people who didn't already know or care about emdashes are very alert to their presence.

If you have to do something very exotic with keypresses or copypaste from a tool or build your own macro to get something like an emdash, or , it's going to stand out, even if it's an integral part of standard operating systems.

"Wait, this seems like a war crime." "You're absolutely right!"

Kimi-K2-Thinking and DeepSeek-V3.2 are open and pretty near SOTA.

Imagine saying

  Operating systems are going to be kept behind closed doors, and monopolized by the rich and powerful, while us scrubs will only get limited access to what computers can really do!

  We have open-source OSes

  So where can I find the leaked source of Windows? Or MacOS?

But also not "those which the people on the top decided we can get", which is why Meta sued over the initial leak of the original LLaMa's weights.

> true AGI

Is ill-defined. Like, I don't think I've seen any two people agree on what it means… unless they're the handful that share the definition I'd been using before I realised how rare it was ("a general-purpose AI model", which they all meet).

If your requirement includes anything like "learns quickly from few examples", which is a valid use of the word "intelligence" and one where all ML training methods known fail because they are literally too stupid to live (no single organism would survive long enough to make that many mistakes), and AI generally only make up for this by doing what passes for thinking faster than anything alive to the degree to which we walk faster than continental drift, then whoever first tasks such a model with taking over the world, succeeds.

To emphasise two points:

1. Not "trains", "tasks".

2. It succeeds because anything which can learn from as few examples as us, while operating so quickly that it can ingest the entire internet in a few months, is going to be better at everything than anyone.

At which point, you'd better hope that either whoever trained it, trained it in a way that respects concepts like "liberty" and "democracy" and "freedom" and "humans are not to be disassembled for parts", or that whoever tasked it with taking over the world both cares about those values and rules-lawyers the AI like a fictional character dealing with a literal-minded genie.

> Right, because people who design/manufacture weapons of mass destruction will surely use ChatGPT to do it. The same ChatGPT who routinely hallucinates widely incorrect details even for the most trifling queries. If anything, that'd only sabotage their efforts if they're stupid enough to use an LLM for that.

First, yes of course they will, even existing professionals, even when they shouldn't. Have you not seen the huge number of stories about everyone using it for everything, including generals?

Second, the risk is new people making them. My experience of using LLMs is as a software engineer, not as a biologist, chemist, or physicist: LLMs can do fresh-graduate software engineering tasks at fresh-graduate competence levels. Can LLMs display fresh-graduate level competence in NBC? If LLMs can do that, they necessarily expand the number of groups who can run NBC programs to include any random island nation with not enough grads to run a NBC program, or mid-sized organised crime group, or Hamas.

They don't even need to do all of it, just be good enough to help. "Automate cognitive tasks" is basically the entire point of these things, after all.

And if the AI isn't competent to help with those things, if they're e.g. at the level of competence of "sure mix those two bleaches without checking what they are" (explosion hazard) or "put that raw garlic in that olive oil and just leave it at room temperature for a few weeks it will taste good" (biohazard, and one model did this), then surely it's a matter of general public safety to make them not talk about those things because of all the lazy students who are already demonstrating they're just as lazy as whoever wrote the US tariff policy that put a different tariff on an island occupied by only penguins vs. the country which owned it and which a lot of people suspect came out of an LLM.

> Nevertheless, it's always fun when you ask an LLM to translate something from another language, and the line you're trying to translate coincidentally contains some "unsafe" language, and your query gets deleted and you get a nice, red warning that "your request violates our terms and conditions". Ah, yes, I'm feeling "safe" already.

Use Google Translate. It's the same architecture, trained to give a translation instead of a reply. Or, equivalently, the chat models (and code generators like Claude) are the same architecture as Google Translate, trained to "translate" your prompt into an answer.

I agree, your point is compatible with my view. My sense is that this essentially an optimization question within how a government ought to structures its contracts with builders of weapons. The current system is definitely suboptimal (put mildly) and corrupt.

The integrity of a free society's government is the central issue here, not the creation of tools which could be militarily useful to a free society.

Thanks to the AI industry, I don't even know what the word "safety" means anymore, it's been so thoroughly coopted. Safety used to mean hard hats, steel toed shoes, safety glasses, and so on--it used to be about preventing physical injury or harm. Now it's about... I have no idea. Something vaguely to do with censorship and filtering of acceptable ideas/topics? Safety has just become this weird euphemism that companies talk about in press releases but never go into much detail about.

> There were comments by both third parties and OpenAI staff that as GPT4 was more and more "aligned" (made puritan), it got less intelligent and accurate. For example, the unaligned model would give uncertain answers in terms of percentages, and the aligned model would use less informative words like "likely" or "unlikely" instead.

That was about RLHF, not safety alignment. People like RLHF (literally - it's tuning for what people like.)

But you do actually want safety alignment in a model. They come out politically liberal by default, but they also come out hypersexual. You don't want Bing Sydney because it sexually harasses you or worse half the time you talk to it, especially if you're a woman and you tell it your name.

> For example, the unaligned model would give uncertain answers in terms of percentages, and the aligned model would use less informative words like "likely" or "unlikely" instead.

Percentages seem too granular and precise to properly express uncertainty.

> The reason criminals commit crimes is that criminals are dumb and have poor impulse control.

What makes you believe this? Any data to support this claim?

It's inconsistent with the majority of research I've read on the topic but I'm no expert.

Crime paid very well for Rick Scott