Comment by PaulKeeble

When I work places where interruptions are expected I change how I do my work. So an observer may see them costing me less time but that’s amortized across the rest of my work as a consequence of leaning into interruption. It’s still there, it’s just spread out more.

> This is a really common problem with science reporting in general. Its often the case that the news will say things about the paper that aren't in the paper, often they say something that is completely the opposite of what the paper actually represents with its data.

I wonder if perhaps a part of LLM hallucinations can be explained by them being provided such reporting and having it (mistakenly) tagged as high-quality training data.

  > This is a really common problem with science reporting in general.

1) it certainly contributes to a significant part of the g̶r̶o̶w̶i̶n̶g̶ distrust in science. People are getting their science from the news, not the horse's mouth. If you report something silly while saying "scientists say" then people will point the finger at the scientists more than they will point to the reporter. The scientists aren't writing in plain English after all[0]. Things like chocolate being healthy for you, red meat causing cancer, or machine learning quantum blackholes. There's always elements of truth to these things but truth is not sensational. Truth and complexity go hand in hand. As complexity decreases you will have to sacrifice accuracy. This is a tough balance to play[1].

2) The public isn't very scientifically literate and it is easy to misconstrue meaning. Hell, even scientists routinely struggle with this stuff. Let's take the red meat issue as an example. It's true, but a lot of those studies were looking at daily intake of 50g or 100g and results like "25% higher risk of rectal cancer" at the higher end of the estimates. For context, a Costco hotdog is 110g while a Nathan's Hotdog is 48. We're talking about 1-2 hotdogs per day. We're also talking about a percent increase in risk, not a percent risk. The CDC site says approximately 3.9% of men and women will be diagnosed with colorectal cancer. If that's our baseline then a 25% increase is a 4.9% risk. That's a disgusting amount of hotdogs to move from a 4% cancer risk to a 5%. Concerning on a national level but not on a personal. This feeds back to #1 as people are interpreting the news as saying "eating hotdogs makes you likely to get cancer" while people observe heavy hotdog eaters around them not getting cancer. Their observation wouldn't run counter to what the research says but it will against the narrative on the news. The incongruence between observation and understanding does justify mistrust. But there's just more ways to misinterpret something than there are to interpret correctly (relates to [2]). This failure mode becomes self-reinforcing. Enough that I think anyone that spends any time on the internet will be aware of it. (Communicating is fucking hard, communicating accurately is even harder)

3) (Perhaps the worst part) Scientists are primarily measured by their citations (count or "h-index"[3]). Unless you do something groundbreaking[4] (which is rare) then this is the main way to "measure" performance. A great way to boost citations is getting media attention. Unfortunately there's just a lot of papers published and a primary driver of citation count is knowledge of a paper's existence. You don't need to be an Avi Loeb type (it doesn't hurt) when we're talking about small numbers. If a Cal State grad student and a MIT grad student were to publish the same paper we'd expect the latter to have more citations due to the latter's greater visibility. MIT has a media wing and these papers are much more easily picked up by larger news orgs. This is why so many scientists use platforms like Twitter. Because your work doesn't mean anything (to your personal success and ability to continue doing your work) if you can't get enough citations. There's an obvious slippery slope here... One that can create a feedback loop to misreporting. The fiercer the competition the the more risky this situation becomes. It's really easy to do slight embellishments of your work. No one is checking at time of publication. Replication happens later and the system devalues replication. Plus, while replicating it is much easier to assume you've made a mistake rather than the paper was in error (or in serious error).

All this is to say that shit is messy. And I don't think any of it is particularly any one person's fault. More an emergent phenomena through compounding effects. Little things here and there add up as we talk about millions of events and many years. I know we all want things to be simple, and simplicity has a lot of benefits, but it also can be a big trap. "As simple as possible, but no simpler" does not mean something isn't extremely complex[5]. It's a trap that makes people think they can read a few lines from Wikipedia and understand something (read the whole article, it's still not enough). A trap with growing consequences to a world that grows in complexity[6]

[0] And I don't think w̶e̶ they should. Papers are a peer-to-peer communication network. Open and visible, but that's how the peer-to-peer communication takes place. Expert-to-public communication has traditionally been done through news or other science communicators. Asking scientists to write papers to the general public is like asking you to communicate to your coworker about your code as if your coworker knows nothing about code or the context it is running in (all because a layman may overhear). Good luck getting any work done...

[1] While news orgs and science communicators (especially pop sci communicators... ugh...) are doing harm here there are defenses anyone can take. Recognize your understanding is always wrong to some degree. Don't take in information as binary true/false statements but probabilities: e.g. likely true/maybe true/maybe false/likely false. Fundamentally the reason this is a good defense is because it is always a more accurate interpretation. Scientists don't find truth through confirmation but through negation. What I mean is w̶e̶ they rule things out. A scientist converges to truth[2]

[2] This also helps you sniff out conmen from the scientists. The scientist always has some doubt. At first they may show themselves as highly confident but as you press on detail they start weakening language. This isn't foolproof and isn't gonna work for every question, but it is common. Scientists are being more trained on media literacy because there is recognition that while this is the right way to talk to peers it gives the public a sense that they lack expertise rather than are aware of complexity. The best signal is just getting them to talk about their domain. They won't stop and will get very detailed.

[3] Number of papers with greater than N citations. An h-index of 10 is having 10 papers with >= 10 citations. h-index of 100 is 100 papers with >= 100 citations.

[4] If you do something groundbreaking no one gives a shit about citation count. But doing something groundbreaking will surely make your citation count skyrocket (often it also drives your h-index too, as you've simply gained more attention and more people are reading your other works. Your work doesn't change, but visibility does). This fact is often used to justify the usage of citation metrics. Citation metrics are fine, but they're also easy to hack and highly context driven. I mention Avi Loeb, and controversy is beneficial to this metric. Every paper that cites him to say he's wrong is a point for him. Controversy is a way to gather points from a whole new source! Not those building on your work, but those building against your work.

[5] "If you can't explain it simply, you don't understand it well enough" is laughable phrase. The reason you can't teach "a barmaid" Quantum Chromodynamics isn't because you don't understand it. You can't explain it "simply" because you don't understand it. You can't explain it simply because you can't even use the word "color" without a layman thinking a quark is red (can you even get into how it is impossible to have "red" at that scale?).

[6] Progress necessitates an increase in complexity. Look at a Taylor Expansion and it's relationship to computational difficulty. I'll let you all figure this out, my comment is already too long.