Comment by majormajor
Comment by majormajor 2 days ago
> "Superhuman" refers to abilities, qualities, or powers that exceed those naturally found in humans. It implies being greater than normal human capabilities.
How do you know what normal human capabilities are for an unusual task that humans have not trained for? Is identifying the gender of the author of a blog post 80% of the time "extraordinary"? How do I know what a human is capable of doing for that with training?
If a person with no programming experience asked Claude or ChatGPT to produce some code, they'd get better code than their "normal" human capability could produce. So: superhuman coders?
But also today, I have asked Claude and ChatGPT to do coding tasks for me that both models got stuck on. Then I fixed them myself because I've had a lot of training and practice. So: not superhuman? But wait, the model output the broken code faster than I would've. So: superhuman again?
Extraordinary shouldn't be so easily disputable.
LLMs have superhuman breadth and superhuman speed. I haven't seen superhuman depth in any capabilities yet. I've seen them have "better than untrained median person" and often "better than hobbyist" depth. But here the authors claim "superhuman capabilities" which is pretty specificly not just meaning the breadth or speed.
I haven't read the paper, maybe their benchmark is flawed as you say, and there are a lot of ways for it to be flawed. But assuming it is not, I see no problem with using the word superhuman.
Out of curiosity, would you agree with me if I said 'Calculators have superhuman capabilities'? (Not just talking about speed here, since you can easily construct complex enough equations that a human wouldn't be able to solve in their lifetime but the calculator could within minutes).