Comment by pu_pe

Comment by pu_pe 17 hours ago

Nice benchmark but the human comparison is a little lacking. They claim to have surveyed 19 experts, though the vast majority of them have only a master's degree. This would be akin to comparing LLM programming expertise to a sample of programmers with less than 5 years of experience.

I'm also not sure it's a fair comparison to average human results like that. If you quiz physicians on a broad variety of topics, you shouldn't expect cardiologists to know that much about neurology and vice-versa. This is what they did here, it seems.

KSteffensen 17 hours ago

I'll get some downvotes for this but PhD vs master's degree difference is mostly work experience, an element of workload hazing and snobbery.

Somebody with a masters degree and 5 years of work experience will likely know more than a freshly graduated PhD

Reply View 2 replies

698969 14 hours ago

I think the breadth vs depth thing applies here as well, the PhD will know more about the topic they're researching of course.

Reply View | 0 replies
eesmith 17 hours ago

Sure, but all we know is that these "13 have a master’s degree (and are currently enroled in Ph.D. studies)". We only know they have at least "2 years of experience in chemistry after their first university-level course in chemistry."
How does that qualify them as "domain experts"? What domain is their expertise? All of chemistry?

Reply View | 0 replies