Comment by physarum_salad

Comment by physarum_salad 3 days ago

The review paper is dead... so this is a good development. Like you can generate these things in a couple of iterations with AI and minor edits. Preprint servers could be dealing with 1000s of review/position papers over short periods of time and then this wastes precious screening work hours.

It is a bit different in other fields where interpretations or know-how might be communicated in a review paper format that is otherwise not possible. For example, in biology relating to a new phenomena or function.

bee_rider 3 days ago

What are review papers for anyway? I think they are either for

1) new grad students to end up with something nice to publish after reviewing the literature or,

2) older professors to write a big overview of everything that happened in their field as sort of a “bible” that can get you up to speed

The former is useful as a social construct; I mean, hey, new grad students, don’t skimp on your literature review. Finding out a couple years in that folks had already done something sorta similar to my work was absolutely gut-wrenching.

For the latter, I don’t think LLMs are quite ready to replace the personal experiences of a late-career professor, right?

Reply View 2 replies

CamperBob2 2 days ago

Ultimately, a key reason to write these papers in the first place is to guide practitioners in the field, right? Otherwise science itself is just a big (redacted term that can get people shadow-banned for simply using it).
As one of those practitioners, I've found good review/survey papers to be incredibly valuable. They call my attention to the important publications and provide at least a basic timeline that helps me understand how the field has evolved from the beginning and what aspects people are focusing on now.
At the same time, I'll confess that I don't really see why most such papers couldn't be written by LLMs. Ideally by better LLMs than we have now, of course, but that could go without saying.

Reply View | 0 replies
trostaft 2 days ago

I've found (good) review papers invaluable as an academic. They're really useful as a fast ladder to getting up to speed in a new area. Usually they have a great literature review (with the important papers to read afterward), a curated list of results important to understand, and good intuition about how to reason. It's a compactification of what I would have to otherwise gain by working in an area for years. No replacement for it, of course, but does make it easier attain.
I don't understand the appeal of an (majorly-)LLM generated review paper. A good review paper is a hard task to write well, and frankly the only good ones I've read have come from authors who are at apex of their field (and are, in particular, strong writers). The 'lossy search' of an LLM is probably an outstanding tool for _refining_ a review paper, but for fully generating it? At least not with current LLMs.

Reply View | 0 replies

JumpCrisscross 3 days ago

> you can generate these things in a couple of iterations with AI

The problem is you can’t. Not without careful review of the output. (Certainly not if you’re writing about anything remotely novel and thus useful.)

But not everyone knows that, which turns private ignorance into a public review problem.

Reply View 1 reply

physarum_salad 3 days ago

Are review papers centred on novel research? I get what you mean ofc but most are really mundane overviews. In good review papers the authors offer novel interpretations/directions but even then it involves a lot of grunt work too.

Reply View | 0 replies

awestroke 3 days ago

A good review paper is infinitely better than an llm managing to find a few papers and making a summary. A knowledgeable researcher knows which papers are outdated and can make a trustworthy review paper, an LLM can't easily do that yet

Reply View 2 replies

physarum_salad 3 days ago

Ok I take your point. However, it is possible to generate a middling review paper combining ai generated slop and edits. Maybe we would be tricked by it in certain circumstances. I don't mean to imply these outputs are something I would value reading. I am just arguing in favour of the proposed approach of arXiv.

Reply View | 1 reply
- JumpCrisscross 3 days ago
  
  > it is possible to generate a middling review paper combining ai generated slop and edits
  If you’re an expert. If you’re not, you’ll publish, best case, bullshit. (Worst case lies.)
  
  Reply View | 0 replies

bulubulu 2 days ago

Review papers are summarizations to recent updates in the field that deserve fellow researchers' attention. Such works should be done annually or at most quarterly in my opinion, to include only time-tested results. If hundreds of review papers are published every month, I am afraid that their quality in terms of paper selection and innovative interpretation/direction will not be much higher than the content generated by LLM, even if written word-to-word by a real scientist.

LLMs are good at plainly summarizing from the public knowledge base. Scientists should invest their time in contributing new knowledge to public base instead of doing the summarization.

Reply View 0 replies