Comment by bluGill
I think it is more like the clerk would say "There never was a US vs Wilson" (well there probably was given how common that name is, but work with me). The AI doesn't have a concept of maybe I misunderstood the question. AI would likely give you a good summary if the case happened, but if it didn't it makes up a case.
Yes. That is precisely the problem with using LLMs. They wantonly make up text that has no basis in reality. That is the one and only problem I have with them.