Comment by dizzant

You’re right, shallowly — the quality of their implementation bears on these results.

One could read this paper as Salesforce publicly weighing their own reputation for wielding existing tools with competence against the challenges they met getting those tools to work. Seemingly they would not want to sully that reputation by publishing a half-baked experiment, easily refuted by a competitor to their shame? It’s not conclusive, but it is relevant evidence about the state of LLMs today.