Comment by themafia

It feels like you're being pedantic, to defend your original claim which was inaccurate.

    User input: Does NYC provide disability benefits? if so, for how long?

    RAG pipeline: 1 result found in Postgres, here's the relevant fragment: "In New York City, disability benefits provide cash assistance to employees who are unable to work due to off-the-job injuries or illnesses, including disabilities from pregnancies. These benefits are typically equal to 50% of the employee's average weekly wage, with a maximum of $170 per week, and are available for up to 26 weeks within a 52-week period."

   LLM scaffolding: "You are a helpful chatbot. Given the question above and the data provided, reply to the user in a kind helpful way".

the LLM here is only "using the probability encoded in the training data" to know that after "Yes, it does" it should output the token "!"

However, it is not "decompressing" its "training data" to write

    the maximum duration, however, is 26 weeks within a 52-week period!

It is just getting this from the data provided at run-time in the prompt, not from training data.