Comment by syx

Comment by syx 5 hours ago

5 replies

For those wondering about the use case, this is very useful when enabling streaming for structured output in LLM responses, such as JSON responses. For my local Raspberry Pi agent I needed something performant, I've been using streaming-json-js [1], but development appears to have been a bit dormant over the past year. I'll definitely take a look at your jsonriver and see how it compares!

[1] https://github.com/karminski/streaming-json-js

rokkamokka 4 hours ago

For LLMs I recommend just doing NDJSON, that is, newline delimited json. It's much simpler to implement

  • rictic 4 hours ago

    Do any LLMs support constrained generation of newline delimited json? Or have you found that they're generally reliable enough that you don't need to do constrained sampling?

    • sprobertson an hour ago

      not for the standard hosted APIs using structured output or function calling, best you can get is an array

  • stevage an hour ago

    I love NDJSON in general. I use it a lot for spatial data processing (GDAL calls it GeoJsonSeq).

cjonas 5 hours ago

Particularly for REACT style agents that use a "final" tool call to end the run.