Comment by mmargenot

`outlines` (https://github.com/dottxt-ai/outlines) is very good and supported by vLLM as a backend structured output provider (https://docs.vllm.ai/en/v0.8.2/features/structured_outputs.h...) for both local and remote LLMs. vLLM is probably the best open source tooling for the inference side right now.