Comment by littlestymaar

Comment by littlestymaar 2 days ago

3 replies

View on Hacker News

> VLMs are hugely significant. Not only are they great for product use cases, giving users the ability to ask questions with images, but they're how we gather the synthetic training data to build image and video animation models. We couldn't do that at scale without VLMs. No human annotator would be up to the task of annotating billions of images and videos at scale and consistently.

Weren't Dall-E, Midjourney and Stable diffusion built before VLM became a thing?

tomrod 2 days ago

These are in the same space, but are diffusion models that match text to picture outputs. VLMs are common in the space, but to my understanding work in reverse, extract text from images.

Reply View 2 replies
  • vlovich123 2 days ago

    The modern VLMs are more powerful. Instead of invoking text to image or image to text as a tool, the models are trained as multimodal models and it’s a single transformer model where the latent space between text and image is blurred. So you can say something like “draw me an image with the instructions from this image” and without any tool calling it’ll read the image, understand the text instructions contained therein and execute that.

    There’s no diffusion anywhere which is kind of dying out except as maybe purpose-built image editing tools.

    Reply View 1 reply