Comment by james2doyle

Comment by james2doyle 3 days ago

I was playing with the new IBM Granite models. They are quick/small and they do seem accurate. You can even try them online in the browser because they are small enough to be loaded via the filesystem: https://huggingface.co/spaces/ibm-granite/Granite-4.0-Nano-W...

Not only are they a lot more recent than gemma, they seem really good at tool calling, so probably good for coding tools. I haven’t personally tried it myself for that.

The actual page is here: https://huggingface.co/ibm-granite/granite-4.0-h-1b

firefax 3 days ago

Interesting. Is there a way to load this into Ollama? Doing things in browser is a cool flex, but my interest is specifically in privacy respecting LLMs -- my goal is to run the most powerful one I can on my personal machine, with the end goal being those little queries I used to send to "the cloud" can be done offline, privately.

Reply View 2 replies

fultonn 3 days ago

> Is there a way to load this into Ollama?
Yes, the granite 4 models are on ollama:
https://ollama.com/library/granite4
> but my interest is specifically in privacy respecting LLMs -- my goal is to run the most powerful one I can on my personal machine
The HF Spaces demo for granite 4 nano does run on your local machine, using Transformers.js and ONNX. After downloading the model weights you can disconnect from the internet and things should still work. It's all happening in your browser, locally.
Of course ollama is preferable for your own dev environment. But ONNX and transformers.js is amazingly useful for edge deployment and easily sharing things with non-technical users. When I want to bundle up a little demo for something I typically just do that instead of the old way I did things (bundle it all up on a server and eat the inference cost).

Reply View | 1 reply
- firefax 2 days ago
  
  Thanks for this pointer and explanation, I appreciate it.
  Also my "dev enviornment" is vi -- I come from infosec (so basically a glorified sysadmin) so I'm mostly making little bash and python scripts, so I'm learning a lot of new things about software engineering as I explore this space :-)
  Edit: Hey which of the models on that page were you referring to? I'm grabbing one now that's apparently double digit GB? Or were you saying they're not CPU/ram intensive but still a bit big?
  
  Reply View | 0 replies

brendoelfrendo 3 days ago

Not the person you replied to, but thanks for this recommendation. These look neat! I'm definitely going to give them a try.

Reply View 0 replies