Comment by mhitza

Comment by mhitza 4 hours ago

9 replies

Should be "that you can train for $100"

Curios to try it someday on a set of specialized documents. Though as I understand the cost of running this is whatever GPU you can rent with 80GB of VRAM. Which kind of leaves hobbyists and students out. Unless some cloud is donating gpu compute capacity.

Onavo 4 hours ago

A GPU with 80GB VRAM costs around $1-3 USD an hour on commodity clouds (i.e. the non-Big 3 bare metal providers e.g. https://getdeploying.com/reference/cloud-gpu/nvidia-h100). I think it's accessible to most middle class users in first world countries.

  • antinomicus 3 hours ago

    Isn’t the whole point to run your model locally?

    • theptip 3 hours ago

      No, that’s clearly not a goal of this project.

      This is a learning tool. If you want a local model you are almost certainly better using something trained on far more compute. (Deepseek, Qwen, etc)

    • yorwba 3 hours ago

      The 80 GB are for training with a batch size of 32 times 2048 tokens each. Since the model has only about 560M parameters, you could probably run it on CPU, if a bit slow.

    • simonw 2 hours ago

      You can run a model locally on much less expensive hardware. It's training that requires the really big GPUs.

    • jsight 2 hours ago

      I'd guess that this will output faster than the average reader can read, even while using only CPU inferencing on a modern-ish CPU.

      The param count is small enough that even cheap (<$500) GPUs would work too.

portaouflop 4 hours ago

If I have let’s say 40gb RAM does it not work at all or just take twice as long to train?

  • typpilol 4 hours ago

    Won't work at all. Or if it does it'll be so slow since it'll have to go to the disk for every single calculation so it won't ever finish.

    • karpathy 2 hours ago

      It will work great with 40GB GPU, probably a bit less than twice slower. These are micro models of a few B param at most and fit easily during both training and inference.