Comment by Havoc

Comment by Havoc 3 months ago

Sounds really good & human! Got a fair bit of unexpected artifacts though. e.g. 3 seconds hissing noise before dialogue. And music in background when I added (happy) in an attempt to control tone. Also don't understand how to control the S1 and S2 speakers...is it just random based on temp?

> TODO Docker support

Got this adapted pretty easily. Just latest nvidia cuda container, throw python and modules on it and change server to serve on 0.0.0.0. Does mean it pulls the model every time on startup though which isn't ideal

dragonwriter 3 months ago

> Also don't understand how to control the S1 and S2 speakers...

Do a clip with the speakers you want as the audio prompt, add the text of that clip (with speaker tags) of the clip at the beginning of your text prompt, and it clones the voices from your audio prompt for the output.

Reply View 0 replies

toebee 3 months ago

Thank you for the kind words! Dia wasn’t fine tuned on certain speaker, so you will get random voices every time you run it, unless you add a prompt / fix the seed.

The outputs are a bit unstable, might need to add cleaner training data and run longer training sessions. Hopefully we can do something like OAI Whisper and update with better performing checkpoints!

Reply View 0 replies

yjftsjthsd-h 3 months ago

> Does mean it pulls the model every time on startup though which isn't ideal

Surely it just downloads to a directory that can be volume mapped?

Reply View 2 replies

Havoc 3 months ago

Yep. I just didn't spend the time to track down the location tbh. Plus huggingface usually does links to a cache folder that I don't recall the location of
Literally got cuda containers working earlier today so haven't spent a huge amount of time figuring things out

Reply View | 1 reply
- genewitch 3 months ago
  
  Its in a dot folder in your home dir on Linux and in %appdata% on windows.
  
  Reply View | 0 replies