Comment by ghc Comment by ghc 6 months ago 2 replies Copy Link View on Hacker News Can someone please enlighten me how this is any different from Mixture of Experts? Because I don't see any difference at all.
Copy Link liuliu 6 months ago Collapse Comment - The router is manually designed (see their cem function). Also, the experts are not separate weights, just different scales of it's singular values. Reply View | 1 reply Copy Link ghc 6 months ago Parent Collapse Comment - Thank you, I was missing that second part. Reply View | 0 replies
Copy Link ghc 6 months ago Parent Collapse Comment - Thank you, I was missing that second part. Reply View | 0 replies
The router is manually designed (see their cem function). Also, the experts are not separate weights, just different scales of it's singular values.