Comment by godelski

Comment by godelski 2 days ago

14 replies

View on Hacker News

Yes but it doesn't generalize very well. Even on simple features like gender. If you go look at embeddings you'll find that man and woman are neighbors, just as king and queen are[0]. This is a better explanation for the result as you're just taking very small steps in the latent space.

Here, play around[1]

  mother - parent + man = woman
  father - parent + woman = man
  father - parent + man = woman
  mother - parent + woman = man
  woman - human + man = girl
Or some that should be trivial

  woman - man + man = girl
  man - man + man = woman
  woman - woman + woman = man
Working in very high dimensions is funky stuff. Embedding high dimensions into low dimensions results in even funkier stuff

[0] https://projector.tensorflow.org/

[1] https://www.cs.cmu.edu/~dst/WordEmbeddingDemo/

yellowcake0 2 days ago

so addition is not associative?

Reply View 12 replies
  • godelski 2 days ago

    I think you're missing the point

    Reply View 11 replies
    • yellowcake0 a day ago

      It's a pretty exotic type of addition that would lead to the second set of examples, just trying to get an idea of its nature.

      Reply View 10 replies
      • godelski a day ago

        Calling it addition is hairy here. Do you just mean an operator? If so, I'm with you. But normally people are expecting addition to have the full abelian group properties, which this certainly doesn't. It's not a ring because it doesn't have the multiplication structure. But it also isn't even a monoid[0] since, as we just discussed, it doesn't have associativity nor unitality.

        There is far less structure here than you are assuming, and that's the underlying problem. There is local structure and so the addition operation will work as expected when operating on close neighbors, but this does greatly limit the utility.

        And if you aren't aware of the terms I'm using here I think you should be extra careful. It highlights that you are making assumptions that you weren't aware were even assumptions (an unknown unknown just became a known unknown). I understand that this is an easy mistake to make since most people are not familiar with these concepts (including many in the ML world), but this is also why you need to be careful. Because even those that do are probably not going to drop these terms when discussing with anyone except other experts as there's no expectation that others will understand them.

        [0] https://ncatlab.org/nlab/show/monoid

        Reply View 9 replies