Comment by williamtrask 4 days ago
I love homomorphic encryption, but why can't they just do a neural search locally?
- iOS Photos -> Vectors
- Search Query "Dog photos" -> Vectors
- Result (Cosine Similarity): Look some dog photos!
iPhones have plenty of local storage and compute power for doing this kind of thing when the phone is idle. And cosine similarity can work quickly at runtime.
I discuss this in the post:
> This seems like a lot of data the client is getting anyway. I don’t blame you for questioning if the server is actually needed. The thing is, the stored vectors that are compared against are by far the biggest storage user. Each vector can easily be multiple kilobytes. The paper discusses a database of 35 million entries divided across 8500 clusters.
Here's an open source iOS app[1] that does that. Incidentally, it's built using Apple's own MobileCLIP[2] models.
[1]: https://github.com/fguzman82/CLIP-Finder2 [2]: https://github.com/apple/ml-mobileclip
Apparently they only do the cloud HE song and dance for landmarks, which is too big of a data set to realistically keep on-device.