Comment by catoc
That is the only implementation I can think of that might make me trust a third party with confidential data. Of course these massive transformers are already insanely computer intensive and adding FHE would make that orders of magnitude worse.
I agree but that's why it needs to be researched more. Some changes in architecture may be able to address some performance problems. It could lead to algorithmic optimizations or even specialized hardware for this.