Comment by kibwen
It's one thing to wish that apps would put their data anywhere except dumping it in your home dir, but this is exactly why I hate the XDG spec. I want all data for a program--be it the configuration or the cache or the binary itself--to be in a single directory such that 1) "uninstalling" the program, completely and in isolation, is nothing more than just deleting that single directory, and 2) any program not doing arbitrary file I/O can entirely function while having access to only its installation directory, and nothing else on the filesystem.
This approach couples together everything though, in such a way there's no standard manner of wiping cache but not your app, configuration, etc.
XDG may not be perfect but wiping related data for apps following it is straightforward. There are a few directories to delete instead of 1, but still consistently structured at least.