Comment by bjourne
I believe the Rust compiler does whole program analysis or, at least, something very similar in spirit. Hence, unless you completely change how the language works it can't be as fast as a simpler compiler. You can work around it by having a daemon that incrementally compiles only changed parts, but then you need to have that daemon running and it's never as smooth as one command that compiles quickly. In other words, yes, I think you are right. Slow compile times reduce adoption.
None of the passes rustc does are whole-program. LLVM does some whole-program passes, but that's also true for C++.