Security
When you use X11, you allow any program running on your computer to access anything on your screen and clipboard, collect your keystrokes and type. It's trivial to implement a keylogger, for example. Do not buy into the whole "no viruses on Linux" thing, it's not true and likely to become even less and less true, as desktop Linux is becoming popular.
Wayland at least tries to put some barriers in place against this.
Can anyone break even AES-128 without years of supercomputer time?