I would really recommend using phone jack and cable, as they are 4 wire, and don't have the issue with hot connection shorting VCC to pins it shouldn't be sent to.

Or use something else. TRRS is used because it's easy to use, low profile-ish, and frequently used. Something like this may be a better option, even.

Also, have you seen the variant of the 5x6 that I'm using, with a trackball in the thumb cluster on the right side?