You could definitely stack switches like that, but it'll get messy. I don't think it's worth doing, especially since there are 4P4T and even 4P5T switches out there, just a much more limited selection. And there are also 4P3T switches of a similar form factor if you still want a 3-way output selector; you'd just need to find matching D-shaft knobs.

For jacks, it looks like you're either going to spend a bit of money, or you're going to have plastic jacks. That's not the end of the world (and isolates all the jacks, which is important if using 4P switches instead of just 3P), but it is a bit chintzy. Fortunately, the quality of the jacks shouldn't change the sound any. These would be easy. Here's a much nicer set, though from the datasheet I can't tell if the panel mounting is insulated. If it's not, and you mount them in a metal project box, all the device grounds will be tied together. I don't actually know whether this is a problem, but it's something I've always avoided; if you wind up with this situation there'd be no point in the 4P switches and you could drop back to 3P (still 4 or 5T, though).

For RCA, I've used these and been very happy.