You could use something like this:
http://www.amazon.com/Belkin-Speaker-and-Headphone-Splitter/dp/B000067RC4 if you only have one device outputing a signal at a time.
Or this:
http://electronicsusa.com/mk1and2.html
Or build something yourself:
http://www.instructables.com/id/Audio-Switcher-Box-18quot-stereo-jacks/
However, when I did this in the past for some reason the audio was much quieter than I expected. I needed an amp to make it loud enough to enjoy.