You can easily mute other people by accessing the participant list on the right hand side of the party view.
To mute a single person, click the speaker icon next to their name.
To mute all other participants, click the "Unmute all" text at the very top of the participant list. This is particularly useful if you are streaming content from your computer using Computer Audio.
If you want to unmute people, just do the same thing as above but in reverse.
Enjoy the silence!