Unmuting doesn't work in Xfce?
I had this problem too after installing Debian Testing (Jessie) which now comes with Xfce by default, but you may encounter it on any system running Xfce DE.

Every time I muted the audio using the mute key on my keyboard, everything worked find - the audio was indeed muted. But when I tried to unmute using the same key, the audio indicator showed the unmuted state but audio was still muted.
After some investigation I understood that the only way to unmute my system was entering the command:

alsamixer

in the terminal and then press 'M' to toggle the mute state.
Apparently, the Audio Mixer was muting PulseAudio AND Alsa, but unmuting ONLY Alsa.

If you run the following command, it will list all the properties under xfce4-mixer:

xfconf-query -lc xfce4-mixer

In my case, what I saw is only one mixer under /sound-cards/ entry. It was Alsa, PulseAudio just wasn't there.
Also, in the audio mixer I saw only the Alsa entry and couldn't even control the PulseAudio mixer as you can see here:

From what I understood, xfce4-mixer (the Audio Mixer) was missing an optional dependency to be able to work with PulseAudio.
This dependency is: gstreamer0.10-pulseaudio.
After installing the above dependency, using (on debian):
sudo apt-get install gstreamer0.10-pulseaudio
the Audio Mixer looked a lot better:

But, running:

xfconf-query -c xfce4-mixer -p /active-card

to query the active card used by xfce4-mixer still returned "HDANVidiaAlsamixer".
So the last thing I needed to do is change the "active card" in xfce4-mixer too by running:

xfconf-query -c xfce4-mixer -p /active-card -s PlaybackBuiltinAudioAnalogStereoPulseAudioMixer

After that, toggling mute finally worked.