Sorry for not mentioning that I am using analog output. I thought it doesn't matter cause some days ago I thought to also have the problem via HDMI. But yesterday I could not reproduce it on HDMI audio output anymore.
Anyway, the passthrough setting does not make a difference for me on the analog output. I think that is expected though. Thanks anyway!