You only need a single 12-24V power supply to power the Raspberry Pi and the Amp4. With 4 Ohm speakers, we recommend 12-20V. Higher operating voltages are only recommended with 8 Ohm speakers.
What's the power supply of your RPi / HiFiBerry system? Maybe you don't get enough wattage for A/V.