RPi4 doesn't have a clock, so LE needs network connectivity and a working network stack to reach an NTP server to set it during the boot. If you do have an RTC, then you would likely have to enable it with an overlay, check with who sold the device. It will likely need NTP to initially set the clock.
I don't know anything about HifiBerry, but pointing out the obvious that it also depends on your media source. If your media source is not 5.1, is poor quality, then garbage in/garbage out. I'll leave it to DAC users to comment, I just use audio over HDMI.