OK, then you're probably right: The RPi must down-sample the stream somewhere. An RPi doesn't have a very fast data bus, so I think the down-sampling happens due to hardware limitation. LE usually deals with A/V streams, so my advice is to use an audio-optimized OS. Such an OS might be slow on video playback, but has the ability to deal with higher audio bit rates.
There already is an LE audio mod for RPi 2/3, which can deal with higher audio bit rates. Maybe those devs will switch to RPi 4 next.