I think USB data rate is too high to stay in-sync.
There is no quick solution for this. You could wait until LE is efficient enough, or you could buy a HiFiBerry DAC. Because HiFiBerry DAC uses GPIO instead of USB, it can handle high audio data rates more efficient.