Ok, so reading that yours is playing back files of that size ok made me check a little further. One file in particular that buffered a lot that I expected to play ok is a 21gb file that is 53 minutes in total. The average bitrate there is well within what it should be capable of, not sure of the peak bitrate.
I did a quick check to get an idea of the actual connection speed using speedtest.net and I seem to be getting 80-90Mbit (My internet connection is 240Mbit so it should be a pretty good indication). I'll investigate further tomorrow.steve_192: Thanks, I played around with the buffer settings a lot already and I tried pretty aggressive settings in the end but all it does is increase the time before the problems occur.
Are any of the 4K files you are having problems with in the public domain and freely downloadable so I can test?