Soooo... first thanks for the help and second, how come gpu_mem=96 isn't the default instead of 76?
The Pi's hardware video decoder is designed for level High Profile, level 4.1 H.264 encodes, as used by Blu-Ray.
We'll play any file encoded to that format with the default gpu_mem.
My guess if your file has been badly encoded (possibly a large number of reference frames requiring level 5).
You may be able to play these with increased gpu_mem (but it's not guaranteed). Best to avoid these encodes if you can.