Ha
I fixed it myself.
Thanks to some suggested threads below I put gpu_mem=96 in config.txt and now it's working!
Also I noticed that disabling hardware decoding also solved the issue but that wasn't what I was looking for.
Soooo... first thanks for the help and second, how come gpu_mem=96 isn't the default instead of 76?