I had a similar problem with my RP4, where CEC worked only sparsely or very delayed (up to 20 minutes!!). It took me some hours (and some hdmi cables+adapters) to figure out, that on the other micro-hdmi port all was working well.
So make sure, you tried both ports on the RasPi...