Enabling "HDMI ULTRA HD Deep Color" is recommended as that'll make it possible to drive the display with 4kp60 (without it it'll be limited to 4kp30 on the RPi4). Other than that it doesn't make a difference for now.
I meant checking the picture settings (i.e. where you can select picture mode, brightness, contrast etc).
If the difference for non-4k videos is minor these settings are probably fine.
I still suspect you are playing HDR 4k videos (eg HDR10, HLG or DolbyVision). In that case the difference will be indeed huge as RPi/Kodi can't currently output HDR. This means the maximum brightness, contrast etc will be far lower than if you play these natively on your TV.
BTW: if you play HDR videos on your TV you may also notice a "DolbyVision" or similar popup and a change of available picture modes in the TV's settings.
Basic support for HDR output has just landed in the Linux kernel and there are still quite some bits missing (in RPi video driver and Kodi) until HDR will be supported.
so long,
Hias