From what I can see with View Mode and Zoom:
Starting from "Normal" mode if you enable "Zoom" mode it starts with "Zoom amount = 1.0" and when you make any change to the amount value the selected mode automatically changes to "Custom" mode. I've zoomed a 4:3/SD letterbox video so the sides are closer to the edge of my 16:9 screen resulting in "Zoom amount = 1.15" and if you toggle between "Normal" and "Custom" modes the zoom value is persistently applied.
However, if I now switch back to "Zoom" the display switches to a mode similar to "Wide zoom" and not the current "Custom" value that I defined. This probably makes sense in the context of not wanting to automatically revert to "Zoom amount = 1.0" when going back to "Zoom" as this would presumably change the "Custom" value automatically too. However, if I now move to change "Zoom amount" the wide zoom mode is shown until the amount value is adjusted; only then the screen switches to the "Custom" amount value. I also note that if I adjust "Zoom amount" back to 1.0 the mode auto-switches back to "Normal" mode again.
This is probably old code and "not a bug" since "code is working as designed" but the design could use improvement. Unless there is some use-case I'm not understanding I think it would be more logical to simply have "Zoom" respect the current "Zoom amount" value defined and eliminate "Custom" mode.
As this is nothing specific to LibreELEC I'd suggest you report this to Kodi developers via their forum or GitHub issues.