CNAME should work without a problem. I use DNS to name "services" and then CNAME them to the host, so Kodi uses "kodi--fs1" which points to my server.
My guess is machine #2 is not using DNS, but probably using mDNS (Avahi). So what I do is (on the server) create mDNS entries as well via a systemd unit avahi-alias. So you might want to make sure your machine #2 is using a FQDN or your DHCP server is pushing the DNS search order. Which will be a problem since the hostname is embedded in the database.
LE also prefers mDNS over DNS, so you may have to create a systemd unit on your server hosting your file server to create avahi entries. I have a systemd unit called avahi-alias:
# cat /etc/systemd/system/[email protected]
[Unit]
Description=Publish %i as alias for %H.local via mdns
[Service]
Type=simple
ExecStart=/bin/bash -c "/usr/bin/avahi-publish -a -R %i $(avahi-resolve -4 -n %H.local | cut -f 2)"
[Install]
WantedBy=multi-user.target
It is enabled as
sudo systemctl enable --now [email protected]
kodi-fs1 being the mDNS alias, ".local" is the mDNS domain namespace. The simplest solution is already from chewitt, which is change the hostname of your file server. This solution would also require your old file server to be offline, or Avahi not publishing there.
The other simple solution is creating a hosts entry on the LE device in /storage/.config/hosts.conf. You would have to dig into what's causing the problem here to figure out the appropriate solution, it's probably as simple as using avahi-resolve, nslookup, ping, on the LE device to figure out what's being resolved and what is resolving it.