NFS performance issue

JurKub · November 21, 2021 at 7:41 AM

Hi guys,

when VDR-Office is doing more the 2 recordings at the same time replay on VDR-Living is chopping, both using NFS.

If I do the repay on VDR-Living using Samba it works fine.

This is a rough drawing of my LibreELEC setup.

NFS mountings:

VDR-Office:

/storage/videos -> FhemRaspi:/srv/vdr/video

VDR-Living:

/storage/videos -> FhemRaspi:/srv/vdr/video

FhemRaspi /etc/exports:

Code

#################################################################################
#                                                                               #
#   The following configuration file is generated automatically by the yaVDR    #
#     system. Don't change this file as every update of yaVDR will overwrite    #
#         the local changes. Instead put your required customizations           #
#       into /etc/yavdr/templates_custom/ based on the original templates       #
#                      under /usr/share/yavdr/templates.                        #
#                                                                               #
#            http://www.yavdr.org/developer-zone/template-overview/             #
#                                                                               #
#                                                                               #
#################################################################################
# /etc/exports: the access control list for filesystems which may be exported
#        to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#
/srv      *(rw,fsid=0,sync,no_subtree_check,no_root_squash,anongid=666,anonuid=666)
#
# /srv      *(rw,fsid=0,sync,no_subtree_check,all_squash,anongid=666,anonuid=666)
/srv/share/vdr      *(rw,fsid=0,async,no_subtree_check,all_squash,anongid=666,anonuid=666)

/srv/audio      *(rw,fsid=1,async,no_subtree_check,all_squash,anongid=666,anonuid=666)

/srv/video      *(rw,fsid=2,async,no_subtree_check,all_squash,anongid=666,anonuid=666)
/srv/picture      *(rw,fsid=3,async,no_subtree_check,all_squash,anongid=666,anonuid=666)

Display More

Any idea what I need to change to get it working all using NFS ?

Thx

JurKub

**Klojum** · November 21, 2021 at 8:31 AM

What OS is running on the RPi4 server with 4TB?

Handling files is a relatively demanding job on a CPU for a RPi, both USB30 and Gigabit ports really only work at half speeds in my experience. So depending on the overall bitrate of the VDR streams, it's possible that the RPi4 is simply not up to the task, despite having set up NFS correctly.

JurKub · November 21, 2021 at 10:43 AM

Quote from Klojum

What OS is running on the RPi4 server with 4TB?

Linux fhemraspi 5.10.63-v7l+ #1459 SMP Wed Oct 6 16:41:57 BST 2021 armv7l GNU/Linux

Quote from Klojum

Handling files is a relatively demanding job on a CPU for a RPi, both USB30 and Gigabit ports really only work at half speeds in my experience. So depending on the overall bitrate of the VDR streams, it's possible that the RPi4 is simply not up to the task, despite having set up NFS correctly.

I wondering using NFS for recording and Samba for replay at the same time everything woks fine.

The problem exists only when using NFS for recording and for replay at the same time !

**Klojum** · November 21, 2021 at 12:06 PM

Quote from JurKub

I wondering using NFS for recording and Samba for replay at the same time everything woks fine.

There is only one way to find that out.

frakkin64 · November 21, 2021 at 12:13 PM

I am running Tvheadend with 6 tuners (2 tuners are over Ethernet, and 4 tuners are in a PCI card in the computer) on an AMD 8-Core machine with 20TB storage, it has no problem recording all tuners at the same time, plus serving 2 medacenter devices over NFS. It serves both my media library and Tvheadend content (which is over HTSP [TCP]).

Have you ran bcmstat.sh on your Pi to get more insight? I would run it with: "bcmstat.sh xce ieth0", you might need to change the eth0 to your network device. I would run that on your Pi hosting the storage, it will give you good visibility to your RX/TX network plus CPU/MEM usage. You might need to check your kernel settings for the number of NFS daemons running (on a Debian based system this is in /etc/default/nfs-kernel-server, changing RPCNFSDCOUNT). I run with 8 myself which is probably overkill since I don't have the overhead of writing to NFS over the network from Tvheadend like you do.

This article has some advice on tuning nfsd threads:

Document Display | HPE Support Center

JurKub · November 21, 2021 at 1:27 PM

Quote from Klojum

There is only one way to find that out.

would you tell me how

**Klojum** · November 21, 2021 at 1:52 PM

Quote from JurKub

would you tell me how

What you earlier suggested:

Quote from JurKub

using NFS for recording and Samba for replay at the same time

JurKub · November 21, 2021 at 1:57 PM

Quote from JurKub

using NFS for recording and Samba for replay at the same time everything woks fine.

this feels like a workaround but not like a solution

**Klojum** · November 21, 2021 at 2:25 PM

Sure... But "if it looks stupid but it works, then it isn't stupid".

JurKub · November 21, 2021 at 2:28 PM

Quote from Klojum

Sure... But "if it looks stupid but it works, then it isn't stupid".

OK I give up

frakkin64 · November 21, 2021 at 2:42 PM

Quote from JurKub

I wondering using NFS for recording and Samba for replay at the same time everything woks fine.
The problem exists only when using NFS for recording and for replay at the same time !

One other thing to consider here is perhaps NFS locking is an issue, especially if you are trying to watch a file that is being written to (i.e. trying to watch a live recording being written by VDR). Probably worthwhile to ask this question on the VDR forums, as you will probably find someone with similar setups. Tvheadend & MythTV work completely differently, everything is streamed by Tvheadend or MythTV, client doesn't access the recorded media files directly.

There is just too much opportunity for speculation here with very little data, unfortunately.

JurKub · November 21, 2021 at 2:54 PM

Quote from frakkin64

I am running Tvheadend with 6 tuners (2 tuners are over Ethernet, and 4 tuners are in a PCI card in the computer) on an AMD 8-Core machine with 20TB storage, it has no problem recording all tuners at the same time, plus serving 2 medacenter devices over NFS. It serves both my media library and Tvheadend content (which is over HTSP [TCP]).
Have you ran bcmstat.sh on your Pi to get more insight? I would run it with: "bcmstat.sh xce ieth0", you might need to change the eth0 to your network device. I would run that on your Pi hosting the storage, it will give you good visibility to your RX/TX network plus CPU/MEM usage. You might need to check your kernel settings for the number of NFS daemons running (on a Debian based system this is in /etc/default/nfs-kernel-server, changing RPCNFSDCOUNT). I run with 8 myself which is probably overkill since I don't have the overhead of writing to NFS over the network from Tvheadend like you do.
This article has some advice on tuning nfsd threads:
https://support.hpe.com/hpesc/public/d…mr_na-c02239048

I don't think the Ethernet port is the issue, because using NFS for recording and Samba for replay at the same time everything woks fine

/etc/default/nfs-kernel-server

Code

# Number of servers to start up
RPCNFSDCOUNT=8

# Runtime priority of server (see nice(1))
RPCNFSDPRIORITY=0

# Options for rpc.mountd.
# If you have a port-based firewall, you might want to set up
# a fixed port here using the --port option. For more information, 
# see rpc.mountd(8) or http://wiki.debian.org/SecuringNFS
# To disable NFSv4 on the server, specify '--no-nfs-version 4' here
RPCMOUNTDOPTS="--manage-gids"

# Do you want to start the svcgssd daemon? It is only required for Kerberos
# exports. Valid alternatives are "yes" and "no"; the default is "no".
NEED_SVCGSSD=""

# Options for rpc.svcgssd.
RPCSVCGSSDOPTS=""

Display More

I've tied sync and async as well in

/etc/exports

Code

#################################################################################
#                                                                               #
#   The following configuration file is generated automatically by the yaVDR    #
#     system. Don't change this file as every update of yaVDR will overwrite    #
#         the local changes. Instead put your required customizations           #
#       into /etc/yavdr/templates_custom/ based on the original templates       #
#                      under /usr/share/yavdr/templates.                        #
#                                                                               #
#            http://www.yavdr.org/developer-zone/template-overview/             #
#                                                                               #
#                                                                               #
#################################################################################
# /etc/exports: the access control list for filesystems which may be exported
#        to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#
/srv      *(rw,fsid=0,sync,no_subtree_check,no_root_squash,anongid=666,anonuid=666)
#
# /srv      *(rw,fsid=0,sync,no_subtree_check,all_squash,anongid=666,anonuid=666)
/srv/share/vdr      *(rw,fsid=0,async,no_subtree_check,all_squash,anongid=666,anonuid=666)

/srv/audio      *(rw,fsid=1,async,no_subtree_check,all_squash,anongid=666,anonuid=666)

/srv/video      *(rw,fsid=2,async,no_subtree_check,all_squash,anongid=666,anonuid=666)
/srv/picture      *(rw,fsid=3,async,no_subtree_check,all_squash,anongid=666,anonuid=666)

Display More

JurKub · November 21, 2021 at 3:12 PM

Quote from frakkin64

One other thing to consider here is perhaps NFS locking is an issue, especially if you are trying to watch a file that is being written to (i.e. trying to watch a live recording being written by VDR). Probably worthwhile to ask this question on the VDR forums, as you will probably find someone with similar setups. Tvheadend & MythTV work completely differently, everything is streamed by Tvheadend or MythTV, client doesn't access the recorded media files directly.
There is just too much opportunity for speculation here with very little data, unfortunately.

I don't try to watch a live recording being written by (another) VDR.

It doesn't matter if I watch a VDR recording or another mp4 video, same issue.

I'm more than happy to provide as much data you like, just ask for what you need to see

frakkin64 · November 21, 2021 at 3:13 PM

Quote from JurKub

I don't think the Ethernet port is the issue, because using NFS for recording and Samba for replay at the same time everything woks fine

OK, so speculation it is then. This is where my part in the conversation ends. Good luck.

GDPR-7 · November 21, 2021 at 9:35 PM

JurKub

- just my 2 ct -

would playing with wsize and rsize have an affect ?

see:

nfs(5) - Linux man page

moons ago I played with it to see how to get the best throughput.

it turned out that the defaults were not that ideal.

at that time the defaults were: rsize=1048576,wsize=1048576

I got the best with: rsize=32768,wsize=32768

but -as said - it's years ago: old NFS server (3.x ?), old disk, ...

write: read: in MB/s

=================

6,91 49,68 NFS-default wsize, rsize)

35,31 81,45 rsize=8192,wsize=8192

34,90 101.75 rsize=16384,wsize=16384

49.86 110.62 rsize=32768,wsize=32768 (nearly full GB-LAN speed [ 125 MB/s (theoretically)]

JurKub · November 22, 2021 at 8:59 AM

Quote from JoeAverage

JurKub
- just my 2 ct -
would playing with wsize and rsize have an affect ?
see:
https://linux.die.net/man/5/nfs
moons ago I played with it to see how to get the best throughput.
it turned out that the defaults were not that ideal.
at that time the defaults were: rsize=1048576,wsize=1048576
I got the best with: rsize=32768,wsize=32768
but -as said - it's years ago: old NFS server (3.x ?), old disk, ...
write: read: in MB/s
=================
6,91 49,68 NFS-default wsize, rsize)
35,31 81,45 rsize=8192,wsize=8192
34,90 101.75 rsize=16384,wsize=16384
49.86 110.62 rsize=32768,wsize=32768 (nearly full GB-LAN speed [ 125 MB/s (theoretically)]

Display More

rsize and wsize don't change nevertheless what I'm entering

Code

10.0.0.53:/srv/vdr/video on /storage/videos type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.53,mountvers=3,mountproto=tcp,local_lock=none,addr=10.0.0.53)
/

**heitbaum** · November 22, 2021 at 10:32 AM

Does iftop / nmon show anything useful?

GDPR-7 · November 22, 2021 at 4:41 PM

another tool to measure bandwidth, MSS/MTU, etc. could be iPerf3:

iPerf - The TCP, UDP and SCTP network bandwidth measurement tool

usage:

iPerf - iPerf3 and iPerf2 user documentation

P.S.

maybe it's worth to check on what speed the network interfaces are currently running ?

Background:

got a new mainboard with 2.5 Gbit Intel LAN (driver: igc) running in an 1 Gbit LAN with dhcp and sometimes I got 100 Mbit only on the igc.

- it could be the kernel (5.15.y) , it could be the new firmware on the router too, ..., no idea ... -

question: is your experiences a regression ?