Posts by camelreef

Quote from HiassofT

Code

Feb 10 17:00:10 kodiplayer smbd[6089]:   smb2_sendfile_send_data: sendfile failed for file Transit/Incoming/Game.of.Thrones.S03.2160p.BluRay.REMUX.HEVC.DTS-HD.MA.TrueHD.7.1.Atmos-FGT/Game.of.Thrones.S03E06.The.Climb.2160p.BluRay.REMUX.HEVC.DTS-HD.MA.TrueHD.7.1.Atmos-FGT.mkv (Connection reset by peer) for client ipv4:10.25.25.1:46624. Terminating

"Connection reset by peer" means the client (i.e. the other machine) unexpectedly closed the connection.

Might be worth looking into what the client is doing and also testing if the issue happens with other clients (eg a Windows box).

so long,

Hias

I would theorise that this is the client resetting things because all hell broke loose at some point, and not the root problem. It is a pretty rare occurrence.

I am, however testing both your and my theories, moving large files back and forth using another client and watching the server's logs.

Just a reminder from another thread, I had tested my client using another Samba server, one running on my old-ish Lenovo-EMC NAS, without any hiccup whatsoever, as I naturally wanted to find out if it was a server or client issue.

Quote from JoeAverage

have you tried to do what is written in the log ?

Code

Feb 10 16:39:35 kodiplayer smbd[5958]:   If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting

Should I really push that issue upstream now? Is LE using the latest and greatest from Samba? Is the issue really a Samba bug?

I have no problem contacting the Samba guys and following up, but I feel that there is a bit more that needs gathering before that.

Also, LE is using Samba 4.13.17, samba.org announces that the latest stable is 4.15.5. I can't judge the LE decision to use an older version, but there must be a good one. I'm pretty sure that if I contact upstream, the first reaction will be to ask me to use the latest version first and go from there, as mentioned in the log message.

After a while I get smbd becoming unstable and segfaulting.

Still not correlated with the client in the timestamps

http://ix.io/3PeH

Nico

Quote from HiassofT

Can you test with a simple, stock network configuration? i.e. plain LE installation with just wired (or wifi) network enabled but not both wifi and wired at the same time.

I'm suspecting network / routing could be the culprit, that test should help to narrow it down.

so long,

Hias

Single network, wired, was how it was until very recently, and the problem was exactly the same.

I have changed to that second backend network only once I thought that the new bootloader/firmware/kernel/samba combo had stabilised things, in error, when I thought that I could put the kit in production at my friend's.

Single network or two networks show exactly the same behaviour.

In any case, this is all very simple networking, a directly connected class C on one interface, with the default route, another directly connected class C on another interface. The mount is using the IP address on that directly connected class C network. There is no need for any IP forwarding to be turned on.

Nothing is confusing, the routing table is exactly as it should be, with proper metrics, as is the IP assignment on the interfaces.

kodiplayer:~ # ip rout sh
0.0.0.0 dev eth0 scope link 
default via 192.168.1.1 dev wlan0 
8.8.4.4 dev eth0 scope link 
8.8.8.8 dev eth0 scope link 
10.25.25.0/24 dev eth0 scope link  src 10.25.25.2 
192.168.1.0/24 dev wlan0 scope link  src 192.168.1.136 
192.168.1.1 dev wlan0 scope link 


kodiplayer:~ # ip add sh
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP8000> mtu 1500 qdisc mq qlen 1000
    link/ether e4:5f:01:86:99:96 brd ff:ff:ff:ff:ff:ff
    inet 10.25.25.2/24 brd 10.25.25.255 scope global eth0
       valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP8000> mtu 1500 qdisc fq_codel qlen 1000
    link/ether e4:5f:01:86:99:98 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.136/24 brd 192.168.1.255 scope global wlan0
       valid_lft forever preferred_lft forever

The issue is not network related.

Quote from HiassofT

Please try to add "debugging" to the end of cmdline.txt, this enables extra debug logging (for connman and other services) which might provide important information. Then upload logs as usual (journalctl should be the most important).

so long,

Hias

Here is the journalctl with the system booted with the debugging option:

http://ix.io/3Pd4

NOT enlightening...

For info, here are the timestamps when the client complained about the share being unavailable:

Feb 10 12:05:07 grabber kernel: CIFS: VFS: \\10.25.25.2 sends on sock 00000000c37af84c stuck for 15 seconds
Feb 10 12:05:07 grabber kernel: CIFS: VFS: \\10.25.25.2 Error -11 sending data on socket to server
Feb 10 12:05:31 grabber kernel: CIFS: VFS: \\10.25.25.2 sends on sock 0000000083fa5a17 stuck for 15 seconds
Feb 10 12:05:31 grabber kernel: CIFS: VFS: \\10.25.25.2 Error -11 sending data on socket to server
Feb 10 12:05:31 grabber kernel: CIFS: VFS: Send error in read = -11
Feb 10 12:05:47 grabber kernel: CIFS: VFS: \\10.25.25.2 sends on sock 0000000083fa5a17 stuck for 15 seconds
Feb 10 12:05:47 grabber kernel: CIFS: VFS: \\10.25.25.2 Error -11 sending data on socket to server



Feb 10 12:06:13 grabber kernel: CIFS: VFS: \\10.25.25.2 sends on sock 0000000024556f67 stuck for 15 seconds
Feb 10 12:06:13 grabber kernel: CIFS: VFS: \\10.25.25.2 Error -11 sending data on socket to server
Feb 10 12:06:26 grabber kernel: CIFS: VFS: Send error in read = -11
Feb 10 12:06:26 grabber kernel: CIFS: VFS: \\10.25.25.2 trying to return 10 credits to old session

What else could I turn on for you that could enable something more useful?

Nico

Quote from HiassofT

Please try to add "debugging" to the end of cmdline.txt, this enables extra debug logging (for connman and other services) which might provide important information. Then upload logs as usual (journalctl should be the most important).

so long,

Hias

Doing it now, but, of course, there has been absolutely zero smb drops yet!

I'm now cross-compiling an LE11 nightly-20220120-f7f2fd5 image. Seeing if that one works well or not could be a nice data point.

Ah, crap, I can't... Debian is not hosting http://ftp.debian.org/debian/pool/ma….26.orig.tar.gz anymore...

Darnit!

Quote from HiassofT

Please try to add "debugging" to the end of cmdline.txt, this enables extra debug logging (for connman and other services) which might provide important information. Then upload logs as usual (journalctl should be the most important).

so long,

Hias

It shall be done tomorrow!

Nico

Quote from heitbaum

As a note - the Ntfs version being used in nightlies is now ntfs3 - https://github.com/LibreELEC/LibreELEC.tv/pull/5838 - which is the Paragon Kernel version. but it is a different version to the ntfs3g that we and other distributions were using before kernel 5.15.

Sorry! Force of habit...

Hello all!

I run an RPi4 h/w rev 1.5 with LE11 nightly-20220207-72b6d0d (RPi-LE).

Wi-Fi is used to connected to my home network and the internet. The Ethernet port is used for a direct link to another RPi (preferred technology order changed in connman).

It has a 5TB USB3 HDD connected to it with single partition formatted in NTFS, mounted by ntfs3g. That hard drive's partition is shared over Samba. The drive's content is 4K HDR videos with HBR audio, directly connected for performance purposes.

There is another Raspberry Pi (RPi-Mgr), running DietPi, connected to the first using the private Ethernet link, mounting the above shared drive through this link, vers=3.1.1 (tried going down to 2.0 with the same results). This Pi manages the content on the USB hard drive.

Randomly, and quite regularly, RPi-Mgr reports that the mounted network share stops responding, which creates havoc higher up in the software stack.

I have tried using a network mount using SMB from my NAS, using the exact same workflow, with ideal results. So RPi-Mgr can be considered fine.

So I've taken a look at RPi-LE, the Samba server.

I may have been pretty annoying in another post, thinking that the h/w rev and firmware could be problematic, apparently in error... Oooops...

I started turning the smbd logs to level 10 (stupidly verbose) about everything and went about the normal workflow, looking for the time the client reports the server as unresponsive. I then grepped the server smbd logs for that time. Uh oh... No result!

So i went into the file, massively dense logs, and searched for the timestamp. Nada, zilch, zip, rien, nothing! There is a gap in all logs for about 20s! No information appears before or after indicative of any ill reaction. It's as if smbd took a break for an espresso cup and came back, as if nothing was wrong... Nothing weird appears in the journalctl logs about the rest of the system either.

Now, that is annoying... There is something that is clearly happening, but I can't find any verbosity about it.

No data, no help, no fix...

What could I do to generate anything that would be useful?

I have an identical pair of RPis, one running LE11 nightly-20220120-f7f2fd5 on Raspberry Pi 4 Model B Rev 1.2, without any issue at all. That on is in production, so I'm not too tempted to upgrade it. I just wish I had an installable image of that version handy to put on the dev kit as a comparison point.

Nico

Guys, HiassofT,

I still have issues with my Samba server going down.

It's not as often as before, but it can really become unproductive.

Is there anything on the pipeline that could help? Anything I could provide/do to help chase it down?

Thanks!

Nico

Quote from chewitt

If you poke around in Escalade's GitHub repo you'll find some prior art for adding an NFS server. It's not something we've ever felt the need for in core images as Samba works well enough and most folks will just get confused. FWIW, I've always used SMB and never had any issues (other than occasional self-inflicted mistakes). It might not be "fastest" but it's always been "fast enought" .. over Ethernet of course.

For the purpose of my little stack, where it's used for offline/background stuff quite invisible from the user, SMB works well enough.

But it's got to work, and that rev. 1.5 Pi 4 did not play ball!

Quote from frakkin64

Ahh, I get it. Catering to a more novice user base that is familiar with Windows. I don't envy the tech support you will have to endure.

Good to hear. Sounds like it was a combination of a few things, firmware bump to perhaps address the rev 1.5 board firmware timeouts/splats, and Samba bump/mount option changes, care to summarize what you think solved it? It may help others that come across this problem.

Does LE11 have NFS server capabilities? I don't believe so, or at least it wasn't there the last time I've looked. If it made its way in, I'm switching to it instantly, dropping Samba!

Tech support is light. They use Kodi and a couple of very well made web pages through a small portal Iv integrated on the non LE RPi.

In any case, I always have my remote access for system maintenance, which takes away a lot of potential pain.

So far, so good!

As for what fixed it? Well...

1. a lot has changed in one go: firmware, bootloader and Samba (I didn't see a version bump for Samba, though), at least. I don't believe that I can point at the main reason in any educated way. Guts say firmware...
2. it's not completely fixed, there are still drops, but shorter and more spaced in time, so the apps relatively gracefully swallow them

Quote from HiassofT

FYI: new nightly with kernel/firmware update is now available https://test.libreelec.tv/LibreELEC-RPi4…203-d383c10.tar

Things are much better! There is a very clear improvement But... it's not completely fixed, there are still drops, but shorter and more spaced in time, so the apps using the samba mount relatively gracefully swallow them without degrading into painful death

Quote from noggin

Yes - in this case the video is output as 8-bit (usually RGB), but accompanied with an HDMI flag to tell your TV that the video isn't to be displayed in SDR but is instead in a PQ HDR dynamic range (which is what HDR10 content uses), and that triggers your TV into an HDR mode. You may see banding that you wouldn't see with a full 10-bit path.

More recent nightly LE 11 builds will ensure the full 10-bit source is preserved and output, and remove the 8-bit bottleneck.

I can attest that it is indeed working beautifully!

Quote from HiassofT

FYI: new nightly with kernel/firmware update is now available https://test.libreelec.tv/LibreELEC-RPi4…203-d383c10.tar

so long,

Hias

I may be jumping the gun, but... I would already have had a CIFS mount dropout by now... It appears to be working! Woohoo!

I'm, like always, impressed! Open Source rocks, you guys make it so!

Thanks a ton!

Nico

Quote from ghtester

This is NOT the same file.

Rev 1.5 RPi 4 - LE11 nightly-20220203-5347c5e

LibreELEC-RPi4.arm-11.0-nightly-20220203-d383c10.tar

Ther are now 2 Nightly builds dated 20220203 so you have to look at the filename tail...

Believe to HiassofT and give it a try. `

Oh, yes! Apologies offered!

I'm so used to the clockwork daily releases that I did not consider a double release day!

As Mulder used to say... I want to believe!

Quote from frakkin64

So you ran Samba on Raspberry Pi OS and did the same testing and everything was fine? Interesting. I kind of got lost in all your updates.

Well, the other thing to point out is there should be a Samba bump to 4.13.17 in tomorrow's nightly along with HiassofT's updates, so you can maybe look out for that as well. Personally, I don't use Samba at all with Kodi/LE, just NFS. Samba has a lot of protocol overhead, that NFS doesn't have. Scanning videos on a large library takes a long time over the network with Samba. I also only use the Pi to play media, that's all, no docker crap, no samba server crap (it's disabled), not even cron.

Beyond that, perhaps you need to fire up Wireshark and do some packet capturing to see what is happening at a protocol level. I assume the only problem left is this Samba issue? I am assuming the crash on kodiplayer is because the connection was reset from the peer.

I totally agree with you about NFS vs Samba. There is no contest. I'm an old Sun Solaris hand, NFS is second nature to me. I use NFS from my NAS on my home system (see sig)!

But the current kits I'm building for family and friends need to be compact and self sustaining. I need the HDD connected to the LE Pi for I/O perf on 4K HDR media files (it also removes the scanning pain you mention), as it only plays media. The traffic over the network share is less critical wrt to perfs, but it does have to work!

Let's see what that new build brings to the table before pulling out the protocol analyser! I'm trusting the team!

Quote from frakkin64

CIFS issue probably has nothing to do with firmware. I assume you are using Samba on "kodiplayer" and it's crashing from what is noted above.

This to me suggests the client "grabber"(?) is the one with the issues:

Code

smb2_sendfile_send_data: sendfile failed for file Transit/Incoming/The.Book.of.Boba.Fett.S01E06.2160p.WEB-DL.x265.10bit.HDR.DDP5.1.Atmos-BobarBiriyani[rartv]/The.Book.of.Boba.Fett.S01E06.Chapter.6.2160p.WEB-DL.DDP5.1.Atmos.HDR.HEVC-BobarBiriyani.mkv (Connection reset by peer) for client ipv4:10.25.25.1:44272

Connection reset by peer is initiated by the peer -- they send an RST packet. Seems like a network issue perhaps, because 7 minutes earlier CIFS is "reconnecting" -- what was the cause of that? Maybe on the client see if vers=3.0 works on the mount options?

True. kodiplayer is the server (LE11 nightly), grabber the client (DietPi).

I thought that the issue was with the client, like you, but I ruled it out by using my home NAS as the server. All went well.

I confirmed that it was a LE11 server issue when I fresh installed kodiplayer with Raspberry Pi OS. Same server hardware, different server software, same client hardware and software. Same network in all cases.

I also have a copycat production setup working seamlessly. The software is exactly the same, configured exactly the same way.

The only difference between the 2 kits is the newer hardware revision of the Pi 4 server.

It's all pretty much pointing at a problem on the server.

But I'm not one to ignore suggestions, I will totally test with vers=3.0 and report back!

Quote from HiassofT

FYI: new nightly with kernel/firmware update is now available https://test.libreelec.tv/LibreELEC-RPi4…203-d383c10.tar

so long,

Hias

oh, this is what I've been running all day long, as mentioned on post #18 (RE: RPi 4 2 GB losing USB - Firmware issue?). Can you hear my optimism getting shattered?

[deleted post content]

scratch that, 1D10T error... See next post!

OK, new fixup/start files from the next branch (https://github.com/raspberrypi/firmware/tree/next/boot)!

It boots!

With new fixup/start:

kodiplayer:~ # dmesg | grep firmware
[    0.110299] raspberrypi-firmware soc:firmware: Attached to firmware from 2022-01-25T13:48:40, variant start_x
[    0.113642] raspberrypi-firmware soc:firmware: Firmware hash is 94562b1518ca82ece28042cca1e5cdbbb43c8bda

So newer firmware than the original (2022-01-06T15:40:20), which is good.

But the remote CIFS client still drops. Oh well...

I'll wait for the full package, including kernel, inside the next nightly!

11.0-nightly-20220204, right?

Thanks again!

Nico

Quote from ghtester

I would say it's not an absolute nonsense - it might work but of course it's not guaranteed. All versions (bootloader, firmware and OS) should match together (to be at supported levels). You can test it or wait, I suppose that LE developers are going to upgrade the RPi 4B's firmware as well but it may take some time.

start.elf and fixup.dat, right?

Nico

Quote from ghtester

This is NOT a firmware, it's a bootloader, stored in RPi 4B's EEPROM. Until it's upgraded/downgraded automatically by OS or manually, it stays there even you boot from another microSD card or HDD.

The firmware files are stored on boot media's partition (.elf files etc.). So they are different with every boot media used.

I don't know which firmware version uses latest LE Nightly but you can compare them with files on RPi OS which works for you.

I learn every day!

OK, I'll look at the boot message, generate md5sums on both systems and report back.

Could I steal the elf files from RPi OS and use them on my LE11 nightly, or is this the stupidest thing you've heard today?