LibreElec Rpi4 TVHat TVHeadend - Recording Problems

    I think I found the cause of the issue (not enough RAM): :)

    Code
    Mar 31 14:31:58.021401 PiKodi kernel: Out of memory: Killed process 1694 (tvheadend) total-vm:1429592kB, anon-rss:1282680kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:2780kB oom_score_adj:0

    So create a swap file:

    Thread

    [How to] Enable swapfile on LibreELEC

    From @escalade in Remix

    By default LibreELEC does not have a swapfile.

    You can enable a swapfile by:

    cp /etc/swap.conf /storage/.config

    then edit it and change

    SWAP_ENABLED="no" to SWAP_ENABLED="yes".

    SWAPFILESIZE="128" to your required swapfile size - normally 4 x RAM but RAM should be enough for a normal LE system.

    When you reboot, a swapfile the size of SWAPFILESIZE is generated in /storage/.config/swapfile.
    Iridium
    April 14, 2019 at 6:09 PM

    Also check that drive:

    Code
    Feb 16 18:10:57.820749 PiKodi kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s
Feb 16 18:10:57.821280 PiKodi kernel: sd 0:0:0:0: [sda] tag#0 Sense Key : 0x3 [current] 
Feb 16 18:10:57.821583 PiKodi kernel: sd 0:0:0:0: [sda] tag#0 ASC=0x11 ASCQ=0x0 
Feb 16 18:10:57.821896 PiKodi kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 00 0a 00 00 00 08 00
Feb 16 18:10:57.822242 PiKodi kernel: critical medium error, dev sda, sector 2560 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2

    I do not have timeshift enabled on the pi4 - although I do have it set to ram only in the settings. I never found it to work properly on the pi4 so disabled it. Timeshift works on my athlon box (8GB Ram). I'll have a look at the other issues.

    Freeview HD in the UK uses Huffman encoding for the EPG information (which will display as garbage as above if not correctly decoded). TV Headend has support for this but you have to enable the Freeview UK EPG module in TV Headend for Freeview HD EPG data to be decoded correctly. Freeview SD uses standard EIT EPG standards and works fine without the Freeview UK EPG enabled (confusingly).

    Two reasons for files splitting I can think of :

    1. Reception errors (if you get too many the recording will stop, but I think it will then re-start as if it's missed the beginning?)

    2. Some broadcasters (C5 in the UK in particular) interrupt movies to broadcast a news bulletin - meaning you'll get two recordings not one. If you access the recordings folder outside of TV Headend (i.e. add it as a VIDEOS->FILES location in Kodi) you can play both parts - but TV Headend's UI and PVR Add-on may only show you one.

    -1 on the end of the filename is used to preserve both parts.

    I had also edited the config.txt to gpu_mem=96 at the same time I added the advancedsettings.xml.

    I used umount on /sda1 (my usb stick 32GB) and ran fsck - errors found and fixed, remounted and then rebooted the pi4.

    With the debug mode still on the top left of the screen displays activity.

    The MEM fluctuates for example 1595188/1871644, 1502444/1871644.

    I'm not sure if the 1871644 (KB) is supposed to represent the whole ram capacity (which should be 2GB --> 2097152 KB) or the usable capacity after some reserve. This difference seems to be about 220 MB.

    Should I really have a swap file? I thought it was a bad idea on both SDCards and usb sticks. I can also put in a larger usb stick than the current 32 GB if needed.

    Quote from boragthung

    Should I really have a swap file? I thought it was a bad idea on both SDCards and usb sticks. I can also put in a larger usb stick than the current 32 GB if needed.

    First repair / replace that broken drive, and analyze the outcome.

    If you're still unhappy afterwards, create a swap file, or buy an RPi with more RAM. Your choice. :)

    I recorded another programme just now (on Ch5HD 11:38 - 12:43) leaving my TV (downstairs) on - and using putty on my main PC (upstairs) ran htop to monitor. The Mem used increased as the recording went on, reaching around 1.9GB the last I looked at it.

    It seemed to be recording OK in the web gui but then at the time it was due to finish recording the gui info read:

    'There seems to be a problem with the live update feed from Tvheadend. Trying to reconnect...'

    The web page refreshed and had to login again. This exact thing also happens if I try to play a live stream using the TV icon in the epg guide - opening the details and using play does work with external player as does the play icons from config/channels and services page.

    Looking at the recorded file (only one) shows it only actually recorded for 32m45s, not the full 65 - ish minutes as listed in the completed recordings. I didn't notice any problem during the recording at this 32m point and it seemed to be recording right to the end. Here's the latest log. I can see another out of memory killing TVheadend, but this occurs a minute after the end point.

    https://paste.libreelec.tv/holy-stingray.log


    noggin ---> I only have OTA - Freeview set, all others are disabled. I realise the gobbledy gook txt is some encoding which I think only seems to occur on HD as it is clear on SD. Your point 2. I have discovered by my investigations into why things weren't working.

    Split films on CH5 among others - I initially found last year that setting a recording only recorded the first part - so then I used to set both parts to record and would end up with one file the full length and one with just the first part. I do most of my recordings on my other LibreElec (freesat) box.

    So on the pi I set the latest (2 part) recording choosing the first part only and it did record the whole thing (until the problem I am getting occurred losing the last few minutes). It is just in the client TV guide (and the webgui guide) it does not show the 'record icon' on the second part. But looking at the upcoming recordings I can see recording set correctly for the whole length. Maybe this was something that was fixed in the last year?

    Start by replacing the broken 32GB drive

    Code
    Apr 06 12:51:23.642517 PiKodi kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s
Apr 06 12:51:23.643238 PiKodi kernel: sd 0:0:0:0: [sda] tag#0 Sense Key : 0x3 [current] 
Apr 06 12:51:23.643529 PiKodi kernel: sd 0:0:0:0: [sda] tag#0 ASC=0x11 ASCQ=0x0 
Apr 06 12:51:23.643802 PiKodi kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 03 95 67 88 00 00 68 00
Apr 06 12:51:23.644082 PiKodi kernel: critical medium error, dev sda, sector 60123016 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 2

    I wouldn't be surprised if the dying drive causes file buffers to increase until the kernel kills tvheadend as the drive cannot keep up with the amount of data that tvheadend wants to write.

    so long,

    Hias

    Well I had run the fsck to fix the errors so looks like they have re-occurred. I was also testing the 2-part film recording - set one up in the pi web gui choosing the first part, no rec icon in the second part but the scheduled recording showed the correct start and end times for the 2 parts together. But only the first part recorded. Possible error as before.

    I was going to ask what would cause the memory usage to increase as the recording progresses, wondering if it might be a memory leak - but perhaps it is caused by this file buffers issue you mention because of the faulty flash drive. This was a brand new flash drive when I installed it.

    I might replace the usb stick with an external usb-caddied hard drive rather than another solid state. I have a few old HDD's lying around but they are not the AV type, just standard internal HDD. This pi box is only used to record occasionally, but I would still like to get to the bottom of the issue.

    I have set another 2 part recording on my freesat box in the same way to see if that completes both halves. This records fine and the only issues I have with that box is it sometimes completely freezes on playback of files, usually after a skip forward or backwards (through adverts),

    Quote from boragthung

    I have set another 2 part recording on my freesat box in the same way to see if that completes both halves. This records fine and the only issues I have with that box is it sometimes completely freezes on playback of files, usually after a skip forward or backwards (through adverts),

    The "out of memory error" is still present at your latest log:

    Code
    kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=tvheadend,pid=1193,uid=0
Apr 06 12:44:17.906747 PiKodi kernel: Out of memory: Killed process 1193 (tvheadend) total-vm:1353420kB, anon-rss:1200636kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:2620kB oom_score_adj:0

    Create a swap file now (double size of your RPi RAM). It can't hurt for a few test recordings.

    Quote from boragthung

    Well I had run the fsck to fix the errors so looks like they have re-occurred. I was also testing the 2-part film recording - set one up in the pi web gui choosing the first part, no rec icon in the second part but the scheduled recording showed the correct start and end times for the 2 parts together. But only the first part recorded. Possible error as before.

    I was going to ask what would cause the memory usage to increase as the recording progresses, wondering if it might be a memory leak - but perhaps it is caused by this file buffers issue you mention because of the faulty flash drive. This was a brand new flash drive when I installed it.

    I might replace the usb stick with an external usb-caddied hard drive rather than another solid state. I have a few old HDD's lying around but they are not the AV type, just standard internal HDD. This pi box is only used to record occasionally, but I would still like to get to the bottom of the issue.

    I have set another 2 part recording on my freesat box in the same way to see if that completes both halves. This records fine and the only issues I have with that box is it sometimes completely freezes on playback of files, usually after a skip forward or backwards (through adverts),


    I'd suggest starting with a new, clean, uSD card (uSD cards do fail) and a stock LibreElec install, and ensure you enable the Freeview UK EPG module so you get proper EPG details for the Freeview HD channes on the PSB3/BBCB mux. The Freeview HD EPG may also help with split recordings (or Series Recording rather than basic recording might). I think I usually just make sure I have set recordings for both parts.

    Personally I don't use anything smaller than 32GB these days - and usually use Samsung EVOs.

    Also - just checking you are using a Raspberry Pi-branded PSU or one you absolutely know is reliable for use with a Pi (the Pis take quite a lot of current and are quite particular about voltage and voltage drops across thin cables). It may not be relevant - but I've seen so many Pis do weird things with dodgy powers supplies and dying uSD cards it's something I always check.

    I've never used an AV-labelled drive for TV Headend recording purposes (they are really designed to be quieter as they sit under TVs in PVRs) and any reasonably decent HDD should be fine. I'd recommend externally powered 3.5" drives over bus-powered 2.5" drives though - as PSU issue can be a pain to sort out. Also - some of the latest large capacity external HDDs go to sleep automatically which can cause problems with TV Headend. (I had an issue recently with TV Headend running on an x86 box with OpenMediaVault caused by that)

    TV streams are very low bandwidth so even USB 2.0 connectivity is usually more than adequate for external HDDs - but USB 3.0 makes managing recordings a lot nicer (HD Video is usually <20Mbs - often <10Mbs after all)

    I'd also avoid using web playback as a way of checking recordings - it relies too much on browser playback support. I'd always check by playing the files in VLC using NFS, CIFS/SMB/SAMBA or SFTP to access the recordings on the Pi.

    Edited once, last by noggin (April 7, 2024 at 10:08 AM).

    I have set up an external (powered) HDD to replace the usb flash stick and the latest recording had no problems. I shall set up some more to see how it goes. I will also set up a swap file on that now. The Rpi power is the official one to be used (from PiHut) and I have the TVHat case and a flirc. I have always only used the Freeview epg module, disabling the others.

    As for AV hard drives - I have one in my main libreelec (freesat) box in the cabinet under the telly, which is an AM1 AMD Kabini APU and it works very well for simultaneous record and playback. The only problem with this APU is it does not play back DTS-MA, which I think is a well known shortcoming.

    I don't use web playback to check recordings - I only use it to check live streams and normally as 'passed through' to the external media player. It is only the TV icon button that does not work on the Rpi4 - this works OK on my Athlon box but only with the webtv-vp8-vorbis-webm stream profile and passthrough. Not that I really use that tv button. If I want to stream any channel from the web browser I would choose the play button from config/channels and it plays via external player (either VLC or MPC-BE).

    Thanks for your help everyone. I'll report back with any success or more problems.

    Quote from boragthung

    I have set up an external (powered) HDD to replace the usb flash stick and the latest recording had no problems. I shall set up some more to see how it goes. I will also set up a swap file on that now. The Rpi power is the official one to be used (from PiHut) and I have the TVHat case and a flirc. I have always only used the Freeview epg module, disabling the others.

    If you have the "Over the air : UK : Freeview" EPG enabled then you should get clear EPG data for the HD channels.

    I do get good epg data for all channels on Freeview. Freesat can get a bit mixed up for some reason, with wrong channel info given for some channels at times. Also strange is some channels that have +1 versions seem to yield better data on the +1 versions than the main one, like series and episode info.

    One thing I am not certain about on Freesat epg is the priorities. I have 3 enabled in order of preference --> OTA: UK Freesat (7), Open TV Sky UK (5) and OTA UK Freesat (EIT) (1) on my Athlon box. This means any information not gathered by Freesat will be filled in by the Sky, or possibly overwritten? I have mux schedulers set up to tune into the necessary transponder to get the 7 day scheduled epg.

    I have set a different order on the TVH docker on my unraid server --> Freesat(EIT) (6), Freesat (5) and Sky (2). Yes I have multiple satellite cards.

    The thinking here is that maybe the EIT Now/Next might update to cover programme overruns while recording. It doesn't seem to work like that though from what I have seen. I know there is epg running state in recording config, but I don't trust broadcasters to use that properly. This dates back to my experience with a VHS recorder (in the1990's) which had PDC (Programme delivery control). This was never implemented correctly and it often cut off the very beginning or end of a programme, which defeats the purpose of it.

    Anyway update on the recording problem. I threw the usb stick away as it seemed to be having internal problems. Instead there is now an externally powered usb caddy with an old 2TB Samsung HD2040UI in it. I set up the swap file to be on there but I had the problem of the swap service not starting and also samba not starting. So I disabled the swap file all together. I also noticed I had an error in the advancedsettings file so corrected that.

    I now have the recordings being made properly and also monitored one of them in action using htop on the secure shell - there didn't seem to be the constant increasing memory use, so it is pretty certain to me that the faulty usb stick was causing this issue as mentioned by HiassofT.

    I'm really glad to have got this sorted out now as I have been baffled by it for some time. Thanks for everyone's contribution.

    Quote from boragthung

    The thinking here is that maybe the EIT Now/Next might update to cover programme overruns while recording. It doesn't seem to work like that though from what I have seen. I know there is epg running state in recording config, but I don't trust broadcasters to use that properly. This dates back to my experience with a VHS recorder (in the1990's) which had PDC (Programme delivery control). This was never implemented correctly and it often cut off the very beginning or end of a programme, which defeats the purpose of it.

    The main BBC, ITV, C4 and C5 channels - all played out by Red Bee Media - do correctly update the present/following (aka EITpf 'Now/Next') data that drives accurate recording in TV Headend on Sky, Freesat and Freeview platforms. (Virgin's platform doesn't support it).

    For BBC channels the EITpf accurate recording triggers follow the same rules as the old VHS PDC used to, and is updated to start recording on the BBC ident before the show (so any content warnings are recorded), and to record the item in the junction after the show too (such as a pointer to the BBC's Action Line support service if the show has contained content that could trigger some people to seek support - such as domestic abuse, addiction portrayals etc.)

    It works pretty well in my experience - though for non-Red Bee Channels it's a bit more hit and miss - though most of them are clock-time linked anyway (Talking Pictures isn't)

    So you think it would be a good idea to enable the epg running state option? The current setting has epg update window set at 24 hours. Should this be left at that or altered?

    On my Freeview box I have OTA settings:

    45 4 * * *
    15 14 * * *
    45 18 * * *

    The internal grabber cron is:

    4 */12 * * *

    and I have epg saved to disc every hour.

    I always have a 7 day epg - the only problem is when things overun. A great example is when Wimbledon is on - this can cause delays, complete programme changes and even complete channel changes of programmes. I'm not sure how to get a 'dynamic' epg to cover these type of problems.


    You misunderstand me - I use the accurate recording option of 'Use EPG Running State' in the Recording Config - which uses the triggers for Rec start and stop in the EITpf DVB data.

    I have used TV Headend for 10+ years and never touched the cron settings - they're all on the defaults. The defaults have always downloaded the Freeview/Freeview HD and Freesat or Sky EPGs OK for me - though I've not got recordings regularly set for shows that are likely to be disrupted by sport (like the BBC Ten O'Clock News for instance)

    Wimbledon is going to be an issue as shows switch channel and the EPG EITpf stuff doesn't cope with that (Freeview+ and Sky+ also struggle with channel switches for the same reason) I think that's a known issue. AutoRecs might work if the EPG is updated but I don't think you can run an EPG update every minute to catch those very last minute EPG changes (as opposed to EITpf changes) - but Wimbledon is a known issue.

    However football delays and overruns like the Eurovision Song Contest are often handled OK from memory.