Schedules Direct for Tvheadend addon

1st Official Post

    Schedules Direct for Tvheadend (sd4tvh) addon is now ready for testing!!
    Thanks to astrilchuk and primaeval for their work taht I merged/morphed into this. The code is not optimized and needs further consolidation, but it has reached a point of needed some feedback from other users.

    The sd4tvh addon creates an xmltv grabber for tvheadend using the Schedules Direct JSON API. It also contains a kodi interface to manage your Schedules Direct lineups and configure a channel filter. The main purpose of the filter is to reduce download time and to fix a Schedules Direct timeout issue for those who use cable lineups (as these often start with 1000 channels).

    The channel filter uses a file to limit the number of channels downloaded from SD. This speeds up the time it takes to generate your xmltv.xml file and reduces the size of you sdcache.db file. There is an untested hdhomerun filter than can be used in place of the file based filter - if you have an hdhomerun setup. This should recognize the channels you have configured in the hdhomerun system and only download the necessary guide data. **As I do not have an hdhomerun - feedback is welcomed.

    Addon and source link is at the bottom of this post.

    To get started:

    • Download and install the sd4tvh addon
    • Run the addon and select Configure Settings and Options

      • Input your Schedules Direct username/password
      • Set the Number of Days to Download as desired (may want to set at 1 until you are sure everything is working)
      • If you have an HDHomerun setup

        • select Discover or IP in the HDHomerun Channel Filter

          • Discover - should find your device on the network
          • IP - if you know the hdhomerun ip address you can enter it here
        • Disable the File Based Channel Filter setting
      • If you do not have an HDHomerun

        • Leave the HDHomerun Channel Filter set to None
        • Leave the File Based Channel Filter enabled
      • Select OK
    • Select the Add Schedules Direct Provider Lineup (sd4tvh uses the JSON API for SD so the lineups on the SD website are not relevant)
      • Follow prompts to select country / zip code / lineup
    • You can review/remove your existing lineups by selecting Remove Schedules Direct Provider Lineup

      • List of subscribed lineups will display
      • Select lineup you wish to remove
    • Select Add & Remove Channels from Lineup ***(Not needed if using HDHomerun filter)*** and follow prompts

      • Select the lineup you wish to add/remove channels for
      • If there are new channels (all show here the first time) the New Channels Found prompt appears
        • Select which channels to include in downloads
      • The Excluded Channels List prompt shows channels that will not be downloaded (any not selected above will now be here)
        • Select which channels to include in downloads
      • The Included Channels List propmt shows channels that will be downloaded
        • Select which channels you do NOT want included in downloads
      • Channels will now be saved to a file called filter.cfg in the addon userdata folder - this text file can be manually edited if desired
    • Login to the tvheadend web interface to configure the sd4tvh grabber (I will include basic tvh setup info for those who need it)

      • Go to the CONFIGURATION > CHANNEL/EPG > EPG GRABBER MODULES tab
      • Select the Internal: XMLTV: tv_grab_sd4tvh is a simple wrapper for sd4tvh.py grabber (if missing - reboot)

        • click Enabled
        • click Save
        • click Re-run internal EPG Grabbers
        • tvheadend will now run the sd4tvh grabber and get a list of the channels (you can monitor progress in the bottom log window)
      • Go to the CONFIGURATION > CHANNEL/EPG > CHANNELS tab

        • For each channel, double click in the EPG source column and select the correct schedules direct channel feed
        • click Save
      • Return to the CONFIGURATION > CHANNEL/EPG > EPG GRABBER MODULES tab

        • click Re-run internal EPG Grabbers
        • tvheadend will now run the sd4tvh grabber and get a program list for the channels (you can monitor progress in the bottom log window)
        • When it finishes running, the Electronic Program Guide tab should be populated

    You can change your lineups and channels whenever you want, but to get the data to updated in tvheadend you need to rerun the internal grabber. The cron is set to update twice a day - you can adjust the cron time on the CONFIGURATION > CHANNEL/EPG > EPG GRABBER tab.

    I'm hoping to add some of the options from my zap2xml addon in the near future.
    Please provide any and all issues or feedback!! Thanks :)


    1/10/2017 - inital version 0.2.1

    Download the addon: Releases · edit4ever/script.module.sd4tvh · GitHub

    Source Code: GitHub - edit4ever/script.module.sd4tvh: Schedules Direct for Tvhheadend


    It's an issue with the ñ character -- likely it's an issue with all non-ascii 128 charcters. I had a similar issue with my zap2xml grabber...now I just have to go back and figure out what I did to make it work!

    edit4ever, I am struggling modifying the channel name. Example, my channel 2.4 shows WJBKDT4, I want to have it show "H&I". I can get it to change temporarily, but reboot my device it reverts back to the WJBKDT4. I have even edited the "Channels will now be saved to a file called filter.cfg in the addon userdata folder - this text file can be manually edited if desired", but still does not work.....

    Anyway, thanks for this fantastic add-on.

    Edit: I even tried to edit the xmltv.xml file, but that did not help either........ arggggggg!!!!!, actually, it is not a big deal, I JUST want it to show the channel names.......

    Edited once, last by clarkss12 (March 15, 2017 at 5:59 PM).

    This should be a setting in tvheadend. Check to make sure that you have unchecked "update channel name" on the CONFIGURATION > CHANNEL/EPG > EPG GRABBER tab. Then go to the CONFIGURATION > CHANNEL/EPG > CHANNELS tab and change the name of the channel there. You don't want to change the name in the filter.cfg as that always references the data that is on the downloaded guide. You just want to change what is displayed in the kodi interface - which is read from the tvheadend epg.

    Hope that helps - and thanks for the feedback!

    Quote from edit4ever


    This should be a setting in tvheadend. Check to make sure that you have unchecked "update channel name" on the CONFIGURATION > CHANNEL/EPG > EPG GRABBER tab. Then go to the CONFIGURATION > CHANNEL/EPG > CHANNELS tab and change the name of the channel there. You don't want to change the name in the filter.cfg as that always references the data that is on the downloaded guide. You just want to change what is displayed in the kodi interface - which is read from the tvheadend epg.

    Hope that helps - and thanks for the feedback!

    Yes!!!!!!!!!!!, that worked, I have been tinkering all morning with this stupid name thingy. Now another problem, when I change the name in the CHANNELS tab, I lose the EPG source, every time I change a channel name. I have to edit a second time to set the EPG source again.

    Quote from edit4ever


    It's an issue with the ñ character -- likely it's an issue with all non-ascii 128 charcters. I had a similar issue with my zap2xml grabber...now I just have to go back and figure out what I did to make it work!


    I tried entering the code below at the beginning of the schedulesdirect.py file to ensure it was using utf-8, but that didn't help. I just don't know enough about python to troublshoot it.
    [code=php]
    import sys
    reload(sys)
    sys.setdefaultencoding('utf-8')
    [/php]

    Edited once, last by spydah (March 16, 2017 at 6:27 PM).

    It has to do with escaped vs unescaped characters. I should have time in the next day or so to investigate. This version was pieced together from various work from other people so I just need to go back and see how the data is being processed. Since I'm not currently using Schedules Direct - when I think I have a fix I'll post the update and maybe you can test it. Thanks!

    spydah - let's start by using only the Dish network lineup.

    Please remove the other two lineups. Then delete the filter.cfg file from your addon_data/script.module.sd4tvh folder. Then you can rerun the sd4tvh addon and see if you can set the channels for the Dish lineup.

    Let me know if that works - and we'll go from there. Thanks!

    Deleted the other two lineups and removed the filter.cfg as you suggested. Manually running the sd4tvh.py still produces the UnicodeEncodeError when trying to list the channels in a lineup. Full error is below.


    [code=php]
    Traceback (most recent call last):
    File "sd4tvh.py", line 407, in <module>
    main()
    File "sd4tvh.py", line 399, in main
    app.manage()
    File "sd4tvh.py", line 68, in manage
    self._sd.manage()
    File "/storage/.kodi/addons/script.module.sd4tvh/libschedulesdirect/schedulesdirect.py", line 314, in manage
    self._list_lineup_channels()
    File "/storage/.kodi/addons/script.module.sd4tvh/libschedulesdirect/schedulesdirect.py", line 402, in _list_lineup_channels
    print(u"{0}\t{1.callsign} '{1.name}'".format(channel.channel, channel.station))
    UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in position 28: ordinal not in range(128)
    [/php]
    [hr]
    Did some more makeshift troubleshooting. The following code removed the error when listing channels, but the pi crashes when I tried to run it to generate the xmltv.xml file. I'm not sure if its a real fix or my pi has something else going on with it.


    [code=php]
    import codecs
    import sys
    UTF8Writer = codecs.getwriter('utf8')
    sys.stdout = UTF8Writer(sys.stdout)
    [/php]


    I got the tip from here

    Edited once, last by spydah (March 19, 2017 at 8:11 AM).

    I've come to the conclusion that the code I posted does handle the unicode error, but it seems my PI is running out of memory trying to store all of the JSON data before flushing it out to xml. Any suggestions on how to keep the memory usage down?

    Edited once, last by spydah (March 21, 2017 at 4:43 PM).

    One way would be to limit the number of days of data that is downloaded. The next way is to limit the number of channels being downloaded, using the channel filter part. First we should determine if you can get a base amount of data in. Why not start with a few channels on the dish lineup and see if it will download a full two weeks of data.

    - did you get a different error from a system log that you can share?

    Edited once, last by edit4ever (March 21, 2017 at 11:56 PM).

    It works if I use only the dish network lineup and limit it to a single day. If I add another lineup it crashes. When I get it back up the sd4tvh.log file is empty.

    I ran top as it was running the grabber and what I observed was the python process used 75% of the memory on my RP3 and 99% of the CPU right before it hung. I'm guessing the memory usage is the bigger issue of the two. I suspect the script caches all the JSON data to memory before parsing it to XML. So, even with channel filters it still has to grab the entire lineup data set.

    So, as a test I copied my .kodi directory off to another linux system with 4GB of ram and ran "python sd4tvh.py -u user -p password --filter --filter-path /storage/.kodi/userdata/addon_data/script.module.sd4tvh/filter.cfg" and it worked just fine.

    I've just added some swap space to my libreelec instance and am running it again. It's taking FOREVER. Will update when complete.

    First thanks for all your work, I have read through some of your threads and looks like you are paving the way.

    I'm pretty new to TVH only using it a few weeks now.

    I would like to try your module, but does TVH have to be installed on the same box as LibreElec?

    My plan was to run LE on several boxes and have a stand alone box for TVH (Right now using a laptop to test it, but will be moving to Rasp-Pi soon).

    I ultimately need to Filter channels from S.D. somehow because I only need certain ones in my guide and this looked like a good way to go about doing it.

    No problem running on different box. I run my tvheadend on a Pi...that system also has my SD addon. That Pi is running LE, so I could also use it as a front end if I want - but I don't. I let it run as the backend and then I have my phones, laptop and Fire TV running Kodi that connect to that. Running LE on the Pi made setup a lot easier.

    You could setup a Pi with Linux or Raspian install and manuall setup tvh and a different type of SD grabber...I just find the LE makes it all so simple!

    I was worried if I run LE & TVH on the same box it might chew up too much resources. I'm using Pi 3's, what is your opinion? I will have a couple more on the way in a few days so I can experiment with them.

    I have found another SD grabber for TVH that would allow multiple lineups and filter of channels, do you know of any?
    Right now I pull in an xmltv.xml with file grabber but it always has some stuff missing. So I wanted to try SD, but need multiple lineups to complete my guide.

    Even if I go another route, I'm still willing to help with testing your app along with the other members. Early retired Comp-Sci guy here...I never used python before, but once you know one the rest are easy to pick up.