tvheadend, ATSC and epg...

    Quote from stepher


    Couldn't get it working using the 0.6.1 update. However, had the chance to disable program icon download and that brought things back to "normal". Things are good again :)

    The "fix" should have been adding str() to the icon fn line 349. I think I was rushing and did it wrong. Try changing the icon section starting at line 348 in zap2xml.py to the following:

    Code
    if "-I" in options:
                    fn = os.path.join(cacheDir,"I" + str(cp) + ".js.gz")
                    if not os.path.isfile(fn):
                        data = getURL(urlRoot + "gridDetailService?rtype=pgmimg&pgmId=" + cp)
                        if data: #sometimes we fail to get the url try to keep going
                            wbf(fn, data)
                            log.pout("[i] Parsing: " + cp,'info')
                    parseJSONI(fn)[/i]


    If that doesn't work - I'd love to see the log error. Thanks!

    Edited once, last by edit4ever (January 10, 2017 at 4:08 PM).

    As near as I can tell, the code changes you provided works. I updated the zap2xml.py file (commenting out the "bad" lines), (re)enabled downloading of program icons and restarted tvheadend. Waited for the results to happen and it checked out great. Thx.

    Iirc the updated code from the zip file had a line (348, I believe) of "cp = str(cp)" and that did not seem to work. The new code seems to work (so far), but I should have a better idea about that tomorrow morning (channel info updates happen at 3:30AM).

    Appreciate the work and effort to help....cheers....

    Quote from edit4ever

    The "fix" should have been adding str() to the icon fn line 349. I think I was rushing and did it wrong. Try changing the icon section starting at line 348 in zap2xml.py to the following:

    Code
    if "-I" in options:
                    fn = os.path.join(cacheDir,"I" + str(cp) + ".js.gz")
                    if not os.path.isfile(fn):
                        data = getURL(urlRoot + "gridDetailService?rtype=pgmimg&pgmId=" + cp)
                        if data: #sometimes we fail to get the url try to keep going
                            wbf(fn, data)
                            log.pout("[i] Parsing: " + cp,'info')
                    parseJSONI(fn)[/i]


    If that doesn't work - I'd love to see the log error. Thanks!

    Just confirming that my tvguide.xml file was updated last night at it's scheduled time (actually, early this morning) and it has the programming schedule data for 14 days out. I'm good!!

    Thx and cheers....

    Quote from stepher


    As near as I can tell, the code changes you provided works. I updated the zap2xml.py file (commenting out the "bad" lines), (re)enabled downloading of program icons and restarted tvheadend. Waited for the results to happen and it checked out great. Thx.

    Iirc the updated code from the zip file had a line (348, I believe) of "cp = str(cp)" and that did not seem to work. The new code seems to work (so far), but I should have a better idea about that tomorrow morning (channel info updates happen at 3:30AM).

    Appreciate the work and effort to help....cheers....

    Hi!

    Since a few days I'm fiddling around with Zap2XML and everything around it. I red this thread, but I have not found my answer yet.
    I just try to understand how this all works together.

    In TVH web interface, when I click on "Re-run internal EPG grabbers". Looks like nothing happens. Via SSH I check the zap2xml log file, or the tvxml.xml file. And they don't get updated.

    If I manually run [...]/tv_grab_zap2xml then I can see the log file and xml file change. That works fine.

    But on Kodi, I don't see results quickly in the TV Guide, I don't even know WHEN the changes will appears. Looks like random, but sure it's not.

    Can I do manual stuff ? Or only rely on some more official ways like the "Re-run" button ?

    Thanks,
    Éric
    LibreELEC 7.9.010 :: TVHeadEnd 4.2 :: Zap2XML 0.60

    In tvh have you enabled the zap2xml grabber and saved that setting? Have you setup your zap2it or tvguide username/password in the zap2xml add on settings? Have you setup favorite channels on the zap2it/tvguide website? In the tvh web interface have you connected your channels to the source epg zap2xml channels?

    All these need to be done to have everything run and be automated. Based on the fact that you can run it manually - and I assume that generated the correct xmltv.xml file - the first or last item above is likely your issue.

    Quote from edit4ever


    In tvh have you enabled the zap2xml grabber and saved that setting? Have you setup your zap2it or tvguide username/password in the zap2xml add on settings? Have you setup favorite channels on the zap2it/tvguide website? In the tvh web interface have you connected your channels to the source epg zap2xml channels?

    All these need to be done to have everything run and be automated. Based on the fact that you can run it manually - and I assume that generated the correct xmltv.xml file - the first or last item above is likely your issue.

    Hi, I think I did all that...

    • In TVH I enabled zap2xml grabber and disabled all the other one.


      

    • I use Zap2It with my usernamer/password.


      

    • I setup my favorite channels on the website.


      

    • I also manually connected my channels to the source EPG zap2xml channels.



    Maybe you can't help me. But what is the best way for me to kinda trace/debug my problems ?

    • How can I see if TVH really trigger the zap2xml (from what I see, that doesn't happens) ?


      

    • And once the xmltv.xml file is generated (I can see it when I launch zap2xml manually) when does TVH pick it up ?


      

    • Finally, when/if TVH parse the xmltv.xml file, when will I see it in Kodi's TV Guide ?



    Probably too much questions... Sorry... I don't know exactly with which one I should start... Probably with the trigger of zap2xml from TVH.

    Thanks again! :)

    Quote


    How can I see if TVH really trigger the zap2xml (from what I see, that doesn't happens) ?

    There is a tvh log called service.log in the tvh addon userdata folder. ((you can also watch this log when you rerun the internal grabber in the bottom log window of the web interface - click the double up arrows in the bottom right corner of the screen) In that log you should see something like:

    2017-01-10 15:54:41.000 [ INFO] xmltv: /storage/.kodi/addons/script.module.zap2xml/bin/tv_grab_zap2xml: grab /storage/.kodi/addons/script.module.zap2xml/bin/tv_grab_zap2xml
    2017-01-10 15:54:41.007 [ INFO] spawn: Executing "/storage/.kodi/addons/script.module.zap2xml/bin/tv_grab_zap2xml"
    2017-01-10 15:56:32.363 [ INFO] xmltv: /storage/.kodi/addons/script.module.zap2xml/bin/tv_grab_zap2xml: grab took 111 seconds
    2017-01-10 15:56:34.445 [ INFO] xmltv: /storage/.kodi/addons/script.module.zap2xml/bin/tv_grab_zap2xml: parse took 2 seconds
    2017-01-10 15:56:34.446 [ INFO] xmltv: /storage/.kodi/addons/script.module.zap2xml/bin/tv_grab_zap2xml: channels tot= 25 new= 0 mod= 0
    2017-01-10 15:56:34.446 [ INFO] xmltv: /storage/.kodi/addons/script.module.zap2xml/bin/tv_grab_zap2xml: brands tot= 0 new= 0 mod= 0
    2017-01-10 15:56:34.446 [ INFO] xmltv: /storage/.kodi/addons/script.module.zap2xml/bin/tv_grab_zap2xml: seasons tot= 5545 new= 5545 mod= 5545
    2017-01-10 15:56:34.446 [ INFO] xmltv: /storage/.kodi/addons/script.module.zap2xml/bin/tv_grab_zap2xml: episodes tot= 5778 new= 4001 mod= 4006


    Quote


    And once the xmltv.xml file is generated (I can see it when I launch zap2xml manually) when does TVH pick it up ?

    If all of the above is seen in the tvh log - the zap2xml grabber ran, made the xmltv.xml file and tvh read that file in. There should now be data in the epg of the tvh web interface. If it did not complete there should be a zap2xml.log file in the script.module.zap2xml userdata folder. Check it and post the last items - which should be an error of some kind.


    Quote


    Finally, when/if TVH parse the xmltv.xml file, when will I see it in Kodi's TV Guide ?

    This depends on how you have configured the pvr settings in kodi. Normally they only update every few hours - so to test, if you see data in the eog of the tvh web interface = just reboot kodi and it should load in.


    Let me know what you find when you check the logs.

    Edited once, last by edit4ever (January 16, 2017 at 4:03 PM).

    First of all, a big thank you. Your help and your patience are greatly appreciated.

    Now, my bad. Found my problem, in part with all the great tips you gave me.
    I added argument in the "Extra arguments" in the "EPG Grabber Module" tab in TVH for zap2xml.
    When I removed them, it all worked fine.

    I try to add arguments because I was looking for long descriptions in the EPG. But now it's fine. I think I setup that in the zap2xml addon settings.

    In fact there's one channel that doesn't "want" to get in the EPG. And I don't know why. For now it's not a big problem since I don't watch it very much.
    But that's strange. In the xmltv.xml file, I can see the channel listed there, and all the associated programmes.
    In TVH I linked the EPG channel with the channel. But still, I can't see it in TVH's "Electronic Program Guide" tab. (and not on Kodi, of course).

    Don't have much time today to "play" with it. Will check it more deeply tomorrow.

    And thanks again. I'm sure I'll have more questions soon! :D

    Glad you got it!

    You can setup all the detail information you want in the zap2xml addon configure settings under the Extra Details Order tab.

    Just put what ever you want and whatever separators you want and the show description will be updated to conatin all that info.

    Again - once you make that change - you will need to rerun the internal grabber and if you want kodi to update it's epg to match. Reboot again.

    Double check the epg source closely in the tvh web interface. There may be two channels with similar names/numbers.

    enjoy!

    Ooops...may have spoken too soon. Things seemed to be fine for a few days (update out to 14 days) and then the last update was 2 days ago ( Jan 14, 2017). Did a tvheadend restart (still no update) and here's the resulting log file output:

    Have done no (obvious) updates so no idea what might have changed to cause this. Looks like another type mismatch. Hold that revision update :(

    Thx and cheers....

    Quote from edit4ever


    Awesome! Thanks for the update...I'll put out a revised file with the fixed code.

    Hi!

    I found the "problem" for the weird channel behaviour. The channel is WPTZ 5.1 (and 5.2 and 5.3) (if that helps somehow) .

    Strangely, when I look inside the xmltv.xml file, all programmes starts 10 days later. The 1st programme is due on the 26th of January. If I check on zap2it website, all the informations are there for today (the 16th of January).

    IDK if you can help. The problem is probably in zap2it.py script. That's probably over my own knowledge.

    Quote from stepher


    File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 352, in on_td
    data = getURL(urlRoot + "gridDetailService?rtype=pgmimg&pgmId=" + cp)
    TypeError: cannot concatenate 'str' and 'int' objects

    There must be some program numbers that have a character that is messing with the putting together of the string.

    My guess is if we force that whoel section to use str(cp) it will fix it... try this starting at 348:

    if "-I" in options:
    fn = os.path.join(cacheDir,"I" + str(cp) + ".js.gz")
    if not os.path.isfile(fn):
    data = getURL(urlRoot + "gridDetailService?rtype=pgmimg&pgmId=" + str(cp))
    if data: #sometimes we fail to get the url try to keep going
    wbf(fn, data)
    log.pout(" Parsing: " + str(cp),'info')
    parseJSONI(fn)
    [hr]

    Quote from esenterre


    Strangely, when I look inside the xmltv.xml file, all programmes starts 10 days later. The 1st programme is due on the 26th of January. If I check on zap2it website, all the informations are there for today (the 16th of January).


    That's really strange. The data on the zap2it site looks correct. The downloads happen as a complete station group in 6 hour blocks...so it shouldn't happen.

    You could try deleting the xmltv.xml file and in the .kodi/addons/script.module.zap2xml/cache folder try deleting all of the files that end with .html.gz - these are the bulk guide data downloads. There shouldn't be that many (depending on how many days you set to download - each file is a 6 hour block) All the other files are the pictures and extra details if you've set those to download.

    If you're trying to navigate these folders from a windows computer (instead of from ssh) you will need to add the .kodi folder to your samba.conf file. Open that file (or if you haven't made one yet - rename the samba.conf.sample file to samba.conf and open it) and add the following ta the bottom:

    [kodi]
    path = /storage/.kodi
    available = yes
    browsable = yes
    public = yes
    writable = yes
    root preexec = mkdir -p /storage/.kodi

    This will allow you to browse the .kodi folder from windows and easily get to the addons folder.

    If you delete the cache files / reboot and let zap2xml run again I'm hoping it clears up. Otherwise - I'm not sure why that one channel would have strange data.

    Edited once, last by edit4ever (January 16, 2017 at 11:13 PM).

    OK. Back up and running (again). Thx. However.....been there, done that :) I'll wait to confirm if it's really fixed this time.

    One other thing....in the section of code above the one that you suggested to be modified (starts on line 342 for me "if cp != -1 and "-D" in options:") there are a few references to the variable "cp". I assume typing is not an issue in this section (I know, if it ain't broke, don't fix it).

    Appreciate the help in getting back on track. Of course I'll be back if something else pops up :)

    Cheers....


    Quote from edit4ever


    [quote='stepher','https://207.154.221.202/thread/?postID=28851#post28851']
    File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 352, in on_td
    data = getURL(urlRoot + "gridDetailService?rtype=pgmimg&pgmId=" + cp)
    TypeError: cannot concatenate 'str' and 'int' objects

    There must be some program numbers that have a character that is messing with the putting together of the string.

    My guess is if we force that whoel section to use str(cp) it will fix it... try this starting at 348:

    if "-I" in options:
    fn = os.path.join(cacheDir,"I" + str(cp) + ".js.gz")
    if not os.path.isfile(fn):
    data = getURL(urlRoot + "gridDetailService?rtype=pgmimg&pgmId=" + str(cp))
    if data: #sometimes we fail to get the url try to keep going
    wbf(fn, data)
    log.pout(" Parsing: " + str(cp),'info')
    parseJSONI(fn)
    [hr]

    Edited once, last by stepher (January 17, 2017 at 5:15 PM).

    Awesome it worked! Thanks! Another tip in my sleeve.

    Just a quick update to let you know things are still working fine.Thx. When it gets to a week+, then I'll feel more comfortable.

    What's a bit baffling for me is how the 1st code change a couple of weeks ago worked for a bit and then failed. That seems a bit odd.

    Anyway, we're good now and I'll let you know when I hit the week milestone :)

    Cheers and thx again....

    Wish I had more good news...but alas :(

    It looked like I had some subnet issue last night, but wasn't clear if that was related to the problem. So I restarted tvheadend just to be sure. Results are pretty much the same.

    Here's the output from the zap2xml.log file:

    ....snip.....
    [53/56] Parsing: /home/osmc/.kodi/addons/script.module.zap2xml/cache/1487080800000.html.gz
    [54/56] Parsing: /home/osmc/.kodi/addons/script.module.zap2xml/cache/1487102400000.html.gz
    Exception
    error<type 'exceptions.ValueError'>
    Traceback (most recent call last):
    File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2132, in <module>
    main()
    File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2042, in main
    parseGrid(fn) #data read in and unicode it
    File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 1094, in parseGrid
    p.feed(b)
    File "/usr/lib/python2.7/HTMLParser.py", line 117, in feed
    self.goahead(0)
    File "/usr/lib/python2.7/HTMLParser.py", line 161, in goahead
    k = self.parse_starttag(i)
    File "/usr/lib/python2.7/HTMLParser.py", line 327, in parse_starttag
    self.handle_starttag(tag, attrs)
    File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 716, in handle_starttag
    globals()['on_%s' % tag](self, tag, attrs)
    File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 347, in on_td
    parseJSOND(fn)
    File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 863, in parseJSOND
    t = json.loads(b)
    File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
    File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
    ValueError: No JSON object could be decoded


    Quote from edit4ever


    awesome - thanks for the update. If all is well after you get through a full week+ - I'll update the addon with the fix.