tvheadend, ATSC and epg...

stepher · February 10, 2017 at 1:53 AM

I did try to look at them using WinZip (from a Win10 sys) and, while I could open a file, it showed as empty. Is there another Win10 based program that allows me to view the files? Or should I be using something on the Debian/OSMC system to do that (like, would nano work)?

Quote from edit4ever

No - the files in the cache folder are just the downloads. They always have their data...They just need to be uncompressed first.

**edit4ever** · February 10, 2017 at 2:24 AM

I use expresszip - you should have the a file with the same name and just the .js extension.

That file can be opened by any text editor - although I use atom as it is built for looking at code.

cabbresson · February 11, 2017 at 11:39 PM

Quote from stepher

What nips at me is not the fact that it stopped working after I upgraded to OSMC/Kodi 17 (maybe it should but with permissions, etal, it's not a surprise) but rather that it started working again this morning, and I didn't do anything (tho, maybe I changed 1 of the options and just don't remember

Mine has also stopped working after upgrading from 7.0.3 to libreelec 7.95.3. I restarted my rpi2 and tvheadend downloaded the guide just fine after that. Dont know what happened either.

stepher · February 11, 2017 at 11:52 PM

I wonder if some of the problem is an occasional "glitch" when downloading the website the program guide data.

My plan (when I get the time) is to experiment with different settings (xtra program details, program/station icon download, etc.) and make sure the problem *isn't/is* (always) on my end.

Quote from cabbresson

Mine has also stopped working after upgrading from 7.0.3 to libreelec 7.95.3. I restarted my rpi2 and tvheadend downloaded the guide just fine after that. Dont know what happened either.

**esenterre** · February 13, 2017 at 3:03 PM

I also have problems with my zap2xml I don't know if it's since I upgraded to 7.95.3 or not.

- I emptied cache folder. But that didn't helped.
- I disabled the extra information and that worked. But now I'm missing these extra informations :-\

I remember seeing "error 500" in the log file when it wasn't working. So it looks like the addon wasn't able to retrieve some content from the web. But shouldn't it manage that ?

Tell me if I can help in some ways too ?

Éric.

zap2xml 0.6.0
LibreElec 7.95.3

**edit4ever** · February 13, 2017 at 3:13 PM

ok - let's try something.

If you can edit the zap2xml.py file - look at the following section starting at line 341:

Code

if cp != -1 and "-D" in options:
                    fn = os.path.join(cacheDir,cp + ".js.gz")
                    if not os.path.isfile(fn):
                        data = getURL(urlRoot + "gridDetailService?pgmId=" + cp)
                        wbf(fn, data)
                        log.pout("[D] Parsing: " + cp,'info')
                    parseJSOND(fn)
                if "-I" in options:
                    fn = os.path.join(cacheDir,"I" + cp + ".js.gz")
                    if not os.path.isfile(fn):
                        data = getURL(urlRoot + "gridDetailService?rtype=pgmimg&pgmId=" + cp)
                        if data: #sometimes we fail to get the url try to keep going
                            wbf(fn, data)
                            log.pout("[I] Parsing: " + str(cp),'info')
                    parseJSONI(fn)

Display More

Change it to the following:

Code

if cp != -1 and "-D" in options:
                    fn = os.path.join(cacheDir,cp + ".js.gz")
                    if not os.path.isfile(fn):
                        data = getURL(urlRoot + "gridDetailService?pgmId=" + cp)
                        wbf(fn, data)
                        log.pout("[D] Parsing: " + cp,'info')
                    if os.path.isfile(fn):
                        parseJSOND(fn)
                if "-I" in options:
                    fn = os.path.join(cacheDir,"I" + cp + ".js.gz")
                    if not os.path.isfile(fn):
                        data = getURL(urlRoot + "gridDetailService?rtype=pgmimg&pgmId=" + cp)
                        if data: #sometimes we fail to get the url try to keep going
                            wbf(fn, data)
                            log.pout("[I] Parsing: " + str(cp),'info')
                    if not os.path.isfile(fn):
                        parseJSONI(fn)

Display More

I'm thinking if we only parse if the file exists - that may solve the error. The change is for both the program details and the program icon downloads.

Let me know if you are able to test.

stepher · February 15, 2017 at 6:01 PM

2/15/17 - Quick update

My tvguide.xml file continues to be updated as programmed. I have not had the chance to experiment (hopefully in the next few days) so I'm not sure at the moment of what's enabled/disabled in the add-on.

Let you know when I have a chance to look at things again. I may also incorporate your recently suggested changes to the zap2xml.py file and see what happens.

Cheers.....

Quote from edit4ever

ok - let's try something.

If you can edit the zap2xml.py file - look at the following section starting at line 341:

Code

if cp != -1 and "-D" in options:
                    fn = os.path.join(cacheDir,cp + ".js.gz")
                    if not os.path.isfile(fn):
                        data = getURL(urlRoot + "gridDetailService?pgmId=" + cp)
                        wbf(fn, data)
                        log.pout("[D] Parsing: " + cp,'info')
                    parseJSOND(fn)
                if "-I" in options:
                    fn = os.path.join(cacheDir,"I" + cp + ".js.gz")
                    if not os.path.isfile(fn):
                        data = getURL(urlRoot + "gridDetailService?rtype=pgmimg&pgmId=" + cp)
                        if data: #sometimes we fail to get the url try to keep going
                            wbf(fn, data)
                            log.pout("[I] Parsing: " + str(cp),'info')
                    parseJSONI(fn)

Display More

Change it to the following:

Code

if cp != -1 and "-D" in options:
                    fn = os.path.join(cacheDir,cp + ".js.gz")
                    if not os.path.isfile(fn):
                        data = getURL(urlRoot + "gridDetailService?pgmId=" + cp)
                        wbf(fn, data)
                        log.pout("[D] Parsing: " + cp,'info')
                    if os.path.isfile(fn):
                        parseJSOND(fn)
                if "-I" in options:
                    fn = os.path.join(cacheDir,"I" + cp + ".js.gz")
                    if not os.path.isfile(fn):
                        data = getURL(urlRoot + "gridDetailService?rtype=pgmimg&pgmId=" + cp)
                        if data: #sometimes we fail to get the url try to keep going
                            wbf(fn, data)
                            log.pout("[I] Parsing: " + str(cp),'info')
                    if not os.path.isfile(fn):
                        parseJSONI(fn)

Display More

I'm thinking if we only parse if the file exists - that may solve the error. The change is for both the program details and the program icon downloads.

Let me know if you are able to test.

stepher · February 18, 2017 at 6:23 PM

2/21/17 Quick Update:

Had a chance to check my tvguide.xml file and looks like it updated normally early this morning. So, disabling the 2 options I mention in yesterday's update seems to solve my problem.

2/20/17 (later) Update:

Had a chance to disable xtra program details (and Download Program Icons option at the same time) and restarted TVH. Back to working just fine.

Sorry to say your recommended code changes to the zap2xml.py file did not help *me* at all.

2/20/17 Update:

I enabled "Xtra Program Details" yesterday and this morning found that my tvguide.xml file was NOT updated. I'm going to disable this option and see what happens.

2/18/17 Update:

Noticed this morning that my tvguide.xml file was last updated on Thurs., 2/16/17. I had not done much of anything to the system (except watch some movies/tv shows and deleted them after I watched them).

However, a couple of times I had to do a hard reset when the system locked up trying to fast forward on a program. I noticed when that happened, the settings I had made (i.e. xtra program details, etc.) were set back to default after the reset(s). This may have been a source of the issue.

Before doing anything else, I decided to update my zap2xml.py file with your new Python code suggestions below. I then restarted TVH and my tvguide.xml file was updated. I did this thru ssh and did not make any changes to the RPI 3/HTPC system.

So, back to tvguide.xml update watch

BTW - I believe the cause for the lock ups may be due to recently developed power supply issues (I occasionally get the "lightning bolt" flashing in the upper right corner of the screen I'll have to install a better power supply.

Plan to test the different options with the new Python code. Will keep you posted on how things are going. Thx for your continued support.

Cheers....

Quote from stepher

2/15/17 - Quick update

My tvguide.xml file continues to be updated as programmed. I have not had the chance to experiment (hopefully in the next few days) so I'm not sure at the moment of what's enabled/disabled in the add-on.

Let you know when I have a chance to look at things again. I may also incorporate your recently suggested changes to the zap2xml.py file and see what happens.

Cheers.....

**edit4ever** · February 23, 2017 at 1:23 AM

Looks like my system has now succumb to the extra details json error. The earlier code change did not help. I'm not sure if this is an issue with an LE update or a change to the way screener/zap2it is hosting the data. I'll have to do some testing - but may not get to this for a bit. At least we know it's not limited to a single system or user issue.

In the meantime, you can turn off the extra details and it will load the basic guide.

stepher · February 23, 2017 at 2:53 AM

I'm glad you've been able to reproduce the problem. Not glad you have the same issue but nice to know it's not unique to my system

I thought I noticed that when I didn't do anything, the system went thru a non-update hiccup, so I wonder if it's at the website that it's some kind of protocol glitch that shows up every once-in-a-while when it comes to the xtra program details. My emphasis is on "I didn't do anything" which doesn't mean the RPI 3/OSMC didn't do something when I wasn't looking.

Something else I've noticed (and this is not really germane to our discussion here) is that when my system locks up and I have to do a hard reset (need to code and install a reset button), some of my settings (I'm using the new Confluence skin) are reset to default (like weather on the home screen, hiding "Pictures" on the home screen, etc.). Need to experiment with that once I've stabilized my RPI 3 P/S.

I can be patient and I appreciate whatever time and effort you are putting into this. Thx!! And I'm happy to continue as your "beta site"

Cheers....

Quote from edit4ever

Looks like my system has now succumb to the extra details json error. The earlier code change did not help. I'm not sure if this is an issue with an LE update or a change to the way screener/zap2it is hosting the data. I'll have to do some testing - but may not get to this for a bit. At least we know it's not limited to a single system or user issue.

In the meantime, you can turn off the extra details and it will load the basic guide.

**edit4ever** · February 24, 2017 at 3:36 PM

Well - my system ran correctly again last night with extra details...so it looks like I'm going to have to wait for another error to appear so i can troubleshoot. I want to narrow down whether the issue is with missing data or changed data. BUt until I have another error - I don't have a place to look.

If yours fails again - please provide the log so I can look at the point of failure.

Thanks!

default_user8 · February 24, 2017 at 4:14 PM

#First thanks for your work, but i'm having some issues. First thing i ran into was the script was calling for xmltv.xml instead of settings.xml(i'm running latest libreelec build and TVH 4.2 on an old laptop). #But that was an easy fix, the problem i'm having now is that my EPG source isn't showing up when i try to assign it under the channels tab. Anyone else having this issue?

Nevermind, my username was wrong, now it's working like it's supposed to.

stepher · February 24, 2017 at 4:58 PM

My did, as well (and has been running correctly). I'll have to set the xtra program details and download program icons options (separately, of course) and see what the results are. I'll post the log when I get the (bad) results.

Thx and cheers.....

Quote from edit4ever

Well - my system ran correctly again last night with extra details...so it looks like I'm going to have to wait for another error to appear so i can troubleshoot. I want to narrow down whether the issue is with missing data or changed data. BUt until I have another error - I don't have a place to look.

If yours fails again - please provide the log so I can look at the point of failure.

Thanks!

cabbresson · February 24, 2017 at 8:31 PM

Here is my bad log file hope it helps

[JSON] bad zap2xml grab - Pastebin.com

**edit4ever** · February 24, 2017 at 9:53 PM

Quote from cabbresson

Here is my bad log file hope it helps

[JSON] bad zap2xml grab - Pastebin.com

OK - it failed on the /storage/.kodi/addons/script.module.zap2xml/cache/1487419200000.html.gz file. That file does exists on the zap2it site - so it should have downloaded. Try to open that file on your system and see if it contains data. It should look like this:

Code

zctv.gd.schedule['SH013074920000'] = {"program": {
  "description": "Inside the lives of America's best and brightest youth.",
  "genres": [
    "Children"
  ],
  "seriesId": "EP01307492",
  "originalAirDate": "1285416000000",
  "episodeUrl": "http://tvschedule.zap2it.com/tv/the-young-icons/EP013074920000?aid=tvschedule",
  "schedule": 0,
  "episodeTimesUrl": "http://tvschedule.zap2it.com/tv/the-young-icons/EP01307492?aid=tvschedule",
  "type": "se",
  "flagship": false,
  "duration": 0,
  "isMyFave": false,
  "canMyFave": true
}}

Display More

If the file doesn't exist - or contains no information. Then we at least have a starting point. Thanks!
[hr]
OK - maybe if we don't write the details file if it contains no data it will keep the error from happening.

Let's try changing the following: (depending on which version you're running, this should start at line 341 or line 332)

Code

if cp != -1 and "-D" in options:
                    fn = os.path.join(cacheDir,cp + ".js.gz")
                    if not os.path.isfile(fn):
                        data = getURL(urlRoot + "gridDetailService?pgmId=" + cp)
                        wbf(fn, data)
                        log.pout("[D] Parsing: " + cp,'info')
                    if os.path.isfile(fn):
                        parseJSOND(fn)

To this:

Code

if cp != -1 and "-D" in options:
                    fn = os.path.join(cacheDir,cp + ".js.gz")
                    if not os.path.isfile(fn):
                        data = getURL(urlRoot + "gridDetailService?pgmId=" + cp)
                        if data:
                            wbf(fn, data)
                            log.pout("[D] Parsing: " + cp,'info')
                    if os.path.isfile(fn):
                        parseJSOND(fn)

If you're having an error with your current grab - change the code and try running the grab again. I've changed mine and I'm still waiting to see if I get the failure again.

cabbresson · February 25, 2017 at 3:08 AM

I don't have that file on my system, but the log I upload is a few days old. I'll turn the extra details back on and get a new log file.
[hr]
I have a new log.
[JSON] bad zap2xml grab 2 - Pastebin.com
[hr]
I changed line 341 like you said and I'm getting an error right after rerunning the grab manually.

log [JSON] bad zap2xml grab line 341 changed - Pastebin.com

**edit4ever** · February 25, 2017 at 2:04 PM

Interesting - let's make sure your code is the same as mine. Rename your zap2xml.py file to zap2xml.bak and download this version.

zap2xml.zip

Extract the zap2xml.py file and put it in place of the one you renamed.

Now rerun the grabber. You shold still get the error - but I would like to see that log for comparison.

Thanks!

stepher · February 25, 2017 at 4:26 PM

Re-enabled xtra program details last night and let the system go thru normal updating (@3:30 AM). Bad news...no update....good news....no update and here's the section of the log file (under .kodi/userdata/addon_data/script.module.zap2xml) where it cast the error. I also noticed in the 1st part of the log file the code now deletes the old cached files ("Deleting old cached file: ....."). Is that new or something I just haven't noticed until now?

Thx and cheers....

Code

...snip....
Getting: http://tvschedule.zap2it.com/tvlistings/gridDetailService?pgmId=EP024717260009
[D] Parsing: EP024717260009
Getting: http://tvschedule.zap2it.com/tvlistings/gridDetailService?pgmId=EP024717260011
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Getting: http://tvschedule.zap2it.com/tvlistings/gridDetailService?pgmId=EP024717260011
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Getting: http://tvschedule.zap2it.com/tvlistings/gridDetailService?pgmId=EP024717260011
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Failed to download within 3 retries.
:Function: getURL :Line: 228
error<class 'mechanize._response.httperror_seek_wrapper'>
Exception
error<type 'exceptions.AttributeError'>
Traceback (most recent call last):
  File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2131, in <module>
    main()
  File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2041, in main
    parseGrid(fn) #data read in and unicode it
  File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 1093, in parseGrid
    p.feed(b)
  File "/usr/lib/python2.7/HTMLParser.py", line 117, in feed
    self.goahead(0)
  File "/usr/lib/python2.7/HTMLParser.py", line 161, in goahead
    k = self.parse_starttag(i)
  File "/usr/lib/python2.7/HTMLParser.py", line 327, in parse_starttag
    self.handle_starttag(tag, attrs)
  File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 715, in handle_starttag
    globals()['on_%s' % tag](self, tag, attrs)
  File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 345, in on_td
    wbf(fn, data)
  File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 233, in wbf
    d = data.encode('utf-8')  # turn into bytes/string so f.write doesn't try to make it ascii'
AttributeError: 'NoneType' object has no attribute 'encode'

Display More

Quote from edit4ever

Well - my system ran correctly again last night with extra details...so it looks like I'm going to have to wait for another error to appear so i can troubleshoot. I want to narrow down whether the issue is with missing data or changed data. BUt until I have another error - I don't have a place to look.

If yours fails again - please provide the log so I can look at the point of failure.

Thanks!

tvheadend, ATSC and epg...

Similar Threads

[8.0.2c] LibreELEC 8.0 builds for KI Plus / KII Pro / KI Pro

Options for PVR backend in LE8