Posts by stepher

I was going to provide you with the contents of the EP024717260011.html.gz file, but that file doesn't exist (looked in ~/.kodi/addons/script.module.zap2xml/cache/). What is there is EP024717260011.js.gz, but that file seems empty. There are also a number of XXXXXX...XXX.html.gz files (the XXXX.XXXX of which are all digits and which seem to provide an html view of the TV Listing grid), but I'll have to figure out what the EP024717260011 resolves to for the XXXXX.....XXX.html.gz file name....and right now my OSMC sys crashed and I'll have to do a hard reset

As for no. of stations...I've pretty much had that from the beginning, so when it's worked, it's been fine. When it hasn't, well....

Quote from edit4ever

Wow - you have a big list of stations! If you're not actually using all 79 stations, you might want to edit the list down so only what you need is downloaded.

Let me know how your testing goes. In the meantime, I'll still try to look for any issues with the falied download section of the code. Thanks for the update!

OK. Now I'm really perplexed (Hate it when things work all of a sudden and I have no idea why). Checked my tvguide.xml file this morning and it was updated early this morning (~3:30AM) just like it is programmed to be. Looked in the zap2xml.log file and this was at the end:

-------------------------------------------------------------------------------

....snip....

Downloaded 15119541 bytes in 5073 http requests.
Writing XML file: /home/osmc/.xmltv/tvguide.xml
Completed in 988s (Parse: 825s) 79 stations, 10409 programs, 35870 scheduled.

-------------------------------------------------------------------------------

Not sure what, if anything, changed. Time to do some "scientific experimentation" with the "xtra program data" and other options and see if any of those are still having some kind of effect. I'll check back in if I notice anything of interest.

Thx and cheers.....

Quote from edit4ever

That looks like it failed when it couldn't download the EP024717260011 episode information. That file doesn't exist - if you try to open the link screener gives you an error. I'll need to look through the code - it should have a way to deal with failed downloads, but something seems like it isn't working for you. Let's find out what program this is. If you open the .html.gz files you can search for EP024717260011 and see what show it is. Then we can go onto the screener site and see if the data there works or not.

I don't have OSMC, I run LE - although I believe they are mostly the same. I am running the latest LE 7.95 beta 2, which is kodi 17, on a raspberry pi 3 and it all is running fine. I don't seem to be able to duplicate the issue on my end. It likely just means there is a problem with the screener listing in your market - but the program should be able to handle that...so on to investigating!

OK. New (maybe, old) problem. I updated to OSMC/Kodi 17 and this problem seems to have occurred after the update. Any insight as to zap2xml and Kodi 17 compatibility? I cleared the cache directory and did a restart of tvheadend (also fairly certain I disabled channel icon download, but need to recheck if I disabled "xtra program details" and get back on that). Here's the last part of the zpa2xml.log file:

------------------------------------------------------------------------
....snip.....

Getting: gridDetailService?pgmId=EP024717260009
[D] Parsing: EP024717260009
Getting: gridDetailService?rtype=pgmimg&pgmId=EP024717260009
[I] Parsing: EP024717260009
Getting: gridDetailService?pgmId=EP024717260011
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Getting: gridDetailService?pgmId=EP024717260011
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Getting: gridDetailService?pgmId=EP024717260011
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Failed to download within 3 retries.
:Function: getURL :Line: 228
error<class 'mechanize._response.httperror_seek_wrapper'>
Exception
error<type 'exceptions.AttributeError'>
Traceback (most recent call last):
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2132, in <module>
main()
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2042, in main
parseGrid(fn) #data read in and unicode it
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 1094, in parseGrid
p.feed(b)
File "/usr/lib/python2.7/HTMLParser.py", line 117, in feed
self.goahead(0)
File "/usr/lib/python2.7/HTMLParser.py", line 161, in goahead
k = self.parse_starttag(i)
File "/usr/lib/python2.7/HTMLParser.py", line 327, in parse_starttag
self.handle_starttag(tag, attrs)
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 716, in handle_starttag
globals()['on_%s' % tag](self, tag, attrs)
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 345, in on_td
wbf(fn, data)
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 233, in wbf
d = data.encode('utf-8') # turn into bytes/string so f.write doesn't try to make it ascii'
AttributeError: 'NoneType' object has no attribute 'encode'

2/5/17 UPDATE: Been ~3 days and guide updates are working as expected.

Does seem to be something related to program (channel?) icons and downloading them correctly. I continue to keep that function disabled and will report back in about a week or so.

Thx and cheers....

-------------------------------------------------------------------------

Here's what the channel info looks like:

<channel id="I19571.labs.zap2it.com">
<display-name>2.1 KTVUDT</display-name>
<display-name>2.1</display-name>
<display-name>KTVUDT</display-name>
<icon src="http://images.zap2it.com/station_logo/fox_affiliate.gif" />
</channel>

...and the program info (I assume channel="I65632.labs.zap2it.com" below tracks back to the proper channel id in the channel info listing):

<programme start="20170202060000 -0800" stop="20170202140000 -0800" channel="I65632.labs.zap2it.com">
<title lang="en">What &apos;s On</title>
<desc lang="en">What &apos;s On </desc>
<episode-num system="dd_progid">SH01615902.0000</episode-num>
<previously-shown />
</programme>

Quote from edit4ever

stepher Can you check your xmltv.xml file and see if the channel numbers have been replaced with "0"??

My test download of your lineup gave me that instead of the channel numbers, so I'm wondering if something changed with the way they list the channels. Since I run my system with the channel name first option enabled, that may be why I haven't had an issue.

OTA

Quote from edit4ever

I mean - Over The Air or Cable or Satellite??? Usually in each zipcode - there are multiple providers (lineups) to choose from. You likely selected this when you customized your channels on the zap2it/screener site.

ZipCode: 94086
Lineup? Do you mean which channels? If so, what's the easiest way to get that list to you?

Quote from edit4ever

This is strange as it is running on two of my systems without issue. There must be some other code in your lineup that I don't have in mine.

What zipcode and lineup are you using? I'll try to recreate the issue on my end.

It seems as if the 'xtra prgram details' option continues to present problems. As soon as I disabled it and went thru a reboot, my tvguide.xml file was updated as expected.

I'll keep it turned off until if, and when, you've got a viable coding solution that works with 'xtra prgram details' enabled.

Quote from edit4ever

OK - let's try to narrow down the issue. Turn off the download extra program details option and let's see if it will pull basic guide info.

Cleared the cache (Just want to confirm you meant "~\.kodi\addons\script.module.zap2xml\cache\") of 20K+ items and restarted tvh. Results below (looks about the same). No updates or system changes that I initiated.

Thx and cheers....

....snip....
Getting: gridDetailService?pgmId=EP024717260009
[D] Parsing: EP024717260009
Getting: gridDetailService?rtype=pgmimg&pgmId=EP024717260009
Parsing: EP024717260009
Getting: gridDetailService?pgmId=EP024717260011
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Getting: gridDetailService?pgmId=EP024717260011
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Getting: gridDetailService?pgmId=EP024717260011
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Failed to download within 3 retries.
:Function: getURL :Line: 228
error<class 'mechanize._response.httperror_seek_wrapper'>
Exception
error<type 'exceptions.AttributeError'>
Traceback (most recent call last):
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2132, in <module>
main()
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2042, in main
parseGrid(fn) #data read in and unicode it
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 1094, in parseGrid
p.feed(b)
File "/usr/lib/python2.7/HTMLParser.py", line 117, in feed
self.goahead(0)
File "/usr/lib/python2.7/HTMLParser.py", line 161, in goahead
k = self.parse_starttag(i)
File "/usr/lib/python2.7/HTMLParser.py", line 327, in parse_starttag
self.handle_starttag(tag, attrs)
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 716, in handle_starttag
globals()['on_%s' % tag](self, tag, attrs)
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 345, in on_td
wbf(fn, data)
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 233, in wbf
d = data.encode('utf-8') # turn into bytes/string so f.write doesn't try to make it ascii'
AttributeError: 'NoneType' object has no attribute 'encode'

Quote from edit4ever

That's an interesting error - looks like the problem was specifically with this download: 1487102400000.html.gz

You can try deleting that file or all the XXXXXXXX.html.gz files from the cache folder and see if it gets better data.

I am still up and running on my system - so I don't believe there was a change to the Screener site that would have broken it.

I should probably look at adding a "delete cache" function in the program so when there are errors it can run a clena download.

Let me know if clearing the cache file works.

Wish I had more good news...but alas

It looked like I had some subnet issue last night, but wasn't clear if that was related to the problem. So I restarted tvheadend just to be sure. Results are pretty much the same.

Here's the output from the zap2xml.log file:

....snip.....
[53/56] Parsing: /home/osmc/.kodi/addons/script.module.zap2xml/cache/1487080800000.html.gz
[54/56] Parsing: /home/osmc/.kodi/addons/script.module.zap2xml/cache/1487102400000.html.gz
Exception
error<type 'exceptions.ValueError'>
Traceback (most recent call last):
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2132, in <module>
main()
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2042, in main
parseGrid(fn) #data read in and unicode it
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 1094, in parseGrid
p.feed(b)
File "/usr/lib/python2.7/HTMLParser.py", line 117, in feed
self.goahead(0)
File "/usr/lib/python2.7/HTMLParser.py", line 161, in goahead
k = self.parse_starttag(i)
File "/usr/lib/python2.7/HTMLParser.py", line 327, in parse_starttag
self.handle_starttag(tag, attrs)
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 716, in handle_starttag
globals()['on_%s' % tag](self, tag, attrs)
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 347, in on_td
parseJSOND(fn)
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 863, in parseJSOND
t = json.loads(b)
File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Quote from edit4ever

awesome - thanks for the update. If all is well after you get through a full week+ - I'll update the addon with the fix.

Just a quick update to let you know things are still working fine.Thx. When it gets to a week+, then I'll feel more comfortable.

What's a bit baffling for me is how the 1st code change a couple of weeks ago worked for a bit and then failed. That seems a bit odd.

Anyway, we're good now and I'll let you know when I hit the week milestone

Cheers and thx again....

Quote from stepher

OK. Back up and running (again). Thx. However.....been there, done that I'll wait to confirm if it's really fixed this time.

One other thing....in the section of code above the one that you suggested to be modified (starts on line 342 for me "if cp != -1 and "-D" in options:") there are a few references to the variable "cp". I assume typing is not an issue in this section (I know, if it ain't broke, don't fix it).

Appreciate the help in getting back on track. Of course I'll be back if something else pops up

Cheers....

There must be some program numbers that have a character that is messing with the putting together of the string.

My guess is if we force that whoel section to use str(cp) it will fix it... try this starting at 348:

if "-I" in options:
fn = os.path.join(cacheDir,"I" + str(cp) + ".js.gz")
if not os.path.isfile(fn):
data = getURL(urlRoot + "gridDetailService?rtype=pgmimg&pgmId=" + str(cp))
if data: #sometimes we fail to get the url try to keep going
wbf(fn, data)
log.pout(" Parsing: " + str(cp),'info')
parseJSONI(fn)
[hr]

Display More

OK. Back up and running (again). Thx. However.....been there, done that I'll wait to confirm if it's really fixed this time.

One other thing....in the section of code above the one that you suggested to be modified (starts on line 342 for me "if cp != -1 and "-D" in options:") there are a few references to the variable "cp". I assume typing is not an issue in this section (I know, if it ain't broke, don't fix it).

Appreciate the help in getting back on track. Of course I'll be back if something else pops up

Cheers....

Quote from edit4ever

[quote='stepher','https://207.154.221.202/thread/?postID=28851#post28851']
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 352, in on_td
data = getURL(urlRoot + "gridDetailService?rtype=pgmimg&pgmId=" + cp)
TypeError: cannot concatenate 'str' and 'int' objects

There must be some program numbers that have a character that is messing with the putting together of the string.

My guess is if we force that whoel section to use str(cp) it will fix it... try this starting at 348:

if "-I" in options:
fn = os.path.join(cacheDir,"I" + str(cp) + ".js.gz")
if not os.path.isfile(fn):
data = getURL(urlRoot + "gridDetailService?rtype=pgmimg&pgmId=" + str(cp))
if data: #sometimes we fail to get the url try to keep going
wbf(fn, data)
log.pout(" Parsing: " + str(cp),'info')
parseJSONI(fn)
[hr]

Ooops...may have spoken too soon. Things seemed to be fine for a few days (update out to 14 days) and then the last update was 2 days ago ( Jan 14, 2017). Did a tvheadend restart (still no update) and here's the resulting log file output:

**** 
NOTE: Line 352 in my zap2xml.py file may be a different line number than yours. I have 2 deprecated lines (from previous revision) commented out above line 352.
---------------------------------------------------------------------------------------------


....snip.....
2017-01-16 13:34:58,968 [48/56] Parsing: /home/osmc/.kodi/addons/script.module.zap2xml/cache/1485612000000.html.gz
2017-01-16 13:34:59,040 Exception
2017-01-16 13:34:59,043 Traceback (most recent call last):
  File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2041, in main
    parseGrid(fn) #data read in and unicode it
  File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 1093, in parseGrid
    p.feed(b)
  File "/usr/lib/python2.7/HTMLParser.py", line 117, in feed
    self.goahead(0)
  File "/usr/lib/python2.7/HTMLParser.py", line 161, in goahead
    k = self.parse_starttag(i)
  File "/usr/lib/python2.7/HTMLParser.py", line 327, in parse_starttag
    self.handle_starttag(tag, attrs)
  File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 715, in handle_starttag
    globals()['on_%s' % tag](self, tag, attrs)
  File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 352, in on_td
    data = getURL(urlRoot + "gridDetailService?rtype=pgmimg&pgmId=" + cp)
TypeError: cannot concatenate 'str' and 'int' objects

Have done no (obvious) updates so no idea what might have changed to cause this. Looks like another type mismatch. Hold that revision update

Thx and cheers....

Quote from edit4ever

Awesome! Thanks for the update...I'll put out a revised file with the fixed code.

Just confirming that my tvguide.xml file was updated last night at it's scheduled time (actually, early this morning) and it has the programming schedule data for 14 days out. I'm good!!

Thx and cheers....

Quote from stepher

As near as I can tell, the code changes you provided works. I updated the zap2xml.py file (commenting out the "bad" lines), (re)enabled downloading of program icons and restarted tvheadend. Waited for the results to happen and it checked out great. Thx.

Iirc the updated code from the zip file had a line (348, I believe) of "cp = str(cp)" and that did not seem to work. The new code seems to work (so far), but I should have a better idea about that tomorrow morning (channel info updates happen at 3:30AM).

Appreciate the work and effort to help....cheers....

As near as I can tell, the code changes you provided works. I updated the zap2xml.py file (commenting out the "bad" lines), (re)enabled downloading of program icons and restarted tvheadend. Waited for the results to happen and it checked out great. Thx.

Iirc the updated code from the zip file had a line (348, I believe) of "cp = str(cp)" and that did not seem to work. The new code seems to work (so far), but I should have a better idea about that tomorrow morning (channel info updates happen at 3:30AM).

Appreciate the work and effort to help....cheers....

Quote from edit4ever

The "fix" should have been adding str() to the icon fn line 349. I think I was rushing and did it wrong. Try changing the icon section starting at line 348 in zap2xml.py to the following:

Code

if "-I" in options:
                    fn = os.path.join(cacheDir,"I" + str(cp) + ".js.gz")
                    if not os.path.isfile(fn):
                        data = getURL(urlRoot + "gridDetailService?rtype=pgmimg&pgmId=" + cp)
                        if data: #sometimes we fail to get the url try to keep going
                            wbf(fn, data)
                            log.pout("[i] Parsing: " + cp,'info')
                    parseJSONI(fn)[/i]

If that doesn't work - I'd love to see the log error. Thanks!

Couldn't get it working using the 0.6.1 update. However, had the chance to disable program icon download and that brought things back to "normal". Things are good again

Thx and cheers....

Quote from stepher

I could uninstall 0.6.0 but couldn't get 0.6.1 to install. So I reinstalled 0.6.0 and then extracted zap2xml.py manually, fixed any pathing and then just did a copy over the old zap2xml.py (Hope that didn't miss anything). I ran tv_grab_zap2xml from ssh and also restarted tvheadend. Running manually (from ssh), none of the LEDs on the HDHomerun tuner flashed and my tvguide.xml never got updated. The log file(s) seemed to get updated, but stopped in what looks like an incomplete scanning process somewhere along the way. Restarting tvheadend got the tuner LEDs flashing and filled the log file (incomplete also) but still no update to my tvguide.xml file.

I'll admit I've been a bit rushed this morning so I'll try another go at it later on today. Haven't had a chance to turn off the program icon update option so I'll try that next.

Assuming I modified no other files, was copying the zap2xml.py file enuf to solve the problem (since it seemed specific to the zap2xml.py code)? Or is there something else that needs to be updated by doing a more "normal" 0.6.1 script install?

Thx and cheers.....

I could uninstall 0.6.0 but couldn't get 0.6.1 to install. So I reinstalled 0.6.0 and then extracted zap2xml.py manually, fixed any pathing and then just did a copy over the old zap2xml.py (Hope that didn't miss anything). I ran tv_grab_zap2xml from ssh and also restarted tvheadend. Running manually (from ssh), none of the LEDs on the HDHomerun tuner flashed and my tvguide.xml never got updated. The log file(s) seemed to get updated, but stopped in what looks like an incomplete scanning process somewhere along the way. Restarting tvheadend got the tuner LEDs flashing and filled the log file (incomplete also) but still no update to my tvguide.xml file.

I'll admit I've been a bit rushed this morning so I'll try another go at it later on today. Haven't had a chance to turn off the program icon update option so I'll try that next.

Assuming I modified no other files, was copying the zap2xml.py file enuf to solve the problem (since it seemed specific to the zap2xml.py code)? Or is there something else that needs to be updated by doing a more "normal" 0.6.1 script install?

Thx and cheers.....

Quote from stepher

I'll test the update 1st and, if that doesn't work, disable the "download program icons" and let you know the results either way.

Thanks and cheers....

I'll test the update 1st and, if that doesn't work, disable the "download program icons" and let you know the results either way.

Thanks and cheers....

Quote from edit4ever

Disable the "download program icons" setting. I believe something may have changed on the screener site that is messing that up.

After you disable - you can rerun the internal grabber and it should work again.

I think I can just convert the the 'cp' to a string and it will work again - but I'll have to do some testing.

---
Try updating to this version and test to see if the program icons download:

The system had been working great. But something since 1/4/17 has been acting up again I haven't done any updates or made any changes (except for adding a "advancedsettings.xml to solve a streaming buffer issue). There might have been an update to something, but I'm fairly certain I didn't initiate it.

My tvguide.xml file hasn't been updated since 1/4/17. The zap2xml.log file under ~/.kodi/userdata/addon_data/script.module.zap2xml shows the following (the zap2xml.log file under ~/.kodi/addons/script.module.zap2xml appears to have the same log data):

....snip....
[38/56] Parsing: /home/osmc/.kodi/addons/script.module.zap2xml/cache/1484726400000.html.gz
[39/56] Parsing: /home/osmc/.kodi/addons/script.module.zap2xml/cache/1484748000000.html.gz
Exception
error<type 'exceptions.TypeError'>
Traceback (most recent call last):
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2130, in <module>
main()
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2040, in main
parseGrid(fn) #data read in and unicode it
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 1092, in parseGrid
p.feed(b)
File "/usr/lib/python2.7/HTMLParser.py", line 117, in feed
self.goahead(0)
File "/usr/lib/python2.7/HTMLParser.py", line 161, in goahead
k = self.parse_starttag(i)
File "/usr/lib/python2.7/HTMLParser.py", line 327, in parse_starttag
self.handle_starttag(tag, attrs)
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 714, in handle_starttag
globals()['on_%s' % tag](self, tag, attrs)
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 350, in on_td
fn = os.path.join(cacheDir,"I" + cp + ".js.gz")
TypeError: cannot concatenate 'str' and 'int' objects

Looks like a variable type issue in the .py file.

Anything obvious to you?

Thanks and cheers....