tvheadend, ATSC and epg...

Cleared the cache (Just want to confirm you meant "~\.kodi\addons\script.module.zap2xml\cache\") of 20K+ items and restarted tvh. Results below (looks about the same). No updates or system changes that I initiated.

Thx and cheers....

....snip....
Getting: gridDetailService?pgmId=EP024717260009
[D] Parsing: EP024717260009
Getting: gridDetailService?rtype=pgmimg&pgmId=EP024717260009
Parsing: EP024717260009
Getting: gridDetailService?pgmId=EP024717260011
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Getting: gridDetailService?pgmId=EP024717260011
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Getting: gridDetailService?pgmId=EP024717260011
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Failed to download within 3 retries.
:Function: getURL :Line: 228
error<class 'mechanize._response.httperror_seek_wrapper'>
Exception
error<type 'exceptions.AttributeError'>
Traceback (most recent call last):
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2132, in <module>
main()
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2042, in main
parseGrid(fn) #data read in and unicode it
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 1094, in parseGrid
p.feed(b)
File "/usr/lib/python2.7/HTMLParser.py", line 117, in feed
self.goahead(0)
File "/usr/lib/python2.7/HTMLParser.py", line 161, in goahead
k = self.parse_starttag(i)
File "/usr/lib/python2.7/HTMLParser.py", line 327, in parse_starttag
self.handle_starttag(tag, attrs)
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 716, in handle_starttag
globals()['on_%s' % tag](self, tag, attrs)
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 345, in on_td
wbf(fn, data)
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 233, in wbf
d = data.encode('utf-8') # turn into bytes/string so f.write doesn't try to make it ascii'
AttributeError: 'NoneType' object has no attribute 'encode'

Quote from edit4ever

That's an interesting error - looks like the problem was specifically with this download: 1487102400000.html.gz

You can try deleting that file or all the XXXXXXXX.html.gz files from the cache folder and see if it gets better data.

I am still up and running on my system - so I don't believe there was a change to the Screener site that would have broken it.

I should probably look at adding a "delete cache" function in the program so when there are errors it can run a clena download.

Let me know if clearing the cache file works.

It seems as if the 'xtra prgram details' option continues to present problems. As soon as I disabled it and went thru a reboot, my tvguide.xml file was updated as expected.

I'll keep it turned off until if, and when, you've got a viable coding solution that works with 'xtra prgram details' enabled.

Quote from edit4ever

OK - let's try to narrow down the issue. Turn off the download extra program details option and let's see if it will pull basic guide info.

ZipCode: 94086
Lineup? Do you mean which channels? If so, what's the easiest way to get that list to you?

Quote from edit4ever

This is strange as it is running on two of my systems without issue. There must be some other code in your lineup that I don't have in mine.

What zipcode and lineup are you using? I'll try to recreate the issue on my end.

OTA

Quote from edit4ever

I mean - Over The Air or Cable or Satellite??? Usually in each zipcode - there are multiple providers (lineups) to choose from. You likely selected this when you customized your channels on the zap2it/screener site.

2/5/17 UPDATE: Been ~3 days and guide updates are working as expected.

Does seem to be something related to program (channel?) icons and downloading them correctly. I continue to keep that function disabled and will report back in about a week or so.

Thx and cheers....

-------------------------------------------------------------------------

Here's what the channel info looks like:

<channel id="I19571.labs.zap2it.com">
<display-name>2.1 KTVUDT</display-name>
<display-name>2.1</display-name>
<display-name>KTVUDT</display-name>
<icon src="http://images.zap2it.com/station_logo/fox_affiliate.gif" />
</channel>

...and the program info (I assume channel="I65632.labs.zap2it.com" below tracks back to the proper channel id in the channel info listing):

<programme start="20170202060000 -0800" stop="20170202140000 -0800" channel="I65632.labs.zap2it.com">
<title lang="en">What &apos;s On</title>
<desc lang="en">What &apos;s On </desc>
<episode-num system="dd_progid">SH01615902.0000</episode-num>
<previously-shown />
</programme>

Quote from edit4ever

stepher Can you check your xmltv.xml file and see if the channel numbers have been replaced with "0"??

My test download of your lineup gave me that instead of the channel numbers, so I'm wondering if something changed with the way they list the channels. Since I run my system with the channel name first option enabled, that may be why I haven't had an issue.

OK. New (maybe, old) problem. I updated to OSMC/Kodi 17 and this problem seems to have occurred after the update. Any insight as to zap2xml and Kodi 17 compatibility? I cleared the cache directory and did a restart of tvheadend (also fairly certain I disabled channel icon download, but need to recheck if I disabled "xtra program details" and get back on that). Here's the last part of the zpa2xml.log file:

------------------------------------------------------------------------
....snip.....

Getting: gridDetailService?pgmId=EP024717260009
[D] Parsing: EP024717260009
Getting: gridDetailService?rtype=pgmimg&pgmId=EP024717260009
[I] Parsing: EP024717260009
Getting: gridDetailService?pgmId=EP024717260011
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Getting: gridDetailService?pgmId=EP024717260011
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Getting: gridDetailService?pgmId=EP024717260011
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Failed to download within 3 retries.
:Function: getURL :Line: 228
error<class 'mechanize._response.httperror_seek_wrapper'>
Exception
error<type 'exceptions.AttributeError'>
Traceback (most recent call last):
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2132, in <module>
main()
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2042, in main
parseGrid(fn) #data read in and unicode it
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 1094, in parseGrid
p.feed(b)
File "/usr/lib/python2.7/HTMLParser.py", line 117, in feed
self.goahead(0)
File "/usr/lib/python2.7/HTMLParser.py", line 161, in goahead
k = self.parse_starttag(i)
File "/usr/lib/python2.7/HTMLParser.py", line 327, in parse_starttag
self.handle_starttag(tag, attrs)
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 716, in handle_starttag
globals()['on_%s' % tag](self, tag, attrs)
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 345, in on_td
wbf(fn, data)
File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 233, in wbf
d = data.encode('utf-8') # turn into bytes/string so f.write doesn't try to make it ascii'
AttributeError: 'NoneType' object has no attribute 'encode'

OK. Now I'm really perplexed (Hate it when things work all of a sudden and I have no idea why). Checked my tvguide.xml file this morning and it was updated early this morning (~3:30AM) just like it is programmed to be. Looked in the zap2xml.log file and this was at the end:

-------------------------------------------------------------------------------

....snip....

Downloaded 15119541 bytes in 5073 http requests.
Writing XML file: /home/osmc/.xmltv/tvguide.xml
Completed in 988s (Parse: 825s) 79 stations, 10409 programs, 35870 scheduled.

-------------------------------------------------------------------------------

Not sure what, if anything, changed. Time to do some "scientific experimentation" with the "xtra program data" and other options and see if any of those are still having some kind of effect. I'll check back in if I notice anything of interest.

Thx and cheers.....

Quote from edit4ever

That looks like it failed when it couldn't download the EP024717260011 episode information. That file doesn't exist - if you try to open the link screener gives you an error. I'll need to look through the code - it should have a way to deal with failed downloads, but something seems like it isn't working for you. Let's find out what program this is. If you open the .html.gz files you can search for EP024717260011 and see what show it is. Then we can go onto the screener site and see if the data there works or not.

I don't have OSMC, I run LE - although I believe they are mostly the same. I am running the latest LE 7.95 beta 2, which is kodi 17, on a raspberry pi 3 and it all is running fine. I don't seem to be able to duplicate the issue on my end. It likely just means there is a problem with the screener listing in your market - but the program should be able to handle that...so on to investigating!

I was going to provide you with the contents of the EP024717260011.html.gz file, but that file doesn't exist (looked in ~/.kodi/addons/script.module.zap2xml/cache/). What is there is EP024717260011.js.gz, but that file seems empty. There are also a number of XXXXXX...XXX.html.gz files (the XXXX.XXXX of which are all digits and which seem to provide an html view of the TV Listing grid), but I'll have to figure out what the EP024717260011 resolves to for the XXXXX.....XXX.html.gz file name....and right now my OSMC sys crashed and I'll have to do a hard reset

As for no. of stations...I've pretty much had that from the beginning, so when it's worked, it's been fine. When it hasn't, well....

Quote from edit4ever

Wow - you have a big list of stations! If you're not actually using all 79 stations, you might want to edit the list down so only what you need is downloaded.

Let me know how your testing goes. In the meantime, I'll still try to look for any issues with the falied download section of the code. Thanks for the update!

I'm glad to continue to act as a kind of "beta tester" for you (as long as it doesn't get too out of hand If we can figure out my issues, then hopefully everyone else can benefit from it. This is all about DIY, right?

What nips at me is not the fact that it stopped working after I upgraded to OSMC/Kodi 17 (maybe it should but with permissions, etal, it's not a surprise) but rather that it started working again this morning, and I didn't do anything (tho, maybe I changed 1 of the options and just don't remember Anyways, like I said, scientific experimentation and see what inputs change the output.

Let you know when I get useful data....

BTW - Should the XXXXX.js.gz files be empty once my tvguide.xml file has been updated?

Thx and cheers.....

Quote from edit4ever

The xxxxxx.html.gz files contain the basic grid in 6 hour chunks. So if you have set the addon to download 1 day, you will have 4 of those files. When you have either the xtra description download or the program icon download enabled - the program parses (scans) those files and pulls the episode information and uses that to download the extra program details (generating the xxxxxxxxx.js.gz file) and the episode program icon (generating the Ixxxxxxxxx.js.gz file). Then all of that info is parsed to generate the final xmltv.xml file.

Apparently you're having an issue where it wants to download the xtra program detail or program icon and screener says the file doesn't exist. In theory the program should just move on - but apparently there is a piece of code missing. The weird thing is that should never happen. If there is an episode scheduled, it has a number and should have information to download. It's as though screener has the worng episode number in the program schedule for on of the programs in your listing.

Anyway - we'll keep trying to fix it! (or if you want - you can just get the schedules direct service for $25 for the year. That addon works great and has lots of good data!)