tvheadend, ATSC and epg...

Just to update. Mine is working good so far but I don't download icons like you. Will keep an eye on it and let you know.

Thanks!

If you refer back to a recent previous posting, I mentioned that I inadvertently enabled Download Program Icons (when I intended it to remain disabled when I tested xtra program details being disabled, as well) and I had a successful tvguide.xml update. So, I don't believe my problem is with the Program Icons.

I just went thru 2 tests (1 tvheadend restart and 1 system reboot) with Program Icons disabled and xtra program details enabled and got the same (well, almost in that it made it further than last time I posted) error cast. Here's the error part of the log results:

...snip....
[27/56] Parsing: /home/osmc/.kodi/addons/script.module.zap2xml/cache/1488700800000.html.gz
[28/56] Parsing: /home/osmc/.kodi/addons/script.module.zap2xml/cache/1488722400000.html.gz
Exception
error<type 'exceptions.ValueError'>
Traceback (most recent call last):
  File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2132, in <module>
    main()
  File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 2042, in main
    parseGrid(fn) #data read in and unicode it
  File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 1094, in parseGrid
    p.feed(b)
  File "/usr/lib/python2.7/HTMLParser.py", line 117, in feed
    self.goahead(0)
  File "/usr/lib/python2.7/HTMLParser.py", line 161, in goahead
    k = self.parse_starttag(i)
  File "/usr/lib/python2.7/HTMLParser.py", line 327, in parse_starttag
    self.handle_starttag(tag, attrs)
  File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 716, in handle_starttag
    globals()['on_%s' % tag](self, tag, attrs)
  File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 349, in on_td
    parseJSOND(fn)
  File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 863, in parseJSOND
    t = json.loads(b)
  File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Quote from edit4ever

Oops - I did the fix the code in the icon section. That's where it failed for you. Turn off program icons for now and when I get back to my system I will fix that section.

Sorry about that!

No on lines 865-873 looking like you what you've listed below. I had downloaded the file from the link you provided in msg. #197 (zap2xml.zip). That's water under the bridge, tho.

Late yesterday I had the chance to download and install zap2xml (2).zip. Did a tvheadend restart and, voile', updated tvguide.xml file. Had to do a system reboot for some reason and my tvguide.xml was again updated as tvheadend restarted. Just checked again this morning and system is still updating as expected.

So, looks like things are working again. Thx.

BTW - Before all of the above, I disabled Download Program Icons because I'm not sure I really care about those icons, only the station icons. However, I may revisit that sometime down the road.

wrt contents of the log file...to clarify...I am getting the Parsing statements (see below for most recent log file contents). What seemed to disappear was a statement before *each* parsing line about "Getting: tvschedule.......". That's been gone for a couple of iterations of the code at this point. As a side note, my parsing channel ids (?) have varying start letters: EP, SH and MV (not sure if that's significant at all).

What I am seeing is (what appears to be random, but may not be) are lines like this:

....snip....
[D] Parsing: SH019264110000
[54/56] Parsing: /home/osmc/.kodi/addons/script.module.zap2xml/cache/1489327200000.html.gz
Getting: TV Listings Grid, TV Guide and TV Schedule, Where to Watch TV Shows - Screener
[D] Parsing: SH016159020000
....snip....

...AND.....

....snip....
[D] Parsing: SH012378150000
[D] Parsing: SH010452440/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py:617: UserWarning: gzip transfer encoding is experimental!
br.set_handle_gzip(True)
000
[D] Parsing: SH016354690000
....snip....

I haven't looked closely thru the whole file, but a fast scan shows these kinds of lines spread throughout the file (but not a lot of them).

I'll download ver. 4 and let you know the results. It seems like things are getting fairly close. I really appreciate your effort in working thru this (don't you have a wife/husband or kids or job or a life to look after (big grin

Cheers and thx....

Quote from edit4ever

Can you confirm that lines 865-873 of your zap2xml.py file (in .kodi/addons/script.module.zap2xml) looks like this:

Code

def parseJSOND(fn):
    global programs, cp
    with gzip.open(fn,"rb") as f:
        e = f.read(1)
        b = f.read()
        f.close()
    if not e:
        log.pout("[D] Skipping: " + cp,'info')
    else:

If so - then one of the issues we may be running into is a change to the amount of data that zap2it/screener is allowing to be read/downloaded from their server before sending a blank response. However, it's strange that your zap2xml log didn't have any lines that refer to the details parsing.

Those should look like - [D] Parsing: EP000191865565
[hr]
OK - This version of zapxml.py uses a different method to skip empty files. If the earlier version isn't working for you - you can try this one. Unfortunately, my lineup doesn't seem to have any empty error files to test with - so I'm relying on your feedback to try out fixes.

No good - figured out a way to test (created an empty details file) - and it didn't work. Stand by...
[hr]
4th time's the charm! I tested this on my end with an empty details file and it finsihed the parsing. When a details or icon download file is empty it will show like this:

[D] Skipping: EP000021770103 2017-02-26 16:34:55,004

And should just keep going with the rest of the files.

Here's the updated zap2xml.py file. Please let me know if you see any errors - I think I've got it this time.

Thanks!

Display More

OK. I updated zap2xml.py and ran (restarted) tvheadend. The tvguide.xml file updated just fine. I'll give it at least a few more "normal" updates just to be sure. WRT update frequency....I get the reduction in network activity by doing multiple updates on a daily basis. Something else I can experiment with once I'm sure the system is stable.

In the log file in .kodi/userdata/add_on/script.module.zap2xml none of the entries have any time stamps (and there was 1 "Skipped:" entry). However, in the log file in .kodi/addons/script.module.zap2xml the date/time stamps are there, except at the beginning of the line, not the end. Is that by design?

Are you a technical consultant so this PVR stuff is an easy distraction for you? Or are you non-technical and this is truly a hobby of love?

Cheers...

Quote from edit4ever

Thanks for the update! Version 4 should work for both details and icons - so feel free to test.

Fortunately my wife and I run our own consulting business - so I get to have lots of time to work on hobbies like this!

3/5/17 Quick Update:

My tvguide.xml file still being updated as set in the schedule. So things are still good.

What I noticed this morning in the xap2xml.log file was the following:

....snip....
[D] Parsing: EP024717260046
Getting: gridDetailService?pgmId=EP024717260035
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Getting: gridDetailService?pgmId=EP024717260035
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Getting: gridDetailService?pgmId=EP024717260035
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Failed to download within 3 retries.
:Function: getURL :Line: 228
error<class 'mechanize._response.httperror_seek_wrapper'>
[D] Failed to download: EP024717260035
[D] Parsing: SH000046300000
....snip....

I assume this is what I should see when a specific update cannot be downloaded. Saw this same logging of repeated attempts 1 or 2 times more in the file. The final lines of the file:

....snip....
Downloaded 1972995 bytes in 521 http requests.

Writing XML file: /home/osmc/.xmltv/tvguide.xml

Completed in 411s (Parse: 198s) 79 stations, 9693 programs, 34851 scheduled.

which seems to indicate that even tho there were a few errors, the tvguide.xml file was still updated...which I believe is what we were looking for.

Cheers....

------------------------------------------------------------------------------------------

2 days in...so far so good I'll post status again at the end of a week. Now I can go focus on a few other aspects of the PVR. I haven't yet cut the cord, but I'm getting very close...maybe another month, possibly 2, at most.

Funny that you come from a non-technical space. I, too, am a former marketing exec, but in the high tech domain (Silicon Valley). I also have an engineering degree, and while it was focused on the h/w side, I've done quite a bit of dabbling in s/w...a little bit of hi level here and there (still threatening to learn Python), but mostly at the CPU level ("diddling bits and bytes", as they say

I'm glad and very appreciative you took this on. I feel zap2xml is turning into a really nice addon. Maybe, just maybe, I'll actually get Python learned and I can look at contributing new features, as well.

Thx and cheers....

Quote from edit4ever

Thanks again for the update. This is all totally a hobby. I didn't know the first thing about python and other code until I started playing with this and modifying Kodi skins.

As for the different logs - I'm not sure how or why that happens. This project was orginally done by fasteddycurrent - which you can see here: GitHub - FastEddyCurrent/zap2xml: zap2xml in Python 2.7 for use on the RaspberryPI - it was he who converted the the original zap2xml perl script into python.

I basically took what he did and created a kodi addon and then started adding features I wanted to see. And now here we are. I'm just a former tv marketing exec who likes to play. I'm driven by the goal of wanting cheaper/better dvr service and this year I finally cut the cord. If i get around to it, I'll have to look at how the log system is structured.

BTW - the "Skipped:" entry means the code fix worked. That indicates that there was an empty or missing details file that would have previously crashed the grabber. Glad to see it completed!

What's different in 0.6.5? Can I just replace the zap2xml.py file (with obvious needed pathing changes) or do I need to do an install from within OSMC?

Quote from edit4ever

Someone asked for the zipped addon....I'm travelling - so I haven't been able to run any tests...but since the feedback from my code fixes seems to be fine - you can try this:

Please let me know if you have issues. Thanks!

Will install when I get home. Do you know If I install from zip will it still update from your repo in the future?

Good timing...I had unzipped the file and was just about to update the .py file and install.

"Use me and abuse me"...as your control control

Safe travels....

Quote from edit4ever

This is the same as the version you're running since you copied the zap2xml.py file over manually. Someone just asked for the version that could be installed from the addon menu. You don't have a need to update. Actually - I prefer if you don't update, in case someone finds an issue with the zipped file. I'm travelling and every now and then I mess up pulling in the updated files! This way your version acts as my control. Thanks!

Thank you for all of your hard work on this addon it's working great.

UPDATE 3/20/17:

Things are working so well, I didn't remember to check for a file update till now. And it's still working fine. I suspect you can call this a solid "Release Candidate".

Thx for your work and effort in this.

Cheers....

3/17/17
Just another quick update.....over a week into it and my tvguide.xml file continues to be updated as scheduled. I don't want to jinx things, but I believe you've worked out the "kinks"

Cheers....