Posts by stepher

    Been a while, but thought I'd check in once more just to confirm with you.....

    My tvguide.xml file has been updating without any issues. My guess is this is a releasable version (assuming you haven't already done so).

    Great work and appreciate the effort.

    Thx again and cheers....


    UPDATE 3/20/17:

    Things are working so well, I didn't remember to check for a file update till now. And it's still working fine. I suspect you can call this a solid "Release Candidate".

    Thx for your work and effort in this.

    Cheers....

    3/17/17
    Just another quick update.....over a week into it and my tvguide.xml file continues to be updated as scheduled. I don't want to jinx things, but I believe you've worked out the "kinks" :)

    Cheers....

    Good timing...I had unzipped the file and was just about to update the .py file and install.

    "Use me and abuse me"...as your control control :)

    Safe travels....


    This is the same as the version you're running since you copied the zap2xml.py file over manually. Someone just asked for the version that could be installed from the addon menu. You don't have a need to update. Actually - I prefer if you don't update, in case someone finds an issue with the zipped file. I'm travelling and every now and then I mess up pulling in the updated files! ;) This way your version acts as my control. Thanks!

    What's different in 0.6.5? Can I just replace the zap2xml.py file (with obvious needed pathing changes) or do I need to do an install from within OSMC?


    Someone asked for the zipped addon....I'm travelling - so I haven't been able to run any tests...but since the feedback from my code fixes seems to be fine - you can try this:


    Please let me know if you have issues. Thanks!

    3/5/17 Quick Update:

    My tvguide.xml file still being updated as set in the schedule. So things are still good.

    What I noticed this morning in the xap2xml.log file was the following:

    ....snip....
    [D] Parsing: EP024717260046
    Getting: gridDetailService?pgmId=EP024717260035
    :Function: getURL :Line: 225
    error<class 'mechanize._response.httperror_seek_wrapper'>
    Getting: gridDetailService?pgmId=EP024717260035
    :Function: getURL :Line: 225
    error<class 'mechanize._response.httperror_seek_wrapper'>
    Getting: gridDetailService?pgmId=EP024717260035
    :Function: getURL :Line: 225
    error<class 'mechanize._response.httperror_seek_wrapper'>
    Failed to download within 3 retries.
    :Function: getURL :Line: 228
    error<class 'mechanize._response.httperror_seek_wrapper'>
    [D] Failed to download: EP024717260035
    [D] Parsing: SH000046300000
    ....snip....

    I assume this is what I should see when a specific update cannot be downloaded. Saw this same logging of repeated attempts 1 or 2 times more in the file. The final lines of the file:

    ....snip....
    Downloaded 1972995 bytes in 521 http requests.

    Writing XML file: /home/osmc/.xmltv/tvguide.xml

    Completed in 411s (Parse: 198s) 79 stations, 9693 programs, 34851 scheduled.

    which seems to indicate that even tho there were a few errors, the tvguide.xml file was still updated...which I believe is what we were looking for.

    Cheers....

    ------------------------------------------------------------------------------------------

    2 days in...so far so good :) I'll post status again at the end of a week. Now I can go focus on a few other aspects of the PVR. I haven't yet cut the cord, but I'm getting very close...maybe another month, possibly 2, at most.

    Funny that you come from a non-technical space. I, too, am a former marketing exec, but in the high tech domain (Silicon Valley). I also have an engineering degree, and while it was focused on the h/w side, I've done quite a bit of dabbling in s/w...a little bit of hi level here and there (still threatening to learn Python), but mostly at the CPU level ("diddling bits and bytes", as they say ;)

    I'm glad and very appreciative you took this on. I feel zap2xml is turning into a really nice addon. Maybe, just maybe, I'll actually get Python learned and I can look at contributing new features, as well.

    Thx and cheers....


    Thanks again for the update. This is all totally a hobby. I didn't know the first thing about python and other code until I started playing with this and modifying Kodi skins.

    As for the different logs - I'm not sure how or why that happens. This project was orginally done by fasteddycurrent - which you can see here: GitHub - FastEddyCurrent/zap2xml: zap2xml in Python 2.7 for use on the RaspberryPI - it was he who converted the the original zap2xml perl script into python.

    I basically took what he did and created a kodi addon and then started adding features I wanted to see. And now here we are. I'm just a former tv marketing exec who likes to play. :) I'm driven by the goal of wanting cheaper/better dvr service and this year I finally cut the cord. If i get around to it, I'll have to look at how the log system is structured.

    BTW - the "Skipped:" entry means the code fix worked. That indicates that there was an empty or missing details file that would have previously crashed the grabber. Glad to see it completed!

    OK. I updated zap2xml.py and ran (restarted) tvheadend. The tvguide.xml file updated just fine. I'll give it at least a few more "normal" updates just to be sure. WRT update frequency....I get the reduction in network activity by doing multiple updates on a daily basis. Something else I can experiment with once I'm sure the system is stable.

    In the log file in .kodi/userdata/add_on/script.module.zap2xml none of the entries have any time stamps (and there was 1 "Skipped:" entry). However, in the log file in .kodi/addons/script.module.zap2xml the date/time stamps are there, except at the beginning of the line, not the end. Is that by design?

    Are you a technical consultant so this PVR stuff is an easy distraction for you? Or are you non-technical and this is truly a hobby of love?

    Cheers...


    Thanks for the update! Version 4 should work for both details and icons - so feel free to test.

    Fortunately my wife and I run our own consulting business - so I get to have lots of time to work on hobbies like this! :)

    No on lines 865-873 looking like you what you've listed below. I had downloaded the file from the link you provided in msg. #197 (zap2xml.zip). That's water under the bridge, tho.

    Late yesterday I had the chance to download and install zap2xml (2).zip. Did a tvheadend restart and, voile', updated tvguide.xml file. Had to do a system reboot for some reason and my tvguide.xml was again updated as tvheadend restarted. Just checked again this morning and system is still updating as expected.

    So, looks like things are working again. Thx.

    BTW - Before all of the above, I disabled Download Program Icons because I'm not sure I really care about those icons, only the station icons. However, I may revisit that sometime down the road.

    wrt contents of the log file...to clarify...I am getting the Parsing statements (see below for most recent log file contents). What seemed to disappear was a statement before *each* parsing line about "Getting: tvschedule.......". That's been gone for a couple of iterations of the code at this point. As a side note, my parsing channel ids (?) have varying start letters: EP, SH and MV (not sure if that's significant at all).

    What I am seeing is (what appears to be random, but may not be) are lines like this:

    ....snip....
    [D] Parsing: SH019264110000
    [54/56] Parsing: /home/osmc/.kodi/addons/script.module.zap2xml/cache/1489327200000.html.gz
    Getting: TV Listings Grid, TV Guide and TV Schedule, Where to Watch TV Shows - Screener
    [D] Parsing: SH016159020000
    ....snip....

    ...AND.....

    ....snip....
    [D] Parsing: SH012378150000
    [D] Parsing: SH010452440/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py:617: UserWarning: gzip transfer encoding is experimental!
    br.set_handle_gzip(True)
    000
    [D] Parsing: SH016354690000
    ....snip....

    I haven't looked closely thru the whole file, but a fast scan shows these kinds of lines spread throughout the file (but not a lot of them).

    I'll download ver. 4 and let you know the results. It seems like things are getting fairly close. I really appreciate your effort in working thru this (don't you have a wife/husband or kids or job or a life to look after (big grin ;)

    Cheers and thx....


    If you refer back to a recent previous posting, I mentioned that I inadvertently enabled Download Program Icons (when I intended it to remain disabled when I tested xtra program details being disabled, as well) and I had a successful tvguide.xml update. So, I don't believe my problem is with the Program Icons.

    I just went thru 2 tests (1 tvheadend restart and 1 system reboot) with Program Icons disabled and xtra program details enabled and got the same (well, almost in that it made it further than last time I posted) error cast. Here's the error part of the log results:



    Oops - I did the fix the code in the icon section. That's where it failed for you. Turn off program icons for now and when I get back to my system I will fix that section.

    Sorry about that!

    OK. I updated with the new zap2xml.py file, enabled xtra program details and restarted TVH. Sorry for the bad news but actually seems worse than before. Also noticed no msgs about clearing the cache at the beginning of the log file. Here's the log error:

    And here I am :)

    (Should have been "disabled :( <Re-enabled> download xtra program details and download was successful and complete (no errors cast) this morning. I had planned to separately test both xtra program details and download program icons. However, I'm fairly certain I inadvertently left program icons enabled last night. So it's also been confirmed OK anyways. Looks like xtra program details is the culprit (for me).

    Before you get too concerned, tho, I'm not sure if I have the latest code you've produced. I updated my zap2xml.py file using the code you posted in msg #186. Do I need to update zap2zml.py using the file you reference in msg #197 and retest?

    -----------------------------------------------------------------------

    BTW - I'm curious if there a good reason to update 2x (or more) a day. Maybe to ensure a valid download is completed (altho, if an error is cast on the 2nd/later download, doesn't do much good)?

    Thx and cheers....

    Re-enabled xtra program details last night and let the system go thru normal updating (@3:30 AM). Bad news...no update....good news....no update and here's the section of the log file (under .kodi/userdata/addon_data/script.module.zap2xml) where it cast the error. I also noticed in the 1st part of the log file the code now deletes the old cached files ("Deleting old cached file: ....."). Is that new or something I just haven't noticed until now?

    Thx and cheers....



    Well - my system ran correctly again last night with extra details...so it looks like I'm going to have to wait for another error to appear so i can troubleshoot. I want to narrow down whether the issue is with missing data or changed data. BUt until I have another error - I don't have a place to look.

    If yours fails again - please provide the log so I can look at the point of failure.

    Thanks!

    My did, as well (and has been running correctly). I'll have to set the xtra program details and download program icons options (separately, of course) and see what the results are. I'll post the log when I get the (bad) results.

    Thx and cheers.....


    Well - my system ran correctly again last night with extra details...so it looks like I'm going to have to wait for another error to appear so i can troubleshoot. I want to narrow down whether the issue is with missing data or changed data. BUt until I have another error - I don't have a place to look.

    If yours fails again - please provide the log so I can look at the point of failure.

    Thanks!

    I'm glad you've been able to reproduce the problem. Not glad you have the same issue :( but nice to know it's not unique to my system :)

    I thought I noticed that when I didn't do anything, the system went thru a non-update hiccup, so I wonder if it's at the website that it's some kind of protocol glitch that shows up every once-in-a-while when it comes to the xtra program details. My emphasis is on "I didn't do anything" which doesn't mean the RPI 3/OSMC didn't do something when I wasn't looking.

    Something else I've noticed (and this is not really germane to our discussion here) is that when my system locks up and I have to do a hard reset (need to code and install a reset button), some of my settings (I'm using the new Confluence skin) are reset to default (like weather on the home screen, hiding "Pictures" on the home screen, etc.). Need to experiment with that once I've stabilized my RPI 3 P/S.

    I can be patient and I appreciate whatever time and effort you are putting into this. Thx!! And I'm happy to continue as your "beta site" :)

    Cheers....


    Looks like my system has now succumb to the extra details json error. The earlier code change did not help. I'm not sure if this is an issue with an LE update or a change to the way screener/zap2it is hosting the data. I'll have to do some testing - but may not get to this for a bit. At least we know it's not limited to a single system or user issue.

    In the meantime, you can turn off the extra details and it will load the basic guide.

    2/21/17 Quick Update:

    Had a chance to check my tvguide.xml file and looks like it updated normally early this morning. So, disabling the 2 options I mention in yesterday's update seems to solve my problem.

    2/20/17 (later) Update:

    Had a chance to disable xtra program details (and Download Program Icons option at the same time) and restarted TVH. Back to working just fine.

    Sorry to say your recommended code changes to the zap2xml.py file did not help *me* at all.

    2/20/17 Update:

    I enabled "Xtra Program Details" yesterday and this morning found that my tvguide.xml file was NOT updated. I'm going to disable this option and see what happens.

    2/18/17 Update:

    Noticed this morning that my tvguide.xml file was last updated on Thurs., 2/16/17. I had not done much of anything to the system (except watch some movies/tv shows and deleted them after I watched them).

    However, a couple of times I had to do a hard reset when the system locked up trying to fast forward on a program. I noticed when that happened, the settings I had made (i.e. xtra program details, etc.) were set back to default after the reset(s). This may have been a source of the issue.

    Before doing anything else, I decided to update my zap2xml.py file with your new Python code suggestions below. I then restarted TVH and my tvguide.xml file was updated. I did this thru ssh and did not make any changes to the RPI 3/HTPC system.

    So, back to tvguide.xml update watch :)

    BTW - I believe the cause for the lock ups may be due to recently developed power supply issues (I occasionally get the "lightning bolt" flashing in the upper right corner of the screen :( I'll have to install a better power supply.

    Plan to test the different options with the new Python code. Will keep you posted on how things are going. Thx for your continued support.

    Cheers....


    2/15/17 - Quick update

    My tvguide.xml file continues to be updated as programmed. I have not had the chance to experiment (hopefully in the next few days) so I'm not sure at the moment of what's enabled/disabled in the add-on.

    Let you know when I have a chance to look at things again. I may also incorporate your recently suggested changes to the zap2xml.py file and see what happens.

    Cheers.....

    2/15/17 - Quick update

    My tvguide.xml file continues to be updated as programmed. I have not had the chance to experiment (hopefully in the next few days) so I'm not sure at the moment of what's enabled/disabled in the add-on.

    Let you know when I have a chance to look at things again. I may also incorporate your recently suggested changes to the zap2xml.py file and see what happens.

    Cheers.....


    ok - let's try something.

    If you can edit the zap2xml.py file - look at the following section starting at line 341:

    Change it to the following:

    I'm thinking if we only parse if the file exists - that may solve the error. The change is for both the program details and the program icon downloads.

    Let me know if you are able to test.

    I wonder if some of the problem is an occasional "glitch" when downloading the website the program guide data.

    My plan (when I get the time) is to experiment with different settings (xtra program details, program/station icon download, etc.) and make sure the problem *isn't/is* (always) on my end.

    Mine has also stopped working after upgrading from 7.0.3 to libreelec 7.95.3. I restarted my rpi2 and tvheadend downloaded the guide just fine after that. Dont know what happened either.

    I did try to look at them using WinZip (from a Win10 sys) and, while I could open a file, it showed as empty. Is there another Win10 based program that allows me to view the files? Or should I be using something on the Debian/OSMC system to do that (like, would nano work)?

    No - the files in the cache folder are just the downloads. They always have their data...They just need to be uncompressed first.

    I'm glad to continue to act as a kind of "beta tester" for you (as long as it doesn't get too out of hand :( If we can figure out my issues, then hopefully everyone else can benefit from it. This is all about DIY, right? :)

    What nips at me is not the fact that it stopped working after I upgraded to OSMC/Kodi 17 (maybe it should but with permissions, etal, it's not a surprise) but rather that it started working again this morning, and I didn't do anything (tho, maybe I changed 1 of the options and just don't remember :( Anyways, like I said, scientific experimentation and see what inputs change the output.

    Let you know when I get useful data....

    BTW - Should the XXXXX.js.gz files be empty once my tvguide.xml file has been updated?

    Thx and cheers.....


    The xxxxxx.html.gz files contain the basic grid in 6 hour chunks. So if you have set the addon to download 1 day, you will have 4 of those files. When you have either the xtra description download or the program icon download enabled - the program parses (scans) those files and pulls the episode information and uses that to download the extra program details (generating the xxxxxxxxx.js.gz file) and the episode program icon (generating the Ixxxxxxxxx.js.gz file). Then all of that info is parsed to generate the final xmltv.xml file.

    Apparently you're having an issue where it wants to download the xtra program detail or program icon and screener says the file doesn't exist. In theory the program should just move on - but apparently there is a piece of code missing. The weird thing is that should never happen. If there is an episode scheduled, it has a number and should have information to download. It's as though screener has the worng episode number in the program schedule for on of the programs in your listing.

    Anyway - we'll keep trying to fix it! (or if you want - you can just get the schedules direct service for $25 for the year. That addon works great and has lots of good data!)