tvheadend, ATSC and epg...

    Quote from StitchedChin


    Yeah, I swapped between those options just to see what they would do, but it just changed those entries from 0 KGTVDT to KGTVDT 0. I had it setup originally how you suggested.

    I've done a clean test and the code is pulling channel numbers...so something is not quite right up on your setup.

    I would suggest deleting the zap2xml cache folder under userdata and rerunning.

    As a test of the code - you can unzip the current version in windows. Open a command prompt from that folder and run the code by entering:

    Code
    python zap2xml.py -u username -p password

    replacing username and password with your account

    This will run the code with basic options and you should quickly get an xmltv.xml file you can check. That should have the channel numbers in it.

    Edited once, last by edit4ever (November 2, 2016 at 6:31 PM).

    Haha, wow, that was a painful learning experience. I installed python at work on a Windows machine, at least I can say a learned a bit more about python. Compatibility between python 2 and 3 sure does break a lot of things. edit4ever is definitely a fitting title for a programmer. Once I finally got the xmltv file to generate, the channel numbers did show up here, so I'll do a fresh install at home on the odroid. I have a feeling I didn't delete the zap2xml script folders before I installed 0.4.6. Usually something I like to do, but not a habit. I think the cache folder is hidden when browsing from Windows explorer, too, so I may had missed what had built up in there. I'll check that out before I delete everything. Thanks for the help and patience!

    Oh, and some titles were still coming back with episode # only from last night's recordings, but much less than before. I'll see if I have a guess on why when I take a closer look.

    Quote from StitchedChin


    I think the cache folder is hidden when browsing from Windows explorer, too, so I may had missed what had built up in there. I'll check that out before I delete everything.

    You should only have to delete the cache folder and then rerun the grabber...so start there before deleting everything.

    Cache folder is under the .kodi/addons/script.module.zap2xml folder - so yes it is hidden. However, if you're going to play around with addons, you can see that folder in windows by adding the following to your samba.conf file (which is under the Configfiles folder)

    Code
    [kodi]
  path = /storage/.kodi
  available = yes
  browsable = yes
  public = yes
  writable = yes
  root preexec = mkdir -p /storage/.kodi

    If you only have the samba.conf.sample file - just remove the .sample part and edit the file adding the above code to the bottom. After a reboot, you should see the kodi folder from windows. Of course, be careful when working in this folder - but it makes it easy to work on addons! :)

    That worked, channels are back to normal, thanks! I'll have to keep learning python to see what those files are doing. I made the samba file adjustment, too, definitely much easier to double check the addons and installation. Thanks again!

    I have a new beta release which adds a new option to add and sort extra details in the description field. This is useful if the skin you are using doesn't display all the data you want. You can now add/adjust that data in the description (plot) field. While the proper way to display data is editing the skin - not everyone will be comfortable with that. So here's an alternative option for you!

    Currently I have the following elements that can be added/sorted:

    None / BULLET / HYPHEN / Plot Description / New-Live Indicator / HD Indicator / CC Indicator / Season-Episode Number / TV Rating / Original Air Date / Program Title / Episode Title / Episode Title in Quotes / Cast / Movie Release Year / LINE BREAK

    If there are other suggestions for fields or characters, please let me know.

    Release: script.module.zap2xml-0.5.0.zip

    Source: GitHub - edit4ever/script.module.zap2xml at xdetails

    Also let me know if you see any issues!

    I've been following this very closely and really appreciate edit4ever's effort to get the epg grabber working. I'm using an RPi3 running LibreELEC (Krypton) v7.90.008 ALPHA with TvH 4.1.2236 and a HDHR Connect tuner. I had issues with the update to Screenertv.com with the season and episode numbers but the 0.5.0 update fixed it. I'm curious how the additional elements affect the EPG in TvH. Can you explain a little what these are for and how I would use them?

    Thanks again for the effort!

    Quote from cloudfire


    I'm curious how the additional elements affect the EPG in TvH. Can you explain a little what these are for and how I would use them?

    Basically - depending on what skin you are using to display your epg...you may want to see additional info but the skin doesn't have the option to show it. For example, the Confluence skin doesn't show the original air date of an episode in the epg screen. You have to go into the epsiode info screen to see that.

    If you want to see the original airdate in the normal epg screen, you can use the "append xtra details to the program description" option. Then on the "xtra details order" tab, you can add the plot and original air dates to the list. You can even add a bullet or hyphen between them. Or add a line break between them and the original air date will appear below the plot description.

    All of the items that you add to the "xtra details order" tab will be written in the plot description field in the order you select. This modifies the original plot description when writing the xmltv.xml file that tvheadend reads. Then this info will be displayed in your skin anywhere the plot description is normally seen.

    You can always turn it on, try a few items to see how they look, adjust or turn it off if you don't like it!

    Please keep in mind that depending on how you have your guide data refresh setup in kodi, you may need to reboot or even clear guide data in order to read in the new info. Also, once you make the changes to the xtra details list, you need to rerun the grabber (which you can trigger in tvheadend) to get the updated xmltv.xml file to write out.

    If that was confusing - please feel free ask any questions!!

    P.S. I (and Schmell) are working on an issue with Spanish language (and other) characters. I think I have a fix, but can't test it until I return home this weekend.

    Edited once, last by edit4ever (November 9, 2016 at 12:02 AM).

    Thanks for the info. This explains a lot. I don't have my RPi3 hooked to a TV, only to my router. I access the TvH through the webpage to schedule recordings and make changes. Basically I have it setup as a network DVR that I pull the recordings from and watch elsewhere.

    [SOLVED] Looks like it's been a while since any postings here....

    1st off, thanks for the great work!! At least, until it stopped working :(

    I'm on OSMC/RPI3/TVH/zap2xml. I finally got it up and running and it was working fine...until a most recent system (OSMC or unbuntu) update (~11/26/16). Now, nothing shows up in the TV Shows/Guide window and my tvguide.xml (changed that from the default) file hasn't been updated since 11/26/16 (I discovered this ~1 week later).

    In the zap2xml.log file I get the following:

    Traceback (most recent call last):
    File "/home/osmc/.kodi/addons/script.module.zap2xml/zap2xml.py", line 92, in <module>
    import mechanize
    ImportError: No module named mechanize

    I changed the path in the 2nd line because there is no "storage" directory on my machine.

    I manually ran tv_grab_zap2xml and it seems to go out and bring a lot of data in. However, my tvguide.xml still does not get updated.

    Any suggestions how to solve this?

    Thanks and cheers.....

    Edited once, last by stepher (December 17, 2016 at 6:40 PM).

    Appreciate the reply.

    I've reinstalled twice (uninstalled zap2xml, then mechanize) and rebooted, as well. Keep in mind it was working before an OSMC update, so I'm not sure what was broken by the update.

    I looked in ~/.kodi/addons and there is a script.module.mechanize folder. Underneath that is the lib/mechanize path. In mechanize there are a number of .py files. So, at 'import mechanize' is python looking for a specific file titled "mechanize" or does python understand the folder "mechanize" contains the files? Also in zap2xml.py, I changed the line "mechLib = 'script.module.mechanize/lib" to "mechLib = 'script.module.mechanize/lib/mechanize/'" to get directly to the .py routines folder. No help.

    It really feels like a "pathing" issue but I don't know enuf to be sure. Actually, I took a look at the 'import' command and it tells python to look for a module. Not sure if a folder/directory constitutes a module. So, maybe I'm actually missing a piece.

    I'm open to more ideas/suggestions....

    Quote from edit4ever

    I would suggest you uninstall and reinstall the add on. Looks like mechanize is missing. A reinstall should add that dependency automatically.

    I assume you are running version .5.0?

    Edited once, last by stepher (December 14, 2016 at 10:57 PM).

    Quote from stepher


    It really feels like a "pathing" issue but I don't know enuf to be sure. Actually, I took a look at the 'import' command and it tells python to look for a module. Not sure if a folder/directory constitutes a module. So, maybe I'm actually missing a piece.

    I'm open to more ideas/suggestions....

    I'm actually thinking the issue is that kodi path - as you are using OSMC. I'm only familiar with LibreELEC and OpenELEC. You mention that you have a ~/.kodi/addons folder. The addon references the kodi path as /storage/.kodi/addons/

    Perhaps try changing the kodi path to /home/osmc/.kodi/addons/ in the zap2xml.py file??

    Sorry. Should have mentioned that I saw the /storage/.kodi/addons and did change that. Also am running .5.0.

    Keep in mind, it was up and running for a few weeks without any problems. I feel like I've been thru all this before, tho can't remember the specific things I did (which I'm sure included some, or all, of what we've discussed here).

    I'm going back to do some more experimenting and see if some different happens. Otherwise, I'll have to try a new scraper.

    Let me know if you think of anything else. Thanks.

    Quote from edit4ever

    I'm actually thinking the issue is that kodi path - as you are using OSMC. I'm only familiar with LibreELEC and OpenELEC. You mention that you have a ~/.kodi/addons folder. The addon references the kodi path as /storage/.kodi/addons/

    Perhaps try changing the kodi path to /home/osmc/.kodi/addons/ in the zap2xml.py file??


    [hr]
    Just realized you never answered my question about what file specifically "import mechanize" is trying to load. Or is that not a file, but just a path to a collection of "include" files from the lib/mechanize folder. I went back and looked in the lib/mechanize folder and saw a _mechanize.py file (among a bunch of others). Is that the specific file? If so, then the name is off.

    Also, I don't really remember messing with this part of the zap2xml.py file to get it to work the 1st time.

    Quote from stepher


    Sorry. Should have mentioned that I saw the /storage/.kodi/addons and did change that. Also am running .5.0.

    Keep in mind, it was up and running for a few weeks without any problems. I feel like I've been thru all this before, tho can't remember the specific things I did (which I'm sure included some, or all, of what we've discussed here).

    I'm going back to do some more experimenting and see if some different happens. Otherwise, I'll have to try a new scraper.

    Let me know if you think of anything else. Thanks.

    Edited once, last by stepher (December 15, 2016 at 12:59 AM).

    OK. Still haven't solved the problem. It looks like multiple issues but may turn out to be only one.

    1. I can see tv_grab_zap2xml retrieving data, but it looks like the data is outdated (Nothing past 12/9/16) which is the latest date of any data in my tvguide.xml file so that may be the reason nothing is getting displayed. It's all in the past.
    2. Even tho tv_grab_zap2xml is capturing the info, it is still not updating my tvguide.xml file (modification date has not changed).
    3. "import mechanize" is still casting an error, and this may be the whole enchilada if I can figure out why this is happening.

    As an experiment, in zap2xml.py I trimmed out everything past the "import mechanize" command and ran tv_grab_zap2xml. Needless to say that no data was retrieved, but the same error about "import mechanize" still showed up. I read up on the python import command and realized that the folder "mechanize" becomes a source of include files (module) when a "__init__.py" file is included in the folder. I confirmed that the file does indeed exist.

    Then I tried a simple experiment. I did a basic "cd" to the ......script.module.mechanize/lib/mechanize directory and it went right to it. I know this isn't quite the same as accessing mechanize as a module thru a python program, but I wanted to make sure basic pathing wasn't an issue.

    I checked permissions and ownership and for the files in lib/mechanize all files are osmc:osmc, -rw-r--r--. In script.module.zap2xml the bin folder shows osmc:osmc, drwxr-xr-x, zap2xml.log osmc:video, -rw-rw-rw and teh xzap2xml is osmc:osmc, -rw-r--r--r (which seems consistent for .py files). My tvguide.xml file is osmc:osmc, -rwx-r-xr-x. I'm not completely sure but could this be the update issue? The user and group are not given write permissions. I'll go check this last part out.

    So, does anything I've shared here seem off? While this might seem like a pathing issue, it still may be but not for reasons I may understand. My guess is something is happening (or not) which may not allow the mechanize folder to be seen as a module from zap2xml.py code.

    If I can't get this fixed in the next few days, I'll have to change to something I can get to work (I really liked your addon, tho :(

    Any help or suggestions would be greatly appreciated. Thanks and cheers.....

    What is the path to your addons? for example - on my system it is: /storage/.kodi/addons/

    I'm guessing it is: /home/osmc/.kodi/addons

    If that is so - try adding the following before the import mechanize line:

    sys.path.append('/home/osmc/.kodi/addons/script.module.mechanize')

    Or try chaning the kodiPath = '/storage/.kodi/addons/' line to: kodiPath = '/home/osmc/.kodi/addons/'

    The addon was originally written for OE/LE installations and hasn't been optimized for other installs. Weird that it worked for you before then stopped working after an update.

    Edited once, last by edit4ever (December 15, 2016 at 5:31 PM).

    I may still try your suggestions. If it really wasn't supposed to work on my system, any recommendations on what I should be using?

    Had some time to kill :( so I decided to go back to the beginning, so to speak, in case one of my files got corrupted somehow :(

    Here's where I've gotten to so far: I uninstalled both zap2xml and mechanize then made sure any "remnant" directories or xml files were deleted from the system. Did a system reboot. Reinstalled zap2xml (and, thus, mechanize). Edited all the references to be consistent with my system. Did another reboot and reset TVH to the zap2xml scraper.

    Once I saved the scraper choice (again) in tvh, it then went off and gathered the data. Still nothing in my tvguide.xml file, but the "import mechanize" error went away...one step closer. What I am now getting in the ~/.kodi/userdata/addon_data/script.module.zap2xml/zap2xml.log file:

    ...snip....
    Getting: gridDetailService?pgmId=EP012252460038
    [D] Parsing: EP012252460038
    Getting: tvschedule.zap2

    These were the last 3 lines in the file (and, yes, the last line is all that was written into the log file). It's like the grabber just stopped mid-process (no errors, tho). I even reduced the no. of days to 1 to minimize the churning time.

    This kind of explains why my tvguide.xml file is empty, but doesn't give me any clue why it just stopped like that.

    In the other zap2xml.log file located in ~/.kodi/addons/script.module.zap2xml were these last 3 lines (all the previous ones were similar but with different pgmId nos.):

    ...snip....
    2016-12-15 17:36:56,107 Getting: gridDetailService?pgmId=SH003304050000
    2016-12-15 17:36:56,239 [D] Parsing: SH003304050000
    2016-12-15 17:36:56,241 Getting: gridDetailService?rtype=pgmimg&pgmId=SH003304050000

    Anything catch your attention?

    Thanks and cheers....

    Quote from stepher


    ...snip....
    2016-12-15 17:36:56,107 Getting: gridDetailService?pgmId=SH003304050000
    2016-12-15 17:36:56,239 [D] Parsing: SH003304050000
    2016-12-15 17:36:56,241 Getting: gridDetailService?rtype=pgmimg&pgmId=SH003304050000

    Anything catch your attention?

    That is strange as it looks like it stopped mid download. Usually this is a timeout issue - but there should have been an indication of a server timeout error.

    Try turning off the Download Extra Program Details option for now. That will stop the -D option which downloads and parses the extra show information. That way we can see if it will just pull a standard listing.

    Fixed it (finally), I think!!! But not quite sure why (more familiar with Fedora than with Debian). Here's how I got (back) here:

    1. tvguide.xml not being written to with guide info - I tend to believe this was the heart of the problem....but was thinking it was a path issue. May have been, but not in the way I thought (or understood :( Frustratingly curious by this point, I deleted tvguide.xml and then ran tv_grab_zap2xml. Results were tvguide.xml not found. Never saw this error before so apparently tvguide.xml was being found, but not necessarily the one (or in the way) I believed it was. Created a new tvguide.xml, ran tv_grab_zap2xml again and "File not found" error went away again...still no info written to the file, tho.

    2. Import mechanize error - For this I went back to look (yet, again) at zap2xml.py. When this problem started and I was thinking pathing issue, I changed the kodiPath variable to an absolute reference (/home/osmc/....) and deleted the /storage directory reference because that doesn't exist on my system. Still, running tv_grab_zap2xml produced the "import mechanize" error. Decided it was time to start fresh. I extracted a clean copy of zap2xml.py from the .zip file. Again made the changes to kodiPath. This time, tho, I used Notepad++ set for the python language to edit and save. Restarted tvheadend and watched the HDHomeRun tuner LEDs flash.

    I have Samba enabled on my RPI 3/OSMC so I could check file properties from my Win 10 system. Since I knew tvguide.xml was new and was 0 bytes, I had a reference point. Giving tvheadend a bit of time to do its stuff, I checked properties of tvguide.xml. Lo, and behold, it showed ~2MB and a very recent modified date/time. Went downstairs and checked Live TV/Guide and everything was back (Yayyyy!). I changed days to 14 and checked it again this morning (update set for 3AM) and the program guide had info out to 12/30/16.

    The only thing I can think of is the recent system update glitched something to do with accessing *my* tvguide.xml file. Apparently the guide data was getting written somewhere (no obvious error notices), just not into my file. The "import mechanize" error is on me and just complicated things (I got other odd errors along the way from executing this program). I had been editing the zap2xml.py file with a different editor and that may have introduced some odd control/invisible chars causing problems for Python. Using Notepad++ in Python mode seemed to work just fine.

    When I get the chance I'll go sniff around the Debian/OSMC logs and see if there are any error outputs that might relate to this problem.

    So, thanks for the assistance...at least, until the next time ;)

    Edited once, last by stepher (December 17, 2016 at 6:45 PM).