| Both sides previous revision
Previous revision
Next revision
|
Previous revision
|
refresh_on_demand [2012/04/01 00:49]
jhb50 |
refresh_on_demand [2012/08/06 14:06] (current)
jhb50 |
| **HOW TO REFRESH ONLINE FOLDERS ON DEMAND** | **HOW TO REFRESH ONLINE FOLDERS ON DEMAND FROM THE CLIENT** |
| |
| __THE NEED FOR REFRESH ON DEMAND__ | __THE NEED FOR REFRESH ON DEMAND__ |
| |
| The Serviio Console and various Remote Device Console implementations offer the ability to force refresh Online Sources On Demand, | By default, Serviio will automatically refresh the content "Online sources" ("feeds"), with each restart and on a periodic basis according to the "Feed expiry interval (hours)" set in the console. This is adequate for "Online Sources" that are relatively static, but there are some Online sources" ("feeds") that are dynamic, with their "feed items" constantly changing, so the "feed" must be refreshed in order for Serviio to obtain and display the latest set of "feed items". Sports Event sites such as HaHaSport and News sites such as Video.AP are two examples of this. |
| however no capability is offered to initiate forced refreshes on demand from the Folder Menu on the client device. | |
| |
| While most Online Sources are relatively static there are some sources that are constantly changing and it is necessary to refresh in order to obtain the currently available set of feed items. One approach is to create a WebResource groovy for the source and initiate refreshes on periodic basis consistent with the known dynamics of the source. The problem with this approach is the continuing use of resources to maintain the current set of feed items even though they may never be used. The better approach is to force refresh the Folder menu just prior to its use. | There are two ways to initiate those more frequent refreshes. |
| |
| The Online Source HAHASPORT.COM is an example. This site maintains and provides a list of scheduled events by sport, and provides links to embedded player streams of events as they go live. To maintain currency of a Serviio menu, the list of scheduled events must be reviewed at least hourly since new events can be added or removed at any time; live events must be detected minutes prior to their scheduled start, and if not yet started again at their scheduled start and a few minutes thereafter; and then live events must be removed a few minutes after their scheduled end time. This constant stream of refreshes particularly when multiple sport sources are added is beyond the capacity of most systems and refreshes are constantly backlogged due to insufficient time to access the changing feed items. | One approach is to create a WebResource groovy for the feed and set the expiry date to initiate refreshes of that feed on periodic basis consistent with the known dynamics of the source for example just before the start of the next scheduled sports event or every few minutes for a news feed. |
| |
| The HAHASPORT.GROOVY therefore implements the ability initiate On Demand Refreshes from the Online Menu rather than refresh on a scheduled basis, and may be reviewed as an example of the following methodology. | The Online Source ("feed") HAHASPORT.COM and its WebResource groovy is an example. This site maintains and provides a list of scheduled events by sport, and provides links to embedded player streams of events as they go live ("feed items"). To maintain currency of a Serviio menu, the list of scheduled events must be reviewed by the groovy at least hourly since new events ("feed items") can be added or removed at any time; live events ("feed items") must be detected minutes prior to their scheduled start, and if not yet started, again at their scheduled start and a few minutes thereafter; and then live events ("feed items") must be removed a few minutes after their scheduled end time. |
| |
| __ON DEMAND METHODOLOGY__ | The problem with this approach is the continual use of resources by the groovy to refresh and maintain the current set of feed items even though they may never be used. Secondly, this constant stream of refreshes particularly when multiple sport feeds are enabled, is beyond the capacity of most systems and feed refreshes are constantly backlogged due to insufficient time to access the changing feed items. |
| |
| The prerequisite is the development of a WebResource groovy for the online source and the identification of a permanently available online source link that may be used to provide a Refresh Menu Item stream. | The better approach is to be able to "Force Refresh" a feed just prior to its use. However the problem with this approach is that if a significant number of new feed items have been created since the previous refresh, it may take Serviio many minutes to refresh and make the feed available again, since each new item can take 30 seconds or more to add. Another problem is that while the Serviio Server Console and various Remote Device Console implementations offer the ability to "Force Refresh" Online Sources On Demand, no capability is offered to initiate a "Force Refresh" on demand from the menus on a Client device. so users must access the server or obtain a device which has 3rd party remote console support. |
| |
| The WebResource groovy should creates the Refresh item as the first menu item, using an appropriate thumbnail and title and the Refresh Menu item stream. | The ultimate solution therefore is to implement these dynamic feeds with a Web Resource that refreshes on a periodic basis in order to minimize the number of new feed items that will need to be added by a "Force Refresh", and to provide a "Force Refresh" capability from the menus on the client device, that may be used just prior to using the dynamic feed. |
| |
| The WebResource groovy must also generate a value for "expires on" for all items equal to the current time plus 30 days to prevent any automatic Serviio Refreshes from occuring. | The facility to initiate periodic refreshes in a Web Resource plugin groovy, more frequently than the console default refresh value, is documented in the "Serviio Online Resource plugin implementation Guide" under "Date expiresOn". Date is based on the current time {long curTimea = System.currentTimeMillis()} plus the milliseconds to the desired expiry time. Note that expiry will be on the earliest of that Date and the console default, and the refresh will actually be performed by Serviio 5 minutes ahead of that time. |
| |
| The ServiioService.exe.vmoptions file also needs to be edited to cause ffmpeg.exe transcoding requests to be sent to a script which first checks for the link to the Refresh Menu item stream and initiates an Online Library Refresh of the appropriate library id using CURL before calling ffmpeg.exe to transcode the requested stream. | The hahasports.groovy implements this option via a refresh=nnn parameter that removes the automatic refresh based on event times and only refreshes the feed every nnn minutes. Other dynamic feeds should implement a similar capability. A "Force Refresh" from the server or a remote console, or from a client device menu may then be used to refresh the feeds prior to opening them. |
| |
| __THE HAHASPORT IMPLEMENTATION__ | The facility to initiate "Force Refresh" from an online menu on a client device consists of a WebResource Refresh.groovy that generates a menu of "Force Refresh" feed items for all enabled WebResource feeds in the Serviio online library. Clicking on a listed feed item initiates a "Force Refresh" of that feed by intercepting the call to play the video associated with that menu item. The feed is then removed from the list of Online menu feeds while it refreshes and it becomes available again by opening the Online menu once the refresh is complete. The implementation is documented below. |
| |
| The HAHASPORT.GROOVY uses the c- value designating the sport from the ResourceUrl to request different Refresh Menu item streams for each sport. Similarly the script checks for each of these streams to determine the online library id to refresh. | __ON DEMAND METHODOLOGY__ |
| |
| The HAHASPORT.GROOVY uses "%%https://sites.google.com/site/serviiorss/refresh.jpg%%" as the thumbnail, "Click to Refresh" as the videotitle and "%%rtsp://a1709.l1856953708.c18569.g.lm.akamaistream.net:554/D/1709/18569/vn/reflector:53708%%" (which is the view from the ISS) as the Refresh Menu item stream where the n in vn corresponds to the c- value in the ResourceUrl. | The ability to implement "Force Refresh" feed items in the Refresh.groovy is based on 6 prerequisite one time implementations: |
| |
| With these changes clicking on the Refresh menu item, initiates a refresh of the corresponding online source while the Refresh Menu Item stream plays. If one waits for at least a minute for the refresh to execute before stopping playback, the refreshed menu will be displayed on return and the desired item selected for playback, else the menu will be unavailable on return and Serviio will exit and require a restart to access the updated menu once available. | 1) The installation of CURL.EXE which provides the abilty to access the Serviio repository data and initiate forced refreshes per the latest "Serviio REST interface specification.pdf". The curl.exe may be obtained from http://curl.haxx.se/dlwiz/?type=bin and use the wizard to select the correct package for your operating system and install it in the appropriate system library. I used the generic Win64-ssl package and copied curl.exe into C:\windows\sysWOW64. W32 users should get the generic Win32 package and copy curl.exe into C:\windows\system32 |
| |
| The HAHASPORT.GROOVY is dependent upon the attached ffmpeg.bat script with the appropriate online library id's added as determined by using CURL to dump the repository as documented in the script, and the script is similarly dependent upon the change to ServiioService.exe.vmoptions as documented in the script. (Note I am sure more advanced scripts can be created by those conversant with other scripting tools than bats) | 2) The availability of a live Stream URL which can be used as the trigger to initiate calls to curl.exe to initiate individual library refresh requests. The live Stream URL for the International Space Station (ISS) feed %%"rtsp://a1709.l1856953708.c18569.g.lm.akamaistream.net:554/D/1709/18569/vnnn/reflector:53708"%% is currently used by RefreshWebResources.groovy as the live stream associated with each "Force Refresh" feed item, where the "v Value" nnn is the online library id to be refreshed. |
| |
| After listing the available live streams, the HAHASPORT.GROOVY also generates a final item in each menu, showing either the scheduled start time of the next event or the absence of any more events in the schedule for that sport. | 3) The availability of a "Force Refresh" icon to be used by RefreshWebResources.groovy for each "Force Refresh" "feed item" associated with the ISS live stream. The icon %%https://sites.google.com/site/serviiorss/refresh.jpg%% is currently used which displays "CLICK TO REFRESH FOLDER" and the "feed item" title is used to display any additional information. |
| | |
| __USING THE HAHASPORT GROOVY__ | |
| | |
| To use the HAHASPORT.GROOVY {{:hahasport.groovy.zip}} just add the following Resource URL's as WebResources in the Serviio Console, together with the appropriate labels. | |
| | |
| %%http://www.hahasport.com/c-1.html%% FOOTBALL | |
| %%http://www.hahasport.com/c-3.html%% HOCKEY | |
| %%http://www.hahasport.com/c-4.html%% TENNIS | |
| %%http://www.hahasport.com/c-6.html%% BASKETBALL | |
| %%http://www.hahasport.com/c-7.html%% BASEBALL | |
| %%http://www.hahasport.com/c-8.html%% GOLF | |
| %%http://www.hahasport.com/c-9.html%% CYCLING | |
| %%http://www.hahasport.com/c-10.html%% MOTORSPORTS | |
| %%http://www.hahasport.com/c-11.html%% OTHER | |
| | |
| Since each event may have multiple streams available each from a different source, the estreams parameter may be used to cause multiple streams to be displayed for each event if available. estreams may equal 1 thru 9 with 9 signifing that all available streams should be displayed, for example %%http://www.hahasport.com/c-1.html?estreams=9%% | |
| | |
| Note also that since HAHASPORT.COM sources its streams from many different sources that there may be streams the groovy does not decode for display in the Serviio menu. For details on the stream decoding set the onlinelibrary log to debug mode. | |
| | |
| To access the current events for a given sport, open the menu and click on the Refresh menu item which will initiate a refresh of the online source while the Refresh Menu Item stream plays. Wait for at least a minute for the refresh to execute before stopping playback and the refreshed menu will be displayed on return and the desired item selected for playback. If playback of the Refresh Menu Item stream is stopped before the refresh is complete the menu will be unavailable on return and Serviio will exit and require a restart to access the updated menu once available. | |
| | |
| __THE SCRIPT__ | |
| | |
| <code> | |
| @echo off | |
| : | |
| : To intercept Serviio FFmpeg Calls install this bat as "C:\Program Files\Serviio\lib\ffmpeg.bat" | |
| : then Edit "C:\Program Files\Serviio\bin\ServiioService.exe.vmoptions" | |
| : and change the Dffmpeg.location variable to | |
| : -Xmx384m | |
| : -Dffmpeg.location="C:\Program Files\Serviio\lib\ffmpeg.bat" | |
| : | |
| : Finally install Curl as required in C:\windows\system32 and use | |
| : Curl -X GET "http://localhost:23423/rest/repository" > C:\temp\repository.txt | |
| : to get the repostitory ids for each c value in http://www.hahasport.com/c-n.html and enter the ids below | |
| : eg: set C1=54 | |
| | |
| set C1= | |
| set C2= | |
| set C3= | |
| set C4= | |
| set C5= | |
| set C6= | |
| set C7= | |
| set C8= | |
| set C9= | |
| set C10= | |
| set C11= | |
| | |
| : The following code will then be executed before passing control back to Serviio's ffmpeg. | |
| | |
| set A=%1 | |
| shift | |
| set B=%1 | |
| shift | |
| set C=%1 | |
| shift | |
| set D=%1 | |
| shift | |
| set E=%1 | |
| shift | |
| set F=%1 | |
| shift | |
| set G=%1 | |
| shift | |
| set H=%1 | |
| shift | |
| set I=%1 | |
| shift | |
| set J=%1 | |
| shift | |
| set K=%1 | |
| shift | |
| set L=%1 | |
| shift | |
| set M=%1 | |
| shift | |
| set N=%1 | |
| shift | |
| set O=%1 | |
| shift | |
| set P=%1 | |
| shift | |
| set Q=%1 | |
| shift | |
| set R=%1 | |
| shift | |
| set S=%1 | |
| shift | |
| set T=%1 | |
| shift | |
| set U=%1 | |
| shift | |
| set V=%1 | |
| shift | |
| set W=%1 | |
| shift | |
| set X=%1 | |
| shift | |
| set Y=%1 | |
| shift | |
| set Z=%1 | |
| shift | |
| set AA=%1 | |
| shift | |
| set BB=%1 | |
| shift | |
| set CC=%1 | |
| shift | |
| set DD=%1 | |
| shift | |
| set EE=%1 | |
| shift | |
| set FF=%1 | |
| shift | |
| set GG=%1 | |
| shift | |
| set HH=%1 | |
| shift | |
| set II=%1 | |
| shift | |
| set JJ=%1 | |
| shift | |
| set KK=%1 | |
| shift | |
| set LL=%1 | |
| shift | |
| set MM=%1 | |
| shift | |
| set NN=%1 | |
| shift | |
| set OO=%1 | |
| shift | |
| set PP=%1 | |
| shift | |
| set QQ=%1 | |
| shift | |
| set RR=%1 | |
| shift | |
| set SS=%1 | |
| shift | |
| set TT=%1 | |
| shift | |
| set UU=%1 | |
| shift | |
| set VV=%1 | |
| shift | |
| set WW=%1 | |
| shift | |
| set XXY=%1 | |
| set CODE=NONE | |
| echo %A% %B% %C% %D% %E% %F% %G% %H% %I% %J% %K% %L% %M% %N% %O% %P% %Q% %R% %S% %T% %U% %V% %W% %X% %Y% %Z% %AA% %BB% %CC% %DD% %EE% %FF% %GG% %HH% %II% %JJ% %KK% %LL% %MM% %NN% %OO% %PP% %QQ% %RR% %SS% %TT% %UU% %VV% %WW% %XX% %YY% %ZZ% >> "C:\Program Files\Serviio\log\ffmpeg.log" | |
| :COM Note Serviio appends ?tcp to rtsp links. | |
| if .%B% == .rtsp://a1709.l1856953708.c18569.g.lm.akamaistream.net:554/D/1709/18569/v1/reflector:53708?tcp set CODE=%C1% | |
| if .%B% == .rtsp://a1709.l1856953708.c18569.g.lm.akamaistream.net:554/D/1709/18569/v2/reflector:53708?tcp set CODE=%C2% | |
| if .%B% == .rtsp://a1709.l1856953708.c18569.g.lm.akamaistream.net:554/D/1709/18569/v3/reflector:53708?tcp set CODE=%C3% | |
| if .%B% == .rtsp://a1709.l1856953708.c18569.g.lm.akamaistream.net:554/D/1709/18569/v4/reflector:53708?tcp set CODE=%C4% | |
| if .%B% == .rtsp://a1709.l1856953708.c18569.g.lm.akamaistream.net:554/D/1709/18569/v6/reflector:53708?tcp set CODE=%C6% | |
| if .%B% == .rtsp://a1709.l1856953708.c18569.g.lm.akamaistream.net:554/D/1709/18569/v7/reflector:53708?tcp set CODE=%C7% | |
| if .%B% == .rtsp://a1709.l1856953708.c18569.g.lm.akamaistream.net:554/D/1709/18569/v8/reflector:53708?tcp set CODE=%C8% | |
| if .%B% == .rtsp://a1709.l1856953708.c18569.g.lm.akamaistream.net:554/D/1709/18569/v9/reflector:53708?tcp set CODE=%C9% | |
| if .%B% == .rtsp://a1709.l1856953708.c18569.g.lm.akamaistream.net:554/D/1709/18569/v10/reflector:53708?tcp set CODE=%C10% | |
| if .%B% == .rtsp://a1709.l1856953708.c18569.g.lm.akamaistream.net:554/D/1709/18569/v11/reflector:53708?tcp set CODE=%C11% | |
| ::if .%CODE%==.NONE "C:\Program Files\Serviio\lib\ffmpeg.exe" %A% %B% %C% %D% %E% %F% %G% %H% %I% %J% %K% %L% %M% %N% %O% %P% %Q% %R% %S% %T% %U% %V% %W% %X% %Y% %Z% %AA% %BB% %CC% %DD% %EE% %FF% %GG% %HH% %II% %JJ% %KK% %LL% %MM% %NN% %OO% %PP% %QQ% %RR% %SS% %TT% %UU% %VV% %WW% %XX% %YY% %ZZ% | |
| if NOT .%CODE%==.NONE echo REFRESH %CODE% >> "C:\Program Files\Serviio\log\ffmpeg.log" | |
| if NOT .%CODE%==.NONE curl -X POST "http://localhost:23423/rest/action" -H "Content-Type: text/xml" -d "<action><name>forceOnlineResourceRefresh</name><parameter>%CODE%</parameter></action>" | |
| "C:\Program Files\Serviio\lib\ffmpeg.exe" %A% %B% %C% %D% %E% %F% %G% %H% %I% %J% %K% %L% %M% %N% %O% %P% %Q% %R% %S% %T% %U% %V% %W% %X% %Y% %Z% %AA% %BB% %CC% %DD% %EE% %FF% %GG% %HH% %II% %JJ% %KK% %LL% %MM% %NN% %OO% %PP% %QQ% %RR% %SS% %TT% %UU% %VV% %WW% %XX% %YY% %ZZ% | |
| exit | |
| |
| | 4) The addition of the line -Dffmpeg.location="C:\Program Files\Serviio\lib\ffmpeg.bat" to the file C:\Program Files\Serviio\bin\ServiioService.exe.vmoptions to cause all Serviio ffmpeg calls to be sent to ffmpeg.bat This needs to be repeated after each Serviio new install, which will restore that file. |
| | After the addition it will contain: |
| | <code>-Xmx512m |
| | -Dffmpeg.location="C:\Program Files\Serviio\lib\ffmpeg.bat" |
| </code> | </code> |
| | 5) The installation of the attached ffmpeg.bat as C:\Program Files\Serviio\lib\ffmpeg.bat which will detect the ffmpeg calls containing the ISS URL, extract the "vValue" nnn and issue the curl command to refresh the Serviio online library nnn, before calling ffmpeg to process the original ffmpeg call. Note the ffmpeg.bat also logs each ffmpeg call in C:\Program Files\Serviio\log\ffmpeg.log which I have found useful for debugging. The bat may be edited to comment out that line as noted in the bat. |
| |
| | 6) The addition of WebResource to the Library Online Sources section of the console using the dummy url http://www.refresh.com |
| |
| | With these changes, a Refresh menu will be created with items showing the library id and name of each active Web Resource online source together with a "Click to Refresh Folder" icon. Clicking on an item, initiates playback of the ISS stream while the feed refreshes. Note that a refresh can require up to 30 seconds to cache a new item, but with hourly automatic updates many of the new events should already be cached. After waiting an appropriate period, playback may be stopped, the Refresh and Online Folders exited, and the Online Folder reopened. If the refresh is complete the Refreshed folder should appear. If not repeat the closing and reopening of the Online Folder until it does. |
| |
| | __The FFMPEG.BAT - Updated for Serviio 1.0__ |
| | Download {{:ffmpeg.bat.zip}} |
| |
| | **How to setup Refresh on demand from client for Linux and Synology NAS:** [[refresh_on_demand_linux|refresh_on_demand_linux]] |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| |
| |
