Serviio Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
Online Sources Quick Refresh Method
===================================
This wiki entry describes a method that may be used in web resource plugins to significantly reduce the time required to refresh the plugin feed, by caching the first extracted urls and delaying subsequent extracts to playback time. It allows subsequent refreshes to be completed in a few seconds which allows for more frequent refreshes and currency of available items.

Serviio supports access to online sources with a 3 step process. This is more fully described in the previous wiki entry http://wiki.serviio.org/doku.php?id=online_logic

1) The acquisition of the online items and their titles within a feed. (WebResourceContainer extractItems)

2) The extract of the stream url associated with each item. (ContentURLContainer extractUrl)

3) The determination and caching of the attributes (metadata) of each extracted stream using ffmpeg. 

Live streams defined in the console only need step 3.

Online RSS/Atom feeds defined in the console already contain a list of items and so only need steps 2 and 3.

Web Resources defined in the console need all 3 steps.

After the initial acquisition of items, extract of the stream urls, and caching of the attributes, Serviio needs to periodiically refresh the feeds in order to maintain the currency of items in the online menus. This means a re-execution of the above steps as appropriate to each online source, either at the earliest of the console default expiry value or the "expires on" date of any item within a feed.

To optimize that refresh processing, Serviio caches the step 3 attributes of each valid stream, as returned by ffmpeg, in a database, the first time a stream is processed, so that they are available on subsequent refreshes without the need to access ffmpeg again.

Serviio also re-extracts the stream url in step 2 for each item each time a feed is refreshed, so that the latest url is immediately available for playback. This process involves accessing one or more urls and takes multiple elapsed seconds for each item. Due to the finite number of seconds available, this limits the possible number of enabled feeds and their refresh frequency. 

Since it is possible on some feeds for a stream url to change between extract time and playback time, extracted streams may also be declared as "expires immediately" which will cause step 2 to be run and the url extracted again prior to playback. 

This "expires immediately" attribute can be used to further optimize refresh processing, by defering the step 2 re-extract of urls until the streams are played back, by implementing the following "url cache" methodology within a plugin. The cache allows the information found during the first extract of a url as stored in the ContentURLContainer, to be used to generate the extracted url by rebuilding the ContentURLContainer on subsequent refreshes without repeating the step 2 extract process.  This allows step 2 for all the items in a feed to be completed in a few seconds rather than multiple minutes when extracted, and makes the frequent refresh of many feeds and items practical.

The method consists of creating 2 new global lists (new and old) available to both step 1 and step 2, containing the WebResource item information of all items whose urls were previously extracted. During step 1, any existing "new list" of previously extracted items is purged of expired items and limited to 300 items because some items have no expiry date, and a corresponding "old list" is created. Each WebResource item is then given a unique "item key" by the plugin, then checked to see if that "item key" is in the "old list" of items with previously extracted urls, and if so a generate flag in the WebResource item information is set true and its index in the "old list" is added to the WebResource item information for use in step 2. During step 2 either the existing item information in the "old list" index is used to generate the url and reset the WebResource item generate flag to false if the generate flag is true; or a new url is extracted, the ContentUrl information and the index to the end of the "new list" is added to the WebResource item information and the WebResource item is appended to the "new list".

The method also supports the re-extraction of a url for any current item by Serviio when "expiresImmediately" urls are "try(ed) again" and Step 2 is re-executed. Because the generate flag in the WebResource item information was reset after any previous generate of a previously extracted url, and because the index to the item in the "new list" is contained in the WebResource item information , Step 2 will re-extract the url, update the ContentUrl information in the WebResource item information, and update the previous item in-place using the "new list" index.

This method is currently implemented in the HaHaSport, Stopstream, ILive, Coolsport, SkysportsPlus and APVideo plugins and makes practical the refresh of these feeds and the update of available events/channels every 30 minutes without significant delay. 

The method could be incorporated within Serviio by maintaining the 2 new lists internally and adding the list purge logic at the start of ExtractItems, the match logic after each WebResource item, the generate logic at the start of ExtractUrls, and the save/update logic after each ContentURLContainer. A ticket proposing that feature has been raised.
Serviio Wiki

User Tools

Site Tools

Page Tools
