User Tools

Site Tools


quick_refresh

This is an old revision of the document!


Online Sources Quick Refresh Method

This wiki entry describes a method that may be used in web resource plugins to significantly reduce the time required to refresh the plugin feed, by caching the first extracted urls and delaying subsequent extracts to playback time. It allows subsequent refreshes to be completed in a few seconds which allows for more frequent refreshes and currency of available items.

Serviio supports access to online sources with a 3 step process.

1) The acquisition of the online items and their titles within a feed.

2) The extract of the stream url associated with each item.

3) The determination of the attributes of each extracted stream using ffmpeg.

Live streams defined in the console only need step 3.

Online RSS/Atom feeds defined in the console already contain a list of items and so only need steps 2 and 3.

Web Resources defined in the console need all 3 steps.

After the initial acquisition of items and extract of feeds, Serviio needs to periodiically refresh the feeds in order to maintain the currency of items in the online menus. This means a reexecution of the above steps as appropriate to each online source either at the earliest of the console default expiry value or the “expires on” date of any item within a feed.

To optimize that refresh processing, Serviio caches the step 3 attributes of each valid stream as returned by ffmpeg, in a database, the first time a stream is processed, so that they are available without the need to access ffmpeg again on subsequent refreshes.

Serviio also reextracts the stream url in step 2 for each item each time a feed is refreshed, so that the latest url is immediately available for playback. This process involves accessing one or more urls and takes multiple elapsed seconds for each item. Due to the finite number of seconds available, this limits the possible number of feeds and their refresh frequency.

Since it is possible on some feeds for a stream url to change between extract time and playback time, extracted streams may also be declared as “expires immediately” which will cause step 2 to be run and the url extracted again prior to playback.

To further optimize refresh processing, this “expires immediately” attribute can be used defer the step 2 reextract of urls until the streams are played back, by implementing the following “url cache” methodology within a plugin. The cache allows the first extracted url to be used to generate the extracted url on subsequent refreshes rather than repeat the step 2 extract process. This allows step 2 for all the items in a feed to be completed in a few seconds rather than multiple minutes when extracted, and makes the frequent refresh of many feeds and items practical.

The method consists of creating 2 global lists (new and old) available to both step 1 and step 2. During step 1, each item is added to the old list, and during step 2 each extracted url is appended to the old list item and the old list item is added to the new list.

On subsequent refreshes, the new list is purged of any expired or non “expires immediately” items, and the unexpired items are used to create an old list. Step 1 is then run and if an item is already in the old list, a generate flag is set. If an item is new, the generate flag is not set. Step 2 is then run for each item, and the extracted url is generated from the saved old list information rather than extracted as is done for those items without the generate flag. Both the generated items and extracted items are used to create a new list.

This method is currently implemented in the hahasport, stopstream, ilive, coolsport and skysportplus plugins and makes practical the refresh of these feeds and the update of available events/channels every 30 minutes without significant delay.

quick_refresh.1359939917.txt.gz · Last modified: 2013/02/04 01:05 by jhb50