User Tools

Site Tools


quick_refresh

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision
Previous revision
quick_refresh [2013/02/04 01:05]
jhb50 created
quick_refresh [2013/02/10 19:20] (current)
jhb50
Line 3: Line 3:
 This wiki entry describes a method that may be used in web resource plugins to significantly reduce the time required to refresh the plugin feed, by caching the first extracted urls and delaying subsequent extracts to playback time. It allows subsequent refreshes to be completed in a few seconds which allows for more frequent refreshes and currency of available items. This wiki entry describes a method that may be used in web resource plugins to significantly reduce the time required to refresh the plugin feed, by caching the first extracted urls and delaying subsequent extracts to playback time. It allows subsequent refreshes to be completed in a few seconds which allows for more frequent refreshes and currency of available items.
  
-Serviio supports access to online sources with a 3 step process.+Serviio supports access to online sources with a 3 step process. This is more fully described in the previous wiki entry http://wiki.serviio.org/doku.php?id=online_logic
  
-1) The acquisition of the online items and their titles within a feed.+1) The acquisition of the online items and their titles within a feed. (WebResourceContainer extractItems)
  
-2) The extract of the stream url associated with each item.+2) The extract of the stream url associated with each item. (ContentURLContainer extractUrl)
  
-3) The determination of the attributes of each extracted stream using ffmpeg. +3) The determination and caching of the attributes (metadata) of each extracted stream using ffmpeg. 
  
 Live streams defined in the console only need step 3. Live streams defined in the console only need step 3.
Line 17: Line 17:
 Web Resources defined in the console need all 3 steps. Web Resources defined in the console need all 3 steps.
  
-After the initial acquisition of items and extract of feeds, Serviio needs to periodiically refresh the feeds in order to maintain the currency of items in the online menus. This means a reexecution of the above steps as appropriate to each online source either at the earliest of the console default expiry value or the "expires on" date of any item within a feed.+After the initial acquisition of itemsextract of the stream urls, and caching of the attributes, Serviio needs to periodiically refresh the feeds in order to maintain the currency of items in the online menus. This means a re-execution of the above steps as appropriate to each online sourceeither at the earliest of the console default expiry value or the "expires on" date of any item within a feed.
  
-To optimize that refresh processing, Serviio caches the step 3 attributes of each valid stream as returned by ffmpeg, in a database, the first time a stream is processed, so that they are available without the need to access ffmpeg again on subsequent refreshes.+To optimize that refresh processing, Serviio caches the step 3 attributes of each valid streamas returned by ffmpeg, in a database, the first time a stream is processed, so that they are available on subsequent refreshes without the need to access ffmpeg again.
  
-Serviio also reextracts the stream url in step 2 for each item each time a feed is refreshed, so that the latest url is immediately available for playback. This process involves accessing one or more urls and takes multiple elapsed seconds for each item. Due to the finite number of seconds available, this limits the possible number of feeds and their refresh frequency. +Serviio also re-extracts the stream url in step 2 for each item each time a feed is refreshed, so that the latest url is immediately available for playback. This process involves accessing one or more urls and takes multiple elapsed seconds for each item. Due to the finite number of seconds available, this limits the possible number of enabled feeds and their refresh frequency. 
  
 Since it is possible on some feeds for a stream url to change between extract time and playback time, extracted streams may also be declared as "expires immediately" which will cause step 2 to be run and the url extracted again prior to playback.  Since it is possible on some feeds for a stream url to change between extract time and playback time, extracted streams may also be declared as "expires immediately" which will cause step 2 to be run and the url extracted again prior to playback. 
  
-To further optimize refresh processing, this "expires immediately" attribute can be used defer the step 2 reextract of urls until the streams are played back, by implementing the following "url cache" methodology within a plugin. The cache allows the first extracted url to be used to generate the extracted url on subsequent refreshes rather than repeat the step 2 extract process.  This allows step 2 for all the items in a feed to be completed in a few seconds rather than multiple minutes when extracted, and makes the frequent refresh of many feeds and items practical.+This "expires immediately" attribute can be used to further optimize refresh processing, by defering the step 2 re-extract of urls until the streams are played back, by implementing the following "url cache" methodology within a plugin. The cache allows the information found during the first extract of a url as stored in the ContentURLContainer, to be used to generate the extracted url by rebuilding the ContentURLContainer on subsequent refreshes without repeating the step 2 extract process.  This allows step 2 for all the items in a feed to be completed in a few seconds rather than multiple minutes when extracted, and makes the frequent refresh of many feeds and items practical.
  
-The method consists of creating 2 global lists (new and old) available to both step 1 and step 2. During step 1, each item is added to the old list, and during step 2 each extracted url is appended to the old list item and the old list item is added to the new list.+The method consists of creating 2 new global lists (new and old) available to both step 1 and step 2, containing the WebResource item information of all items whose urls were previously extracted. During step 1, any existing "new list" of previously extracted items is purged of expired items and limited to 300 items because some items have no expiry date, and a corresponding "old list" is created. Each WebResource item is then given a unique "item key" by the plugin, then checked to see if that "item key" is in the "old list" of items with previously extracted urls, and if so a generate flag in the WebResource item information is set true and its index in the "old list" is added to the WebResource item information for use in step 2. During step 2 either the existing item information in the "old list" index is used to generate the url and reset the WebResource item generate flag to false if the generate flag is true; or a new url is extracted, the ContentUrl information and the index to the end of the "new list" is added to the WebResource item information and the WebResource item is appended to the "new list".
  
-On subsequent refreshes, the new list is purged of any expired or non "expires immediatelyitems, and the unexpired items are used to create an old list. Step is then run and if an item is already in the old list, a generate flag is set. If an item is newthe generate flag is not set. Step 2 is then run for each item, and the extracted url is generated from the saved old list information rather than extracted as is done for those items without the generate flag. Both the generated items and extracted items are used to create a new list.+The method also supports the re-extraction of a url for any current item by Serviio when "expiresImmediately" urls are "try(ed) again" and Step is re-executed. Because the generate flag in the WebResource item information was reset after any previous generate of a previously extracted url, and because the index to the item in the "new listis contained in the WebResource item information , Step 2 will re-extract the url, update the ContentUrl information in the WebResource item information, and update the previous item in-place using the "new list" index.
  
-This method is currently implemented in the hahasportstopstreamilivecoolsport and skysportplus plugins and makes practical the refresh of these feeds and the update of available events/channels every 30 minutes without significant delay.  +This method is currently implemented in the HaHaSportStopstreamILiveCoolsport, SkysportsPlus and APVideo plugins and makes practical the refresh of these feeds and the update of available events/channels every 30 minutes without significant delay.  
 + 
 +The method could be incorporated within Serviio by maintaining the 2 new lists internally and adding the list purge logic at the start of ExtractItems, the match logic after each WebResource item, the generate logic at the start of ExtractUrls, and the save/update logic after each ContentURLContainer. A ticket proposing that feature has been raised.
  
  
quick_refresh.1359939917.txt.gz · Last modified: 2013/02/04 01:05 by jhb50