User Tools

Site Tools


quick_refresh

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
quick_refresh [2013/02/09 17:48]
jhb50
quick_refresh [2013/02/10 19:20] (current)
jhb50
Line 3: Line 3:
 This wiki entry describes a method that may be used in web resource plugins to significantly reduce the time required to refresh the plugin feed, by caching the first extracted urls and delaying subsequent extracts to playback time. It allows subsequent refreshes to be completed in a few seconds which allows for more frequent refreshes and currency of available items. This wiki entry describes a method that may be used in web resource plugins to significantly reduce the time required to refresh the plugin feed, by caching the first extracted urls and delaying subsequent extracts to playback time. It allows subsequent refreshes to be completed in a few seconds which allows for more frequent refreshes and currency of available items.
  
-Serviio supports access to online sources with a 3 step process.+Serviio supports access to online sources with a 3 step process. This is more fully described in the previous wiki entry http://wiki.serviio.org/doku.php?id=online_logic
  
 1) The acquisition of the online items and their titles within a feed. (WebResourceContainer extractItems) 1) The acquisition of the online items and their titles within a feed. (WebResourceContainer extractItems)
Line 27: Line 27:
 This "expires immediately" attribute can be used to further optimize refresh processing, by defering the step 2 re-extract of urls until the streams are played back, by implementing the following "url cache" methodology within a plugin. The cache allows the information found during the first extract of a url as stored in the ContentURLContainer, to be used to generate the extracted url by rebuilding the ContentURLContainer on subsequent refreshes without repeating the step 2 extract process.  This allows step 2 for all the items in a feed to be completed in a few seconds rather than multiple minutes when extracted, and makes the frequent refresh of many feeds and items practical. This "expires immediately" attribute can be used to further optimize refresh processing, by defering the step 2 re-extract of urls until the streams are played back, by implementing the following "url cache" methodology within a plugin. The cache allows the information found during the first extract of a url as stored in the ContentURLContainer, to be used to generate the extracted url by rebuilding the ContentURLContainer on subsequent refreshes without repeating the step 2 extract process.  This allows step 2 for all the items in a feed to be completed in a few seconds rather than multiple minutes when extracted, and makes the frequent refresh of many feeds and items practical.
  
-The method consists of creating 2 new global lists (new and old) available to both step 1 and step 2, containing the WebResource item information of all items whose urls were previously extracted. During step 1, any existing "new list" of previously extracted items is purged of expired items and a corresponding "old list" is created. Each WebResource item is given an "item key", then checked to see if the "item key" is in the "old list" of items with previously extracted urls, and if so a generate flag in the WebResource item information is set true and its index in the "old list" is added to the WebResource item information for use in step 2. During step 2 either the existing item information in the "old list" index is used to generate the url and reset the WebResource item generate flag to falseif the generate flag is true; or a new url is extracted, the ContentUrl information and the index to the end of the "new list" is added to the WebResource item information and the WebResource item is appended to the "new list".+The method consists of creating 2 new global lists (new and old) available to both step 1 and step 2, containing the WebResource item information of all items whose urls were previously extracted. During step 1, any existing "new list" of previously extracted items is purged of expired items and limited to 300 items because some items have no expiry date, and a corresponding "old list" is created. Each WebResource item is then given a unique "item key" by the plugin, then checked to see if that "item key" is in the "old list" of items with previously extracted urls, and if so a generate flag in the WebResource item information is set true and its index in the "old list" is added to the WebResource item information for use in step 2. During step 2 either the existing item information in the "old list" index is used to generate the url and reset the WebResource item generate flag to false if the generate flag is true; or a new url is extracted, the ContentUrl information and the index to the end of the "new list" is added to the WebResource item information and the WebResource item is appended to the "new list".
  
-The method also supports the re-extraction of a url for any current item by Serviio when "expiresImmediately" urls are "try(ed) again" by re-executing Step 2. Because the generate flag in the WebResource item information was reset after any previous generate of a previously extracted url, and because the index to the item in the "new list" is contained in the WebResource item information , Step 2 will re-extract the url, update the ContentUrl information in the WebResource item information, and store the updated item in the "new list"index.+The method also supports the re-extraction of a url for any current item by Serviio when "expiresImmediately" urls are "try(ed) again" and Step 2 is re-executed. Because the generate flag in the WebResource item information was reset after any previous generate of a previously extracted url, and because the index to the item in the "new list" is contained in the WebResource item information , Step 2 will re-extract the url, update the ContentUrl information in the WebResource item information, and update the previous item in-place using the "new list" index.
  
 This method is currently implemented in the HaHaSport, Stopstream, ILive, Coolsport, SkysportsPlus and APVideo plugins and makes practical the refresh of these feeds and the update of available events/channels every 30 minutes without significant delay.  This method is currently implemented in the HaHaSport, Stopstream, ILive, Coolsport, SkysportsPlus and APVideo plugins and makes practical the refresh of these feeds and the update of available events/channels every 30 minutes without significant delay. 
  
-The method could be incorporated within Serviio by maintaining the 2 new lists internally and adding the list purge logic at the start of ExtractItems, the match logic after each WebResource item, the generate logic at the start of ExtractUrls, and the save/update logic after each ContentURLContainer. A ticket to that end has been raised.+The method could be incorporated within Serviio by maintaining the 2 new lists internally and adding the list purge logic at the start of ExtractItems, the match logic after each WebResource item, the generate logic at the start of ExtractUrls, and the save/update logic after each ContentURLContainer. A ticket proposing that feature has been raised.
  
  
quick_refresh.1360432119.txt.gz ยท Last modified: 2013/02/09 17:48 by jhb50