Full-Text RSS

Full-Text RSS works by taking an inputted URL, parsing the content, and creating a full-text feed out of it.

The feed can then be followed for changes via online or desktop feed readers, working just like any other feed, updating whenever a change is detected on the original page.

Full-Text RSS in its full glory is available under two commercial licenses. A free version is available, but the commercial versions yield better extraction results.

What is new in this release:

• Open Graph properties og:title, og:type, og:url, og:image, and og:description now returned if found in the page being processed
• Bug fix: certain XPath expressions weren't being evaluated correctly when HTML5 parsing was enabled
• Cookie handling now only on redirects - fixes issue with certain sites (thanks to Dave Vasilevsky)
• Compatibility test will no longer show HHVM as incompatible - Full-Text RSS worked with HHVM 3.7.1 in our tests (but without Tidy support and no automatic site config updates)
• Humble HTTP Agent updated to support version 2 of PHP's HTTP extension
• HTML5-PHP library updated
• Site config files can now include HTTP headers (user-agent, cookie, referer), e.g. http_header(user-agent): PHP/5.6
• Config option removed: $options->user_agents - use site config files.
• Site config files which use single_page_link can now follow it with if_page_contains: XPath to make it conditional.
• Minimum supported PHP version is now 5.3. If you must use PHP 5.2, please download Full-Text RSS 3.4
• Site config files updated for better extraction
• Other minor fixes/improvements

What is new in version 3.4:

• New request parameter: siteconfig lets you submit extraction rules directly in request
• New request paramter: accept=(auto|feed|html) determines what we'll accept as a response (deprecates html=1 parameter)
• New request parameter: key_redirect=0 to prevent HTTP redirect to hide API key
• Site config files can now contain native_ad_clue: [xpath] to check for elements which signify that the article is a native ad
• New config option: remove_native_ads - set to true and when we notice native ads (see above) we'll remove them from the output (only when processing feeds, doesn't affect output when input URL points to an HTML page).
• Feed output will include Native Ad for articles which appear to be native ads.
• New config option: user_submitted_config to determine whether siteconfig parameter is enabled or not
• Feed output now includes with URL of the generated feed
• Feed output now includes with URL of the original (input) URL
• Feed output now includes with URL to subscribe to the generated feed (using subtome.com)
• Feed preview stylesheet (feed.xsl) now presents a subscribe to feed link
• Fixed character encoding issue for certain texts
• Fixed character encoding issue for certain characters in HTML5 parsing mode

What is new in version 3.3:

• New HTML5 parser: HTML5Lib has been replaced by HTML5-PHP (the old one had too many problems)
• New config option: cache time ($options->cache_time)
• New config option: enable/disable single-page retrieval ($options->singlepage)
• New config option: allow HTML parser override through querystring ($options->allow_parser_override)
• New request parameter: parser - use it to force new HTML5 parser to be used, &parser=html5php (it will be slower)
• Expanded debug request parameter: &debug=rawhtml (shows original response headers and body), &debug=parsedhtml (shows response body after parsing)
• APC stats page now expects APCu (older version of APC still supported, but stats within admin area won't be viewable)
• Auto update of site-specific extraction rules fixed
• Content security HTTP headers now used for the feed preview
• Request parameters and response examples now listed in a table on the index page (new Request Parameters tab)
• Compatibility test file updated to show if HTML5-PHP parser is supported (PHP 5.3 dependency), and to test for HHVM (not yet supported)
• Config option removed: $options->registration_key
• Preserve TTL element in RSS 2.0 feeds
• Other minor fixes/improvements

What is new in version 3.2:

• Full content can now be excluded from the output (pass &content=0 in querystring, see $options->content in config file for more info)
• Site config files can now be automatically updated from our GitHub repository (URL to call visible in admin area)
• Site config files updated for better extraction
• PHP Readability updated to be more lenient when pruning HTML
• Language detection library updated
• HTML meta refresh redirects now also followed
• APC stats (if APC is available on your server) now visible in admin area
• Bug fix: Duplicate find_string and replace_string values in site config files no longer removed (thanks Fabrizio!)
• Bug fix: MIME type actions now applied when following single page URLs
• Other minor fixes/improvements

What is new in version 3.1:

• Allow multiple elements (previously only one was preserved)
• Bug fix: No more self-closing iframe elements
• Bug fix: Fixed manifest.yml to prevent error message when deploying to AppFog
• Other minor fixes/improvements

What is new in version 3.0:

• Multi-page supportnext_page_link now supported in site config (enable/disable with $options->multipage)
• HTML5 parser availableuse parser: html5lib in site config, also see $options->allowed_parsers
• Updated site patterns for better extraction
• New global site config to be applied to all sites (global.txt)
• Strip 'http://' prefix when API key is supplied
• Site config merging (custom + standard + fingerprint + global)
• Site config command replace_string(find): replace can now be split over two lines: find_string: find, replace_string: replace
• YouTube and Vimeo URLs now return iframe embed code
• We now look for OpenGraph title and date elements
• Improved extraction from AJAX pageswe now look for AJAX triggers embedded in HTML, per Google spec
• JSONP supportuse &format=json&callback=functionName in querystring
• New config option to enable Cross-Origin Resource Sharing (CORS): $option->cors
• New config option to enable XSS filtering, if required: $option->xss_filter
• Zend_Cache updated
• Smart cachingexperimental feature to store cache IDs in APC first, and write output to disk on subsequent request (see $options->smart_cache)
• Easier cloud deploymanifest.yml added for AppFog
• APC caching of site config files to improve performance, if APC availablesee $options->apc
• Site config editor in admin/easily find, edit, test, and test site config files, or add new ones
• Debug mode to see what's happening behind the scenessee $options->debug
• Removed deprecated config options: restrict, message_to_prepend_with_key, message_to_append_with_key, error_message_with_key
• Removed extraction with CSS via querystring
• Removed config option: $options->alternative_url
• Bug fix: allow extraction of a single element
• Bug fix: redirect handling improved

Requirements:

• PHP 5.2 or higher