Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Feed polling and caching

SparkleMuffin periodically makes HTTP requests to update Atom and RSS feeds. This creates two requirements:

  • do not put unnecessary load on the remote servers;
  • do not perform unnecessary database updates when the remote content has not changed.

SparkleMuffin uses HTTP caching features from the HTTP specification. It also performs extra checks on the feed content.

HTTP Conditional Requests

When responding to an HTTP request, a remote server may set the following headers:

  • ETag: the current entity tag for the selected representation (usually a hash of the feed data);
  • Last-Modified: the date and time the origin server last modified the selected representation.

When present, SparkleMuffin stores these values in the database. It uses them to set the following headers in later requests:

  • If-None-Match: the value of the ETag header from the previous response;
  • If-Modified-Since: the value of the Last-Modified header from the previous response.

The remote server responds in one of two ways:

  • 200 OK: the content changed. SparkleMuffin updates the feed and its entries;
  • 304 Not Modified: nothing changed. SparkleMuffin only updates the feed’s ETag and Last-Modified headers.

Feed content hash

A remote server can send a different ETag or Last-Modified value even when the feed content has not changed. It can also send neither header. To handle this, SparkleMuffin:

  • computes and stores a hash of the feed data, using the xxHash non-cryptographic hash function;
  • compares the hash of the feed data with the value stored in the database;
  • returns early if the hashes match, to avoid unnecessary database updates.

Reference

Feed caching

RFCs

HTTP Conditional Requests

Non-cryptographic hash functions