SparkleMuffin

SparkleMuffin is a self-hosted bookmark manager and feed aggregator.

It provides a minimalist interface thait aims at being simple to use, clutter-free and accessible.

Documentation

The documentation is split in two main sections:

  • a User Guide that showcases SparkleMuffin's features and how to use them;
  • a Developer Guide that provides information on how SparkleMuffin works, and how to contribute to the project.

Contributing

SparkleMuffin is free and open-source, licensed under the MIT License. You can find the source code on Github, and issues and feature requests can be posted on the issue tracker.

If you would like to contribute, please read the Contributing guide!

What status is the project currently in?

SparkleMuffin is still alpha software, and should not be considered ready for production use.

Change Log

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog and this project adheres to Semantic Versioning.

v0.4.3 - 2025-01-05

Fixed

Feeds

  • Ensure truncating entry descriptions does not result in invalid UTF-8 code points

v0.4.2 - 2024-12-21

Changed

Feeds

  • When deleting a category or subscription, propagate the deletion to feeds with no remaining subscriptions

Security

v0.4.1 - 2024-12-14

Security

v0.4.0 - 2024-12-10

Added

Database

  • Add PostgreSQL integration tests for feed operations

Changed

Database

  • Split PostgreSQL repository into dedicated domain repositories
  • Update testcontainers configuration to use a tmpfs volume and disable WAL features to speed up integration tests

Feeds

  • Update page title to display the subscription alias (if set) or the feed title
  • Update listed entries to display the subscription alias (if set) or the feed title

WWW

  • Relocate HTTP packages to internal/http
  • Relocate version detection helpers to internal/version

v0.3.1 - 2024-12-07

Fixed

Feeds

  • Fix HTML templates after renaming querying models

v0.3.0 - 2024-12-07

Added

Database

  • Add dedicated tests for PostrgeSQL database migrations (up/down)

Documentation

  • Add custom CSS to display wider content on large screens

Feeds

  • For each entry in the list, display the title of the corresponding feed
  • Save and display feed descriptions
  • Extract keywords (significant terms) from entry content/description with TextRank
  • Add full-text search based on feed and entry metadata
  • Store and compare the hash (xxHash64) of the feed data to avoid unnecessary database upserts
  • Document the feed polling and caching strategy
  • Allow users to set an alias title for feed subscriptions

Changed

CI

  • Lint and format SQL files with SQLFluff
  • Publish HTML documentation to Github Pages

Documentation

  • Update documentation structure to follow the Diátaxis approach
  • Disable mdBook file auto-creation
  • Check for broken links with mdbook-linkcheck

www

  • Update the home page
  • Render HTTP 4xx errors as HTML views

Fixed

Docker

  • Install the ca-certificates package in the Docker image for TLS connections

Feeds

  • Ensure entry publication and update dates are non-zero
  • Ensure entry publication and update dates are not in the (far) future (limit: 2 days)
  • In the subscription edit form, ensure the correct category is selected

v0.2.0 - 2024-11-14

Added

Feeds

  • Subscribe to Atom and RSS feeds
  • Categorize subscriptions
  • Display subscriptions
  • Periodically synchronize subscriptions
  • Import existing subscriptions from OPML
  • Export subscriptions

Changed

Bookmarks

  • Enforce CSRF validation for import and export forms

PostgreSQL

  • Update repository helpers

Packaging & automation

  • Build with Go 1.23
  • Update direct and transitive dependencies

v0.1.1 - 2024-01-26

Initial release

Added

Bookmarks

  • Create and manage users of the application
  • Create and manage bookmarks to Web pages (links)
  • Display bookmarks and bookmark tags
  • Import existing bookmarks
  • Search bookmarks by keywords (full-text search)

Command-line & configuration

  • Add a sparklemuffin root command to handle common program configuration
  • Add a createadmin subcommand to create users with administrator privileges
  • Add a migrate subcommand to manage database migrations
  • Add a run subcommand to start all services
  • Add a version subcommand to display the running version (featuring Git version information)
  • Allow configuring services via:
    • application defaults
    • configuration file
    • command-line flags
    • environment variables

Observability

  • Setup structured logging (formats: console, JSON)
  • Expose Prometheus metrics:
    • Go runtime
    • HTTP requests
    • Application build and version information

Packaging & automation

  • Package the application as a Docker container
  • Provide Docker Compose configuration for:
    • local development
    • example usage
  • Add mdBook documentation
  • Add Make targets to:
    • run static analysis tools (linters)
    • run unitary tests
    • run integration tests
    • generate coverage reports
    • run live-reload development servers
    • build HTML documentation
  • Add Github Actions workflows:
    • CI: build the application, run linters, run tests
    • Copywrite: ensure license headers are present in source files
    • Docker: build and publish Docker image to the Github Container Registry (GHCR)

User Guide

How-to Guides

Practical step-by-step guides to help you achieve a specific goal. Most useful when you're trying to get something done.

Run SparkleMuffin locally

Goals

This guide shows how to run SparkleMuffin locally on your computer, in development mode, with debug logging enabled.

This setup is only fit for testing purposes, and should not be used as-is in production (or a public-facing server).

Prerequisites

If you are not familiar with Docker or Docker Compose, take a look at the quickstart guides:

Run services with Docker Compose

Create a new directory and download the example Docker Compose configuration:

$ mkdir -p sparklemuffin
$ cd sparklemuffin
$ wget https://raw.githubusercontent.com/virtualtam/sparklemuffin/main/docker-compose.yml

Pull the Docker images for PostgreSQL and SparkleMuffin:

$ docker compose pull

Run PostgreSQL and SparkleMuffin:

$ docker compose up -d

Apply database migrations

Use the sparklemuffin migrate command to apply database migrations:

$ docker compose exec sparklemuffin sparklemuffin migrate

Create a first administrator user

Use the sparklemuffin createadmin command to create the admin@local.dev user; a strong password will be generated automatically for you:

$ docker compose exec sparklemuffin \
    sparklemuffin createadmin \
		--displayname Admin \
		--email admin@dev.local \
		--nickname admin

Log in to the SparkleMuffin

You are now ready to access SparkleMuffin's Web interface by opening http://localhost:8080 in your Web browser, and log in using the credentials from the previous step:

  • login: admin@dev.local;
  • password: use the password generated by the sparklemuffin createadmin command.

Cleanup - When you are done using SparkleMuffin

Stop the services

Stop running containers with:

$ docker compose stop

Stop the services, remove containers and PostgreSQL data volume

$ docker compose down -v

Reference Guides

Features

Bookmarks

SparkleMuffin allows you to:

  • save, tag and search your Web bookmarks;
  • import your existing bookmarks from a Web browser or another bookmarking application, using the Netscape Bookmark File Format;
  • share your public bookmarks so they can be accessed by a Web browser or as an Atom feed.

Feeds

SparkleMuffin allows you to:

  • subscribe to Atom and RSS feeds;
  • import your existing feed subscriptions using the OPML File Format.

Web interface

SparkleMuffin aims at providing a Web interface that is:

  • simple to use;
  • minimalist and clutter-free;
  • accessible on Web browsers for both desktop and mobile devices;
  • light on Javascript.

Database

SparkleMuffin is backed by a PostgreSQL database, which allows multiple users to log in to the same instance, and eases administration, backup and maintenance.

Roadmap

See the changelog for past releases, and the milestones for upcoming features.

Configuration

Configuration formats

SparkleMuffin can be configured by:

  • using a configuration file;
  • setting POSIX flags, e.g. --log-level debug;
  • setting environment variables, e.g. SPARKLEMUFFIN_LOG_LEVEL=debug.

Configuration variable precedence

Configuration variables are evaluated in this order:

  • program defaults (lowest precedence);
  • configuration file;
  • command-line flags;
  • environment variables (highest precedence).

Naming convention for configuration variables

All configuration variables are specified as program flags (see command-line flags), from which environment variables names and configuration file keys are derived:

Command-line flagEnvironment VariableConfiguration File
--exampleSPARKLEMUFFIN_EXAMPLEexample: true
--log-level debugSPARKLEMUFFIN_LOG_LEVEL=debuglog-level: debug

Configuration file

  • TODO: add CLI flag to specify a configuration file
  • TODO: add CLI command to generate a configuration file with default values
  • TODO: specify configuration file format (TOML?)
  • TODO: add commented configuration file to SCM
  • TODO: add commented configuration file to docs (section on this page)

Command-line flags

SparkleMuffin is provided as a single-binary command-line application, that provides commands to:

  • run the Web server
  • create administrator users
  • apply database migrations
  • display information about how the program was built
  • etc.

To see which commands and global flags are available, run the sparklemuffin --help command:

$ sparklemuffin --help

SparkleMuffin - Web Bookmark Manager

Usage:
  sparklemuffin [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  createadmin Create a user with administration privileges
  help        Help about any command
  migrate     Initialize database and run migrations
  run         Start the HTTP server
  version     Display the prorgam version

Flags:
      --db-addr string       Database address (host:port) (default "localhost:15432")
      --db-name string       Database name (default "sparklemuffin")
      --db-password string   Database password (default "sparklemuffin")
      --db-sslmode string    Database sslmode (default "disable")
      --db-user string       Database user (default "sparklemuffin")
  -h, --help                 help for sparklemuffin
      --hmac-key string      Secret key for HMAC session token hashing (default "hmac-secret-key")
      --log-level string     Log level (trace, debug, info, warn, error, fatal, panic) (default "info")

Use "sparklemuffin [command] --help" for more information about a command.

To get more information about a specific command, run sparklemuffin <command> --help:

$ sparklemuffin run --help

Start the HTTP server

Usage:
  sparklemuffin run [flags]

Flags:
      --csrf-key string              Secret key for CSRF token hashing (default "csrf-secret-key")
  -h, --help                         help for run
      --listen-addr string           Listen to this address (host:port) (default "0.0.0.0:8080")
      --metrics-listen-addr string   Listen to this address for Prometheus metrics (host:port) (default "127.0.0.1:8081")
      --public-addr string           Public HTTP address (if behind a proxy) (default "http://localhost:8080")

Global Flags:
      --db-addr string       Database address (host:port) (default "localhost:15432")
      --db-name string       Database name (default "sparklemuffin")
      --db-password string   Database password (default "sparklemuffin")
      --db-sslmode string    Database sslmode (default "disable")
      --db-user string       Database user (default "sparklemuffin")
      --hmac-key string      Secret key for HMAC session token hashing (default "hmac-secret-key")
      --log-level string     Log level (trace, debug, info, warn, error, fatal, panic) (default "info")

Observability

Structured Logs

SparkleMuffin logs information on the standard error (stderr) stream, using a structured log message format (JSON or logfmt).

The log format and level can be specified via configuration.

Example logs: program startup

Format: console

2024-01-20T17:03:52+01:00 INF configuration: no file found config_paths=["/etc","/home/dev/.config","."]
2024-01-20T17:03:53+01:00 INF database: successfully created connection pool database_addr=localhost:15432 database_driver=pgx database_name=sparklemuffin
2024-01-20T17:03:53+01:00 INF global: setting up services log_level=info version=devel
2024-01-20T17:03:53+01:00 INF metrics: listening for HTTP requests metrics_addr=127.0.0.1:8081
2024-01-20T17:03:53+01:00 INF sparklemuffin: listening for HTTP requests http_addr=0.0.0.0:8080
2024-01-20T17:04:44+01:00 INF handle request duration_ms=0.750875 host=localhost:8080 method=GET path=/ remote_addr=127.0.0.1:51440 request_id=localhost.local/96bRV2ceWt-000001 size=1187 status=200
2024-01-20T17:04:44+01:00 INF handle request duration_ms=4.369792 host=localhost:8080 method=GET path=/static/awesomplete.css remote_addr=127.0.0.1:51441 request_id=localhost.local/96bRV2ceWt-000004 size=167 status=200
2024-01-20T17:04:44+01:00 INF handle request duration_ms=5.682958 host=localhost:8080 method=GET path=/static/easymde.css remote_addr=127.0.0.1:51442 request_id=localhost.local/96bRV2ceWt-000003 size=931 status=200
2024-01-20T17:04:44+01:00 INF handle request duration_ms=7.072792 host=localhost:8080 method=GET path=/static/www.css remote_addr=127.0.0.1:51440 request_id=localhost.local/96bRV2ceWt-000002 size=5402 status=200

Format: json

{"level":"info","config_paths":["/etc","/home/dev/.config","."],"time":"2024-01-22T22:56:30+01:00","message":"configuration: no file found"}
{"level":"info","database_driver":"pgx","database_addr":"localhost:15432","database_name":"sparklemuffin","time":"2024-01-22T22:56:30+01:00","message":"database: successfully created connection pool"}
{"level":"info","log_level":"info","version":"devel","time":"2024-01-22T22:56:30+01:00","message":"global: setting up services"}
{"level":"info","metrics_addr":"127.0.0.1:8081","time":"2024-01-22T22:56:30+01:00","message":"metrics: listening for HTTP requests"}
{"level":"info","http_addr":"0.0.0.0:8080","time":"2024-01-22T22:56:30+01:00","message":"sparklemuffin: listening for HTTP requests"}
{"level":"info","duration_ms":69.373458,"host":"localhost:8080","method":"GET","path":"/bookmarks","remote_addr":"127.0.0.1:50857","request_id":"localhost.local/dqPgOolnvF-000001","size":26808,"status":200,"time":"2024-01-22T22:57:12+01:00","message":"handle request"}
{"level":"info","duration_ms":5.552375,"host":"localhost:8080","method":"GET","path":"/static/awesomplete.css","remote_addr":"127.0.0.1:50857","request_id":"localhost.local/dqPgOolnvF-000002","size":236,"status":200,"time":"2024-01-22T22:57:12+01:00","message":"handle request"}
{"level":"info","duration_ms":12.052417,"host":"localhost:8080","method":"GET","path":"/static/easymde.css","remote_addr":"127.0.0.1:50860","request_id":"localhost.local/dqPgOolnvF-000003","size":1000,"status":200,"time":"2024-01-22T22:57:12+01:00","message":"handle request"}

Prometheus Metrics

SparkleMuffin exposes Prometheus metrics, providing useful information that can be used for monitoring and alerting.

These metrics are exposed by default on http://0.0.0.0:8081/metrics; the host and port can be specified via configuration.

Available Metrics

Developer Guide

How-to Guides

Practical step-by-step guides to help you achieve a specific goal. Most useful when you're trying to get something done.

Getting the source code

Clone and enter the Git repository:

$ git clone git@github.com:virtualtam/sparklemuffin.git
$ cd sparklemuffin

Static Analysis

Dependencies

Install development utilities

Install Go linters, vulnerability detection and license check tools:

$ make dev-install-tools

Install SQLFluff:

$ make dev-install-sqlfluff

Run linters

Go

Check Go sources with golangci-lint:

$ make lint

Check Go source headers with copywrite:

$ make copywrite

Check Go sources and go.mod for vulnerabilities:

$ make vulncheck

SQL Migrations

Check SQL files with SQLFluff:

$ make lint-sql

Format SQL files with SQLFluff:

$ make format-sql

Running tests

Dependencies

Run tests

Run unitary and integration tests:

$ make test

Run unitary and integration tests with race detection enabled:

$ make race

Code coverage reports

Run unitary and integration tests with code coverage enabled:

$ make cover

Generate an HTML report and open it in your Web browser:

$ make coverhtml

Compiling

Build the application with:

$ make build

The resulting binary will be located under build/sparklemuffin:

$ ls -lah build/

total 32M
drwxr-xr-x  2 dev dev 4.0K Nov  4 19:25 .
drwxr-xr-x 10 dev dev 4.0K Nov  5 00:07 ..
-rwxr-xr-x  1 dev dev  32M Nov  4 19:25 sparklemuffin

Live Development Server

Prerequisites

Run a local development server

The helper Make targets will run a local development server, with:

  • the PostgreSQL database running as a Docker container;
  • the SparkleMuffin application running locally in development mode (via go run).

The application server will be reloaded every time a source file is changed on the disk, thanks to watchexec.

For more information about how services are configured and started, see:

Run a local development server:

$ make live

== Starting database
docker compose -f docker-compose.dev.yml up --remove-orphans -d
[+] Building 0.0s (0/0)                                                                    docker:default
[+] Running 1/1
 ✔ Container sparklemuffin-postgres-1  Started                                                       0.0s
== Watching for changes... (hit Ctrl+C when done)
2023-11-03T10:26:52+01:00 INF configuration: no file found config_paths=["/etc","/home/dev/.config","."]
2023-11-03T10:26:52+01:00 INF database: successfully created connection pool database_addr=localhost:15432 database_driver=pgx database_name=sparklemuffin
2023-11-03T10:26:52+01:00 INF global: setting up services log_level=info version=devel
2023-11-03T10:26:52+01:00 INF metrics: listening for HTTP requests metrics_addr=127.0.0.1:8081
2023-11-03T10:26:52+01:00 INF sparklemuffin: listening for HTTP requests http_addr=0.0.0.0:8080

Run a local development server, with the Go race detector enabled:

$ make live-race

== Starting database
docker compose -f docker-compose.dev.yml up --remove-orphans -d
[+] Building 0.0s (0/0)                                                                                                                                                                               docker:default
[+] Running 1/0
 ✔ Container sparklemuffin-postgres-1  Running                                                                                                                                                                  0.0s
== Watching for changes... (hit Ctrl+C when done)
2023-11-03T10:27:38+01:00 INF configuration: no file found config_paths=["/etc","/home/dev/.config","."]
2023-11-03T10:27:38+01:00 INF database: successfully created connection pool database_addr=localhost:15432 database_driver=pgx database_name=sparklemuffin
2023-11-03T10:27:38+01:00 INF global: setting up services log_level=info version=devel
2023-11-03T10:27:38+01:00 INF metrics: listening for HTTP requests metrics_addr=127.0.0.1:8081
2023-11-03T10:27:38+01:00 INF sparklemuffin: listening for HTTP requests http_addr=0.0.0.0:8080

Run database migrations

$ make dev-migrate

go run ./cmd/sparklemuffin migrate
2023-11-03T10:31:53+01:00 INF configuration: no file found config_paths=["/etc","/home/dev/.config","."]
2023-11-03T10:31:53+01:00 INF database: successfully created connection pool database_addr=localhost:15432 database_driver=pgx database_name=sparklemuffin
2023-11-03T10:31:53+01:00 INF successfully opened database connection database_addr=localhost:15432 database_driver=pgx database_name=sparklemuffin
2023-11-03T10:31:53+01:00 INF migrate: the database schema is up to date database_addr=localhost:15432 database_driver=pgx

Create a first administrator user

Create the user account with:

$ make dev-admin

go run ./cmd/sparklemuffin createadmin \
        --displayname Admin \
        --email admin@dev.local \
        --nickname admin
2023-11-03T10:34:50+01:00 INF configuration: no file found config_paths=["/etc","/home/dev/.config","."]
2023-11-03T10:34:50+01:00 INF database: successfully created connection pool database_addr=localhost:15432 database_driver=pgx database_name=sparklemuffin
2023-11-03T10:34:50+01:00 INF admin user successfully created email=admin@dev.local nickname=admin
Generated password: Qj3Qkeq4GpmEOrzjRv36VqVPQVymztbE4nlQ9u8KhjE=

Then open the application in your Web browser:

  • access http://localhost:8080;
  • login using the generated credentials:
    • Email address: admin@dev.local
    • Password: use the password generated by the make dev-admin command

Stop local services

$ docker compose stop

[+] Stopping 1/1
 ✔ Container sparklemuffin-postgres-1  Stopped

Remove containers and application data

Stop and remove application containers (without removing data volumes):

$ docker compose down

Stop and remove application containers, and remove data volumes:

$ docker compose down -v

Generating the HTML documentation

Prerequisites

HTML Documentation

Build the HTML documentation with:

$ make docs

mdbook build docs
2023-11-05 16:19:04 [INFO] (mdbook::book): Book building has started
2023-11-05 16:19:04 [INFO] (mdbook::book): Running the html backend

The generated website will be located under docs/book/html.

Live-reload server

Start mdBook's live-reload server with:

$ make live-docs

mdbook serve docs
2023-11-05 16:19:25 [INFO] (mdbook::book): Book building has started
2023-11-05 16:19:25 [INFO] (mdbook::book): Running the html backend
2023-11-05 16:19:25 [INFO] (mdbook::cmd::serve): Serving on: http://localhost:3000
2023-11-05 16:19:25 [INFO] (mdbook::cmd::watch): Listening for changes...
2023-11-05 16:19:25 [INFO] (warp::server): Server::run; addr=[::1]:3000
2023-11-05 16:19:25 [INFO] (warp::server): listening on http://[::1]:3000
  • The generated website will be located under docs/book/html;
  • The live server can be accessed by opening http://localhost:3000 in a Web browser.

Reference

Reference Guides

Nitty-gritty technical descriptions of how SparkleMuffin works. Most useful when you need detailed information about SparkleMuffin's implementation, database structure or APIs.

Contributing to SparkleMuffin

SparkleMuffin is still in an early stage of development, and as such, you may find some rough edges and missing features.

The goal of the project is to experiment with writing a Web-based tool that can be used to save, tag and retrieve bookmarks for the Websites you visit, while keeping its interface simple to use, clutter-free and responsive for mobile devices.

If this sounds interesting to you, please keep on reading!

Providing feedback

Issues and improvements

The best way to provide feedback about the project and its documentation is via the Github issue tracker.

Here are some good ways to start contributing:

  • reporting bugs in the application;
  • reporting missing or outdated documentation;
  • providing suggestions to improve accessibility;
  • providing sugegstions to improve User Interface (UI) and User Experience (UX).
Before submitting a new issue, please check the existing issues (open and closed) for similar reports, known issues and planned work.

Feel free to attach any relevant information that may be helpful in fixing the issue, such as:

  • application logs;
  • application configuration (please redact any secret or sensitive information first!);
  • screenshots;
  • links to relevant documentation, articles or previous issues.

Feature requests

Feel free to open a new issue on the Github issue tracker to discuss improvements to existing features, or request new features.

Security issues

  • TODO: email address for security reports

Submitting patches

Documentation

To contribute to the documentation, you need to setup a local development environment:

This will allow you to build the documentation locally and test your changes before submitting a Pull Request.

Bug Fixes

  • TODO: contribution workflow

Improvements and new features

Improvements to existing features, and requests for new features, must be discussed first by opening a new issue on the Github issue tracker.

This will ensure:

  • the improvement or feature is aligned with the project's goals;
  • the contribution's quality is on-par with the SparkleMuffin codebase.

Project Structure

Overview

The Go source code is broken down into several top-level packages:

.
├── cmd       # Command-line application
├── internal  # Private packages and test helpers
└── pkg       # Domain packages

cmd - Command-line application

cmd
└── sparklemuffin
    ├── command    # Command-line application commands and sub-commands (CLI parser)
    ├── config     # Configuration utilities
    └── main.go    # Command-line entrypoint

internal - Application-specific and private packages

internal
├── hash            # Cryptographically secure hash helpers
├── http            # HTTP servers: metrics, Web application
├── paginate        # Pagination utilities
├── rand            # Cryptographically secure pseudo-random helpers
├── repository
│   └── postgresql  # PostgreSQL database persistence layer (repository)
├── test            # Helpers for unitary and integration tests
└── version         # Version detection utilities

pkg - Domain packages

pkg
├── bookmark  # Web bookmark management
├── feed      # Feed subscription management
├── session   # User session persistence
└── user      # User and permission management

Documentation Structure

Markdown sources

The documentation is a static Website generated from Markdown files using mdBook.

The documentation resources are part of the SparkleMuffin repository, and located under the docs/ directory:

docs/
├── book       # Generated Website (not tracked in Git)
├── book.toml  # mdBook configuration
└── src        # Markdown source files

Sections and page categories

There are two main sections:

  • a User Guide that showcases SparkleMuffin's features and how to use them;
  • a Developer Guide that provides information on how SparkleMuffin works, and how to contribute to the project.

Pages are organized into four categories, according to the Diátaxis approach:

  • Tutorials: learning-oriented lessons that take you through a series of steps to use a feature;
  • How-to Guides: practical step-by-step guides to help you achieve a specific goal;
  • Reference Guides: details about how SparkleMuffin works;
  • Concept Guides: thoughts and reflections about how why things work the way they do.

Reference

Development Tools

Git

The source code is tracked using the Git Source Code Management tool, and available on Github at github.com/virtualtam/sparklemuffin.

To get started with using Git and Github:

Go

SparkleMuffin is mainly written with the Go programming language.

See go.mod for the minimum version of Go required by SparkleMuffin.

Linux

The recommended way of installing Go is via your Linux distribution's package manager.

macOS

The recommended way of installing Go is via the Homebrew community packages:

$ brew install go

Windows

The recommended way of installing Go is via winget:

$ winget install --id=GoLang.Go

Manual installation (advanced users)

To install a specific version of Go, see:

Docker

Docker is used to:

  • Package the application as easy-to-run Docker images;
  • Run database integration tests with Testcontainers;
  • Spin a local development environment with Docker Compose

A recent version of Docker is required to build Docker images locally, as we leverage:

GNU Make

A Makefile is provided for convenience to help running tests, linters, generate documentation and spin local development environments.

mdBook

mdBook is used to generate a static HTML documentation from Markdown files.

Watchexec

watchexec is used to live-reload the development server when source files have been changed on the disk.

Continuous Integration

Github Actions Workflows

CI Workflow

This workflow runs when:

  • new commits are pushed to the main Git branch;
  • new Git tags are pushed;
  • Pull Requests are created or updated.

It runs all continuous integration tasks:

  • Documentation build;
  • SQL linter (static code analysis);
  • Go linters (static code analysis);
  • Go unitary and integration tests;
  • Go build.

Docker Workflow

This workflow runs when:

  • new commits are pushed to the main Git branch;
  • new Git tags are pushed.

It builds and tags the SparkleMuffin production Docker images, and pushes them to the Github Container Registry (GHCR) at ghcr.io/virtualtam/sparklemuffin.

Documentation workflow

This workflow runs when:

  • new commits are pushed to the main Git branch;
  • new Git tags are pushed;
  • Pull Requests are created or updated.

It generates the HTML documentation with mdBook.

When new commits are pushed to the main Git branch and the CI workflow is successful, the documentation is uploaded to Github Pages and can be accessed here: SparkleMuffin Documentation.

Database

  • TODO: PostgreSQL database
  • TODO: migrations

Netscape Bookmark Parser

The Netscape Bookmark File Format is a format commonly used by Web browsers and Web bookmarking applications to export and import bookmarks.

It has a very loose specification (i.e. no DTD nor XSL Stylesheet), and may be assimilated to XML "with some quirks":

  • some elements have an opening tag, but no closing tag:
    • <DT> items;
    • <DD> item descriptions;
  • some elements have an opening and closing tag:
    • <A>...</A> bookmarks;
    • <H1>...</H1> export title;
    • <H3>...</H3> folder name;
  • some elements have surprising opening and closing tags:
    • <DL><p>...</DL><p> item lists;
  • depending on the implementation:
    • elements may (or may not) be capitalized;
    • some elements may (or may not) be nested;
    • some attributes may (or may not) be present.

Example Netscape Bookmark Document

<!DOCTYPE NETSCAPE-Bookmark-file-1>
<!-- This is an automatically generated file.
     It will be read and overwritten.
     DO NOT EDIT! -->
<TITLE>Bookmarks</TITLE>
<H1>Bookmarks</H1>
<DL><p>
    <DT><H3 ADD_DATE="1622567473" LAST_MODIFIED="1627855786">Favorites</H3>
    <DD>Add bookmarks here
    <DL><p>
        <DT><A HREF="https://domain.tld" ADD_DATE="1641057073" PRIVATE="1">Test Domain</A>
        <DT><A HREF="https://test.domain.tld" ADD_DATE="1641057073" LAST_MODIFIED="1646172586" PRIVATE="1">Test Domain II</A>
        <DD>Second test
    </DL><p>
</DL><p>

Go Parser

SparkleMuffin uses virtualtam/netscape-go to parse (unmarshal) and export (marshal) bookmarks using the Netscape Bookmark File Format.

This allows users to import or synchronize their existing bookmarks to SparkleMuffin, and to export them for usage with another bookmarking service.

virtualtam/netscape-go is provided as a standalone library in the hope other users may find it useful.

It leverages:

OPML Feed Subscription Parser

Outline Processor Markup Language (OPML) is a format commonly used by feed aggregators and feed readers to export and import subscriptions to Atom and RSS feeds.

It has a permissive specification, and each feed aggregator or reader may:

  • specify extra attributes;
  • use non-standard attributes or attribute formats (e.g. to format dates and time);
  • use a nested structure to represent subscriptions, categories and directories.

Specifications

Example OPML Document

<?xml version="1.0" encoding="UTF-8"?>

<opml version="1.0">
    <head>
        <title>My subscriptions in feedly Cloud</title>
    </head>
    <body>
        <outline text="Programming" title="Programming">
            <outline type="rss" text="Elixir Lang" title="Elixir Lang" xmlUrl="https://feeds.feedburner.com/ElixirLang" htmlUrl="http://elixir-lang.org"/>
            <outline type="rss" text="Python Insider" title="Python Insider" xmlUrl="https://feeds.feedburner.com/PythonInsider" htmlUrl="https://pythoninsider.blogspot.com/"/>
        </outline>
        <outline text="Games" title="Games">
            <outline type="rss" text="Vintage Story" title="Vintage Story" xmlUrl="https://www.vintagestory.at/blog.html/?rss=1" htmlUrl="https://www.vintagestory.at/blog.html/"/>
        </outline>
    </body>
</opml>

Go Parser

SparkleMuffin uses virtualtam/opml-go to parse (unmarshal) and export (marshal) feed subscriptions using the OPML file format.

This allows users to import or synchronize their existing subscriptions to SparkleMuffin, and to export them for usage with another feed aggregator or feed reader.

virtualtam/opml-go is provided as a standalone library in the hope other users may find it useful.

Conceptual Guides

Big-picture explanations of higher-level SparkleMuffin concepts. Most useful for building understanding of a particular topic.

Feed polling and caching

As SparkleMuffin periodically makes HTTP requests to update Atom and RSS feeds, we need to ensure:

  • we do not put unnecessary load on the remote servers;
  • we do not perform unnecessary database updates if the remote content has not changed.

To this effect, we leverage features from the HTTP specification to benefit from remote server caching, and perform additional checks on the feed content.

HTTP Conditional Requests

When responding to an HTTP request, a remote server may set the following headers:

  • ETag: the current entity tag for the selected representation (usually a hash of the feed data));
  • Last-Modified: a timestamp indicating the date and time at which the origin server believes the selected representation was last modified.

When present, we store these values in the database, and use them to set the following headers in subsequent requests:

  • If-None-Match: the value of the ETag header from the previous response;
  • If-Modified-Since: the value of the Last-Modified header from the previous response.

Depending on whether the feed has changed since the last request, the remote server will then respond with:

  • 200 OK: the content has changed, we update the feed and its entries;
  • 304 Not Modified: there are no changes, we only update the feed's ETag and Last-Modified headers.

Feed content hash

As a remote server may send a different ETag or Last-Modified value without the feed content being modified, or not send any of these headers at all, we:

  • compute and store a hash of the feed data using the xxHash non-cryptographic hash function;
  • compare the hash of the feed data with what we already have in the database;
  • return early if the hashes match, to avoid unnecessary database updates.

Reference

Feed caching

RFCs

HTTP Conditional Requests

Non-cryptographic hash functions