I got Mozilla's syncstorage-rs working; free yourself from python2

2023-03-27

You may not know this, but you can run your own sync server for Firefox Sync, which lets you keep all your sync’d data entirely on your own servers. The github:mozilla-services/syncserver has been what you use to do this for awhile now, but it’s written in Python 2, which is becoming increasingly non-existent on modern Linux distributions since Python 2 is end-of-life. The replacement is github:mozilla-services/syncstorage-rs, a new rust implementation. I had some trouble just following the docs, but I did some code diving to find the missing details. Here’s what I got working on my machine.

At the end of this post you should have a mariadb or mysql database up and running, and you’ll be syncing your firefox instances through your very own syncstorage-rs server. These docs expect you to know how to install, set up, and manage your own SQL database, because I’m not prepared to educate a beginner on how to do that. What you won’t have at the end of this post is your own instance of the mozilla authentication provider. That means that you’ll still log in through a Mozilla Firefox Account, using an existing email and password for that if you have one (though you could also delete and recreate your account if you wanted).

You can theoretically set up your own auth provider too, but I haven’t looked into how, because ultimately all your device data will go through your own server regardless of what auth provider you use. It’d be a benefit for device metadata privacy though, which would be nice. Regardless, setting up your own auth server is out of scope for this post.

I don’t have a good understanding of the stability of this software right now. I’m just some girl on the internet; I don’t work at mozilla. Use this at your own risk, and if it ends up deleting all your data, well hey that’s probably good to know but I can’t do anything about it.

Also, I’m writing this against syncstorage-rs commit hash f416d8a8c44c4c294f9403b40f136bda85bdd709 from March 7, 2023. These instructions may not be applicable to other versions, newer or older.

So let’s get on with it.

1. Dependencies

First decide on a URL and port that you want to run the syncserver on. I’ll be using

http://umbreon.eq:8000

In this post. Anywhere you see that, replace it with your syncserver’s location.

To build syncstorage-rs, you need the mysql or mariadb client, and you also need development headers for it. Some distros split up the client/server, some bundle them all together. You’ll also need the database server if you intend to run the database on the same system as syncstorage-rs. Here’s some example commands for various distros:

# gentoo, with +server (default) on mariadb
emerge dev-db/mariadb

# arch linux, client + server
pacman -S mariadb

# arch linux, just the client
pacman -S mariadb-clients mariadb-libs

# void linux
xbps-install mariadb

# debianish (mariadb-server is the server pkg)
apt install libmariadb-dev

# rhelish
dnf intsall mariadb-devel

You also need Python 3 and virtualenv (or whatever you prefer to use instead of virtualenv, if you have preferences).

Oh and rust of course. Go grab rust from rustup or wherever you get your rust toolchains.

Once you have rust, you’ll also need diesel to run the database migrations to initialize the database.

cargo install diesel_cli --no-default-features --features 'mysql'

2. Clone and Build

Now you need to clone the syncstorage-rs project and install the main server program:

git clone https://github.com/mozilla-services/syncstorage-rs
cd syncstorage-rs
cargo install --path ./syncserver --no-default-features --features=syncstorage-db/mysql --locked

In the same directory, set up a python virtualenv and install these two sets of requirements. You need to run the syncstorage server from the environment because it runs some python itself as part of the authentication process. You also need this environment to run some of the commands we’ll be using to populate the database.

virtualenv venv
source venv/bin/activate
pip3 install -r requirements.txt
pip3 install -r tools/tokenserver/requirements.txt

3. Initialize the Database

Now we can initialize the databases syncstorage / tokenserver (both bundled up in syncserver) need. This assumes you’ve already done basic setup on your mysql/mariadb database and have a root user you can access.

Create the user and databases:

SYNCSTORAGE_PW="$(cat /dev/urandom | base32 | head -c64)"
printf 'Use this for the syncstorage user password: %s\n' "$SYNCSTORAGE_PW"

# login as root sql user using whatever creds you set up for that
# this sets up a user for sync storage and sets up the databases
mysql -u root -p <<EOF
CREATE USER "syncstorage"@"localhost" IDENTIFIED BY "$SYNCSTORAGE_PW";
CREATE DATABASE syncstorage_rs;
CREATE DATABASE tokenserver_rs;
GRANT ALL PRIVILEGES on syncstorage_rs.* to syncstorage@localhost;
GRANT ALL PRIVILEGES on tokenserver_rs.* to syncstorage@localhost;
EOF

Run the migrations to setup the initial database structure. From the syncstorage-rs folder:

# syncstorage db
$HOME/.cargo/bin/diesel --database-url "mysql://syncstorage:${SYNCSTORAGE_PW}@localhost/syncstorage_rs" migration --migration-dir syncstorage-mysql/migrations run

# tokenserver db
$HOME/.cargo/bin/diesel --database-url "mysql://syncstorage:${SYNCSTORAGE_PW}@localhost/tokenserver_rs" migration --migration-dir tokenserver-db/migrations run

Add the sync endpoint to the services table in the tokenserver db:

mysql -u syncstorage -p"$SYNCSTORAGE_PW" <<EOF
USE tokenserver_rs
INSERT INTO services (id, service, pattern) VALUES
    (1, "sync-1.5", "{node}/1.5/{uid}");
EOF

Now you need to add a “node”. A node is any instance of syncserver to which a client can be allocated. You probably only want one node and that node is the server we’re setting up literally right now. You’ll also specify the user capacity, which indicates how many separate firefox accounts can use your server to sync. If it’s just you using this, you could set this to 1.

# the 10 is the user capacity.
SYNC_TOKENSERVER__DATABASE_URL="mysql://syncstorage:${SYNCSTORAGE_PW}@localhost/tokenserver_rs" \
    python3 tools/tokenserver/add_node.py \
    http://umbreon.eq:8000 10

4. Set up your config file

There’s a sample config file in config/local.example.toml, but we need to change most of the URLs because we want to run against mozilla’s prod environment instead of their staging environment. Rather than tell you how to edit that, just run this command to generate a good file.

MASTER_SECRET="$(cat /dev/urandom | base32 | head -c64)"
METRICS_HASH_SECRET="$(cat /dev/urandom | base32 | head -c64)"
cat > config/local.toml <<EOF
master_secret = "${MASTER_SECRET}"

# removing this line will default to moz_json formatted logs
human_logs = 1

host = "localhost" # default
port = 8000 # default

syncstorage.database_url = "mysql://syncstorage:${SYNCSTORAGE_PW}@localhost/syncstorage_rs"
syncstorage.enable_quota = 0
syncstorage.enabled = true
syncstorage.limits.max_total_records = 1666 # See issues #298/#333

# token
tokenserver.database_url = "mysql://syncstorage:${SYNCSTORAGE_PW}@localhost/tokenserver_rs"
tokenserver.enabled = true 
tokenserver.fxa_email_domain = "api.accounts.firefox.com"
tokenserver.fxa_metrics_hash_secret = "${METRICS_HASH_SECRET}"
tokenserver.fxa_oauth_server_url = "https://oauth.accounts.firefox.com"
tokenserver.fxa_browserid_audience = "https://token.services.mozilla.com"
tokenserver.fxa_browserid_issuer = "https://api.accounts.firefox.com"
tokenserver.fxa_browserid_server_url = "https://verifier.accounts.firefox.com/v2"
EOF

5. Securing your connections

I personally don’t feel like leaving this thing running on the open internet for anyone to use. You open yourself up to all sorts of fun possibilities like your hard drive filling up from other peoples’ data, or someone discovering a vulnerability in the service and using is to hack your server. I’m running mine on a local network with my own VPN setup to make that work, but this is also a good application for tailscale if you use that (I don’t). I heard tailscale recently got beta support for custom OIDC providers, neat!

The config file generated by the command I gave you above restricts the server to run on localhost so that you don’t get a server open to the entire world just by copy-pasting commands out of this post. You can set it to a specific IP to listen on a particular instance, but it’ll be serving over unencrypted HTTP, so you only really want to do this if you’re putting it on a VPN interface instead of an internet-accessible IP. If you want TLS encryption, reverse proxy it behind nginx or caddy or something like that.

6. Run the server

It’s finally time! Make sure that whenever you run the syncserver you’re doing it from the python virtualenv so it can run the python needed for the authentication.

~/.cargo/bin/syncserver --config=config/local.toml

6.1 A Note on Time

The authentication process is very sensitive to time. If your server is more than a couple seconds behind what the global NTP network agrees the current time is, authentication will just silently fail, and you won’t know why your browser won’t authenticate correctly, it’s very confusing to debug. The root cause is that when the sync server’s time is behind, the JWT token will be considered valid in the future, but not now. The sync server silently eats this error:

tokenserver-auth/src/verify.py

        except (ClientError, TrustError):
                    return None

7. Configure Firefox

First, log out of Firefox Sync if you’re logged in. I’m not 100% sure if this is necessary, but it’s what I did. Then open about:config.

Set identity.sync.tokenserver.uri to http://umbreon.eq:8000/1.0/sync/1.5, replacing http://umbreon.eq:8000 with your sync server’s location.

Restart Firefox.

Log in to Firefox Sync like normal. Then configure another Firefox the same way and log in there too. If everything goes well, data should transfer over! If not everything goes well, then you might see some things sync and not others, or nothing happen at all.

If you need to debug, about:sync-log is your friend.

8. Make it persistent

Now you should probably set up a system service to start it up automatically at boot. I’ll leave that for you to figure out how to do, because statistically you’re probably on a systemd-based system, but I don’t have any of those handy to try out a systemd service. Here’s some resources that can get you started though.

Remember, whatever script you write to start the service up, it needs to activate the virtualenv first! And you probably want to run it from the syncstorage-rs folder as the working directory, though I’m not 100% sure that’s necessary.

systemd - arch linux wiki
openrc - openrc docs
runit - runit docs

My friend also sent me this sample systemd service from her setup, which you can maybe adapt to your needs.

[Unit]
Description=Mozilla Firefox Sync
Wants=mysql.service
After=network.target mysql.service

[Service]
Environment="VIRTUAL_ENV=/path/to/syncstorage-rs/venv"
ExecStart=/path/to/syncstorage/binary --config=/path/to/config.toml
Restart=on-abort

User=syncstorage
Group=syncstorage
UMask=007

NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true

[Install]
WantedBy=multi-user.target