This is a simple, not-so-production-ready implementation of a sync engine. We use SQLite on the client and server, and use a simplified version of Replicache's "push" and "pull" model to sync changes.
This project is a teaching tool, not the bedrock of your next side hustle. I hope it becomes your springboard to implement a robust sync engine of your own!
Ensure you have Node.js v20 or later installed on your machine, along with pnpm for package management.
Then, install dependencies and start the development server:
pnpm install
pnpm dev
These are the primary files that drive the sync engine:
/
├── src/
│ ├── lib/ # Utilities to query the server and client databases
│ ├── pages/
│ │ ├── api/
│ │ │ ├── push.ts # Push mutations to the server
│ │ │ └── pull.ts # Pull updates from the server
│ ├── App.tsx # The client-side app
│ ├── migrations.ts # Migrations to initialize the database
│ └── queries.ts # Queries to read and write data on the server and client
This project uses a simplified version of Replicache's "push" and "pull" model to sync changes.
The queries.ts
file contains all of the queries and mutations that the client can perform. When a mutation is performed, the client will send the name of the mutation along with its arguments to the server.
push
is triggered by the client when the user performs runs a mutation (say, createIssue()
). The client sends the name and arguments of the mutation to the server, which then executes the mutation against its own database. The server will also add a mutation_log
entry to store a running history of mutations that have been performed.
pull
is run periodically by the client to sync changes from the server. The client will send a pull request to the server, which will return a list of mutations that have been performed since the client last pulled. The client will then execute these mutations against its own database.
To track which mutations have been performed, the server maintains a mutation_log
table and a lastLogId
cookie attached to each request. The server will check for any new logs since the lastLogId
, return them to the client, and update the lastLogId
to the end of the log.
The client will receive two important pieces of information when it performs a "pull":
mutations
: A list of mutations to apply to the client databaseflushCount
: The number of mutations sent by the client that the server has "acknowledged" since the last pull
To apply these changes, the client will perform a "rebase" of its local state on top of the latest server state. The client tracks two separate databases to perform this safely: a base db
modeled against the server state, and an optimisticDb
where all client mutations are applied.
When the client receives a "pull" response, it will:
- Update the base
db
with the list ofmutations
from the server - Flush acknowledged mutations from the client's running log of mutations
- Overwrite the
optimisticDb
with the updateddb
- Replay any mutations that were not acknowledged by the server (aka the remaining mutations from step 2) on top of the
optimisticDb
See lib/client.ts
for the full implementation.
To make this project simple, we made a lot of assumptions. You'll definitely need to address these before using this in production!
-
All data is accessible to the user, once you implement an authentication system. Organization-level and row-level permissions are another can of worms that frameworks like Zero can help you solve.
-
The user is online for the duration of the session.
push
is called whenever the client makes a change, and we don't implement any retry logic if it fails. You'll likely want a queue backed up by your SQLite database to ensure the client can retry after a network error. -
Breaking database migrations will cause data loss. We don't handle database migrations in this project, and suggest using the
DB_RESET
environment variable to force reset your database. To handle database changes properly, we suggest using the expand and contract model with a "versioning" system for your database schema. Zero provides a robust implementation of this.
This project can be deployed as a standalone server. We use Astro with the Node.js adapter, though you can use Astro's adapter system to deploy to other runtimes like Deno or Bun.
Run a production build with pnpm build
and start the production server with pnpm start
. You can also use pnpm preview
to run the production server locally.
Be sure to update your .env.prod
file with a path to wherever you would like your SQLite database to be stored. The example .env.prod
file assumes you will use /data/database.sqlite3
as the database path.
Here is a step-by-step guide to deploy to Railway:
- Clone this project to a new GitHub repository
- Create a new Railway project from this repository using the default settings (
pnpm build
to build andpnpm start
to run) - From the project dashboard, hit
cmd + k
and search for "Services -> Volumes" - Create a new volume and use the path
/data/