Dev blog: One server decides

The problem

One truth, and it keeps time.

Keel puts many pilots in one world with a market, structures, and PvP. If any PC could decide its own hit or its own cargo, every fight and every price would sit one edited client away from a lie.

So the rule is short. The server owns hits, sectors, inventory, core, and which contacts a scanner can see. The client may animate a move. It may not decide a hit.

The second problem was time. Our first loop did its work and then slept a full 50 milliseconds, so the world ran slow and fell further behind as pilots joined. Our performance vote post said the same thing from the pilot's side: the game got laggier and laggier until it disconnected.

What the old loop cost

0.78xWorld speed with 10 pilots. A comment in the loop gives the number.

0.4xWorld speed with 25 pilots, from our vote post.

20 msAbout how long one password hash froze the world on a login, from a code comment.

12 msAbout how long the world waited on a save for each order, from a code comment.

What we wanted instead

Three rules for a server that is in charge.

We wrote these down before we tuned anything. Each one has code that holds it.

An order asks

The client sends orders such as move, attack, and stance. The server checks each one, decides what happens, and answers in the next snapshot. The route you preview is a guess. The path you get is the server's.

You get your own view

Each pilot gets a snapshot built for that pilot: the contacts in your sectors, the blips from your scanner, and the sectors you have charted. What the server does not send, the client cannot draw.

Time is the server's

The world steps every 50 milliseconds whether anyone clicks or not. Clients are told what time did. They do not set the pace.

What we built

A 50 ms step, ten snapshots a second, and a budget we can check.

The server is one Python process. One thread runs the world, and each connected client has a thread of its own for reading and writing. A load check holds the step to half its budget.

50 msThe fixed server step, 20 a second.

10Snapshots a pilot gets each second, one for every second step.

2Rounds a second that also carry the cold sections.

25 msThe load check's limit for the 95th percentile of one step, half the budget.

One second, ten snapshot rounds

12345678910

Flying: every round, cold on every fifth

HHHHCHHHHC

Docked and idle: every fifth round only

Schematic drawn for this post, not a capture. H is a round with the hot sections, which change often. C is a round that adds the cold sections, 48 keys that change rarely. An order sends a round at once, with the cold sections, on the next step. Which two rounds are cold differs from client to client, because each client starts at its own offset.

As a pilot lives it

You click. The server answers on the next step.

Here is one session from the pilot's seat, with what the server does at each point. How a shot lands is its own story. Here it is enough that the server decides every hit.

Dial and trustThe game opens TCP and starts TLS. It trusts one certificate, the one built into the client. A server it cannot match ends the handshake.
Log inThe login names the protocol and asks for a deflate stream and a ship cache. The server hashes the password outside its world lock and answers with one full snapshot.
Click a hexThe client sends a move order and draws a preview route. The server checks the sector and the fog route, sets the path, and answers with a line like the first one below. The path in the next snapshot is the real one.
Time passesEvery 50 ms step the server moves ships, runs NPCs, and decides each hit. Ten times a second it sends you what changed. Sound cues come from what changed between two snapshots.
The link stallsThe game pings every 2 seconds. Ten seconds of silence is a stall, and the game logs back in on its own. The server waits up to 60 seconds on a blocked send.

LogPath 2. Arrival about 3.2 s. Esc cancels the next jump.

Login pageThe secure connection failed. This client may not match the server's build. Get the newest build from the itch page.

LogThe connection stalled. Reconnecting.

Login pageConnection lost. Logging back in as soon as the server answers.

ServerToo many orders at once. Some were dropped.

ServerThis account logged in from another window.

The first line is the server's wording with its own numbers filled in, as the capture at the top of this post shows. The next three are the client's, from main.gd. The last two are the server's, from keel_server.py. The orders limit is 25 a second with a burst of 60.

Coming in 0.5.3

Remember me, without remembering your password.

Tick Remember me on the login card. The server sends one token, once, and keeps only its SHA-256 hash beside the account.
The game stores your pilot name and the token on your PC, and never the password. A later login sends the name and the token.
A token lasts 30 days, and an account keeps at most 8. A password reset drops all of them. A logout that asks to forget drops the one on that device.
A bad token counts as a failed login, so the same limits apply as for a password. Only a refusal that says the token is bad makes the game forget it, so a ban or a busy server does not wipe it.
The token is a key to your account on that PC until it expires. On a shared PC, leave the box unticked.

The token in numbers

30 daysHow long a token lasts.

8Tokens an account keeps, the oldest going first.

32 bytesOf randomness in each token.

0Passwords stored on your PC.

How it reaches us

Only one leg of the road is a tunnel.

Pilots do not dial our PC directly. This is the road, hop by hop, five stops and a pin, with the word that fits each one. The words matter, because they say who can read what.

Your gameDials a public name on port 443 over TCP and starts TLS. The name goes into the handshake so the next hop can route by it.the word: client
The public relayReads only that name and passes the bytes on. It does not decrypt them.the word: relay, not proxy
The link to the hostWireGuard carries the bytes from the relay to our PC. This is the only hop that earns the word tunnel.the word: tunnel
The local daemonHands the connection to the server on loopback and puts a PROXY header in front, so the server learns your real address. Nothing here opens your bytes.the word: forwarder
The serverReads the header within 5 seconds, then ends Keel's own TLS itself, within 10, with its own certificate and TLS 1.2 or newer.the word: endpoint
The pinThe client trusts only the certificate built into it. Keel's TLS runs from the first stop to the fifth, so the relay and the daemon carry bytes they cannot read.the word: end to end

HeaderThe PROXY header is where the word proxy belongs. A connection that skips it is dropped, except one from this PC's own loopback, which is how the host plays.

AddressThe server listens on loopback only. The header lets the sign-up, login, and connection limits see each pilot's own address, not the daemon's.

StallsA stalled relay to host leg can leave your socket open at the relay. That is why the game treats silence, and not an error, as the sign of a stall.

PinThe certificate lists the public name and was issued for 3650 days. Its key is not in git and stays on the host. A new key means a new client build for everyone.

Under the hood

A lock, a loop, and a set of marks.

The server is 49 Python files and 36,605 lines, and 13,804 of those lines are tests and checks. The network side is one file, 1,242 lines. The rules are world.py, 4,256 lines, and the modules mixed into it.

Networkkeel_server.py accepts sockets, reads the PROXY header, ends TLS, runs the 50 ms loop, and sends snapshots. Sending never holds the world lock.

Worldworld.py and its mixins hold the rules. handle takes one order, tick runs one step, and save and save_world write the SQLite file.

Viewsnapshot.py builds one pilot's view in 286 lines. It is where what a scanner can see is decided.

One order, from the wire to your screen

ReadA client thread reads a line, refuses one over 16,384 characters, and spends one of its order tokens._io, _spend, _on_line
HashA login password is hashed outside the world lock, with PBKDF2 at 120,000 rounds, so a login never freezes the world._prehash
DecideUnder the one world lock the server runs the order and marks the client dirty.handle, _op_done
StepThe loop steps the world every 50 ms, and the step decides every hit._loop, tick, _tick_shared
BuildA snapshot round builds the pilot's view, then drops every section the client already holds.snapshot, _push_snapshot
SendThe client's own thread encodes, deflates, and sends. A newer snapshot replaces an unsent one.queue, _flush

The main functions

Server._loopkeel_server.py

A fixed 50 ms step. It catches up owed time by at most 3 steps, and past that the world slows down instead of spiralling. It saves every online pilot and the shared world every 10 seconds.

World.tick, _tick_sharedworld.py

Splits the time into 50 ms steps, runs each online account, then the shared world: NPCs, storms, incursions, and the market. The world simulates while at least one account is logged in.

World.handleworld.py

Every in-game order comes through this one function. It finds the op by name, checks it, and changes the world. Login, logout, and the hello are handled one level up, in the server.

World.snapshotsnapshot.py

Builds one pilot's view. Hot sections come every round and the cold ones only when asked.

Server._push_snapshotkeel_server.py

Marks each section as bytes and drops the ones the client already has. A code comment puts a marshal mark at about 6 times cheaper than a sorted JSON dump. Ship fits, stats, and slots are cached per ship.

Client.queue, Server._flushkeel_server.py

One snapshot waits per client, and a newer one merges into it. The client's thread encodes and deflates it. A send blocked for 60 seconds drops the client.

read_proxy_header, proxy_addresskeel_server.py

Reads PROXY version 1 or 2 and returns the real address. Anything that is not a header raises, so a connection that skipped the relay is dropped.

World._disk, save, save_world, _backupworld.py

One kept SQLite connection in WAL mode. Each account is one JSON row and the shared world is one more. A backup is copied at start, and three are kept.

_heartbeat_loop, _write_heartbeatkeel_server.py

Its own thread. Every 20 seconds it writes a small file by temp file and rename, so a reader never sees half a file. Its fields include the time, the start time, the version, and the online count. A watcher reads it each minute and calls the server down after 90 seconds of silence.

issue_remember, login_rememberaccess.py

Coming in 0.5.3. A random token, only its hash stored, and a constant-time compare on use.

_tick_connection, _heard_nothingmain.gd

The client's connect, TLS, and stall logic. It trusts one certificate, calls 10 seconds of silence a stall, and retries with a pause that doubles up to 4 seconds.

Four choices we are happy with

One lock, and sending stays outside it.A slow client cannot stall the world. Each client has its own thread and at most one pending snapshot.

A snapshot is a patch.The server drops every section the client already holds. The client merges what arrives into what it has, key by key.

Steps are fixed and catch-up is capped.The world keeps time, and there is a hard limit on how far behind it can fall before it slows down.

The save is plain.One JSON row per account in SQLite, a schema number of 6 for accounts and 1 for the world, and three backups. A test runs the shipped seed save through the migrations.

The client side

The server decides. The client draws and asks.

main.gd merges each snapshot into the last one, key by key, and keeps each ship's fit, stats, and slots from the last time they were sent.
Sound cues are made by comparing two snapshots. A comment in main.gd says it plainly: the server decided every event.
The route preview is a client copy of the server's route rule, so a preview can be wrong. The path in the snapshot is the one that counts.
The top bar pings every 2 seconds and shows the round trip. It turns red with an exclamation mark from 250 ms.

What the code looks like in numbers

22,801Lines of server code outside the tests and checks.

48Cold keys in the snapshot.

16,891Lines in the 33 client GDScript files, 6,016 of them in main.gd.

3Startup backups of the save kept.

What the tests prove

What is pinned, and what is not.

The unit tests and the client self-test run on every commit. The longer checks run by hand before a release, because they take minutes. The counts below come from reading the files today. Each check script prints its own total when it runs, and we did not run them for this post.

548Unit test methods in test_alpha.py, against 380 at the live build.

38Checks in alpha_check: a real server on a scratch save and socket pilots, over TLS when the pair is on the machine.

98Check calls in playtest_check: four plain TCP pilots through 18 areas. A few sit in loops, so a run counts more.

111Places where the client self-test can add a failure.

Pinned

ProxyHeaderTest: version 1, version 2, local, and garbage headers. A host client on loopback needs no header. A proxied tester gets their own sign-up limit.
test_option_b_deflate_after_the_marker_and_hash_outside_the_lock: the plain deflate marker, then a zlib stream that decodes to a snapshot, and a stale password check that is refused.
test_a_remember_token_is_issued_hashed_used_expired_and_revoked: only the hash is stored, a ban still refuses it, an expired token fails, and a password reset drops them all. Its neighbour pins the limit of 8 devices.
HeartbeatTest, ConnectionLogTest, and HealthcheckTest: the beat fields, which save writes a beat, the stall note, log rotation, and a certificate pin that must match the client's byte for byte.
test_the_connection_close_saves_the_user_then_notes_the_drop and test_shipped_seed_migrates_to_the_current_schema: a drop saves the pilot, and the shipped seed save migrates to the current schema.
alpha_check: a bad name and a duplicate name are refused, pilots in one sector see each other, a dropped pilot reconnects to the same state, kick and ban work, and after a restart the account and its purchase are still there.

Not pinned, or thin

No test crosses the real relay. The health check sends one hello through it when the host or the watcher runs it. The tests only run it against a private server or a stand-in.
No unit test opens a TLS connection, and the health check tests say in their own words that the TLS path was checked by hand. alpha_check uses TLS only when the certificate pair is on the machine, and the key is not in git. playtest_check is plain TCP.
No test names the hot and cold marks. The socket pilots merge each snapshot into the last, so a section that never re-sent would look stale, but nothing counts the bytes saved.
The load check runs in one process, with no sockets, TLS, or deflate. It builds a full snapshot for every pilot on every step, which is twice the live rate. That is harder on the world and easier on the wire.
The last p95 in our commit log is 19.9 ms, from 2026-10-20, against a limit of 25. That is a margin of about 5 ms. We have not put a number from today in this post.
We found no test that kills the server mid-save. Our restarts are clean shutdowns, so a hard crash can lose the seconds since the last save, and we have not measured how often that happens.
Our vote post estimated 100 to 110 pilots, from tests on a fast desktop. The server caps connections at 64, and our checks use 3, 4, and 8 pilots.

What we learned and what is next

Small lessons from keeping a world on time.

Cap the time you owe.The first loop slept after its work and fell behind as pilots joined. The fixed step catches up by at most 3 steps. Past that the world slows, which is honest, instead of racing to catch up.

Never hold the lock for slow work.A password hash and a save were both taken while the world lock was held. We hash outside the lock now and keep one SQLite connection open in WAL mode. The 0.5.1 patch notes put the result at about 90 percent less server work.

Silence is the signal.A relay can keep your socket open after its leg to the host has died, so no error arrives. The client pings every 2 seconds, and since 0.5.2.1 it calls 10 seconds of quiet a stall. The 0.5.2.1 notes say the new connection log is there to track down what is left of the drops.

Where we would push next

The remember token and its checkbox on the login card are coming in 0.5.3.
A test that pins the cadence constants and the hot and cold marks. Our draft test spec already lists a pin on the constants as to write.

Time a move from click to snapshot under the load check. The same draft asks for 100 ms and leaves open where to measure it.
Bring the load check's header up to date, because it still says 20 snapshots a second. The plan for 0.5.3 also asks for a load check pass with the biggest incursions, Layer 6 and Rift 6, before it ships. The option that forces one landed on 2026-10-22.

Join the open alpha

Come and put the server to work.

Keel is free to play on PC, and the open alpha is where the server earns its keep. Find us in the Keel Discord, tell us what a bad link or a slow evening did to your session, and file a report from inside the game when something looks wrong.

Players whose reports become fixes are credited in game. If the game says it is reconnecting, note the time and tell us. The time is what lets us find your drop in the host's connection log.