Context / the project: I’m building Protoverse, a tick-based space strategy game backend, mostly as a learning project to go deep on Go concurrency + a real multi-service architecture The idea: players send commands over gRPC (MoveFleet, Attack), those get queued in Redis, and a background tick engine resolves them every 10 seconds , applying movement/combat, writing results to Postgres, and publishing events over Redis pub/sub that get bridged into a live gRPC stream to clients.
It’s been a genuinely great way to actually use transactions, locking, and Redis primitives for real instead of just reading about them. Full loop (command to queue to tick to resolve and to live push) is working end to end with a single tick-engine instance.
Where I’m stuck: I want the tick engine to be safe to run as multiple replicas (for HA / horizontal scaling), and I’m not fully sure my current approach actually guarantees that, or if I’m missing an obvious failure mode.
What I’ve built so far
Popping due actions from Redis (atomic via Lua):
local key = KEYS[1]
local now = ARGV[1]
local limit = ARGV[2]
local due = redis.call('ZRANGEBYSCORE', key, '-inf', now, 'LIMIT', 0, limit)
if #due > 0 then
redis.call('ZREM', key, unpack(due))
end
return due
My reasoning: since Redis runs the whole script atomically, two tick-engine instances calling this at the “same time” can’t both get the same batch of action IDs — whichever call Redis processes first empties the set for those members before the second call runs.
Resolving each action (row-locked Postgres transaction):
func (r *Repository) TxGetFleetForUpdate(ctx context.Context, tx pgx.Tx, fleetID uuid.UUID) (*domain.Fleet, error) {
// ... SELECT ... FOR UPDATE on both the fleet and its ships
}
Each action resolution runs inside WithTx, and loads the fleet(s) involved with FOR UPDATE, so if two different actions in the same tick batch touch the same fleet, the second transaction blocks until the first commits.
My actual question: given the Lua script guarantees no two instances get the same action ID twice, is the FOR UPDATE locking on top of that actually necessary/sufficient, or redundant? My mental model is:
-
Lua script : no two instances ever process the same action
-
FOR UPDATE: protects against two different actions (e.g. two separate attacks against the same fleet, popped in the same batch by the same instance, or split across two instances) racing on the same fleet’s HP
Is #2 real, or am I overthinking it? And is there a cleaner/more idiomatic pattern for this than what I’ve got e.g. would most people reach for SELECT ... FOR UPDATE SKIP LOCKED instead, or a proper distributed lock (Redlock-style), or just… not bother and run a single tick-engine instance with a supervisor restarting it on failure (simpler, but no horizontal scaling)?
Repo’s public if it helps to see the actual code: github.com/Atul-Koundal/protoverse (see internal/tickengine/ and internal/queue/queue.go).
Appreciate any pointers especially if there’s a well-known pattern for this I should just be using instead of half-inventing one.