Building a Distributed DevOps Platform Without etcd or Raft

I just shipped cluster mode for Kovanex — our self-hosted DevOps platform. Two nodes, gossip protocol, zero shared state. Here's what we built and why.

The Problem

Most distributed systems need a consensus layer. Kubernetes uses etcd. Nomad uses Raft. Docker Swarm uses Raft. All of them require a quorum — a minimum number of nodes that must agree before anything happens.

For a self-hosted DevOps platform that starts as a single binary on a single machine, this is overkill. We needed something that:

The Architecture

We went with a gossip-based approach (SWIM protocol via HashiCorp's memberlist library). Each node is fully independent:

Node A (prod)                    Node B (test)
┌──────────────────┐            ┌──────────────────┐
│ gRPC Server      │◄──gossip──►│ gRPC Server      │
│ Git SSH          │   :7946    │ Git SSH          │
│ kovadb (embedded)│            │ kovadb (embedded)│
│ CI/CD Runner     │            │ CI/CD Runner     │
│ Health Matrix    │            │ Health Matrix    │
│ Scheduler        │            │ Scheduler        │
└──────────────────┘            └──────────────────┘

No shared database. Each node embeds its own kovadb — a pure-Go document + graph + vector store, no separate process. Git repositories ARE the replication layer — a concept borrowed from Git's distributed nature itself.

Health Matrix — Not Just a Health Check

Most systems report "healthy" or "unhealthy". We built a time-quantized metrics matrix with weighted average convolution:

metric     │ weight │ Node A  │ Node B
───────────┼────────┼─────────┼────────
cpu        │  0.15  │   4.6%  │   2.1%
memory     │  0.15  │  23.1%  │  15.4%
disk       │  0.10  │  85.3%  │  62.1%
goroutines │  0.05  │     37  │     24
db_latency │  0.20  │   0ms   │   0ms
grpc_p95   │  0.15  │   0ms   │   0ms

Each metric is normalized to 0.0–1.0 (inverted — lower resource usage = higher score), then weighted and convolved into a single health score. Node A scores 88%, Node B scores 92%.

This score feeds directly into the task scheduler.

Task Scheduling — A Scoring Function

When a CI/CD task needs to run, the scheduler picks a node:

score = health(node) × capacity(node) × locality(node, repo) × label_match(node, task)

Four strategies: BEST_FIT (default), ROUND_ROBIN, LOCALITY (prefer repo owner), SPREAD (distribute evenly).

Production Test Results

We deployed the cluster across two nodes in the same datacenter:

Metric                  Node A   Node B
──────────────────────  ───────  ───────
Role                    full     full
Health score            88%      92%
Runner slots            4        2
Gossip convergence      <1s      <1s
Total cluster health    90%      —

The gossip layer converged in under 1 second. Health metrics are collected every 5 seconds. Failure detection thresholds: 30s suspected, 60s dead.

Client Failover

Both CLI and CI/CD runners support seed lists:

# CLI
KOVANEX_ADDR=node-a:9000,node-b:9000 kovanex-ctl status

# Runner
KOVANEX_SERVER=node-a:9000,node-b:9000

On disconnect, clients rotate to the next seed immediately (500ms), then pause 5s after exhausting all seeds. The Go SDK includes a gRPC custom resolver that discovers cluster nodes via ListNodes and sets up round-robin balancing automatically.

What's NOT Clustered

Being explicit about boundaries is just as important as the features:

The Code

17 RPCs in the ClusterService, ~2,500 lines of Go across 7 files:

The entire cluster layer is optional — set CLUSTER_NODE_ID in your environment to enable it. Without it, the platform runs as a single standalone node, exactly as before.

What's Next

Kovanex is an open-source, self-hosted DevOps platform (Go + gRPC, with an embedded kovadb store). 21 gRPC services, 190+ RPCs, built-in Vault, CI/CD, Git hosting, Kanban, and now distributed clustering. See the cluster architecture diagram.