Architecture

Universe separates orchestration concerns across two node roles, Master and Wrapper, connected by a Hazelcast cluster. The Master owns all authoritative state and exposes a REST API; Wrappers own nothing persistently but execute every concrete action: copying templates, replacing variables, and managing OS processes. Both roles run from the same fat JAR, and a single configuration flag decides which role a node assumes at startup.

Master / Wrapper topology

When isMasterNode: true, the startup sequence in UniverseApplication registers a ResilienceMembershipListener, loads all instance configurations from ./configuration/ into the Hazelcast configurations map, starts the Ktor HTTP server, and launches InstanceCountEnforcer to maintain minimumServiceCount guarantees. When isMasterNode: false, the node joins the cluster using the Master’s masterAddress:masterPort, registers the built-in runtime providers (tmux, screen, process), loads extensions, and begins listening for task objects dispatched over IExecutorService.

flowchart LR
subgraph master["Master Node"]
  direction TB
  api["Ktor REST API<br/>POST · PUT /api/instances"]
  hz["Hazelcast<br/>IMap state · IExecutor dispatch"]
  cmd["Console Commands<br/>cluster · instance · config"]
end
subgraph wrapper["Wrapper Node(s)"]
  direction TB
  router["TaskRouter · IExecutor<br/>routes deploy / stop / sync"]
  tmpl["Template Manager<br/>copy tree · replace %VARIABLE%"]
  runtime["RuntimeProvider<br/>screen · tmux · process · docker"]
end
master ==>|Hazelcast cluster| wrapper
style master stroke:#7aa2f7,stroke-width:1.5px
style wrapper stroke:#7dcfff,stroke-width:1.5px

Component reference

Hazelcast task dispatch

Task objects are defined in the api module and serialised to JSON strings before submission to IExecutorService. The full dispatch path for a POST /api/instances request is:

1. Ktor route handler receives {"configurationName": "default"}.
2. Master selects a target Wrapper from Configuration.nodes (matched against live Hazelcast members).
3. Master writes a new InstanceInfo with state = CREATING into the instances map.
4. Master wraps a DeployInstanceTask as a UniverseCallableTask JSON payload.
5. TaskDispatcher.dispatch(task, targetMemberUUID) submits the callable to IExecutorService.
6. Hazelcast serialises and routes the callable to the target member.
7. Wrapper deserialises the payload in TaskDeserializer and extracts the DeployInstanceTask.
8. TaskRouter calls TemplateManager (copy + variable replace), then RuntimeProvider.start().
9. Wrapper writes state = ONLINE directly to the instances map via ClusterStateService.putInstance().

Instance lifecycle

1. CREATING

   The Master writes the InstanceInfo record to the instances map with state = CREATING immediately after selecting a Wrapper and before dispatching the task. The instance is already visible in GET /api/instances at this point.

2. ONLINE

   Once the Wrapper has copied templates, replaced variables, and started the process via RuntimeProvider.start(), it writes the updated InstanceInfo (with state = ONLINE, the allocated port, and the process PID) directly to the shared map via ClusterStateService.putInstance(). Wrappers send periodic heartbeats to keep lastHeartbeat current.

3. OFFLINE

   If a Wrapper disconnects from the cluster (network partition, crash, container stop), ResilienceMembershipListener on the Master marks every instance owned by that Wrapper as OFFLINE. The records are not deleted, preserving history and letting InstanceRecoveryService attempt re-attachment on reconnect.

4. STOPPED

   A deliberate stop via DELETE /api/instances/{id}, instance stop <id>, or PATCH /api/instances/{id}/lifecycle?target=stop dispatches a StopInstanceTask to the owning Wrapper. After RuntimeProvider.stop() completes, the state becomes STOPPED and the working directory in ./running/<id>/ is cleaned up (unless static: true).

Data flow: creating an instance

The full sequence when you run POST /api/instances:

sequenceDiagram
participant C as Client
participant M as Master · REST API
participant H as Hazelcast cluster
participant W as Wrapper
C->>M: POST /api/instances
Note over M: resolve config · select wrapper<br/>allocate port · write CREATING
M->>H: submit DeployInstanceTask
H->>W: deliver Callable to target member
Note over W: TaskDeserializer → TaskRouter
W->>W: TemplateManager copies tree · replaces vars
W->>W: RuntimeProvider.start()
W->>H: putInstance(state ONLINE)
H-->>C: InstanceInfo · ONLINE · port 25565

Port allocation strategy

PortAllocator checks three sources before committing to a port, preventing conflicts across concurrent deployments and external services:

1. In-memory allocations already claimed by this JVM during the current session.
2. Cluster-wide map scan over all ONLINE and CREATING instances, skipping their allocatedPort values.
3. OS-level probe that attempts a ServerSocket bind and a TCP connect on each candidate port to catch services already listening on the machine.

Module reference

Explore further