> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vibetunnel.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Hq

# HQ Mode Documentation

HQ (Headquarters) mode allows multiple VibeTunnel servers to work together in a distributed setup, where one server acts as the central HQ and others register as remote servers.

## Overview

In HQ mode:

* **HQ Server**: Acts as a central aggregator and router
* **Remote Servers**: Individual VibeTunnel servers that register with the HQ
* **Clients**: Connect to the HQ server and can create/manage sessions on any remote

## How It Works

### 1. Registration Flow

When a remote server starts with HQ configuration:

1. It generates a unique bearer token
2. Registers itself with the HQ using Basic Auth (HQ credentials)
3. Provides its ID, name, URL, and bearer token
4. HQ stores this information in its `RemoteRegistry`

### 2. Session Management

**Creating Sessions:**

* Clients must specify a `remoteId` when creating sessions through HQ
* HQ forwards the request to the specified remote using the bearer token
* The remote creates the session locally
* HQ tracks which sessions belong to which remote

The web New Session form lists registered machines and requires one as the target. Working-directory paths are interpreted on that selected machine. Because repository discovery, directory browsing, and Git worktree helpers are local filesystem operations, the form hides those HQ-router helpers in distributed mode; enter the machine's path directly.

**Session Operations:**

* All session operations (get info, send input, kill, etc.) are proxied through HQ
* HQ checks its registry to find which remote owns the session
* Requests are forwarded with bearer token authentication

### 3. Health Monitoring

* HQ performs health checks every 15 seconds on all registered remotes
* Health check: `GET /api/health` with 5-second timeout
* Failed remotes are automatically unregistered
* Remotes refresh their registration every 60 seconds and automatically rejoin after sleep or a network interruption

### 4. Session Discovery

* Remote servers watch their control directory for new sessions
* When sessions are created/deleted, remotes notify HQ via `/api/remotes/{name}/refresh-sessions`
* HQ fetches the latest session list from the remote and updates its registry

## Setup

### Running an HQ Server

```bash theme={null}
# Basic HQ server
vibetunnel-server --hq --username admin --password secret

# HQ server on custom port
vibetunnel-server --hq --port 8080 --username admin --password secret
```

### Running Remote Servers

```bash theme={null}
# Remote server registering with HQ
vibetunnel-server \
  --username local-user \
  --password local-pass \
  --hq-url https://hq.example.com \
  --hq-username admin \
  --hq-password secret \
  --name production-1

# For local development (allow HTTP)
vibetunnel-server \
  --hq-url http://localhost:4020 \
  --hq-username admin \
  --hq-password secret \
  --name dev-remote \
  --allow-insecure-hq

# Advertise an address that the HQ can resolve and reach
vibetunnel-server \
  --hq-url https://hq.example.com \
  --hq-username admin \
  --hq-password secret \
  --name tailnet-remote \
  --remote-url http://100.64.0.5:4020
```

### Command-Line Options

**HQ Server Options:**

* `--hq` - Enable HQ mode
* `--username` - Admin username for HQ access
* `--password` - Admin password for HQ access

**Remote Server Options:**

* `--hq-url` - URL of the HQ server
* `--hq-username` - Username to authenticate with HQ
* `--hq-password` - Password to authenticate with HQ
* `--name` - Unique name for this remote server
* `--remote-url` - HTTP(S) origin the HQ uses to connect back to this remote; defaults to the bind address or hostname
* `--allow-insecure-hq` - Allow HTTP connections to HQ (dev only)
* `--no-hq-auth` - Disable HQ authentication (testing only)

## API Endpoints

### HQ-Specific Endpoints

**List Remotes:**

```http theme={null}
GET /api/remotes
Authorization: Basic <base64(username:password)>

Response:
[
  {
    "id": "uuid",
    "name": "production-1",
    "url": "http://remote1:4020",
    "registeredAt": "2025-01-17T10:00:00.000Z",
    "lastHeartbeat": "2025-01-17T10:15:00.000Z",
    "sessionIds": ["session1", "session2"]
  }
]
```

**Register Remote (called by remotes):**

```http theme={null}
POST /api/remotes/register
Authorization: Basic <HQ credentials>
Content-Type: application/json

{
  "id": "unique-id",
  "name": "remote-name",
  "url": "http://remote:4020",
  "token": "bearer-token-for-hq-to-use"
}
```

**Refresh Sessions (called by remotes):**

```http theme={null}
POST /api/remotes/{remoteName}/refresh-sessions
Authorization: Basic <HQ credentials>
Content-Type: application/json

{
  "action": "created" | "deleted",
  "sessionId": "session-id"
}
```

### Session Management Through HQ

**Create Session on Remote:**

```http theme={null}
POST /api/sessions
Content-Type: application/json

{
  "command": ["bash"],
  "remoteId": "remote-uuid",  // Specify which remote
  "name": "My Session"
}
```

**All Standard Endpoints Work Transparently:**

* `GET /api/sessions` - Aggregates from all remotes
* `GET /api/sessions/:id` - Proxied to owning remote
* `POST /api/sessions/:id/input` - Proxied to owning remote
* `DELETE /api/sessions/:id` - Proxied to owning remote
* `GET /ws` (WebSocket) - Unified v3 terminal stream proxied/fanned-out via HQ

## Authentication Flow

1. **Client → HQ**: Standard authentication (Basic Auth or JWT)
2. **HQ → Remote**: Bearer token (provided by remote during registration)
3. **Remote → HQ**: Basic Auth (HQ credentials)

## WebSocket Support

* Unified terminal transport: `/ws` (WebSocket v3 framing)
* HQ keeps one upstream `/ws` connection per remote and fans out frames to clients
* Subscription flags are aggregated per session (stdout/snapshots/events)

## Implementation Details

### Key Components

**RemoteRegistry** (`src/server/services/remote-registry.ts`):

* Maintains map of registered remotes
* Tracks session ownership (which sessions belong to which remote)
* Performs periodic health checks
* Handles registration/unregistration

**HQClient** (`src/server/services/hq-client.ts`):

* Used by remote servers to register with HQ
* Handles registration and cleanup
* Manages bearer token generation

**Session Routes** (`src/server/routes/sessions.ts`):

* Checks if running in HQ mode
* For remote sessions, forwards requests using bearer token
* For local sessions, handles normally

**Control Directory Watcher** (`src/server/services/control-dir-watcher.ts`):

* Watches for new/deleted sessions
* Notifies HQ about session changes
* Triggers session list refresh on HQ

### Session Tracking

* Each remote maintains its own session IDs
* HQ tracks which sessions belong to which remote
* Session IDs are not namespaced - they remain unchanged
* The `source` field in session objects indicates the remote name

## Testing

The e2e tests in `src/test/e2e/hq-mode.e2e.test.ts` demonstrate:

1. Starting HQ server and multiple remotes
2. Remote registration
3. Creating sessions on specific remotes
4. Proxying session operations
5. WebSocket buffer aggregation
6. Cleanup and unregistration

## Limitations

* Remotes must be network-accessible from the HQ server
* If the default hostname is not resolvable from HQ, set `--remote-url` to a reachable origin such as a tailnet address
* Health checks use a fixed 15-second interval
* No built-in load balancing (clients must specify remoteId)
* Bearer tokens are generated per server startup (not persistent)

## Security Considerations

* Always use HTTPS in production (use `--allow-insecure-hq` only for local dev)
* Use HTTPS for `--remote-url` on untrusted networks because HQ sends the remote bearer token on callback requests
* Bearer tokens are sensitive - they allow HQ to execute commands on remotes
* Remote listing and registration responses expose only public machine metadata, never callback bearer tokens
* HQ credentials should be strong and kept secure
* Consider network isolation between HQ and remotes
* Remotes should not be directly accessible from the internet