Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions pages/clustering/high-availability.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,11 @@ recommended configuration patterns.
Recommended practices for running a robust, reliable, and well-observed HA
deployment.

### [Coordinator authentication](/clustering/high-availability/coordinator-authentication)

Secure the cluster's control plane with single sign-on, Raft-replicated
coordinator roles, and the `COORDINATOR_READ` / `COORDINATOR_WRITE` privileges.

### [HA commands reference guide](/clustering/high-availability/ha-commands-reference)

A complete reference of all commands for managing coordinators, registering
Expand Down
1 change: 1 addition & 0 deletions pages/clustering/high-availability/_meta.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
export default {
"how-high-availability-works": "Under the hood",
"querying-the-cluster-in-high-availability": "Querying the cluster in HA",
"coordinator-authentication": "Coordinator authentication",
"setup-ha-cluster-docker": "Set up HA cluster with Docker",
"setup-ha-cluster-docker-compose": "Set up HA cluster with Docker Compose",
"setup-ha-cluster-k8s": "Set up HA cluster with K8s",
Expand Down
591 changes: 591 additions & 0 deletions pages/clustering/high-availability/coordinator-authentication.mdx

Large diffs are not rendered by default.

183 changes: 180 additions & 3 deletions pages/clustering/high-availability/ha-commands-reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,17 @@ LEADERSHIP`](#yield-leadership), which must be run directly on the current
leader.
</Callout>

<Callout type="info">
From Memgraph 3.13, coordinators can enforce privileges on these queries. Every
query on this page requires either `COORDINATOR_READ` (read-only introspection)
or `COORDINATOR_WRITE` (everything mutating). A session that connected with
basic auth carries full `COORDINATOR_WRITE`, so nothing changes unless you
enable [SSO on coordinators](/clustering/high-availability/coordinator-authentication).
See the [privilege
reference](/clustering/high-availability/coordinator-authentication#which-privilege-each-query-requires)
for the per-query mapping.
</Callout>

### `ADD COORDINATOR`

Adds a coordinator to the cluster.
Expand Down Expand Up @@ -332,7 +343,9 @@ informational notification that the request was submitted.
- Failover of data instances is not triggered, but the new leader recomputes
cluster state, so a cluster that was already missing a MAIN can fail over as
part of the leadership change.
- No privilege is required to run this query on the coordinators.
- Requires `COORDINATOR_WRITE`. A basic-auth session carries it implicitly; an
[SSO session](/clustering/high-availability/coordinator-authentication) needs
a role that has been granted it.

{<h4 className="custom-header"> Typical use cases </h4>}

Expand Down Expand Up @@ -553,7 +566,11 @@ survives coordinator restarts and leader re-elections, and is honored across
failovers: a newly promoted MAIN comes up read-only when the cluster is in
read-only mode, instead of silently accepting writes.

No privilege is required to run this query on the coordinators.
`SET COORDINATOR SETTING` requires `COORDINATOR_WRITE` and `SHOW COORDINATOR
SETTINGS` requires `COORDINATOR_READ`. A basic-auth session carries both
implicitly; an [SSO
session](/clustering/high-availability/coordinator-authentication) needs a role
that has been granted them.

<Callout type="info">
Enabling read-only mode blocks **all write sources** on the MAIN — user Cypher
Expand All @@ -575,10 +592,170 @@ cluster self-heals to the requested state.
</Callout>


## Coordinator role and privilege management

<Callout type="info">
These queries are **Memgraph Enterprise** features and require a valid license.
They exist so that [SSO
identities](/clustering/high-availability/coordinator-authentication) have
something to map onto. Coordinators have no users — only roles.
</Callout>

Coordinator roles are stored in the **Raft-replicated cluster state**, not in
the auth store, so they survive restarts, follower catch-up and leader failover.
Like the cluster registration queries, all of these can be run on any
coordinator: writes are transparently forwarded to the leader and committed
through the Raft log, and `SHOW ROLES` / `SHOW PRIVILEGES FOR ROLE` are strong
reads served by the leader.

### `CREATE ROLE`

Creates a coordinator role. New roles start with **no** privileges.

```cypher
CREATE ROLE ifNotExists? roleName;
```

{<h4 className="custom-header"> Behavior & implications </h4>}

- Errors if the role already exists, unless `IF NOT EXISTS` is given.
- The role name must match the `--auth-user-or-role-name-regex` pattern,
otherwise the query fails with `Invalid role name '<name>'.`
- Requires `COORDINATOR_WRITE`.

{<h4 className="custom-header"> Example </h4>}

```cypher
CREATE ROLE dba;
CREATE ROLE IF NOT EXISTS analyst;
```

### `DROP ROLE`

Removes a coordinator role.

```cypher
DROP ROLE roleName;
```

{<h4 className="custom-header"> Behavior & implications </h4>}

- Errors with `Role '<name>' doesn't exist.` if the role is not present.
- Takes effect on **already-connected sessions immediately** — privileges are
re-derived from the committed role set on every query, so a session that
authenticated with the dropped role is denied its next privileged query
without needing to reconnect.
- Requires `COORDINATOR_WRITE`.

{<h4 className="custom-header"> Example </h4>}

```cypher
DROP ROLE analyst;
```

### `SHOW ROLES`

Lists the coordinator roles. Returns one `role` column, name only.

```cypher
SHOW ROLES;
```

{<h4 className="custom-header"> Behavior & implications </h4>}

- Strong read served by the leader. If no leader can be reached, the query fails
rather than returning possibly-stale local state.
- Requires `COORDINATOR_READ`.

### `GRANT` / `REVOKE` coordinator privileges

Grants or revokes a coordinator privilege on a role. Coordinators support
exactly two privileges: `COORDINATOR_READ` and `COORDINATOR_WRITE`, where
`COORDINATOR_WRITE` is a superset of `COORDINATOR_READ`.

```cypher
GRANT ( ALL PRIVILEGES | COORDINATOR_READ | COORDINATOR_WRITE [, ...] ) TO ROLE? roleName;
REVOKE ( ALL PRIVILEGES | COORDINATOR_READ | COORDINATOR_WRITE [, ...] ) FROM ROLE? roleName;
```

{<h4 className="custom-header"> Behavior & implications </h4>}

- `GRANT ALL PRIVILEGES` grants **both** coordinator privileges;
`REVOKE ALL PRIVILEGES` removes both.
- Errors with `Role '<name>' doesn't exist.` if the role is not present.
- Only `COORDINATOR_READ` and `COORDINATOR_WRITE` may appear in the privilege
list. Any other privilege, `DENY` in any form, a `USER` target, fine-grained
access control (`ON NODES` / `ON EDGES`), property permissions and
`GRANT DATABASE` are all rejected on a coordinator.
- Like `DROP ROLE`, a `REVOKE` applies to already-connected sessions on their
next query.
- Requires `COORDINATOR_WRITE`.

{<h4 className="custom-header"> Example </h4>}

```cypher
GRANT ALL PRIVILEGES TO dba;
GRANT COORDINATOR_READ TO analyst;
REVOKE COORDINATOR_WRITE FROM analyst;
```

### `SHOW PRIVILEGES FOR ROLE`

Reports the privileges granted to a coordinator role, one per row.

```cypher
SHOW PRIVILEGES FOR ROLE? roleName;
```

{<h4 className="custom-header"> Behavior & implications </h4>}

- A role with no grants returns no rows.
- The trailing `ON MAIN | CURRENT | DATABASE <db>` clause is **rejected** —
coordinators have no databases.
- `SHOW PRIVILEGES FOR USER <user>` is rejected — coordinators have no users.
- Strong read served by the leader.
- Requires `COORDINATOR_READ`.

{<h4 className="custom-header"> Example </h4>}

```cypher
SHOW PRIVILEGES FOR ROLE dba;
```

```plaintext
+---------------------+
| privilege |
+---------------------+
| COORDINATOR_READ |
| COORDINATOR_WRITE |
+---------------------+
```

### `SHOW CURRENT USER` and `SHOW CURRENT ROLE`

Report the identity of the current session.

```cypher
SHOW CURRENT USER;
SHOW CURRENT ROLE;
```

{<h4 className="custom-header"> Behavior & implications </h4>}

- **No privilege and no license are required** — these are self-service queries
that reveal only the session's own identity.
- `SHOW CURRENT USER` returns the principal the SSO module reported. It is
session-local and works even when the leader is unreachable. A basic-auth
passthrough session returns `null`.
- `SHOW CURRENT ROLE` returns the session's roles filtered against the leader's
committed role set, so a dropped role stops being reported. A basic-auth
passthrough session has no roles and returns `null`.

## Error handling

If a Raft log commit fails for any cluster operation (register, unregister,
promote, demote, add coordinator), the error message will indicate:
promote, demote, add coordinator, role or privilege change), the error message
will indicate:

> Writing to Raft log failed. Please retry the operation.

Expand Down
105 changes: 95 additions & 10 deletions pages/clustering/high-availability/how-high-availability-works.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -255,24 +255,109 @@ in the cluster to ensure high availability, with timeouts.

| RPC message request | source | target | timeout |
|--------------------------|-------------|----------------| -----------------|
| `ShowInstancesReq` | Coordinator | Coordinator | |
| `DemoteMainToReplicaReq` | Coordinator | Data instance | |
| `PromoteToMainReq` | Coordinator | Data instance | |
| `RegisterReplicaOnMainReq` | Coordinator | Data instance | |
| `UnregisterReplicaReq` | Coordinator | Data instance | |
| `EnableWritingOnMainReq` | Coordinator | Data instance | deprecated |
| `GetDatabaseHistoriesReq` | Coordinator | Data instance | |
| `ShowInstancesReq` | Coordinator | Coordinator | 10s |
| `DemoteMainToReplicaReq` | Coordinator | Data instance | 10s |
| `PromoteToMainReq` | Coordinator | Data instance | 10s |
| `RegisterReplicaOnMainReq` | Coordinator | Data instance | 10s |
| `UnregisterReplicaReq` | Coordinator | Data instance | 10s |
| `ReplicationLagReq` | Coordinator | Data instance | 5s |
| `GetDatabaseHistoriesReq` | Coordinator | Data instance | 10s |
| `StateCheckReq` | Coordinator | Data instance | 5s |
| `SwapMainUUIDReq` | Coordinator | Data instance | |
| `SwapMainUUIDReq` | Coordinator | Data instance | 10s |
| `UpdateDataInstanceConfigReq` | Coordinator | Data instance | 10s |
| `FrequentHeartbeatReq` | Main | Replica | 5s |
| `HeartbeatReq` | Main | Replica | |
| `SystemRecoveryReq` | Main | Replica | 5s |
| `HeartbeatReq` | Main | Replica | 10s |
| `SystemRecoveryReq` | Main | Replica | 30s |
| `PrepareCommitRpc` | Main | Replica | proportional |
| `FinalizeCommitReq` | Main | Replica | 10s |
| `SnapshotRpc` | Main | Replica | proportional |
| `WalFilesRpc` | Main | Replica | proportional |
| `CurrentWalRpc` | Main | Replica | proportional |

<Callout type="info">
`EnableWritingOnMainReq` was **removed in Memgraph 3.13**. It was never sent —
writing on a newly promoted MAIN is enabled through the `writing_enabled` flag
carried inside `PromoteToMainRpc`. Its Prometheus counters were removed along
with it.
</Callout>

{<h5 className="custom-header"> Follower-to-leader forwarding timeouts </h5>}

Cluster management queries can be run on any coordinator; a follower forwards
them to the leader over RPC. From Memgraph 3.13, each of these RPCs has an
explicit timeout. They run on the caller's **Bolt session thread**, so without
one, a session would block forever against a leader that is reachable but stuck.

The four instance operations must outlast the work they trigger on the leader,
or a follower would report failure for an operation the leader has already
committed to Raft. Their budgets are the sum of that work plus headroom, where a
Raft commit is capped at 3 seconds and each RPC from the leader to a data
instance is capped by its entry in the table above.

| RPC message request | source | target | timeout | Budget breakdown |
|--------------------------|-------------|-------------|---------|------------------|
| `RegisterInstanceReq` | Coordinator | Coordinator | 30s | Raft commit + demote the new replica + register it |
| `UnregisterInstanceReq` | Coordinator | Coordinator | 20s | Raft commit + one RPC to the current MAIN |
| `DemoteInstanceReq` | Coordinator | Coordinator | 20s | Raft commit + one RPC to the current MAIN |
| `SetInstanceToMainReq` | Coordinator | Coordinator | 60s | Raft commit + one `SwapMainUUID` per other instance + promote the new MAIN |
| `AddCoordinatorReq` | Coordinator | Coordinator | 10s | Raft commit only |
| `RemoveCoordinatorReq` | Coordinator | Coordinator | 10s | Raft commit only |
| `UpdateConfigReq` | Coordinator | Coordinator | 10s | Raft commit only |
| `ForceResetReq` | Coordinator | Coordinator | 60s | Unbounded leader-side work — see the note below |
| `SetCoordinatorSettingReq` | Coordinator | Coordinator | 10s | Raft commit only |
| `GetRoutingTableReq` | Coordinator | Coordinator | 10s | Read served by the leader |
| `CoordReplLagReq` | Coordinator | Coordinator | 10s | Read served by the leader |
| `CreateRoleReq` | Coordinator | Coordinator | 10s | Raft commit only |
| `DropRoleReq` | Coordinator | Coordinator | 10s | Raft commit only |
| `GrantPrivilegeReq` | Coordinator | Coordinator | 10s | Raft commit only |
| `RevokePrivilegeReq` | Coordinator | Coordinator | 10s | Raft commit only |
| `GetRolesReq` | Coordinator | Coordinator | 10s | Read served by the leader |
| `GetRolePrivilegesReq` | Coordinator | Coordinator | 10s | Read served by the leader |

<Callout type="warning">
`SetInstanceToMainReq` sends one `SwapMainUUID` per other instance, so its cost
grows with the number of data instances. The 60s budget comfortably covers five
instances. Beyond that, a follower can time out before the leader answers — no
fixed value bounds it. Run `SET INSTANCE ... TO MAIN` directly on the leader in
very large clusters.

`ForceResetReq` triggers a reconciliation that retries under a 1s–60s backoff
for as long as the coordinator stays leader, so the leader-side work has no
upper bound at all. The 60s budget only keeps a genuinely wedged leader from
blocking the session — hitting it does **not** mean the reset failed, and
[`FORCE RESET CLUSTER
STATE`](/clustering/high-availability/ha-commands-reference#force-reset-cluster-state)
is safe to re-run.
</Callout>

The role and privilege RPCs are used by [coordinator
authentication](/clustering/high-availability/coordinator-authentication):
`GetRolesReq` in particular is sent on **every query of an SSO session**, because
coordinator privileges are re-derived from the leader's committed role set
rather than cached at login.

{<h5 className="custom-header"> System transaction timeouts </h5>}

MAIN-to-REPLICA system-delta RPCs are sent while committing a system
transaction, so they must not block indefinitely either. From Memgraph 3.13 each
carries a **10 second** timeout; a timeout marks the REPLICA as `BEHIND` and
defers to system recovery.

| RPC message request | source | target | timeout |
|---------------------|--------|---------|---------|
| `UpdateAuthDataReq` | Main | Replica | 10s |
| `DropAuthDataReq` | Main | Replica | 10s |
| `FinalizeSystemTxReq` | Main | Replica | 10s |
| `CreateDatabaseReq` | Main | Replica | 10s |
| `DropDatabaseReq` | Main | Replica | 10s |
| `SuspendDatabaseReq` | Main | Replica | 10s |
| `ResumeDatabaseReq` | Main | Replica | 10s |
| `RenameDatabaseReq` | Main | Replica | 10s |
| `TenantProfileReq` | Main | Replica | 10s |
| `SetParameterReq` | Main | Replica | 10s |
| `UnsetParameterReq` | Main | Replica | 10s |
| `DeleteAllParametersReq`| Main | Replica | 10s |

## Intra-cluster TLS

By default, the communication between instances in a high-availability cluster
Expand Down
Loading