How to Fix 503 Service Unavailable - Raft consensus unavailable (Weaviate)

Quick Answer: This error occurs when a Weaviate cluster loses consensus among its Raft nodes, typically due to a majority of nodes going down, severe network partitioning, or resource exhaustion. The fastest fix is to check pod health, ensure network connectivity between cluster nodes, and restart unhealthy nodes to reform a quorum.

What Causes This Error

• A majority of nodes in the Raft consensus group are offline or unreachable.
• Severe network partitioning preventing nodes from communicating with the Raft leader.
• Node resource exhaustion (CPU, memory, or disk space) causing frequent crashes.
• Incorrect configuration of node discovery or cluster addressing endpoints.

Step-by-Step Fixes

Fix 1: Fix 1: Check Cluster Node Status and Health

Inspect your container orchestration platform (e.g., Kubernetes pods) to see which Weaviate replicas are running or crashing.,Check the logs of the unhealthy Weaviate pods for panic messages, disk errors, or network timeouts.,Restart any crashed or non-responsive follower nodes to allow them to rejoin the cluster.

Fix 2: Fix 2: Verify Network Connectivity and DNS

Ensure that inter-node communication ports (typically gRPC and HTTP ports used by Weaviate) are open and accessible.,Verify that Kubernetes services or internal DNS records resolving cluster member addresses are functioning correctly.

Fix 3: Fix 3: Free Up Insufficient Cluster Resources

Check node resource utilization using monitoring tools (like Prometheus/Grafana or kubectl top nodes).,Scale up node memory or CPU allocations if nodes are being OOM-killed.,Ensure underlying persistent volumes have sufficient free disk space.

Advanced Fixes

Advanced Fix 1: Advanced: Force Raft Recovery or Reconfigure Quorum

If quorum is permanently lost and data recovery is needed, inspect the Raft data directory state.,Use Weaviate's administrative recovery procedures or restore from a recent snapshot if the consensus state is unrecoverable.

FAQs

Q: What is Raft consensus in Weaviate?

A: Raft is a consensus algorithm used in Weaviate's distributed deployment mode to manage metadata, schema replication, and coordination across multiple cluster nodes.

Q: Will I lose data if Raft consensus is unavailable?

A: Not necessarily. Data already written to disk is preserved, but the cluster will reject read/write operations until a valid leader is elected and quorum is restored.