Cluster Operations

Rolling Restart

The OpenSearch cluster is very resilient -- it tries to make sure it has all of its indices appropriately replicated at all times. It can handle nodes dropping out and being added -- but sometimes we don't want that. If nodes are simply restarted (to change settings) or the system is rebooted (for patching), we don't want to have OpenSearch move all its indices around.

This process works for a whole cluster restart (not recommended unless absolutely necessary) or individual node restarts. If individual nodes are being restarted, only do one node at a time to avoid cluster panic should all shards of an index go offline on the nodes that get restarted. The cluster should recover, but why stress it unnecessarily.

The process is described in the Elastic documentation, and summarized below.

Disable Shard Allocation

When you shut down a data node, the allocation process waits for index.unassigned.node_left.delayed_timeout (by default, one minute) before starting to replicate the shards on that node to other nodes in the cluster, which can involve a lot of I/O. Since the node is shortly going to be restarted, this I/O is unnecessary. You can avoid racing the clock by disabling allocation of replicas before shutting down data nodes:

curl -X PUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}'

Flush Cache

This will prevent data loss by flushing any in-process activities to disk:

curl -X POST "localhost:9200/_flush/synced?pretty"

Shut Down/Restart Nodes

Now that things have settled out, we can stop/restart the opensearch services. Note that a full cluster shutdown should be avoided, since any activity on the cluster as it is going down or coming up could cause errors to the user. Use the systemd commands to stop/start/restart the nodes:

# do this on all nodes
sudo systemctl stop opensearch
 ... do required activity on all nodes ...
sudo systemctl start opensearch

... or ...

# restart a single node
sudo systemctl restart opensearch

If it was a complete cluster shutdown, all nodes must rejoin the cluster before proceeding. In either case, wait until the cluster (or node) achieves a "YELLOW" state before proceeding ... this means that all primary shards are accounted for, but replicas may not be.

Enable Shard Allocation

As each node re-starts, it tries to identify which shards it had before. Unless something has happened to disrupt the node's data, it will simply reactivate those shards (most likely as replica shards if it is a single-node restart). To enable the cluster to get back to normal shard management, we remove the block we set above:

curl -X PUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}'

Follow Through

When doing any activity that disrupts the cluster operations enough to cause it to drop into a "YELLOW" or "RED" state, wait until the cluster is "GREEN" again before proceeding. In the case of a full rolling restart (one node at a time), the cluster should be "GREEN" before proceeding to the next node.