Editing Cluster Operations (section)

== Certificate Updates ==
Given that the standard lifetime for certificates these days is one year, this is  something that will need to be accomplished several times during the lifetime of a cluster.

This process does not require downtime for the cluster, but care must be taken to ensure that the mix of old and new certificates does not cripple the cluster.  This results in up to three complete cluster restarts (using the '''Rolling Restart''' method described above), but it does not disrupt the operation of the cluster as long as nothing exceptional happens, and the cluster is allowed to stabilize between each node restart.

=== Create New Certificates ===
Using the process outlined in [https://wiki.williams-net.org/wiki/OpenSearch_Cluster_Installation#Create_Certificates OpenSearch Cluster Creation], create the required certificates and keys.  If at all possible, keep the DNs on all certificates the same as the ones they are replacing ... it reduces the number of restarts and minimizes the edits of the config files.  It also helps if the node certificate/key files have the same names ... for the same reasons.

Send each node its new certificate/keys and the admin certificate/key -- or make them available for remote retrieval.  If user certificates were created, send them to the system(s) where they will be used.

=== Create CA Bundle ===
Using a CA bundle with the old and new CAs will allow us to do a gradual transition and avoid downtime.  Fortunately, creating a bundle for the CAs is easy
 cat <old CA> <new CA> >> rootCAbundle.crt
Again, if there is currently a bundle in place, use the same name for the bundle file and you'll save yourself a lot of editing.  If the REST API uses a different CA bundle already that supports other (external) CAs, create a second CA bundle that includes the external CAs, the old internal CA, and the new internal CA.

Send this new CA bundle to all cluster nodes, dashboard instances, and logstash instances.

=== Update CA Files ===
For each cluster node:

* Copy the new CA bundle in place (in the opensearch <code>config</code> directory) making sure that permissions allow for read access to the <code>opensearch</code> user.  
* DO NOT copy the node certificates yet.  
* Edit <code>opensearch.yml</code> if needed:
** Update the name of the CA to the new CA bundle file (if needed) in both the <code>ssl</code> and <code>transport</code> parameters, using the appropriate bundle for each parameter (see above)
** Add the DNs (DO NOT replace them yet) for the node and admin certificates if the new DNs are different from what is in the file

* Perform a '''Rolling Restart''' of the node, verifying that there are no SSL/certificate errors, and waiting for "GREEN" status before proceeding 

Now do the same for all dashboards, logstash instances, and any other clients that use the CA to validate cluster identify:

* Install the CA bundle as appropriate, ensuring that the client has read access to the CA bundle.
* Update the CA filename in the configuration file (if needed)
* DO NOT update user certificates at this point
* Restart the client (if needed) and verify proper operation

=== Update Node Certificates ===
For each node in the cluster:

* Copy the node certificate and key and the admin certificate and key into the opensearch <code>config</code> directory, verifying that the <code>opensearch</code> user has read permissions
* Edit <code>opensearch.yml</code> if needed to correct certificate file names
* DO NOT remove extra DNs from the admin or node sections of the file yet
* Perform a '''Rolling Restart''' of the node, waiting for the cluster to achieve "GREEN" status before going to the next node

=== Update Client Certificates ===
For each client that uses certificate authentication:

* Copy the client certificate bundle (.p12 file) into place
* Edit the configuration file if needed to reflect the new file
* Restart the client (if needed) to activate the new configuration

=== Clean Up ===
At this point we can remove any old DNs from the <code>opensearch.yml</code> configurations and in general remove all traces of the old certificates in the cluster nodes and clients.  If a restart is warranted to verify that nothing critical was deleted, that should be done at this time.

It is not necessary to replace the new CA bundle with the CA certificate -- it serves no purpose as the old cert is just along for the ride at this point, and once it reaches its expiration date, will not be used.  If it must be done, use the same process above that was used to install the CA bundle, performing a rolling restart on each node as it is completed.