Editing OpenSearch Cluster Installation (section)

== Configure OpenSearch ==
Configuration varies depending on the type of node, but these are the places that need to be touched.

=== <code>jvm.options</code> ===
Set the heap size.  Common practice is that the heap should be between 4GB and half the available memory, but no higher than 32GB.  A minimum configuration might be (assuming the hot data nodes have 32GB installed RAM:

* 4G for master and cold data nodes
* 16G for hot data nodes

This must be done individually for each node -- create the file <code>config/jvm.options.d/heapsize.options</code> with the heap size options:
 -Xms4g
 -Xmx4g
... with, of course, the proper heap size for that node.

=== <code>opensearch.yml</code> ===
While the OpenSearch software can be rather forgiving for low-security environments, if you turn on full hostname verification, it is rather strict.  The biggest thing to remember is that '''SAN entries in the node certificates aren't always respected.'''  What this boils down to is that if full verification is turned on, you cannot use IP addresses in the <code>opensearch.yml</code> (or any other configuration file such as the logstash pipelines).

Each node will have a unique version of opensearch.yml due to certificate file names, the role(s) for the node, and the temperature (for data nodes).  The significant parameters in the opensearch.yml file are:
* <code>cluster.name</code> - must be identical in all nodes
* <code>node.name</code>- can be the hostname, but doesn't have to be (must be unique)
* <code>path.data</code> - data nodes only -- must be unique to the node (i.e if using shared storage, each node must have its own directory on the shared volume)
* <code>path.logs</code> - must be unique to the node (same as above)
* <code>network.host</code> -- host's FQDN where all traffic will be received
**Can be a single hostname or an array of hostnames in <code>["name", "name"]</code> format
**If you are doing transport-level host verification this CANNOT be an IP -- OpenSearch can't handle the SANs in the certificates needed to make it work
* <code>http.port</code> -- leave it as the default (<code>9200</code>) but uncomment it to lock it in
* <code>discovery.seed_hosts</code> -- FQDN for all the masters (same for all nodes) -- only used for cluster bootstrap
* <code>cluster.initial_master_nodes</code> -- put the master node names here (not the hostnames) -- same for all nodes
* <code>node.[master|data|ingest|remote]</code> - set the type of the node (specify all four)
* <code>node.attr.temp</code> - (data nodes only) defines data storage tiers (<code>hot</code>/<code>warm</code>/<code>cold</code>)
The stock configuration file has most of these specified -- edit the file to fit the configuration of the node.

The security configuration must be added next.  Since we are not using the demo security as our template, all the relevant entries must be added now.  Copy this block into the end of the stock opensearch.yml, editing as needed to provide the relevant certificate file names:
 node.master: true
 node.ingest: true
 node.data: false
 #node.remote: false
 
 #node.attr.temp: hot
 
 ######## Start OpenSearch Security Configuration ########
 plugins.security.ssl.transport.pemcert_filepath: <node>.crt
 plugins.security.ssl.transport.pemkey_filepath: <node>-pkcs8.key
 plugins.security.ssl.transport.pemtrustedcas_filepath: <CAcert>
 plugins.security.ssl.transport.enforce_hostname_verification: true
 
 plugins.security.ssl.http.enabled: true
 plugins.security.ssl.http.pemcert_filepath: <node>.crt
 plugins.security.ssl.http.pemkey_filepath: <node>-pkcs8.key
 plugins.security.ssl.http.pemtrustedcas_filepath: <CAcert>
 
 plugins.security.allow_unsafe_democertificates: false
 plugins.security.allow_default_init_securityindex: true
 
 # at least one cert -- must be an EXACT copy of the DN for the certificates to be used for admin auth
 plugins.security.authcz.admin_dn:
   - CN=Admin,O=OpenSearch,C=US
 
 # these must be EXACT copies of the DN for the node certificates
 plugins.security.nodes_dn:
   - CN=master-1.opensearch.localnet,O=OpenSearch,C=US
   - CN=data-hot-1.opensearch.localnet,O=OpenSearch,C=US
 
 # this is all boiler plate ... no modifications needed
 plugins.security.audit.type: internal_opensearch
 plugins.security.enable_snapshot_restore_privilege: true
 plugins.security.check_snapshot_restore_write_privileges: true
 plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
 plugins.security.system_indices.enabled: true
 # this next bit should ALL be on one line ...
 plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
 
 node.max_local_storage_nodes: 3
 
 ######## End OpenSearch Security Configuration ########
Note that there are two sections where the DNs from the certificates must be provided to enable admin authentication for security configuration, and for validating the nodes that will be part of the cluster.  You can easily show the DNs for your certificates using this command:
 openssl x509 -subject -nameopt RFC2253 -noout -in <cert file name>
The DN for the admin cert and the DNs for the node certs go in separate sections as shown above.  Add all the cluster nodes into the nodes_dn section -- this must be the same for all nodes or they will not be able to communicate.

=== Certificates ===
Copy the following certificates/keys that were created previously into the config directory of each node:

* <CAcert>
* <node>.crt
* <node>-pkcs8.key
* admin.crt
* admin-pkcs8.crt
Verify that the <code>opensearch</code> user is able to at least read all of the certs/keys, or the cluster will not function.

=== Security Plugin Configuration ===
While the security configuration files exist on all the nodes in the cluster, they are only used once and only by the node where the security initialization is performed.  With that in mine, pick one node (the first master node will do nicely) to edit the configuration files as needed.  There are many things that can be done with these files, but two items in particular should be addressed.  The files are located in the <code>plugins/opensearch-security/securityconfig</code> directory in the opensearch distribution.

==== Authentication ====
<code>config.yml</code> is where the authentication options are established.  This will need to be configured to support your authentication options.  The OpenSearch documentation describes the different options [https://opensearch.org/docs/latest/security-plugin/configuration/configuration/ here].

The default configuration supports basic username/password authentication, but if you wish to use certificate authentication for API clients or ingest pipelines using logstash, you need to enable both '''http''' and '''transport''' under <code>clientcert_auth_domain</code>.  Proxy authentication and OpenID Connect authentication is possible as well ... see the documentation.

==== Admin user password ====
The default password for the admin user is well-known, and should be changed for a production environment.  The OpenSearch documentation provides instructions on how to [https://opensearch.org/docs/latest/security-plugin/configuration/yaml/#internal_usersyml generate passwords] for the <code>internal_users.yml</code> file.  Note that '''the admin password cannot be set through the dashboard or the API''' -- it must be set in this file, and then (re)loaded to take effect.

=== Security Plugin Initialization ===
Once all security configuration files are finished, they must be loaded into the OpenSearch cluster.  This can only be done once enough of the cluster is up to provide a master node and at least one data node, as the security configuration is loaded into an index in the cluster.  See the instructions below for specific sequencing.  This command will initialize the security plugin:
 plugins/opensearch-security/tools/securityadmin.sh \
     -cacert config/<CA cert> -cert config/admin.crt -key config/admin-pkcs8.key \
     -cd plugins/opensearch-security/securityconfig/ -host master-1.opensearch.localnet -icl -nhnv
... but again -- wait to do this until the right time below.

=== Starting The Node ===
Subject to the constraints described in the next section, the only thing left is to start the opensearch service:
 sudo systemctl enable --now opensearch
Again, this must be done on each node in the correct sequence ... see below.