Editing OpenSearch Cluster Installation

These instructions describe the creation of a multi-node OpenSearch cluster using the tarball distribution.  The configuration instructions below assume a cluster with the following components:

* Multiple master/ingest nodes
* Multiple hot data nodes
* Multiple cold data nodes

The instructions should be adaptable to any multi-node configuration, and even to a uniform node configuration (i.e. three master/ingress/data nodes) -- though that simple configuration does not need the multiple steps required for the more complex deployment.

Read through the entire document before doing anything, as there is a specific order in which things must be done for a successful installation.

== Node Preparation ==
Each node to be used in the cluster must be prepared to support the OpenSearch installation.

=== Create Opensearch User ===
Debian/Ubuntu:
 sudo adduser --system -uid 968 --shell /bin/bash --gecos 'OpenSearch User'  --group  --disabled-password --home /opt/opensearch opensearch
Fedora/RHEL:
 sudo adduser --system --uid 968 --shell /bin/bash --home-dir /opt/opensearch -m opensearch
Then add your normal user to the group:
 sudo usermod -aG opensearch ewilliam

=== Set vm.max_map_count ===
Create <code>/etc/sysctl.d/vm.maxmap_count.conf</code> (as root) and add the following line:
 vm.max_map_count=262144
Run the following command as root to load the new configuration:
 sysctl --system

=== Systemd Unit File ===
One of the big things missing from the tarball distribution is a systemd unit file.  Fortunately, it isn't difficult to create a simple one that will work.

Create a file <code>/etc/systemd/system/opensearch.service</code> with the following contents:
 [Unit]
 Description=OpenSearch
 Documentation=<nowiki>https://opensearch.org/docs/</nowiki>
 Wants=network-online.target
 After=network-online.target
 
 [Service]
 Type=exec
 RuntimeDirectory=opensearch
 PrivateTmp=true
 Environment=OPENSEARCH_HOME=/opt/opensearch/opensearch
 Environment=OPENSEARCH_PATH_CONF=/opt/opensearch/opensearch/config
 Environment=PID_DIR=/opt/opensearch/opensearch
 Environment=OPENSEARCH_SD_NOTIFY=true
 EnvironmentFile=-/opt/opensearch/opensearch/opensearch.env
 
 WorkingDirectory=/opt/opensearch/opensearch
 
 User=opensearch
 Group=opensearch
 
 ExecStart=/opt/opensearch/opensearch/bin/opensearch -p ${PID_DIR}/opensearch.pid --quiet
 
 # StandardOutput is configured to redirect to journalctl since
 # some error messages may be logged in standard output before
 # elasticsearch logging system is initialized. Elasticsearch
 # stores its logs in /var/log/elasticsearch and does not use
 # journalctl by default. If you also want to enable journalctl
 # logging, you can simply remove the "quiet" option from ExecStart.
 StandardOutput=journal
 StandardError=inherit
 
 # Specifies the maximum file descriptor number that can be opened by this process
 LimitNOFILE=65535
 
 # Specifies the maximum number of processes
 LimitNPROC=4096
 
 # Specifies the maximum size of virtual memory
 LimitAS=infinity
 
 # Specifies the maximum file size
 LimitFSIZE=infinity
 
 # Disable timeout logic and wait until process is stopped
 TimeoutStopSec=0
 
 # SIGTERM signal is used to stop the Java process
 KillSignal=SIGTERM
 
 # Send the signal only to the JVM rather than its control group
 KillMode=process
 
 # Java process is never killed
 SendSIGKILL=no
 
 # When a JVM receives a SIGTERM signal it exits with code 143
 SuccessExitStatus=143
 
 # Allow a slow startup before the systemd notifier module kicks in to extend the timeout
 TimeoutStartSec=75
 
 [Install]
 WantedBy=multi-user.target

== Create Certificates ==
There are three types of certificates used in OpenSearch:

* CA certificate - one for the whole cluster to validate everything else
* Node certificate/key - identifies the node for internal cluster communications (transport layer) and data ingest (http layer)
** the CN must be the FQDN of the node
** even though OpenSearch doesn't seem to use them at this point, there should be SANs for the full FQDN, the short hostname, and the node's IP address
* User certificate/key - identifies a specific OpenSearch user
** the CN is a username in OpenSearch (explicit or implicit)
** username is used to determine roles/privileges in the cluster
** Need at least one cert for admin authentication

Node and User certificates must all be signed by the cluster CA, and '''all keys must be in PKCS8 format'''.  If user certificates are to be used for authentication by logstash, they must be imported (along with the key and CA) into a PKCS12 (.p12) keystore.  The OpenSearch documentation has a good piece on [https://opensearch.org/docs/latest/security-plugin/configuration/generate-certificates/ how to create certificates].

In summary, create the following certificates/keys

# CA certificate/key
# For each cluster node:
#* <node>.crt
#* <node>-pkcs8.key
# For each user:
#* <user>.crt
#* <user>-pkcs8.key
#* <user>.p12

All these should be created and available for the configuration of each cluster node below.

== Install OpenSearch Tarball ==
The OpenSearch tarball must be installed on each node in the cluster; there is no difference at this point between the installations on the different types of nodes.  Assuming the deployment will use the <code>opensearch</code> user created above ... change to the <code>opensearch</code> user (from root):
 su - opensearch
 cd ~
Run the following commands as user <code>opensearch</code> (from the opensearch home directory <code>/opt/opensearch</code>):
 wget https://artifacts.opensearch.org/releases/bundle/opensearch/1.2.0/opensearch-1.2.0-linux-x64.tar.gz
 tar -xvzf opensearch-1.2.0-linux-x64.tar.gz
== Configure OpenSearch ==
Configuration varies depending on the type of node, but these are the places that need to be touched.

=== <code>jvm.options</code> ===
Set the heap size.  Common practice is that the heap should be between 4GB and half the available memory, but no higher than 32GB.  A minimum configuration might be (assuming the hot data nodes have 32GB installed RAM:

* 4G for master and cold data nodes
* 16G for hot data nodes

This must be done individually for each node -- create the file <code>config/jvm.options.d/heapsize.options</code> with the heap size options:
 -Xms4g
 -Xmx4g
... with, of course, the proper heap size for that node.

=== <code>opensearch.yml</code> ===
While the OpenSearch software can be rather forgiving for low-security environments, if you turn on full hostname verification, it is rather strict.  The biggest thing to remember is that '''SAN entries in the node certificates aren't always respected.'''  What this boils down to is that if full verification is turned on, you cannot use IP addresses in the <code>opensearch.yml</code> (or any other configuration file such as the logstash pipelines).

Each node will have a unique version of opensearch.yml due to certificate file names, the role(s) for the node, and the temperature (for data nodes).  The significant parameters in the opensearch.yml file are:
* <code>cluster.name</code> - must be identical in all nodes
* <code>node.name</code>- can be the hostname, but doesn't have to be (must be unique)
* <code>path.data</code> - data nodes only -- must be unique to the node (i.e if using shared storage, each node must have its own directory on the shared volume)
* <code>path.logs</code> - must be unique to the node (same as above)
* <code>network.host</code> -- host's FQDN where all traffic will be received
**Can be a single hostname or an array of hostnames in <code>["name", "name"]</code> format
**If you are doing transport-level host verification this CANNOT be an IP -- OpenSearch can't handle the SANs in the certificates needed to make it work
* <code>http.port</code> -- leave it as the default (<code>9200</code>) but uncomment it to lock it in
* <code>discovery.seed_hosts</code> -- FQDN for all the masters (same for all nodes) -- only used for cluster bootstrap
* <code>cluster.initial_master_nodes</code> -- put the master node names here (not the hostnames) -- same for all nodes
* <code>node.[master|data|ingest|remote]</code> - set the type of the node (specify all four)
* <code>node.attr.temp</code> - (data nodes only) defines data storage tiers (<code>hot</code>/<code>warm</code>/<code>cold</code>)
The stock configuration file has most of these specified -- edit the file to fit the configuration of the node.

The security configuration must be added next.  Since we are not using the demo security as our template, all the relevant entries must be added now.  Copy this block into the end of the stock opensearch.yml, editing as needed to provide the relevant certificate file names:
 node.master: true
 node.ingest: true
 node.data: false
 #node.remote: false
 
 #node.attr.temp: hot
 
 ######## Start OpenSearch Security Configuration ########
 plugins.security.ssl.transport.pemcert_filepath: <node>.crt
 plugins.security.ssl.transport.pemkey_filepath: <node>-pkcs8.key
 plugins.security.ssl.transport.pemtrustedcas_filepath: <CAcert>
 plugins.security.ssl.transport.enforce_hostname_verification: true
 
 plugins.security.ssl.http.enabled: true
 plugins.security.ssl.http.pemcert_filepath: <node>.crt
 plugins.security.ssl.http.pemkey_filepath: <node>-pkcs8.key
 plugins.security.ssl.http.pemtrustedcas_filepath: <CAcert>
 
 plugins.security.allow_unsafe_democertificates: false
 plugins.security.allow_default_init_securityindex: true
 
 # at least one cert -- must be an EXACT copy of the DN for the certificates to be used for admin auth
 plugins.security.authcz.admin_dn:
   - CN=Admin,O=OpenSearch,C=US
 
 # these must be EXACT copies of the DN for the node certificates
 plugins.security.nodes_dn:
   - CN=master-1.opensearch.localnet,O=OpenSearch,C=US
   - CN=data-hot-1.opensearch.localnet,O=OpenSearch,C=US
 
 # this is all boiler plate ... no modifications needed
 plugins.security.audit.type: internal_opensearch
 plugins.security.enable_snapshot_restore_privilege: true
 plugins.security.check_snapshot_restore_write_privileges: true
 plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
 plugins.security.system_indices.enabled: true
 # this next bit should ALL be on one line ...
 plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
 
 node.max_local_storage_nodes: 3
 
 ######## End OpenSearch Security Configuration ########
Note that there are two sections where the DNs from the certificates must be provided to enable admin authentication for security configuration, and for validating the nodes that will be part of the cluster.  You can easily show the DNs for your certificates using this command:
 openssl x509 -subject -nameopt RFC2253 -noout -in <cert file name>
The DN for the admin cert and the DNs for the node certs go in separate sections as shown above.  Add all the cluster nodes into the nodes_dn section -- this must be the same for all nodes or they will not be able to communicate.

=== Certificates ===
Copy the following certificates/keys that were created previously into the config directory of each node:

* <CAcert>
* <node>.crt
* <node>-pkcs8.key
* admin.crt
* admin-pkcs8.crt
Verify that the <code>opensearch</code> user is able to at least read all of the certs/keys, or the cluster will not function.

=== Security Plugin Configuration ===
While the security configuration files exist on all the nodes in the cluster, they are only used once and only by the node where the security initialization is performed.  With that in mine, pick one node (the first master node will do nicely) to edit the configuration files as needed.  There are many things that can be done with these files, but two items in particular should be addressed.  The files are located in the <code>plugins/opensearch-security/securityconfig</code> directory in the opensearch distribution.

==== Authentication ====
<code>config.yml</code> is where the authentication options are established.  This will need to be configured to support your authentication options.  The OpenSearch documentation describes the different options [https://opensearch.org/docs/latest/security-plugin/configuration/configuration/ here].

The default configuration supports basic username/password authentication, but if you wish to use certificate authentication for API clients or ingest pipelines using logstash, you need to enable both '''http''' and '''transport''' under <code>clientcert_auth_domain</code>.  Proxy authentication and OpenID Connect authentication is possible as well ... see the documentation.

==== Admin user password ====
The default password for the admin user is well-known, and should be changed for a production environment.  The OpenSearch documentation provides instructions on how to [https://opensearch.org/docs/latest/security-plugin/configuration/yaml/#internal_usersyml generate passwords] for the <code>internal_users.yml</code> file.  Note that '''the admin password cannot be set through the dashboard or the API''' -- it must be set in this file, and then (re)loaded to take effect.

=== Security Plugin Initialization ===
Once all security configuration files are finished, they must be loaded into the OpenSearch cluster.  This can only be done once enough of the cluster is up to provide a master node and at least one data node, as the security configuration is loaded into an index in the cluster.  See the instructions below for specific sequencing.  This command will initialize the security plugin:
 plugins/opensearch-security/tools/securityadmin.sh \
     -cacert config/<CA cert> -cert config/admin.crt -key config/admin-pkcs8.key \
     -cd plugins/opensearch-security/securityconfig/ -host master-1.opensearch.localnet -icl -nhnv
... but again -- wait to do this until the right time below.

=== Starting The Node ===
Subject to the constraints described in the next section, the only thing left is to start the opensearch service:
 sudo systemctl enable --now opensearch
Again, this must be done on each node in the correct sequence ... see below.

== Cluster Bootstrapping ==
Starting up a multi-node cluster is a bit tricky:

* Master-only nodes have no data
* Data-only nodes aren't master eligible
* Cluster initialization requires indexes to be available

'''Note that this process does NOT use the demo configuration -- put all the security plugin items in the <code>opensearch.yml</code> file from the beginning'''.  This will cause the <code>opensearch-tar-install.sh</code> script to skip the security initialization -- this is intentional.

What this means is that you have to stand up multiple nodes before anything starts working ... the process should be:

# Install and start what will be the 'primary' master ... reference point for cluster startup operations
# Install and start one of the 'hot' data nodes
# Run the 'securityadmin.sh' script (using the command above) to initialize the security subsystem
# Install/start the other master nodes so that there is resiliency as nodes go up and down
# install/start the other data nodes to allow replication of the system indexes.

== Install OpenSearch Dashboards ==
Finally, install the OpenSearch Dashboards using the tarball as well.

Run the following commands as opensearch user:
 wget [https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/1.1.0/opensearch-dashboards-1.1.0-linux-x64.tar.gz https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/1.2.0/opensearch-dashboards-1.2.0-linux-x64.tar.gz]
 tar -zxf opensearch-dashboards-1.2.0-linux-x64.tar.gz

Configure the <code>opensearch-dashboards.yml</code> file as needed for your situation.

Create file <code>/etc/systemd/system/opensearch-dashboards.service</code> (as root) with the following contents:
 [Unit]
 Description=OpenSearch-Dasboards
 
 [Service]
 Type=simple
 User=opensearch
 Group=opensearch
 ExecStart=/opt/opensearch/opensearch-dashboards-1.2.0/bin/opensearch-dashboards
 Restart=always
 WorkingDirectory=/opt/opensearch/opensearch-dashboards-1.2.0
 
 [Install]
 WantedBy=multi-user.target
Run the following commands as root:
 systemctl daemon-reload
 systemctl start opensearch-dashboards.service 
 systemctl enable opensearch-dashboards.service

== References ==

* https://cryptopoolparty.com/blog/cardano-stake-pool-monitoring-ofo-stack/
** Describes how to use OpenSearch to monitor another application -- but portions were used to create this document
*https://github.com/opensearch-project/dashboards-reports/releases/tag/chromium-1.12.0.0
**download for headless chrome for generating PDF/PNG reports