OpenSearch Cluster Installation

These instructions describe the creation of a multi-node OpenSearch cluster using the tarball distribution. The configuration instructions below assume a cluster with the following components:

• Multiple master/ingest nodes
• Multiple hot data nodes
• Multiple cold data nodes

The instructions should be adaptable to any multi-node configuration, and even to a uniform node configuration (i.e. three master/ingress/data nodes) -- though that simple configuration does not need the multiple steps required for the more complex deployment.

Read through the entire document before doing anything, as there is a specific order in which things must be done for a successful installation.

Node Preparation

Each node to be used in the cluster must be prepared to support the OpenSearch installation.

Create Opensearch User

Debian/Ubuntu:

sudo adduser --system -uid 968 --shell /bin/bash --gecos 'OpenSearch User'  --group  --disabled-password --home /opt/opensearch opensearch

Fedora/RHEL:

sudo adduser --system --uid 968 --shell /bin/bash --home-dir /opt/opensearch -m opensearch

Then add your normal user to the group:

sudo usermod -aG opensearch ewilliam

Set vm.max_map_count

Create /etc/sysctl.d/vm.maxmap_count.conf (as root) and add the following line:

vm.max_map_count=262144

Run the following command as root to load the new configuration:

sysctl --system

Systemd Unit File

One of the big things missing from the tarball distribution is a systemd unit file. Fortunately, it isn't difficult to create a simple one that will work.

Create a file /etc/systemd/system/opensearch.service with the following contents:

[Unit]
Description=OpenSearch
Documentation=https://opensearch.org/docs/
Wants=network-online.target
After=network-online.target

[Service]
Type=exec
RuntimeDirectory=opensearch
PrivateTmp=true
Environment=OPENSEARCH_HOME=/opt/opensearch/opensearch
Environment=OPENSEARCH_PATH_CONF=/opt/opensearch/opensearch/config
Environment=PID_DIR=/opt/opensearch/opensearch
Environment=OPENSEARCH_SD_NOTIFY=true
EnvironmentFile=-/opt/opensearch/opensearch/opensearch.env

WorkingDirectory=/opt/opensearch/opensearch

User=opensearch
Group=opensearch

ExecStart=/opt/opensearch/opensearch/bin/opensearch -p ${PID_DIR}/opensearch.pid --quiet

# StandardOutput is configured to redirect to journalctl since
# some error messages may be logged in standard output before
# elasticsearch logging system is initialized. Elasticsearch
# stores its logs in /var/log/elasticsearch and does not use
# journalctl by default. If you also want to enable journalctl
# logging, you can simply remove the "quiet" option from ExecStart.
StandardOutput=journal
StandardError=inherit

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65535

# Specifies the maximum number of processes
LimitNPROC=4096

# Specifies the maximum size of virtual memory
LimitAS=infinity

# Specifies the maximum file size
LimitFSIZE=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=0

# SIGTERM signal is used to stop the Java process
KillSignal=SIGTERM

# Send the signal only to the JVM rather than its control group
KillMode=process

# Java process is never killed
SendSIGKILL=no

# When a JVM receives a SIGTERM signal it exits with code 143
SuccessExitStatus=143

# Allow a slow startup before the systemd notifier module kicks in to extend the timeout
TimeoutStartSec=75

[Install]
WantedBy=multi-user.target

Create Certificates

There are three types of certificates used in OpenSearch:

• CA certificate - one for the whole cluster to validate everything else
• Node certificate/key - identifies the node for internal cluster communications (transport layer) and data ingest (http layer)
  • the CN must be the FQDN of the node
  • even though OpenSearch doesn't seem to use them at this point, there should be SANs for the full FQDN, the short hostname, and the node's IP address
• User certificate/key - identifies a specific OpenSearch user
  • the CN is a username in OpenSearch (explicit or implicit)
  • username is used to determine roles/privileges in the cluster
  • Need at least one cert for admin authentication

Node and User certificates must all be signed by the cluster CA, and all keys must be in PKCS8 format. If user certificates are to be used for authentication by logstash, they must be imported (along with the key and CA) into a PKCS12 (.p12) keystore. The OpenSearch documentation has a good piece on how to create certificates.

In summary, create the following certificates/keys

1. CA certificate/key
2. For each cluster node:
   • <node>.crt
   • <node>-pkcs8.key
3. For each user:
   • <user>.crt
   • <user>-pkcs8.key
   • <user>.p12

All these should be created and available for the configuration of each cluster node below.

Install OpenSearch Tarball

The OpenSearch tarball must be installed on each node in the cluster; there is no difference at this point between the installations on the different types of nodes. Assuming the deployment will use the opensearch user created above ... change to the opensearch user (from root):

su - opensearch
cd ~

Run the following commands as user opensearch (from the opensearch home directory /opt/opensearch):

wget https://artifacts.opensearch.org/releases/bundle/opensearch/1.2.0/opensearch-1.2.0-linux-x64.tar.gz
tar -xvzf opensearch-1.2.0-linux-x64.tar.gz

Configure OpenSearch

Configuration varies depending on the type of node, but these are the places that need to be touched.

jvm.options

Set the heap size. Common practice is that the heap should be between 4GB and half the available memory, but no higher than 32GB. A minimum configuration might be (assuming the hot data nodes have 32GB installed RAM:

• 4G for master and cold data nodes
• 16G for hot data nodes

This must be done individually for each node -- create the file config/jvm.options.d/heapsize.options with the heap size options:

-Xms4g
-Xmx4g

... with, of course, the proper heap size for that node.

opensearch.yml

While the OpenSearch software can be rather forgiving for low-security environments, if you turn on full hostname verification, it is rather strict. The biggest thing to remember is that SAN entries in the node certificates aren't always respected. What this boils down to is that if full verification is turned on, you cannot use IP addresses in the opensearch.yml (or any other configuration file such as the logstash pipelines).

Each node will have a unique version of opensearch.yml due to certificate file names, the role(s) for the node, and the temperature (for data nodes). The significant parameters in the opensearch.yml file are:

• cluster.name - must be identical in all nodes
• node.name- can be the hostname, but doesn't have to be (must be unique)
• path.data - data nodes only -- must be unique to the node (i.e if using shared storage, each node must have its own directory on the shared volume)
• path.logs - must be unique to the node (same as above)
• network.host -- host's FQDN where all traffic will be received
  • Can be a single hostname or an array of hostnames in ["name", "name"] format
  • If you are doing transport-level host verification this CANNOT be an IP -- OpenSearch can't handle the SANs in the certificates needed to make it work
• http.port -- leave it as the default (9200) but uncomment it to lock it in
• discovery.seed_hosts -- FQDN for all the masters (same for all nodes) -- only used for cluster bootstrap
• cluster.initial_master_nodes -- put the master node names here (not the hostnames) -- same for all nodes
• node.[master|data|ingest|remote] - set the type of the node (specify all four)
• node.attr.temp - (data nodes only) defines data storage tiers (hot/warm/cold)

The stock configuration file has most of these specified -- edit the file to fit the configuration of the node.

The security configuration must be added next. Since we are not using the demo security as our template, all the relevant entries must be added now. Copy this block into the end of the stock opensearch.yml, editing as needed to provide the relevant certificate file names:

node.master: true
node.ingest: true
node.data: false
#node.remote: false

#node.attr.temp: hot

######## Start OpenSearch Security Configuration ########
plugins.security.ssl.transport.pemcert_filepath: <node>.crt
plugins.security.ssl.transport.pemkey_filepath: <node>-pkcs8.key
plugins.security.ssl.transport.pemtrustedcas_filepath: <CAcert>
plugins.security.ssl.transport.enforce_hostname_verification: true

plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: <node>.crt
plugins.security.ssl.http.pemkey_filepath: <node>-pkcs8.key
plugins.security.ssl.http.pemtrustedcas_filepath: <CAcert>

plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true

# at least one cert -- must be an EXACT copy of the DN for the certificates to be used for admin auth
plugins.security.authcz.admin_dn:
  - CN=Admin,O=OpenSearch,C=US

# these must be EXACT copies of the DN for the node certificates
plugins.security.nodes_dn:
  - CN=master-1.opensearch.localnet,O=OpenSearch,C=US
  - CN=data-hot-1.opensearch.localnet,O=OpenSearch,C=US

# this is all boiler plate ... no modifications needed
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
# this next bit should ALL be on one line ...
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

node.max_local_storage_nodes: 3

######## End OpenSearch Security Configuration ########

Note that there are two sections where the DNs from the certificates must be provided to enable admin authentication for security configuration, and for validating the nodes that will be part of the cluster. You can easily show the DNs for your certificates using this command:

openssl x509 -subject -nameopt RFC2253 -noout -in <cert file name>

The DN for the admin cert and the DNs for the node certs go in separate sections as shown above. Add all the cluster nodes into the nodes_dn section -- this must be the same for all nodes or they will not be able to communicate.

Certificates

Copy the following certificates/keys that were created previously into the config directory of each node:

• <CAcert>
• <node>.crt
• <node>-pkcs8.key
• admin.crt
• admin-pkcs8.crt

Verify that the opensearch user is able to at least read all of the certs/keys, or the cluster will not function.

Security Plugin Configuration

While the security configuration files exist on all the nodes in the cluster, they are only used once and only by the node where the security initialization is performed. With that in mine, pick one node (the first master node will do nicely) to edit the configuration files as needed. There are many things that can be done with these files, but two items in particular should be addressed. The files are located in the plugins/opensearch-security/securityconfig directory in the opensearch distribution.

Authentication

config.yml is where the authentication options are established. This will need to be configured to support your authentication options. The OpenSearch documentation describes the different options here.

The default configuration supports basic username/password authentication, but if you wish to use certificate authentication for API clients or ingest pipelines using logstash, you need to enable both http and transport under clientcert_auth_domain. Proxy authentication and OpenID Connect authentication is possible as well ... see the documentation.

Admin user password

The default password for the admin user is well-known, and should be changed for a production environment. The OpenSearch documentation provides instructions on how to generate passwords for the internal_users.yml file. Note that the admin password cannot be set through the dashboard or the API -- it must be set in this file, and then (re)loaded to take effect.

Security Plugin Initialization

Once all security configuration files are finished, they must be loaded into the OpenSearch cluster. This can only be done once enough of the cluster is up to provide a master node and at least one data node, as the security configuration is loaded into an index in the cluster. See the instructions below for specific sequencing. This command will initialize the security plugin:

plugins/opensearch-security/tools/securityadmin.sh \
    -cacert config/<CA cert> -cert config/admin.crt -key config/admin-pkcs8.key \
    -cd plugins/opensearch-security/securityconfig/ -host master-1.opensearch.localnet -icl -nhnv

... but again -- wait to do this until the right time below.

Starting The Node

Subject to the constraints described in the next section, the only thing left is to start the opensearch service:

sudo systemctl enable --now opensearch

Again, this must be done on each node in the correct sequence ... see below.

Cluster Bootstrapping

Starting up a multi-node cluster is a bit tricky:

• Master-only nodes have no data
• Data-only nodes aren't master eligible
• Cluster initialization requires indexes to be available

Note that this process does NOT use the demo configuration -- put all the security plugin items in the opensearch.yml file from the beginning. This will cause the opensearch-tar-install.sh script to skip the security initialization -- this is intentional.

What this means is that you have to stand up multiple nodes before anything starts working ... the process should be:

1. Install and start what will be the 'primary' master ... reference point for cluster startup operations
2. Install and start one of the 'hot' data nodes
3. Run the 'securityadmin.sh' script (using the command above) to initialize the security subsystem
4. Install/start the other master nodes so that there is resiliency as nodes go up and down
5. install/start the other data nodes to allow replication of the system indexes.

Install OpenSearch Dashboards

Finally, install the OpenSearch Dashboards using the tarball as well.

Run the following commands as opensearch user:

wget https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/1.2.0/opensearch-dashboards-1.2.0-linux-x64.tar.gz
tar -zxf opensearch-dashboards-1.2.0-linux-x64.tar.gz

Configure the opensearch-dashboards.yml file as needed for your situation.

Create file /etc/systemd/system/opensearch-dashboards.service (as root) with the following contents:

[Unit]
Description=OpenSearch-Dasboards

[Service]
Type=simple
User=opensearch
Group=opensearch
ExecStart=/opt/opensearch/opensearch-dashboards-1.2.0/bin/opensearch-dashboards
Restart=always
WorkingDirectory=/opt/opensearch/opensearch-dashboards-1.2.0

[Install]
WantedBy=multi-user.target

Run the following commands as root:

systemctl daemon-reload
systemctl start opensearch-dashboards.service 
systemctl enable opensearch-dashboards.service

References

• https://cryptopoolparty.com/blog/cardano-stake-pool-monitoring-ofo-stack/
  • Describes how to use OpenSearch to monitor another application -- but portions were used to create this document
• https://github.com/opensearch-project/dashboards-reports/releases/tag/chromium-1.12.0.0
  • download for headless chrome for generating PDF/PNG reports