OpenSearch Cluster Installation

These instructions describe the creation of a multi-node OpenSearch cluster using the tarball distribution. The configuration instructions below assume a cluster with the following components:

• Multiple master/ingest nodes
• Multiple hot data nodes
• Multiple cold data nodes

The instructions should be adaptable to any multi-node configuration, and even to a uniform node configuration (i.e. three master/ingress/data nodes) -- though that simple configuration does not need the multiple steps required for the more complex deployment.

Read through the entire document before doing anything, as there is a specific order in which things must be done for a successful installation.

Node Preparation

Each node to be used in the cluster must be prepared to support the OpenSearch installation.

Create Opensearch User

Debian/Ubuntu:

sudo adduser --system -uid 968 --shell /bin/bash --gecos 'OpenSearch User'  --group  --disabled-password --home /opt/opensearch opensearch

Fedora/RHEL:

sudo adduser --system --uid 968 --shell /bin/bash --home-dir /opt/opensearch -m opensearch

Then add your normal user to the group:

sudo usermod -aG opensearch ewilliam

Set vm.max_map_count

Create /etc/sysctl.d/vm.maxmap_count (as root) and add the following line:

vm.max_map_count=262144

Run the following commands as root to load the new configuration:

sysctl --system

Install Java 11 (optional)

The OpenSearch distro comes with Java 8 integrated into the packages. If this won't work for some reason, run the following commands as root:

apt update
apt install -y openjdk-11-jdk

Similar commands can be used on a Fedora/RHEL system.

Systemd Unit File

One of the big things missing from the tarball distribution is a systemd unit file. Fortunately, it isn't difficult to create a simple one that will work.

Create a file /etc/systemd/system/opensearch.service with the following contents:

[Unit]
Description=OpenSearch
Documentation=https://opensearch.org/docs/
Wants=network-online.target
After=network-online.target

[Service]
RuntimeDirectory=opensearch
WorkingDirectory=/opt/opensearch/opensearch-1.1.0
User=opensearch
Group=opensearch

ExecStart=/opt/opensearch/opensearch-1.1.0/opensearch-tar-install.sh

# StandardOutput is configured to redirect to journalctl since
# some error messages may be logged in standard output before
# elasticsearch logging system is initialized. Elasticsearch
# stores its logs in /var/log/elasticsearch and does not use
# journalctl by default. If you also want to enable journalctl
# logging, you can simply remove the "quiet" option from ExecStart.
StandardOutput=journal
StandardError=inherit

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65535

# Specifies the maximum number of processes
LimitNPROC=4096

# Specifies the maximum size of virtual memory
LimitAS=infinity

# Specifies the maximum file size
LimitFSIZE=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=0

# SIGTERM signal is used to stop the Java process
KillSignal=SIGTERM

# Send the signal only to the JVM rather than its control group
KillMode=process

# Java process is never killed
SendSIGKILL=no

# When a JVM receives a SIGTERM signal it exits with code 143
SuccessExitStatus=143

# Allow a slow startup before the systemd notifier module kicks in to extend the timeout
TimeoutStartSec=75

[Install]
WantedBy=multi-user.target

Create Certificates

There are three types of certificates used in OpenSearch:

• CA certificate - one for the whole cluster to validate everything else
• Node certificate/key - identifies the node for internal cluster communications (transport layer) and data ingest (http layer)
  • the CN must be the FQDN of the node
  • even though OpenSearch doesn't seem to use them at this point, there should be SANs for the full FQDN, the short hostname, and the node's IP address
• User certificate/key - identifies a specific OpenSearch user
  • the CN is a username in OpenSearch (explicit or implicit)
  • username is used to determine roles/privileges in the cluster
  • Need at least one cert for admin authentication

Node and User certificates must all be signed by the cluster CA, and all keys must be in PKCS8 format. If user certificates are to be used for authentication by logstash, they must be imported (along with the key and CA) into a PKCS12 (.p12) keystore. The OpenSearch documentation has a good piece on how to create certificates.

In summary, create the following certificates/keys

1. CA certificate/key
2. For each cluster node:
   • <node>.crt
   • <node>-pkcs8.key
3. For each user:
   • <user>.crt
   • <user>-pkcs8.key
   • <user>.p12

All these should be created and available for the configuration of each cluster node below.

Install Opensearch Tarball

The OpenSearch tarball must be installed on each node in the cluster; there is no difference at this point between the installations on the different types of nodes. Assuming the deployment will use the opensearch user created above ... change to the opensearch user (from root):

su - opensearch
cd ~

Run the following commands as user opensearch (from the opensearch home directory /opt/opensearch):

wget https://artifacts.opensearch.org/releases/bundle/opensearch/1.2.0/opensearch-1.2.0-linux-x64.tar.gz
tar -xvzf opensearch-1.2.0-linux-x64.tar.gz

Configure OpenSearch

Configuration varies depending on the type of node, but these are the places that need to be touched.

jvm.options

Set the heap size. Common practice is that the heap should be between 4GB and half the available memory, but no higher than 32GB. A minimum configuration might be (assuming the hot data nodes have 32GB installed RAM:

• 4G for master and cold data nodes
• 16G for hot data nodes

This must be done individually for each node.

opensearch.yml

While the OpenSearch software can be rather forgiving for low-security environments, if you turn on full hostname verification, it is rather strict. The biggest thing to remember is that SAN entries in the node certificates aren't always respected. What this boils down to is that if full verification is turned on, you cannot use IP addresses in the opensearch.yml (or any other configuration file such as the logstash pipelines).

Each node will have a unique version of opensearch.yml due to certificate file names, the role(s) for the node, and the temperature (for data nodes). The significant parameters in the opensearch.yml file are:

• cluster.name - must be identical in all nodes
• node.name- can be the hostname, but doesn't have to be (must be unique)
• path.data - data nodes only -- must be unique to the node (i.e if using shared storage, each node must have its own directory on the shared volume)
• path.logs - must be unique to the node (same as above)
• network.host -- host's FQDN where all traffic will be received
  • Can be a single hostname or an array of hostnames in ["name", "name"] format
  • If you are doing transport-level host verification this CANNOT be an IP -- OpenSearch can't handle the SANs in the certificates that make it work
• http.port -- leave it as the default (9200) but uncomment it to lock it in
• discovery.seed_hosts -- FQDN for all the masters (same for all nodes) -- only used for cluster bootstrap
• cluster.initial_master_nodes -- put the master node names here (not the hostnames) -- same for all nodes
• node.[master|data|ingest|remote] - set the type of the node (specify all four)
• node.attr.temp - (data nodes only) defines data storage tiers (hot/warm/cold)

The stock configuration file has most of these specified -- edit the file to fit the configuration of the node.

The security configuration must be added next. Since we are not using the demo security as our template, all the relevant entries must be added now. Copy this block into the end of the stock opensearch.yml, editing as needed to provide the relevant certificate file names:

node.master: true
node.ingest: true
node.data: false
#node.remote: false

#node.attr.temp: hot

######## Start OpenSearch Security Configuration ########
plugins.security.ssl.transport.pemcert_filepath: <node>.crt
plugins.security.ssl.transport.pemkey_filepath: <node>-pkcs8.key
plugins.security.ssl.transport.pemtrustedcas_filepath: <CAcert>
plugins.security.ssl.transport.enforce_hostname_verification: true

plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: <node>.crt
plugins.security.ssl.http.pemkey_filepath: <node>-pkcs8.key
plugins.security.ssl.http.pemtrustedcas_filepath: <CAcert>

plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true

# at least one cert -- must be an EXACT copy of the DN for the certificates to be used for admin auth
plugins.security.authcz.admin_dn:
  - CN=Admin,O=OpenSearch,C=US

# these must be EXACT copies of the DN for the node certificates
plugins.security.nodes_dn:
  - CN=master-1.opensearch.localnet,O=OpenSearch,C=US
  - CN=data-hot-1.opensearch.localnet,O=OpenSearch,C=US

# this is all boiler plate ... no modifications needed
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
# this next bit should ALL be on one line ...
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

node.max_local_storage_nodes: 3

######## End OpenSearch Security Configuration ########

Note that there are two sections where the DNs from the certificates must be provided to enable admin authentication for security configuration, and for validating the nodes that will be part of the cluster. You can easily show the DNs for your certificates using this command:

openssl x509 -subject -nameopt RFC2253 -noout -in <cert file name>

The DN for the admin cert and the DNs for the node certs go in separate sections as shown above. Add all the cluster nodes into the nodes_dn section -- this must be the same for all nodes or they will not be able to communicate.

Security Plugin Configuration

While the security configuration files exist on all the nodes in the cluster, they are only used once and only by the node where the security initialization is performed. With that in mine, pick one node (the first master node will do nicely) to edit the configuration files as needed. There are many things that can be done with these files, but two items in particular should be addressed.

Authentication

On the initial master node only, configure /opt/opensearch/opensearch-1.1.0/plugins/opensearch-security/securityconfig/config.yml to allow certificate authentication. Enable both http and transport under clientcert_auth_domain

admin user password

The default password for the admin user is well-known, and should be changed for a production environment. The OpenSearch documentation provides

Certificates

Copy the following certificates/keys from /work/osdata/certs to the config directory:

• <CAcert>
• <node>.crt
• <node>-pkcs8.key
• admin.crt
• admin-pkcs8.crt

Starting The Node

Subject to the constraints described in the next section, the only thing left is to start the opensearch service:

sudo systemctl enable --now opensearch

Cluster Bootstrapping

Starting up a multi-node cluster is a bit tricky:

• Master-only nodes have no data
• Data-only nodes aren't master eligible
• Cluster initialization requires indexes to be available

Note that this process does NOT use the demo configuration -- put all the security plugin items in the opensearch.yml file from the beginning. This will cause the opensearch-tar-install.sh script to skip the security initialization -- this is intentional.

What this means is that you have to stand up multiple nodes before anything starts working ... the process should be:

1. Install and start what will be the 'primary' master ... reference point for cluster startup operations
2. Install and start one of the 'hot' data nodes
3. Run the 'securityadmin.sh' script to initialize the security subsystem
4. Install/start the other master nodes so that there is resiliency as nodes go up and down
5. install/start the other data nodes to allow replication of the system indexes.

Install OpenSearch Dashboards

Run the following commands as opensearch user:

wget https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/1.1.0/opensearch-dashboards-1.1.0-linux-x64.tar.gz
tar -zxf opensearch-dashboards-1.1.0-linux-x64.tar.gz
rm opensearch-dashboards-1.1.0-linux-x64.tar.gz

Create file /etc/systemd/system/opensearch-dashboards.service (as root) with the following contents:

[Unit]
Description=OpenSeach-Dasboards

[Service]
Type=simple
User=opensearch
Group=opensearch
ExecStart=/opt/opensearch/opensearch-dashboards-1.1.0/bin/opensearch-dashboards
Restart=always
WorkingDirectory=/opt/opensearch/opensearch-dashboards-1.1.0

[Install]
WantedBy=multi-user.target

Run the following commands as root:

systemctl daemon-reload
systemctl start opensearch-dashboards.service 
systemctl enable opensearch-dashboards.service

Next Steps

• Cluster Monitoring
  • Cerebro - implements much of the cluster monitoring capability from the commercial Kibana that's missing in OpenSearch Dashboards
    • installed on pro1 for now, but may put in a docker container/kubernetes deployment ...
• Data Ingest
  • Logstash - aggregate and process data through pipelines before sending to OpenSearch
  • Metricbeat - system data from Linux/Windows systems
  • Filebeat - capturing text from logfiles
    • capture OpenSearch logs into OpenSearch for better visibility and analysis
    • capture syslogs from the log server (pro6) for all systems
  • NVIDIA - simple script to get GPU data

References

• https://cryptopoolparty.com/blog/cardano-stake-pool-monitoring-ofo-stack/
  • Describes how to use OpenSearch to monitor another application -- but portions are very useful
• https://github.com/opensearch-project/dashboards-reports/releases/tag/chromium-1.12.0.0
  • download for headless chrome for generating PDF/PNG reports