OpenSearch Cluster Installation

Preparation

Do this on each cluster node (unless otherwise noted

Create Opensearch User

Debian/Ubuntu:

sudo adduser --system -uid 968 --shell /bin/bash --gecos 'OpenSearch User'  --group  --disabled-password --home /opt/opensearch opensearch

Fedora:

sudo adduser --system --uid 968 --shell /bin/bash --home-dir /opt/opensearch -m opensearch

Then add your normal user to the group:

sudo usermod -aG opensearch ewilliam

Set vm.max_map_count

Create /etc/sysctl.d/vm.maxmap_count (as root) and add the following line:

vm.max_map_count=262144

Run the following commands as root:

sysctl --system

Install Java 11 (optional)

The OpenSearch distro comes with Java 8 integrated into the packages. If this won't work for some reason, run the following commands as root:

apt update
apt install -y openjdk-11-jdk

Certificates

There are three types of certificates used in OpenSearch:

• CA certificate - one for the whole cluster to validate everything else
• Node certificate/key - identifies the node for internal cluster communications (transport layer) and data ingest (http layer)
  • the CN must be the FQDN of the node
  • even though opensearch doesn't seem to use them at this point, there should be SANs for the full FQDN, the short hostname, and the node's IP address
• User certificate/key - identifies a specific opensearch user
  • the CN is a username in opensearch (explicit or implicit)
  • username is used to determine roles/privileges in the cluster
  • Need at least one cert for admin authentication

Node and User certificates must all be signed by the cluster CA, and all keys must be in PKCS8 format. If user certificates are to be used by logstash, they must be imported (along with the key and CA) into a PKCS12 (.p12) keystore.

In summary, create the following certificates/keys

1. CA certificate/key
2. For each cluster node:
   • <node>.crt
   • <node>-pkcs8.key
3. For each user:
   • <user>.crt
   • <user>-pkcs8.key
   • <user>.p12

All these should be created and available for the configuration of each cluster node below.

Systemd Unit File

Create a systemd unit file in /etc/systemd/system/opensearch.service with the following contents:

[Unit]
Description=OpenSearch
Documentation=https://opensearch.org/docs/
Wants=network-online.target
After=network-online.target

[Service]
RuntimeDirectory=opensearch
WorkingDirectory=/opt/opensearch/opensearch-1.1.0
User=opensearch
Group=opensearch

ExecStart=/opt/opensearch/opensearch-1.1.0/opensearch-tar-install.sh

# StandardOutput is configured to redirect to journalctl since
# some error messages may be logged in standard output before
# elasticsearch logging system is initialized. Elasticsearch
# stores its logs in /var/log/elasticsearch and does not use
# journalctl by default. If you also want to enable journalctl
# logging, you can simply remove the "quiet" option from ExecStart.
StandardOutput=journal
StandardError=inherit

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65535

# Specifies the maximum number of processes
LimitNPROC=4096

# Specifies the maximum size of virtual memory
LimitAS=infinity

# Specifies the maximum file size
LimitFSIZE=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=0

# SIGTERM signal is used to stop the Java process
KillSignal=SIGTERM

# Send the signal only to the JVM rather than its control group
KillMode=process

# Java process is never killed
SendSIGKILL=no

# When a JVM receives a SIGTERM signal it exits with code 143
SuccessExitStatus=143

# Allow a slow startup before the systemd notifier module kicks in to extend the timeout
TimeoutStartSec=75

[Install]
WantedBy=multi-user.target

Install Opensearch Tarball

Change to the opensearch user (from root):

su - opensearch
cd ~

Run the following commands as opensearch user (from the opensearch home directory /opt/opensearch):

wget https://artifacts.opensearch.org/releases/bundle/opensearch/1.0.0/opensearch-1.0.0-linux-x64.tar.gz
tar -xvzf opensearch-1.0.0-linux-x64.tar.gz
rm opensearch-1.0.0-linux-x64.tar.gz

Create the systemd unit file (since the tarball install doesn't provide one:

Configure OpenSearch

Configuration varies depending on the type of node, but these are the places that need to be touched.

jvm.options

Set the heap size:

• 4G for master and cold data nodes
• 16G for hot data nodes

In general, it shouldn't be more than 50% of available memory

config.yml

On the initial master node only, configure /opt/opensearch/opensearch-1.1.0/plugins/opensearch-security/securityconfig/config.yml to allow certificate authentication. Enable both http and transport under clientcert_auth_domain

opensearch.yml:

While the OpenSearch software can be rather forgiving for low-security environments, if you turn on full hostname verification, it is a bit of a pain. The biggest thing to remember is that while it claims to be able to handle SAN certificates, IT CAN'T! What this boils down to is that if full verification is turned on, you cannot use IP addresses in the opensearch.yml (or any other configuration file such as the logstash pipelines).

• cluster.name - must be identical in all nodes
• node.name- can be the hostname, but doesn't have to be (must be unique)
• path.data - data nodes only -- must be unique to the node (i.e if using shared storage, each node must have its own directory on the shared volume)
• path.logs - must be unique to the node (same as above)
• network.host -- host's FQDN where all traffic will be received -- if you are doing transport-level host verification this CANNOT be an IP -- it can't handle the SANs in the certificates that make it work
• http.port -- leave it as the default (9200) but uncomment it to lock it in
• discovery.seed_hosts -- FQDN for all the masters (same for all nodes) -- only used for cluster bootstrap
• cluster.initial_master_nodes -- put the master node names here (not the hostnames) -- same for all nodes
• node.[master|data|ingest|remote] - set the type of the node (specify all four)
• node.attr.temp - (data nodes only) defines data storage tiers (hot/warm/cold)

Copy this block into the end of the stock opensearch.yml:

node.master: true
node.ingest: true
node.data: false
#node.remote: false

#node.attr.temp: hot

######## Start OpenSearch Security Configuration ########
plugins.security.ssl.transport.pemcert_filepath: <node>.crt
plugins.security.ssl.transport.pemkey_filepath: <node>-pkcs8.key
plugins.security.ssl.transport.pemtrustedcas_filepath: <CAcert>
plugins.security.ssl.transport.enforce_hostname_verification: true

plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: <node>.crt
plugins.security.ssl.http.pemkey_filepath: <node>-pkcs8.key
plugins.security.ssl.http.pemtrustedcas_filepath: <CAcert>

plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true

# at least one cert -- must be an EXACT copy of the DN for the certificates to be used for admin auth
plugins.security.authcz.admin_dn:
  - CN=Admin,O=WilliamsNet,ST=OH,C=US

# these must be EXACT copies of the DN for the node certificates
plugins.security.nodes_dn:
  - CN=poggin.williams.localnet,O=WilliamsNet,ST=OH,C=US
  - CN=caspian.williams.localnet,C=US,O=WilliamsNet

# this is all boiler plate ... no modifications needed
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
# this next bit should ALL be on one line ...
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

node.max_local_storage_nodes: 3

######## End OpenSearch Security Configuration ########

Certificates

Copy the following certificates/keys from /work/osdata/certs to the config directory:

• <CAcert>
• <node>.crt
• <node>-pkcs8.key
• admin.crt
• admin-pkcs8.crt

Starting The Node

Subject to the constraints described in the next section, the only thing left is to start the opensearch service:

sudo systemctl enable --now opensearch

Cluster Bootstrapping

Starting up a multi-node cluster is a bit tricky:

• Master-only nodes have no data
• Data-only nodes aren't master eligible
• Cluster initialization requires indexes to be available

Note that this process does NOT use the demo configuration -- put all the security plugin items in the opensearch.yml file from the beginning.

What this means is that you have to stand up multiple nodes before anything starts working ... the process should be:

1. Install what will be the 'primary' master ... reference point for cluster startup operations
2. Install one of the 'hot' data nodes
3. Run the 'securityadmin.sh' script to initialize the security subsystem
4. Install the other master nodes so that there is resiliency as nodes go up and down
5. install the other data nodes to allow replication of the system indexes.