OpenSearch Cluster Installation - Revision history

DrEdWilliams: /* Install OpenSearch Dashboards */

2021-11-30T22:18:01Z

Install OpenSearch Dashboards

← Older revision		Revision as of 22:18, 30 November 2021
Line 269:		Line 269:
	Create file <code>/etc/systemd/system/opensearch-dashboards.service</code> (as root) with the following contents:		Create file <code>/etc/systemd/system/opensearch-dashboards.service</code> (as root) with the following contents:
	[Unit]		[Unit]
	Description=~~OpenSeach~~-Dasboards		Description=OpenSearch-Dasboards

	[Service]		[Service]

DrEdWilliams: /* Systemd Unit File */

2021-11-30T20:32:32Z

Systemd Unit File

← Older revision		Revision as of 20:32, 30 November 2021
Line 37:		Line 37:

	[Service]		[Service]
			Type=exec
	RuntimeDirectory=opensearch		RuntimeDirectory=opensearch
			PrivateTmp=true
			Environment=OPENSEARCH_HOME=/opt/opensearch/opensearch
			Environment=OPENSEARCH_PATH_CONF=/opt/opensearch/opensearch/config
			Environment=PID_DIR=/opt/opensearch/opensearch
			Environment=OPENSEARCH_SD_NOTIFY=true
			EnvironmentFile=-/opt/opensearch/opensearch/opensearch.env

	WorkingDirectory=/opt/opensearch/opensearch		WorkingDirectory=/opt/opensearch/opensearch

	User=opensearch		User=opensearch
	Group=opensearch		Group=opensearch

	ExecStart=/opt/opensearch/opensearch/opensearch-~~tar~~-~~install.sh~~		ExecStart=/opt/opensearch/opensearch/bin/opensearch -p ${PID_DIR}/opensearch.pid --quiet

	# StandardOutput is configured to redirect to journalctl since		# StandardOutput is configured to redirect to journalctl since

DrEdWilliams: /* Systemd Unit File */

2021-11-26T15:55:06Z

Systemd Unit File

← Older revision		Revision as of 15:55, 26 November 2021
Line 38:		Line 38:
	[Service]		[Service]
	RuntimeDirectory=opensearch		RuntimeDirectory=opensearch
	WorkingDirectory=/opt/opensearch/opensearch~~-1.1.0~~		WorkingDirectory=/opt/opensearch/opensearch
	User=opensearch		User=opensearch
	Group=opensearch		Group=opensearch

	ExecStart=/opt/opensearch/opensearch~~-1.1.0~~/opensearch-tar-install.sh		ExecStart=/opt/opensearch/opensearch/opensearch-tar-install.sh

	# StandardOutput is configured to redirect to journalctl since		# StandardOutput is configured to redirect to journalctl since

DrEdWilliams: /* jvm.options */

2021-11-26T01:16:44Z

jvm.options

← Older revision		Revision as of 01:16, 26 November 2021
Line 129:		Line 129:
	* 16G for hot data nodes		* 16G for hot data nodes

	This must be done individually for each node.		This must be done individually for each node -- create the file <code>config/jvm.options.d/heapsize.options</code> with the heap size options:
			-Xms4g
			-Xmx4g
			... with, of course, the proper heap size for that node.

	=== <code>opensearch.yml</code> ===		=== <code>opensearch.yml</code> ===

DrEdWilliams: /* Install Opensearch Tarball */

2021-11-25T19:15:34Z

Install Opensearch Tarball

← Older revision		Revision as of 19:15, 25 November 2021
Line 113:		Line 113:
	All these should be created and available for the configuration of each cluster node below.		All these should be created and available for the configuration of each cluster node below.

	== Install ~~Opensearch~~ Tarball ==		== Install OpenSearch Tarball ==
	The OpenSearch tarball must be installed on each node in the cluster; there is no difference at this point between the installations on the different types of nodes. Assuming the deployment will use the <code>opensearch</code> user created above ... change to the <code>opensearch</code> user (from root):		The OpenSearch tarball must be installed on each node in the cluster; there is no difference at this point between the installations on the different types of nodes. Assuming the deployment will use the <code>opensearch</code> user created above ... change to the <code>opensearch</code> user (from root):
	su - opensearch		su - opensearch

DrEdWilliams: /* Install Java 11 (optional) */

2021-11-25T19:14:12Z

Install Java 11 (optional)

← Older revision		Revision as of 19:14, 25 November 2021
Line 25:		Line 25:
	Run the following command as root to load the new configuration:		Run the following command as root to load the new configuration:
	sysctl --system		sysctl --system

	~~=== Install Java 11 (optional) ===~~
	~~The OpenSearch distro comes with Java 8 integrated into the packages. If this won't work for some reason, run the following commands as root:~~
	~~apt update~~
	~~apt install -y openjdk-11-jdk~~

	~~Similar commands can be used on a Fedora/RHEL system.~~

	=== Systemd Unit File ===		=== Systemd Unit File ===

DrEdWilliams: /* Install OpenSearch Dashboards */

2021-11-25T11:37:40Z

Install OpenSearch Dashboards

← Older revision		Revision as of 11:37, 25 November 2021
Line 21:		Line 21:

	=== Set vm.max_map_count ===		=== Set vm.max_map_count ===
	Create <code>/etc/sysctl.d/vm.maxmap_count</code> (as root) and add the following line:		Create <code>/etc/sysctl.d/vm.maxmap_count.conf</code> (as root) and add the following line:
	vm.max_map_count=262144		vm.max_map_count=262144
	Run the following ~~commands~~ as root to load the new configuration:		Run the following command as root to load the new configuration:
	sysctl --system		sysctl --system

Line 148:		Line 148:
	* <code>network.host</code> -- host's FQDN where all traffic will be received		* <code>network.host</code> -- host's FQDN where all traffic will be received
	**Can be a single hostname or an array of hostnames in <code>["name", "name"]</code> format		**Can be a single hostname or an array of hostnames in <code>["name", "name"]</code> format
	**If you are doing transport-level host verification this CANNOT be an IP -- OpenSearch can't handle the SANs in the certificates ~~that~~ make it work		**If you are doing transport-level host verification this CANNOT be an IP -- OpenSearch can't handle the SANs in the certificates needed to make it work
	* <code>http.port</code> -- leave it as the default (<code>9200</code>) but uncomment it to lock it in		* <code>http.port</code> -- leave it as the default (<code>9200</code>) but uncomment it to lock it in
	* <code>discovery.seed_hosts</code> -- FQDN for all the masters (same for all nodes) -- only used for cluster bootstrap		* <code>discovery.seed_hosts</code> -- FQDN for all the masters (same for all nodes) -- only used for cluster bootstrap
Line 260:		Line 260:
	tar -zxf opensearch-dashboards-1.2.0-linux-x64.tar.gz		tar -zxf opensearch-dashboards-1.2.0-linux-x64.tar.gz

	Configure the <code>opensearch-dashboards.yml</code> file as needed for your ~~configuration~~.		Configure the <code>opensearch-dashboards.yml</code> file as needed for your situation.

	Create file <code>/etc/systemd/system/opensearch-dashboards.service</code> (as root) with the following contents:		Create file <code>/etc/systemd/system/opensearch-dashboards.service</code> (as root) with the following contents:

DrEdWilliams: /* Next Steps */

2021-11-24T23:50:31Z

Next Steps

← Older revision		Revision as of 23:50, 24 November 2021
Line 270:		Line 270:
	User=opensearch		User=opensearch
	Group=opensearch		Group=opensearch
	ExecStart=/opt/opensearch/opensearch-dashboards-1.1.0/bin/opensearch-dashboards		ExecStart=/opt/opensearch/opensearch-dashboards-1.2.0/bin/opensearch-dashboards
	Restart=always		Restart=always
	WorkingDirectory=/opt/opensearch/opensearch-dashboards-1.1.0		WorkingDirectory=/opt/opensearch/opensearch-dashboards-1.2.0

	[Install]		[Install]
Line 280:		Line 280:
	systemctl start opensearch-dashboards.service		systemctl start opensearch-dashboards.service
	systemctl enable opensearch-dashboards.service		systemctl enable opensearch-dashboards.service

	~~== Next Steps ==~~

	* '''[[OpenSearch Cluster Monitoring\|Cluster Monitoring]]'''
	** [https://github.com/lmenezes/cerebro Cerebro] - implements much of the cluster monitoring capability from the commercial Kibana that's missing in OpenSearch Dashboards
	*** installed on pro1 for now, but may put in a docker container/kubernetes deployment ...
	* '''[[Data Collection Methods\|Data Ingest]]'''
	** Logstash - aggregate and process data through pipelines before sending to OpenSearch
	** Metricbeat - system data from Linux/Windows systems
	** Filebeat - capturing text from logfiles
	*** capture OpenSearch logs into OpenSearch for better visibility and analysis
	*** capture syslogs from the log server (pro6) for all systems
	** NVIDIA - simple script to get GPU data

	== References ==		== References ==

	* https://cryptopoolparty.com/blog/cardano-stake-pool-monitoring-ofo-stack/		* https://cryptopoolparty.com/blog/cardano-stake-pool-monitoring-ofo-stack/
	** Describes how to use OpenSearch to monitor another application -- but portions ~~are very useful~~		** Describes how to use OpenSearch to monitor another application -- but portions were used to create this document
	*https://github.com/opensearch-project/dashboards-reports/releases/tag/chromium-1.12.0.0		*https://github.com/opensearch-project/dashboards-reports/releases/tag/chromium-1.12.0.0
	**download for headless chrome for generating PDF/PNG reports		**download for headless chrome for generating PDF/PNG reports

DrEdWilliams: DrEdWilliams moved page OpenSearch Cluster to OpenSearch Cluster Installation

2021-11-24T23:45:03Z

DrEdWilliams moved page OpenSearch Cluster to OpenSearch Cluster Installation

← Older revision	Revision as of 23:45, 24 November 2021
(No difference)

DrEdWilliams: /* opensearch.yml */

2021-11-24T23:44:22Z

opensearch.yml

← Older revision		Revision as of 23:44, 24 November 2021
Line 202:		Line 202:
	openssl x509 -subject -nameopt RFC2253 -noout -in <cert file name>		openssl x509 -subject -nameopt RFC2253 -noout -in <cert file name>
	The DN for the admin cert and the DNs for the node certs go in separate sections as shown above. Add all the cluster nodes into the nodes_dn section -- this must be the same for all nodes or they will not be able to communicate.		The DN for the admin cert and the DNs for the node certs go in separate sections as shown above. Add all the cluster nodes into the nodes_dn section -- this must be the same for all nodes or they will not be able to communicate.

	~~=== Security Plugin Configuration ===~~
	While the security configuration files exist on all the nodes in the cluster, they are only used once and only by the node where the security initialization is performed. With that in mine, pick one node (the first master node will do nicely) to edit the configuration files as needed. There are many things that can be done with these files, but two items in particular should be addressed.

	~~==== Authentication ====~~
	'''On the initial master node only''', configure <code>/opt/opensearch/opensearch-1.1.0/plugins/opensearch-security/securityconfig/config.yml</code> to allow certificate authentication. Enable both '''http''' and '''transport''' under <code>clientcert_auth_domain</code>

	~~==== admin user password ====~~
	~~The default password for the admin user is well-known, and should be changed for a production environment. The OpenSearch documentation provides~~

	=== Certificates ===		=== Certificates ===
	Copy the following certificates/keys ~~from /work/osdata/certs to~~ the config directory:		Copy the following certificates/keys that were created previously into the config directory of each node:

	* <CAcert>		* <CAcert>
Line 220:		Line 211:
	* admin.crt		* admin.crt
	* admin-pkcs8.crt		* admin-pkcs8.crt
			Verify that the <code>opensearch</code> user is able to at least read all of the certs/keys, or the cluster will not function.

			=== Security Plugin Configuration ===
			While the security configuration files exist on all the nodes in the cluster, they are only used once and only by the node where the security initialization is performed. With that in mine, pick one node (the first master node will do nicely) to edit the configuration files as needed. There are many things that can be done with these files, but two items in particular should be addressed. The files are located in the <code>plugins/opensearch-security/securityconfig</code> directory in the opensearch distribution.

			==== Authentication ====
			<code>config.yml</code> is where the authentication options are established. This will need to be configured to support your authentication options. The OpenSearch documentation describes the different options [https://opensearch.org/docs/latest/security-plugin/configuration/configuration/ here].

			The default configuration supports basic username/password authentication, but if you wish to use certificate authentication for API clients or ingest pipelines using logstash, you need to enable both '''http''' and '''transport''' under <code>clientcert_auth_domain</code>. Proxy authentication and OpenID Connect authentication is possible as well ... see the documentation.

			==== Admin user password ====
			The default password for the admin user is well-known, and should be changed for a production environment. The OpenSearch documentation provides instructions on how to [https://opensearch.org/docs/latest/security-plugin/configuration/yaml/#internal_usersyml generate passwords] for the <code>internal_users.yml</code> file. Note that '''the admin password cannot be set through the dashboard or the API''' -- it must be set in this file, and then (re)loaded to take effect.

			=== Security Plugin Initialization ===
			Once all security configuration files are finished, they must be loaded into the OpenSearch cluster. This can only be done once enough of the cluster is up to provide a master node and at least one data node, as the security configuration is loaded into an index in the cluster. See the instructions below for specific sequencing. This command will initialize the security plugin:
			plugins/opensearch-security/tools/securityadmin.sh \
			-cacert config/<CA cert> -cert config/admin.crt -key config/admin-pkcs8.key \
			-cd plugins/opensearch-security/securityconfig/ -host master-1.opensearch.localnet -icl -nhnv
			... but again -- wait to do this until the right time below.

	=== Starting The Node ===		=== Starting The Node ===
	Subject to the constraints described in the next section, the only thing left is to start the opensearch service:		Subject to the constraints described in the next section, the only thing left is to start the opensearch service:
	sudo systemctl enable --now opensearch		sudo systemctl enable --now opensearch
			Again, this must be done on each node in the correct sequence ... see below.

	== Cluster Bootstrapping ==		== Cluster Bootstrapping ==
Line 238:		Line 249:
	# Install and start what will be the 'primary' master ... reference point for cluster startup operations		# Install and start what will be the 'primary' master ... reference point for cluster startup operations
	# Install and start one of the 'hot' data nodes		# Install and start one of the 'hot' data nodes
	# Run the 'securityadmin.sh' script to initialize the security subsystem		# Run the 'securityadmin.sh' script (using the command above) to initialize the security subsystem
	# Install/start the other master nodes so that there is resiliency as nodes go up and down		# Install/start the other master nodes so that there is resiliency as nodes go up and down
	# install/start the other data nodes to allow replication of the system indexes.		# install/start the other data nodes to allow replication of the system indexes.

	== Install OpenSearch Dashboards ==		== Install OpenSearch Dashboards ==
			Finally, install the OpenSearch Dashboards using the tarball as well.

	Run the following commands as opensearch user:		Run the following commands as opensearch user:
	wget https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/1.1.0/opensearch-dashboards-1.1.0-linux-x64.tar.gz		wget [https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/1.1.0/opensearch-dashboards-1.1.0-linux-x64.tar.gz https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/1.2.0/opensearch-dashboards-1.2.0-linux-x64.tar.gz]
	~~tar~~ -~~zxf~~ opensearch-dashboards-1.1.0-linux-x64.tar.gz		tar -zxf opensearch-dashboards-1.2.0-linux-x64.tar.gz
	rm opensearch-dashboards-1.1.0-linux-x64.tar.gz
			Configure the <code>opensearch-dashboards.yml</code> file as needed for your configuration.

	Create file <code>/etc/systemd/system/opensearch-dashboards.service</code> (as root) with the following contents:		Create file <code>/etc/systemd/system/opensearch-dashboards.service</code> (as root) with the following contents:
	[Unit]		[Unit]