Editing OpenSearch

== WilliamsNet OpenSearch Architecture ==
For convenience, and to enable this cluster to be usable as a Test and Development System for the HPCMP OpenSearch cluster, the configuration and naming conventions here will mirror that used in the HPCMP environment:

* Indices will be named using this style:  <code><nowiki><data source>-<schema version>-<system name>-YYYY.MM.DD</nowiki></code>
* Data will be routed through logstash to enable processing and augmentation prior to indexing in the cluster

=== OpenSearch Cluster & Dashboards ===
The OpenSearch Cluster is composed of these node types:
{| class="wikitable"
!Node Type
!Servers
!System
RAM (GB)
!Heap
Size (GB)
|-
|Master/Ingest
|poggin
|16
|4
|-
|
|aravis
|8
|4
|-
|
|lamppost
|8
|4
|-
|Hot Data
|caspian
|72
|16
|-
|
|belisar
|72
|16
|-
|
|uvilas
|72
|16
|-
|Cold Data
|calormen
|16
|6
|-
|
|trumpkin
|12
|6
|-
|Dashboards
|pro1
|40
|
|-
|Monitoring
|pro1 (cerebro)
|40
|
|}
All nodes have access to the <code>/work</code> filesystem (NFS shared by '''pro6'''), and use <code>/work/osdata/<hostname></code> for logs and data (if appropriate).

All communications with (port 9200) and within (port 9300) the cluster is SSL encrypted using certificates generated based on a self-signed CA.  All certs are located in <code>/work/osdata/certs</code>.

=== Aggregation and Processing ===
Logstash is installed on '''calormen''' and serves as the primary ingest route for the cluster.  Additionally, another instance of logstash is installed on '''aslan''' for testing/validation purposes.

=== Data Collection ===

* All servers should have metricbeat installed, feeding data to logstash on calormen
* Servers with CUDA-capable NVIDIA GPUs have the NVIDIA collection script installed:
** '''poggin''' (Quadro GP100)
** '''aslan''' (GTX 1080 Ti)
** '''rogin''' (GTX 1080)
** '''compute1''' (GTX 1070)
** '''griffle''' (GTX 1060, GTX 950)
** '''trumpkin''' (GTX 950)

=== Cluster Monitoring ===
The OpenSearch cluster is monitored using the methods described in the [[Cluster Monitoring]] page:

* OpenSearch node logs are collected by a single instance of filebeat running on '''poggin''', accessing the logs directly from the shared filesystem.  The log data is sent through logstash on '''calormen''', where the JSON log data is expanded and time-tagged
* Dashboards and visualizations are set up to view the log data and the audit data

=== Authentication ===

* User authentication to Dashboards is username/password using the OpenSearch internal user database
* API authentication uses username/password and certificate authentication methods
* Authentication of nodes in the cluster uses node certificates and the DNs contained in the <code>opensearch.yml</code> configuration files
*All certs are located in <code>/work/osdata/certs</code>

== OpenSearch How-Tos ==
Topics related to creating, maintaining, and using an elastic cluster

* [[OpenSearch Cluster|Installing an OpenSearch Cluster & Dashboards]]
* [[Data Collection Methods]]
* [[OpenSearch Cluster Monitoring|Cluster Monitoring]]
*[[Cluster Operations]]