OpenSearch: Difference between revisions
Jump to navigation
Jump to search
DrEdWilliams (talk | contribs) mNo edit summary Tag: visualeditor |
DrEdWilliams (talk | contribs) Tag: visualeditor |
||
| (2 intermediate revisions by the same user not shown) | |||
| Line 65: | Line 65: | ||
| | | | ||
|} | |} | ||
All nodes have access to the /work filesystem (NFS shared by pro6), and use /work/osdata/<hostname> for logs and data (if appropriate). | All nodes have access to the <code>/work</code> filesystem (NFS shared by '''pro6'''), and use <code>/work/osdata/<hostname></code> for logs and data (if appropriate). | ||
All communications with (port 9200) and within (port 9300) the cluster is SSL encrypted using certificates generated based on a self-signed CA. All certs are located in <code>/work/osdata/certs</code>. | |||
=== Aggregation and Processing === | === Aggregation and Processing === | ||
Logstash is installed on calormen and serves as the primary ingest route for the cluster. | Logstash is installed on '''calormen''' and serves as the primary ingest route for the cluster. Additionally, another instance of logstash is installed on '''aslan''' for testing/validation purposes. | ||
=== Data Collection === | === Data Collection === | ||
All servers should have metricbeat installed, feeding data to logstash on calormen | |||
* All servers should have metricbeat installed, feeding data to logstash on calormen | |||
* Servers with CUDA-capable NVIDIA GPUs have the NVIDIA collection script installed: | |||
** '''poggin''' (Quadro GP100) | |||
** '''aslan''' (GTX 1080 Ti) | |||
** '''rogin''' (GTX 1080) | |||
** '''compute1''' (GTX 1070) | |||
** '''griffle''' (GTX 1060, GTX 950) | |||
** '''trumpkin''' (GTX 950) | |||
=== Cluster Monitoring === | |||
The OpenSearch cluster is monitored using the methods described in the [[Cluster Monitoring]] page: | |||
* OpenSearch node logs are collected by a single instance of filebeat running on '''poggin''', accessing the logs directly from the shared filesystem. The log data is sent through logstash on '''calormen''', where the JSON log data is expanded and time-tagged | |||
* Dashboards and visualizations are set up to view the log data and the audit data | |||
=== Authentication === | |||
* User authentication to Dashboards is username/password using the OpenSearch internal user database | |||
* API authentication uses username/password and certificate authentication methods | |||
* Authentication of nodes in the cluster uses node certificates and the DNs contained in the <code>opensearch.yml</code> configuration files | |||
*All certs are located in <code>/work/osdata/certs</code> | |||
== OpenSearch How-Tos == | == OpenSearch How-Tos == | ||
| Line 79: | Line 102: | ||
* [[Data Collection Methods]] | * [[Data Collection Methods]] | ||
* [[OpenSearch Cluster Monitoring|Cluster Monitoring]] | * [[OpenSearch Cluster Monitoring|Cluster Monitoring]] | ||
*[[Cluster Operations]] | |||
Latest revision as of 12:51, 26 November 2021
WilliamsNet OpenSearch Architecture[edit]
For convenience, and to enable this cluster to be usable as a Test and Development System for the HPCMP OpenSearch cluster, the configuration and naming conventions here will mirror that used in the HPCMP environment:
- Indices will be named using this style:
<data source>-<schema version>-<system name>-YYYY.MM.DD - Data will be routed through logstash to enable processing and augmentation prior to indexing in the cluster
OpenSearch Cluster & Dashboards[edit]
The OpenSearch Cluster is composed of these node types:
| Node Type | Servers | System
RAM (GB) |
Heap
Size (GB) |
|---|---|---|---|
| Master/Ingest | poggin | 16 | 4 |
| aravis | 8 | 4 | |
| lamppost | 8 | 4 | |
| Hot Data | caspian | 72 | 16 |
| belisar | 72 | 16 | |
| uvilas | 72 | 16 | |
| Cold Data | calormen | 16 | 6 |
| trumpkin | 12 | 6 | |
| Dashboards | pro1 | 40 | |
| Monitoring | pro1 (cerebro) | 40 |
All nodes have access to the /work filesystem (NFS shared by pro6), and use /work/osdata/<hostname> for logs and data (if appropriate).
All communications with (port 9200) and within (port 9300) the cluster is SSL encrypted using certificates generated based on a self-signed CA. All certs are located in /work/osdata/certs.
Aggregation and Processing[edit]
Logstash is installed on calormen and serves as the primary ingest route for the cluster. Additionally, another instance of logstash is installed on aslan for testing/validation purposes.
Data Collection[edit]
- All servers should have metricbeat installed, feeding data to logstash on calormen
- Servers with CUDA-capable NVIDIA GPUs have the NVIDIA collection script installed:
- poggin (Quadro GP100)
- aslan (GTX 1080 Ti)
- rogin (GTX 1080)
- compute1 (GTX 1070)
- griffle (GTX 1060, GTX 950)
- trumpkin (GTX 950)
Cluster Monitoring[edit]
The OpenSearch cluster is monitored using the methods described in the Cluster Monitoring page:
- OpenSearch node logs are collected by a single instance of filebeat running on poggin, accessing the logs directly from the shared filesystem. The log data is sent through logstash on calormen, where the JSON log data is expanded and time-tagged
- Dashboards and visualizations are set up to view the log data and the audit data
Authentication[edit]
- User authentication to Dashboards is username/password using the OpenSearch internal user database
- API authentication uses username/password and certificate authentication methods
- Authentication of nodes in the cluster uses node certificates and the DNs contained in the
opensearch.ymlconfiguration files - All certs are located in
/work/osdata/certs
OpenSearch How-Tos[edit]
Topics related to creating, maintaining, and using an elastic cluster