Data Collection Methods: Difference between revisions
Jump to navigation
Jump to search
DrEdWilliams (talk | contribs) mNo edit summary Tag: visualeditor |
DrEdWilliams (talk | contribs) Tag: visualeditor |
||
| Line 10: | Line 10: | ||
The elastic suite of applications includes tools that are usable for both the Elastic tools and the open source alternatives (such as OpenSearch): | The elastic suite of applications includes tools that are usable for both the Elastic tools and the open source alternatives (such as OpenSearch): | ||
* Logstash | *Logstash - aggregate and process data through pipelines before sending to OpenSearch | ||
* Filebeat | *Metricbeat - system data from Linux/Windows systems | ||
* | *Filebeat - capturing text from logfiles | ||
**capture OpenSearch logs into OpenSearch for better visibility and analysis | |||
**capture syslogs from the log server (pro6) for all systems | |||
=== Prometheus Exporters: === | === Prometheus Exporters: === | ||
[https://prometheus.io/ Prometheus] implements a pull method vs the push method used by the Elastic products. Metricbeat serves as an interface between the two methods -- one of its modules can read prometheus format metrics -- either produced natively by applications or by specific exporter applications. A list of exporters from the prometheus website can be found at https://prometheus.io/docs/instrumenting/exporters/ | [https://prometheus.io/ Prometheus] implements a pull method vs the push method used by the Elastic products. Metricbeat serves as an interface between the two methods -- one of its modules can read prometheus format metrics -- either produced natively by applications or by specific exporter applications. A list of exporters from the prometheus website can be found at https://prometheus.io/docs/instrumenting/exporters/ | ||
A general-purpose exporter is [https://github.com/prometheus/node_exporter node_exporter] -- it can export a large range of metrics across many areas of system operations. | * A general-purpose exporter is [https://github.com/prometheus/node_exporter node_exporter] -- it can export a large range of metrics across many areas of system operations. | ||
=== Custom Tools === | |||
* NVIDIA - simple script to get GPU data | |||
Latest revision as of 00:15, 18 November 2021
(notes -- work in progress)
Getting data for OpenSearch/ElasticSearch is a matter of:
- Identifying the data to be gathered
- Finding a way to surface and collect that data
- Send the data to the ElasticSearch/OpenSearch cluster
Elastic Products[edit]
The elastic suite of applications includes tools that are usable for both the Elastic tools and the open source alternatives (such as OpenSearch):
- Logstash - aggregate and process data through pipelines before sending to OpenSearch
- Metricbeat - system data from Linux/Windows systems
- Filebeat - capturing text from logfiles
- capture OpenSearch logs into OpenSearch for better visibility and analysis
- capture syslogs from the log server (pro6) for all systems
Prometheus Exporters:[edit]
Prometheus implements a pull method vs the push method used by the Elastic products. Metricbeat serves as an interface between the two methods -- one of its modules can read prometheus format metrics -- either produced natively by applications or by specific exporter applications. A list of exporters from the prometheus website can be found at https://prometheus.io/docs/instrumenting/exporters/
- A general-purpose exporter is node_exporter -- it can export a large range of metrics across many areas of system operations.
Custom Tools[edit]
- NVIDIA - simple script to get GPU data