Editing OpenSearch Cluster Monitoring (section)

=== Logfiles ===
Looking in logfiles for messages can be a difficult prospect as each node independently collects its own output, and the cluster scheduler can assign tasks to different nodes -- and you may not know which node a task is executing on.  The solution is to collect all the logs in one place where they can be searched and correlated -- fortunately, there is a place to do that:  OpenSearch itself.  In an ideal world, there would be a separate cluster to monitor the main cluster -- so that if the main cluster is non-functional, you can analyze logs to identify the cause.  Lacking that, the logs can be collected as part of the main cluster.

To accomplish this, install filebeat on each cluster node (or in a single place if logfiles are collected on a shared filesystem):

* Use the 'elasticsearch' module to collect the different log components (prefer JSON formats) and send to logstash

 # Module: elasticsearch
 # Docs: <nowiki>https://www.elastic.co/guide/en/beats/filebeat/7.15/filebeat-module-elasticsearch.html</nowiki>
 
 - module: elasticsearch
   # Server log
   server:
     enabled: true
     # Set custom paths for the log files. If left empty,
     # Filebeat will choose the paths depending on your OS.
     var.paths:
       - /work/osdata/*/logs/*_server.json
   gc:
     enabled: false
     # Set custom paths for the log files. If left empty,
     # Filebeat will choose the paths depending on your OS.
     #var.paths:
   audit:
     enabled: false
     # Set custom paths for the log files. If left empty,
     # Filebeat will choose the paths depending on your OS.
     #var.paths:
   slowlog:
     enabled: true
     # Set custom paths for the log files. If left empty,
     # Filebeat will choose the paths depending on your OS.
     var.paths:
       - /work/osdata/*/logs/*_index_search_slowlog.json
       - /work/osdata/*/logs/*_index_indexing_slowlog.json
   deprecation:
     enabled: true
     # Set custom paths for the log files. If left empty,
     # Filebeat will choose the paths depending on your OS.
     var.paths:
       - /var/log/elasticsearch/*_deprecation.json  # JSON logs

* Use a pipeline in logstash to parse the JSON into fields, using the logfile timestamp as the timestamp for the record.
 input { 
     beats { 
         port => 5044 
     } 
 }
 filter {
   json {
     source => "message"
     remove_field => [ "message" ]
   }
   date {
     match => [ "timestamp", "ISO8601" ]
   }
 }
 output {
   opensearch {
       ssl => true
       ssl_certificate_verification => true
       cacert => "/etc/logstash/WilliamsNetCA.pem"
       keystore => "/etc/logstash/calormen.p12"
       keystore_password => "xxxxx"
       hosts => ["<nowiki>https://poggin.williams.localnet:9200</nowiki>", "<nowiki>https://aravis.williams.localnet:9200</nowiki>", "<nowiki>https://lamppost.williams.localnet:9200</nowiki>"]
       index => "opensearch-logs-v1-calormen-%{+YYYY.MM.dd}"
   }
 }