diff --git a/helm/examples/metrics/README.md b/helm/examples/metrics/README.md index 387a756eea8..750a35337b5 100644 --- a/helm/examples/metrics/README.md +++ b/helm/examples/metrics/README.md @@ -187,21 +187,25 @@ sink will report metrics from all components to the same index. Therefore, the index must be configured to receive metrics from all components. #### Index Configuration -The index must be created in ElasticSearch before metrics can be reported to it. The name is passed -to the sink as a configuration setting. The index must be created with the following settings: +The sink requires the index to be created and configured fully in ElasticSearch in order for the sink to report +metrics. The sink reads configuration data from the index in order to properly report metrics. Future versions +of the sink may include the capability to create the index if it does not exist. The name of the index is passed +to the sink as a configuration setting. + +The following must be configured in the index in order for the sink to properly report metrics: ##### Dynamic Mapping -The index must be created with dynamic mapping enabled. Dynamic mapping allows framework metric -data types to be stored in the index in their native types. Without dynamic mapping, the ElasticSearch -default mapping does not properly map value to unsigned 64-bit integers. +The index must be configured with dynamic mapping enabled. Dynamic mapping allows storing of framework metric +values using native types. Without dynamic mapping, the ElasticSearch default mapping does not properly map +values to unsigned 64-bit integers. -To create an index with dynamic mapping, use the following object when creating the index: +To create an index with dynamic mapping, use the following object, or other means, when creating the index: ```code json { "mappings": { "dynamic_templates": [ { - "hpcc_metric_count_to_unsigned_long": { + "hpcc_metrics_count_suffix": { "match": "*_count", "mapping": { "type": "unsigned_long" @@ -209,7 +213,7 @@ To create an index with dynamic mapping, use the following object when creating } }, { - "hpcc_metric_gauge_to_unsigned_long": { + "hpcc_metrics_gauge_suffix": { "match": "*_gauge", "mapping": { "type": "unsigned_long" @@ -217,7 +221,7 @@ To create an index with dynamic mapping, use the following object when creating } }, { - "hpcc_metric_histogram_to_histogram": { + "hpcc_metrics_histogram_suffix": { "match": "*_histogram", "mapping": { "type": "histogram" @@ -229,11 +233,11 @@ To create an index with dynamic mapping, use the following object when creating } ``` -Note that there may be other means for adding the required dynamic mapping to the index. - -The _match_ values above are representative of typical values. The actual values may -vary depending on the cluster configuration and shared use of the index across multiple clusters. -In all cases, the configuration of the sink must match that of the dynamic mappings in the index. +The object names beginning with "hpcc_metric_" represent the mappings for the three affected hpcc metrics types. +The _match_ value for each defines the expected suffix for each HPCC metric value, based on type, when metrics +are reported and indexed. The object names above are the default values used by the sink. The index can be +configured with different object names (for environment flexibility), but the object names must be provided to the +sink as additional configuration settings (defined below). #### Enabling ElasticSearch Metrics Sink for Kubernetes To enable reporting of metrics to ElasticSearch, add the metric configuration settings to @@ -255,13 +259,16 @@ Make a copy and modify as needed for your installation. The ElasticSearch sink defines the following settings **Host** -The host settings define the ElasticSearch server to which metrics are reported. The settings are: +The host settings define the server hosting ElasticSearch to which metrics are reported. The settings are: * domain - The domain or IP address of the ElasticSearch server. (required) * protocol - The protocol used to connect to the ElasticSearch server. (default: https) * port - The port number of the ElasticSearch server. (default: 9200) -* certificateFilePath - Path to the file containing the certificate used to connect to the ElasticSearch server. (optional) - +* certificateFilePath - Path to the file containing the certificate used to connect to the ElasticSearch server. +(optional) +* connectTimeout - The time in seconds to wait for a connection to be established to the (default: 5) +* readTimeout - The time in seconds to wait for a response from the server for a read operation. (default: 5) +* writeTimeout - The time in seconds to wait for a response from the server for a write operation. (default: 5) **Authentication** @@ -276,20 +283,35 @@ the ElasticSearch server. (optional, valid for Kubernetes only) * credentialsVaultId - The vault ID containing the credentials used to authenticate to the ElasticSearch server. (optional, valid for Vault only) -For **basic** authentication, the following settings are required, regardless if the credentials are stored -as a secret or in the environment.xml file. +For **basic** authentication, the following settings are required. If a secret or vault is defined, these +values are store there and are not required in the configuration file. Otherwise, they are required in the +configuration file (environment.xml). * username - The username used to authenticate to the ElasticSearch server. * password - The password used to authenticate to the ElasticSearch server. When stored in the environment.xml file, it shall be encrypted using standard environment.xml encryption. **Index** -The index settings define the index where metrics are indexed. The settings are: +The index must be created and configred in ElasticSearch before the sink will load and report +metrics. The following settings describe what must be configured in ElasticSearch. * name - The name of the index to which metrics are reported. (required) -* countMetricSuffix - The suffix used to identify count metrics. (default: count) -* gaugeMetricSuffix - The suffix used to identify gauge metrics. (default: gauge) -* histogramMetricSuffix - The suffix used to identify histogram metrics. (default: histogram) +* countSuffixMappingName - See below. (default: hpcc_metrics_count_suffix) +* gaugeSuffixMappingName - See below. (default: hpcc_metrics_gauge_suffix) +* histogramSuffixMappingName - See below. (default: hpcc_metrics_histogram_suffix) + +The _*SuffixMappingName_ values are object names in the index _dynamic_templates_ object of the +index's _mappings_ object. These MUST be configured in the index. The mapping name objects contain +a _match_ member whose value is used as the suffix for the related HPCC metric type. The format +is expected to be "*" where "*" is part of the defined match pattern syntax defined by +ElasticSearch and "" is the suffix appended to each HPCC metric based on type. For example, +If a _match_ value is defined as "*_count", then the sink would add "_count" to the end of each +metric when indexing measurements during a report. This ensures that each HPCC metric is stored in +the index with the correct type. Please refer to the ElasticSearch documentation for more information +on dynamically mapping types when indexing documents. + +For convenience, if the index dynamic templates configuration settings do not include a gauge +suffix setting, the value for the count suffix setting is used. Standard periodic metric sink settings are also available. @@ -306,10 +328,11 @@ Add the following to the environment.xml configuration file (note that some valu - + - + getProp("@certificateFilePath", certificateFilePath); // Get authentication settings, if present - Owned pAuthConfigTree = pSettingsTree->getPropTree("authentication"); + Owned pAuthConfigTree = pHostConfigTree->getPropTree("authentication"); if (!pAuthConfigTree) return true; @@ -143,6 +132,12 @@ bool ElasticMetricSink::getHostConfig(const IPropertyTree *pSettingsTree) decrypt(password, encryptedPassword.str()); //MD5 encrypted in config } } + + // Read optional timeout values + connectTimeout = pHostConfigTree->getPropInt("@connectionTimeout", connectTimeout); + readTimeout = pHostConfigTree->getPropInt("@readTimeout", readTimeout); + writeTimeout = pHostConfigTree->getPropInt("@writeTimeout", writeTimeout); + return true; } @@ -162,26 +157,167 @@ bool ElasticMetricSink::getIndexConfig(const IPropertyTree *pSettingsTree) return false; } - // Initialize standard suffixes - if (!pIndexConfigTree->getProp("@countMetricSuffix", countMetricSuffix)) - countMetricSuffix.append("count"); + intializeElasticClient(); + + if (!validateIndex()) + return false; + + return getDynamicMappingSuffixesFromIndex(pIndexConfigTree); +} - if (!pSettingsTree->getProp("@gaugeMetricSuffix", gaugeMetricSuffix)) - gaugeMetricSuffix.append("gauge"); - if (!pSettingsTree->getProp("@histogramMetricSuffix", histogramMetricSuffix)) - histogramMetricSuffix.append("histogram"); +bool ElasticMetricSink::getDynamicMappingSuffixesFromIndex(const IPropertyTree *pIndexConfigTree) +{ + StringBuffer countSuffixMappingName; + if (!pIndexConfigTree->getProp("@countSuffixMappingName", countSuffixMappingName)) + countSuffixMappingName.append("hpcc_metrics_count_suffix"); - return true; + StringBuffer gaugeSuffixMappingName; + if (!pIndexConfigTree->getProp("@gaugeSuffixMappingName", gaugeSuffixMappingName)) + gaugeSuffixMappingName.append("hpcc_metrics_gauge_suffix"); + + StringBuffer histogramSuffixMappingName; + if (!pIndexConfigTree->getProp("@histogramSuffixMappingName", histogramSuffixMappingName)) + histogramSuffixMappingName.append("hpcc_metrics_histogram_suffix"); + + std::string endpoint; + endpoint.append("/").append(indexName.str()).append("/_mapping"); + httplib::Result res = pClient->Get(endpoint.c_str(), elasticHeaders); + + if (res == nullptr) + { + httplib::Error err = res.error(); + WARNLOG("ElasticMetricSink: Unable to connect to ElasticSearch host '%s', httplib Error = %d", elasticHostUrl.str(), err); + return false; + } + + if (res->status != 200) + { + WARNLOG("ElasticMetricSink: Error response status = %d, unable to retrieve mapping for index '%s'", res->status, indexName.str()); + return false; + } + + nlohmann::json data = nlohmann::json::parse(res->body); + + auto indexConfig = data[indexName.str()]; + if (indexConfig.is_null()) + { + WARNLOG("ElasticMetricSink: Unable to load configuration for Index '%s'", indexName.str()); + return false; + } + + auto mappings = indexConfig["mappings"]; + if (mappings.is_null()) + { + WARNLOG("ElasticMetricSink: Required 'mappings' section does not exist in Index '%s'", indexName.str()); + return false; + } + + auto dynamicTemplates = mappings["dynamic_templates"]; + if (dynamicTemplates.is_null()) + { + WARNLOG("ElasticMetricSink: Required 'dynamic_templates' section does not exist in Index '%s'", indexName.str()); + return false; + } + + // validate type is as expected from the data + if (dynamicTemplates.is_array()) + { + for (auto & dynamicTemplate : dynamicTemplates.items()) + { + auto mappingObject = dynamicTemplate.value().get(); + + for (const auto& it: mappingObject) + { + auto const &mappingName = it.first; + auto kvPairs = it.second; + auto const &matchValue = kvPairs["match"]; + if (mappingName == countSuffixMappingName.str()) + { + if (!convertPatternToSuffix(matchValue.get().c_str(), countMetricSuffix)) + { + WARNLOG("ElasticMetricSink: Invalid count suffix pattern"); + return false; + } + } + else if (mappingName == histogramSuffixMappingName.str()) + { + if (!convertPatternToSuffix(matchValue.get().c_str(), histogramMetricSuffix)) + { + WARNLOG("ElasticMetricSink: Invalid histogram suffix pattern"); + return false; + } + } + else if (mappingName == gaugeSuffixMappingName.str()) + { + if (!convertPatternToSuffix(matchValue.get().c_str(), gaugeMetricSuffix)) + { + WARNLOG("ElasticMetricSink: Invalid gauge suffix pattern"); + return false; + } + } + } + } + + // If no gauge suffix pattern configured, use the count suffix pattern + if (gaugeMetricSuffix.isEmpty()) + gaugeMetricSuffix.append(countMetricSuffix); + + // All three must be configured + return !countMetricSuffix.isEmpty() && !gaugeMetricSuffix.isEmpty() && !histogramMetricSuffix.isEmpty(); + } + return false; } -bool ElasticMetricSink::prepareToStartCollecting() +bool ElasticMetricSink::convertPatternToSuffix(const char *pattern, StringBuffer &suffix) { + if (pattern && (strlen(pattern) > 2) && pattern[0] == '*') + { + suffix.append(pattern + 1); + return true; + } return false; } +void ElasticMetricSink::intializeElasticClient() +{ + elasticHeaders = { + {"Content-Type", "application/json"}, + {"Accept", "application/json"} + }; + + // Add authorization header if needed + if (streq(authenticationType, "basic")) + { + StringBuffer token; + StringBuffer usernamePassword; + usernamePassword.append(username).append(":").append(password); + JBASE64_Encode(usernamePassword.str(), usernamePassword.length(), token, false); + StringBuffer basicAuth; + basicAuth.append("Basic ").append(token); + elasticHeaders.insert({"Authorization", basicAuth.str()}); + } + + pClient = std::make_shared(elasticHostUrl.str()); + + // Add cert path if needed + if (!certificateFilePath.isEmpty()) + pClient->set_ca_cert_path(certificateFilePath.str()); + + pClient->set_connection_timeout(connectTimeout); + pClient->set_read_timeout(readTimeout); + pClient->set_write_timeout(writeTimeout); +} + + +bool ElasticMetricSink::prepareToStartCollecting() +{ + return configurationValid; +} + + void ElasticMetricSink::doCollection() { @@ -193,3 +329,24 @@ void ElasticMetricSink::collectingHasStopped() } + +bool ElasticMetricSink::validateIndex() +{ + std::string endpoint; + endpoint.append("/").append(indexName.str()); + auto res = pClient->Get(endpoint.c_str(), elasticHeaders); + + if (res == nullptr) + { + httplib::Error err = res.error(); + WARNLOG("ElasticMetricSink: Unable to connect to ElasticSearch host '%s, httplib Error = %d", elasticHostUrl.str(), err); + return false; + } + + else if (res->status != 200) + { + WARNLOG("ElasticMetricSink: Error response status = %d accessing Index '%s'", res->status, indexName.str()); + return false; + } + return true; +} diff --git a/system/metrics/sinks/elastic/elasticSink.hpp b/system/metrics/sinks/elastic/elasticSink.hpp index 8045416a1cb..3b483ad2fa9 100644 --- a/system/metrics/sinks/elastic/elasticSink.hpp +++ b/system/metrics/sinks/elastic/elasticSink.hpp @@ -18,6 +18,23 @@ #include "jptree.hpp" #include "jstring.hpp" +//including cpp-httplib single header file REST client +// doesn't work with format-nonliteral as an error +// +#if defined(__clang__) || defined(__GNUC__) +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wformat-nonliteral" +#pragma GCC diagnostic ignored "-Warray-bounds" +#endif + +#undef INVALID_SOCKET +#define CPPHTTPLIB_OPENSSL_SUPPORT +#include "httplib.h" + +#if defined(__clang__) || defined(__GNUC__) +#pragma GCC diagnostic pop +#endif + #ifdef ELASTICSINK_EXPORTS #define ELASTICSINK_API DECL_EXPORT #else @@ -36,6 +53,10 @@ class ELASTICSINK_API ElasticMetricSink : public hpccMetrics::PeriodicMetricSink virtual void doCollection() override; bool getHostConfig(const IPropertyTree *pSettingsTree); bool getIndexConfig(const IPropertyTree *pSettingsTree); + bool getDynamicMappingSuffixesFromIndex(const IPropertyTree *pIndexConfigTree); + static bool convertPatternToSuffix(const char *pattern, StringBuffer &suffix); + bool validateIndex(); + void intializeElasticClient(); protected: StringBuffer indexName; @@ -48,5 +69,10 @@ class ELASTICSINK_API ElasticMetricSink : public hpccMetrics::PeriodicMetricSink StringBuffer countMetricSuffix; StringBuffer gaugeMetricSuffix; StringBuffer histogramMetricSuffix; + int connectTimeout = 5; + int readTimeout = 5; + int writeTimeout = 5; bool configurationValid = false; + std::shared_ptr pClient; + httplib::Headers elasticHeaders; };