Skip to content

Commit

Permalink
HPCC-32909 Retrieve index settings from ElasticSearch server
Browse files Browse the repository at this point in the history 
Added code to query configured index on ElasticSearch server for dynamic
mapping template match values for hpcc metric type suffixes.

Signed-Off-By: Kenneth Rowland kenneth.rowland@lexisnexisrisk.com
  • Loading branch information
@kenrowland
kenrowland committed Jan 13, 2025
1 parent eb69d88 commit 20d6c92
Show file tree
Hide file tree
Showing 4 changed files with 262 additions and 56 deletions.
73 changes: 48 additions & 25 deletions helm/examples/metrics/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -187,37 +187,41 @@ sink will report metrics from all components to the same index. Therefore, the
index must be configured to receive metrics from all components.
#### Index Configuration
The index must be created in ElasticSearch before metrics can be reported to it. The name is passed
to the sink as a configuration setting. The index must be created with the following settings:
The sink requires the index to be created and configured fully in ElasticSearch in order for the sink to report
metrics. The sink reads configuration data from the index in order to properly report metrics. Future versions
of the sink may include the capability to create the index if it does not exist. The name of the index is passed
to the sink as a configuration setting.
The following must be configured in the index in order for the sink to properly report metrics:
##### Dynamic Mapping
The index must be created with dynamic mapping enabled. Dynamic mapping allows framework metric
data types to be stored in the index in their native types. Without dynamic mapping, the ElasticSearch
default mapping does not properly map value to unsigned 64-bit integers.
The index must be configured with dynamic mapping enabled. Dynamic mapping allows storing of framework metric
values using native types. Without dynamic mapping, the ElasticSearch default mapping does not properly map
values to unsigned 64-bit integers.
To create an index with dynamic mapping, use the following object when creating the index:
To create an index with dynamic mapping, use the following object, or other means, when creating the index:
```code json
{
"mappings": {
"dynamic_templates": [
{
"hpcc_metric_count_to_unsigned_long": {
"hpcc_metrics_count_suffix": {
"match": "*_count",
"mapping": {
"type": "unsigned_long"
}
}
},
{
"hpcc_metric_gauge_to_unsigned_long": {
"hpcc_metrics_gauge_suffix": {
"match": "*_gauge",
"mapping": {
"type": "unsigned_long"
}
}
},
{
"hpcc_metric_histogram_to_histogram": {
"hpcc_metrics_histogram_suffix": {
"match": "*_histogram",
"mapping": {
"type": "histogram"
Expand All @@ -229,11 +233,11 @@ To create an index with dynamic mapping, use the following object when creating
}
```

Note that there may be other means for adding the required dynamic mapping to the index.

The _match_ values above are representative of typical values. The actual values may
vary depending on the cluster configuration and shared use of the index across multiple clusters.
In all cases, the configuration of the sink must match that of the dynamic mappings in the index.
The object names beginning with "hpcc_metric_" represent the mappings for the three affected hpcc metrics types.
The _match_ value for each defines the expected suffix for each HPCC metric value, based on type, when metrics
are reported and indexed. The object names above are the default values used by the sink. The index can be
configured with different object names (for environment flexibility), but the object names must be provided to the
sink as additional configuration settings (defined below).

#### Enabling ElasticSearch Metrics Sink for Kubernetes
To enable reporting of metrics to ElasticSearch, add the metric configuration settings to
Expand All @@ -255,13 +259,16 @@ Make a copy and modify as needed for your installation.
The ElasticSearch sink defines the following settings

**Host**
The host settings define the ElasticSearch server to which metrics are reported. The settings are:
The host settings define the server hosting ElasticSearch to which metrics are reported. The settings are:

* domain - The domain or IP address of the ElasticSearch server. (required)
* protocol - The protocol used to connect to the ElasticSearch server. (default: https)
* port - The port number of the ElasticSearch server. (default: 9200)
* certificateFilePath - Path to the file containing the certificate used to connect to the ElasticSearch server. (optional)

* certificateFilePath - Path to the file containing the certificate used to connect to the ElasticSearch server.
(optional)
* connectTimeout - The time in seconds to wait for a connection to be established to the (default: 5)
* readTimeout - The time in seconds to wait for a response from the server for a read operation. (default: 5)
* writeTimeout - The time in seconds to wait for a response from the server for a write operation. (default: 5)

**Authentication**

Expand All @@ -276,20 +283,35 @@ the ElasticSearch server. (optional, valid for Kubernetes only)
* credentialsVaultId - The vault ID containing the credentials used to authenticate to the
ElasticSearch server. (optional, valid for Vault only)

For **basic** authentication, the following settings are required, regardless if the credentials are stored
as a secret or in the environment.xml file.
For **basic** authentication, the following settings are required. If a secret or vault is defined, these
values are store there and are not required in the configuration file. Otherwise, they are required in the
configuration file (environment.xml).
* username - The username used to authenticate to the ElasticSearch server.
* password - The password used to authenticate to the ElasticSearch server. When stored in the
environment.xml file, it shall be encrypted using standard environment.xml encryption.

**Index**

The index settings define the index where metrics are indexed. The settings are:
The index must be created and configred in ElasticSearch before the sink will load and report
metrics. The following settings describe what must be configured in ElasticSearch.

* name - The name of the index to which metrics are reported. (required)
* countMetricSuffix - The suffix used to identify count metrics. (default: count)
* gaugeMetricSuffix - The suffix used to identify gauge metrics. (default: gauge)
* histogramMetricSuffix - The suffix used to identify histogram metrics. (default: histogram)
* countSuffixMappingName - See below. (default: hpcc_metrics_count_suffix)
* gaugeSuffixMappingName - See below. (default: hpcc_metrics_gauge_suffix)
* histogramSuffixMappingName - See below. (default: hpcc_metrics_histogram_suffix)

The _*SuffixMappingName_ values are object names in the index _dynamic_templates_ object of the
index's _mappings_ object. These MUST be configured in the index. The mapping name objects contain
a _match_ member whose value is used as the suffix for the related HPCC metric type. The format
is expected to be "*<string>" where "*" is part of the defined match pattern syntax defined by
ElasticSearch and "<string>" is the suffix appended to each HPCC metric based on type. For example,
If a _match_ value is defined as "*_count", then the sink would add "_count" to the end of each
metric when indexing measurements during a report. This ensures that each HPCC metric is stored in
the index with the correct type. Please refer to the ElasticSearch documentation for more information
on dynamically mapping types when indexing documents.

For convenience, if the index dynamic templates configuration settings do not include a gauge
suffix setting, the value for the count suffix setting is used.

Standard periodic metric sink settings are also available.

Expand All @@ -306,10 +328,11 @@ Add the following to the environment.xml configuration file (note that some valu
<metrics name="mymetricsconfig">
<sinks name="myelasticsink" type="elastic">
<settings period="30" ignoreZeroMetrics="1">
<host domain="<domainname>" port="<port>" protocol="http|https">
<host domain="<domainname>" port="<port>" protocol="http|https"
certificateFilePath="<path to cert file>">
<authentication type="basic" username="<username>" password="<password>"/>
</host>
<index name="<index>"/>
<index name="<index>" countSuffixMappingName="<name>" histogramSuffixMappingName="<name>" gaugeSuffixMappingName="<name>/>
<settings/>
</sinks>
</metrics>
Expand Down
12 changes: 6 additions & 6 deletions helm/examples/metrics/elasticsearch_metrics.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,6 @@
# Settings:
# type - sink type (must be elastic for ElasticSearch support)
# name - name for the sink instance
# settings.countMetricSuffix - suffix for count metrics (default: count)
# settings.gaugeMetricSuffix - suffix for gauge metrics (default: gauge)
# settings.histogramMetricSuffix - suffix for histogram metrics (default: histogram)
# settings.host - ElasticSearch host settings
# settings.host.protocol - protocol to use, http or https (default)
# settings.host.domain - host domain
Expand All @@ -19,6 +16,9 @@
# settings.host.authentication.credentialsVaultId - optional vault id for secret containing username password for basic authentication
# settings.index - ElasticSearch index settings
# settings.index.name - index name
# settings.index.countSuffixMappingName - dynamic template mapping object name for count metrics suffixes (default: hpcc_metrics_count_suffix)
# settings.index.gaugeSuffixMappingName - dynamic template mapping object name for gauge metrics suffixes (default: hpcc_metrics_gauge_suffix)
# settings.index.histogramSuffixMappingName - dynamic template mapping object name for histogram metrics suffixes (default: hpcc_metrics_histogram_suffix)
#
# If not overridden, the following suffixes are used by default:
# countMetricSuffix: count
Expand All @@ -31,9 +31,6 @@ global:
- type: elastic
name: myelasticsink
settings:
countMetricSuffix: count
gaugeMetricSuffix: gauge
histogramMetricSuffix: histogram
host:
protocol: https
domain: domain
Expand All @@ -47,3 +44,6 @@ global:
credentialsVaultId: vaultId
index:
name: hpccmetrics
countSuffixMappingName: name
gaugeSuffixMappingName: name
histogramSuffixMappingName: name
Loading

0 comments on commit 20d6c92

Please sign in to comment.