-
Notifications
You must be signed in to change notification settings - Fork 54
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add "bind_all" feature to daemons->endpoints that instructs an ldmsd …
…to listen on all available interfaces for the given endpoint bind_all is a boolean that accepts True/False as arguments Introduce man page detailing YAML configuration syntax as well as documentation for the ldmsd -y option and ldmsd_yaml_parser python executable.
- Loading branch information
1 parent
f598e45
commit 88c887d
Showing
3 changed files
with
361 additions
and
13 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,347 @@ | ||
| .\" Manpage for ldmsd_yaml_parser | ||
| .\" Contact ovis-help@ca.sandia to correct errors or typos. | ||
| .TH man 8 "20 Nov 2024" "ovis-4.4.5" "ldmsd_yaml_parser man page" | ||
| .TH YAML Configuration Quick Guide | ||
|
|
||
| .SH NAME | ||
| ldmsd_yaml_parser \- a python program to parse a YAML configuration file into a v4 LDMS configuration. | ||
|
|
||
| .SH SYNOPSIS | ||
| .B ldmsd [OPTIONS] | ||
|
|
||
| .B ldmsd_yaml_parser | ||
| [OPTIONS] | ||
|
|
||
| .SH DESCRIPTION | ||
| A single ldmsd can be configured using a YAML configuration file. This can be done by running ldmsd directly, or generating a configuration parser with the ldmsd_yaml_parser | ||
|
|
||
| .SH LDMSD YAML OPTIONS | ||
| .TP | ||
| .BI -y, " CONFIG_PATH" | ||
| The path to the YAML configuration file. | ||
| .TP | ||
| .BI -n, " NAME" | ||
| The name of the LDMS daemon. | ||
|
|
||
| .SH LDMSD_YAML_PARSER COMMAND SYNTAX | ||
| .TP | ||
| .BI --ldms_config, " CONFIG_PATH" | ||
| The path to the YAML configuration file. | ||
| .TP | ||
| .BI --daemon_name, " NAME" | ||
| The name of the LDMS daemon to configure. | ||
|
|
||
| .SH YAML CONFIGURATION SYNTAX | ||
|
|
||
| The LDMS YAML configuration syntax allows defining a single LMDSDs configuration up to an entire LDMS cluster configuration in a single file. | ||
| .br | ||
| LDMS YAML cluster configuration is organized into “groups” of dictionaries with relevant configurations for each group. The top level dictionaries are “daemons”, “aggregators”, “samplers”, “plugins”, and “stores”. | ||
| .br | ||
| Time intervals are defined as a unit string, or an integer in microseconds. | ||
| .br | ||
| A unit string is one of the followings: | ||
| us -- microseconds | ||
| ms -- milliseconds | ||
| s -- seconds | ||
| m -- minutes | ||
| h -- hours | ||
| d -- days | ||
| .br | ||
| Several of the keys in this configuration syntax utilize the hostlist format. | ||
| .br | ||
| Please see the hostlist documentation for more information regarding this here: | ||
| .br | ||
| https://readthedocs.org/projects/py-hostlist/downloads/pdf/latest/ | ||
|
|
||
| .SH daemons | ||
| List of dictionaries describing LDMS daemons that are part of the cluster and their endpoints that share transport configurations. The primary keys are "names", "hosts", "endpoints", and "environments". Any keys provided that do not match this string values are assumed to be either CLI commands, or overarching default configuration commands for a daemon. | ||
|
|
||
| .SS names | ||
| Regex of a group of LDMS daemon attributes and endpoints that share transport configuration. These strings are referenced in the samplers and aggregators sections. Hostlist format. | ||
|
|
||
| .SS hosts | ||
| Hostnames on which the LDMS daemons will operate. Must be specified here. May be overridden by a "hosts" definition in the endpoints section. Hostlist format. | ||
|
|
||
| .SS endpoints | ||
| List of dictionaries of endpoint configurations for these daemons. | ||
| .RS | ||
| .TP | ||
| .BR names | ||
| .br | ||
| Unique names by which to identify the transport endpoints. Hostlist format. e.g. "sampler-[1-4]" | ||
| .TP | ||
| .BR [hosts] | ||
| .br | ||
| Hosts for an endpoint may be optionally specified if a certain interface is desired other than the stated hostname in the top level daemons section. Hostlist format. e.g. "node-[1-4]" for 4 LDMSDs | ||
| .TP | ||
| .BR [bind_all] | ||
| .br | ||
| The bind_all boolean may be specified to configure the LDMS daemon to listen on all available interfaces when using this endpoint. The prdcr_add command will use either the top level host definition, or, if defined, the host specified in the endpoints. bind_all accepts <True/False> | ||
| .TP | ||
| .BR ports | ||
| .br | ||
| Daemon endpoint ports on which to communicate. Hostlist or integer format. e.g. "[10001-10004]" | ||
| .br | ||
| If there are two endpoints, and two ports, one will be assigned to each endpoint. | ||
| .TP | ||
| .BR xprt | ||
| .br | ||
| The communication transport for the endpoint. <sock/rdma/ugni> are supported. | ||
| .TP | ||
| .BR auth | ||
| .br | ||
| Dictionary of a authentication domains plugin configuration. | ||
| .RS | ||
| .TP | ||
| .BR name | ||
| .br | ||
| Unique authentication domain name for this authentication configuration. | ||
| .TP | ||
| .BR plugin | ||
| .br | ||
| Name of the authentication domain plugin <ovis/munge> | ||
| .TP | ||
| .BR conf | ||
| .br | ||
| Dictionary of plugin specific configuration options for this authentication domain. | ||
|
|
||
| .SH aggregators | ||
| List of dictionaries defining aggregator configurations, their “peers” i.e. “producers”, that they will be aggregating data from, and the endpoints and daemons on which to communicate. | ||
| .br | ||
| The daemons reference daemon configuration definitions defined in the "daemons" dictionary. | ||
| .br | ||
| The stores reference storage policy names defined in the "stores" top level dictionary. | ||
| .br | ||
| The "plugins" key reference plugin instance names defined in the "plugins" top level dictionary. | ||
| .br | ||
| The primary keys are "names", "hosts", "endpoints", and "environments" | ||
| .br | ||
| Any keys provided that do not match one of these string values are assumed to be either CLI commands, or overarching default configuration commands for a daemon. | ||
|
|
||
| .SS names | ||
| String regex in hostlist format of a group of LDMS daemon attributes and endpoints that share transport configuration in hostlist format. These strings are referenced in the sampler and aggregator configurations. | ||
|
|
||
| .SS hosts | ||
| String regex in hostlist format of hostnames on which the LDMS daemon will operate. Must expand to an equal length as the daemon names, or be evenly divisble. e.g. 2 hostnames for 4 daemons. | ||
|
|
||
| .SS [subscribe] | ||
| List of dictionaries of streams to subscribe producers to. | ||
| .TP | ||
| .BR stream | ||
| .br | ||
| The name of the stream. | ||
| .TP | ||
| .BR regex | ||
| .br | ||
| Regular expression matching producers to subscribe to the stream. | ||
|
|
||
| .SS peers | ||
| List of dictionaries containing producer configurations. This is an alternative method to configuring producers than using prdcr_listen | ||
| .TP | ||
| .BR daemons | ||
| .br | ||
| String of daemon names in hostlist format that references daemon names defined in the top level daemons section. | ||
| .TP | ||
| .BR endpoints | ||
| .br | ||
| String of endpoints in hostlist format that references endpoints defined in the top level daemons section. | ||
| .TP | ||
| .BR reconnect | ||
| .br | ||
| Interval by which the aggregator will attempt to reconnect to a disconnected producer. Unit string format. | ||
| .TP | ||
| .BR type | ||
| .br | ||
| Producer type. Either active or passive. passive is being deprecated. | ||
| .TP | ||
| .BR updaters | ||
| .br | ||
| List of dictionaries of updater policy configurations. | ||
| .RS | ||
| .TP | ||
| .BR mode | ||
| .br | ||
| Updater mode. Accepted strings are <pull|push|onchange|auto> | ||
| "onchange" means the Updater will get an update whenever the set source ends | ||
| a transaction or pushes the update. "push" means the Updater will receive an | ||
| update only when the set source pushes the update. | ||
| 'auto' means the updater will schedule set updates according to the update hint. | ||
| The sets with no hints will not be updated. | ||
| "pull" means the updater will schedule the set updates according to the given interval | ||
| .TP | ||
| .BR interval | ||
| .br | ||
| The update/collect interval at which to update the producer. Unit string format. | ||
| .TP | ||
| .BR [offset] | ||
| .br | ||
| Offset for synchronized aggregation. Optional. Unit string format. | ||
| .TP | ||
| .BR sets | ||
| .br | ||
| List of dictionaries containing regular expressions that match either a schema instance name or a metric set instance name. | ||
| .RS | ||
| .TP | ||
| .BR regex | ||
| .br | ||
| Regular expression to either match instance names or schemas to apply this updater policy too. | ||
| .TP | ||
| .BR field | ||
| .br | ||
| Field to use when matching the regular expression. <schema|inst>. schema matches a schema instance name, and inst matches a metric set instance name. | ||
|
|
||
| .SS prdcr_listen | ||
| An optional alternative configuration for how your aggregators will add producers that is used in conjunction with the top level samplers "advertise" key. When utilizing producer listen, the aggregator will listen until a connection is established by a sampler. When using this configuration, the aggregators configuration information is provided in the samplers section under the key "advertisers". | ||
| .RS | ||
| .TP | ||
| .BR name | ||
| .br | ||
| String name for the producer listener - does not need to be unique across aggregators. | ||
| .TP | ||
| .BR updaters | ||
| .br | ||
| List of dictionaries containing updater policies for the producers that ultimately connect to the producer listener. | ||
| .RS | ||
| .TP | ||
| .BR mode | ||
| .br | ||
| Updater mode. Accepted strings are <pull|push|onchange|auto> | ||
| "onchange" means the Updater will get an update whenever the set source ends | ||
| a transaction or pushes the update. "push" means the Updater will receive an | ||
| update only when the set source pushes the update. | ||
| 'auto' means the updater will schedule set updates according to the update hint. | ||
| The sets with no hints will not be updated. | ||
| "pull" means the updater will schedule the set updates according to the given interval | ||
| and offset values. | ||
| .TP | ||
| .BR interval | ||
| .br | ||
| The update/collect interval at which to update the producer. Unit string format. | ||
| .TP | ||
| .BR [offset] | ||
| .br | ||
| Offset for synchronized aggregation. Optional. Unit string format. | ||
| .TP | ||
| .BR sets | ||
| .br | ||
| List of dictionaries containing regular expressions that match either a schema instance name or a metric set instance name. | ||
| .RS | ||
| .TP | ||
| .BR regex | ||
| .br | ||
| Regular expression to either match instance names or schemas to apply this updater policy too. | ||
| .TP | ||
| .BR field | ||
| .br | ||
| Field to use when matching the regular expression. <schema|inst>. schema matches a schema instance name, and inst matches a metric set instance name. | ||
|
|
||
| .SH samplers | ||
| List of dictionaries defining sampler configurations and the LDMS daemons to apply them to. The daemons reference daemons defined in the top level "daemons" dictionary. Plugins reference instance names of plugins defined in the "plugins" top level dictionary. | ||
| .TP | ||
| .BR daemons | ||
| .br | ||
| String of daemon names in hostlist format that references daemon names defined in the top level daemons section. | ||
| .TP | ||
| .BR plugins | ||
| .br | ||
| List of strings of plugin instance names to load that reference plugin instance names defined in the top level plugins section. String format. | ||
| .TP | ||
| .BR [advertise] | ||
| .br | ||
| Alternative configuration to the aggregators "peers" where the sampler initiates a connection to the aggregator. The producer listener for an advertiser is defined in the top level aggregators section. | ||
| .RS | ||
| .TP | ||
| .BR names | ||
| .br | ||
| String of daemon names in hostlist format to advertise the samplers as. | ||
| .TP | ||
| .BR hosts | ||
| .br | ||
| String of daemon hosts in hostlist format, that references daemon names defined in the top level "daemons" section, for the samplers to advertise to | ||
| .TP | ||
| .BR port | ||
| .br | ||
| String of port(s) in hostlist format of the aggregator daemons that the sampler daemons will attempt to connect to. | ||
| .TP | ||
| .BR reconnect | ||
| .br | ||
| The interval at which the sampler will attempt to reconnect to a disconnected advertiser. Float followed by a unit string. | ||
| .TP | ||
| .BR auth | ||
| .br | ||
| Dictionary of a authentication domains plugin configuration. | ||
| .RS | ||
| .TP | ||
| .BR name | ||
| .br | ||
| Unique authentication domain name for this authentication configuration. | ||
| .TP | ||
| .BR plugin | ||
| .br | ||
| Name of the authentication domain plugin <ovis/munge> | ||
| .TP | ||
| .BR [conf] | ||
| .br | ||
| Optional dictionary of plugin specific configuration options for this authentication domain. | ||
| .RS | ||
| .BR ["path" : "/opt/ovis/secret.conf"] | ||
|
|
||
| .SH stores | ||
| Dictionary of storage policies and their configuration information with each key being a storage policy name. | ||
| .TP | ||
| .BR container | ||
| .br | ||
| File path of the database container. | ||
| .TP | ||
| .BR schema | ||
| .br | ||
| Name of the metric set schema. | ||
| .TP | ||
| .BR plugin | ||
| .br | ||
| Name of a storage plugin that matches a key of a plugin defined in the top level plugins section. | ||
| .TP | ||
| .BR [flush] | ||
| .br | ||
| Optional interval of time that directs flushing of the store to the database. | ||
|
|
||
| .SH plugins | ||
| Dictionary of plugins and their configuration information with each key being a plugin instance name. | ||
| .RS | ||
| .TP | ||
| .BR name | ||
| .br | ||
| The name of a plugin to load. e.g. meminfo | ||
| .TP | ||
| .BR interval | ||
| .br | ||
| The interval at which to sample data. | ||
| .TP | ||
| .BR [offset] | ||
| .br | ||
| Offset (shift) from the sample mark in the same format as intervals. | ||
| Offset can be positive or negative with magnitude up to 1/2 | ||
| the sample interval. The default offset is 0. Collection is always synchronous. | ||
| .TP | ||
| .BR config | ||
| .br | ||
| A list of dictionaries containing plugin configuration options. Each dictionary in the list is a "config" command call, and in this fashion, the YAML configuration mimics running multiple "config" statements in a conventional v4 configuration file. | ||
| .br | ||
| Any plugin-specific configuration options not listed below will be included in the configuration. | ||
| .RS | ||
| .TP | ||
| .BR schema | ||
| .br | ||
| Name of the metric set to use. | ||
| .TP | ||
| .BR [perm] | ||
| .br | ||
| Access permissions for the metric set within the container. e.g. "0440" | ||
| .TP | ||
| .BR [component_id] | ||
| .br | ||
| Unique ID of the component being monitored. If configuring an entire cluster, it's advised to set this to reference an environment variable on the system. | ||
| .TP | ||
| .BR [producer] | ||
| .br | ||
| Producer name must be unique in an aggregator. It is independent of any attributes specified for the metric sets or hosts. A producer name will be generated by the yaml using the hostname of the sampler and the plugin instance name if one is not specified. <hostname>/<plugin_name> |
Oops, something went wrong.