Skip to content

Commit

Permalink
Add ES params
Browse files Browse the repository at this point in the history
  • Loading branch information
@ahma @tarokkk
ahma authored and tarokkk committed Sep 27, 2019
1 parent ffac24f commit f02dc07
Show file tree
Hide file tree
Showing 8 changed files with 367 additions and 11 deletions.
6 changes: 3 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -106,9 +106,9 @@ For complete list of supported plugins pleas checkl the [plugins index](/docs/pl
| Name | Type | Description | Status | Version |
|---------------------------------------------------------|:------:|:-------------------------------------------------------------------------:|---------|-------------------------------------------------------------------------------------------|
| [Alibaba](./docs/plugins/outputs/oss.md) | Output | Store logs the Alibaba Cloud Object Storage Service | GA | [0.0.1](https://github.com/aliyun/fluent-plugin-oss) |
| [Amazon S3](./docs/plugins/outputs/s3.md) | Output | Store logs in Amazon S3 | GA | [1.1.10](https://github.com/fluent/fluent-plugin-s3/releases/tag/v1.1.10) |
| [Azure](./docs/plugins/outputs/azurestore.md) | Output | Store logs in Azure Storega | GA | [0.1.1](https://github.com/htgc/fluent-plugin-azurestorage/releases/tag/v0.1.0) |
| [Google Storage](./docs/plugins/outputs/gcs.md) | Output | Store logs in Google Cloud Storage | GA | [0.4.0.beta1](https://github.com/banzaicloud/fluent-plugin-gcs) |
| [Amazon S3](./docs/plugins/outputs/s3.md) | Output | Store logs in Amazon S3 | GA | [1.1.11](https://github.com/fluent/fluent-plugin-s3/releases/tag/v1.1.11) |
| [Azure](./docs/plugins/outputs/azurestore.md) | Output | Store logs in Azure Storega | GA | [0.1.0](https://github.com/htgc/fluent-plugin-azurestorage/releases/tag/v0.1.0) |
| [Google Storage](./docs/plugins/outputs/gcs.md) | Output | Store logs in Google Cloud Storage | GA | [0.4.0](https://github.com/banzaicloud/fluent-plugin-gcs) |
| [Grafana Loki](./docs/plugins/outputs/loki.md) | Output | Transfer logs to Loki | Testing | [0.2](https://github.com/banzaicloud/fluent-plugin-kubernetes-loki/releases/tag/v0.2) |
| [ElasticSearch](./docs/plugins/outputs/elasticsearch.md) | Output | Send your logs to Elasticsearch | GA | [3.5.5](https://github.com/uken/fluent-plugin-elasticsearch/releases/tag/v3.5.5) |
| [Sumologic](./docs/plugins/outputs/sumologic.md) | Output | Send your logs to Sumologic | GA | [1.5.0](https://github.com/SumoLogic/fluentd-output-sumologic/releases/tag/1.5.0) |
Expand Down
78 changes: 78 additions & 0 deletions charts/logging-operator/templates/logging.banzaicloud.io_clusteroutputs.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -410,28 +410,58 @@ spec:
required:
- timekey
type: object
bulk_message_request_threshold:
description: 'Configure bulk_message request splitting threshold
size. Default value is 20MB. (20 * 1024 * 1024) If you specify
this size as negative number, bulk_message request splitting feature
will be disabled. (default: 20MB)'
type: string
content_type:
description: 'With content_type application/x-ndjson, elasticsearch
plugin adds application/x-ndjson as Content-Type in payload. (default:
application/json)'
type: string
custom_headers:
description: 'This parameter adds additional headers to request.
Example: {"token":"secret"} (default: {})'
type: string
customize_template:
description: Specify the string and its value to be replaced in
form of hash. Can contain multiple key value pair that would be
replaced in the specified template_file. This setting only creates
template and to add rollover index please check the rollover_index
configuration.
type: string
default_elasticsearch_version:
description: This parameter changes that ES plugin assumes default
Elasticsearch version. The default value is 5.
type: string
deflector_alias:
description: Specify the deflector alias which would be assigned
to the rollover index created. This is useful in case of using
the Elasticsearch rollover API
type: string
exception_backup:
description: 'Indicates whether to backup chunk when ignore exception
occurs. (default: true)'
type: boolean
fail_on_putting_template_retry_exceed:
description: 'Indicates whether to fail when max_retry_putting_template
is exceeded. If you have multiple output plugin, you could use
this property to do not fail on fluentd statup.(default: true)'
type: boolean
flatten_hashes:
description: 'Elasticsearch will complain if you send object and
concrete values to the same field. For example, you might have
logs that look this, from different places: {"people" => 100}
{"people" => {"some" => "thing"}} The second log line will be
rejected by the Elasticsearch parser because objects and concrete
values can''t live in the same field. To combat this, you can
enable hash flattening.'
type: boolean
flatten_hashes_separator:
description: Flatten separator
type: string
host:
description: You can specify Elasticsearch host by this parameter.
(default:localhost)
Expand All @@ -448,6 +478,16 @@ spec:
id_key:
description: https://github.com/uken/fluent-plugin-elasticsearch#id_key
type: string
ignore_exceptions:
description: A list of exception that will be ignored - when the
exception occurs the chunk will be discarded and the buffer retry
mechanism won't be called. It is possible also to specify classes
at higher level in the hierarchy. For example `ignore_exceptions
["Elasticsearch::Transport::Transport::ServerError"]` will match
all subclasses of ServerError - Elasticsearch::Transport::Transport::Errors::BadRequest,
Elasticsearch::Transport::Transport::Errors::ServiceUnavailable,
etc.
type: string
include_index_in_url:
description: With this option set to true, Fluentd manifests the
index name in the request URL (rather than in the request body).
Expand All @@ -471,6 +511,14 @@ spec:
description: Specify the index prefix for the rollover index to
be created.
type: string
log_es_400_reason:
description: 'By default, the error logger won''t record the reason
for a 400 error from the Elasticsearch API unless you set log_level
to debug. However, this results in a lot of log spam, which isn''t
desirable if all you want is the 400 error reasons. You can set
this true to capture the 400 error reasons without all the other
debug logs. (default: false)'
type: boolean
logstash_dateformat:
description: 'Set the Logstash date format.(default: %Y.%m.%d)'
type: string
Expand Down Expand Up @@ -589,6 +637,12 @@ spec:
description: If you want to configure SSL/TLS version, you can specify
ssl_version parameter. [SSLv23, TLSv1, TLSv1_1, TLSv1_2]
type: string
suppress_doc_wrap:
description: 'By default, record body is wrapped by ''doc''. This
behavior can not handle update script requests. You can set this
to suppress doc wrapping and allow record body to be untouched.
(default: false)'
type: boolean
tag_key:
description: 'This will add the Fluentd tag in the JSON record.(default:
tag)'
Expand Down Expand Up @@ -644,6 +698,16 @@ spec:
degree of sub-second time precision to preserve from the time
portion of the routed event.
type: string
unrecoverable_error_types:
description: Default unrecoverable_error_types parameter is set
up strictly. Because es_rejected_execution_exception is caused
by exceeding Elasticsearch's thread pool capacity. Advanced users
can increase its capacity, but normal users should follow default
behavior. If you want to increase it and forcibly retrying bulk
request, please consider to change unrecoverable_error_types parameter
from default value. Change default value of thread_pool.bulk.queue_size
in elasticsearch.yml)
type: string
user:
description: User for HTTP Basic authentication. This plugin will
escape required URL encoded characters within %{} placeholders.
Expand All @@ -654,6 +718,20 @@ spec:
with UTC (Coordinated Universal Time). This option allows to use
local time if you describe utc_index to false.(default: true)'
type: boolean
validate_client_version:
description: 'When you use mismatched Elasticsearch server and client
libraries, fluent-plugin-elasticsearch cannot send data into Elasticsearch.
(default: false)'
type: boolean
verify_es_version_at_startup:
description: 'Because Elasticsearch plugin should change behavior
each of Elasticsearch major versions. For example, Elasticsearch
6 starts to prohibit multiple type_names in one index, and Elasticsearch
7 will handle only _doc type_name in index. If you want to disable
to verify Elasticsearch version at start up, set it as false.
When using the following configuration, ES plugin intends to communicate
into Elasticsearch 6. (default: true)'
type: boolean
with_transporter_log:
description: 'This is debugging purpose option to enable to obtain
transporter layer log. (default: false)'
Expand Down
78 changes: 78 additions & 0 deletions charts/logging-operator/templates/logging.banzaicloud.io_outputs.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -410,28 +410,58 @@ spec:
required:
- timekey
type: object
bulk_message_request_threshold:
description: 'Configure bulk_message request splitting threshold
size. Default value is 20MB. (20 * 1024 * 1024) If you specify
this size as negative number, bulk_message request splitting feature
will be disabled. (default: 20MB)'
type: string
content_type:
description: 'With content_type application/x-ndjson, elasticsearch
plugin adds application/x-ndjson as Content-Type in payload. (default:
application/json)'
type: string
custom_headers:
description: 'This parameter adds additional headers to request.
Example: {"token":"secret"} (default: {})'
type: string
customize_template:
description: Specify the string and its value to be replaced in
form of hash. Can contain multiple key value pair that would be
replaced in the specified template_file. This setting only creates
template and to add rollover index please check the rollover_index
configuration.
type: string
default_elasticsearch_version:
description: This parameter changes that ES plugin assumes default
Elasticsearch version. The default value is 5.
type: string
deflector_alias:
description: Specify the deflector alias which would be assigned
to the rollover index created. This is useful in case of using
the Elasticsearch rollover API
type: string
exception_backup:
description: 'Indicates whether to backup chunk when ignore exception
occurs. (default: true)'
type: boolean
fail_on_putting_template_retry_exceed:
description: 'Indicates whether to fail when max_retry_putting_template
is exceeded. If you have multiple output plugin, you could use
this property to do not fail on fluentd statup.(default: true)'
type: boolean
flatten_hashes:
description: 'Elasticsearch will complain if you send object and
concrete values to the same field. For example, you might have
logs that look this, from different places: {"people" => 100}
{"people" => {"some" => "thing"}} The second log line will be
rejected by the Elasticsearch parser because objects and concrete
values can''t live in the same field. To combat this, you can
enable hash flattening.'
type: boolean
flatten_hashes_separator:
description: Flatten separator
type: string
host:
description: You can specify Elasticsearch host by this parameter.
(default:localhost)
Expand All @@ -448,6 +478,16 @@ spec:
id_key:
description: https://github.com/uken/fluent-plugin-elasticsearch#id_key
type: string
ignore_exceptions:
description: A list of exception that will be ignored - when the
exception occurs the chunk will be discarded and the buffer retry
mechanism won't be called. It is possible also to specify classes
at higher level in the hierarchy. For example `ignore_exceptions
["Elasticsearch::Transport::Transport::ServerError"]` will match
all subclasses of ServerError - Elasticsearch::Transport::Transport::Errors::BadRequest,
Elasticsearch::Transport::Transport::Errors::ServiceUnavailable,
etc.
type: string
include_index_in_url:
description: With this option set to true, Fluentd manifests the
index name in the request URL (rather than in the request body).
Expand All @@ -471,6 +511,14 @@ spec:
description: Specify the index prefix for the rollover index to
be created.
type: string
log_es_400_reason:
description: 'By default, the error logger won''t record the reason
for a 400 error from the Elasticsearch API unless you set log_level
to debug. However, this results in a lot of log spam, which isn''t
desirable if all you want is the 400 error reasons. You can set
this true to capture the 400 error reasons without all the other
debug logs. (default: false)'
type: boolean
logstash_dateformat:
description: 'Set the Logstash date format.(default: %Y.%m.%d)'
type: string
Expand Down Expand Up @@ -589,6 +637,12 @@ spec:
description: If you want to configure SSL/TLS version, you can specify
ssl_version parameter. [SSLv23, TLSv1, TLSv1_1, TLSv1_2]
type: string
suppress_doc_wrap:
description: 'By default, record body is wrapped by ''doc''. This
behavior can not handle update script requests. You can set this
to suppress doc wrapping and allow record body to be untouched.
(default: false)'
type: boolean
tag_key:
description: 'This will add the Fluentd tag in the JSON record.(default:
tag)'
Expand Down Expand Up @@ -644,6 +698,16 @@ spec:
degree of sub-second time precision to preserve from the time
portion of the routed event.
type: string
unrecoverable_error_types:
description: Default unrecoverable_error_types parameter is set
up strictly. Because es_rejected_execution_exception is caused
by exceeding Elasticsearch's thread pool capacity. Advanced users
can increase its capacity, but normal users should follow default
behavior. If you want to increase it and forcibly retrying bulk
request, please consider to change unrecoverable_error_types parameter
from default value. Change default value of thread_pool.bulk.queue_size
in elasticsearch.yml)
type: string
user:
description: User for HTTP Basic authentication. This plugin will
escape required URL encoded characters within %{} placeholders.
Expand All @@ -654,6 +718,20 @@ spec:
with UTC (Coordinated Universal Time). This option allows to use
local time if you describe utc_index to false.(default: true)'
type: boolean
validate_client_version:
description: 'When you use mismatched Elasticsearch server and client
libraries, fluent-plugin-elasticsearch cannot send data into Elasticsearch.
(default: false)'
type: boolean
verify_es_version_at_startup:
description: 'Because Elasticsearch plugin should change behavior
each of Elasticsearch major versions. For example, Elasticsearch
6 starts to prohibit multiple type_names in one index, and Elasticsearch
7 will handle only _doc type_name in index. If you want to disable
to verify Elasticsearch version at start up, set it as false.
When using the following configuration, ES plugin intends to communicate
into Elasticsearch 6. (default: true)'
type: boolean
with_transporter_log:
description: 'This is debugging purpose option to enable to obtain
transporter layer log. (default: false)'
Expand Down
Loading

0 comments on commit f02dc07

Please sign in to comment.