User cannot change Shards for some indices (e.g. System, Illuminate)

[drewmiranda-gl]

Graylog users (anyone using graylog, even admins) are disallowed from modifying the Index Set settings for specific Index Sets.

Attempting to save index set settings for the affected index sets yields the following error:

HTTP 400

{
    "type": "ApiError",
    "message": "Custom field mappings cannot be set for events and failures index sets"
}

See

graylog2-server/graylog2-server/src/main/java/org/graylog2/indexer/IndexSetValidator.java

Line 139 in 6864f1d

return Violation.create("Custom field mappings cannot be set for events and failures index sets");

Appears to apply to the following types of Index Template Types:

graylog2-server/graylog2-server/src/main/java/org/graylog2/indexer/indexset/IndexSetConfig.java

Line 67 in 6864f1d

private static final Set<String> TEMPLATE_TYPES_FOR_INDEX_SETS_WITH_IMMUTABLE_FIELD_TYPES = Set.of(

• EVENT_TEMPLATE_TYPE
• FAILURE_TEMPLATE_TYPE
• ILLUMINATE_INDEX_TEMPLATE_TYPE

It is crucial the user be able to modify at least the shard settings in order to specify the correct number of primary shards and replica shards. Not having the ability to do so can make the OpenSearch cluster brittle and unreliable, causing a Red cluster health and cause stability problems for Graylog, such as the inability to ingest new messages or search existing messages.

Expected Behavior

Users are able to modify crucial shard settings for ALL index sets

Current Behavior

Users are unable to modify the above mentioned index sets

Possible Solution

Allow modification of "Details" section for ALL index sets (Index Shards, Index Replicas, Maximum Number of Segments)

Steps to Reproduce (for bugs)

Via Graylog Web Interface

1. Navigate to System / Indices
2. Click Edit for one of the affected Index Set types (e.g. Graylog Events)
3. Scroll down to bottom of page
4. Click Update index set

Context

I suffered a fatal error with my 2 node OpenSearch cluster. To avoid future issues I am reconfiguring all index sets to have at least 1 replica shard so that if an OpenSearch node is offline or a shard has a catastrophic write failure, the OpenSearch cluster won't go into a Red Health state preventing log ingestion.

Your Environment

• Graylog Version: 6.1.4
• Java Version: Bundled
• OpenSearch Version: 2.15.0
• MongoDB Version: 7.0.14
• Operating System: Ubuntu 22.04 LTS
• Browser version: Google Chrome Version 131.0.6778.140 (Official Build) (arm64)

Please let me know if there are any questions.

[tellistone]

Unable to reproduce in master, I am able to change shard config of Graylog Events index

[drewmiranda-gl]

@tellistone this appears to be mostly an issue with illuminate indices. I did confirm i cannot reproduce this with a greenfield 6.1 environment. I CAN save settings for an illuminate indices.

Also is it possible there is something that gets carried over from existing installs vs a greenfield install? How can one work around this issue if they are to encounter it? Is it also possible code changes were made to master that resolved this?

This is very important for configuring a Graylog cluster to be HA.

[drewmiranda-gl]

The error is very strange because it claims i am trying to change the field mappings for illuminate which, not only am i not trying to do, the UX removes the field mapping UI elements and the PUT payload also is not setting any field mappings:

{
    "title": "Fortinet Event Log Messages",
    "description": "Fortinet Event Log Messages",
    "index_prefix": "gl_fortinet",
    "writable": true,
    "can_be_default": true,
    "shards": 1,
    "data_tiering": {
        "type": "hot_only",
        "index_lifetime_min": "P14D",
        "index_lifetime_max": "P30D"
    },
    "replicas": 0,
    "rotation_strategy_class": "org.graylog2.indexer.rotation.strategies.TimeBasedSizeOptimizingStrategy",
    "rotation_strategy": {
        "type": "org.graylog2.indexer.rotation.strategies.TimeBasedSizeOptimizingStrategyConfig",
        "index_lifetime_min": "P7D",
        "index_lifetime_max": "P14D"
    },
    "retention_strategy_class": "org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy",
    "retention_strategy": {
        "type": "org.graylog2.indexer.retention.strategies.DeletionRetentionStrategyConfig",
        "max_number_of_indices": 7
    },
    "index_analyzer": "keyword",
    "index_optimization_max_num_segments": 1,
    "index_optimization_disabled": false,
    "field_type_refresh_interval": 30000,
    "id": "63c1b02d5707651ef182399a",
    "creation_date": "2023-01-13T19:25:33.406Z",
    "index_template_type": "illuminate_content",
    "use_legacy_rotation": true,
    "default": false
}

[drewmiranda-gl]

Found the issue, so this is self-inflected and occurred before users were disallowed from changing illuminate index template settings. Though even though the UX allowed this previously graylog still ignored applying these to illuminate indices. Once the graylog code changes to disallow the UX to change field types for illuminate indices the use can no longer see/remove these mappings other than directly editing the mongo record.

If i remove the following from the record in mongo's index_sets collection, i can then successfully save changes made to index settings for this specific index set.

    "custom_field_mappings" : [
        {
            "field" : "source_ip",
            "type" : "ip"
        },
        {
            "field" : "destination_ip",
            "type" : "ip"
        },
        {
            "field" : "network_bytes",
            "type" : "long"
        }
    ]

Left is my existing real environment and right is a greenfield 6.1 lab

[tellistone]

So we have a subset of customers who will be unable to modify their illuminate index set config (shard count etc) IF they have any custom mappings created before we locked it down.

Would the more appropriate path be to auto-nuke any custom field mappings on these index sets, or to handle the exception of them being in existence?

[drewmiranda-gl]

Would the more appropriate path be to auto-nuke any custom field mappings on these index sets

I think so. From what I tested previously even though the frontend allowed you to set these on illumiante index sets, they never actually got applied.

I checked the old issue
#19766

Note:

Graylog does save this mapping in the index_sets MongoDB collection: