Document options for enriching events when using ESF

[dedemorton]

Describe the enhancement:

We need to add some info to the ESF docs to help customers who want to use Beats-style processors, but cannot because ESF does not support them. We should explain that they can use a custom ingest pipeline instead and provide examples.

Describe a specific use case for the enhancement or feature:

Customers who are migrating away from functionbeat to ESF might have a use case that requires processors. For example, they might be using processors like decode_json_fields, add_fields, and add_cloud_metadata to enrich their data. However ESF does not support Beats-style processors and may not have access to all the fields needed to enrich events with cloud metadata.

Specifically, we need to:

• Add some clarification to the docs about processor operations that are supported by ingest pipelines.

  For functionalities like adding/removing/renaming fields, JSON, and more, ESF leverages the processors available in the ingest pipeline on Elasticsearch. The integrations allow users to customize the default pipelines by using a custom pipeline. We need to explain this and point to the documentation that describes how to use custom pipelines. (There's a tutorial here, but there's probably something even better in the Elasticsearch docs.)

• Note that ESF is an AWS Lambda function, so it only supports the metadata applicable in this execution environment. Given the typical content of a cloud field:

{
  "cloud": {
    "account.id": "123456789012",
    "availability_zone": "us-east-1c",
    "instance.id": "i-4e123456",
    "machine.type": "t2.medium",
    "image.id": "ami-abcd1234",
    "provider": "aws",
    "region": "us-east-1"
  }
}

Here is the same field from ESF:

All the other cloud fields (image.id, availability_zone, machine.type, and instance.id) are specific to EC2 and not applicable to a Lambda runtime. Even if AWS probably runs these functions on some EC2-based compute unit, it is an implementation detail they don’t make available to customers.

Contacts
@zmoog

[marcaurele]

@dedemorton it would be nice at least to be able to add static fields defined in the configuration of the ESF coming from attributes of the deployment.

[dedemorton]

My apologies, but I need to put this back into the backlog. I thought it would be a quick change, but after digging, can see that it's going to require more work. The tutorial as it's written now isn't ideal because it mentions Elastic Agent and Fleet, which might be confusing to people who are using ESF. Also we need to provide examples that require further investigation and testing. If someone on the development team could write the initial content, we could help get it into the docs, but for now this is going back to the backlog.

[VTLee]

Is there any movement on this? We're looking to upgrade from Functionbeat, but it appears the official suggested solution is not a viable solution if you use processors.

[zmoog]

Is there any movement on this? We're looking to upgrade from Functionbeat, but it appears the official suggested solution is not a viable solution if you use processors.

Hey @VTLee, can you share more about your use case and what you're trying to achieve?

[VTLee]

Is there any movement on this? We're looking to upgrade from Functionbeat, but it appears the official suggested solution is not a viable solution if you use processors.

Hey @VTLee, can you share more about your use case and what you're trying to achieve?

We currently send logs using FunctionBeat 7 to push CloudWatch Logs resulting from AWS Lambda executions to LogStash. Our logging admins are deprecating support for beats < v8. So when we looked to upgrade our shipper, we found that FunctionBeat 7 & 8 are no longer supported, and instead, we should go to ESF

ESF only supports LogStash as a technical preview according to the graphic on that page. Additionally, it appears there's no replacement for "processors" to manipulate the data and enrich it before it ships over -- dropping lines we consider junk, truncating huge log lines, parsing common fields (like AWS Lambda "REPORT" lines), etc.

I saw this issue was discussing adding support and/or documentation on how to add processors, and I was hoping that would explain how we could do this from the shipper side (we are not able to manipulate the data once it is received, and they accept the data as sent)