[FR] Add keep metadata check to esql schema test

[eric-forte-elastic]

Pull Request

Issue link(s):

Resolves #5440

Related to #5439

Summary - What I changed

I added a schema test to enforce adding metadata fields to the keep portion of non-aggregate ES|QL queries.

Note unit tests will continue to fail until the rules are updated to match the new keep requirement.

How To Test

Use view rule on an ES|QL rule that does not have the metadata fields _id, _version, _index in its keep line.

E.g.

Checklist

• Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[github-actions]

Enhancement - Guidelines

These guidelines serve as a reminder set of considerations when addressing adding a new schema feature to the code.

Documentation and Context

• Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
• Include additional context or screenshots.
• Ensure the enhancement includes necessary updates to the documentation and versioning.

Code Standards and Practices

• Code follows established design patterns within the repo and avoids duplication.
• Ensure that the code is modular and reusable where applicable.

Testing

• New unit tests have been added to cover the enhancement.
• Existing unit tests have been updated to reflect the changes.
• Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
• Validate that any rules affected by the enhancement are correctly updated.
• Ensure that performance is not negatively impacted by the changes.
• Verify that any release artifacts are properly generated and tested.
• Conducted system testing, including fleet, import, and create APIs (e.g., run make test-cli, make test-remote-cli, make test-hunting-cli)

Additional Schema Related Checks

• Verify that the enhancement works across all relevant environments (e.g., different OS versions).
• Link to the relevant Kibana PR or issue provided
• Test export/import flow:
  • Exported detection rule(s) from Kibana to showcase the feature(s)
  • Converted the exported ndjson file(s) to toml in the detection-rules repo
  • Re-exported the toml rule(s) to ndjson and re-imported into Kibana
• Updated necessary unit tests to accommodate the feature
• Incorporated a comprehensive test rule in unit tests for full schema coverage
• Applied min_compat restrictions to limit the feature to a specified minimum stack version
• Executed all unit tests locally with a test toml rule to confirm passing
• Included Kibana PR implementer as an optional reviewer for insights on the feature
• Implemented requisite downgrade functionality
• Cross-referenced the feature with product documentation for consistency
• Confirm that the proper version label is applied to the PR patch, minor, major.

[eric-forte-elastic]

If #5433 merges and we are fine with keep * then this PR needs to be updated to allow for keep * as well.

Updated: Complete

[Mikaayenson]

Maybe clarify in docs/tests that this is specifically about ensuring those metadata fields are in the output row (i.e., in keep), not just in METADATA

[Mikaayenson]

How does this work with things like keep Esql.* / keep aws.*?

[eric-forte-elastic]

This pattern is used to grab all of the fields that are kept as a single string. It is then parsed later into each individual field to check for the appropriate metadata or presence of an *. For cases like keep Esql.* and keep aws.* they are treated as part of the keep string.

For example, a rule with query like this: (also see example in our unit test)

from logs-endpoint.events.process-*, logs-windows.sysmon_operational-*, logs-system.security-*, logs-windows.*, winlogbeat-*, logs-crowdstrike.fdr*, logs-m365_defender.event-* METADATA _id, _version, _index
| where
    @timestamp > now() - 8 hours and
    event.category == "process" and
    event.type == "start" and
    process.name == "rundll32.exe" and
    process.command_line like "*DavSetCookie*"
| keep Esql.*, aws.*, event.*, host.*, process.*, user.*, *

Will have a keep_pattern.search(query_lower).group(1) of 'esql.*, aws.*, event.*, host.*, process.*, user.*, *\n' which is how we are using the keep_pattern.