Skip to content
This repository was archived by the owner on Apr 2, 2024. It is now read-only.
This repository was archived by the owner on Apr 2, 2024. It is now read-only.

Add glob_pattern to LogstreamerInput #1985

@michaelgibson

Description

@michaelgibson

The LogstreamerInput uses filepath.Walk() for locating files to compare to the file_match regexes.
This is normally fine in a mostly sane directory structure.
In the case of ridiculously large/complex filesystems with numerous levels containing millions of files, this can be somewhat inefficient.

The pattern matching is useful so not suggesting a replacement for regexing.
This change should still follow the same rules for log_directory and file_match.
I propose adding an additional config option to use globbing specifically for the filepath.Walk portion.
https://github.com/mozilla-services/heka/blob/dev/logstreamer/filehandling.go#L280

glob_pattern = "/this/*/path*/would/*take/*/*/*/forever/to/*/walk/*/in/a*/reasonable/*/*/*/timeframe/*"

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions