Skip to content

_count_lines reads the whole file just to count newlines #20

Description

@DearGitLab

backend/app/code/context/repo_scanner.py:232-238:

def _count_lines(self, file_path: str) -> int:
    try:
        with open(file_path, 'r', encoding='utf-8', errors='ignore') as f:
            return sum(1 for _ in f)
    except Exception:
        return 0

For a repo scan, this opens and reads every file in text mode, which on a 1 MB max-size file with errors='ignore' runs UTF-8 decode over the entire buffer just to throw it away. On a large monorepo (e.g. tens of thousands of TS files in a yarn workspace), this is the dominant cost in RepoScanner.scan — and the result (line_count) is only used for display.

Two cheap improvements:

  1. Read in bytes, then count b'\n':

    with open(file_path, 'rb') as f:
        return f.read().count(b'\n')

    Same answer for any file with a trailing newline, slightly off (by 1) for files without — but line_count is informational, not load-bearing.

  2. Stream in chunks for files near the 1 MB cap:

    count = 0
    with open(file_path, 'rb') as f:
        for chunk in iter(lambda: f.read(1 << 16), b''):
            count += chunk.count(b'\n')
    return count

(1) alone gives roughly a 3-4× speedup vs the current implementation in my quick mental check. (2) caps memory usage if the limit is ever raised above 1 MB.

Not a blocker, but RepoScanner.scan is in the hot path for context building (see app.code.context.context_pack.py), so any free wins here pay back at every request.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions