backend/app/code/context/repo_scanner.py:232-238:
def _count_lines(self, file_path: str) -> int:
try:
with open(file_path, 'r', encoding='utf-8', errors='ignore') as f:
return sum(1 for _ in f)
except Exception:
return 0
For a repo scan, this opens and reads every file in text mode, which on a 1 MB max-size file with errors='ignore' runs UTF-8 decode over the entire buffer just to throw it away. On a large monorepo (e.g. tens of thousands of TS files in a yarn workspace), this is the dominant cost in RepoScanner.scan — and the result (line_count) is only used for display.
Two cheap improvements:
-
Read in bytes, then count b'\n':
with open(file_path, 'rb') as f:
return f.read().count(b'\n')
Same answer for any file with a trailing newline, slightly off (by 1) for files without — but line_count is informational, not load-bearing.
-
Stream in chunks for files near the 1 MB cap:
count = 0
with open(file_path, 'rb') as f:
for chunk in iter(lambda: f.read(1 << 16), b''):
count += chunk.count(b'\n')
return count
(1) alone gives roughly a 3-4× speedup vs the current implementation in my quick mental check. (2) caps memory usage if the limit is ever raised above 1 MB.
Not a blocker, but RepoScanner.scan is in the hot path for context building (see app.code.context.context_pack.py), so any free wins here pay back at every request.
backend/app/code/context/repo_scanner.py:232-238:For a repo scan, this opens and reads every file in text mode, which on a 1 MB max-size file with
errors='ignore'runs UTF-8 decode over the entire buffer just to throw it away. On a large monorepo (e.g. tens of thousands of TS files in a yarn workspace), this is the dominant cost inRepoScanner.scan— and the result (line_count) is only used for display.Two cheap improvements:
Read in bytes, then count
b'\n':Same answer for any file with a trailing newline, slightly off (by 1) for files without — but
line_countis informational, not load-bearing.Stream in chunks for files near the 1 MB cap:
(1) alone gives roughly a 3-4× speedup vs the current implementation in my quick mental check. (2) caps memory usage if the limit is ever raised above 1 MB.
Not a blocker, but
RepoScanner.scanis in the hot path for context building (seeapp.code.context.context_pack.py), so any free wins here pay back at every request.