Add hermetic Vim test and benchmark harness#9
Merged
Conversation
tony
force-pushed
the
refactor-ideas-3-test-harness
branch
4 times, most recently
from
a4ef100 to
43b85ca
Compare
why: Support the new libtestvim Python package and benchmark artifacts what: - Add Python bytecode, dist/, .venv/, .pytest_cache/ patterns - Add artifacts/ and benchmark output ignores - Remove autoload/plug.vim* ignore (now vendored)
why: Enable tests to run in isolated environments without affecting user config what: - Add lib#IsTestMode(), lib#ConfigRoot(), lib#PluginRoot(), lib#StateRoot() - Add lib#ResolvePath(), lib#JoinPath(), lib#ConfigPath(), lib#StatePath() - Add lib#FzfRoot() for FZF integration path resolution - Improve lib#SourceIfExists() to use fnameescape()
why: Replace hardcoded ~/.vim/ paths so config works in hermetic test mode what: - Replace hardcoded paths with lib#ConfigPath(), lib#PluginRoot() calls - Add self-locating g:vim_config_root in vimrc - Wrap bare autocmds in augroups - Disable CoC extensions and xrdb in test mode - Guard Biome ALE helper against missing ALE - Move plug#begin/end into plugins.vim
tony
force-pushed
the
refactor-ideas-3-test-harness
branch
from
3eae7db to
2814237
Compare
why: CI and test harness need vim-plug available without network access what: - Add upstream vim-plug (MIT-licensed) to autoload/plug.vim - Source: junegunn/vim-plug @ baa66bc (2025-03-29) Post-0.14.0 unreleased master, 7 commits ahead of tag - SHA256: c2d8998469a049a51225a71128a12917b379822d16b639493e29ea02d8787306 See also: - junegunn/vim-plug@baa66bc - https://github.com/junegunn/vim-plug/blob/master/LICENSE (MIT) - https://github.com/junegunn/vim-plug/releases/tag/0.14.0
why: Provide a reusable Python package for Vim harnessing, profiling, and benchmarks what: - Add pyproject.toml with uv-managed dependencies - Add src/libtestvim/ package: harness, models, specs, serde, CLI, MCP server, parsers, capabilities, artifacts, gitmeta, tmux, util - Include uv.lock for reproducible installs
why: Replace ad-hoc test files with structured, discoverable test suites what: - Add tests/vim/runner.vim test runner with TAP-style output - Add tests/vim/helpers.vim shared assertion utilities - Add core suites: options_and_mappings, filetypes_and_autocmds, startup_and_registration - Add integration suites: gated_plugins, project_root_and_local_commands
why: Python-driven tests for startup benchmarks, tmux smoke tests, and API validation what: - Add conftest.py with shared harness and tmp_home fixtures - Add test_vim_core.py, test_vim_tmux_smoke.py, test_startup_benchmark.py, test_compare_smoke.py, test_libtestvim_interfaces.py - Add fixture corpus: markdown, python, typescript, vim sample files
why: Provide task runner interface and GitHub Actions merge gate what: - Add justfile with sync, test, lint, benchmark, and compare recipes - Add .github/workflows/harness.yml CI workflow for test and lint gates
why: Old test files and Makefile are superseded by the new harness what: - Delete tests/basic.vim, tests/colorscheme.vim, tests/filetype.vim, tests/simplification.vim, tests/harness.vim - Delete Makefile (replaced by justfile)
why: Documentation should reflect the new harness, test structure, and commands what: - Update README.md with new test and benchmark commands - Update CLAUDE.md with harness architecture and test-mode details - Rewrite tests/README.md to document new suite layout
…workflow standards why: Align .vim repo with project-wide agent guidance conventions used by libtmux, libvcs, and ai-workflow-plugins what: - Update header to universal LLM agent audience - Add critical requirements block (all tests must pass) - Add development environment and tooling list - Add development workflow (lint, typecheck, test steps) - Add testing guidelines for dual test stack (Vimscript + pytest) - Add coding standards for Python and Vimscript - Add git commit standards with project-specific scopes (vim, harness, tests, ci) and version-bump format - Add commit quality checklist (do/don't) - Add documentation standards (one command per code block) - Add debugging tips (pause, minimize, document) - Reformat command sections to one-command-per-block pattern
tony
force-pushed
the
refactor-ideas-3-test-harness
branch
from
78d2d81 to
a2117ce
Compare
why: In CI, ~/.fzf doesn't exist so fzf_root resolved to None, causing fzf to install into an ephemeral temp HOME that was destroyed between harness invocations. This broke both the integration test and compare benchmark correctness check. what: - Fall back to plugin_root / "fzf" instead of None when ~/.fzf is absent, keeping fzf in the persistent plugged/ directory alongside other vim-plug plugins
why: uvx treated `vint` as both the package name and command, installing the unrelated PyPI `vint` package (which imports `cerf_api`) instead of using `vim-vint`'s `vint` entrypoint. what: - Change `--with vim-vint` to `--from vim-vint` so uvx resolves the `vint` command from the correct package
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
libtestvimpackage for hermetic Vim runs, telemetry, benchmarks, compare reports, and FastMCP accessjustentrypoints plus uv-managed test dependencies and an accepted startup baselineValidation