From 7dec47aa123d162deea46d6c86a69e52c15756ad Mon Sep 17 00:00:00 2001 From: yosefAlsuhaibani Date: Fri, 30 Jan 2026 09:22:23 -0500 Subject: [PATCH 1/2] chore: update contributing docs with latest uv setup --- docs/contributing/semgrep-contributing.md | 73 ++++++++----------- .../contributing/semgrep-core-contributing.md | 40 +++------- 2 files changed, 41 insertions(+), 72 deletions(-) diff --git a/docs/contributing/semgrep-contributing.md b/docs/contributing/semgrep-contributing.md index 6d88afa0fa..ac253c24d9 100644 --- a/docs/contributing/semgrep-contributing.md +++ b/docs/contributing/semgrep-contributing.md @@ -7,10 +7,10 @@ The `semgrep-cli` name refers to the project that exposes the actual `semgrep` c ## Prerequisite - Python >= 3.10 installed in your local machine. -- [`pipenv`](https://github.com/pypa/pipenv) for managing your virtual +- [`uv`](https://github.com/astral-sh/uv) for managing your virtual environment. - - Install it by following the `pipenv` [documentation](https://pipenv.pypa.io/en/latest/installation.html). - - Ensure `pipenv` is on your `$PATH` before proceeding. + - Install it by following the `uv` [documentation](https://docs.astral.sh/uv/getting-started/installation/) + - Ensure `uv` is on your `$PATH` before proceeding. ## Set up the environment @@ -20,25 +20,22 @@ Most Python development is done inside the `cli` directory: cd cli ``` -Next, initialize and enter the virtual environment. The following command installs developer dependencies, such as `pytest`, and installs `semgrep` in editable mode in the virtual environment. From the `cli` directory, run the following command: +Next, initialize the virtual environment. The following commands installs both the required dependencies and the developer dependencies, such as `pytest`, and installs `semgrep` in editable mode in the virtual environment. From the `cli` directory, run the following command: ```bash -pipenv shell +uv sync +uv build +uv pip install dist/*.whl ``` -By convention, your shell prompt is prepended with `(cli)` when the virtual environment is active. - -Next, install the Python dependencies: +To execute `semgrep` in this virtual environiment, you could activate the virtual environment as follows: ```bash -SEMGREP_SKIP_BIN=true pipenv install --dev +source .venv/bin/activate +semgrep --help ``` -:::info -`SEMGREP_SKIP_BIN` tells the installer that you'll use your own `semgrep-core`; see below.* -::: - -Running `which semgrep` should return a path within your virtual environment. On macOS, this is likely contained within `$HOME/.local/share/virtualenvs/`. +Running `which semgrep` should return a path within your local virtual environment. On macOS, this is likely contained within `cli/.venv/bin/`. ## Get the `semgrep-core` binary @@ -66,36 +63,28 @@ export PATH="${SEMGREP_BREW_CORE_BINARY_PATH}:${PATH}" ## Run `semgrep-cli` -Ensure that you are in the `cli/` directory, and then issue the following command: +Ensure that you are in the `cli/` directory and that you have activated the virtual environment with the installed wheel, then issue the following command: -``` -pipenv run semgrep --help +```bash +semgrep --help ``` To try a simple analysis, run: -``` +```bash echo 'if 1 == 1: pass' | semgrep --lang python --pattern '$X == $X' - ``` You now have Semgrep running locally. -## Install `semgrep` - -You can always run `semgrep` from `cli/`, which will use your latest changes in that directory, but you may also want to install the `semgrep` binary. To do this, run - -``` -pipenv install --dev -``` - If you encounter difficulties, reach out to the [`semgrep` team on Slack](https://go.semgrep.dev/slack). Now you can run `semgrep --help` from anywhere. If you have installed `semgrep-core` from source, there are convenient targets in the root Makefile that let you update all binaries. After you pull, run: -``` -make rebuild +```bash +make ``` See the Makefile in `cli/` @@ -104,9 +93,9 @@ See the Makefile in `cli/` Semgrep uses `mypy` to do static type-checking of its Python code. Therefore, when adding a new Python package, you also need to add typing stubs for that package. This can be done in three steps. For example, suppose you are adding the package `pyyaml` to Semgrep. -1. Install the corresponding package with typing stubs. For this `pyyaml` example, the corresponding package is `types-pyyaml`. In the following command, `--dev` specifies that this package is needed for development but not in production. This command updates `cli/Pipfile` with the typing stubs package, and adds both the typing stubs and the package itself to your `Pipfile.lock`. This allows you to import the package in your code (for example, `import yaml as pyyaml`). +1. Install the corresponding package with typing stubs. For this `pyyaml` example, the corresponding package is `types-pyyaml`. In the following command, `--group dev` specifies that this package is needed for development but not in production. This command updates `cli/pyproject.toml` with the typing stubs package, and adds both the typing stubs and the package itself to your `uv.lock`. This allows you to import the package in your code (for example, `import yaml as pyyaml`). ``` - pipenv install --dev types-pyyaml + uv add --group dev types-pyyaml ``` 2. Add the typing stubs package to `.pre-commit-config.yaml` so that the pre-commit `mypy` hook can find the package. ``` @@ -115,9 +104,9 @@ Semgrep uses `mypy` to do static type-checking of its Python code. Therefore, wh - ... - types-PyYAML ``` -3. Add the original package to `cli/setup.py` in the `install_requires` list variable. You can find the version number either in the `Pipfile.lock` file or by looking up the most recent major version of the package online. +3. Add the original package to `cli/pyproject.toml` in the `dependencies` list. You can find the version number either in the `Pipfile.lock` file or by looking up the most recent major version of the package online. ``` - install_requires = [ + dependencies = [ ... "pyyaml~=6.0", ] @@ -130,7 +119,7 @@ This change makes your package a dependency of published Semgrep. Without this c For a reference build that's known to work, consult the root `Dockerfile` to build Semgrep inside a container. You can check that it builds with -``` +```sh docker build -t semgrep . ``` @@ -140,14 +129,14 @@ docker build -t semgrep . To run tests, run the following command: -``` -pipenv run pytest +```sh +uv run pytest ``` There are some much slower tests that run Semgrep on many open source projects. To run these slow tests, run: ```sh -pipenv run pytest tests/qa +uv run pytest tests/qa ``` If you want to update the tests to match the current output: @@ -158,19 +147,19 @@ make regenerate-tests If you want to run a single test file: -``` -pipenv run pytest path/to/test.py +```sh +uv run pytest path/to/test.py ``` Or run an individual test function: -``` -pipenv run pytest path/to/test.py::test_func_name +```sh +uv run pytest path/to/test.py::test_func_name ``` `semgrep-cli` also includes [`pytest-benchmark`](https://pytest-benchmark.readthedocs.io/en/latest/) to allow for basic benchmarking functionality. Run the following command: -``` -pipenv run pytest --benchmark-only +```sh +uv run pytest --benchmark-only ``` diff --git a/docs/contributing/semgrep-core-contributing.md b/docs/contributing/semgrep-core-contributing.md index 52246a89a8..2aeef63f5d 100644 --- a/docs/contributing/semgrep-core-contributing.md +++ b/docs/contributing/semgrep-core-contributing.md @@ -42,14 +42,6 @@ brew install pkg-config bash Lastly, you will almost certainly want the Python environment for `semgrep-cli` configured before proceeding. Please refer to the [Set up the environment](/contributing/semgrep-contributing#set-up-the-environment) documentation. -Once you've returned here, ensure that your shell is able to enter the Python -virtual environment. - -```bash -cd cli; pipenv shell # enter the virtual environment -cd .. # from within the virtual environment, return to the repo root -``` - ### First-time installation The root `Makefile` contains targets that take care of building the @@ -83,16 +75,10 @@ Unless there is a significant dependency change, you won't need to run `make dev The Semgrep team has provided useful targets to help you build and link the entire semgrep project, including both `semgrep-core` and `semgrep`. You may find these helpful. -To install the latest OCaml binaries and `semgrep` binary after pulling source code changes from Git, run: - -``` -make rebuild -``` - -To install after you make a change locally, run +To build the latest OCaml binaries and `semgrep` binary after pulling source code changes from Git, run: -``` -make build # or just `make` +```sh +make ``` After making either of these targets, `semgrep` runs with all your local changes, OCaml and Python both. @@ -115,12 +101,6 @@ git submodule update --recursive This will update internal dependencies. (We suggest aliasing it to `uu`) -After `tree-sitter` is updated, you may need to reconfigure it. If so, run - -``` -make config -``` - ### Develop `semgrep-core` If you are developing `semgrep-core`, Use `Makefile` in the repository root for `core` and `core-test` targets; the code is primarily in `src/`. @@ -129,25 +109,25 @@ The following assumes you are in the repository root. After you pull or make a change, compile using -``` +```sh make ``` -This will build an executable for `semgrep-core` in `_build/default/src/main/Main.exe` (we suggest aliasing this to `sc`). Try it out by running +This will build an executable for `semgrep-core` in `bin/`. Try it out by running -``` -_build/default/src/main/Main.exe -help +```sh +bin/semgrep-core -help ``` When you are done, test your changes with -``` +```sh make core-test ``` Finally, to update the `semgrep-core` binary used by `semgrep`, run -``` +```sh make copy-core-for-cli ``` @@ -325,7 +305,7 @@ information, for example: ```bash export SEMGREP_CORE_DEBUG=1 export SEMGREP_CORE_PROFILE=1 -pipenv run semgrep -f ../semgrep-core/tests/PERF/ajin.yaml ../semgrep-core/tests/PERF/three.js +uv run semgrep -f ../semgrep-core/tests/PERF/ajin.yaml ../semgrep-core/tests/PERF/three.js ``` will output: ```bash From eb101ea0bbb37b67fe867a274bcc8e92bae78643 Mon Sep 17 00:00:00 2001 From: yosefAlsuhaibani Date: Fri, 30 Jan 2026 09:42:14 -0500 Subject: [PATCH 2/2] Don't rename sc --- docs/contributing/contributing-code.md | 2 +- docs/contributing/semgrep-core-contributing.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/contributing/contributing-code.md b/docs/contributing/contributing-code.md index b436cda46c..e304aa9e64 100644 --- a/docs/contributing/contributing-code.md +++ b/docs/contributing/contributing-code.md @@ -83,7 +83,7 @@ Currently, depending on the flags used, `spacegrep` is invoked both independentl ## Making a change -Semgrep runs on Python versions >= 3.8. If you don't have one of these versions installed, please do so before proceeding. +Semgrep runs on Python versions >= 3.10. If you don't have one of these versions installed, please do so before proceeding. Because the Python and OCaml development paths are relatively independent, the instructions are divided into Python ([semgrep-cli contributing](semgrep-contributing.md)) and OCaml ([semgrep-core contributing](semgrep-core-contributing.md)). diff --git a/docs/contributing/semgrep-core-contributing.md b/docs/contributing/semgrep-core-contributing.md index 2aeef63f5d..17d014e6a6 100644 --- a/docs/contributing/semgrep-core-contributing.md +++ b/docs/contributing/semgrep-core-contributing.md @@ -113,10 +113,10 @@ After you pull or make a change, compile using make ``` -This will build an executable for `semgrep-core` in `bin/`. Try it out by running +This will build an executable for `semgrep-core` in `_build/default/src/main/Main.exe` (we suggest aliasing this to `sc`). Try it out by running ```sh -bin/semgrep-core -help +_build/default/src/main/Main.exe -help ``` When you are done, test your changes with