From 68bd2f18a4f6eb012731cdc7a50bec59f99437cc Mon Sep 17 00:00:00 2001 From: Fang-Pen Lin Date: Mon, 12 Jan 2026 20:39:48 -0800 Subject: [PATCH 1/8] Add e2e test doc --- e2e/README.md | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 e2e/README.md diff --git a/e2e/README.md b/e2e/README.md new file mode 100644 index 00000000..a40b28b6 --- /dev/null +++ b/e2e/README.md @@ -0,0 +1,49 @@ +# End-to-end tests for the CLI project + +The folder under this folder is for our e2e tests of this CLI project. +The goal of the e2e tests is to simulate the user's interaction with Infisical system all the way from the cli to the backend API server and potentially third-party servers to ensure it works as expected. +Please note that the e2e tests is still under active development and it's subject to rapid changes. +The document here could be outdated. +If you find anything here that doesn't make sense or doesn't work as described in the document, it's likely we have already changed it. +Please feel free to reach out to @fangpenlin in Slack channel if you encounter any problem with the e2e tests. + +## Running the test + +The main subject of testing is a CLI executable program, i.e the `infisical` command. +Currently we support two approaches to test the CLI. +The default run method is `subprocess`. +You can set the `CLI_E2E_DEFAULT_RUN_METHOD` environment variable to change the default run method. +For example: + +```bash +export CLI_E2E_DEFAULT_RUN_METHOD=functionCall +``` + +### The `subprocess` run method + +The most straightforward way to run a CLI cmd and test it is to start a new subprocess with the cmd executable, control the input by environment vars, stdin, args and see how it connects to the server and what it outputs (stdout and stderr). +The benefit of this approach is that it simulates what user really does and we can collect the stdout and stderr easily. +The drawback of this approach is that attaching a debugger to the CLI cmd requires some extra work (like finding the PID and attach to it by using the PID from the debugger). +This is the default run method if not specified. + +When using this run method, you need make sure the executable at the default location `./infisical-merge`. +Otherwise, you can also specify the path to the executable by setting `INFISICAL_CLI_EXECUTABLE` environment variable like this + +```bash +export INFISICAL_CLI_EXECUTABLE=/path/to/infisical-merge +``` + +To build the executable, you can go to the root folder of this cli project then run + +```bash +go build . +``` + +### The `functionCall` run method + +The function call method calling the `main` cmd function directly from the e2e test case. +The benefit is that you can attach debugger directly to the cli process without extra effort. +The drawback is that currently we cannot collect stdout and stderr. +Some extra efforts might be needed to update the CLI code to abstract the stdout and stderr output from logs to make it possible. +In the meantime since this is not available, we didn't set it as the default value. +With this run method, since that we are linking the e2e test build with the CLI as a library directly, there's no need to build the executable sperately. From e7b750ee20b31d1ea2ef55ee1590984948cb63e3 Mon Sep 17 00:00:00 2001 From: Fang-Pen Lin Date: Mon, 12 Jan 2026 20:50:30 -0800 Subject: [PATCH 2/8] Update doc --- e2e/README.md | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/e2e/README.md b/e2e/README.md index a40b28b6..0871ac8d 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -47,3 +47,51 @@ The drawback is that currently we cannot collect stdout and stderr. Some extra efforts might be needed to update the CLI code to abstract the stdout and stderr output from logs to make it possible. In the meantime since this is not available, we didn't set it as the default value. With this run method, since that we are linking the e2e test build with the CLI as a library directly, there's no need to build the executable sperately. + +## Setting the `INFISICAL_BACKEND_DIR` value + +To make our CLI test against the actual Infisical API server, we provide easy to use API to spin up the full Infisical stack in docker-compose like this: + +```go +infisical := NewInfisicalService(). + WithBackendEnvironment(types.NewMappingWithEquals([]string{ + // This is needed for the private ip (current host) to be accepted for the relay server + "ALLOW_INTERNAL_IP_CONNECTIONS=true", + })). + Up(t, ctx) +``` + +Because it runs the actual Infisical API server stack, you need to specify `INFISICAL_BACKEND_DIR` value and pointing to the `backend` folder of the [infisical repository](https://github.com/infisical/infisical) to make it works +Like, for example, you have the repo checked out at `/Users/fangpen/workspace/infisical`. +Then you can expose environment variable like this: + +```bash +export INFISICAL_BACKEND_DIR=/Users/fangpen/workspace/infisical/backend +``` + +Please note that the `/backend` path is appended. + +Currently, you need to enable the needed feature flag for running your tests manually in the locally checked out Infisical repo. +If you don't know how to enable those feature flags, please contact the infisical eng-team members, they will show you how to do that. +In the future, we will provide builder API to the `NewInfisicalService()` object for you to simply sepcify which feature you would like to enable. + +## Running the test + +To run the e2e test, you can do the following: + +```bash +# switch to the e2e folder +cd e2e +go test github.com/infisical/cli/e2e-tests/relay +``` + +Combining the exporting environment variables, you might end up with running cmds like this: + +```bash +export INFISICAL_CLI_EXECUTABLE=/path/to/infisical-merge +export INFISICAL_BACKEND_DIR=/path/to/infisical/backend +cd e2e +go test github.com/infisical/cli/e2e-tests/relay +``` + +It's a bit verbose right now, but we will improve the quality of life over time by adding things such as a Makefile to make it much easier. From 53e1c68f3747d6da7d549961c74c620b87616bcc Mon Sep 17 00:00:00 2001 From: Fang-Pen Lin Date: Mon, 12 Jan 2026 20:55:02 -0800 Subject: [PATCH 3/8] More docs --- e2e/README.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/e2e/README.md b/e2e/README.md index 0871ac8d..222312b2 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -95,3 +95,29 @@ go test github.com/infisical/cli/e2e-tests/relay ``` It's a bit verbose right now, but we will improve the quality of life over time by adding things such as a Makefile to make it much easier. + +## Troubleshooting the failing tests due to CLI error + +Sometimes, the e2e tests failed. +To find out why it fails, if it's a failure of the CLI cmd process itself, such as the cmd exits with an error code, you can look at the logs and may found this: + +``` +2026/01/12 11:05:14 INFO Running command as a sub-process executable=/Users/fangpenlin/workspace/cli/infisical-merge args="[relay start --domain http://localhost:54937]" +2026/01/12 11:05:14 INFO Writing stdout to temp file file=/var/folders/wc/g97rf4092_z9wqbp93djvnt00000gn/T/TestRelay_RegistersARelay2199940539/001/stdout.log +2026/01/12 11:05:14 INFO Writing stderr to temp file file=/var/folders/wc/g97rf4092_z9wqbp93djvnt00000gn/T/TestRelay_RegistersARelay2199940539/001/stderr.log +``` + +With the sub-process approach, the stdout and stderr logs should be outputting to the files as shown in the path. +In that case, you may want to inspect the stderr log file like: + +```bash +less /var/folders/wc/g97rf4092_z9wqbp93djvnt00000gn/T/TestRelay_RegistersARelay2199940539/001/stderr.log +``` + +Then you should be able to find otu why it fail. +You can also switch the call method to `functionCall` and setup debugger to trace into the CLI program to find out why they fail. +If you run the cmd test with `functionCall`, it will not write the stdout / stderr to file, but instead, it should print it to the console where you run the tests. + +## Troubleshooting the failing tests due to Infisical backend API errors + +## Use compose containers cache to speed up the development cycle From 47674f7165db44b0d644bd46e54b4c9fecf57119 Mon Sep 17 00:00:00 2001 From: Fang-Pen Lin Date: Mon, 12 Jan 2026 21:02:35 -0800 Subject: [PATCH 4/8] More readme --- e2e/README.md | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/e2e/README.md b/e2e/README.md index 222312b2..12c1802a 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -120,4 +120,37 @@ If you run the cmd test with `functionCall`, it will not write the stdout / stde ## Troubleshooting the failing tests due to Infisical backend API errors +If the errors happen in the backend, to find out what's going on, you can open the Docker Desktop app or use `docker ps` and then `docker logs` to find out what's the error message in the Infisical backend API server. +Please note that by default, the testcontainers library (the library we used to run the docker-compose for the Infisical stack) will start a container called Ryuk for deleting the containers after the test is finished. +Because of that, if you run into an error in the backend reproduced by running the test, the container might already be gone after the test finish. +Then you won't be able to look inside the container and find out what's going on. +To solve the problem, you can set the `TESTCONTAINERS_RYUK_DISABLED` environment to `true` like this to disable the container deleting behavior: + +```bash +TESTCONTAINERS_RYUK_DISABLED=true +``` + ## Use compose containers cache to speed up the development cycle + +More often than not, we may find ourselves in the loop of: + +- Change a few line of CLI code +- Run the test +- Change a few line of CLI code again +- Run the test again +- Change a few line of CLI code once again +- Run the test once again +- ... + +It would be very time consuming to wait for the Infisical backend server to fully bootup for each interation. +To speed up the development cycle, we have a cache system built in. +Here's how it works. +If you have `TESTCONTAINERS_RYUK_DISABLED` set to `true`, each time when the `Up` method of the `InfisicalService()` is called, we will look at the hash value of the compose YAML file generated from the desired environment defined in the e2e test case. +If there's such container already running, we will reuse it by resetting its database instead of starting a new one. +That way, it's much faster than booting up a new compose stack and wait for it to get online. + +If for any reason the cache system is not working or desired, you can disable it by setting `CLI_E2E_DISABLE_COMPOSE_CACHE` value to `1` like this: + +```bash +export CLI_E2E_DISABLE_COMPOSE_CACHE=1 +``` From 89383c0965f6d562ddff3c23388e3937e27362dd Mon Sep 17 00:00:00 2001 From: Fang-Pen Lin Date: Mon, 12 Jan 2026 21:03:57 -0800 Subject: [PATCH 5/8] proofread --- e2e/README.md | 78 +++++++++++++++++++++++++-------------------------- 1 file changed, 39 insertions(+), 39 deletions(-) diff --git a/e2e/README.md b/e2e/README.md index 12c1802a..af8c1c7a 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -1,15 +1,15 @@ # End-to-end tests for the CLI project -The folder under this folder is for our e2e tests of this CLI project. -The goal of the e2e tests is to simulate the user's interaction with Infisical system all the way from the cli to the backend API server and potentially third-party servers to ensure it works as expected. -Please note that the e2e tests is still under active development and it's subject to rapid changes. +This folder contains our e2e tests for this CLI project. +The goal of the e2e tests is to simulate the user's interaction with the Infisical system all the way from the CLI to the backend API server and potentially third-party servers to ensure it works as expected. +Please note that the e2e tests are still under active development and subject to rapid changes. The document here could be outdated. If you find anything here that doesn't make sense or doesn't work as described in the document, it's likely we have already changed it. -Please feel free to reach out to @fangpenlin in Slack channel if you encounter any problem with the e2e tests. +Please feel free to reach out to @fangpenlin on the Slack channel if you encounter any problem with the e2e tests. ## Running the test -The main subject of testing is a CLI executable program, i.e the `infisical` command. +The main subject of testing is a CLI executable program, i.e., the `infisical` command. Currently we support two approaches to test the CLI. The default run method is `subprocess`. You can set the `CLI_E2E_DEFAULT_RUN_METHOD` environment variable to change the default run method. @@ -21,13 +21,13 @@ export CLI_E2E_DEFAULT_RUN_METHOD=functionCall ### The `subprocess` run method -The most straightforward way to run a CLI cmd and test it is to start a new subprocess with the cmd executable, control the input by environment vars, stdin, args and see how it connects to the server and what it outputs (stdout and stderr). -The benefit of this approach is that it simulates what user really does and we can collect the stdout and stderr easily. -The drawback of this approach is that attaching a debugger to the CLI cmd requires some extra work (like finding the PID and attach to it by using the PID from the debugger). +The most straightforward way to run a CLI command and test it is to start a new subprocess with the command executable, control the input by environment vars, stdin, args and see how it connects to the server and what it outputs (stdout and stderr). +The benefit of this approach is that it simulates what a user really does and we can collect the stdout and stderr easily. +The drawback of this approach is that attaching a debugger to the CLI command requires some extra work (like finding the PID and attaching to it by using the PID from the debugger). This is the default run method if not specified. -When using this run method, you need make sure the executable at the default location `./infisical-merge`. -Otherwise, you can also specify the path to the executable by setting `INFISICAL_CLI_EXECUTABLE` environment variable like this +When using this run method, you need to make sure the executable is at the default location `./infisical-merge`. +Otherwise, you can also specify the path to the executable by setting the `INFISICAL_CLI_EXECUTABLE` environment variable like this: ```bash export INFISICAL_CLI_EXECUTABLE=/path/to/infisical-merge @@ -41,16 +41,16 @@ go build . ### The `functionCall` run method -The function call method calling the `main` cmd function directly from the e2e test case. -The benefit is that you can attach debugger directly to the cli process without extra effort. +The function call method calls the `main` command function directly from the e2e test case. +The benefit is that you can attach a debugger directly to the CLI process without extra effort. The drawback is that currently we cannot collect stdout and stderr. Some extra efforts might be needed to update the CLI code to abstract the stdout and stderr output from logs to make it possible. -In the meantime since this is not available, we didn't set it as the default value. -With this run method, since that we are linking the e2e test build with the CLI as a library directly, there's no need to build the executable sperately. +In the meantime, since this is not available, we didn't set it as the default value. +With this run method, since we are linking the e2e test build with the CLI as a library directly, there's no need to build the executable separately. ## Setting the `INFISICAL_BACKEND_DIR` value -To make our CLI test against the actual Infisical API server, we provide easy to use API to spin up the full Infisical stack in docker-compose like this: +To make our CLI test against the actual Infisical API server, we provide an easy-to-use API to spin up the full Infisical stack in docker-compose like this: ```go infisical := NewInfisicalService(). @@ -61,9 +61,9 @@ infisical := NewInfisicalService(). Up(t, ctx) ``` -Because it runs the actual Infisical API server stack, you need to specify `INFISICAL_BACKEND_DIR` value and pointing to the `backend` folder of the [infisical repository](https://github.com/infisical/infisical) to make it works -Like, for example, you have the repo checked out at `/Users/fangpen/workspace/infisical`. -Then you can expose environment variable like this: +Because it runs the actual Infisical API server stack, you need to specify the `INFISICAL_BACKEND_DIR` value and point it to the `backend` folder of the [infisical repository](https://github.com/infisical/infisical) to make it work. +For example, if you have the repo checked out at `/Users/fangpen/workspace/infisical`. +Then you can set the environment variable like this: ```bash export INFISICAL_BACKEND_DIR=/Users/fangpen/workspace/infisical/backend @@ -71,9 +71,9 @@ export INFISICAL_BACKEND_DIR=/Users/fangpen/workspace/infisical/backend Please note that the `/backend` path is appended. -Currently, you need to enable the needed feature flag for running your tests manually in the locally checked out Infisical repo. -If you don't know how to enable those feature flags, please contact the infisical eng-team members, they will show you how to do that. -In the future, we will provide builder API to the `NewInfisicalService()` object for you to simply sepcify which feature you would like to enable. +Currently, you need to enable the needed feature flags for running your tests manually in the locally checked out Infisical repo. +If you don't know how to enable those feature flags, please contact the Infisical engineering team members; they will show you how to do that. +In the future, we will provide a builder API to the `NewInfisicalService()` object for you to simply specify which features you would like to enable. ## Running the test @@ -85,7 +85,7 @@ cd e2e go test github.com/infisical/cli/e2e-tests/relay ``` -Combining the exporting environment variables, you might end up with running cmds like this: +Combining the exported environment variables, you might end up with running commands like this: ```bash export INFISICAL_CLI_EXECUTABLE=/path/to/infisical-merge @@ -98,8 +98,8 @@ It's a bit verbose right now, but we will improve the quality of life over time ## Troubleshooting the failing tests due to CLI error -Sometimes, the e2e tests failed. -To find out why it fails, if it's a failure of the CLI cmd process itself, such as the cmd exits with an error code, you can look at the logs and may found this: +Sometimes, the e2e tests fail. +To find out why they fail, if it's a failure of the CLI command process itself, such as the command exiting with an error code, you can look at the logs and may find this: ``` 2026/01/12 11:05:14 INFO Running command as a sub-process executable=/Users/fangpenlin/workspace/cli/infisical-merge args="[relay start --domain http://localhost:54937]" @@ -114,17 +114,17 @@ In that case, you may want to inspect the stderr log file like: less /var/folders/wc/g97rf4092_z9wqbp93djvnt00000gn/T/TestRelay_RegistersARelay2199940539/001/stderr.log ``` -Then you should be able to find otu why it fail. -You can also switch the call method to `functionCall` and setup debugger to trace into the CLI program to find out why they fail. -If you run the cmd test with `functionCall`, it will not write the stdout / stderr to file, but instead, it should print it to the console where you run the tests. +Then you should be able to find out why it fails. +You can also switch the call method to `functionCall` and set up a debugger to trace into the CLI program to find out why it fails. +If you run the command test with `functionCall`, it will not write the stdout / stderr to a file, but instead, it should print it to the console where you run the tests. ## Troubleshooting the failing tests due to Infisical backend API errors -If the errors happen in the backend, to find out what's going on, you can open the Docker Desktop app or use `docker ps` and then `docker logs` to find out what's the error message in the Infisical backend API server. -Please note that by default, the testcontainers library (the library we used to run the docker-compose for the Infisical stack) will start a container called Ryuk for deleting the containers after the test is finished. -Because of that, if you run into an error in the backend reproduced by running the test, the container might already be gone after the test finish. +If the errors happen in the backend, to find out what's going on, you can open the Docker Desktop app or use `docker ps` and then `docker logs` to find out what the error message is in the Infisical backend API server. +Please note that by default, the testcontainers library (the library we use to run the docker-compose for the Infisical stack) will start a container called Ryuk for deleting the containers after the test is finished. +Because of that, if you run into an error in the backend reproduced by running the test, the container might already be gone after the test finishes. Then you won't be able to look inside the container and find out what's going on. -To solve the problem, you can set the `TESTCONTAINERS_RYUK_DISABLED` environment to `true` like this to disable the container deleting behavior: +To solve the problem, you can set the `TESTCONTAINERS_RYUK_DISABLED` environment variable to `true` like this to disable the container deleting behavior: ```bash TESTCONTAINERS_RYUK_DISABLED=true @@ -134,22 +134,22 @@ TESTCONTAINERS_RYUK_DISABLED=true More often than not, we may find ourselves in the loop of: -- Change a few line of CLI code +- Change a few lines of CLI code - Run the test -- Change a few line of CLI code again +- Change a few lines of CLI code again - Run the test again -- Change a few line of CLI code once again +- Change a few lines of CLI code once again - Run the test once again - ... -It would be very time consuming to wait for the Infisical backend server to fully bootup for each interation. +It would be very time consuming to wait for the Infisical backend server to fully boot up for each iteration. To speed up the development cycle, we have a cache system built in. Here's how it works. -If you have `TESTCONTAINERS_RYUK_DISABLED` set to `true`, each time when the `Up` method of the `InfisicalService()` is called, we will look at the hash value of the compose YAML file generated from the desired environment defined in the e2e test case. -If there's such container already running, we will reuse it by resetting its database instead of starting a new one. -That way, it's much faster than booting up a new compose stack and wait for it to get online. +If you have `TESTCONTAINERS_RYUK_DISABLED` set to `true`, each time the `Up` method of the `InfisicalService()` is called, we will look at the hash value of the compose YAML file generated from the desired environment defined in the e2e test case. +If there's such a container already running, we will reuse it by resetting its database instead of starting a new one. +That way, it's much faster than booting up a new compose stack and waiting for it to get online. -If for any reason the cache system is not working or desired, you can disable it by setting `CLI_E2E_DISABLE_COMPOSE_CACHE` value to `1` like this: +If for any reason the cache system is not working as desired, you can disable it by setting the `CLI_E2E_DISABLE_COMPOSE_CACHE` value to `1` like this: ```bash export CLI_E2E_DISABLE_COMPOSE_CACHE=1 From ba4753606afd3626a7b05dc24225dc873670f4d0 Mon Sep 17 00:00:00 2001 From: Fang-Pen Lin Date: Mon, 12 Jan 2026 21:11:00 -0800 Subject: [PATCH 6/8] Add links --- e2e/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/e2e/README.md b/e2e/README.md index af8c1c7a..c614ba42 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -121,7 +121,7 @@ If you run the command test with `functionCall`, it will not write the stdout / ## Troubleshooting the failing tests due to Infisical backend API errors If the errors happen in the backend, to find out what's going on, you can open the Docker Desktop app or use `docker ps` and then `docker logs` to find out what the error message is in the Infisical backend API server. -Please note that by default, the testcontainers library (the library we use to run the docker-compose for the Infisical stack) will start a container called Ryuk for deleting the containers after the test is finished. +Please note that by default, the [testcontainers library](https://github.com/testcontainers/testcontainers-go) (the library we use to run the docker-compose for the Infisical stack) will start a container called Ryuk for deleting the containers after the test is finished. Because of that, if you run into an error in the backend reproduced by running the test, the container might already be gone after the test finishes. Then you won't be able to look inside the container and find out what's going on. To solve the problem, you can set the `TESTCONTAINERS_RYUK_DISABLED` environment variable to `true` like this to disable the container deleting behavior: @@ -130,6 +130,8 @@ To solve the problem, you can set the `TESTCONTAINERS_RYUK_DISABLED` environment TESTCONTAINERS_RYUK_DISABLED=true ``` +To learn more about the behaivor of Ryuk from testcontainers, please read [their document here](https://golang.testcontainers.org/features/garbage_collector/#ryuk). + ## Use compose containers cache to speed up the development cycle More often than not, we may find ourselves in the loop of: From d1b2b145a90a841579404c831121dd37e0bf7370 Mon Sep 17 00:00:00 2001 From: Fang-Pen Lin Date: Mon, 12 Jan 2026 21:15:38 -0800 Subject: [PATCH 7/8] Update e2e/README.md Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --- e2e/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/e2e/README.md b/e2e/README.md index c614ba42..aa9a5a6b 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -130,7 +130,7 @@ To solve the problem, you can set the `TESTCONTAINERS_RYUK_DISABLED` environment TESTCONTAINERS_RYUK_DISABLED=true ``` -To learn more about the behaivor of Ryuk from testcontainers, please read [their document here](https://golang.testcontainers.org/features/garbage_collector/#ryuk). +To learn more about the behavior of Ryuk from testcontainers, please read [their document here](https://golang.testcontainers.org/features/garbage_collector/#ryuk). ## Use compose containers cache to speed up the development cycle From 9b5c66923fb87eb8ba1b475619066310d30f3c35 Mon Sep 17 00:00:00 2001 From: Fang-Pen Lin Date: Mon, 12 Jan 2026 21:15:52 -0800 Subject: [PATCH 8/8] Update e2e/README.md Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --- e2e/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/e2e/README.md b/e2e/README.md index aa9a5a6b..103a5a6a 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -75,7 +75,7 @@ Currently, you need to enable the needed feature flags for running your tests ma If you don't know how to enable those feature flags, please contact the Infisical engineering team members; they will show you how to do that. In the future, we will provide a builder API to the `NewInfisicalService()` object for you to simply specify which features you would like to enable. -## Running the test +## Running the tests - Complete Example To run the e2e test, you can do the following: