Add runtests subcommand

[shoo]

As with many official tools, projects often require integration testing for CI/CD. In such cases, it is useful to run integration tests in a unified format.
The runtests subcommand I added is similar to test/run-unittest.sh or test/run-unittest.d so integration tests of dub projects can be run dub itself.
To test dub, run the following command:

dub run -- runtests --skip-build

[github-actions]

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 8

Total warnings: 0

Build statistics:

 statistics (-before, +after)
-executable size=5293368 bin/dub
-rough build time=62s
+executable size=5364928 bin/dub
+rough build time=63s

Full build output

DUB version 1.38.0, built on Jul  4 2024
LDC - the LLVM D compiler (1.39.0):
  based on DMD v2.109.1 and LLVM 18.1.6
  built with LDC - the LLVM D compiler (1.39.0)
  Default target: x86_64-unknown-linux-gnu
  Host CPU: znver3
  http://dlang.org - http://wiki.dlang.org/LDC


  Registered Targets:
    aarch64     - AArch64 (little endian)
    aarch64_32  - AArch64 (little endian ILP32)
    aarch64_be  - AArch64 (big endian)
    amdgcn      - AMD GCN GPUs
    arm         - ARM
    arm64       - ARM64 (little endian)
    arm64_32    - ARM64 (little endian ILP32)
    armeb       - ARM (big endian)
    avr         - Atmel AVR Microcontroller
    bpf         - BPF (host endian)
    bpfeb       - BPF (big endian)
    bpfel       - BPF (little endian)
    hexagon     - Hexagon
    lanai       - Lanai
    loongarch32 - 32-bit LoongArch
    loongarch64 - 64-bit LoongArch
    mips        - MIPS (32-bit big endian)
    mips64      - MIPS (64-bit big endian)
    mips64el    - MIPS (64-bit little endian)
    mipsel      - MIPS (32-bit little endian)
    msp430      - MSP430 [experimental]
    nvptx       - NVIDIA PTX 32-bit
    nvptx64     - NVIDIA PTX 64-bit
    ppc32       - PowerPC 32
    ppc32le     - PowerPC 32 LE
    ppc64       - PowerPC 64
    ppc64le     - PowerPC 64 LE
    r600        - AMD GPUs HD2XXX-HD6XXX
    riscv32     - 32-bit RISC-V
    riscv64     - 64-bit RISC-V
    sparc       - Sparc
    sparcel     - Sparc LE
    sparcv9     - Sparc V9
    spirv       - SPIR-V Logical
    spirv32     - SPIR-V 32-bit
    spirv64     - SPIR-V 64-bit
    systemz     - SystemZ
    thumb       - Thumb
    thumbeb     - Thumb (big endian)
    ve          - VE
    wasm32      - WebAssembly 32-bit
    wasm64      - WebAssembly 64-bit
    x86         - 32-bit X86: Pentium-Pro and above
    x86-64      - 64-bit X86: EM64T and AMD64
    xcore       - XCore
   Upgrading project in /home/runner/work/dub/dub/
    Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.39.0/x64/ldc2-1.39.0-linux-x86_64/bin/ldc2 for x86_64.
    Building dub 1.39.0-beta.1+commit.9.gc8c3cbb0: building configuration [application]
source/dub/internal/dyaml/composer.d(210,43): Deprecation: cannot access overlapped field `Event.implicit` with unsafe bit patterns in `@safe` code
source/dub/internal/dyaml/composer.d(232,43): Deprecation: cannot access overlapped field `Event.implicit` with unsafe bit patterns in `@safe` code
source/dub/internal/dyaml/composer.d(336,43): Deprecation: cannot access overlapped field `Event.implicit` with unsafe bit patterns in `@safe` code
source/dub/internal/dyaml/event.d(196,5): Deprecation: cannot access overlapped field `Event.explicitDocument` with unsafe bit patterns in `@safe` code
source/dub/internal/dyaml/event.d(214,5): Deprecation: cannot access overlapped field `Event.explicitDocument` with unsafe bit patterns in `@safe` code
source/dub/internal/dyaml/event.d(241,5): Deprecation: cannot access overlapped field `Event.implicit` with unsafe bit patterns in `@safe` code
source/dub/internal/dyaml/event.d(148,5): Deprecation: cannot access overlapped field `Event.implicit` with unsafe bit patterns in `@safe` code
source/dub/internal/dyaml/event.d(148,5): Deprecation: cannot access overlapped field `Event.implicit` with unsafe bit patterns in `@safe` code
     Linking dub
STAT:statistics (-before, +after)
STAT:executable size=5364928 bin/dub
STAT:rough build time=63s

[rikkimax]

Ideally these two would be merged together by looking for the json/sdl comment at start of file.

- `*.d` : The D language source code is executed by rdmd.
- `*.script.d` : Runs the D language source code as a file as a single-file DUB project.

[shoo]

Ideally these two would be merged together by looking for the json/sdl comment at start of file.

- `*.d` : The D language source code is executed by rdmd.
- `*.script.d` : Runs the D language source code as a file as a single-file DUB project.

Although not impossible, only the minimum necessary functions have been implemented in this PR.
This change would be able to be enhanced later without breaking changes.

Here is a list of others that I can think of that would be nice to have but have not yet been implemented:

• .fail_build / .no_build / .no_test / .no_run / .min_frontend / .testignore / --testdir These could be better configured, for example integrated into the project settings.
• When running tests, it would be useful to have an environment variable that contains the path to the binary file under test.
• A way to output the test results in CSV format, etc., for easier handling by other tools would be helpful.
• It would be useful to have an option to specify the directory where the coverage results are stored.
• An option to run each test in parallel may be useful to speed up the process.

If you can think of any very important enhancements, glitches, or future enhancements that might require breaking changes, please let me know.

[rikkimax]

Support directory based dub projects.

A directive to pick the directory by default on where the tests are instead of test or provided each time by the CLI argument.

[shoo]

Support directory based dub projects.

If you are talking about test cases, they are already implemented.
The added test test/runtests/test/test01 is a directory based dub project.

A directive to pick the directory by default on where the tests are instead of test or provided each time by the CLI argument.

It is difficult to detect with certainty which directory contains test cases.
For example, dub includes three directories (test, examples, scripts) with similar file characteristics, but it is difficult to mechanically identify which directory is for integration testing. Furthermore, if this were to be automatically deduced, it would be confusing to the user, unless it were deduced with fairly simple logic.

[rikkimax]

Support directory based dub projects.

If you are talking about test cases, they are already implemented. The added test test/runtests/test/test01 is a directory based dub project.

No, I'm talking about test/runtests/dub.sdl.

A directive to pick the directory by default on where the tests are instead of test or provided each time by the CLI argument.

It is difficult to detect with certainty which directory contains test cases. For example, dub includes three directories (test, examples, scripts) with similar file characteristics, but it is difficult to mechanically identify which directory is for integration testing. Furthermore, if this were to be automatically deduced, it would be confusing to the user, unless it were deduced with fairly simple logic.

I didn't suggest automatic detection. I suggested a directive in your dub file to specify the directory to use.

[shoo]

No, I'm talking about test/runtests/dub.sdl .

More information, please. Are you saying that you want to recognize the directory as being for the dub project without the dub.sdl?

I didn't suggest automatic detection. I suggested a directive in your dub file to specify the directory to use.

I see. However, I considered this but decided not to include it in this PR. I believe that adding dub project configuration directives with a higher priority than the default but lower than command line arguments can be added in a future update, not this time.

[rikkimax]

No, I'm talking about test/runtests/dub.sdl .

More information, please. Are you saying that you want to recognize the directory as being for the dub project without the dub.sdl?

Below the tests directory, below that is either files that you're running. I would like it to also support dub projects at that level.

It could be a misunderstanding on my part, but that changelog doesn't imply it.

[shoo]

Below the tests directory, below that is either files that you're running. I would like it to also support dub projects at that level.

It could be a misunderstanding on my part, but that changelog doesn't imply it.

When you run dub runtests at the root of a dub project, it will execute directory based dub projects contained within the test directory. This behavior should align with your expectations. However, in the case of the test/runtests directory, it is excluded from the tests because it contains a .no_build file.
It might be confusing, but the test case I created runs test/runtests.script.d with test/runtests as the current directory. This test executes the test cases in test/runtests/test/* by running the dub runtests command.

[Geod24]

First thing that comes to mind: We already have a special configuration for unittest, why not do that for integration tests ? Why a special command ?

[shoo]

First thing that comes to mind: We already have a special configuration for unittest, why not do that for integration tests ? Why a special command ?

The idea of having a special configuration selected when running runtests subcommand might be a good one. By having a directive to replace --testdir, such as allowing --config to switch test directories, we could achieve good synergy.
On the other hand, I still think a special subcommand is necessary. Just as there is a dub test subcommand instead of dub run --config unittest -- --DRT-testmode=test-only .