Skip to content

feat(selections): Add the ability to read integrity tag from selections#3026

Merged
Geod24 merged 1 commit intodlang:masterfrom
Geod24:mlang/IntegrityTag
May 14, 2025
Merged

feat(selections): Add the ability to read integrity tag from selections#3026
Geod24 merged 1 commit intodlang:masterfrom
Geod24:mlang/IntegrityTag

Conversation

@Geod24
Copy link
Member

@Geod24 Geod24 commented May 14, 2025

The dub.selections.json file can now contains integrity tags matching the SRI specifications, allowing dub (and other tools, e.g. Nix) to better validate that the downloaded archive matches the expected version.

However, Dub will not yet write the integrity tag, as it would result in a bad user experience. Since dub tries hard to reuse packages present on the filesystem, doing a dub upgrade could wipe the integrity tag (or not populate it) if the package is already present on the system, an issue which would manifest itself quite often for popular packages.

In order to solve this issue, we could store the integrity tag on disk, however this can be done in another PR as such package metadata would be useful for other purposes as well.

@github-actions
Copy link

github-actions bot commented May 14, 2025

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 0

Total warnings: 0

Build statistics:

 statistics (-before, +after)
-executable size=5055872 bin/dub
-rough build time=60s
+executable size=5105200 bin/dub
+rough build time=61s
Full build output
DUB version 1.39.0, built on Mar 20 2025
LDC - the LLVM D compiler (1.40.1):
  based on DMD v2.110.0 and LLVM 19.1.7
  built with LDC - the LLVM D compiler (1.40.1)
  Default target: x86_64-unknown-linux-gnu
  Host CPU: znver3
  http://dlang.org - http://wiki.dlang.org/LDC


  Registered Targets:
    aarch64     - AArch64 (little endian)
    aarch64_32  - AArch64 (little endian ILP32)
    aarch64_be  - AArch64 (big endian)
    amdgcn      - AMD GCN GPUs
    arm         - ARM
    arm64       - ARM64 (little endian)
    arm64_32    - ARM64 (little endian ILP32)
    armeb       - ARM (big endian)
    avr         - Atmel AVR Microcontroller
    bpf         - BPF (host endian)
    bpfeb       - BPF (big endian)
    bpfel       - BPF (little endian)
    hexagon     - Hexagon
    lanai       - Lanai
    loongarch32 - 32-bit LoongArch
    loongarch64 - 64-bit LoongArch
    mips        - MIPS (32-bit big endian)
    mips64      - MIPS (64-bit big endian)
    mips64el    - MIPS (64-bit little endian)
    mipsel      - MIPS (32-bit little endian)
    msp430      - MSP430 [experimental]
    nvptx       - NVIDIA PTX 32-bit
    nvptx64     - NVIDIA PTX 64-bit
    ppc32       - PowerPC 32
    ppc32le     - PowerPC 32 LE
    ppc64       - PowerPC 64
    ppc64le     - PowerPC 64 LE
    r600        - AMD GPUs HD2XXX-HD6XXX
    riscv32     - 32-bit RISC-V
    riscv64     - 64-bit RISC-V
    sparc       - Sparc
    sparcel     - Sparc LE
    sparcv9     - Sparc V9
    spirv       - SPIR-V Logical
    spirv32     - SPIR-V 32-bit
    spirv64     - SPIR-V 64-bit
    systemz     - SystemZ
    thumb       - Thumb
    thumbeb     - Thumb (big endian)
    ve          - VE
    wasm32      - WebAssembly 32-bit
    wasm64      - WebAssembly 64-bit
    x86         - 32-bit X86: Pentium-Pro and above
    x86-64      - 64-bit X86: EM64T and AMD64
    xcore       - XCore
    xtensa      - Xtensa 32
   Upgrading project in /home/runner/work/dub/dub/
    Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.40.1/x64/ldc2-1.40.1-linux-x86_64/bin/ldc2 for x86_64.
    Building dub 1.39.0-rc.1+commit.53.g781e55bb: building configuration [application]
     Linking dub
STAT:statistics (-before, +after)
STAT:executable size=5105200 bin/dub
STAT:rough build time=61s

@Geod24
Copy link
Member Author

Geod24 commented May 14, 2025

Testing (with the version (none) commented):

$ rm -rf ~/.dub/packages/vibe-d/
$ ./bin/dub upgrade
    Upgrading project in /home/mlang/projects/dlang/dub/
    Fetching vibe-d 0.10.2 (getting selected version)
$ git diff
-               "vibe-d": "0.10.1",
+               "vibe-d": {"integrity":"sha512-dzTTxl87eYMV3RyyaIak3PRBXd40c3IgwHI80gDGsCEDSdv9LHVeUy5vmoITpPmCtEjHQP5nDWrILnnqbyPkOA==","version":"0.10.2"},

Then I changed the selections file to have "version":"0.10.1" in order to trigger a mismatch.

$ ./bin/dub build
    Fetching vibe-d 0.10.1 (getting selected version)
Error Hash of downloaded package does not match integrity tag for vibe-d@0.10.1 - This can happen if the version has been re-tagged

I also attempted to build with an earlier dub:

 % dub upgrade
     Warning /home/mlang/projects/dlang/dub/dub.selections.json(12:13): versions[vibe-d].integrity: Key is not a valid member of this section. There are 3 valid keys: version, path, repository
   Upgrading project in /home/mlang/projects/dlang/dub/

This shows as a warning but doesn't impair functionality.

Copy link
Member

@CyberShadow CyberShadow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! Thank you very much for working on this!

Approving as even just the ability to understand and verify hashes is useful on its own - with this, we could start adding them to dub.selections.json manually or with third-party tools.

Code LGTM beyond the obvious syntax error.

@Geod24
Copy link
Member Author

Geod24 commented May 14, 2025

Another example while building:

diff --git a/dub.selections.json b/dub.selections.json
index 1732d821..299bf252 100644
--- a/dub.selections.json
+++ b/dub.selections.json
@@ -9,8 +9,8 @@
                "stdx-allocator": "2.77.5",
                "taggedalgebraic": "0.11.23",
                "vibe-container": "1.4.0",
-               "vibe-core": "2.9.6",
-               "vibe-d": "0.10.1",
+               "vibe-core": {"integrity":"sha512-srDYpMJhpw0SMbDhGttu7RrLjiCXsYOZJOh8wFyxU/rOXO7Qydt+4hkG/rMJfQki3o3YnDZOMY9naXeAZutRdw==","version":"2.9.6"},
+               "vibe-d": {"integrity":"sha512-vwQ9tYTjLb981j41+3GZZUgKXm/5PlKpmY2bplRSUM8ajL03++LGm/TcfFFarJrHex8CTb5ZLWdiY1fFAOSkSw==","version":"0.10.1"},
                "vibe-http": "1.2.1",
                "vibe-inet": "1.1.0",
                "vibe-serialization": "1.0.7",
% wget -q -O - https://github.com/vibe-d/vibe.d/archive/refs/tags/v0.10.1.zip | openssl dgst -binary -sha512 | base64
vwQ9tYTjLb981j41+3GZZUgKXm/5PlKpmY2bplRSUM8ajL03++LGm/TcfFFarJrHex8CTb5ZLWdi
Y1fFAOSkSw==

The `dub.selections.json` file can now contains integrity tags matching the
SRI specifications, allowing dub (and other tools, e.g. Nix) to better validate
that the downloaded archive matches the expected version.

However, Dub will not yet write the integrity tag, as it would result in a bad
user experience. Since `dub` tries hard to reuse packages present on the
filesystem, doing a `dub upgrade` could wipe the integrity tag (or not populate it)
if the package is already present on the system, an issue which would manifest itself
quite often for popular packages.

In order to solve this issue, we could store the integrity tag on disk,
however this can be done in another PR as such package metadata would be
useful for other purposes as well.
@Geod24 Geod24 force-pushed the mlang/IntegrityTag branch from b7d198f to 1260495 Compare May 14, 2025 15:10
@Geod24
Copy link
Member Author

Geod24 commented May 14, 2025

Regarding writing them: We could also cheat and rely on the index having them (#3023).
But I've wanted to have dub gc for a while now and it would most likely require package metadata.

@Geod24 Geod24 merged commit b4a73be into dlang:master May 14, 2025
32 checks passed
@Geod24 Geod24 deleted the mlang/IntegrityTag branch May 14, 2025 15:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants