Skip to content

[DX-3383] Optimize Docker Caching#21705

Open
kalverra wants to merge 29 commits intodevelopfrom
moreDockerCacheFix
Open

[DX-3383] Optimize Docker Caching#21705
kalverra wants to merge 29 commits intodevelopfrom
moreDockerCacheFix

Conversation

@kalverra
Copy link
Copy Markdown
Collaborator

@kalverra kalverra commented Mar 25, 2026

.github PR: smartcontractkit/.github#1487

Optimizes Docker builds in CI by

  • Refactoring Dockerfile for more parallel execution
  • Fixes some caching bugs
  • Introduces smarter caching along with Dockerfile refactor
Approach Typical Case Worst Case
Old (CI) ~5m ~5m
New (CI) ~2m30s ~5m
Old (local) 2s 2m36s
New (local) 500ms 3m34s
image

This also updates some old actions versions, so there's some noise there. If reviewing, only focus on the Dockerfile changes.

Image Verification

Since we mess with the Dockerfile and a few build steps, I wanted to verify the image is effectively the same. We do so in this PR: #21731

Original Build

.dkr.ecr..amazonaws.com/chainlink-integration-tests:f64dc28d6c19749e5a4ba7b7f8233933ffe7a733
##[endgroup]
##[group]Installed binaries
total 1.3G
-rwxr-xr-x 1 root root 256M Mar 26 18:37 chainlink
-rwxr-xr-x 1 root root 130M Mar 26 18:35 chainlink-evm
-rwxr-xr-x 1 root root 126M Mar 26 18:35 chainlink-ocr3-capability
-rwxr-xr-x 1 root root 64M Mar 26 18:36 chainlink-cosmos
-rwxr-xr-x 1 root root 47M Mar 26 18:35 chainlink-solana
-rwxr-xr-x 1 root root 42M Mar 26 18:36 chainlink-starknet
-rwxr-xr-x 1 root root 37M Mar 26 18:37 evm
-rwxr-xr-x 1 root root 36M Mar 26 18:35 chainlink-sui
-rwxr-xr-x 1 root root 36M Mar 26 18:35 chainlink-aptos
-rwxr-xr-x 1 root root 35M Mar 26 18:36 chainlink-ton
-rwxr-xr-x 1 root root 35M Mar 26 18:36 solana
-rwxr-xr-x 1 root root 35M Mar 26 18:35 chainlink-tron
-rwxr-xr-x 1 root root 33M Mar 26 18:34 log-event-trigger
-rwxr-xr-x 1 root root 33M Mar 26 18:35 chainlink-medianpoc
-rwxr-xr-x 1 root root 33M Mar 26 18:37 cron
-rwxr-xr-x 1 root root 32M Mar 26 18:36 http_trigger
-rwxr-xr-x 1 root root 32M Mar 26 18:36 readcontract
-rwxr-xr-x 1 root root 32M Mar 26 18:36 consensus
-rwxr-xr-x 1 root root 32M Mar 26 18:36 confidential-http
-rwxr-xr-x 1 root root 32M Mar 26 18:36 http_action
-rwxr-xr-x 1 root root 32M Mar 26 18:36 mock
-rwxr-xr-x 1 root root 31M Mar 26 18:35 chainlink-mercury
-rwxr-xr-x 1 root root 26M Mar 26 18:35 chainlink-feeds
-rwxr-xr-x 1 root root 22M Mar 26 18:34 dlv
drwxr-xr-x 1 root root 60 Mar 26 18:37 .
drwxr-xr-x 1 root root 60 Feb 17 02:02 ..
##[endgroup]
##[group]Binary checksums (SHA256)
f48f2cc8346012cafbf2e5edc6bc74cf40661776f5c9ce18dafb47037605e82b /usr/local/bin/chainlink
a3d2507c69b1392380f88287a065b71ddcee3419f63ef419788c97d3189890fb /usr/local/bin/chainlink-aptos
d13ddc5e6f78b257bdf2a67f708883cb64e2121c5e9ae129bdd72d2ff935fc32 /usr/local/bin/chainlink-cosmos
461e6baf847e9bb9a007ac741da96f3ee669ee67d261014eca2e6e857fa5c856 /usr/local/bin/chainlink-evm
4931c54d8c831faa9f32810091bf19cf64a7f8b98f9476cf8b67fd84765efa33 /usr/local/bin/chainlink-feeds
5db0d73d4918f5ba1a66fc51b98b28f6a6ce8ef78e32fc0d8079afd381e0ad00 /usr/local/bin/chainlink-medianpoc
8efb5db3f494347f58d8e1704792bfa2e4298e6c26ae8bdc691bc6fc148f9b4d /usr/local/bin/chainlink-mercury
a1c13920e4235dae025bf18fbfb1b0b60840add4fe24ac45f93a52b913896f04 /usr/local/bin/chainlink-ocr3-capability
b7ab5c46546877a198ed5703553428351228b15f26a1662ae7ea84eddd402519 /usr/local/bin/chainlink-solana
7453c575a675bd088e0c6d842b66941309150f10ebdd69db3c0bd3b35bcde9e4 /usr/local/bin/chainlink-starknet
7502690a98accb07e6694ea24887bd9636b8af72adaf0df589d806e927c0f237 /usr/local/bin/chainlink-sui
931326d3e076c8f1b6820e1ee8cbd21492872c05099a6326125ebba3d86927c9 /usr/local/bin/chainlink-ton
25b1373324601c9dfbe306a45bf9c8e3deaf8586fd8e7f868e6b2db97f9ff74b /usr/local/bin/chainlink-tron
7e88882052fffd113e4080c86bd477ffd7509b4fd13e28728d7569693710444e /usr/local/bin/confidential-http
5dfe0a96995fb96057daa28a27d548e4192de77aa251b11847a494e4e587aee3 /usr/local/bin/consensus
c5ad412e336e49b261b0b5ae3a2f322964b1f2cb1f06106203d4d273fbf0a28c /usr/local/bin/cron
91803e942a02dda6be2d611ecadfa38592805948af3cbe4c0252ff3ad32e5b2a /usr/local/bin/dlv
e38e0e0d6522376957e1d727a1ef7c71fb00407b8f2f357c385f698a9d681d47 /usr/local/bin/evm
c6dc9c7c89c90671e543b48a08e92970b32cfe52155575ec27fe69a4304ac42a /usr/local/bin/http_action
a3d00711ce995c360084fd9d6f2bbec13a44734d085dc8d0b487ad85c3ab6a74 /usr/local/bin/http_trigger
7f7e9330298149c5e2b430f93522a3f24c000262f4f0bc4b4e9450840a6dca4b /usr/local/bin/log-event-trigger
360887df6c553227981ec02dc83614fde3dc690277582edb7091fdd53dd9f776 /usr/local/bin/mock
1aa329f128a3a4d54160f6d2483e96d1cb86377c74edf0ceea4e18b2f0e528e9 /usr/local/bin/readcontract
e26ced2ec6ee5aec75fa2ed7404f199134c309ea9ef8dc6ea60012eb6afc19ad /usr/local/bin/solana
##[endgroup]
##[group]Shared libraries
No shared libraries found
##[endgroup]
##[group]Chainlink version
chainlink version 2.40.0@f64dc28d6c19749e5a4ba7b7f8233933ffe7a733
##[endgroup]
##[group]Environment variables
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBIAN_FRONTEND=noninteractive
CL_DOCKER_TAG=f64dc28d6c19749e5a4ba7b7f8233933ffe7a733
CL_SOLANA_CMD=chainlink-solana
CL_MEDIAN_CMD=
CL_EVM_CMD=
CL_CHAIN_DEFAULTS=
XDG_CACHE_HOME=/home/chainlink/.cache
GOCOVERDIR=/var/tmp/go-coverage

##[endgroup]
##[group]Image config
{
"Hostname": "",
"Domainname": "",
"User": "chainlink",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"6688/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"DEBIAN_FRONTEND=noninteractive",
"CL_DOCKER_TAG=f64dc28d6c19749e5a4ba7b7f8233933ffe7a733",
"CL_SOLANA_CMD=chainlink-solana",
"CL_MEDIAN_CMD=",
"CL_EVM_CMD=",
"CL_CHAIN_DEFAULTS=",
"XDG_CACHE_HOME=/home/chainlink/.cache",
"GOCOVERDIR=/var/tmp/go-coverage"
],
"Cmd": [
"local",
"node"
],
"Healthcheck": {
"Test": [
"CMD-SHELL",
"curl -f http://localhost:6688/health || exit 1"
]
},
"ArgsEscaped": true,
"Image": "",
"Volumes": null,
"WorkingDir": "/home/chainlink",
"Entrypoint": [
"chainlink"
],
"OnBuild": null,
"Labels": {
"org.opencontainers.image.created": "2026-03-26T18:34:07.933Z",
"org.opencontainers.image.description": "node of the decentralized oracle network, bridging on and off-chain computation",
"org.opencontainers.image.licenses": "NOASSERTION",
"org.opencontainers.image.ref.name": "ubuntu",
"org.opencontainers.image.revision": "f64dc28d6c19749e5a4ba7b7f8233933ffe7a733",
"org.opencontainers.image.source": "https://github.com/smartcontractkit/chainlink",
"org.opencontainers.image.title": "chainlink",
"org.opencontainers.image.url": "https://github.com/smartcontractkit/chainlink",
"org.opencontainers.image.version": "f64dc28d6c19749e5a4ba7b7f8233933ffe7a733"
}
}
##[endgroup]

New Image

Digest: sha256:bd255c3012ae9e8530675c7d8d0aff839667c2a503a2bd92373b5cd1e5e346d2
Status: Downloaded newer image for .dkr.ecr..amazonaws.com/chainlink-integration-tests:cb63350bf5f58052409208b3c555dd9385129916
.dkr.ecr..amazonaws.com/chainlink-integration-tests:cb63350bf5f58052409208b3c555dd9385129916
##[endgroup]
##[group]Installed binaries
total 1.3G
-rwxr-xr-x 1 root root 256M Mar 26 18:53 chainlink
-rwxr-xr-x 1 root root 130M Mar 26 18:52 chainlink-evm
-rwxr-xr-x 1 root root 126M Mar 26 18:52 chainlink-ocr3-capability
-rwxr-xr-x 1 root root 64M Mar 26 18:52 chainlink-cosmos
-rwxr-xr-x 1 root root 47M Mar 26 18:52 chainlink-solana
-rwxr-xr-x 1 root root 42M Mar 26 18:52 chainlink-starknet
-rwxr-xr-x 1 root root 37M Mar 26 18:53 evm
-rwxr-xr-x 1 root root 36M Mar 26 18:52 chainlink-sui
-rwxr-xr-x 1 root root 36M Mar 26 18:52 chainlink-aptos
-rwxr-xr-x 1 root root 35M Mar 26 18:52 chainlink-ton
-rwxr-xr-x 1 root root 35M Mar 26 18:53 solana
-rwxr-xr-x 1 root root 35M Mar 26 18:52 chainlink-tron
-rwxr-xr-x 1 root root 33M Mar 26 18:52 log-event-trigger
-rwxr-xr-x 1 root root 33M Mar 26 18:52 chainlink-medianpoc
-rwxr-xr-x 1 root root 33M Mar 26 18:53 cron
-rwxr-xr-x 1 root root 32M Mar 26 18:53 http_trigger
-rwxr-xr-x 1 root root 32M Mar 26 18:53 readcontract
-rwxr-xr-x 1 root root 32M Mar 26 18:53 consensus
-rwxr-xr-x 1 root root 32M Mar 26 18:53 confidential-http
-rwxr-xr-x 1 root root 32M Mar 26 18:53 http_action
-rwxr-xr-x 1 root root 32M Mar 26 18:53 mock
-rwxr-xr-x 1 root root 31M Mar 26 18:52 chainlink-mercury
-rwxr-xr-x 1 root root 26M Mar 26 18:52 chainlink-feeds
-rwxr-xr-x 1 root root 22M Mar 26 18:51 dlv
drwxr-xr-x 1 root root 60 Mar 26 18:53 .
drwxr-xr-x 1 root root 60 Feb 17 02:02 ..
##[endgroup]
##[group]Binary checksums (SHA256)
fce028e4204f11afea00adaa51b664c7e8571cf0c7f78eaf47bd7dd6992db997 /usr/local/bin/chainlink
a3d2507c69b1392380f88287a065b71ddcee3419f63ef419788c97d3189890fb /usr/local/bin/chainlink-aptos
d13ddc5e6f78b257bdf2a67f708883cb64e2121c5e9ae129bdd72d2ff935fc32 /usr/local/bin/chainlink-cosmos
461e6baf847e9bb9a007ac741da96f3ee669ee67d261014eca2e6e857fa5c856 /usr/local/bin/chainlink-evm
4931c54d8c831faa9f32810091bf19cf64a7f8b98f9476cf8b67fd84765efa33 /usr/local/bin/chainlink-feeds
5db0d73d4918f5ba1a66fc51b98b28f6a6ce8ef78e32fc0d8079afd381e0ad00 /usr/local/bin/chainlink-medianpoc
8efb5db3f494347f58d8e1704792bfa2e4298e6c26ae8bdc691bc6fc148f9b4d /usr/local/bin/chainlink-mercury
a1c13920e4235dae025bf18fbfb1b0b60840add4fe24ac45f93a52b913896f04 /usr/local/bin/chainlink-ocr3-capability
b7ab5c46546877a198ed5703553428351228b15f26a1662ae7ea84eddd402519 /usr/local/bin/chainlink-solana
7453c575a675bd088e0c6d842b66941309150f10ebdd69db3c0bd3b35bcde9e4 /usr/local/bin/chainlink-starknet
7502690a98accb07e6694ea24887bd9636b8af72adaf0df589d806e927c0f237 /usr/local/bin/chainlink-sui
931326d3e076c8f1b6820e1ee8cbd21492872c05099a6326125ebba3d86927c9 /usr/local/bin/chainlink-ton
25b1373324601c9dfbe306a45bf9c8e3deaf8586fd8e7f868e6b2db97f9ff74b /usr/local/bin/chainlink-tron
7e88882052fffd113e4080c86bd477ffd7509b4fd13e28728d7569693710444e /usr/local/bin/confidential-http
5dfe0a96995fb96057daa28a27d548e4192de77aa251b11847a494e4e587aee3 /usr/local/bin/consensus
c5ad412e336e49b261b0b5ae3a2f322964b1f2cb1f06106203d4d273fbf0a28c /usr/local/bin/cron
91803e942a02dda6be2d611ecadfa38592805948af3cbe4c0252ff3ad32e5b2a /usr/local/bin/dlv
e38e0e0d6522376957e1d727a1ef7c71fb00407b8f2f357c385f698a9d681d47 /usr/local/bin/evm
c6dc9c7c89c90671e543b48a08e92970b32cfe52155575ec27fe69a4304ac42a /usr/local/bin/http_action
a3d00711ce995c360084fd9d6f2bbec13a44734d085dc8d0b487ad85c3ab6a74 /usr/local/bin/http_trigger
7f7e9330298149c5e2b430f93522a3f24c000262f4f0bc4b4e9450840a6dca4b /usr/local/bin/log-event-trigger
360887df6c553227981ec02dc83614fde3dc690277582edb7091fdd53dd9f776 /usr/local/bin/mock
1aa329f128a3a4d54160f6d2483e96d1cb86377c74edf0ceea4e18b2f0e528e9 /usr/local/bin/readcontract
e26ced2ec6ee5aec75fa2ed7404f199134c309ea9ef8dc6ea60012eb6afc19ad /usr/local/bin/solana
##[endgroup]
##[group]Shared libraries
No shared libraries found
##[endgroup]
##[group]Chainlink version
chainlink version 2.40.0@cb63350bf5f58052409208b3c555dd9385129916
##[endgroup]
##[group]Environment variables
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBIAN_FRONTEND=noninteractive
CL_DOCKER_TAG=cb63350bf5f58052409208b3c555dd9385129916
CL_SOLANA_CMD=chainlink-solana
CL_MEDIAN_CMD=
CL_EVM_CMD=
CL_CHAIN_DEFAULTS=
XDG_CACHE_HOME=/home/chainlink/.cache
GOCOVERDIR=/var/tmp/go-coverage

##[endgroup]
##[group]Image config
{
"Hostname": "",
"Domainname": "",
"User": "chainlink",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"6688/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"DEBIAN_FRONTEND=noninteractive",
"CL_DOCKER_TAG=cb63350bf5f58052409208b3c555dd9385129916",
"CL_SOLANA_CMD=chainlink-solana",
"CL_MEDIAN_CMD=",
"CL_EVM_CMD=",
"CL_CHAIN_DEFAULTS=",
"XDG_CACHE_HOME=/home/chainlink/.cache",
"GOCOVERDIR=/var/tmp/go-coverage"
],
"Cmd": [
"local",
"node"
],
"Healthcheck": {
"Test": [
"CMD-SHELL",
"curl -f http://localhost:6688/health || exit 1"
]
},
"ArgsEscaped": true,
"Image": "",
"Volumes": null,
"WorkingDir": "/home/chainlink",
"Entrypoint": [
"chainlink"
],
"OnBuild": null,
"Labels": {
"org.opencontainers.image.created": "2026-03-26T18:50:58.810Z",
"org.opencontainers.image.description": "node of the decentralized oracle network, bridging on and off-chain computation",
"org.opencontainers.image.licenses": "NOASSERTION",
"org.opencontainers.image.ref.name": "ubuntu",
"org.opencontainers.image.revision": "cb63350bf5f58052409208b3c555dd9385129916",
"org.opencontainers.image.source": "https://github.com/smartcontractkit/chainlink",
"org.opencontainers.image.title": "chainlink",
"org.opencontainers.image.url": "https://github.com/smartcontractkit/chainlink",
"org.opencontainers.image.version": "cb63350bf5f58052409208b3c555dd9385129916"
}
}
##[endgroup]

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 25, 2026

✅ No conflicts with other open PRs targeting develop

@trunk-io
Copy link
Copy Markdown

trunk-io bot commented Mar 25, 2026

Static BadgeStatic BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

@kalverra kalverra mentioned this pull request Mar 26, 2026
@kalverra kalverra marked this pull request as ready for review March 26, 2026 19:23
@kalverra kalverra requested review from a team as code owners March 26, 2026 19:23
Copilot AI review requested due to automatic review settings March 26, 2026 19:24
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Risk Rating: HIGH

Optimizes CI Docker image build performance by restructuring Chainlink Dockerfiles to improve layer cache reuse and by adjusting workflow/action configuration to use updated actions and (for integration image builds) scoped caching.

Changes:

  • Refactor core/chainlink.Dockerfile and plugins/chainlink.Dockerfile into multiple stages (deps-base, deps, build-remote-plugins, build-local-plugins, build-chainlink, build-delve) to increase parallelism and cache stability.
  • Introduce GO_OVERRIDE_DEPS-driven dependency override behavior during Docker builds.
  • Update multiple GitHub workflows/actions (notably actions/checkout and actions/upload-artifact major versions) and add .dockerignore entries to reduce build context.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
plugins/chainlink.Dockerfile Split build into cache-friendly stages; add dependency override logic; copy outputs from parallel stages into final image.
core/chainlink.Dockerfile Same multi-stage refactor for production Docker image; add override logic and parallel artifact copying.
.github/workflows/integration-tests.yml Update checkout version; pin ctf-build-image action commit and add cache-scope wiring.
.github/workflows/* (many) Bump actions/checkout to v6 and actions/upload-artifact to v7 across workflows; minor formatting/spelling fixes.
.github/actions/* Update composite actions to use newer checkout/upload-artifact versions.
.dockerignore Exclude additional directories (e.g., .github/, docs/, fuzz/) from Docker build context.

Areas requiring scrupulous human review:

  • The new GO_OVERRIDE_DEPS handling in both Dockerfiles (it modifies module files during the build and is currently placed before the source tree is present).
  • The correctness of the “remote plugins w/o full source tree” stage assumption (i.e., that make install-plugins-* continues to work with only the copied manifests/scripts + GNUmakefile + go.mod/go.sum).
  • Final image assembly order (multiple COPY --from=... /gobins/ /usr/local/bin/) to ensure no unexpected binary overwrites.

Suggested reviewers (per .github/CODEOWNERS):

  • For Dockerfile + CI workflow changes: @smartcontractkit/devex-cicd, @smartcontractkit/core, @smartcontractkit/foundations
  • For broader .github/** workflow updates: @smartcontractkit/devex-tooling in addition to the above

pavel-raykov
pavel-raykov previously approved these changes Mar 26, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated 4 comments.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 2 comments.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 4 comments.

@cl-sonarqube-production
Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants