migrate to moby modules by thaJeztah · Pull Request #13078 · docker/compose

thaJeztah · 2025-07-21T18:51:11Z

depends on pkg/compose: align classic builder implementation with docker/cli #13278
testing Create github.com/moby/moby/api and github.com/moby/moby/client module moby/moby#50280
testing migrate to moby/moby/api and moby/moby/client modules cli#6196
testing vendor: github.com/moby/moby/api v1.52.0, moby/client v0.1.0 buildx#3326

What I did

Related issue

(not mandatory) A picture of a cute animal, if possible in relation to what you did

ndeloof · 2025-07-21T18:56:05Z

Nice to see moby API as a dedicated module, so we don't get transitive dependencies from engine inside Compose ! ❤️

cmd/compose/bridge.go

thaJeztah · 2025-07-21T18:58:32Z

pkg/compose/internal/urlutil/urlutil.go

@@ -0,0 +1,89 @@
+// Package urlutil provides helper function to check if a given build-context


Added this fork temporarily; didn't realise compose itself also uses it (not only through cli); the CLI fork is "internal", so probably needs to made public, or we keep a fork here 😞

thaJeztah · 2025-07-21T18:58:58Z

Remaining uses of docker/docker;

tree -d vendor/github.com/docker/docker
vendor/github.com/docker/docker
├── internal
│   └── lazyregexp
├── pkg
│   ├── jsonmessage
│   ├── namesgenerator
│   ├── pidfile
│   ├── process
│   ├── progress
│   ├── stdcopy
│   ├── streamformatter
│   └── stringid
└── registry

13 directories

thaJeztah · 2025-07-21T19:00:26Z

Build failure looks like it may be an incompatible change in buildx itself?

 > [build 1/1] RUN --mount=type=bind,target=.     --mount=type=cache,target=/root/.cache     --mount=type=cache,target=/go/pkg/mod     --mount=type=bind,from=osxcross,src=/osxsdk,target=/xx-sdk     xx-go --wrap &&     if [ "$(xx-info os)" == "darwin" ]; then export CGO_ENABLED=1; fi &&     make build GO_BUILDTAGS="e2e" DESTDIR=/out &&     xx-verify --static /out/docker-compose:
1.027 GO111MODULE=on go build  -trimpath -tags "e2e" -ldflags "-w -X github.com/docker/compose/v2/internal.Version=fa50b46" -o "/out/docker-compose" ./cmd
38.90 # github.com/docker/buildx/build
38.90 /go/pkg/mod/github.com/tha!jeztah/buildx@v0.2.1-0.20250721183044-80b40a761230/build/provenance.go:135:16: pred1.ConvertToSLSA02 undefined (type *"github.com/moby/buildkit/solver/llbsolver/provenance/types".ProvenancePredicateSLSA1 has no field or method ConvertToSLSA02)
67.83 make: *** [Makefile:61: build] Error 1

38.90 /go/pkg/mod/github.com/tha!jeztah/buildx@v0.2.1-0.20250721183044-80b40a761230/build/provenance.go:135:16: pred1.ConvertToSLSA02 undefined (type

thaJeztah · 2025-07-21T19:10:48Z

OK, it's because buildx depends on an unreleased version of BuildKit, so go modules considers it to be older than the latest release and won't update; https://github.com/docker/buildx/blob/3f4bf829d8c5b92a8f670609417d9aa4c0b6be98/go.mod#L32

github.com/moby/buildkit v0.23.0-rc1.0.20250618182037-9b91d20367db // master

ndeloof · 2025-07-22T06:04:18Z

jsonmessage
this one should be part of the API, don't you think ?

thaJeztah · 2025-07-22T06:09:46Z

Yes, or at least the type; the utilities could be "client" - those packages are messy.

Also looking at things that are in the api, but shouldn't (api/types/plugins/logdriver)

thaJeztah · 2025-07-23T20:32:59Z

Also included a (WIP) branch for docker/cli#6202

Before/After;

Details

Before:

tree -d vendor/github.com/docker/cli
vendor/github.com/docker/cli
├── cli
│   ├── command
│   │   ├── completion
│   │   ├── container
│   │   ├── formatter
│   │   │   └── tabwriter
│   │   ├── image
│   │   │   └── build
│   │   │       └── internal
│   │   │           ├── git
│   │   │           └── urlutil
│   │   └── inspect
│   ├── compose
│   │   ├── interpolation
│   │   ├── loader
│   │   ├── schema
│   │   │   └── data
│   │   ├── template
│   │   └── types
│   ├── config
│   │   ├── configfile
│   │   ├── credentials
│   │   ├── memorystore
│   │   └── types
│   ├── connhelper
│   │   ├── commandconn
│   │   ├── internal
│   │   │   └── syntax
│   │   └── ssh
│   ├── context
│   │   ├── docker
│   │   └── store
│   ├── debug
│   ├── flags
│   ├── hints
│   ├── streams
│   ├── trust
│   └── version
├── cli-plugins
│   ├── hooks
│   ├── manager
│   ├── metadata
│   ├── plugin
│   └── socket
├── internal
│   ├── jsonstream
│   ├── lazyregexp
│   ├── prompt
│   └── tui
├── opts
│   └── swarmopts
├── pkg
│   └── kvfile
└── templates

55 directories

After:

tree -d vendor/github.com/docker/cli
vendor/github.com/docker/cli
├── cli
│   ├── command
│   │   ├── completion
│   │   ├── container
│   │   ├── formatter
│   │   │   └── tabwriter
│   │   ├── image
│   │   │   └── build
│   │   │       └── internal
│   │   │           ├── git
│   │   │           └── urlutil
│   │   └── inspect
│   ├── config
│   │   ├── configfile
│   │   ├── credentials
│   │   ├── memorystore
│   │   └── types
│   ├── connhelper
│   │   ├── commandconn
│   │   ├── internal
│   │   │   └── syntax
│   │   └── ssh
│   ├── context
│   │   ├── docker
│   │   └── store
│   ├── debug
│   ├── flags
│   ├── hints
│   ├── streams
│   ├── trust
│   └── version
├── cli-plugins
│   ├── hooks
│   ├── manager
│   ├── metadata
│   ├── plugin
│   └── socket
├── internal
│   ├── jsonstream
│   ├── lazyregexp
│   ├── prompt
│   ├── tui
│   └── volumespec
├── opts
├── pkg
│   └── kvfile
└── templates

48 directories

thaJeztah · 2025-08-05T22:52:03Z

Down to two packages; both possibly candidates for github.com/moby/sys;

tree -d vendor/github.com/docker/docker
vendor/github.com/docker/docker
└── pkg
    ├── pidfile
    └── process

4 directories

CLI still needs sorting out;

Details

tree -d vendor/github.com/docker/cli
vendor/github.com/docker/cli
├── cli
│   ├── command
│   │   ├── completion
│   │   ├── container
│   │   ├── formatter
│   │   │   └── tabwriter
│   │   ├── image
│   │   │   └── build
│   │   │       └── internal
│   │   │           ├── git
│   │   │           └── urlutil
│   │   ├── inspect
│   │   └── system
│   │       └── pruner
│   ├── config
│   │   ├── configfile
│   │   ├── credentials
│   │   ├── memorystore
│   │   └── types
│   ├── connhelper
│   │   ├── commandconn
│   │   ├── internal
│   │   │   └── syntax
│   │   └── ssh
│   ├── context
│   │   ├── docker
│   │   └── store
│   ├── debug
│   ├── flags
│   ├── hints
│   ├── streams
│   ├── trust
│   └── version
├── cli-plugins
│   ├── hooks
│   ├── manager
│   ├── metadata
│   ├── plugin
│   └── socket
├── internal
│   ├── jsonstream
│   ├── lazyregexp
│   ├── prompt
│   ├── registry
│   ├── tui
│   └── volumespec
├── opts
├── pkg
│   └── kvfile
└── templates

51 directories

thaJeztah · 2026-01-22T10:21:20Z

cmd/compose/compose.go

-	// filter out useless commandConn.CloseWrite warning message that can occur
-	// when using a remote context that is unreachable: "commandConn.CloseWrite: commandconn: failed to wait: signal: killed"
-	// https://github.com/docker/cli/blob/e1f24d3c93df6752d3c27c8d61d18260f141310c/cli/connhelper/commandconn/commandconn.go#L203-L215
-	logrus.AddHook(logutil.NewFilter([]logrus.Level{


These were removed in docker/cli#6644

thaJeztah · 2026-01-22T10:42:12Z

pkg/compose/push.go

-		text = jm.Progress.String()
+		// FIXME(thaJeztah): what's the replacement for Progress.String()?
+		// text = jm.Progress.String()


This one probably still needs to be looked into.

@ndeloof looks like the data is used in different fields for toPushProgressEvent than in toPullProgressEvent - slightly confused which data should go in which field;

In toPullProgressEvent, we set the Details field, and Text is set to jm.Status, but in toPushEvent, we don't use the Details field;

-func toPullProgressEvent(parent string, jm jsonstream.Message, events api.EventProcessor) { - if jm.ID == "" || jm.Progress == nil { +func toPushProgressEvent(prefix string, jm jsonstream.Message, events api.EventProcessor) { + if jm.ID == "" { + // skipped return } - var ( - details string + text string + status = api.Working total int64 - percent int current int64 - status = api.Working + percent int ) - - switch jm.Status { - case PreparingPhase, WaitingPhase, PullingFsPhase: - percent = 0 - case DownloadingPhase, ExtractingPhase, VerifyingChecksumPhase: - if jm.Progress != nil { + if isDone(jm) { + status = api.Done + percent = 100 + } + if jm.Error != nil { + status = api.Error + text = jm.Error.Message + } + if jm.Progress != nil { + // FIXME(thaJeztah): what's the replacement for Progress.String()? + // text = jm.Progress.String() + if jm.Progress.Total != 0 { current = jm.Progress.Current total = jm.Progress.Total if jm.Progress.Total > 0 { @@ -436,32 +161,25 @@ func toPullProgressEvent(parent string, jm jsonstream.Message, events api.EventP } } } - case DownloadCompletePhase, AlreadyExistsPhase, PullCompletePhase: - status = api.Done - percent = 100 - } - - if strings.Contains(jm.Status, "Image is up to date") || - strings.Contains(jm.Status, "Downloaded newer image") { - status = api.Done - percent = 100 - } - - if jm.Error != nil { - status = api.Error - details = jm.Error.Message - } else { - details = units.HumanSize(float64(jm.Progress.Current)) } events.On(api.Resource{ + ParentID: prefix, ID: jm.ID, - ParentID: parent, + Text: text, + Status: status, Current: current, Total: total, Percent: percent, - Status: status, - Text: jm.Status, - Details: details, }) }

thaJeztah · 2026-01-22T10:43:59Z

pkg/compose/pull.go

-	progress = jm.Progress.String()
+	// FIXME(thaJeztah): what's the replacement for Progress.String()?
+	// progress = jm.Progress.String()


Same for this one

codecov · 2026-01-26T08:56:27Z

Codecov Report

❌ Patch coverage is 48.85599% with 380 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pkg/dryrun/dryrunclient.go	0.00%	218 Missing ⚠️
pkg/compose/create.go	66.66%	38 Missing and 23 partials ⚠️
cmd/compose/list.go	0.00%	15 Missing and 2 partials ⚠️
pkg/bridge/transformers.go	17.64%	13 Missing and 1 partial ⚠️
pkg/compose/cp.go	56.66%	13 Missing ⚠️
pkg/compose/generate.go	0.00%	13 Missing ⚠️
pkg/compose/pull.go	54.54%	8 Missing and 2 partials ⚠️
pkg/compose/events.go	0.00%	5 Missing ⚠️
pkg/compose/watch.go	80.00%	5 Missing ⚠️
cmd/compose/stats.go	0.00%	4 Missing ⚠️
... and 12 more

📢 Thoughts on this report? Let us know!

thaJeztah · 2026-02-04T14:28:49Z

pkg/compose/create_test.go

 func TestDefaultNetworkSettings(t *testing.T) {
 	t.Run("returns the network with the highest priority when service has multiple networks", func(t *testing.T) {
+		t.Skip("FIXME: test is failing (but for legacy API versions?)")
 		service := composetypes.ServiceConfig{


I put this as a separate commit, but this test can probably be removed, as I for legacy API versions ("1.43"), which the client no longer supports; https://github.com/moby/moby/blob/client/v0.2.2/client/client.go#L112-L114

I forgot what the exact failure was, so temporarily enabled it again;

=== Failed === FAIL: pkg/compose TestDefaultNetworkSettings/returns_the_network_with_the_highest_priority_when_service_has_multiple_networks (0.00s) create_test.go:225: assertion failed: expected map[myProject_myNetwork1:%!s(*network.EndpointSettings=&{0xc00017efa0 [] [myProject-myService-1 myService] map[] 0 {{0 0} {<nil>}} {{0 0} {<nil>}} [] 0 {{0 0} {<nil>}} {{0 0} {<nil>}} 0 []}) myProject_myNetwork2:%!s(*network.EndpointSettings=&{0xc00017ef50 [] [myProject-myService-1 myService] map[] 0 {{0 0} {<nil>}} {{0 0} {<nil>}} [] 0 {{0 0} {<nil>}} {{0 0} {<nil>}} 0 []})] (length 2) to have length 1 --- FAIL: TestDefaultNetworkSettings/returns_the_network_with_the_highest_priority_when_service_has_multiple_networks (0.00s) === FAIL: pkg/compose TestDefaultNetworkSettings (0.00s)

So it expects a single network, but returns 2?

create_test.go:225: assertion failed: expected map[ myProject_myNetwork1: %!s(*network.EndpointSettings=&{0xc00017efa0 [] [myProject-myService-1 myService] map[] 0 {{0 0} {<nil>}} {{0 0} {<nil>}} [] 0 {{0 0} {<nil>}} {{0 0} {<nil>}} 0 []}) myProject_myNetwork2: %!s(*network.EndpointSettings=&{0xc00017ef50 [] [myProject-myService-1 myService] map[] 0 {{0 0} {<nil>}} {{0 0} {<nil>}} [] 0 {{0 0} {<nil>}} {{0 0} {<nil>}} 0 []}) ] (length 2) to have length 1

Ah, right, yes, indeed: that's code that was only ran on API < 1.44, but that conditional code is removed, so it will no longer return a single network (which was because older daemons didn't support that);

compose/pkg/compose/create.go

Lines 523 to 539 in af0029a

// Starting from API version 1.44, the Engine will take several EndpointsConfigs

// so we can pass all the extra networks we want the container to be connected to

// in the network configuration instead of connecting the container to each extra

// network individually after creation.

if versions.GreaterThanOrEqualTo(version, APIVersion144) {

if len(service.Networks) > 1 {

serviceNetworks := service.NetworksByPriority()

for _, networkKey := range serviceNetworks[1:] {

mobyNetworkName := project.Networks[networkKey].Name

epSettings := createEndpointSettings(project, service, serviceIndex, networkKey, links, useNetworkAliases)

endpointsConfig[mobyNetworkName] = epSettings

}

}

if primaryNetworkEndpoint.MacAddress == "" {

primaryNetworkEndpoint.MacAddress = service.MacAddress

}

}

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Test that the network with the highest priority is returned as "primary" network, and other networks as extra networks. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

…ture Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

- use an intermediate serviceNetworks slice so that we don't have to call service.NetworksByPriority multiple times. - shift the primary network from the slice (if any), so that we can drop some checks for "additional networks" - group code related to setting up the primary network as first step, then append remaining networks. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

… render individual layers Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>

Format layer progress details with minimal efforts as new UI does not render individual layers

thaJeztah requested a review from a team as a code owner July 21, 2025 18:51

thaJeztah requested review from glours and ndeloof July 21, 2025 18:51

thaJeztah marked this pull request as draft July 21, 2025 18:51

thaJeztah changed the title ~~Bump moby cli~~ WIP: migrate to moby modules Jul 21, 2025

thaJeztah force-pushed the bump_moby_cli branch 2 times, most recently from a5b199b to 73bb946 Compare July 21, 2025 18:55

thaJeztah commented Jul 21, 2025

View reviewed changes

cmd/compose/bridge.go Outdated Show resolved Hide resolved

thaJeztah commented Jul 21, 2025

View reviewed changes

thaJeztah mentioned this pull request Jul 21, 2025

Create github.com/moby/moby/api and github.com/moby/moby/client module moby/moby#50280

Merged

thaJeztah force-pushed the bump_moby_cli branch 2 times, most recently from 83ca48f to 13827c4 Compare July 21, 2025 19:20

thaJeztah force-pushed the bump_moby_cli branch from 13827c4 to 52f0ad8 Compare July 23, 2025 20:32

thaJeztah force-pushed the bump_moby_cli branch 3 times, most recently from d2146d4 to 3a7124d Compare July 24, 2025 09:24

thaJeztah force-pushed the bump_moby_cli branch 3 times, most recently from 3b1ec4c to 6cd56af Compare August 5, 2025 22:51

thaJeztah force-pushed the bump_moby_cli branch from 6cd56af to 8925b9c Compare August 5, 2025 23:13

thaJeztah force-pushed the bump_moby_cli branch 2 times, most recently from d7c329f to 04ac165 Compare August 28, 2025 21:12

thaJeztah mentioned this pull request Jan 19, 2026

build(deps): bump github.com/moby/go-archive from 0.1.0 to 0.2.0 #13473

Closed

thaJeztah force-pushed the bump_moby_cli branch 3 times, most recently from 083a7c9 to cb07967 Compare January 22, 2026 10:19

thaJeztah commented Jan 22, 2026

View reviewed changes

thaJeztah force-pushed the bump_moby_cli branch 3 times, most recently from ce597b5 to 8797fa0 Compare January 26, 2026 08:45

thaJeztah force-pushed the bump_moby_cli branch from 8797fa0 to b6784a8 Compare January 27, 2026 10:25

thaJeztah force-pushed the bump_moby_cli branch from b6784a8 to 3dbbaa6 Compare February 4, 2026 14:24

thaJeztah commented Feb 4, 2026

View reviewed changes

thaJeztah force-pushed the bump_moby_cli branch from 3dbbaa6 to 8bf3cdc Compare February 4, 2026 14:32

thaJeztah added 3 commits February 4, 2026 15:38

go.mod: bump github.com/moby/moby/api v1.53.0, moby/client v0.2.2

22854d9

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

pkg/bridge: remove uses of go-connections

bc31828

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

pkg/compose: remove unused consts

eff4cdb

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah force-pushed the bump_moby_cli branch 3 times, most recently from 1a7e8e8 to 6510727 Compare February 4, 2026 16:42

thaJeztah added 3 commits February 4, 2026 17:47

update TestDefaultNetworkSettings

29e0611

Test that the network with the highest priority is returned as "primary" network, and other networks as extra networks. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

pkg/compose: defaultNetworkSettings: return early for unsupported fea…

0373747

…ture Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah force-pushed the bump_moby_cli branch from 6510727 to fae6d96 Compare February 4, 2026 16:48

thaJeztah closed this Feb 4, 2026

thaJeztah reopened this Feb 4, 2026

Format layer progress details with minimal efforts as new UI does not…

f7cc8d5

… render individual layers Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>

ndeloof mentioned this pull request Feb 5, 2026

Format layer progress details with minimal efforts as new UI does not render individual layers thaJeztah/compose#335

Merged

Merge pull request #335 from ndeloof/bump_moby_cli

62c9042

Format layer progress details with minimal efforts as new UI does not render individual layers

		@@ -0,0 +1,89 @@
		// Package urlutil provides helper function to check if a given build-context

	// Starting from API version 1.44, the Engine will take several EndpointsConfigs
	// so we can pass all the extra networks we want the container to be connected to
	// in the network configuration instead of connecting the container to each extra
	// network individually after creation.
	if versions.GreaterThanOrEqualTo(version, APIVersion144) {
	if len(service.Networks) > 1 {
	serviceNetworks := service.NetworksByPriority()
	for _, networkKey := range serviceNetworks[1:] {
	mobyNetworkName := project.Networks[networkKey].Name
	epSettings := createEndpointSettings(project, service, serviceIndex, networkKey, links, useNetworkAliases)
	endpointsConfig[mobyNetworkName] = epSettings
	}
	}
	if primaryNetworkEndpoint.MacAddress == "" {
	primaryNetworkEndpoint.MacAddress = service.MacAddress
	}
	}

Conversation

thaJeztah commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ndeloof commented Jul 21, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented Jul 21, 2025

Uh oh!

thaJeztah commented Jul 21, 2025

Uh oh!

thaJeztah commented Jul 21, 2025

Uh oh!

ndeloof commented Jul 22, 2025

Uh oh!

thaJeztah commented Jul 22, 2025

Uh oh!

thaJeztah commented Jul 23, 2025

Uh oh!

thaJeztah commented Aug 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

thaJeztah commented Jul 21, 2025 •

edited

Loading

codecov bot commented Jan 26, 2026 •

edited

Loading