cas

A content-addressable file cache with download verification.

Files are downloaded from pluggable remote sources, stored locally by their SHA-256 content hash, and verified before being committed to the cache. The cache is safe for concurrent use — simultaneous requests for the same checksum share a single download, and callers can cancel via context without affecting in-progress downloads.

Requires Go 1.25 or later.

Installation

go get github.com/acycl/cas

Source packages are installed separately to avoid pulling in unnecessary dependencies:

go get github.com/acycl/cas/gcs
go get github.com/acycl/cas/s3

The HTTPS source lives in the root module and requires no additional dependencies:

import "github.com/acycl/cas/https"

Usage

GCS

client, _ := storage.NewClient(ctx)
d, _ := transfermanager.NewDownloader(client)
src := gcs.NewSource(d)
cache := cas.New("/var/cache/files", src)

m, _ := cas.NewManifest(
    cas.File("file.txt", "gs://bucket/file.txt", "ab12cd34..."),
)
f, _ := cache.Open(ctx, m, "file.txt")
defer f.Close()
data, _ := io.ReadAll(f)

S3

cfg, _ := config.LoadDefaultConfig(ctx)
client := awss3.NewFromConfig(cfg)
d := manager.NewDownloader(client)
src := s3.NewSource(d)
cache := cas.New("/var/cache/files", src)

m, _ := cas.NewManifest(
    cas.File("file.txt", "s3://bucket/file.txt", "ab12cd34..."),
)
f, _ := cache.Open(ctx, m, "file.txt")
defer f.Close()
data, _ := io.ReadAll(f)

HTTPS

src := https.NewSource()
cache := cas.New("/var/cache/files", src)

m, _ := cas.NewManifest(
    cas.File("file.txt", "https://example.com/file.txt", "ab12cd34..."),
)
f, _ := cache.Open(ctx, m, "file.txt")
defer f.Close()
data, _ := io.ReadAll(f)

Manifest

A Manifest is a standalone type that maps names to remote files. It has no dependency on a Cache, so it can be defined at package scope, loaded from configuration, or shared across cache instances:

m, _ := cas.NewManifest(
    cas.File("model.bin", "gs://bucket/model.bin", "ab12cd34..."),
    cas.File("config.json", "s3://bucket/config.json", "ef56ab78..."),
)

f, _ := cache.Open(ctx, m, "model.bin")
defer f.Close()

Use Validate to check at startup that all manifest URIs have registered sources, without downloading anything:

if err := cache.Validate(m); err != nil {
    log.Fatal(err)
}

Custom sources

Implement the Source interface to add support for any protocol:

type Source interface {
    Scheme() string
    Download(ctx context.Context, dst *os.File, u *url.URL) error
}

Pass custom sources directly to New:

cache := cas.New("/var/cache/files", mySource)