Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ A comprehensive Go SDK for building [flowctl](https://github.com/withObsrvr/flow
- **Three Complete SDKs**: Source, Processor, and Consumer
- **Event-First API**: Work with strongly-typed `*flowctlv1.Event` objects
- **Automatic Control Plane Integration**: Registration, heartbeats, and discovery
- **Historical Chunk Reporting**: Report bounded ledger work-unit progress, completion, verification, and failures
- **Built-in Observability**: Health checks, metrics, and logging
- **Production-Ready**: Graceful shutdown, error handling, and backpressure
- **Developer-Friendly**: 70-85% less code than manual gRPC implementation
Expand Down Expand Up @@ -143,6 +144,37 @@ if err != nil {
}
```

## Historical Chunk Reporting

For bounded historical workers, use the component reporter to emit flowctl chunk state:

```go
import (
"github.com/withObsrvr/flowctl-sdk/pkg/component"
flowctlpb "github.com/withobsrvr/flowctl/proto"
)

cfg := component.ConfigFromEnv()
reporter, err := component.NewReporter(ctx, cfg)
if err != nil {
return err
}
defer reporter.Close()

_ = reporter.Register(ctx, flowctlpb.ServiceType_SERVICE_TYPE_SOURCE, map[string]string{
"network": "pubnet",
})

_ = reporter.ReportChunkProgress(ctx, chunkStart, chunkEnd, "extract", nil, nil)
_ = reporter.ReportChunkCompleted(ctx, chunkStart, chunkEnd, true, map[string]int64{
"ledgers": chunkEnd - chunkStart + 1,
}, map[string]string{"passed": "true"})
```

The reporter reads the standard `flowctl run` component environment contract:
`ENABLE_FLOWCTL`, `FLOWCTL_ENDPOINT`, `FLOWCTL_COMPONENT_ID`, `FLOWCTL_RUN_ID`,
`FLOWCTL_ATTEMPT`, and `FLOWCTL_HEARTBEAT_INTERVAL_MS`.

## Custom Metrics

Track custom metrics easily:
Expand Down
68 changes: 40 additions & 28 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,25 +3,31 @@ module github.com/withObsrvr/flowctl-sdk
go 1.26.1

require (
github.com/withObsrvr/flow-proto v0.0.0-20251209215201-bd54ee3e43e9
github.com/withObsrvr/flow-proto v0.1.3
google.golang.org/grpc v1.79.2
)

require (
github.com/lib/pq v1.10.9
github.com/stellar/go-stellar-sdk v0.5.0
github.com/withobsrvr/flowctl v0.0.10
google.golang.org/protobuf v1.36.11
gopkg.in/yaml.v3 v3.0.1
)

require (
cloud.google.com/go v0.114.0 // indirect
cloud.google.com/go/auth v0.5.1 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.2 // indirect
cel.dev/expr v0.25.1 // indirect
cloud.google.com/go v0.116.0 // indirect
cloud.google.com/go/auth v0.13.0 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.6 // indirect
cloud.google.com/go/compute/metadata v0.9.0 // indirect
cloud.google.com/go/iam v1.1.8 // indirect
cloud.google.com/go/storage v1.42.0 // indirect
github.com/Microsoft/go-winio v0.6.1 // indirect
cloud.google.com/go/iam v1.2.2 // indirect
cloud.google.com/go/monitoring v1.21.2 // indirect
cloud.google.com/go/storage v1.49.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.30.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.48.1 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.48.1 // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/aws/aws-sdk-go v1.45.27 // indirect
github.com/aws/aws-sdk-go-v2 v1.36.5 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.11 // indirect
Expand All @@ -45,57 +51,63 @@ require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/cncf/xds/go v0.0.0-20251210132809-ee656c7534f5 // indirect
github.com/creachadair/jrpc2 v1.2.0 // indirect
github.com/creachadair/mds v0.13.4 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/djherbis/fscache v0.10.1 // indirect
github.com/envoyproxy/go-control-plane/envoy v1.36.0 // indirect
github.com/envoyproxy/protoc-gen-validate v1.3.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-errors/errors v1.5.1 // indirect
github.com/go-jose/go-jose/v4 v4.1.3 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/google/s2a-go v0.1.7 // indirect
github.com/google/s2a-go v0.1.8 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
github.com/googleapis/gax-go/v2 v2.12.4 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.4 // indirect
github.com/googleapis/gax-go/v2 v2.14.1 // indirect
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/klauspost/compress v1.17.6 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 // indirect
github.com/pelletier/go-toml v1.9.5 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_golang v1.17.0 // indirect
github.com/prometheus/client_model v0.5.0 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.45.0 // indirect
github.com/prometheus/procfs v0.12.0 // indirect
github.com/segmentio/go-loggly v0.5.1-0.20171222203950-eb91657e62b2 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/spiffe/go-spiffe/v2 v2.6.0 // indirect
github.com/stellar/go-xdr v0.0.0-20260312225820-cc2b0611aabf // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/stretchr/testify v1.11.1 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
go.opentelemetry.io/otel v1.39.0 // indirect
go.opentelemetry.io/otel/metric v1.39.0 // indirect
go.opentelemetry.io/otel/trace v1.39.0 // indirect
golang.org/x/crypto v0.46.0 // indirect
go.opentelemetry.io/contrib/detectors/gcp v1.39.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.54.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 // indirect
go.opentelemetry.io/otel v1.42.0 // indirect
go.opentelemetry.io/otel/metric v1.42.0 // indirect
go.opentelemetry.io/otel/sdk v1.42.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.42.0 // indirect
go.opentelemetry.io/otel/trace v1.42.0 // indirect
golang.org/x/crypto v0.48.0 // indirect
golang.org/x/exp v0.0.0-20231006140011-7918f672742d // indirect
golang.org/x/mod v0.30.0 // indirect
golang.org/x/net v0.48.0 // indirect
golang.org/x/net v0.50.0 // indirect
golang.org/x/oauth2 v0.34.0 // indirect
golang.org/x/sync v0.19.0 // indirect
golang.org/x/sys v0.39.0 // indirect
golang.org/x/text v0.32.0 // indirect
golang.org/x/time v0.5.0 // indirect
golang.org/x/tools v0.39.0 // indirect
google.golang.org/api v0.183.0 // indirect
google.golang.org/genproto v0.0.0-20240528184218-531527333157 // indirect
golang.org/x/sys v0.41.0 // indirect
golang.org/x/text v0.34.0 // indirect
golang.org/x/time v0.8.0 // indirect
google.golang.org/api v0.215.0 // indirect
google.golang.org/genproto v0.0.0-20241118233622-e639e219e697 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20251202230838-ff82c1b0f217 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20251202230838-ff82c1b0f217 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260209200024-4cfbd4190f57 // indirect
gopkg.in/djherbis/atime.v1 v1.0.0 // indirect
gopkg.in/djherbis/stream.v1 v1.3.1 // indirect
)
Loading
Loading