Skip to content

developer-guide.md

Latest commit

 

History

History
159 lines (116 loc) · 3.97 KB

developer-guide.md

File metadata and controls

159 lines (116 loc) · 3.97 KB

Developer Guide

This guide covers Git DRS internals, architecture, and development information.

Architecture Overview

Git DRS integrates with Git through several mechanisms:

Git Hooks Integration

Pre-commit Hook: git drs precommit

  • Triggered automatically before each commit
  • Processes all staged files
  • Creates DRS records for new files
  • Only processes files that don't already exist on the DRS server
  • Prepares metadata for later upload during push

Pre-push Hook: git drs pre-push-prepare (internal)

  • Triggered automatically before each push
  • Stages pending metadata for new/changed files
  • The hook prepares the repo for git drs push to complete upload and registration

Managed Push/Pull

  • git drs push performs the register/upload workflow directly through the syfon client stack
  • git drs pull performs the download workflow directly through the syfon client stack

File Processing Flow

1. Developer: git add file.bam
2. Developer: git commit -m "Add data"
3. Git Hook: git drs precommit
   - Creates DRS object metadata
   - Stores in .git/drs/ directory
4. Developer: git push
5. Git Hook: git drs pre-push-prepare
   - Stages pending metadata for DRS verify
6. Git DRS:
   - `git drs push` runs register/upload directly
   - `git drs pull` runs download directly

Current Data Path

Git DRS no longer uses a custom transfer agent.

  • Upload path (primary): git drs push discovers local pointers, bulk-registers missing objects, checks validity, and uploads missing bits.
  • Download path (primary): git drs pull resolves object records and downloads into local object storage.

Repository Structure

Core Components

cmd/                    # CLI command implementations
├── initialize/         # Repository initialization
├── push/               # Register/upload workflow
├── pull/               # Download workflow
├── prepush/            # Pre-push metadata staging hook
├── precommit/         # Pre-commit hook
├── addurl/            # Cloud object URL reference handling
└── ...

client/                # DRS client implementations
├── interface.go       # Client interface definitions
├── DRS.go         # Gen3/DRS client
└── drs-map.go        # File mapping utilities

config/                # Configuration management
└── config.go         # Config file handling

drs/                   # DRS object utilities
├── object.go         # DRS object structures
└── util.go           # Utility functions

lfs/                   # Pointer utilities
└── lfs.go            # Pointer/discovery helpers

utils/                 # Shared utilities
├── common.go         # Common functions
├── lfs-track.go      # Tracking utilities
└── util.go           # General utilities

Configuration System

Repository Configuration: .git/drs/config.yaml

current_server: gen3
servers:
  gen3:
    endpoint: "https://data.example.org/"
    profile: "myprofile"
    project: "project-123"
    bucket: "data-bucket"

DRS Object Management

Objects are stored in .git/drs/objects/ during pre-commit and referenced during push/pull workflows.

Development Setup

Prerequisites

  • Go 1.26.2+
  • Access to a DRS server for testing

Building from Source

# Clone repository
git clone https://github.com/calypr/git-drs.git
cd git-drs

# Install dependencies
go mod download

# Build
go build

# Install locally
export PATH=$PATH:$(pwd)

Development Workflow

  1. Make changes to source code
  2. Build and test: 
    go build
go test ./...
  3. Test with real repository: 
    cd /path/to/test-repo
/path/to/git-drs/git-drs --help

Debugging and Logging

Log Locations

  • Commit logs: .git/drs/git-drs.log
  • Push/Pull logs: .git/drs/git-drs.log

Testing

Unit Tests

# Test specific functionality
go test ./utils -run TestTrack

Integration Tests

WIP