DysonFS

aka. Dyson Network File System

Go implementation of the Dyson Network file service.

Modes

• master: HTTP API, gRPC, uploads, file serving, health check
• worker: post-upload media processing, derived file generation, cleanup
• storage: optional local storage node for filesystem-backed deployments

CLI

Use the first positional argument as the command.

go run ./cmd master
go run ./cmd migrate-legacy --config ./config.toml --legacy-dsn "$LEGACY_DATABASE_DSN"
go run ./cmd reanalyze-missing --config ./config.toml
go run ./cmd validate-storage --config ./config.toml

Logging

• ZEROLOG_PRETTY=true enables console-style pretty logs
• LOG_LEVEL=debug|info|warn|error sets the log level

Run

go run ./cmd master

Legacy migration

Use the one-shot migrator to import data from the old C# database into the new schema:

go run ./cmd migrate-legacy --config ./config.toml --legacy-dsn "$LEGACY_DATABASE_DSN"

Flags:

• --dry-run to simulate without writing
• --skip-derived to skip thumbnail/compression child reconstruction
• --batch-size to tune import batch size
• --continue-on-error to keep going after row-level failures

Metadata reanalysis

Repair missing image/video metadata from stored source files:

go run ./cmd reanalyze-missing --config ./config.toml

It shows a preview first, then asks for confirmation before changing anything.

Flags:

• --reanalyze-limit to cap the preview/repair batch size
• --file-id to target one or more file IDs, comma-separated
• --preview-count to control how many candidates are shown first
• --yes to skip the confirmation prompt

Upload API

Both direct upload and chunked upload creation accept the same metadata payload:

{
  "hash": "...",
  "file_name": "clip.mov",
  "file_size": 12345,
  "content_type": "video/quicktime",
  "pool_id": "...",
  "expired_at": "2026-05-17T12:34:56Z",
  "chunk_size": 5242880,
  "parent_id": "...",
  "overwrite_id": "...",
  "fast_mode": true,
  "usage": "...",
  "application_type": "...",
  "index": true
}

• direct upload uses multipart form data with the same field names, plus file
• index controls whether the file is indexed; defaults to false, but automatically becomes true when parent_id points to an indexed folder
• parent_id is optional and can still be resolved server-side when omitted
• overwrite_id is optional; when set, the upload replaces the content of an existing file instead of creating a new cloud_files row
• fast_mode is optional; when used with overwrite_id, the server tries to overwrite the existing backing object in place
• fast mode is applied only when the target object_id is referenced by exactly one live file
• if the target object is shared, fast mode automatically falls back to recreate-and-swap behavior
• overwrite keeps the existing file record and swaps in a new file_object after analysis completes
• overwrite deletes stale derived children so thumbnails/compressed variants can be regenerated from the new source
• hash is stored on the created file/task when provided
• upload quota is checked before task creation or direct upload processing
• quota refusal returns 403 Forbidden
• if pool_id is omitted, the quota check uses the resolved default pool

Direct overwrite example:

curl -X POST "http://localhost:8080/api/files/upload/direct" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -F "file=@./edited.txt" \
  -F "overwrite_id=FILE_ID" \
  -F "fast_mode=true"

Chunked overwrite task creation example:

{
  "overwrite_id": "FILE_ID",
  "fast_mode": true,
  "file_name": "ignored-on-overwrite",
  "file_size": 12345,
  "content_type": "text/plain",
  "chunk_size": 5242880
}

Fast mode notes:

• in fast mode, the existing file_object record is updated in place instead of creating a new one
• the server updates hash, size, mime_type, and analyzed metadata on that existing object
• stale derived children are removed before worker regeneration

File List Filters

These endpoints share the same list query parser:

• GET /api/files/me
• GET /api/files/root/children
• GET /api/files/:id/children
• GET /api/files/unindexed

Base query params:

• offset
• take
• order: date, name, size
• orderDesc: true|false or 1|0
• query: case-insensitive substring match on file name
• name: exact file name match
• extension: file extension, with or without the leading .
• usage
• application_type
• content_type: exact response MIME type match
• mime_type: alias of content_type
• pool_id
• parent_id
• indexed: true|false or 1|0
• recycled: true|false or 1|0
• is_folder: true|false or 1|0
• has_thumbnail: true|false or 1|0
• has_compression: true|false or 1|0
• min_size
• max_size
• created_after: RFC3339 or YYYY-MM-DD
• created_before: RFC3339 or YYYY-MM-DD
• updated_after: RFC3339 or YYYY-MM-DD
• updated_before: RFC3339 or YYYY-MM-DD

Notes:

• content_type / mime_type matches the file response MIME type, which comes from the backing object or the folder sentinel type
• /api/files/unindexed also accepts pool as an alias of pool_id
• /api/files/unindexed defaults to recycled=false when the query param is omitted
• size filters use the backing object size in bytes

Example:

curl "http://localhost:8080/api/files/me?content_type=image/png&extension=png&has_thumbnail=1&min_size=1024"

WOPI / Collabora CODE

DysonFS can expose WOPI host endpoints for Collabora Online / CODE.

Config:

[wopi]
enabled = true
publicUrl = "https://files.example.com"
collaboraUrl = "https://collabora.example.com"
tokenTtl = "15m"
requireProof = false
proofCacheTtl = "1h"

Notes:

• publicUrl must be the externally reachable DysonFS base URL used by Collabora for WOPI callbacks
• DysonFS builds the WOPI callback base directly from publicUrl, for example https://files.example.com/wopi/files/:id
• collaboraUrl must point at the Collabora server base URL
• proof validation is not implemented yet; keep requireProof = false

Launch endpoint:

• POST /api/files/:id/edit creates a WOPI-backed editing session for the authenticated user
• the response includes action_url, method, form_fields, and wopi_src
• the client should POST form_fields to action_url, typically into an iframe or popup
• Collabora callbacks may authenticate with either access_token form/query parameters or Authorization: Bearer <access_token>

Example response:

{
  "action_url": "https://collabora.example.com/browser/edit?WOPISrc=https%3A%2F%2Ffiles.example.com%2Fwopi%2Ffiles%2FFILE_ID",
  "action": "edit",
  "method": "POST",
  "form_fields": {
    "access_token": "TOKEN",
    "access_token_ttl": "1770000000000"
  },
  "wopi_src": "https://files.example.com/wopi/files/FILE_ID",
  "expires_at": "2026-05-29T12:00:00Z"
}

Example client flow:

<iframe id="collabora-frame" name="collabora-frame"></iframe>
<script>
  async function openEditor(fileId, token) {
    const response = await fetch(`/api/files/${fileId}/edit`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${token}`,
      },
    });
    if (!response.ok) throw new Error("failed to create edit session");

  const session = await response.json();
  const form = document.createElement("form");
  form.method = session.method || "POST";
  form.action = session.action_url;
  form.target = "collabora-frame";

  for (const [key, value] of Object.entries(session.form_fields || {})) {
      const input = document.createElement("input");
      input.type = "hidden";
      input.name = key;
      input.value = value;
      form.appendChild(input);
    }

    document.body.appendChild(form);
    form.submit();
    form.remove();
  }
</script>

WOPI endpoints:

• GET /wopi/files/:id
• GET /wopi/files/:id/contents
• POST /wopi/files/:id/contents
• POST /wopi/files/:id

Quota And Billing

Quota values are reported in MB.

Base quota is the sum of:

• leveling quota
• perk quota

Leveling quota uses account.profile.level, clamped to 0..120, with these milestones:

• Lv0: 512MB
• Lv10: 1GB
• Lv60: 5GB
• Lv120: 10GB

Between those milestones, the quota is interpolated piecewise.

Perk quota is added on top of leveling quota:

• perk 1: 10GB
• perk 2: 25GB
• perk 3: 50GB

Extra quota comes from active quota_records and is added after the base quota.

GET /api/billing/quota returns:

{
  "based_quota": 15360,
  "extra_quota": 25,
  "total_quota": 15385
}

GET /api/billing/usage returns:

{
  "used_quota": 300,
  "total_quota": 15385,
  "total_file_count": 2,
  "total_usage_bytes": 209715200
}

Usage accounting rules:

• used_quota is billable usage in MB, not raw bytes
• raw file bytes are returned separately as total_usage_bytes
• pool billing cost_multiplier affects billable usage and quota checks
• the multiplier is applied per file based on the file's pool

Folders

Folders are created with POST /folders.

Request body:

{
  "name": "Projects",
  "parent_id": "..."
}

• A folder is stored as a cloud_files row with is_folder = true and indexed = true
• parent_id is optional
• The current implementation does not yet validate that the parent exists or is a folder
• Root folders automatically get a private read permission record

Permission Management

Files expose read/write/manage permissions through GET /files/:id/permissions and PUT /files/:id/permissions.

• No file permission rows means the file is public
• A private permission row with read makes a file private by default
• Permission checks inherit from ancestor folders
• PUT /files/:id/permissions replaces the full permission set in one batch

Request body:

{
  "items": [
    {
      "id": "...",
      "file_id": "...",
      "subject_type": "account",
      "subject_id": "...",
      "permission": "read"
    },
    {
      "id": "...",
      "file_id": "...",
      "subject_type": "scope",
      "subject_id": "files.manage",
      "permission": "manage"
    }
  ]
}

• subject_type can be public, private, account, or scope
• permission is typically read, write, or manage
• Send the full desired list; omitted rows are removed

File Update Operations

Rename a file with PATCH /api/files/:id:

{
  "name": "renamed-file.txt"
}

• rename updates only the file name
• rename currently requires the file owner or a superuser

Batch File Operations

Batch operations use POST /api/files/<operation>/batch with a JSON body.

Recycle files:

{
  "file_ids": ["file-id-1", "file-id-2"]
}

Restore files:

{
  "file_ids": ["file-id-1", "file-id-2"]
}

Purge files:

{
  "file_ids": ["file-id-1", "file-id-2"]
}

Move files into a parent:

{
  "file_ids": ["file-id-1", "file-id-2"],
  "parent_id": "...",
  "indexed": true
}

Available operations:

• POST /api/files/recycle/batch
• POST /api/files/restore/batch
• POST /api/files/delete/batch
• POST /api/files/move/batch

Notes:

• file_ids is required for every batch operation
• parent_id is optional for move; omit it or set it to null to move files back to the root
• indexed is optional for move; set to true to mark files as indexed, false to mark as unindexed, or omit to leave unchanged

File Listings

Files have an indexed flag that controls visibility in the file tree:

• Indexed files appear in GET /api/files/root/children and GET /api/files/:id/children (the normal folder browse experience)
• Unindexed files only appear in GET /api/files/unindexed (a flat listing of orphaned uploads)
• Folders are always indexed
• The indexed flag can be set at upload time via the index field, or changed later via the move batch endpoint

List responses include extra metadata for navigation and access UI:

• children_count for immediate child count
• permission_status for current access state

Performance notes:

• Child counts and inherited permission status are resolved in batches for list responses to avoid per-file query fan-out.
• Postgres should have a composite index on cloud_files(parent_id, deleted_at) so children_count lookups do not fall back to full table scans.
• Postgres should have a composite index on file_permissions(file_id, permission, deleted_at) so inherited permission-source lookups stay index-backed.
• These indexes are declared in the GORM models and are created by AutoMigrate, but existing deployments need a restart or migration run before the new indexes appear.
• If file-list or permission logs still show slow SQL after rollout, verify the indexes exist with pg_indexes and inspect the hot queries with EXPLAIN (ANALYZE, BUFFERS).

Example:

"permission_status": {
  "readable": true,
  "writable": false,
  "manageable": false,
  "visibility": "private",
  "inherited_from": "..."
}

Storage validation

Validate file_objects.storage_key against remote S3 objects and clean up orphans:

go run ./cmd validate-storage --config ./config.toml --yes

It snapshots remote keys first, then compares the snapshot against the database in batches.

Flags:

• --validate-snapshot to choose the snapshot file path
• --validate-prefix to limit the remote listing prefix
• --validate-batch to control DB batch size
• --yes to skip the confirmation prompt

Config

Use --config or CONFIG_PATH for a TOML config file.

Key settings:

• app.name
• database.dsn
• http.port
• grpc.port
• grpc.useTLS
• grpc.certFile
• grpc.keyFile
• storage.tempDir
• storage.localDir
• auth.target
• auth.useTLS
• passport.target
• passport.useTLS
• quota.leveling.level1
• quota.leveling.level10
• quota.leveling.level60
• quota.leveling.level120
• nats.url
• mode.master
• mode.worker
• mode.storage

Pool storage is configured with [[pools]] and seeded into the database at startup.

Example:

[[pools]]
id = "500e5ed8-bd44-4359-bc0a-ec85e2adf447"
name = "Default"
default = true
hidden = false

[pools.storage]
endpoint = "http://minio:9000"
bucket = "dyson-files"
enableSigned = true
enableSsl = false
secretId = "minio"
secretKey = "minio123"

Notes

• Public read is the default.
• Explicit ACL rows restrict access when present.
• master resolves storage from pool config stored in the database.
• worker listens for file upload events and builds thumbnails, blurhash, and other derived artifacts.
• master can use S3 directly; local storage is still supported.
• The Docker image expects ffmpeg and libvips runtime packages.
• The E2EE file route has been removed.