Enable File deletion

[quinnwai]

Is your feature request related to a problem? Please describe.
As a user, when I run git rm to remove the file from the repo, I should be able to remove the data from CALYPR as well.

Describe the solution you'd like
Ideally, I could do a git rm and then during git push, the file is removed automatically for me (more dev work)

Describe alternatives you've considered

• At minimum, I should be able to do a git rm, commit, push and then just do a git drs delete separately to sync it up. This is manual and is on the user to maintain server state which seems silly
• The LFS way right now would be to just do a prune repo where only the files that exist in the repo are kept in indexd for that gen3 project. This also has some pitfalls as a repo to gen3 project mapping is not enforced nor would it necessarily be one-to-one

Additional context

• during git push, we don't know which files are deleted. LFS only tells us which files need to be uploaded, so we would have to do that calculation at some point (either during pre-commit or during pre-push)
• there are a couple edge cases here as to whether we delete just the indexd record (DRS record), just the file, or both, eg...
  • indexd records exist for that hash but none of them match the project
  • indexd record in that project exists but there's two
  • indexd record exists for this project AND one other that user doesn't have access to

[quinnwai]

Design Doc

Currently...

• git drs delete <hash-type> <oid>
• no way to delete actual data file in the bucket

Approach

On git drs delete, delete the corresponding index always and delete the file if nothing else in indexd points to it:

git drs delete git/repo/path/to/file

Why?

• When doing a git rm, commit, and push, LFS doesn’t register that files were deleted
• This means we would have to implement a “parse each git commit for the relevant rm” which could be an time-consuming operation for large file pushes that might not even have any files to remove
• Given that pushes happen really often and deletes happens relatively infrequently, I’d rather make it manual for now

Steps

1. Given file hash, get its index records
2. Given records, get records with a matching gen3 project
3. Find record matching the user-inputted git url
   1. if >1 records in project, keep the file + delete index
   2. if only that 1 record in project, delete index + file via fence delete/<file_id>

Out of scope for now: If remote add exists for files, then don’t ever delete (see note here)

Edge cases

We need to determine edge cases for if we need to delete only the indexd record and not the file. Given a particular file hash and particular bucket url…

1. File hash has no indexd records in any project
   • delete index + file
2. File hash has an indexd record exists for a project user doesn’t have access to
   • File URI will not match as it is based on the DRS ID and the hash
   • Will not encounter this use case, as UUIDs are unique
3. Index record exists for a project user has access to
   • same as above, will not be encountered
4. indexd record exists for this current project
   1. if >1, delete index only
   2. if =1, delete index + file

Related Issues

• Git repo to gen3 project mapping: you could easily change the config to be a different project and cause some damage. No guardrails gating the user from this

[bwalsh]

Just FYI on how "stock" git lfs handles this. TL/DR - it is server dependent

---

Short version: deleting an LFS-tracked file in a new commit does not delete its large blob from your LFS server or other clones. It only removes the pointer file from the repo at that commit. Actual bytes stick around as long as any reachable Git ref (branch/tag) in any remote still points to a commit that referenced them.

Here’s how removal really works and what you can do:

What happens when you git rm

• git rm path && git commit && git push removes the pointer file at HEAD.
• The LFS object (the big content in .git/lfs/objects / remote LFS store) remains, because older commits/branches still reference it.

Cleaning up locally

• Prune unneeded local LFS objects:

  git lfs prune            # keeps objects referenced by recent commits/reflog
  git lfs prune --force    # be more aggressive (use with care)

• If you want the working copy to drop downloaded binaries (but keep pointers), re-checkout the pointer state:

  git lfs fetch --all
  git lfs checkout --to=-

  (In practice, just rely on git lfs prune for space.)

Cleaning up on the remote LFS server

1. Ensure no refs point to the unwanted file:

   • Delete any branches/tags that include it, or

   • Rewrite history to remove the file entirely (so no commit references it):

     • Quick LFS-specific rewrite (convert past file versions into nothing or into LFS/non-LFS):

       # Move large history into LFS (if it wasn’t) or adjust paths
       git lfs migrate import --include="path/or/pattern"

     • To purge a path from history, use Git’s history rewrite (e.g., git filter-repo):

       git filter-repo --path path/to/huge.bin --invert-paths

       Then force-push:

       git push --force --all
       git push --force --tags

2. After no refs point to the object, most LFS servers will eventually garbage-collect it. Some have explicit admin/prune jobs or retention windows. (You may need a repo admin to run/server-side prune or wait for scheduled GC.)

Stopping future tracking

• If you no longer want a pattern in LFS:

  git lfs untrack "*.big"
  git add .gitattributes
  git commit -m "Stop tracking *.big with LFS"

  New files matching that pattern won’t be put in LFS (existing history is unaffected unless you rewrite it).

Common scenarios & recipes

A) I deleted the file in a new commit; I just want local disk space back

git lfs prune

B) I pushed a huge file by mistake and want it gone from Git history and LFS storage

# Remove it from all history
git filter-repo --path path/to/huge.bin --invert-paths
git push --force --all
git push --force --tags

# (Optional) ask the LFS server admin to run GC / wait for its GC window

C) The file should stay in history but be tracked by LFS (not regular Git)

git lfs track "*.bin"
git add .gitattributes
git commit -m "Track *.bin with LFS"
git lfs migrate import --include="*.bin"
git push --force --all
git push --force --tags

D) Make sure no one can re-introduce the big file type

• Add a CI check or pre-receive hook to block large files outside LFS.
• Document .gitattributes and enforce via code owners/branch protection.

Gotchas

• Any fork or other remote that still has refs to the old commits keeps the LFS objects alive server-side.
• Collaborators must rebase/reset onto the rewritten history after a purge.
• Some hosts keep a retention period for unreferenced LFS objects before final deletion.

---

Short answer: there is no Git-LFS-standard GC. Server-side garbage collection of LFS objects is implementation-specific. Once all Git refs that ever pointed at an LFS object are gone (branches/tags/forks/PRs), each hosting platform has its own background jobs or admin tools—some automatic, some manual—to eventually delete the now-orphaned blobs.

Here’s how the major servers handle it:

The general rule

• LFS blobs become eligible for deletion only after no reachable commit in any ref on the server points to a pointer that references them (which usually means history rewrite + force-push and cleaning up branches/tags/forks/PRs). The LFS spec doesn’t define GC; hosts implement their own cleanup. ([Jira]1)

GitHub.com / GHES

• For sensitive-data purges, GitHub Support’s workflow (after you’ve removed all refs and PRs/forks references) includes: run server GC and purge orphaned LFS objects. Outside of that Support path, cleanup is internal/opaque and happens on GitHub’s schedule. ([GitHub Docs]2)

GitLab (SaaS / self-managed)

• GitLab provides rake tasks and background jobs to clean up orphaned LFS files once they’re unreferenced. Admins can run or schedule these tasks (e.g., gitlab:cleanup:orphan_lfs_files) to reclaim storage. ([GitLab Docs]3)

Gitea / Forgejo

• There are cron tasks for LFS metadata and repo GC, but users report gaps where unused LFS files aren’t deleted even after running them—so behavior can require admin scripts or updates. ([Gitea]4)

Bitbucket Server / Cloud

• Atlassian notes that Git LFS lacks a built-in GC comparable to git gc, and requests for first-class server-side LFS pruning exist. In practice, deletion depends on the host’s internal processes or admin intervention. ([Jira]1)

Azure DevOps (Server)

• Some environments don’t automatically clean LFS objects even after repo deletion; admins may need to follow manual cleanup steps. ([Microsoft Learn]5)

---

Practical playbook (works everywhere)

1. Make objects unreferenced: rewrite history to remove the files (e.g., git filter-repo --path path --invert-paths), delete branches/tags that carried them, and ensure forks/PRs don’t keep references. Force-push.
2. Wait or trigger server cleanup:

• 🚧 🚧 🚧 🚧 🚧 🚧 🚧
  git drs: TODO
  🚧 🚧 🚧 🚧 🚧 🚧 🚧

• GitHub: if it’s a sensitive-data case, open a Support ticket—they will run GC and purge LFS orphans. Otherwise storage drops on their schedule. ([GitHub Docs]2)

• GitLab: run the orphan-LFS cleanup rake task or rely on scheduled jobs. ([GitLab Docs]3)

• Others: check your platform’s admin docs; many don’t expose LFS GC explicitly. ([Jira]1)

3. Local cleanup (developers): run git lfs prune to reclaim workstation disk; this is local only and does not affect the server. ([GitHub]6)

[quinnwai]

Thanks for the feedback / fyi! Seems like you're covering 2 important things: first, when should we do deletes? Second, for the cases in which we do, how do we want to handle them?

So given that Git LFS doesn't do clean up of the version history, a couple questions come to mind that I wanna discuss with the team:

1. How should indexd represent Git repo, by copying its current current state or storing its entire git history?
2. Given this, how do we want to proceed with Git DRS object deletion?

A couple use cases

• I want to revert back to a previous commit for the data and still have my files accessible and downloadable
• I want to remove files to clear up space in my bucket cause quotas
• I want to remove all evidence of the file because it's a PHI file that's been checked-in (actually was done)

[quinnwai]

Not slated for first official cutover for git-drs, we will work on this when user-facing concern or encounter size restrictions

[quinnwai]

yeah this is just being done via the standard lfs way which is you can clean it up if you want to yourself. See delete and delete-project commands. They also have checks now to not delete without confirmation, giving you a preview of what you're deleting and which remote server you're targeting. Close for now unless there's a need for implicit deletions.

[matthewpeterkort]

added to docs/TODO