Prevent situations where Git LS files are added but not properly committed or pushed (workflow/committed-vs-pushed)

[bwalsh]

Title:

Current: Check to ensure all LFS files are committed and pushed, use drs (indexd) as system-of-record for file info

• PRO:
  • clean separation of concerns
• CON:
  • tags and meta data occur in different commit, push cycle than lfs files
  • user training, adapt expectations of git workflow (tagging and meta data happens after file push)

Alternative: Check all git lfs added files are committed, add tags to local, temporary drs directory. Use combination of local drs directory and drs server info for tagging and metadata initialization

• PRO:
  • tags and meta occur in same git workflow, no user expectation management/training
• CON:
  • same logic spread across git-drs, add-tag, and meta init/validate

---

Description:

We should add a safeguard to prevent situations where Git LFS files are added but not properly committed or pushed. This can lead to broken DRS manifests, missing metadata, or confusing behavior for collaborators.

---

Current Solution:

Implement a script or pre-push CI check that verifies:

1. All LFS-tracked files that are added (git add) are also committed.

   • Detect staged but uncommitted LFS files using:

     git lfs ls-files --cached

   • If any files are staged but not yet committed, block the [push, add-link, or meta init] and print a clear error.

2. All LFS-tracked files in commits are pushed to the remote.

   • Use git lfs status to detect LFS objects pending push.
   • If any LFS objects are pending, block the [add-link, or meta init] and print a clear warning.

---

Optional Enhancements:

• Integrate this check into:

  • Pre-push Git hook
  • CI workflows (e.g., GitHub Actions)

• Print helpful instructions if failures are detected (e.g., reminder to run git lfs push).

---

References:

• git lfs ls-files --cached

• git lfs status

• Related Issues:

  • gen3_util#127
  • [git-drs#13](https://github.com/bmeg/git-drs/issues/13)

---

Acceptance Criteria:

• Attempting to push with uncommitted LFS files triggers a warning or blocks the push.
• Attempting to push with LFS objects pending upload triggers a warning or blocks the push.
• Documentation is added describing the new check.

[bwalsh]

[quinnwai]

I'd say maybe the alternative is that we can make metadata a post-add / pre-commit action so that metadata can only be added for files that are already staged. That doesn't remove all possible problems that could occur with file data being out-of-sync with metadata and vice versa, but aligns the metadata creation better with a git workflow (ie add all the files in some sequence, commit files, push files).

Question: how did we frame/solve this problem in g3t? Any lessons learned to draw off of there?

---

All LFS-tracked files that are added (git add) are also committed.

What would be the reasoning to want to enforce this? I can see how people would be frustrated with not having a clean repository and being unable to push because they wanna do a test file or they wanna do their data upload in stages.

---

I still don't fully understand this issue, I think what you're trying to get at is the question of: how do we integrate our metadata validation with our existing Git (+ custom Git LFS) workflow?

[bwalsh]

@quinnwai thanks for the careful read

I still don't fully understand this issue, I think what you're trying to get at is the question of:

What will tags and meta init read from?

• As far as I know, the indexd update doesn't completely happen until push. Is that still the case?
• See Practical Strategies to Preserve Pointer Extensions #18 after today's conversation, it seems that where the sha256->extended attributes mapping are stored is still up in the air?

[bwalsh]

@quinnwai Follow up:

• For the tagging use case: the tagger will need to update either:
  a) .git/index + pointer files,
  b) the local .drs cache or
  c) the indexd/drs server
• For the meta data use case: will need to read [drs-url, id, etc] from either a,b,c (above)

Preferably, the answer will be a, b or c not all/any if we can avoid it.

Therefore the decision on what these use cases read, specifies when - and impacts the users workflow.

[matthewpeterkort]

implementing a guard for this in #232