Native tag editor frontend dist recovery

agent_allinone.md

@@ -0,0 +1,159 @@
+# Agent All-in-One Handoff: Native Tag Editor
+
+## Project
+
+Repository: `D:\ai\lora-scripts-next`
+
+Active branch for this work: `editor`
+
+This branch is intentionally separate from `main` because the native tag editor UI touches the trainer WebUI shell and can regress navigation, caching, and route behavior.
+
+## Must Read First
+
+Read these files before making changes:
+
+- `doc/local/AGENT_INTERNAL.md`
+- `doc/local/HANDOFF_DATASET_EDITOR_2026-05-29.md`
+- `.cursor/rules/00-project-overview.mdc`
+- `.cursor/rules/embedded-service-ports.mdc`
+- `docs/repo-layout.md`
+- `docs/portable-packaging-git-update.md`
+- `docs/team/risk-memo.md`
+- `docs/team/backlog-priorities.md`
+
+## Hard Constraints
+
+- Do not push this UI work directly to `main`.
+- Do not run npm build for the current frontend work. Edit `frontend/dist/` directly.
+- Do not change portable contract directory names or the portable launch chain unless explicitly requested.
+- Do not delete or commit local HuggingFace lock files under `huggingface/hub/.locks/`.
+- Old Gradio tag editor must remain opt-in only through `--enable-legacy-tageditor`.
+- `/dataset-editor.html` is the standalone native editor fallback/debug page.
+- `/native-tageditor.html` is the native editor embedded in the trainer shell.
+- `/tageditor.html` is the classic/legacy tag editor page.
+- Keep changes scoped to dataset/tag editor integration unless the user explicitly asks otherwise.
+
+## Current Dev Server
+
+Common command:
+
+```powershell
+python gui.py --dev --skip-prepare-environment --disable-tensorboard --disable-train-monitor --port 28182
+```
+
+Useful URLs:
+
+- `http://127.0.0.1:28182/tagger.html`
+- `http://127.0.0.1:28182/tageditor.html`
+- `http://127.0.0.1:28182/native-tageditor.html`
+- `http://127.0.0.1:28182/dataset-editor.html`
+- `http://127.0.0.1:28182/other/settings.html`
+
+## What This Branch Contains
+
+- Native dataset/tag editor backend API in `mikazuki/dataset_editor.py`.
+- Native editor frontend in `frontend/dist/dataset-editor.html` and `frontend/dist/assets/dataset-editor.*`.
+- Trainer-embedded native editor entry in `frontend/dist/assets/dataset-editor-entry.js`.
+- Separate navigation entries:
+  - Classic Tag Editor: `经典标签编辑`, `/tageditor.md`
+  - Native Tag Editor: `原生标签编辑`, `/native-tageditor.html`
+- Dataset editor tagging controls for local/API tagging, tag/natural-language caption mode, model selection, conflict handling, and settings integration.
+- Local tagger model storage helpers in `mikazuki/tagger/local_models.py`.
+- Preferred local tagger model layout:
+  - `tagger-models/wd14/<model-key>/`
+  - `tagger-models/vlm/<model-key>/`
+  - Legacy flat `tagger-models/<model-key>/` remains compatible.
+- Portable packaging updates to create/copy the tagger model directories.
+- Cache-control workaround in `mikazuki/app/application.py` for patched dist core assets.
+- Regression tests for route/sidebar JSON parsing and Chinese nav labels.
+
+## Important Recent Bugs
+
+### VuePress sidebar JSON and mojibake
+
+The VuePress app bundle `frontend/dist/assets/app.547295de.js` contains theme sidebar data as:
+
+```js
+const WE=JSON.parse(`...`)
+```
+
+Manual edits previously broke this JSON and then corrupted Chinese menu text. The current fix stores that JSON with ASCII-only `\uXXXX` escapes so runtime text still renders Chinese while file writes are less likely to corrupt encoding.
+
+If navigation becomes 404-like, stuck on `Loading...`, or shows mojibake, first inspect and parse this `WE=JSON.parse(...)` block.
+
+Relevant test:
+
+```powershell
+python -m pytest tests\test_dataset_editor_api.py::test_vuepress_theme_sidebar_json_stays_parseable -q
+```
+
+### Native editor route loading the classic editor
+
+Root cause found on 2026-05-30:
+
+- `/native-tageditor.html` existed and returned HTTP 200.
+- The HTML loaded `dataset-editor-entry.js`, so the native editor injection path was present.
+- But `frontend/dist/assets/app.547295de.js` mapped the VuePress page key `v-native-tageditor` to the classic editor page data chunk:
+
+```js
+"v-native-tageditor":()=>wt(()=>import("./tageditor.html.66da263e.js"),[]).then(({data:e})=>e)
+```
+
+That classic chunk exports frontmatter like:
+
+```json
+{"type":"iframe","subtype":"tageditor"}
+```
+
+So VuePress could render the old iframe/service warning on the native route. This was not primarily a port issue.
+
+Fix currently on branch:
+
+- Added `frontend/dist/assets/native-tageditor.html.native.js` as dedicated page data for `v-native-tageditor`.
+- Patched `frontend/dist/assets/app.547295de.js` so `v-native-tageditor` imports `native-tageditor.html.native.js`.
+- Patched `frontend/dist/native-tageditor.html` so it preloads the native page data chunk instead of old `tageditor.html.*.js` chunks.
+- Added regression coverage in `tests/test_dataset_editor_api.py::test_native_tageditor_uses_native_vuepress_page_data`.
+
+Safe verification snippets for this minified bundle should avoid `Select-String` on the whole single-line file. Use bounded boolean checks instead:
+
+```powershell
+$text = Get-Content frontend/dist/assets/app.547295de.js -Raw -Encoding UTF8
+$text.Contains('"v-native-tageditor":()=>wt(()=>import("./native-tageditor.html.native.js")')
+$text.Contains('"v-native-tageditor":()=>wt(()=>import("./tageditor.html.66da263e.js")')
+```
+
+## Verification Commands
+
+Run before claiming the branch is healthy:
+
+```powershell
+python -m pytest tests\test_dataset_editor_api.py tests\test_tagger_progress_api.py tests\test_portable_packaging_scripts.py -q
+```
+
+Last known result before this handoff:
+
+```text
+50 passed, 4 warnings
+```
+
+Browser smoke checks:
+
+- Open `/tagger.html`, confirm no 404 and sidebar Chinese is readable.
+- Open `/native-tageditor.html`, confirm it uses the trainer shell and native editor.
+- Open `/dataset-editor.html`, confirm standalone fallback still loads.
+- Open `/other/settings.html`, confirm tagger API settings are present.
+
+## Known Local Noise
+
+Do not commit these runtime files:
+
+- `huggingface/hub/.locks/models--SmilingWolf--wd-convnext-tagger-v3/*.lock`
+- `huggingface/hub/.locks/models--SmilingWolf--wd-v1-4-moat-tagger-v2/*.lock`
+
+## Suggested Next Work
+
+1. Continue visual QA of `/native-tageditor.html` in the trainer shell.
+2. Use a real dataset to test scan, thumbnails, selection, batch tagging, save, undo, and redo.
+3. Improve native editor UI carefully, one small change at a time, with browser checks after each change.
+4. Keep the classic editor and native editor separate in navigation.
+5. Avoid broad frontend shell rewrites until the current editor flow is stable.

build-scripts/03-copy-project.ps1

@@ -31,6 +31,7 @@ $excludeDirs = @(
     "logs",
     "output",
     "huggingface",
+    "tagger-models",
     "sd-models",
     "wd14_tagger_model",
     "train",
@@ -58,6 +59,9 @@ New-Item -ItemType Directory -Path (Join-Path $BuildDir "sd-models") -Force | Ou
 New-Item -ItemType Directory -Path (Join-Path $BuildDir "output") -Force | Out-Null
 New-Item -ItemType Directory -Path (Join-Path $BuildDir "logs") -Force | Out-Null
 New-Item -ItemType Directory -Path (Join-Path $BuildDir "huggingface") -Force | Out-Null
+New-Item -ItemType Directory -Path (Join-Path $BuildDir "tagger-models") -Force | Out-Null
+New-Item -ItemType Directory -Path (Join-Path $BuildDir "tagger-models\wd14") -Force | Out-Null
+New-Item -ItemType Directory -Path (Join-Path $BuildDir "tagger-models\vlm") -Force | Out-Null
 
 Write-Host "用户目录创建完成" -ForegroundColor Green
 

build-scripts/build_portable.ps1

@@ -333,7 +333,11 @@ Write-Host ""
 Write-Host "[3/6] Bundling default WD tagger (wd14-convnextv2-v2, ~388 MB)..." -ForegroundColor Cyan
 
 $hfHome = Join-Path $portableDir "huggingface"
+$taggerModelsDir = Join-Path $portableDir "tagger-models"
 New-Item -ItemType Directory -Path $hfHome -Force | Out-Null
+New-Item -ItemType Directory -Path $taggerModelsDir -Force | Out-Null
+New-Item -ItemType Directory -Path (Join-Path $portableDir "tagger-models\wd14") -Force | Out-Null
+New-Item -ItemType Directory -Path (Join-Path $portableDir "tagger-models\vlm") -Force | Out-Null
 $prefetchScript = Join-Path $sdtDir "scripts\prefetch_default_tagger.py"
 
 if (-not (Test-Path $prefetchScript)) {
@@ -342,28 +346,29 @@ if (-not (Test-Path $prefetchScript)) {
     Write-Host "  WARNING: no Python for tagger prefetch; tagger will download on first tag run" -ForegroundColor Yellow
 } else {
     $env:HF_HOME = $hfHome
+    $env:MIKAZUKI_TAGGER_MODELS_DIR = $taggerModelsDir
     # pip writes progress to stderr; with $ErrorActionPreference Stop that aborts the build.
     $prevEap = $ErrorActionPreference
     $ErrorActionPreference = 'Continue'
     $prefetchExit = 1
     if (Get-Command python -ErrorAction SilentlyContinue) {
         & python -m pip install -q huggingface_hub 2>&1 | Out-Null
-        & python $prefetchScript --hf-home $hfHome --if-missing
+        & python $prefetchScript --hf-home $hfHome --tagger-models-dir $taggerModelsDir --if-missing
         $prefetchExit = $LASTEXITCODE
     } elseif (Test-Path $pythonExe) {
         if (Test-Path $getPipPath) {
             Write-Host "  Bootstrapping pip in embedded Python..."
             & $pythonExe $getPipPath --no-warn-script-location 2>&1 | Out-Null
         }
         & $pythonExe -s -m pip install -q huggingface_hub 2>&1 | Out-Null
-        & $pythonExe -s $prefetchScript --hf-home $hfHome --if-missing
+        & $pythonExe -s $prefetchScript --hf-home $hfHome --tagger-models-dir $taggerModelsDir --if-missing
         $prefetchExit = $LASTEXITCODE
     }
     $ErrorActionPreference = $prevEap
     if ($prefetchExit -ne 0) {
         Write-Host "  WARNING: tagger prefetch failed; 7z may ship without offline tagger" -ForegroundColor Yellow
     } else {
-        Write-Host "  Cached under huggingface/hub/" -ForegroundColor Green
+        Write-Host "  Cached under huggingface/hub/ and tagger-models/" -ForegroundColor Green
     }
 }
 
@@ -450,7 +455,7 @@ foreach ($bat in @("Update-SD-Trainer.bat", "Download-Anima-Model.bat")) {
 Write-Host ""
 Write-Host "[5/6] Creating user directories and README..." -ForegroundColor Cyan
 
-foreach ($d in @("sd-models", "output", "logs", "huggingface")) {
+foreach ($d in @("sd-models", "output", "logs", "huggingface", "tagger-models", "tagger-models\wd14", "tagger-models\vlm")) {
     $p = Join-Path $portableDir $d
     New-Item -ItemType Directory -Path $p -Force | Out-Null
     [System.IO.File]::WriteAllText((Join-Path $p ".gitkeep"), "")
@@ -463,8 +468,10 @@ $readme += "  1. Double-click run_gui.bat`r`n"
 $readme += "  2. First launch requires internet (downloads ~3 GB of PyTorch)`r`n"
 $readme += "  3. Open http://127.0.0.1:28000 in browser`r`n`r`n"
 $readme += "Tagging:`r`n"
-$readme += "  Default WD tagger (wd14-convnextv2-v2) is bundled under huggingface/`r`n"
-$readme += "  (~400 MB). Use the built-in Tagger page without extra model download.`r`n`r`n"
+$readme += "  Default WD tagger (wd14-convnextv2-v2) is bundled under tagger-models/wd14/`r`n"
+$readme += "  (~400 MB). Put extra WD/CL tag models in tagger-models/wd14/<model-key>/`r`n"
+$readme += "  Future VLM caption models can be placed under tagger-models/vlm/<model-key>/`r`n"
+$readme += "  with the files required by that model, such as model.onnx and selected_tags.csv.`r`n`r`n"
 $readme += "Directories:`r`n"
 $readme += "  run_gui.bat      - Stable entrypoint for portable users`r`n"
 $readme += "  run_gui_portable.bat - Legacy shim (logic in SD-Trainer/scripts/portable/)`r`n"
@@ -473,6 +480,7 @@ $readme += "  SD-Trainer/      - Project files`r`n"
 $readme += "  sd-models/       - Put your models here`r`n"
 $readme += "  output/          - Training output`r`n"
 $readme += "  logs/            - Logs`r`n`r`n"
+$readme += "  tagger-models/   - Local tagger models`r`n`r`n"
 $readme += "Update:`r`n"
 $readme += "  update\update_sd_trainer.bat       - Shortcut to Update-SD-Trainer.bat`r`n"
 $readme += "  update\update_dependencies.bat     - Update Python packages`r`n`r`n"

docs/api/dataset-tagging.md

@@ -62,6 +62,23 @@
 - `https://hf-mirror.com`
 - `https://modelscope.cn`
 
+## 本地打标模型目录
+
+WebUI 会优先检查本地 `tagger-models/<family>/<model-key>/`，文件齐全时不会访问 Hugging Face。默认模型目录为：
+
+```text
+tagger-models/wd14/wd14-convnextv2-v2/
+  model.onnx
+  selected_tags.csv
+```
+
+整合包会把默认 WD 模型放入该目录。用户也可以手动下载其它模型并放到对应的分类子目录：
+
+- `tagger-models/wd14/<model-key>/`：WD14 / CL 系列，主要用于 tag 打标。
+- `tagger-models/vlm/<model-key>/`：预留给 VLM / 自然语言描述模型。
+
+为兼容已安装用户，旧的一层目录 `tagger-models/<model-key>/` 仍可识别。如果文件不完整，系统会继续回退到原有 `huggingface/` 缓存或在线下载流程。
+
 **成功**：`status: success`，`message: 模型下载已开始`  
 **失败**：已有任务进行中、未知模型等返回 `status: fail`。