Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,8 @@ hivewatch.init(
- `runs/<run_id>.jsonl` for the complete event history
- `runs/<run_id>.map.json` for map-ready metadata that can be loaded directly later

Serve the dashboard separately:
Serve the dashboard separately. Without `--run-id`, the server watches the run
directory and streams new events as they arrive:

```bash
hivewatch map run --runs-dir runs --port 7070
Expand All @@ -78,7 +79,10 @@ Open one specific saved run in static mode:
hivewatch map run --runs-dir runs --run-id run-abc123
```

The bundled `examples/hivewatch_map.html` viewer loads map metadata first and falls back to the JSONL-derived event history for older runs. This keeps local development and later replay workflows compatible with the same viewer.
The bundled viewer loads map metadata first and falls back to the JSONL-derived
event history for older runs. It displays non-geographic client metadata in the
sidebar automatically. Prefix a client metadata key with `_` when the value
should remain in the saved artifact but stay hidden from the map card.

### Package layout

Expand Down Expand Up @@ -190,6 +194,11 @@ hivewatch.init(emitters=[MyEmitter()])
| `lat` / `lng` / `country` | float/str | Client location metadata for map visualization |
| `base_round` | int | For asynchronous FL, staleness is `round - base_round` |

Unknown client fields are preserved in `ClientUpdate` objects and the local
JSONL/`.map.json` artifacts. The map dashboard displays visible non-geographic
fields automatically. Use a leading underscore for values that should be stored
but hidden from the bundled map viewer, for example `_debug_score`.

## Logged Metrics

### Weights & Biases
Expand Down Expand Up @@ -236,7 +245,8 @@ FL Server
hivewatch
├── WandbEmitter → wandb.ai dashboard
└── MLflowEmitter → MLflow UI (localhost:5000)
├── MLflowEmitter → MLflow UI (localhost:5000)
└── SSEEmitter → local JSONL/.map.json artifacts and map dashboard
```

`hivewatch` does not depend on a specific transport layer or FL framework. Applications bridge their training framework to `hivewatch` in the same way they would bridge it to another experiment tracking backend.
Expand Down
4 changes: 3 additions & 1 deletion docs/appfl.rst
Original file line number Diff line number Diff line change
Expand Up @@ -51,4 +51,6 @@ HiveWatch understands common FL metadata such as:
* ``lat``, ``lng``, ``city``, and ``country`` for map views

Unknown keys are preserved in the client payload, so you can attach
framework-specific metadata without breaking the core schema.
framework-specific metadata without breaking the core schema. The bundled map
viewer displays non-geographic client metadata automatically. Prefix a key with
``_`` when it should be saved but hidden from the map card.
15 changes: 15 additions & 0 deletions docs/emitters.rst
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,21 @@ This emitter writes:
* ``runs/<run_id>.jsonl`` with the full event stream
* ``runs/<run_id>.map.json`` with map-ready metadata

Set ``serve_map=False`` when training and dashboard serving should be separate.
This is useful for batch jobs, shared run directories, or environments where a
long-running dashboard process already exists.

Server metadata, such as the server location or host label, can be recorded
with the runtime API:

.. code-block:: python

hw.set_server_metadata(city="Chicago", country="US", lat=41.7, lng=-87.9)

Unknown client metadata is preserved in both artifacts. Fields prefixed with
``_`` remain available in the saved data but are hidden by the bundled map
viewer.

Weights & Biases Emitter
========================

Expand Down
6 changes: 3 additions & 3 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ forcing you into a single training framework.
With HiveWatch you can:

* log client and round metrics from custom training loops,
* stream and replay runs locally with the built-in map dashboard,
* stream, inspect, and replay runs locally with the built-in map dashboard,
* send metrics to Weights & Biases, MLflow, or both, and
* integrate observability into APPFL workflows with minimal glue code.

Expand Down Expand Up @@ -55,8 +55,8 @@ HiveWatch is organized around a few clear technical components.

Map Dashboard
^^^^^^^^^^^^^
Visualizes client geography and run progress from saved ``.jsonl`` and
``.map.json`` artifacts.
Visualizes client geography, training metadata, and replay progress
from saved ``.jsonl`` and ``.map.json`` artifacts.

.. grid-item-card::

Expand Down
39 changes: 35 additions & 4 deletions docs/map.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@ Map Dashboard
=============

HiveWatch ships with a local dashboard flow based on saved run artifacts and a
small HTTP server.
small HTTP server. The same viewer supports live monitoring and replay of
completed runs.

Generate run artifacts
----------------------
Expand All @@ -27,16 +28,19 @@ Serve the dashboard

hivewatch map run --runs-dir runs --port 7070

Without ``--run-id``, the server watches the run directory and publishes new
JSONL events as they arrive.

Useful flags
------------

* ``--host`` changes the bind address.
* ``--run-id`` opens one saved run in static replay mode.
* ``--map-path`` points at a custom HTML viewer.
* ``--poll-interval`` controls how often the runs directory is rescanned.

The bundled viewer in the ``examples/`` directory loads map metadata first and
falls back to the raw event history for older runs. That means the same viewer
works for both live monitoring and deferred replay.
The bundled viewer loads ``.map.json`` metadata first and falls back to the raw
event history for older runs.

Common map commands
-------------------
Expand All @@ -46,6 +50,9 @@ Common map commands
# Watch a runs directory for live updates
hivewatch map run --runs-dir runs --port 7070

# Open one completed run in static replay mode
hivewatch map run --runs-dir runs --run-id run-abc123

# Serve a custom viewer HTML file
hivewatch map run --runs-dir runs --map-path /path/to/viewer.html

Expand All @@ -58,3 +65,27 @@ Deferred viewing
Because HiveWatch persists map metadata separately from the raw JSONL log, you
can train first and inspect later. This keeps a live workflow and a replay
workflow compatible with the same dashboard interface.

Client metadata in the map
--------------------------

The sidebar shows client fields from the run artifacts. Geo and system identity
fields such as ``lat``, ``lng``, ``city``, ``country``, ``ip``, and
``client_id`` are used internally or omitted from the card. Other scalar
metadata is displayed automatically.

Use a leading underscore for metadata that should be preserved but hidden from
the bundled viewer:

.. code-block:: python

hw.log_client_update(
client_id="client-1",
round=round_num,
current_local_steps=200,
blocking=True,
_debug_score=0.92,
)

The map also includes a draggable event log. Drag it by the header to move it;
double-click the header to reset its saved position.
6 changes: 6 additions & 0 deletions docs/quickstart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ Minimal example
local_accuracy=0.70 + round_num * 0.03,
local_loss=0.90 - round_num * 0.08,
num_samples=500,
current_local_steps=200,
bytes_sent=8192,
lat=41.88,
lng=-87.63,
Expand All @@ -46,6 +47,11 @@ This creates two run artifacts under ``runs/``:
* ``<run_id>.jsonl`` with the full event stream.
* ``<run_id>.map.json`` with map-ready metadata for replay and deferred viewing.

Client metadata that is not part of the core schema is preserved. The map
dashboard displays non-geographic client fields automatically; prefix a field
with ``_`` when it should remain in the artifact but stay hidden from the map
card, for example ``_debug_score``.

Serve the dashboard
-------------------

Expand Down
5 changes: 3 additions & 2 deletions src/hivewatch/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,9 @@ def main(argv=None):
args = parser.parse_args(argv)

if args.command == "map" and args.map_command == "run":
# Disable watch mode when loading a specific run
watch_mode = args.run_id
# Static run replay reads the selected JSONL directly; the directory
# watcher is only needed for live dashboards.
watch_mode = args.run_id is None

server = MapServer(
host=args.host,
Expand Down
18 changes: 1 addition & 17 deletions src/hivewatch/emitters/sse_emitter.py
Original file line number Diff line number Diff line change
Expand Up @@ -308,20 +308,4 @@ def _start_server(self):

@staticmethod
def _client_dict(c: ClientUpdate) -> dict:
return {
"client_id": c.client_id,
"round": c.round,
"lat": c.lat,
"lng": c.lng,
"city": c.city,
"country": c.country,
"local_accuracy": c.local_accuracy,
"local_loss": c.local_loss,
"num_samples": c.num_samples,
"gradient_norm": c.gradient_norm,
"bytes_sent": c.bytes_sent,
"train_time_sec": c.train_time_sec,
"cpu_pct": c.cpu_pct,
"ram_mb": c.ram_mb,
"status": c.status,
}
return c.to_dict()
Loading