diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json index 9bf73f1e8..f1ad5c6f2 100644 --- a/.devcontainer/devcontainer.json +++ b/.devcontainer/devcontainer.json @@ -73,7 +73,8 @@ "figma.figma-vscode-extension", "mike-lischke.vscode-antlr4", "ms-vsliveshare.vsliveshare", - "Google.gemini-cli-vscode-ide-companion" + "Google.gemini-cli-vscode-ide-companion", + "MermaidChart.vscode-mermaid-chart" ] } }, diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 55689c9d8..6b040a21a 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -28,6 +28,9 @@ everything pre-installed. - jsonschema/: JSON schemas for data validation. - antlr: Description of search grammar - lib/gen: Output of generated code +- docs/ARCHITECTURE.md: Architecture diagrams + +For a visual overview of how these components interact in the cloud and during local development, see the [Architecture Diagrams](./docs/ARCHITECTURE.md). ## Running locally diff --git a/GEMINI.md b/GEMINI.md index 945f9702c..d3d645150 100644 --- a/GEMINI.md +++ b/GEMINI.md @@ -1,6 +1,6 @@ # Gemini Code Assist Configuration for webstatus.dev - + This document provides context to Gemini Code Assist to help it generate more accurate and project-specific code suggestions. @@ -163,14 +163,14 @@ The goal is to retrieve a specific feature's data and serve it via the REST API. ### 3.4. User Authentication and Notifications -This section describes the components related to user accounts, authentication, and notification features. +This section describes the components related to user accounts, authentication, and the end-to-end notification ecosystem. -- **Authentication**: User authentication is handled via Firebase Authentication on the frontend, which integrates with GitHub as an OAuth provider. When a user signs in, the frontend receives a Firebase token and a GitHub token. The GitHub token is sent to the backend during a login sync process. +- **Authentication**: User authentication is handled via Firebase Authentication on the frontend, which integrates with GitHub as an OAuth provider. On login, the backend's `/v1/users/me/ping` endpoint synchronizes the user's verified GitHub emails with their notification channels. - **User Profile Sync**: On login, the backend synchronizes the user's verified GitHub emails with their notification channels in the database. This is an example of a transactional update using the mapper pattern. The flow is as follows: 1. A user signs in on the frontend. The frontend sends the user's GitHub token to the backend's `/v1/users/me/ping` endpoint. 2. The `httpserver.PingUser` handler receives the request. It uses a `UserGitHubClient` to fetch the user's profile and verified emails from the GitHub API. 3. The handler then calls `spanneradapters.Backend.SyncUserProfileInfo` with the user's profile information. - 4. The `Backend` adapter translates the `backendtypes.UserProfile` to a `gcpspanner.UserProfile` and calls `gcpspanner.Client.SyncUserProfileInfo`. + 4. The `Backend` adapter translates the `backendtypes.UserProfile` to a `gcpspanner.UserProfile` and calls `gcpspanner.Client.SyncUserProfileInfo` (which uses transactional helpers to create/update notification channels and states). 5. `SyncUserProfileInfo` starts a `ReadWriteTransaction`. 6. Inside the transaction, it fetches the user's existing `NotificationChannels` and `NotificationChannelStates`. 7. It then compares the existing channels with the verified emails from GitHub. @@ -178,8 +178,49 @@ This section describes the components related to user accounts, authentication, - Emails that were previously disabled are re-enabled using `updateWithTransaction`. - Channels for emails that are no longer verified are disabled, also using `updateWithTransaction`. 8. The entire set of operations is committed atomically. If any step fails, the entire transaction is rolled back. -- **Notification Channels**: The `NotificationChannels` table stores the destinations for notifications (e.g., email addresses). Each channel has a corresponding entry in the `NotificationChannelStates` table, which tracks whether the channel is enabled or disabled. +- **Notification Channels**: The `NotificationChannels` table stores the destinations for notifications. Currently, two types are supported: `email` and `webhook` (specifically Slack webhooks). + - **Email Channels:** Are provisioned automatically through the GitHub profile sync. Manual creation or updates of email channels are rejected by the API because they rely on GitHub's verification process. + - **Webhook Channels:** Users can manually create and update webhook channels. + - **Limits:** There is a strict maximum of 25 notification channels per user. + - Each channel has a corresponding entry in the `NotificationChannelStates` table, which tracks whether the channel is enabled or disabled. - **Saved Search Subscriptions**: Authenticated users can save feature search queries and subscribe to receive notifications when the results of that query change. This is managed through the `UserSavedSearches` and `UserSavedSearchBookmarks` tables. + - **API**: A full CRUD API (`/v1/users/me/subscriptions`) and adapter layer exist to manage the creation, reading, updating, and deletion of subscriptions. + - **Individual Feature Subscriptions**: Users can subscribe to updates for specific individual features. This is implemented by creating a subscription to a **System Managed Saved Search** associated with that feature. + - **Limits**: There is a strict maximum of **25 subscriptions** per user. + - **Forward-Compatible Models**: Subscription triggers are forward-compatible. If a trigger stored in the database is no longer a valid API enum, its `value` must be set to `unknown` and the original string preserved in a `raw_value` field to prevent the API from failing on old data. + - **Pagination**: Pagination logic for `SavedSearchSubscriptions` and `NotificationChannelDeliveryAttempts` correctly sorts by `UpdatedAt` or `AttemptTimestamp` (descending) with `ID` as a tie-breaker (ascending) to ensure consistent results. +- **System Managed Saved Searches**: These are automatically created and updated by the system (e.g., when a feature is added or its search criteria changes). They are stored in the `SystemManagedSavedSearches` table and are used to support "Subscribe to this feature" functionality. + - **Redirects & Moves**: When features are moved, split, or redirected, the system migrates and deduplicates associated subscriptions to ensure users continue to receive relevant notifications. +- **Notification Pipeline Architecture**: + 1. **Ingestion/Workflow**: Detects changes in feature data. + 2. **Event Producer**: Receives ingestion events, calculates diffs between snapshots, and publishes `FeatureDiffEvent` messages. Uses DLQs for robust error handling. + 3. **Push Delivery Worker**: Consumes diff events. Its `Dispatcher` filters them against user subscriptions and triggers (using a visitor pattern), then queues specific delivery jobs (e.g., `EmailDeliveryJob`). + 4. **Email Worker**: Consumes delivery jobs, renders HTML templates (including match warnings for moved/split features), and sends emails via Chime. + +### 3.5. Feature Diffing and Data Evolution + +- **Removed vs. Deleted**: The system distinguishes between features that are truly **deleted** from the database and those that are merely **removed** from a specific search result (due to moves, splits, or property changes). `FeatureDiff` contains separate slices for each. +- **Match Warnings**: When a feature no longer matches a user's saved search (but still exists), the notification system flags it with a "Match Warning" (⚠️) to inform the user why it was removed from their scoped digest. +- **Blob Schema Evolution**: State for saved search notifications is stored in GCS blobs (`lib/blobtypes`). + - **Comparator Best Practices**: When adding new fields to storage structs, the `comparator.go` logic must explicitly handle all four states of `OptionallySet` fields (Unset/Unset, Unset/Set, Set/Unset, Set/Set). This prevents "Cold Start" problems where new data fails to trigger initial notifications. + - **Quiet Rollout**: Minimal additions (like a browser status changing from Unset to "Unavailable" without version details) can be flagged for "Quiet Rollout" to avoid spamming users during backfills. + - **Guide for Adding New Fields**: Always update `comparator.go` when adding fields to `blobtypes` to ensure robust comparison. +- **System Managed Saved Searches**: These are automatically created and updated by the system (e.g., when a feature is added or its search criteria changes). They are stored in the `SystemManagedSavedSearches` table and are used to support "Subscribe to this feature" functionality. + - **Redirects & Moves**: When features are moved, split, or redirected, the system migrates and deduplicates associated subscriptions to ensure users continue to receive relevant notifications. +- **Notification Pipeline Architecture**: + 1. **Ingestion/Workflow**: Detects changes in feature data. + 2. **Event Producer**: Receives ingestion events, calculates diffs between snapshots, and publishes `FeatureDiffEvent` messages. Uses DLQs for robust error handling. + 3. **Push Delivery Worker**: Consumes diff events. Its `Dispatcher` filters them against user subscriptions and triggers (using a visitor pattern), then queues specific delivery jobs (e.g., `EmailDeliveryJob`). + 4. **Email Worker**: Consumes delivery jobs, renders HTML templates (including match warnings for moved/split features), and sends emails via Chime. + +### 3.5. Feature Diffing and Data Evolution + +- **Removed vs. Deleted**: The system distinguishes between features that are truly **deleted** from the database and those that are merely **removed** from a specific search result (due to moves, splits, or property changes). `FeatureDiff` contains separate slices for each. +- **Match Warnings**: When a feature no longer matches a user's saved search (but still exists), the notification system flags it with a "Match Warning" (⚠️) to inform the user why it was removed from their scoped digest. +- **Blob Schema Evolution**: State for saved search notifications is stored in GCS blobs (`lib/blobtypes`). + - **Comparator Best Practices**: When adding new fields to storage structs, the `comparator.go` logic must explicitly handle all four states of `OptionallySet` fields (Unset/Unset, Unset/Set, Set/Unset, Set/Set). This prevents "Cold Start" problems where new data fails to trigger initial notifications. + - **Quiet Rollout**: Minimal additions (like a browser status changing from Unset to "Unavailable" without version details) can be flagged for "Quiet Rollout" to avoid spamming users during backfills. + - **Guide for Adding New Fields**: Always update `comparator.go` when adding fields to `blobtypes` to ensure robust comparison. ## 4. Specifications & Generated Code @@ -516,6 +557,6 @@ When you give me this prompt, I will: 1. Read the `GEMINI.md` file to find the last analyzed commit SHA. 2. Use `git log` to find all the commits that have been made since that SHA. -3. Analyze the new commits, applying the "Verify, Don't Assume" principle by consulting relevant sources of truth (e.g., `openapi.yaml` for API changes, migration files for schema changes). -4. Update this document with the new information. -5. Update the last analyzed commit SHA near the top of this file. +3. Analyze the new commits, applying the "Verify, Don't Assume" principle by consulting relevant sources of truth (e.g., `openapi.yaml` for API changes, migration files for schema changes). Use the `get_pull_request` and `get_pull_request_comments` tools to get the pull request information for **every relevant PR** found in the commits. Read the comments to understand the context and architectural decisions, because it is likely Gemini was used for those changes and its lack of knowledge caused those errors when generating the change. Do not assume you only need to check one or two PRs; check them all. +4. Update this document with the new information first. +5. Update the last analyzed commit SHA near the top of this file only after all other updates are complete. diff --git a/README.md b/README.md index ec5e75cdb..1f5367530 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,8 @@ described in a [OpenAPI](./openapi/backend/openapi.yaml) document. The tool provides a [frontend](./frontend/) dashboard written in Typescript to display the data. +To understand how these components interact in the cloud and during local development, please refer to the [Architecture Diagrams](./docs/ARCHITECTURE.md). + ## Search Syntax webstatus.dev provides a powerful search feature to help you find the diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 000000000..670d63f31 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,246 @@ +# Webstatus.dev Architecture & Workflows + +This document contains high-level architectural diagrams for the webstatus.dev ecosystem. These diagrams are intended to help new developers understand the system boundaries, data flows, and local development environment. + +_Note: The nodes in these diagrams are clickable! Click on a component to jump to its source code or relevant GCP documentation._ + +--- + +## 1. High-Level Architecture (Multi-Project Setup) + +To limit the scope of the `allUsers` IAM permission (which makes the frontend and API public), the infrastructure is split across three GCP projects. They are networked together via a Shared VPC in the Host project. + +```mermaid +flowchart TD + subgraph Host_Project ["Project: Host (VPC)"] + VPC["Shared VPC / Cloud Armor"] + end + + subgraph Public_Project ["Project: Public (allUsers Access)"] + FE["Frontend (Cloud Run)"] + BE["Backend API (Cloud Run)"] + end + + subgraph Internal_Project ["Project: Internal (Data & Jobs)"] + direction TB + DB[(Cloud Spanner)] + Cache[(Valkey/Memorystore)] + PS[[Pub/Sub]] + Workflows["Ingestion Jobs (Cloud Run Jobs)"] + Workers["Notification Workers"] + GCS[(GCS State Buckets)] + end + + %% Interaction Flows + User((User/Browser)) --> FE + User --> BE + FE -.-> VPC + BE -.-> VPC + VPC -.-> DB + VPC -.-> Cache + BE --> DB + BE --> Cache + + Workflows --> DB + Workers --> DB + Workers --> PS + Workers --> GCS + + %% Clickable Links + click VPC "https://cloud.google.com/vpc/docs/shared-vpc" "GCP Shared VPC Docs" + click FE "https://github.com/GoogleChrome/webstatus.dev/tree/main/frontend/" "Go to Frontend Source" + click BE "https://github.com/GoogleChrome/webstatus.dev/tree/main/backend/" "Go to Backend Source" + click DB "https://cloud.google.com/spanner/docs" "GCP Spanner Docs" + click Cache "https://cloud.google.com/memorystore/docs/valkey" "GCP Memorystore for Valkey" + click PS "https://cloud.google.com/pubsub/docs" "GCP Pub/Sub Docs" + click Workflows "https://github.com/GoogleChrome/webstatus.dev/tree/main/workflows/" "Go to Workflows Source" + click Workers "https://github.com/GoogleChrome/webstatus.dev/tree/main/workers/" "Go to Workers Source" + click GCS "https://cloud.google.com/storage/docs" "GCP Cloud Storage Docs" +``` + +--- + +## 2. Public-to-Internal Request Flow + +This diagram illustrates how a user's request travels from the browser, hits the public-facing API, and securely queries the internal database using the Spanner Adapter pattern. + +```mermaid +sequenceDiagram + participant U as User / Browser + box "Public Project" #f9f9f9 + participant FE as Frontend (Lit) + participant BE as Backend (Go API) + end + box "Internal Project" #e1f5fe + participant C as Valkey Cache + participant DB as Spanner DB + end + + U->>FE: Load Dashboard + FE->>BE: GET /v1/features + BE->>C: Check Cache + alt Cache Hit + C-->>BE: Return JSON + else Cache Miss + BE->>DB: Query via Spanner Adapter + DB-->>BE: Return Rows + BE->>C: Update Cache + end + BE-->>FE: HTTP 200 JSON + FE-->>U: Render Web Components +``` + +_(Note: Sequence diagrams in Mermaid currently have limited support for external hyperlinks on participants, so we rely on the flowchart diagrams for deep-linking.)_ + +--- + +## 3. Data Ingestion Pipeline (Internal Project) + +Webstatus.dev relies heavily on external data. Cloud Scheduler triggers jobs that download, parse, and synchronize this data into our Spanner database. + +```mermaid +flowchart LR + subgraph External ["External Sources"] + WPT["WPT.fyi API"] + BCD["MDN BCD Repo"] + WF["Web-Features Repo"] + end + + subgraph Internal ["Project: Internal"] + direction TB + Sched["Cloud Scheduler"] --> Job["Ingestion Job (Go)"] + + subgraph Job_Logic ["Job Internal Flow"] + DL["pkg/data/downloader.go"] --> P["pkg/data/parser.go"] + P --> Adp["Spanner Adapter"] + end + + Adp --> DB[(Cloud Spanner)] + end + + WPT --> DL + BCD --> DL + WF --> DL + + %% Clickable Links + click WPT "https://wpt.fyi/" "Web Platform Tests API" + click BCD "https://github.com/mdn/browser-compat-data" "MDN BCD Repository" + click WF "https://github.com/web-platform-dx/web-features" "Web-Features Repository" + click Sched "https://cloud.google.com/scheduler/docs" "GCP Cloud Scheduler" + click Job "https://github.com/GoogleChrome/webstatus.dev/tree/main/workflows/steps/services/" "Go to Workflow Services Source" + click Adp "https://github.com/GoogleChrome/webstatus.dev/tree/main/lib/gcpspanner/spanneradapters/" "Go to Spanner Adapters" + click DB "https://cloud.google.com/spanner" "GCP Spanner" +``` + +--- + +## 4. Notification System Architecture (Event-Driven) + +When data changes (via ingestion workflows) or users update their saved searches, an event-driven architecture processes those changes to deliver email and push notifications. + +```mermaid +flowchart TD + subgraph Internal ["Project: Internal"] + direction TB + DB[(Spanner: Subscriptions & Channels)] + EP["Event Producer Worker"] --> PS[[Pub/Sub: feature-diffs]] + PS --> PDW["Push Delivery Worker"] + PS --> EW["Email Worker"] + + PDW --> DB + EW --> Temp["lib/email (Templates)"] + EW --> Chime["Chime (email service)"] + + EP --> GCS[(GCS: Snapshot State)] + end + + DB -- Trigger Change --> EP + + %% Clickable Links + click DB "https://github.com/GoogleChrome/webstatus.dev/tree/main/lib/gcpspanner/" "Go to Spanner Schema/Mappers" + click EP "https://github.com/GoogleChrome/webstatus.dev/tree/main/workers/event_producer/" "Go to Event Producer Source" + click PS "https://cloud.google.com/pubsub" "GCP Pub/Sub" + click PDW "https://github.com/GoogleChrome/webstatus.dev/tree/main/workers/push_delivery/" "Go to Push Delivery Worker Source" + click EW "https://github.com/GoogleChrome/webstatus.dev/tree/main/workers/email/" "Go to Email Worker Source" + click Temp "https://github.com/GoogleChrome/webstatus.dev/tree/main/lib/email/" "Go to HTML Email Templates" + click GCS "https://cloud.google.com/storage" "GCS Blob Storage" +``` + +--- + +## 5. Local Development & E2E Testing Environment + +Understanding the local dev loop is crucial. We use Skaffold and Minikube to orchestrate live services alongside GCP emulators. The `Makefile` provides targets to populate these emulators with either _fake data_ (for deterministic E2E testing) or _live data_ (for manual workflow testing). + +```mermaid +flowchart TD + subgraph Dev_Machine ["Developer Machine / CI Pipeline"] + direction LR + Make["Makefile"] + Playwright{"Playwright (E2E Tests)"} + end + + subgraph Minikube ["Minikube Local Cluster"] + direction TB + + subgraph Live_Services ["Skaffold Live Services"] + FE["Frontend (Lit/Nginx)"] + BE["Backend API (Go)"] + end + + subgraph Emulators ["GCP Emulators & Mocks"] + Spanner[(Spanner Emulator)] + DS[(Datastore Emulator)] + Auth["Firebase Auth Emulator"] + WM["Wiremock +(Mock GitHub API)"] + Valkey[(Valkey Cache)] + PS[[Pub/Sub Emulator]] + end + + FE <-->|API Calls| BE + FE <-->|Firebase Login| Auth + BE <-->|Read/Write| Spanner + BE <-->|Cache| Valkey + BE <-->|Fetch Profile/Emails +on Login| WM + end + + subgraph Data_Population ["Data Population Strategies"] + direction TB + FakeUsers["make dev_fake_users +(util/cmd/load_test_users)"] + FakeData["make dev_fake_data +(util/cmd/load_fake_data)"] + RealFlows["make dev_workflows +(util/run_job.sh)"] + end + + %% Makefile connections + Make -.->|Triggers| FakeUsers + Make -.->|Triggers| FakeData + Make -.->|Triggers| RealFlows + + %% Population flows + FakeUsers -->|Seeds Test Users| Auth + FakeData -->|Seeds Predictable Entities| Spanner + FakeData -->|Seeds Predictable Entities| DS + RealFlows -->|Ingests Live Data| Spanner + + %% Testing flows + Playwright -.->|Requires| FakeData + Playwright -.->|Requires| FakeUsers + Playwright -->|Runs Tests Against| FE + + %% Clickable Links + click Make "https://github.com/GoogleChrome/webstatus.dev/tree/main/Makefile" "Go to Makefile" + click Playwright "https://github.com/GoogleChrome/webstatus.dev/tree/main/e2e/tests/" "Go to E2E Playwright Tests" + click FE "https://github.com/GoogleChrome/webstatus.dev/tree/main/frontend/src/" "Go to Frontend Source" + click BE "https://github.com/GoogleChrome/webstatus.dev/tree/main/backend/pkg/httpserver/" "Go to Backend Handlers" + click Spanner "https://github.com/GoogleChrome/webstatus.dev/tree/main/.dev/spanner/" "Spanner Emulator Setup" + click WM "https://github.com/GoogleChrome/webstatus.dev/tree/main/.dev/wiremock/" "Wiremock Configuration" + click Auth "https://github.com/GoogleChrome/webstatus.dev/tree/main/.dev/auth/" "Firebase Auth Emulator Setup" + click FakeUsers "https://github.com/GoogleChrome/webstatus.dev/tree/main/util/cmd/load_test_users/" "Go to load_test_users Source" + click FakeData "https://github.com/GoogleChrome/webstatus.dev/tree/main/util/cmd/load_fake_data/" "Go to load_fake_data Source" + click RealFlows "https://github.com/GoogleChrome/webstatus.dev/tree/main/util/run_job.sh" "Go to run_job.sh" +```