Newsletter

This folder contains a local daily newsletter pipeline:

• fetch candidates from configured news queries
• generate a draft daily issue as Markdown in issues/daily
• optionally have an OpenAI model polish the draft into a read-ready issue
• optionally have a second OpenAI model review the issue for release readiness
• render it to a styled HTML email
• build a Jekyll archive site for GitHub Pages
• send it through any SMTP provider
• schedule it on macOS with launchd

Layout

• issues/daily: daily Markdown issues named YYYY-MM-DD-daily-newsletter.md
• config/section_queries.json: feed/search queries used for automated candidate gathering
• data/candidates: raw fetched candidate story data by date
• scripts/send_daily_newsletter.py: renderer and email sender
• scripts/fetch_candidates.py: fetches candidates from configured queries
• scripts/check_pipeline_inputs.py: validates source coverage and market data before any issue draft is written
• scripts/generate_issue.py: generates a draft issue from candidate data
• scripts/ai_generate_issue.py: uses OpenAI to rewrite the draft into a stronger editorial issue
• scripts/review_issue.py: runs quality checks on generated issues before send
• scripts/ai_review_issue.py: uses OpenAI to score and gate the issue for automatic delivery
• scripts/openai_pipeline.py: shared OpenAI API and AI-config helpers
• scripts/build_archive.py: builds a Jekyll source tree for the archive into site/
• scripts/run_daily_pipeline.py: runs fetch, generate, render, archive, and optional send
• scripts/newsletter_command.py: single remote-friendly entrypoint that runs the full pipeline and can optionally commit/push the result
• output: generated HTML previews
• site: generated Jekyll site source for GitHub Pages
• launchd/com.munga.newsletter.daily.plist: example scheduler job
• .github/workflows/publish-newsletter-site.yml: optional GitHub Pages publishing workflow
• .env.example: SMTP configuration template
• config/newsletter_profile.json: default editorial and automation profile for the remote command
• config/codex_daily_automation_prompt.md: ready-to-paste Codex Desktop automation prompt for unattended daily generation
• sources.md: editorial source registry grouped by section and source type
• selection_criteria.md: editorial rubric for story selection tailored to the target reader
• story_scorecard.md: reusable template for scoring and triaging candidate stories
• daily_workflow.md: repeatable end-to-end workflow for gathering, scoring, writing, rendering, and sending an issue
• daily_issue_template.md: reusable Markdown template for building a new daily issue

Setup

1. Copy .env.example to .env.
2. Fill in your SMTP settings and recipient email.
3. Recommended default: fill in NEWSLETTER_AI_API_TOKEN and keep the AI settings enabled. The project is now tuned for a fail-closed, AI-required workflow if you want output at or above the April 1, 2026 benchmark issue quality.
4. Fetch candidates and generate an issue:

python3 scripts/run_daily_pipeline.py --overwrite

5. Review the generated issue in issues/daily if you want a manual check.
6. Render a preview if needed:

python3 scripts/send_daily_newsletter.py --preview-html --latest

7. Send a test message:

python3 scripts/send_daily_newsletter.py --latest

One-command remote run

If you want a single command that fetches sources, generates the issue, rebuilds the archive/search index, and optionally emails and publishes it, use:

python3 scripts/newsletter_command.py run

Useful variants:

python3 scripts/newsletter_command.py run --send
python3 scripts/newsletter_command.py run --date 2026-03-21 --send
python3 scripts/newsletter_command.py run --send --git-commit --git-push
python3 scripts/newsletter_command.py prepare
python3 scripts/newsletter_command.py publish --date 2026-03-21 --git-commit --git-push

That last form is the closest to the full remote-trigger workflow:

• fetch candidates
• run preflight input checks
• generate the newsletter
• run review
• render HTML
• rebuild the Jekyll site and search index
• send the email
• commit and push the updated archive so GitHub Pages refreshes

For Codex-driven editorial runs, the more useful split is:

• python3 scripts/newsletter_command.py prepare Creates a dated editorial packet, issue scaffold, research-notes scaffold, and fresh candidate snapshot.
• python3 scripts/newsletter_command.py publish --date YYYY-MM-DD --git-commit --git-push Reviews an already-written issue, renders the preview, rebuilds the archive, and pushes only if the review gates pass.

For an SSH-style remote trigger, the command would look like:

ssh <machine> 'cd /Users/munga/PycharmProjects/Newsletter && python3 scripts/newsletter_command.py run --send --git-commit --git-push'

Trigger From Phone

The cleanest minimal-intervention setup is now GitHub-based:

1. Push this project to GitHub.
2. Add the required repository secrets:
   • NEWSLETTER_AI_API_TOKEN
   • optional SMTP secrets if you want the email sent automatically
3. Use the workflow generate-newsletter.yml.

You now have two phone-friendly trigger options:

• GitHub app / mobile web: open the Generate Newsletter workflow and tap Run workflow.
• API trigger: send a repository_dispatch event from a phone shortcut or HTTP client.

Example repository_dispatch request:

curl -X POST \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <github-token>" \
  https://api.github.com/repos/<owner>/<repo>/dispatches \
  -d '{
    "event_type": "run-newsletter",
    "client_payload": {
      "issue_date": "2026-03-21",
      "send_email": true
    }
  }'

What that workflow does:

• runs the full newsletter generation pipeline
• rebuilds the archive, search index, sitemap, and current issue pages
• commits the generated newsletter artifacts back to the repo
• pushes the changes
• lets the Pages publish workflow update the live site

That means your phone interaction can be as small as:

• tap Run workflow in GitHub, or
• trigger one saved shortcut / HTTP request

Codex Desktop Automation Prompt

If you want Codex Desktop Automations to drive the repo directly instead of relying only on GitHub Actions, use the prompt in config/codex_daily_automation_prompt.md.

That prompt tells Codex to:

• use the April 1 benchmark and the repo templates as hard references
• start each run with python3 scripts/newsletter_command.py prepare
• use the generated editorial packet plus the approved source list for source-first research
• write the final issue directly rather than trusting the baseline draft blindly
• run python3 scripts/newsletter_command.py publish --git-commit --git-push only after the issue is publication-ready

This is the correct route if you want Codex itself, running in OpenAI's cloud against a linked GitHub repository, to generate the issue and push the changes without putting a provider token into this repo.

The schedule for that route is created in Codex Desktop / Codex cloud, not in the GitHub workflow files in this repository.

Daily schedule on macOS

Copy the sample plist into ~/Library/LaunchAgents and load it:

cp launchd/com.munga.newsletter.daily.plist ~/Library/LaunchAgents/
launchctl unload ~/Library/LaunchAgents/com.munga.newsletter.daily.plist 2>/dev/null || true
launchctl load ~/Library/LaunchAgents/com.munga.newsletter.daily.plist

The sample job runs every day at 07:30 local time. It now calls the full pipeline script, which fetches candidates, generates the issue, optionally applies AI drafting and AI review, renders the preview, builds the archive, and sends the email.

Notes

• The sender script has no third-party Python dependencies.
• --preview-html always writes a browser-openable preview into output/.
• If there is no issue matching today's date, the script falls back to the latest daily issue.
• scripts/fetch_candidates.py uses configured Google News RSS search queries as a feed-based ingestion layer.
• scripts/fetch_candidates.py now also supplements the Google News query layer with direct newsletter discovery from Nature Briefing, MIT Technology Review's The Download, and Superpower Daily, plus newsletter-focused domain queries for Morning Brew and 1440.
• Google News RSS is currently the discovery layer for automated candidate gathering; the intended editorial sources remain the underlying publishers and institutions listed in sources.md.
• scripts/prepare_editorial_packet.py turns sources.md, the benchmark issue, the template, and the current discovery snapshot into a dated run packet for Codex-driven source-first drafting.
• scripts/check_pipeline_inputs.py now fails the pipeline before draft generation when source coverage or live market data falls below minimum thresholds, and it writes blocking review artifacts instead of letting placeholder text flow into a draft issue.
• scripts/generate_issue.py produces a draft issue automatically; you can still edit the Markdown before sending.
• scripts/ai_generate_issue.py upgrades that draft into a more readable issue when NEWSLETTER_AI_API_TOKEN is configured.
• scripts/review_issue.py blocks the pipeline when low-quality feed artifacts or obvious placeholders remain in the draft.
• scripts/ai_review_issue.py adds an editorial release gate and can block automatic delivery if the issue is not ready.
• scripts/newsletter_command.py now propagates quality_policy values from config/newsletter_profile.json, so require_ai and the minimum review score are actually enforced during remote runs.
• The recommended production mode is now: NEWSLETTER_USE_AI=true, NEWSLETTER_REQUIRE_AI=true, and NEWSLETTER_AI_REVIEW_MIN_SCORE=90.
• scripts/build_archive.py builds a Jekyll archive with a current-issue landing page, browseable archive, and client-side keyword search.
• scripts/run_daily_pipeline.py is the one-command daily automation entry point.
• scripts/newsletter_command.py is the remote-friendly wrapper around the pipeline and is the recommended command to trigger over SSH or any other remote shell.
• the daily pipeline now fails closed in two places: preflight blocks draft generation when upstream inputs are too weak, and draft review blocks preview/build/send when low-quality artifacts remain
• if NEWSLETTER_REQUIRE_AI=true, the pipeline also fails closed when the AI editorial layer is unavailable
• config/newsletter_profile.json is where you keep the standing editorial instructions that the AI drafting layer should follow on every run
• sources.md is the current reference for where each newsletter section is drawing material from.
• selection_criteria.md defines how stories should be prioritized and tailored to the reader's interests.
• story_scorecard.md provides a lightweight scoring workflow for deciding whether a story should become a main entry, short take, or be excluded.
• daily_workflow.md ties the sourcing, scoring, drafting, and publishing steps into one repeatable editorial process.
• daily_issue_template.md provides the working structure for drafting each new issue.