Skip to content

📊 Update Equaldex#6123

Draft
paarriagadap wants to merge 37 commits into
data-lgbti-policy-v2from
data-equaldex-update-481707-private
Draft

📊 Update Equaldex#6123
paarriagadap wants to merge 37 commits into
data-lgbti-policy-v2from
data-equaldex-update-481707-private

Conversation

@paarriagadap
Copy link
Copy Markdown
Contributor

@paarriagadap paarriagadap commented May 14, 2026

Summary

Tracks: owid/owid-issues#2396

Update the Equaldex LGBT+ rights dataset from the 2025-04-07 snapshot to 2026-05-14. Refresh of the legal-status data for 16 LGBT+ issues across 195 countries, plus a new tracked issue (hate-crime-protections) and a refactor of the snapshot pipeline so future updates are one command instead of three.

Headline numbers

  • 17 published charts upgraded on staging (etl indicator-upgrade auto ✓).
  • 257 grapher variables ingested (2 new top-level indicators + their categorical count/pop aggregates).
  • Long-format snapshot: 47,350 → 58,030 rows (+22.6%); year range 1950–2025 → 1950–2026.
  • Same-sex marriage now legal in 39 countries (was 38).
  • New: hate-crime-protections issue (~1,200 historical rows).

Snapshot diff
File Old rows New rows Δ
equaldex.csv (long-format) 47,350 58,030 +22.6%
equaldex_current.csv 3,570 3,808 +6.7%
equaldex_indices.csv 238 238 0

Schema: unchanged across all three files.

Categorical surface change (value_formatted column):

  • New Illegal (other penalty) for homosexuality — already covered by the existing CATEGORIES_RENAMING map (no garden change).
  • New Death penalty as punishment for censorship — added a mapping entry so 41 rows aren't silently dropped.
  • Brand-new issue hate-crime-protections — full CATEGORIES_RENAMING block added (six categorical values, ordered liberal → restrictive).

Other changes: Equaldex renamed the region French Southern TerritoriesFrench Southern and Antarctic Lands; .countries.json updated to match.

Year range: equaldex.csv extended from 1950–2025 to 1950–2026; indices file's year_extraction moved from 2025 → 2026.

File size: main long-format CSV grew from 14.1 MB → 19.2 MB (+36%).

Meadow step

etlr succeeded on the first try (0.6 s, no errors). No code changes needed:

  • Dedup-by-(country, year, issue) block absorbed the new hate-crime-protections historical rows without producing duplicate index keys.
  • No mixed-type object-column repacker issues.
  • The carried-over safe_types=False on the main read was already sufficient.

Output diff highlights:

  • Adds the new hate-crime-protections issue with historical coverage back to 1985.
  • Deeper military history for several countries.
  • Corrects adoption values for the United States across 2017–2025 (upstream change).
  • Three current rows lost their 2025 entries for conversion-therapy, housing-discrimination, and intersex-infant-surgery — these still exist historically; the producer simply doesn't have a 2025 "current" reading for them.
Garden step

Two preemptive edits to etl/steps/data/garden/lgbt_rights/2026-05-14/equaldex.py (commit 040844c0f):

  1. CATEGORIES_RENAMING['hate_crime_protections'] — new block, six values ordered from "Sexual orientation and gender identity" (most protective) down to "No protections", matching the convention of the existing discrimination-style blocks. Without this, the new issue would have failed the post-pivot column-set assertion.
  2. CATEGORIES_RENAMING['censorship'] — added "Death penalty as punishment": "Death penalty as punishment" so the new value (40 historical + 1 current row) isn't silently dropped by map_series.

Two new indicator blocks added to the garden .meta.yml:

  • hate_crime_protections (historical) and hate_crime_protections_current — mirror the structure of discrimination / discrimination_current (title with (historical) / (current) suffix, presentation.title_public, display.name, etc.).

Greenland-from-aggregates fix (commit 29aec8659). The legacy Greenland workaround re-appended Greenland's rows inside select_only_sovereign_countries, which meant Greenland was being summed into Europe / World aggregates. The fix moves the snapshot/re-append to run() and does it after both region-aggregation functions, so Greenland still shows on country-level / map charts but no longer contributes to regional totals. Effect on 2025 totals, for example: Europe homosexuality_legal_count 44 → 43; World 111 → 110.

Garden runs clean (2.7 s, only routine PerformanceWarning and region-aggregation NaN warnings). Garden diff vs the 2025 build is consistent with the upstream data changes above — no unexpected NaNs or value-range shifts in the indices.

Country harmonization audit

One real bug surfaced and fixed (commit 390fa0f9d):

  • Equaldex now ships French Southern and Antarctic Lands where it used to send French Southern Territories. The harmonization map's source key was updated; the target stays French Southern Territories (the canonical OWID region name).
  • After the fix: zero harmonization warnings; zero garden-output entities outside OWID's canonical regions/income groups (out of 202 entities).
  • One mapping entry (Pitcairn Islands) targets a non-canonical name, but that entity is filtered out by select_only_sovereign_countries before reaching Grapher, so it's harmless. Mapping kept as documentation.
  • No .excluded_countries.json for this step (N/A).

Full audit: workbench/equaldex/harmonization_audit.md.

Grapher step

Built and uploaded to staging cleanly (no code changes needed). 257 variables ingested, including the two new hate_crime_protections* top-level indicators and their 14 categorical breakdown aggregates (count + population variants). No ghost-variable warnings.

etl indicator-upgrade auto then upgraded 22 charts (17 published + 5 drafts) from the 2025 dataset IDs to the new 2026 IDs on staging.

Metadata quality fixes

Commits 5135b4b08, 8f52f5e4a, da3da26a6, 6e55fc873, a0eb8d493:

  • Added presentation.attribution_short: Equaldex and display.numDecimalPlaces: 0 to definitions.common — propagates to all 257 indicators (previously none had these set).
  • Sentence-cased "Don't Ask, Don't Tell""Don't ask, don't tell" in the military_current description key (consistency with the historical variable and OWID's sentence-case rule).
  • Removed a stray "do" in intersex_infant_surgery_current.description_short ("that do differ from" → "that differ from").
  • Dataset intro bullet added as the first description_key on every indicator, via a reusable {definitions.description_key_dataset_intro} Jinja reference (single source of truth, prepended to the four indicators that carry their own description_key lists).
  • Slimmed index descriptions: ei, ei_legal, ei_po each now carry just two bullets — the shared dataset intro, plus a one-sentence definition of the index combined with a "For more information, visit the [Equaldex Equality Index methodology]" pointer.
  • description_processing now documents the Greenland exception inline on the sovereignty bullet ("Greenland is the one exception: we keep it as its own country row but exclude it from regional aggregates").
  • description_from_producer_legal_index policy count bumped from "15" → "16" to match Equaldex's own weights table on the methodology page (the producer's prose still says "13", but their table lists 16 — we side with the table).

Style guide / typo / Jinja-spacing scans all clean. Indicator metadata coverage check: zero missing-mandatory-field issues across the 257 variables. dataset.update_period_days: 365 is in place. All external URLs resolve.

Greenland handling (NOTE in equaldex.py)

Greenland is treated as a Danish territory by the sovereignty source (Butcher and Griffiths 2020) and by OWID's regions dataset, but Equaldex publishes a Greenland-specific record. The handling:

  • Country-level charts: Greenland appears as its own row (verified — 77 rows, 1950–2026, with the raw status columns populated).
  • Regional aggregates: Greenland is excluded from Europe / World / etc. The snapshot/re-append happens in run() after the two region-aggregation functions, not inside select_only_sovereign_countries as it used to (commit 29aec8659).
  • The # NOTE: comment in equaldex.py remains in place as a heads-up to the next maintainer; ISD's isd_countries table still doesn't include Greenland in any year, so the workaround is still load-bearing.
Snapshot pipeline refactor (engineering)

Replaced the three per-file upload scripts (equaldex.py, equaldex_current.py, equaldex_indices.py) with a single self-contained snapshots/lgbt_rights/2026-05-14/equaldex_extract.py following the un/<ver>/ilostat_extract.py pattern (commits 16375bc6d + 2510c305b):

  • One click command: etls lgbt_rights/<version>/equaldex_extract hits the API, builds all three frames, and uploads via Snapshot(...).create_snapshot(data=df, upload=upload).
  • tenacity retry + joblib.Memory disk cache (.cache/equaldex/<SNAPSHOT_VERSION>/) for resumability. The version is part of the cache path (commit d8dea320d, after a Codex P1) so a future run at a new SNAPSHOT_VERSION starts with a fresh cache instead of silently reusing the previous year's API payloads.
  • Missing EQUALDEX_KEY now hard-errors instead of logging-and-continuing with apiKey=None.
  • country_list.json reformatted one entry per line for reviewable diffs.

Future updates no longer require the manual "write CSVs locally, then etls --path-to-file three times" workflow.

Codex review

One P1 from Codex, addressed and resolved:

  • Cache freshness in equaldex_extract.pyfetch_region was memoized to .cache/equaldex with no freshness key, so a future run at a new SNAPSHOT_VERSION would silently reuse the previous year's API payloads. Fixed in commit d8dea320d by moving the cache root to .cache/equaldex/<SNAPSHOT_VERSION>/ — retries within a single extraction still reuse fetched regions, but next year's run gets a clean namespace. Thread PRRT_kwDOGEqCzM6CT6WI resolved.
Slack announcement draft
# Data update comms draft — Equaldex

Source: `lgbt_rights/2026-05-14/equaldex` · Branch: `data-equaldex-update-481707-private` · Generated: 2026-05-15T08:45:17Z

---

## What dataset(s) did you update?

Equaldex — Equaldex

## When was this data released? When is the next scheduled release / our plan for next update?

Released: rolling — Equaldex is a continuously updated crowdsourced database; data fetched on 2026-05-15. Next: there is no fixed release schedule; OWID re-pulls roughly once a year.

## Who is the data source(s)? Is there anything our users should know about them?

Equaldex. The data is crowdsourced by Equaldex contributors and gathered from the official Equaldex JSON API — "LGBT Rights by Country & Travel Guide | Equaldex. (2026). https://www.equaldex.com/".

## What's the coverage of the data in terms of years and countries/regions?

Covers 1950–2026, 195 countries, plus the seven OWID continental aggregates (Africa, Asia, Europe, North America, Oceania, South America, World). Coverage by issue varies — newer policy areas like hate-crime protections have fewer historical observations than older ones like the legality of same-sex sexual activity.

## How many charts did this update affect?

17 published charts (moderate).

## What does this dataset help our users understand about the world, and why is it important they know that?

- Equaldex tracks LGBT+ rights across 16 distinct legal issues — from the legality of same-sex sexual activity and marriage, to employment and housing discrimination, gender-affirming care, hate-crime protections, military service, and more — for nearly every country in the world. It's the most comprehensive picture of where LGBT+ people are protected by law and where they aren't.
- Legal protection is still highly uneven across regions. Same-sex sexual activity is criminalized in 61 countries (≈ 31% of those covered), and in seven of those, the death penalty is on the books.
- Same-sex marriage has expanded substantially over the last two decades: 39 countries now recognize it, up from 38 in last year's snapshot. At the other end, 99 countries explicitly prohibit it.
- The dataset includes a composite **Equality Index** (with separate legal and public-opinion components) that turns the policy landscape into a single number per country, making it easier to compare countries and track progress over time.
- History reaches back to the 1950s in many cases, so users can see the long arc of legalisation, decriminalisation, and reversals over the post-war period.

## Any important caveats or pitfalls in interpretation that users should know about this data? (optional)

- Equaldex is crowdsourced — coverage and timeliness vary by country, and contributors occasionally revise historical entries. This release retroactively reclassified US `adoption` rights from "Legal" to "Varies by region" for 2017–2025, a reminder that older years can shift between snapshots.
- Where Equaldex doesn't record the date a policy was changed, we assume the current status has held throughout — which can over-extend recently codified policies backward into earlier decades.
- We restrict the dataset to sovereign states (as defined by Butcher and Griffiths 2020), so non-sovereign territories are excluded — with one manual exception for Greenland, which the sovereignty source omits but which we keep because Equaldex publishes a Greenland-specific record.

## Anything interesting to note about this update, including what you had to do? Anything else you'd like to add? (optional)

- **New issue tracked: hate-crime protections.** Equaldex added explicit hate-crime-protection coverage in 2024–2025, and this release brings ~1,200 historical rows of it into OWID for the first time. Two new chart-ready indicators (`hate_crime_protections`, `hate_crime_protections_current`) now sit alongside the existing 14 LGBT+ issues.
- **+1 country recognizing same-sex marriage** since the last snapshot — the legal-marriage tally moves from 38 to 39 countries.
- **One more recent year of coverage:** the dataset's range extended from 1950–2025 to 1950–2026, with 22% more rows in the long-format file.
- **Equaldex renamed `French Southern Territories``French Southern and Antarctic Lands`** — the harmonization map was updated to match.
- **Snapshot pipeline refactor (engineering note):** the three legacy upload scripts were merged into a single self-contained `equaldex_extract.py` that hits the API, builds all three frames, and uploads them in one go. Future updates are now `etls lgbt_rights/<version>/equaldex_extract` without manual file-shuffling.

## Add 1–3 chart views we might use in the public announcement

1. **Legal status of same-sex sexual acts**`same-sex-sexual-acts-equaldex` — flagship indicator, immediately visual on the world map, headline figure for the dataset, still highly contested in 60+ countries
2. **Marriage for same-sex partners**`marriage-same-sex-partners-equaldex` — landmark policy that has expanded substantially in recent years; one more country has legalized it since the 2025 snapshot
3. **LGBT+ Legal Equality Index**`lgbt-legal-equality-index` — composite index summarizing the whole dataset (legality of 15 policies, weighted), single chart that captures the broad picture

## Link to the updated charts as a search result (not a chart collection anymore). Ask Charlie if you need help with this. (optional)

https://ourworldindata.org/search?datasetProducts=Equaldex
Data update post (for OWID /latest)
title: Where in the world are LGBT+ rights protected, and where are they restricted?
excerpt: Explore updated data on LGBT+ rights from Equaldex, plus new coverage of hate-crime protections.
type: announcement
authors: Pablo Arriagada
kicker: Data update

\[+body\]

How does the legal status of LGBT+ people compare across countries, and how has it changed over time?

[Equaldex](https://www.equaldex.com/) is a crowdsourced database that tracks LGBT+ rights laws in 195 countries. It covers 16 distinct legal issues — from the legality of same-sex sexual activity and marriage, to employment and housing discrimination, gender-affirming care, military service, and *hate-crime protections*, which we're now able to chart for the first time.

The picture is mixed. Same-sex sexual activity is legal in 130 countries — about two-thirds of those covered — but criminalized in 61, including seven where the death penalty is still on the books. Same-sex marriage is now recognized in 39 countries, up from 38 last year, while 99 countries explicitly prohibit it.

Keep in mind that this data is crowdsourced and updated by Equaldex contributors, so coverage and timeliness vary by country, and older entries are sometimes revised between releases.

I recently updated our charts with the latest data from Equaldex.

{.cta}
url: https://ourworldindata.org/search?datasetProducts=Equaldex
text: Explore the updated data in our interactive charts
{}

:skip
**👉 Add a picture here.** Attach a chart screenshot to the Google Doc and use the filename below.
:endskip

{.image}
filename: 2026-05-data-update-equaldex.png
{}

\[\]

Test plan

  • Snapshot extract runs cleanly (etls lgbt_rights/2026-05-14/equaldex_extract)
  • Meadow / garden / grapher steps run cleanly with built-in change detection
  • Garden output: 202 entities (195 countries + 7 region aggregates), 1950–2026, no entities outside canonical regions
  • Grapher: 257 variables uploaded to staging
  • etl indicator-upgrade auto upgraded 22 charts (17 published)
  • All metadata checks pass: typos, Jinja spacing, style guide, coverage, URL HEAD
  • Anomalist on staging: please review at staging-site-data-equaldex-update-481707/etl/wizard/anomalist
  • Chart Diff on staging: please review at staging-site-data-equaldex-update-481707/etl/wizard/chart-diff

🤖 Generated with Claude Code

paarriagadap and others added 2 commits May 14, 2026 17:25
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@owidbot
Copy link
Copy Markdown
Contributor

owidbot commented May 14, 2026

Quick links (staging server):

Site Dev Site Preview Admin Wizard Docs Docs Preview

Login: ssh owid@staging-site-data-equaldex-update-481707

chart-diff: ❌
  • 0/23 reviewed charts
  • Modified: 0/22
  • New: 0/1
  • Rejected: 0
  • Data changes: 0
  • Metadata changes: 2
data-diff: ❌ Found differences
= Dataset garden/countries/2023-09-25/isd
  = Table isd
  = Table isd_countries
  = Table isd_regions
    ~ Column population (changed metadata)
+       + licenses:
+       +   - name: CC BY 3.0
+       +     url: https://dataportaal.pbl.nl/downloads/HYDE/HYDE3.2/readme_release_HYDE3.2.1.txt
~ Dataset garden/demography/2023-03-31/population
+   + licenses:
+   +   - name: CC BY 3.0
+   +     url: https://dataportaal.pbl.nl/downloads/HYDE/HYDE3.2/readme_release_HYDE3.2.1.txt
  = Table population
    ~ Column population (changed metadata)
+       + licenses:
+       +   - name: CC BY 3.0
+       +     url: https://dataportaal.pbl.nl/downloads/HYDE/HYDE3.2/readme_release_HYDE3.2.1.txt
    ~ Column world_pop_share (changed metadata)
+       + licenses:
+       +   - name: CC BY 3.0
+       +     url: https://dataportaal.pbl.nl/downloads/HYDE/HYDE3.2/readme_release_HYDE3.2.1.txt
  = Table population_original
    ~ Column population (changed metadata)
+       + licenses:
+       +   - name: CC BY 3.0
+       +     url: https://dataportaal.pbl.nl/downloads/HYDE/HYDE3.2/readme_release_HYDE3.2.1.txt
    ~ Column world_pop_share (changed metadata)
+       + licenses:
+       +   - name: CC BY 3.0
+       +     url: https://dataportaal.pbl.nl/downloads/HYDE/HYDE3.2/readme_release_HYDE3.2.1.txt
~ Dataset garden/hyde/2017/baseline
+   + sources:
+   +   - name: PBL Netherlands Environmental Assessment Agency
+   +     description: |-
+   +       HYDE is an internally consistent combination of updated historical population (gridded) estimates and land use for the past 12,000 years. Categories include cropland, with a new distinction into irrigated and rain fed crops (other than rice) and irrigated and rain fed rice. Also grazing lands are provided, divided into more intensively used pasture, converted rangeland and non-converted natural (less intensively used) rangeland. Population is represented by maps of total, urban, rural population and population density as well as built-up area.
+   +     url: https://www.pbl.nl/en/image/links/hyde
+   +     source_data_url: https://dataportaal.pbl.nl/downloads/HYDE/HYDE3.2/baseline.zip
+   +     date_accessed: '2021-10-01'
+   +     publication_year: 2017
+   + licenses:
+   +   - name: CC BY 3.0
+   +     url: https://dataportaal.pbl.nl/downloads/HYDE/HYDE3.2/readme_release_HYDE3.2.1.txt
  = Table population
    ~ Column population (changed metadata)
-       - {}
+       + licenses:
+       +   - name: CC BY 3.0
+       +     url: https://dataportaal.pbl.nl/downloads/HYDE/HYDE3.2/readme_release_HYDE3.2.1.txt
+ Dataset garden/lgbt_rights/2026-05-11/lgbti_national_policy_dataset
+ + Table lgbti_national_policy_dataset
+   + Column proportion
+ + Table lgbti_national_policy_dataset_combined
+   + Column age_of_consent
+   + Column blood_donations
+   + Column civil_unions
+   + Column constitutional_protections
+   + Column constitutional_protections_gender_identity
+   + Column constitutional_protections_sexual_orientation
+   + Column conversion_therapies
+   + Column death_penalty
+   + Column employment_discrimination
+   + Column employment_discrimination_gender_identity
+   + Column employment_discrimination_sexual_orientation
+   + Column gender_affirming_care
+   + Column gender_assignment_surgeries_on_children
+   + Column gender_marker_change
+   + Column goods_services_discrimination
+   + Column goods_services_discrimination_gender_identity
+   + Column goods_services_discrimination_sexual_orientation
+   + Column hate_crime_protections
+   + Column hate_crime_protections_gender_identity
+   + Column hate_crime_protections_sexual_orientation
+   + Column incitement_to_hatred
+   + Column joint_adoption
+   + Column lgb_military_join
+   + Column lgbt_military
+   + Column lgbt_military_no_enforcement
+   + Column lgbtq_civil_society_restrictions
+   + Column marriage
+   + Column morality_propaganda
+   + Column religious_exemption_laws
+   + Column same_sex_acts
+   + Column third_gender_recognition
+   + Column transgender_military
+ + Table lgbti_national_policy_dataset_combined_regions
+   + Column age_of_consent_equal_count
+   + Column age_of_consent_equal_pop
+   + Column age_of_consent_no_legal_provisions_count
+   + Column age_of_consent_no_legal_provisions_pop
+   + Column age_of_consent_unequal_count
+   + Column age_of_consent_unequal_pop
+   + Column age_of_consent_varies_by_region_count
+   + Column age_of_consent_varies_by_region_pop
+   + Column blood_donations_deferral_or_ban_count
+   + Column blood_donations_deferral_or_ban_pop
+   + Column blood_donations_no_msm_restrictions_count
+   + Column blood_donations_no_msm_restrictions_pop
+   + Column blood_donations_no_policy_count
+   + Column blood_donations_no_policy_pop
+   + Column civil_unions_legally_recognized_count
+   + Column civil_unions_legally_recognized_pop
+   + Column civil_unions_not_recognized_count
+   + Column civil_unions_not_recognized_pop
+   + Column civil_unions_varies_by_region_count
+   + Column civil_unions_varies_by_region_pop
+   + Column constitutional_protections_both_protected_count
+   + Column constitutional_protections_both_protected_pop
+   + Column constitutional_protections_gender_identity_constitutionally_protected_count
+   + Column constitutional_protections_gender_identity_constitutionally_protected_pop
+   + Column constitutional_protections_gender_identity_not_protected_count
+   + Column constitutional_protections_gender_identity_not_protected_pop
+   + Column constitutional_protections_not_protected_count
+   + Column constitutional_protections_not_protected_pop
+   + Column constitutional_protections_sexual_orientation_constitutionally_protected_count
+   + Column constitutional_protections_sexual_orientation_constitutionally_protected_pop
+   + Column constitutional_protections_sexual_orientation_not_protected_count
+   + Column constitutional_protections_sexual_orientation_not_protected_pop
+   + Column constitutional_protections_sexual_orientation_only_count
+   + Column constitutional_protections_sexual_orientation_only_pop
+   + Column conversion_therapies_banned_count
+   + Column conversion_therapies_banned_pop
+   + Column conversion_therapies_not_banned_count
+   + Column conversion_therapies_not_banned_pop
+   + Column conversion_therapies_varies_by_region_count
+   + Column conversion_therapies_varies_by_region_pop
+   + Column death_penalty_death_penalty_applies_count
+   + Column death_penalty_death_penalty_applies_pop
+   + Column death_penalty_no_death_penalty_count
+   + Column death_penalty_no_death_penalty_pop
+   + Column death_penalty_varies_by_region_count
+   + Column death_penalty_varies_by_region_pop
+   + Column employment_discrimination_both_protected_count
+   + Column employment_discrimination_both_protected_pop
+   + Column employment_discrimination_gender_identity_discrimination_prohibited_count
+   + Column employment_discrimination_gender_identity_discrimination_prohibited_pop
+   + Column employment_discrimination_gender_identity_no_protection_count
+   + Column employment_discrimination_gender_identity_no_protection_pop
+   + Column employment_discrimination_gender_identity_only_count
+   + Column employment_discrimination_gender_identity_only_pop
+   + Column employment_discrimination_gender_identity_varies_by_region_count
+   + Column employment_discrimination_gender_identity_varies_by_region_pop
+   + Column employment_discrimination_no_protection_count
+   + Column employment_discrimination_no_protection_pop
+   + Column employment_discrimination_sexual_orientation_discrimination_prohibited_count
+   + Column employment_discrimination_sexual_orientation_discrimination_prohibited_pop
+   + Column employment_discrimination_sexual_orientation_no_protection_count
+   + Column employment_discrimination_sexual_orientation_no_protection_pop
+   + Column employment_discrimination_sexual_orientation_only_count
+   + Column employment_discrimination_sexual_orientation_only_pop
+   + Column employment_discrimination_sexual_orientation_varies_by_region_count
+   + Column employment_discrimination_sexual_orientation_varies_by_region_pop
+   + Column employment_discrimination_varies_by_region_count
+   + Column employment_discrimination_varies_by_region_pop
+   + Column gender_affirming_care_adults_covered__minors_restricted_count
+   + Column gender_affirming_care_adults_covered__minors_restricted_pop
+   + Column gender_affirming_care_covered_for_adults_and_minors_count
+   + Column gender_affirming_care_covered_for_adults_and_minors_pop
+   + Column gender_affirming_care_covered_for_adults_only_count
+   + Column gender_affirming_care_covered_for_adults_only_pop
+   + Column gender_affirming_care_neither_covered_nor_restricted_count
+   + Column gender_affirming_care_neither_covered_nor_restricted_pop
+   + Column gender_affirming_care_restricted_for_both_count
+   + Column gender_affirming_care_restricted_for_both_pop
+   + Column gender_affirming_care_varies_by_region_or_other_count
+   + Column gender_affirming_care_varies_by_region_or_other_pop
+   + Column gender_assignment_surgeries_on_children_non_consensual_surgeries_banned_count
+   + Column gender_assignment_surgeries_on_children_non_consensual_surgeries_banned_pop
+   + Column gender_assignment_surgeries_on_children_not_banned_count
+   + Column gender_assignment_surgeries_on_children_not_banned_pop
+   + Column gender_assignment_surgeries_on_children_varies_by_region_count
+   + Column gender_assignment_surgeries_on_children_varies_by_region_pop
+   + Column gender_marker_change_legally_possible__requirement_unknown_count
+   + Column gender_marker_change_legally_possible__requirement_unknown_pop
+   + Column gender_marker_change_medical_or_psychological_diagnosis_count
+   + Column gender_marker_change_medical_or_psychological_diagnosis_pop
+   + Column gender_marker_change_not_legally_possible_count
+   + Column gender_marker_change_not_legally_possible_pop
+   + Column gender_marker_change_self_declaration_count
+   + Column gender_marker_change_self_declaration_pop
+   + Column gender_marker_change_surgery_and_sterilization_required_count
+   + Column gender_marker_change_surgery_and_sterilization_required_pop
+   + Column gender_marker_change_surgery_required_count
+   + Column gender_marker_change_surgery_required_pop
+   + Column gender_marker_change_varies_by_region_count
+   + Column gender_marker_change_varies_by_region_pop
+   + Column goods_services_discrimination_both_protected_count
+   + Column goods_services_discrimination_both_protected_pop
+   + Column goods_services_discrimination_gender_identity_discrimination_prohibited_count
+   + Column goods_services_discrimination_gender_identity_discrimination_prohibited_pop
+   + Column goods_services_discrimination_gender_identity_no_protection_count
+   + Column goods_services_discrimination_gender_identity_no_protection_pop
+   + Column goods_services_discrimination_gender_identity_only_count
+   + Column goods_services_discrimination_gender_identity_only_pop
+   + Column goods_services_discrimination_gender_identity_varies_by_region_count
+   + Column goods_services_discrimination_gender_identity_varies_by_region_pop
+   + Column goods_services_discrimination_no_protection_count
+   + Column goods_services_discrimination_no_protection_pop
+   + Column goods_services_discrimination_sexual_orientation_discrimination_prohibited_count
+   + Column goods_services_discrimination_sexual_orientation_discrimination_prohibited_pop
+   + Column goods_services_discrimination_sexual_orientation_no_protection_count
+   + Column goods_services_discrimination_sexual_orientation_no_protection_pop
+   + Column goods_services_discrimination_sexual_orientation_only_count
+   + Column goods_services_discrimination_sexual_orientation_only_pop
+   + Column goods_services_discrimination_sexual_orientation_varies_by_region_count
+   + Column goods_services_discrimination_sexual_orientation_varies_by_region_pop
+   + Column goods_services_discrimination_varies_by_region_count
+   + Column goods_services_discrimination_varies_by_region_pop
+   + Column hate_crime_protections_both_covered_count
+   + Column hate_crime_protections_both_covered_pop
+   + Column hate_crime_protections_gender_identity_covered_by_hate_crime_laws_count
+   + Column hate_crime_protections_gender_identity_covered_by_hate_crime_laws_pop
+   + Column hate_crime_protections_gender_identity_not_covered_count
+   + Column hate_crime_protections_gender_identity_not_covered_pop
+   + Column hate_crime_protections_gender_identity_varies_by_region_count
+   + Column hate_crime_protections_gender_identity_varies_by_region_pop
+   + Column hate_crime_protections_not_covered_count
+   + Column hate_crime_protections_not_covered_pop
+   + Column hate_crime_protections_sexual_orientation_covered_by_hate_crime_laws_count
+   + Column hate_crime_protections_sexual_orientation_covered_by_hate_crime_laws_pop
+   + Column hate_crime_protections_sexual_orientation_not_covered_count
+   + Column hate_crime_protections_sexual_orientation_not_covered_pop
+   + Column hate_crime_protections_sexual_orientation_only_count
+   + Column hate_crime_protections_sexual_orientation_only_pop
+   + Column hate_crime_protections_sexual_orientation_varies_by_region_count
+   + Column hate_crime_protections_sexual_orientation_varies_by_region_pop
+   + Column hate_crime_protections_varies_by_region_count
+   + Column hate_crime_protections_varies_by_region_pop
+   + Column incitement_to_hatred_not_prohibited_count
+   + Column incitement_to_hatred_not_prohibited_pop
+   + Column incitement_to_hatred_prohibited_count
+   + Column incitement_to_hatred_prohibited_pop
+   + Column incitement_to_hatred_varies_by_region_count
+   + Column incitement_to_hatred_varies_by_region_pop
+   + Column joint_adoption_not_permitted_count
+   + Column joint_adoption_not_permitted_pop
+   + Column joint_adoption_permitted_count
+   + Column joint_adoption_permitted_pop
+   + Column joint_adoption_varies_by_region_count
+   + Column joint_adoption_varies_by_region_pop
+   + Column lgb_military_join_allowed_count
+   + Column lgb_military_join_allowed_pop
+   + Column lgb_military_join_banned_but_not_enforced_count
+   + Column lgb_military_join_banned_but_not_enforced_pop
+   + Column lgb_military_join_banned_count
+   + Column lgb_military_join_banned_pop
+   + Column lgb_military_join_no_policy_count
+   + Column lgb_military_join_no_policy_pop
+   + Column lgbt_military_allowed_for_lgb_only_count
+   + Column lgbt_military_allowed_for_lgb_only_pop
+   + Column lgbt_military_allowed_for_transgender_only_count
+   + Column lgbt_military_allowed_for_transgender_only_pop
+   + Column lgbt_military_banned_for_lgb_only__not_enforced_count
+   + Column lgbt_military_banned_for_lgb_only__not_enforced_pop
+   + Column lgbt_military_banned_for_lgb_only_count
+   + Column lgbt_military_banned_for_lgb_only_pop
+   + Column lgbt_military_mixed__lgb_allowed__transgender_banned_but_not_enforced_count
+   + Column lgbt_military_mixed__lgb_allowed__transgender_banned_but_not_enforced_pop
+   + Column lgbt_military_no_enforcement_allowed_for_lgb_only_count
+   + Column lgbt_military_no_enforcement_allowed_for_lgb_only_pop
+   + Column lgbt_military_no_enforcement_allowed_for_transgender_only_count
+   + Column lgbt_military_no_enforcement_allowed_for_transgender_only_pop
+   + Column lgbt_military_no_enforcement_banned_for_lgb_only_count
+   + Column lgbt_military_no_enforcement_banned_for_lgb_only_pop
+   + Column lgbt_military_no_enforcement_mixed__lgb_allowed__transgender_banned_count
+   + Column lgbt_military_no_enforcement_mixed__lgb_allowed__transgender_banned_pop
+   + Column lgbt_military_no_enforcement_no_policy_count
+   + Column lgbt_military_no_enforcement_no_policy_pop
+   + Column lgbt_military_no_enforcement_open_service_permitted_count
+   + Column lgbt_military_no_enforcement_open_service_permitted_pop
+   + Column lgbt_military_no_enforcement_service_banned_count
+   + Column lgbt_military_no_enforcement_service_banned_pop
+   + Column lgbt_military_no_policy_count
+   + Column lgbt_military_no_policy_pop
+   + Column lgbt_military_open_service_permitted_count
+   + Column lgbt_military_open_service_permitted_pop
+   + Column lgbt_military_service_banned_count
+   + Column lgbt_military_service_banned_pop
+   + Column lgbtq_civil_society_restrictions_no_restrictions_count
+   + Column lgbtq_civil_society_restrictions_no_restrictions_pop
+   + Column lgbtq_civil_society_restrictions_restrictions_in_effect_count
+   + Column lgbtq_civil_society_restrictions_restrictions_in_effect_pop
+   + Column lgbtq_civil_society_restrictions_varies_by_region_count
+   + Column lgbtq_civil_society_restrictions_varies_by_region_pop
+   + Column marriage_banned_count
+   + Column marriage_banned_pop
+   + Column marriage_legal_count
+   + Column marriage_legal_pop
+   + Column marriage_no_legal_provisions_count
+   + Column marriage_no_legal_provisions_pop
+   + Column marriage_varies_by_region_count
+   + Column marriage_varies_by_region_pop
+   + Column morality_propaganda_no_restrictions_count
+   + Column morality_propaganda_no_restrictions_pop
+   + Column morality_propaganda_restrictions_in_effect_but_not_enforced_count
+   + Column morality_propaganda_restrictions_in_effect_but_not_enforced_pop
+   + Column morality_propaganda_restrictions_in_effect_count
+   + Column morality_propaganda_restrictions_in_effect_pop
+   + Column morality_propaganda_varies_by_region_count
+   + Column morality_propaganda_varies_by_region_pop
+   + Column religious_exemption_laws_no_religious_exemptions_count
+   + Column religious_exemption_laws_no_religious_exemptions_pop
+   + Column religious_exemption_laws_religious_exemptions_in_effect_count
+   + Column religious_exemption_laws_religious_exemptions_in_effect_pop
+   + Column same_sex_acts_criminalized_but_not_enforced_count
+   + Column same_sex_acts_criminalized_but_not_enforced_pop
+   + Column same_sex_acts_criminalized_count
+   + Column same_sex_acts_criminalized_pop
+   + Column same_sex_acts_legal_count
+   + Column same_sex_acts_legal_pop
+   + Column same_sex_acts_no_legal_provisions_count
+   + Column same_sex_acts_no_legal_provisions_pop
+   + Column same_sex_acts_varies_by_region_count
+   + Column same_sex_acts_varies_by_region_pop
+   + Column third_gender_recognition_not_recognized_count
+   + Column third_gender_recognition_not_recognized_pop
+   + Column third_gender_recognition_recognized_count
+   + Column third_gender_recognition_recognized_pop
+   + Column third_gender_recognition_varies_by_region_count
+   + Column third_gender_recognition_varies_by_region_pop
+   + Column transgender_military_allowed_count
+   + Column transgender_military_allowed_pop
+   + Column transgender_military_banned_but_not_enforced_count
+   + Column transgender_military_banned_but_not_enforced_pop
+   + Column transgender_military_banned_count
+   + Column transgender_military_banned_pop
+   + Column transgender_military_no_policy_count
+   + Column transgender_military_no_policy_pop
+ + Table lgbti_national_policy_dataset_regions
+   + Column n_countries_no
+   + Column n_countries_yes
+   + Column pop_no
+   + Column pop_yes
+ Dataset garden/lgbt_rights/2026-05-14/equaldex
+ + Table equaldex
+   + Column homosexuality
+   + Column changing_gender
+   + Column marriage
+   + Column adoption
+   + Column age_of_consent
+   + Column blood
+   + Column censorship
+   + Column conversion_therapy
+   + Column discrimination
+   + Column employment_discrimination
+   + Column housing_discrimination
+   + Column military
+   + Column non_binary_gender_recognition
+   + Column gender_affirming_care
+   + Column intersex_infant_surgery
+   + Column hate_crime_protections
+   + Column homosexuality_current
+   + Column changing_gender_current
+   + Column marriage_current
+   + Column adoption_current
+   + Column age_of_consent_current
+   + Column blood_current
+   + Column censorship_current
+   + Column conversion_therapy_current
+   + Column discrimination_current
+   + Column employment_discrimination_current
+   + Column housing_discrimination_current
+   + Column military_current
+   + Column non_binary_gender_recognition_current
+   + Column gender_affirming_care_current
+   + Column intersex_infant_surgery_current
+   + Column hate_crime_protections_current
+   + Column ei
+   + Column ei_legal
+   + Column ei_po
+   + Column homosexuality_missing_count
+   + Column homosexuality_male_illegal__female_legal_or_uncertain_count
+   + Column homosexuality_illegal__death_penalty_count
+   + Column homosexuality_illegal__prison_or_other_penalty_count
+   + Column homosexuality_legal_count
+   + Column homosexuality_varies_by_region_count
+   + Column homosexuality_ambiguous_count
+   + Column changing_gender_missing_count
+   + Column changing_gender_illegal_count
+   + Column changing_gender_legal__surgery_required_count
+   + Column changing_gender_legal__no_restrictions_count
+   + Column changing_gender_varies_by_region_count
+   + Column changing_gender_legal__medical_diagnosis_required_count
+   + Column changing_gender_ambiguous_count
+   + Column marriage_banned_count
+   + Column marriage_missing_count
+   + Column marriage_civil_union_or_other_partnership_count
+   + Column marriage_legal_count
+   + Column marriage_unrecognized_count
+   + Column marriage_varies_by_region_count
+   + Column marriage_foreign_same_sex_marriages_recognized_only_count
+   + Column marriage_unregistered_cohabitation_count
+   + Column marriage_ambiguous_count
+   + Column adoption_missing_count
+   + Column adoption_individual_only_count
+   + Column adoption_illegal_count
+   + Column adoption_legal_count
+   + Column adoption_second_parent_adoption_only_count
+   + Column adoption_ambiguous_count
+   + Column adoption_varies_by_region_count
+   + Column age_of_consent_missing_count
+   + Column age_of_consent_unequal_count
+   + Column age_of_consent_equal_count
+   + Column age_of_consent_female_equal__male_unequal_or_uncertain_count
+   + Column age_of_consent_varies_by_region_count
+   + Column age_of_consent_ambiguous_count
+   + Column blood_missing_count
+   + Column blood_legal_count
+   + Column blood_banned__1_year_deferral__count
+   + Column blood_varies_by_region_count
+   + Column blood_banned__5_year_deferral__count
+   + Column blood_banned__less_than_6_month_deferral__count
+   + Column blood_banned__indefinite_deferral__count
+   + Column blood_ambiguous_count
+   + Column blood_legal_with_restrictions_count
+   + Column blood_banned__6_month_deferral__count
+   + Column censorship_missing_count
+   + Column censorship_state_enforced_count
+   + Column censorship_imprisonment_as_punishment_count
+   + Column censorship_no_censorship_count
+   + Column censorship_varies_by_region_count
+   + Column censorship_fine_as_punishment_count
+   + Column censorship_death_penalty_as_punishment_count
+   + Column censorship_ambiguous_count
+   + Column conversion_therapy_missing_count
+   + Column conversion_therapy_not_banned_count
+   + Column conversion_therapy_banned_count
+   + Column conversion_therapy_varies_by_region_count
+   + Column conversion_therapy_sexual_orientation_only_count
+   + Column conversion_therapy_ambiguous_count
+   + Column discrimination_missing_count
+   + Column discrimination_no_protections_count
+   + Column discrimination_illegal_count
+   + Column discrimination_illegal_in_some_contexts_count
+   + Column discrimination_varies_by_region_count
+   + Column employment_discrimination_missing_count
+   + Column employment_discrimination_no_protections_count
+   + Column employment_discrimination_sexual_orientation_and_gender_identity_count
+   + Column employment_discrimination_sexual_orientation_only_count
+   + Column employment_discrimination_varies_by_region_count
+   + Column employment_discrimination_gender_identity_only_count
+   + Column employment_discrimination_ambiguous_count
+   + Column housing_discrimination_missing_count
+   + Column housing_discrimination_no_protections_count
+   + Column housing_discrimination_sexual_orientation_and_gender_identity_count
+   + Column housing_discrimination_sexual_orientation_only_count
+   + Column housing_discrimination_varies_by_region_count
+   + Column housing_discrimination_gender_identity_only_count
+   + Column housing_discrimination_ambiguous_count
+   + Column military_missing_count
+   + Column military_illegal_count
+   + Column military_legal_count
+   + Column military_dont_ask__dont_tell_count
+   + Column military_lgb_permitted__transgender_people_banned_count
+   + Column military_ambiguous_count
+   + Column non_binary_gender_recognition_missing_count
+   + Column non_binary_gender_recognition_not_legally_recognized_count
+   + Column non_binary_gender_recognition_varies_by_region_count
+   + Column non_binary_gender_recognition_recognized_count
+   + Column non_binary_gender_recognition_intersex_only_count
+   + Column non_binary_gender_recognition_ambiguous_count
+   + Column gender_affirming_care_missing_count
+   + Column gender_affirming_care_banned_count
+   + Column gender_affirming_care_restricted_count
+   + Column gender_affirming_care_legal__but_restricted_for_minors_count
+   + Column gender_affirming_care_ambiguous_count
+   + Column gender_affirming_care_legal_count
+   + Column gender_affirming_care_legal__but_banned_for_minors_count
+   + Column gender_affirming_care_varies_by_region_count
+   + Column intersex_infant_surgery_missing_count
+   + Column intersex_infant_surgery_not_banned_count
+   + Column intersex_infant_surgery_full_ban_count
+   + Column intersex_infant_surgery_varies_by_region_count
+   + Column intersex_infant_surgery_parental_approval_required_count
+   + Column hate_crime_protections_missing_count
+   + Column hate_crime_protections_no_protections_count
+   + Column hate_crime_protections_sexual_orientation_and_gender_identity_count
+   + Column hate_crime_protections_sexual_orientation_only_count
+   + Column hate_crime_protections_varies_by_region_count
+   + Column hate_crime_protections_ambiguous_count
+   + Column hate_crime_protections_protected_in_some_contexts_count
+   + Column homosexuality_missing_pop
+   + Column homosexuality_male_illegal__female_legal_or_uncertain_pop
+   + Column homosexuality_illegal__death_penalty_pop
+   + Column homosexuality_illegal__prison_or_other_penalty_pop
+   + Column homosexuality_legal_pop
+   + Column homosexuality_varies_by_region_pop
+   + Column homosexuality_ambiguous_pop
+   + Column changing_gender_missing_pop
+   + Column changing_gender_illegal_pop
+   + Column changing_gender_legal__surgery_required_pop
+   + Column changing_gender_legal__no_restrictions_pop
+   + Column changing_gender_varies_by_region_pop
+   + Column changing_gender_legal__medical_diagnosis_required_pop
+   + Column changing_gender_ambiguous_pop
+   + Column marriage_banned_pop
+   + Column marriage_missing_pop
+   + Column marriage_civil_union_or_other_partnership_pop
+   + Column marriage_legal_pop
+   + Column marriage_unrecognized_pop
+   + Column marriage_varies_by_region_pop
+   + Column marriage_foreign_same_sex_marriages_recognized_only_pop
+   + Column marriage_unregistered_cohabitation_pop
+   + Column marriage_ambiguous_pop
+   + Column adoption_missing_pop
+   + Column adoption_individual_only_pop
+   + Column adoption_illegal_pop
+   + Column adoption_legal_pop
+   + Column adoption_second_parent_adoption_only_pop
+   + Column adoption_ambiguous_pop
+   + Column adoption_varies_by_region_pop
+   + Column age_of_consent_missing_pop
+   + Column age_of_consent_unequal_pop
+   + Column age_of_consent_equal_pop
+   + Column age_of_consent_female_equal__male_unequal_or_uncertain_pop
+   + Column age_of_consent_varies_by_region_pop
+   + Column age_of_consent_ambiguous_pop
+   + Column blood_missing_pop
+   + Column blood_legal_pop
+   + Column blood_banned__1_year_deferral__pop
+   + Column blood_varies_by_region_pop
+   + Column blood_banned__5_year_deferral__pop
+   + Column blood_banned__less_than_6_month_deferral__pop
+   + Column blood_banned__indefinite_deferral__pop
+   + Column blood_ambiguous_pop
+   + Column blood_legal_with_restrictions_pop
+   + Column blood_banned__6_month_deferral__pop
+   + Column censorship_missing_pop
+   + Column censorship_state_enforced_pop
+   + Column censorship_imprisonment_as_punishment_pop
+   + Column censorship_no_censorship_pop
+   + Column censorship_varies_by_region_pop
+   + Column censorship_fine_as_punishment_pop
+   + Column censorship_death_penalty_as_punishment_pop
+   + Column censorship_ambiguous_pop
+   + Column conversion_therapy_missing_pop
+   + Column conversion_therapy_not_banned_pop
+   + Column conversion_therapy_banned_pop
+   + Column conversion_therapy_varies_by_region_pop
+   + Column conversion_therapy_sexual_orientation_only_pop
+   + Column conversion_therapy_ambiguous_pop
+   + Column discrimination_missing_pop
+   + Column discrimination_no_protections_pop
+   + Column discrimination_illegal_pop
+   + Column discrimination_illegal_in_some_contexts_pop
+   + Column discrimination_varies_by_region_pop
+   + Column employment_discrimination_missing_pop
+   + Column employment_discrimination_no_protections_pop
+   + Column employment_discrimination_sexual_orientation_and_gender_identity_pop
+   + Column employment_discrimination_sexual_orientation_only_pop
+   + Column employment_discrimination_varies_by_region_pop
+   + Column employment_discrimination_gender_identity_only_pop
+   + Column employment_discrimination_ambiguous_pop
+   + Column housing_discrimination_missing_pop
+   + Column housing_discrimination_no_protections_pop
+   + Column housing_discrimination_sexual_orientation_and_gender_identity_pop
+   + Column housing_discrimination_sexual_orientation_only_pop
+   + Column housing_discrimination_varies_by_region_pop
+   + Column housing_discrimination_gender_identity_only_pop
+   + Column housing_discrimination_ambiguous_pop
+   + Column military_missing_pop
+   + Column military_illegal_pop
+   + Column military_legal_pop
+   + Column military_dont_ask__dont_tell_pop
+   + Column military_lgb_permitted__transgender_people_banned_pop
+   + Column military_ambiguous_pop
+   + Column non_binary_gender_recognition_missing_pop
+   + Column non_binary_gender_recognition_not_legally_recognized_pop
+   + Column non_binary_gender_recognition_varies_by_region_pop
+   + Column non_binary_gender_recognition_recognized_pop
+   + Column non_binary_gender_recognition_intersex_only_pop
+   + Column non_binary_gender_recognition_ambiguous_pop
+   + Column gender_affirming_care_missing_pop
+   + Column gender_affirming_care_banned_pop
+   + Column gender_affirming_care_restricted_pop
+   + Column gender_affirming_care_legal__but_restricted_for_minors_pop
+   + Column gender_affirming_care_ambiguous_pop
+   + Column gender_affirming_care_legal_pop
+   + Column gender_affirming_care_legal__but_banned_for_minors_pop
+   + Column gender_affirming_care_varies_by_region_pop
+   + Column intersex_infant_surgery_missing_pop
+   + Column intersex_infant_surgery_not_banned_pop
+   + Column intersex_infant_surgery_full_ban_pop
+   + Column intersex_infant_surgery_varies_by_region_pop
+   + Column intersex_infant_surgery_parental_approval_required_pop
+   + Column hate_crime_protections_missing_pop
+   + Column hate_crime_protections_no_protections_pop
+   + Column hate_crime_protections_sexual_orientation_and_gender_identity_pop
+   + Column hate_crime_protections_sexual_orientation_only_pop
+   + Column hate_crime_protections_varies_by_region_pop
+   + Column hate_crime_protections_ambiguous_pop
+   + Column hate_crime_protections_protected_in_some_contexts_pop
= Dataset garden/wb/2021-07-01/wb_income
  = Table wb_income_group


Legend: +New  ~Modified  -Removed  =Identical  Details
Hint: Run this locally with etl diff REMOTE data/ --include yourdataset --verbose --snippet

Automatically updated datasets matching excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk are not included

Edited: 2026-05-19 16:17:06 UTC
Execution time: 6.62 seconds

paarriagadap and others added 7 commits May 14, 2026 17:39
Replace the three per-file upload scripts (equaldex.py, equaldex_current.py,
equaldex_indices.py) with a single self-contained equaldex_extract.py:

- One click command (etls lgbt_rights/<version>/equaldex_extract) hits the
  API, builds the three frames, and uploads them directly via
  Snapshot(...).create_snapshot(data=df, upload=upload).
- Per-region API call wrapped with tenacity retry + joblib.Memory disk
  cache so transient failures don't lose progress.
- Missing EQUALDEX_KEY now hard-errors instead of logging-and-continuing
  with apiKey=None.

Removes the manual "write CSVs locally, then run etls --path-to-file three
times" workflow.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pre-commit ruff reformatted a few wrapped lines in the previous commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Update date_accessed to 2026-05-15, date_published to 2026, and
bump citation year to 2026 across all three equaldex snapshot files.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ualdex

- Add `hate_crime_protections` block to CATEGORIES_RENAMING (6 categories,
  liberal → restrictive order, mirroring employment/housing_discrimination)
- Add `Death penalty as punishment` to censorship CATEGORIES_RENAMING block
- Add hate_crime_protections and hate_crime_protections_current indicator
  entries to equaldex.meta.yml (title, description_short, display.name,
  presentation.title_public) following existing discrimination pattern

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…hern and Antarctic Lands

Equaldex now ships this region under "French Southern and Antarctic Lands"
where it previously used "French Southern Territories". Update the source
side of the harmonization map so the entity is captured; the target stays
"French Southern Territories" (the canonical OWID region name).

Without this, the garden step warned about 1 missing + 1 unused mapping entry.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ls, two text fixes

- Add `presentation.attribution_short: Equaldex` and `display.numDecimalPlaces: 0`
  to `definitions.common` so all 257 indicators inherit them (verified in
  the rebuilt grapher catalog: previously none had attribution_short or
  numDecimalPlaces set).
- Sentence-case "Don't Ask, Don't Tell" → "Don't ask, don't tell" in the
  `military_current` description_key bullet, matching the historical
  variable and OWID's sentence-case style guide.
- Drop the stray "do" in `intersex_infant_surgery_current.description_short`
  ("that do differ from" → "that differ from").

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…riting links

Cloudflare-fronted hosts (ourworldindata.org, equaldex.com) can return 404
to curl on URLs that work in browsers. Verified during the equaldex
2026-05-14 update: the embedded chart link returned 404 via curl but 200
via WebFetch and had a fresh Wayback snapshot — the link was fine.

Update the link-verification step:

- Tighten the curl command (`--no-filename` so rg doesn't prepend the
  source path to the URL string, `-L` to follow redirects, `-A` UA,
  longer timeout).
- Add a "non-2xx ≠ broken" subsection: re-check with WebFetch first, then
  Wayback; only act on a real failure; and even then flag-and-ask before
  rewriting an external link.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@paarriagadap
Copy link
Copy Markdown
Contributor Author

@codex review

- Move snapshot+meadow+garden+grapher entries for `lgbt_rights/2025-04-07/equaldex`
  from `dag/equality.yml` to `dag/archive/equality.yml` (preserving the original
  `# Equaldex dataset` section comment).
- Relocate the new `lgbt_rights/2026-05-14/equaldex` block from the bottom of
  `dag/equality.yml` into the slot the old entries occupied, so the section
  stays grouped under its comment.
- Flatten the new block from nested-list form to the modern step-per-key form,
  matching the layout already used by neighbouring entries in this file.

Verified: no remaining references to `lgbt_rights/2025-04-07/equaldex` in the
active DAG; both YAMLs parse cleanly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d53ef9a574

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread snapshots/lgbt_rights/2026-05-14/equaldex_extract.py
paarriagadap and others added 10 commits May 15, 2026 10:39
Wider diffs are reviewable; previously the 238-region list lived on a
single line.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`fetch_region` was memoized to `.cache/equaldex` with no freshness key, so a
future run at a new SNAPSHOT_VERSION would silently reuse the previous year's
API payloads instead of refetching from Equaldex (Codex P1).

Move the cache root to `.cache/equaldex/<SNAPSHOT_VERSION>/` so each refresh
has its own namespace: retries within a single extraction still hit the cache,
but next year's run starts clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ap charts

Move the Greenland snapshot/re-append from `select_only_sovereign_countries`
to `run()` and do it AFTER the region aggregations. Previously Greenland was
added back to the table BEFORE `add_population_weighted_aggregations` and
`add_country_counts_and_population_by_status`, so its rows were summed into
Europe / World totals — incorrect, since the sovereignty source (Butcher and
Griffiths 2020) and OWID's regions dataset both treat Greenland as part of
Denmark.

Now:
- `select_only_sovereign_countries` returns only true sovereign countries.
- Greenland's rows are captured from the harmonized table before the filter.
- They're concatenated back AFTER both aggregation functions, so Greenland
  shows on country-level / map charts but doesn't contribute to Europe or
  World aggregates.

Effect on the 2025 row, for example:
- Europe `homosexuality_legal_count`: 44 → 43
- World `homosexuality_legal_count`: 111 → 110
- Greenland row still present (77 rows, 1950–2026) with raw status columns
  populated; per-status count / pop columns are NaN by design (those are
  region-level indicators).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Shared bullet introducing the dataset (linked producer, ~195-country
coverage, breadth of issues, history depth) is now defined once in
`definitions.description_key_dataset_intro` and surfaced on every
indicator:

- Indicators without their own description_key inherit it from
  `definitions.common.description_key`.
- The four indicators that carry their own description_key
  (`military`, `military_current`, `ei`, `ei_legal`) prepend the shared
  bullet via the `{definitions.description_key_dataset_intro}` Jinja
  reference, so it sits before their indicator-specific bullets.

Verified after rebuild: bullet expands correctly across all 257
indicators (e.g. homosexuality: 1 bullet; military: 2; ei_legal: 6,
intro first).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two follow-ups to the previous commit:

- Reword the shared dataset-intro bullet to lead with "This data comes
  from [Equaldex](https://www.equaldex.com/), …" so the source
  attribution is the first thing a reader sees.
- For the four indicators that carry their own description_key
  (`military`, `military_current`, `ei`, `ei_legal`), put the
  `{definitions.description_key_dataset_intro}` reference as the
  FIRST bullet (was at the end of the list for `ei_legal`), so the
  intro is consistently the opening bullet across all 257 indicators.

Verified after rebuild: position [0] on homosexuality, military
(of 2), ei (of 2), and ei_legal (of 6).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…logy

The Equaldex equality-index methodology page (https://www.equaldex.com/equality-index)
now shows 16 policies in the legal index (Equaldex added hate-crime protections),
not 15. Reflect that across the index indicators and pull in the methodology details
the page provides:

- `ei`: bump "15 policies" → "16 policies". Add caveats that (a) public-opinion data
  isn't available everywhere, so some regions are scored on the legal index alone,
  and (b) the index evolves as data improves and weights are tuned, so cross-release
  comparisons should be cautious.

- `ei_legal`: bump "15 individual policies" → "16" and rewrite the policy list to
  match the methodology page (homosexual activity, marriage, change of legal gender,
  censorship, gender-affirming care, non-binary recognition, hate-crime protections,
  discrimination, employment/housing discrimination, adoption, intersex infant
  surgery, military, blood donations by men who have sex with men, conversion
  therapy, age of consent). Note the top-weighted issues (100 / 60 points) and the
  bottom three (10 points each), and how Equaldex handles unknown / not-applicable /
  within-region-varying laws.

- `ei_po`: previously inherited only the dataset-intro bullet. Add bullets describing
  how the index is built from survey averages, the 75%-per-year time decay applied
  to older surveys, and the coverage caveats (some regions have no survey data;
  global surveys rarely cover transgender topics; conservative regions are surveyed
  on a narrow set of issues).

`description_from_producer_*` blocks left unchanged on purpose — those are the
producer's own wording.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each index now carries two bullets:
- The shared dataset-intro reference (about Equaldex).
- A single combined bullet: a one-sentence definition of the index +
  a "For more information, visit [methodology]" pointer.

Drops the longer methodology / caveat bullets I had added in the previous
commit; details that readers need now live on the producer's methodology
page (already referenced and authoritative).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a bullet to `definitions.common.description_processing` explaining that
Greenland is the one exception to the sovereignty filter: it appears as its
own row (Equaldex tracks it separately) but is excluded from regional
aggregates because the sovereignty source treats it as part of Denmark.

This makes the country-vs-aggregate behaviour explicit on every indicator's
processing description, matching the code in equaldex.py:run().

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Short clause appended to the Butcher-and-Griffiths sovereignty bullet:
"Greenland is the one exception: we keep it as its own country row but
exclude it from regional aggregates." Replaces the separate bullet added
in the previous commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… table (16)

Equaldex's equality-index methodology page lists 16 weighted policies in its
table (it added hate-crime protections), but the producer's prose section
still says "13 different issues" — clearly stale relative to their own table.
Our `description_from_producer_legal_index` originally said "15"; bumping it
to the table count of 16 so the OWID page is internally consistent with
itself and matches reality.

This is the one place where we deviate from strict verbatim reproduction of
the producer's prose because the producer's prose is wrong about their own
data shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@paarriagadap paarriagadap changed the base branch from data-lgbti-policy-v2 to master May 15, 2026 10:35
paarriagadap and others added 4 commits May 15, 2026 11:59
Two concrete additions from the Equaldex iteration:

- **Chart Diff URL** now includes `show_reviewed=` so reviewed charts stay
  visible. Without it the diff list can look empty after the reviewer's
  first pass — surprising and annoying. Document the three filter params
  (`show_reviewed`, `show-narrative-charts`, `show-article-citations`)
  and what each does, since the previous URL only carried two of them.

- **Style guide** picks up four new "avoid" rules captured during the
  draft-and-revise pass with the reviewer:
  - "chart" as a verb
  - "mock up" as the offer-to-build phrasing
  - "<dataset> YYYY-MM-DD update" framing (use "latest X update" instead)
  - Citing the PR number in the body when posting on that same PR

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…dated

Both functions have an explicit deprecation warning in their docstrings
pointing at `paths.regions.add_population(tb)` and
`paths.regions.add_aggregates(tb, ...)` respectively, but the
`detect-outdated-practices` extension wasn't catching either call site.

Adds two new patterns to `vscode_extensions/detect-outdated-practices`,
bumps the extension to 0.0.3, recompiles, repackages, and ships the
new `.vsix` under `install/`. The `/check-outdated-practices` skill
will now surface these on every step pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…s detector

Sweep of etl/data_helpers/geo.py for every docstring that has a "WARNING: ...
deprecated" block. Adds the four remaining public call sites that the
detector wasn't catching yet:

- `geo.add_region_aggregates` → `paths.regions.add_aggregates(tb, ...)`
- `geo.list_countries_in_region` → `paths.regions.get_region(<name>)`
- `geo.list_countries_in_region_that_must_have_data` — deprecated, no
  replacement implemented yet. Flagged so call sites are visible.
- `geo.interpolate_table` → `etl.data_helpers.misc.interpolate_table`

Private helpers (`_load_countries_regions`, `_load_income_groups`,
`_add_population_to_dataframe`) intentionally skipped — internal-only.

Keeps version at 0.0.3 and repackages the .vsix in place.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the five `geo.*` deprecated calls flagged by the outdated-practices
extension with their modern `paths.regions.*` equivalents:

- `geo.add_population_to_table(tb, ds_population=..., warn_on_missing_countries=False)` (×3)
  → `paths.regions.add_population(tb, warn_on_missing_countries=False)`
- `geo.add_regions_to_table(tb, ds_regions=..., regions=..., aggregations=..., frac_allowed_nans_per_year=...)` (×2)
  → `paths.regions.add_aggregates(tb=tb, regions=..., aggregations=..., frac_allowed_nans_per_year=...)`

Cleanup that falls out of the migration:

- `ds_regions = paths.load_dataset("regions")` and `ds_population = paths.load_dataset("population")`
  in `run()` are no longer needed (auto-resolved from the DAG).
- The two helper functions (`add_population_weighted_aggregations` and
  `add_country_counts_and_population_by_status`) drop their `ds_regions: Dataset`
  and `ds_population: Dataset` parameters.
- `from etl.data_helpers import geo` import is removed (unused).
- `Dataset` is dropped from the `owid.catalog` import (no longer referenced).

DAG dependencies on `data://garden/regions/...` and
`data://garden/demography/.../population` remain in place — `paths.regions`
needs them to resolve.

Garden + grapher rebuild clean. Same output as the pre-migration build
(spot-checked Europe `homosexuality_legal_count` = 43 in 2025).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@paarriagadap paarriagadap changed the base branch from master to data-lgbti-policy-v2 May 15, 2026 12:15
Remove the dataset-intro bullet ("This data comes from Equaldex…") from
every indicator. Three pieces deleted together:

- The `definitions.common.description_key` block that injected the intro
  bullet into every indicator without its own list.
- The `definitions.description_key_dataset_intro` Jinja definition.
- The `- "{definitions.description_key_dataset_intro}"` reference at
  the top of the per-indicator description_key lists on `military`,
  `military_current`, `ei`, `ei_legal`, and `ei_po`.

Net effect on the rebuilt catalog:
- Indicators without their own list: 0 description_key bullets (was 1).
- `military` / `military_current`: 1 bullet (the DAT one).
- `ei` / `ei_legal` / `ei_po`: 1 bullet each (index definition + methodology link).

`description_short` and `description_from_producer` carry the dataset-level
context now; the redundant top bullet was crowding the per-indicator copy
without adding new information.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
paarriagadap and others added 12 commits May 15, 2026 17:14
`map_series` was called with both warnings suppressed, so drift between
CATEGORIES_RENAMING and the actual Equaldex categories was completely
silent — phantom dict entries clutter the chart legend's categorical sort,
and a brand-new producer-side category would just NaN-out downstream.

Turn on `warn_on_unused_mappings=True` so phantom entries surface on every
run. Leave `warn_on_missing_mappings=False` because `map_series` lumps NaN
into "missing", which fires on every issue (every country/year without a
value) and would drown out the real signal.

Current run flags four real candidates for follow-up:
- `discrimination[Ambiguous]` — no row in the data
- `intersex_infant_surgery[Ambiguous]` — no row in the data

Plus four entries that look like phantoms on the "current" snapshot but
are legitimate historical statuses (kept):
- `homosexuality[Male illegal, female uncertain]`
- `marriage[Varies by Region]`
- `adoption[Married couples only]`
- `conversion_therapy[Ambiguous]`
- `housing_discrimination[Ambiguous]`

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…phantoms

Two changes:

1. Replace map_series's generic warning text with an explicit pre-loop
   check that names both the table (historical / current) and the issue
   when source values are unmapped or dict keys are phantoms. Output now
   looks like:

     [historical] CATEGORIES_RENAMING['discrimination']: phantom keys not in data: ['Ambiguous']
     [current]    CATEGORIES_RENAMING['homosexuality']: phantom keys not in data: ['Male illegal, female uncertain']

   A phantom on [historical] is a real ghost (drop it); one only on
   [current] is a legitimate historical-only status (keep it).

2. Drop the two real phantoms surfaced by the new check:
   - CATEGORIES_RENAMING['discrimination']['Ambiguous']
   - CATEGORIES_RENAMING['intersex_infant_surgery']['Ambiguous']
   Neither has any row in the snapshot. The chart legend's categorical
   sort no longer carries empty buckets for them.

Historical-pass warnings are now silent; the remaining "[current]"
warnings are expected (real values that just aren't anyone's current
status).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ry phantoms

Restructure `run()` so harmonization and the sovereign-countries filter
(plus the Greenland exception) happen on the long-format meadow tables
BEFORE `make_table_wide_and_map_categories`. The CATEGORIES_RENAMING
coverage warnings now only fire for values that actually reach the
published charts — values that live only on non-sovereign entities like
Vatican City no longer surface in the source-value set.

Two phantom drops surfaced by the cleaner check:

- `CATEGORIES_RENAMING['censorship']['Other punishment']` — the single
  row that ever held this value is Vatican City, which the sovereignty
  filter drops. It never reached any chart anyway.
- `CATEGORIES_RENAMING['adoption']['Married couples only']` — newly
  revealed phantom on the historical pass; only non-sovereign entities
  ever carried this value.

`select_only_sovereign_countries` gains a `keep_extra` argument used
for the Greenland exception — Greenland flows through the same
sovereign-filtered pipeline so its values get mapped, then it gets
snapshotted from the post-merge wide table and excluded from the
regional aggregation pass.

Verified the rebuilt catalog matches:
- 202 entities (sovereign + Greenland + 7 OWID regions).
- Vatican absent; Greenland present.
- Europe 2025 `homosexuality_legal_count` = 43; World 2025 = 110 — same
  as before the restructure.
- `[historical]` phantom warnings: zero remaining.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a `check_phantoms: bool = True` parameter to
`make_table_wide_and_map_categories`. The caller passes `False` for the
"current" snapshot, where a phantom just means "no country currently
holds this status" (still a legitimate historical value) — that's noise,
not actionable.

`[historical]` phantom warnings stay enabled (real drop candidates), and
the unmapped-value warning stays enabled on both tables (the only
safety-critical signal — a brand-new producer-side category not in the
dict).

Net effect: the log channel is now silent under normal conditions, and
fires only when there's a true category-coverage problem.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ruff reformatted the multi-line def signature back onto a single line
after the last commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighter chart subtitles. 14 unique sentences, each on a historical +
current pair (28 lines total):

- "Describes the legislation status of …" → "Legislation status of …"
- "Describes the ability for …" → "Ability for …"
- "Describes the legal status of …" → "Legal status of …"
- "Describes the prohibition of …" → "Prohibition of …"
- "Describes the difference between …" → "Difference between …"
- "Describes the legal recognition of …" → "Legal recognition of …"
- "Describes censorship …" → "Censorship …"

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…cription_short

Append a sentence to both hate_crime_protections (historical) and
hate_crime_protections_current description_short:

  '"Protected in some contexts" means the protection extends only to
   violent crimes (such as assault), or also to lesser offenses (such
   as harassment).'

The category was opaque on its own (Equaldex's site doesn't define it),
and the user wanted the explanation in the chart subtitle rather than
description_key.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`select_only_sovereign_countries` used `pr.merge(tb, tb_sovereign_countries,
on=["country"], how="inner")`, which propagated ISD's origin (Butcher
and Griffiths, International System Dataset 2020) onto every variable
in the equaldex garden output — so every chart would have listed it
in the source attribution.

Replace the merge with an `isin` filter that uses the sovereignty
source only to derive the country set, never touching column metadata.
The same set of countries is kept (verified: 202 entities, Greenland
included; Europe 2025 `homosexuality_legal_count` unchanged at 43).

Note: `paths.regions.add_population(tb)` in the aggregation helpers
still leaks a "Population / Various sources" origin onto the
population-weighted index columns (ei, ei_legal, ei_po). That's a
separate concern.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Equaldex's per-issue indicators now ship aggregates for the four WB
income groups (High-income, Upper-middle-income, Lower-middle-income,
Low-income) alongside the existing continental + World aggregates.
`paths.regions.add_aggregates(...)` auto-resolves the classification
from the `income_groups` dataset listed in the DAG.

Two changes:
- Append the four classification names to `REGIONS` in the garden step.
- Add `data://garden/wb/2025-07-01/income_groups` as a dependency to
  the garden DAG entry.

Verified the rebuilt catalog has 4 new entity rows (one per income
group) with sensible counts — e.g. 2025 `homosexuality_legal_count`:
High-income 55, Upper-middle 38, Lower-middle 15, Low 2 (totals to the
World count of 110).

Total entities now 206 (195 sovereign + Greenland + 7 OWID regions + 4
income groups).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the flat per-step keys with the nested form (announced
internally), where each step is a list-item under its consumer and
multi-line dependencies hang off the parent step:

  data://grapher/lgbt_rights/2026-05-14/equaldex:
    - data://garden/lgbt_rights/2026-05-14/equaldex:
      - data://meadow/lgbt_rights/2026-05-14/equaldex:
        - snapshot://lgbt_rights/2026-05-14/equaldex.csv
        - snapshot://lgbt_rights/2026-05-14/equaldex_indices.csv
        - snapshot://lgbt_rights/2026-05-14/equaldex_current.csv
      - data://garden/demography/2024-07-15/population
      - data://garden/regions/2023-01-01/regions
      - data://garden/wb/2025-07-01/income_groups
      - data://garden/countries/2023-09-25/isd

Dry-run reports "All datasets up to date" — the dependency graph is
unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ions

Rephrase the clarifying sentence on both `hate_crime_protections` and
`hate_crime_protections_current` description_short — "violent crimes
… or also to lesser offenses" was ambiguous; the new wording makes the
exclusion of lesser offenses explicit:

  '"Protected in some contexts" means the protection extends only to
   violent crimes (such as assault) but not to lesser offenses (such
   as harassment).'

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants