Add six new datasets (EHRShot, HIRID, INSPIRE, NWICU, SICdb, eICU) + complete AUMCdb by mmcdermott · Pull Request #299 · Medical-Event-Data-Standard/MEDS-DEV

mmcdermott · 2026-05-12T19:14:09Z

Summary

Replicates the dataset additions from #258 on top of current dev. The upstream branch (May 2025) is too far behind dev to merge directly — it reverts the web-tooling, CI tiers, CLAUDE.md, uv.lock, and other infra that have landed in the meantime. This PR keeps only the dataset/task content and rebases it onto dev as a single commit.

Also makes #243 (NWICU) redundant; that PR can be closed once this lands.

What's added

Six new datasets (src/MEDS_DEV/datasets/<name>/):

EHRShot — Stanford EHR cohort. No automated build — extraction is pre-done; users must prep data manually.
HIRID — Bern ICU via MEDS_extract-HIRID.
INSPIRE — perioperative via MEDS_extract-INSPIRE.
NWICU — Northwestern ICU via NWICU_MEDS.
SICdb — Salzburg ICU via MEDS_extract-SICdb.
eICU — multi-center US ICU via MEDS_extract-eICU (with demo recipe).

AUMCdb completed — was previously just predicates.yaml + README.md; this adds dataset.yaml, requirements.txt, refs.bib, and the full ICU predicate set.

Tasks — mortality/in_icu/first_24h.supported_datasets now includes AUMCdb and NWICU (alongside MIMIC-IV). EHRShot, HIRID, INSPIRE, SICdb, eICU register as datasets but no task references them yet, so they'll just be inert registry entries until tasks are added.

MIMIC-IV README — replaces the previous TODO: Summarize MIMIC-IV stub with the longer description + access-requirements write-up from the upstream PR.

Fixes applied during local review

Beyond replicating the contributor's content, I made these corrections:

UTF-8 mojibake in AUMCdb/predicates.yaml — 4× �mol/l → µmol/l (the predicates would never have matched as-is).
Regex syntax bugs in HIRID, INSPIRE, SICdb predicates — patterns like ^HOSPITAL_ADMISSION* (which matches HOSPITAL_ADMISSIO + zero-or-more Ns) corrected to ^HOSPITAL_ADMISSION//.* to match the working convention used by NWICU/eICU/AUMCdb.
Commented-out predicate stubs removed (3 in INSPIRE, 7 in SICdb, 1 in NWICU, 3 in EHRShot). Dead config; git history preserves the original intent.
MIMIC-IV README — dropped the fictional Supported Tasks list (referenced three task files that don't exist in this repo) and the empty ## MEDS-transformation heading.
Added stub build_demo: echo "Demo not available for this dataset" to dataset.yamls that were missing it, so registry validation passes (matches the pattern HIRID's upstream already used).

Open items needing maintainer / contributor input

Draft because these aren't fixed yet:

MEDS_cohort_dir vs MEDS_output_dir — all new dataset.yamls use MEDS_cohort_dir="{output_dir}"; dev's MIMIC-IV uses MEDS_output_dir="{output_dir}". One of these is stale; needs the extractor-maintainer's call.
HIRID icu_admission / icu_discharge both use ^HOSPITAL_ADMISSION.* / ^HOSPITAL_DISCHARGE.* — same regex as the hospital predicates, so they don't actually distinguish ICU vs hospital events. Likely a content bug in the upstream PR but needs HIRID-MEDS knowledge to fix.
SICdb same issue — icu_admission/icu_discharge collapse to the same ^ADMISSION//.* / ^DISCHARGE//.* as the hospital predicates. The mortality/in_icu task wouldn't work correctly on SICdb if added to supported_datasets (SICdb is not currently listed, so this is latent).
EHRShot has no real build_full (echo "MEDS extraction is pre-done") and no requirements.txt. Worth deciding: (a) document the "bring your own MEDS" expectation prominently, or (b) hold the registry entry until an extractor exists.
EHRShot icu_discharge is the same as hospital_discharge (Visit/DP). May be intentional (EHRShot's OMOP source may not have a separate ICU discharge event) but worth confirming.
TODOs left in predicate files for unit-conversion logic (AUMCdb × 2, INSPIRE × 1, HIRID × 1). Real semantic gaps; predicates with value_min: thresholds will behave wrong outside the source's native units.
Pinned package versions are stale on PyPI for 4 of 6 (aumcdb-meds, hirid-meds, inspire-meds, sicdb-meds, eicu-meds — see review for exact deltas). Contributor's call.
access_policy unset everywhere (defaults to PRIVATE_SINGLE_USE, which is wrong for public PhysioNet datasets). Repo-wide gap, not new in this PR — fine to fix separately.

Test plan

Pre-commit (mdformat, codespell, prettier, ruff) passes.
Full fast test suite passes (53 tests, including test_registry_validation.py for all 8 datasets).
Integration tests for the new dataset / task pairs (AUMCdb + NWICU on mortality/in_icu/first_24h) — these require real source data.

Supersedes / refs

Supersedes More datasets #258 (more-datasets branch by @rvandewater is too stale to merge directly).
Supersedes Add NWICU #243 (NWICU only — strict subset of what's here, with less README content).
Co-authored with @rvandewater, @prockenschaub (per the original branch contributors).

🤖 Generated with Claude Code

Replicates the dataset additions from #258 on top of current dev (the original branch is too far behind to merge directly; this PR keeps only the dataset/task content, not the stale infra reverts). Datasets added (`src/MEDS_DEV/datasets/<name>/`): - EHRShot — Stanford EHR cohort with pre-built MEDS extraction. - HIRID — Bern ICU dataset via MEDS_extract-HIRID. - INSPIRE — perioperative dataset via MEDS_extract-INSPIRE. - NWICU — Northwestern ICU dataset via NWICU_MEDS. - SICdb — Salzburg ICU dataset via MEDS_extract-SICdb. - eICU — multi-center US ICU dataset via MEDS_extract-eICU (with demo). AUMCdb is also completed (was previously just predicates.yaml + README): adds dataset.yaml, requirements.txt, refs.bib, and the full ICU predicate set from the upstream PR. Tasks: mortality/in_icu/first_24h now lists AUMCdb and NWICU under supported_datasets in addition to MIMIC-IV. MIMIC-IV/README.md: pulled the longer description + access-requirements write-up from #258 (replaces the TODO placeholders). Each dataset.yaml has a `build_demo` command — for datasets without a real demo recipe, this is a stub echo so registry validation passes (matching the pattern HIRID already used in the source PR). Co-Authored-By: Robin P. van de Water <rvandewater@users.noreply.github.com> Co-Authored-By: Patrick Rockenschaub <prockenschaub@users.noreply.github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

codecov · 2026-05-13T15:51:32Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
54	1	53	0

View the top 1 failed test(s) by shortest run time

tests.test_0_datasets::test_datasets_configured[eICU]

Stack Traces | 827s run time

request = <SubRequest 'demo_dataset' for <Function test_datasets_configured[eICU]>>
venv_cache = PosixPath('.../tmp/tmpxvwnxk5l/venvs')

    @pytest.fixture(scope="session")
    def demo_dataset(request, venv_cache: Path) -> NAME_AND_DIR:
        dataset_name = request.param
        persistent_cache_dir, (cache_datasets, _, _) = get_and_validate_cache_settings(request)
        reuse_datasets, _, _ = get_and_validate_reuse_settings(request)
    
        do_overwrite = dataset_name not in reuse_datasets
    
        with cache_dir(persistent_cache_dir if dataset_name in cache_datasets else None) as root_dir:
            root_dir = Path(root_dir)
    
            check_fp = root_dir / f".{dataset_name}.check"
            output_dir = root_dir / dataset_name
    
            data_exists = (output_dir / "data").is_dir()
            metadata_exists = (output_dir / "metadata").is_dir()
    
            already_tested = check_fp.exists() and data_exists and metadata_exists
            if do_overwrite or not already_tested:
                venv_dir = venv_cache / "datasets" / dataset_name
    
>               run_command(
                    "meds-dev-dataset",
                    test_name=f"Build {dataset_name}",
                    hydra_kwargs={
                        "dataset": dataset_name,
                        "output_dir": str(output_dir.resolve()),
                        "demo": True,
                        "venv_dir": str(venv_dir.resolve()),
                    },
                )

tests/conftest.py:380: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

script = ['meds-dev-dataset', 'dataset=eICU output_dir=.........................../tmp/tmpcd9bov59/eICU demo=true venv_dir=....../venvs/datasets/eICU']
test_name = 'Build eICU'
hydra_kwargs = {'dataset': 'eICU', 'demo': True, 'output_dir': '.........................../tmp/tmpcd9bov59/eICU', 'venv_dir': '....../venvs/datasets/eICU'}
config_name = None, should_error = False, want_err_msg = None
do_use_config_yaml = False

    def run_command(
        script: Path | str,
        test_name: str,
        hydra_kwargs: dict[str, str] | None = None,
        config_name: str | None = None,
        should_error: bool = False,
        want_err_msg: str | None = None,
        do_use_config_yaml: bool = False,
    ):
        script = ["python", str(script.resolve())] if isinstance(script, Path) else [script]
        command_parts = script
    
        err_cmd_lines = []
    
        if config_name is not None and not config_name.startswith("_"):
            config_name = f"_{config_name}"
    
        if hydra_kwargs is None:
            hydra_kwargs = {}
    
        if do_use_config_yaml:
            if config_name is None:
                raise ValueError("config_name must be provided if do_use_config_yaml is True.")
    
            conf = OmegaConf.create(
                {
                    "defaults": [config_name],
                    **hydra_kwargs,
                }
            )
    
            conf_dir = tempfile.TemporaryDirectory()
            conf_path = Path(conf_dir.name) / "config.yaml"
            OmegaConf.save(conf, conf_path)
    
            command_parts.extend(
                [
                    f"--config-path={conf_path.parent.resolve()!s}",
                    "--config-name=config",
                    "'hydra.searchpath=[pkg://MEDS_transforms.configs]'",
                ]
            )
            err_cmd_lines.append(f"Using config yaml:\n{OmegaConf.to_yaml(conf)}")
        else:
            if config_name is not None:
                command_parts.append(f"--config-name={config_name}")
            command_parts.append(" ".join(dict_to_hydra_kwargs(hydra_kwargs)))
    
        full_cmd = " ".join(command_parts)
        err_cmd_lines.append(f"Running command: {full_cmd}")
        command_out = subprocess.run(full_cmd, shell=True, capture_output=True)
    
        command_errored = command_out.returncode != 0
    
        stderr = command_out.stderr.decode()
        err_cmd_lines.append(f"stderr:\n{stderr}")
        stdout = command_out.stdout.decode()
        err_cmd_lines.append(f"stdout:\n{stdout}")
    
        if should_error:
            err_cmd_str = "\n".join(err_cmd_lines)
            if not command_errored:
                if do_use_config_yaml:
                    conf_dir.cleanup()
                raise AssertionError(f"{test_name} failed as command did not error when expected!\n{err_cmd_str}")
            if want_err_msg is not None and want_err_msg not in stderr:
                if do_use_config_yaml:
                    conf_dir.cleanup()
                raise AssertionError(
                    f"{test_name} failed as expected error message not found in stderr!\n{err_cmd_str}"
                )
        elif not should_error and command_errored:
            if do_use_config_yaml:
                conf_dir.cleanup()
>           raise AssertionError(
                f"{test_name} failed as command errored when not expected!\n" + "\n".join(err_cmd_lines)
            )
E           AssertionError: Build eICU failed as command errored when not expected!
E           Running command: meds-dev-dataset dataset=eICU output_dir=.........................../tmp/tmpcd9bov59/eICU demo=true venv_dir=....../venvs/datasets/eICU
E           stderr:
E           Using CPython 3.11.15 interpreter at: .venv/bin/python
E           Creating virtual environment at: ....../venvs/datasets/eICU
E           Activate with: ....../venvs/datasets/eICU/bin/activate
E           Using Python 3.11.15 environment at: ....../venvs/datasets/eICU
E           Resolved 25 packages in 243ms
E           Downloading pyarrow (46.6MiB)
E           Downloading polars (33.8MiB)
E           Downloading numpy (16.1MiB)
E            Downloaded polars
E            Downloaded pyarrow
E            Downloaded numpy
E           Prepared 16 packages in 1.03s
E           Installed 25 packages in 38ms
E            + antlr4-python3-runtime==4.9.3
E            + attrs==26.1.0
E            + beautifulsoup4==4.14.3
E            + certifi==2026.4.22
E            + charset-normalizer==3.4.7
E            + eicu-meds==0.0.1
E            + filelock==3.29.0
E            + hydra-core==1.3.2
E            + idna==3.15
E            + jsonschema==4.26.0
E            + jsonschema-specifications==2025.9.1
E            + meds==0.3.3
E            + meds-transforms==0.2.4
E            + numpy==2.4.4
E            + omegaconf==2.3.0
E            + packaging==26.2
E            + polars==1.26.0
E            + pyarrow==24.0.0
E            + pyyaml==6.0.3
E            + referencing==0.37.0
E            + requests==2.34.0
E            + rpds-py==0.30.0
E            + soupsieve==2.8.3
E            + typing-extensions==4.15.0
E            + urllib3==2.7.0
E           Error executing job with overrides: ['dataset=eICU', 'output_dir=.........................../tmp/tmpcd9bov59/eICU', 'demo=true', 'venv_dir=....../venvs/datasets/eICU']
E           Traceback (most recent call last):
E             File ".../MEDS-DEV/MEDS-DEV/.venv/bin/meds-dev-dataset", line 10, in <module>
E               sys.exit(main())
E                        ^^^^^^
E             File ".../MEDS-DEV/MEDS-DEV/.venv/lib/python3.11.../site-packages/hydra/main.py", line 94, in decorated_main
E               _run_hydra(
E             File ".../MEDS-DEV/MEDS-DEV/.venv/lib/python3.11.../hydra/_internal/utils.py", line 394, in _run_hydra
E               _run_app(
E             File ".../MEDS-DEV/MEDS-DEV/.venv/lib/python3.11.../hydra/_internal/utils.py", line 457, in _run_app
E               run_and_report(
E             File ".../MEDS-DEV/MEDS-DEV/.venv/lib/python3.11.../hydra/_internal/utils.py", line 223, in run_and_report
E               raise ex
E             File ".../MEDS-DEV/MEDS-DEV/.venv/lib/python3.11.../hydra/_internal/utils.py", line 220, in run_and_report
E               return func()
E                      ^^^^^^
E             File ".../MEDS-DEV/MEDS-DEV/.venv/lib/python3.11.../hydra/_internal/utils.py", line 458, in <lambda>
E               lambda: hydra.run(
E                       ^^^^^^^^^^
E             File ".../MEDS-DEV/MEDS-DEV/.venv/lib/python3.11.../hydra/_internal/hydra.py", line 132, in run
E               _ = ret.return_value
E                   ^^^^^^^^^^^^^^^^
E             File ".../MEDS-DEV/MEDS-DEV/.venv/lib/python3.11.../hydra/core/utils.py", line 260, in return_value
E               raise self._return_value
E             File ".../MEDS-DEV/MEDS-DEV/.venv/lib/python3.11.../hydra/core/utils.py", line 186, in run_job
E               ret.return_value = task_function(task_cfg)
E                                  ^^^^^^^^^^^^^^^^^^^^^^^
E             File ".../MEDS_DEV/datasets/__main__.py", line 42, in main
E               run_in_env(
E             File ".../src/MEDS_DEV/utils.py", line 422, in run_in_env
E               raise RuntimeError(
E           RuntimeError: Command failed with exit code 1:
E           SCRIPT:
E           #!/bin/bash
E           set -e
E           MEDS_extract-eICU do_demo=True raw_input_dir="............/tmp/tmpclsvtm7s/raw" pre_MEDS_dir="......................../tmp/tmpclsvtm7s/pre_MEDS" MEDS_cohort_dir=".........................../tmp/tmpcd9bov59/eICU" log_dir="................../tmpcd9bov59/eICU/.pipeline_logs"
E           STDERR:
E           Error executing job with overrides: ['do_demo=True', 'raw_input_dir=............/tmp/tmpclsvtm7s/raw', 'pre_MEDS_dir=......................../tmp/tmpclsvtm7s/pre_MEDS', 'MEDS_cohort_dir=.........................../tmp/tmpcd9bov59/eICU', 'log_dir=................../tmpcd9bov59/eICU/.pipeline_logs']
E           Traceback (most recent call last):
E             File "....../venvs/datasets/eICU/lib/python3.11.../site-packages/eICU_MEDS/__main__.py", line 87, in main
E               run_command(command_parts, cfg)
E             File "....../venvs/datasets/eICU/lib/python3.11.../site-packages/eICU_MEDS/commands.py", line 80, in run_command
E               raise ValueError(f"Command failed with return code {command_out.returncode}.")
E           ValueError: Command failed with return code 1.
E           
E           Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
E           
E           STDOUT:
E           [2026-05-13 15:37:27,105][eICU_MEDS.__main__][INFO] - Downloading demo data.
E           [2026-05-13 15:45:27,241][eICU_MEDS.download][INFO] - Downloaded: ........./raw/sqlite/eicu_v2_0_1.sqlite3.gz
E           [2026-05-13 15:45:27,451][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/LICENSE.txt
E           [2026-05-13 15:45:27,539][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/SHA256SUMS.txt
E           [2026-05-13 15:45:28,059][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/admissionDx.csv.gz
E           [2026-05-13 15:45:28,751][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/admissiondrug.csv.gz
E           [2026-05-13 15:45:28,984][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/allergy.csv.gz
E           [2026-05-13 15:45:29,356][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/apacheApsVar.csv.gz
E           [2026-05-13 15:45:30,519][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/apachePatientResult.csv.gz
E           [2026-05-13 15:45:30,987][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/apachePredVar.csv.gz
E           [2026-05-13 15:45:31,410][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/carePlanCareProvider.csv.gz
E           [2026-05-13 15:45:31,459][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/carePlanEOL.csv.gz
E           [2026-05-13 15:45:33,242][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/carePlanGeneral.csv.gz
E           [2026-05-13 15:45:33,460][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/carePlanGoal.csv.gz
E           [2026-05-13 15:45:33,514][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/carePlanInfectiousDisease.csv.gz
E           [2026-05-13 15:45:33,563][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/customLab.csv.gz
E           [2026-05-13 15:45:35,555][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/diagnosis.csv.gz
E           [2026-05-13 15:45:35,602][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/hospital.csv.gz
E           [2026-05-13 15:45:37,460][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/infusiondrug.csv.gz
E           [2026-05-13 15:45:46,248][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/intakeOutput.csv.gz
E           [2026-05-13 15:46:14,209][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/lab.csv.gz
E           [2026-05-13 15:46:22,340][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/medication.csv.gz
E           [2026-05-13 15:46:22,412][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/microLab.csv.gz
E           [2026-05-13 15:46:23,874][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/note.csv.gz
E           [2026-05-13 15:46:33,224][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/nurseAssessment.csv.gz
E           [2026-05-13 15:46:36,053][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/nurseCare.csv.gz
E           [2026-05-13 15:48:21,814][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/nurseCharting.csv.gz
E           [2026-05-13 15:48:22,633][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/pastHistory.csv.gz
E           [2026-05-13 15:48:23,349][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/patient.csv.gz
E           [2026-05-13 15:48:28,445][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/physicalExam.csv.gz
E           [2026-05-13 15:48:28,993][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/respiratoryCare.csv.gz
E           [2026-05-13 15:48:38,388][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/respiratoryCharting.csv.gz
E           [2026-05-13 15:48:41,720][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/treatment.csv.gz
E           [2026-05-13 15:48:59,339][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/vitalAperiodic.csv.gz
E           [2026-05-13 15:51:00,898][eICU_MEDS.download][INFO] - Downloaded: ........./tmpclsvtm7s/raw/vitalPeriodic.csv.gz
E           [2026-05-13 15:51:00,899][eICU_MEDS.__main__][INFO] - Running pre_MEDS data wrangling.
E           [2026-05-13 15:51:00,900][eICU_MEDS.pre_MEDS][INFO] - Loading table preprocessors from ....../venvs/datasets/eICU/lib/python3.11.../eICU_MEDS/configs/table_preprocessors.yaml...
E           [2026-05-13 15:51:00,922][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for admissionDx:
E           offset_col: admitdxenteredoffset
E           pseudotime_col: admitDxEnteredTimestamp
E           output_data_cols:
E           - admitdxname
E           - admissiondxid
E           warning_items:
E           - How should we use `admitdxtest`?
E           - How should we use `admitdxpath`?
E           
E           [2026-05-13 15:51:00,922][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for allergy:
E           offset_col: allergyenteredoffset
E           pseudotime_col: allergyEnteredTimestamp
E           output_data_cols:
E           - allergytype
E           - allergyname
E           warning_items:
E           - How should we use `allergyNoteType`?
E           - How should we use `specialtyType`?
E           - How should we use `userType`?
E           - Is `drugName` the name of the drug to which the patient is allergic or the drug
E             given to the patient (docs say 'name of the selected admission drug')?
E           
E           [2026-05-13 15:51:00,923][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for carePlanGeneral:
E           offset_col: cplitemoffset
E           pseudotime_col: carePlanGeneralItemEnteredTimestamp
E           output_data_cols:
E           - cplgroup
E           - cplitemvalue
E           
E           [2026-05-13 15:51:00,923][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for carePlanEOL:
E           offset_col: cpleoldiscussionoffset
E           pseudotime_col: carePlanEolDiscussionOccurredTimestamp
E           warning_items:
E           - Is the DiscussionOffset time actually reliable? Should we fall back on the SaveOffset
E             time?
E           
E           [2026-05-13 15:51:00,924][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for carePlanGoal:
E           offset_col: cplgoaloffset
E           pseudotime_col: carePlanGoalEnteredTimestamp
E           output_data_cols:
E           - cplgoalcategory
E           - cplgoalvalue
E           - cplgoalstatus
E           
E           [2026-05-13 15:51:00,924][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for carePlanInfectiousDisease:
E           offset_col: cplinfectdiseaseoffset
E           pseudotime_col: carePlanInfectDiseaseEnteredTimestamp
E           output_data_cols:
E           - infectdiseasesite
E           - infectdiseaseassessment
E           - responsetotherapy
E           - treatment
E           
E           [2026-05-13 15:51:00,925][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for diagnosis:
E           offset_col: diagnosisoffset
E           pseudotime_col: diagnosisEnteredTimestamp
E           output_data_cols:
E           - icd9code
E           - diagnosispriority
E           - diagnosisstring
E           warning_items:
E           - Though we use it, the `diagnosisString` field documentation is unclear -- by what
E             is it separated?
E           
E           [2026-05-13 15:51:00,925][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for infusionDrug:
E           offset_col: infusionoffset
E           pseudotime_col: infusionEnteredTimestamp
E           output_data_cols:
E           - infusiondrugid
E           - drugname
E           - drugrate
E           - infusionrate
E           - drugamount
E           - volumeoffluid
E           - patientweight
E           
E           [2026-05-13 15:51:00,926][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for lab:
E           offset_col: labresultoffset
E           pseudotime_col: labResultDrawnTimestamp
E           output_data_cols:
E           - labname
E           - labresult
E           - labresulttext
E           - labmeasurenamesystem
E           - labmeasurenameinterface
E           - labtypeid
E           warning_items:
E           - Is this the time the lab was drawn? Entered? The time the result came in?
E           - We **IGNORE** the `labResultRevisedOffset` column -- this may be a mistake!
E           
E           [2026-05-13 15:51:00,926][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for medication:
E           offset_col:
E           - drugorderoffset
E           - drugstartoffset
E           - drugstopoffset
E           pseudotime_col:
E           - drugordertimestamp
E           - drugstarttimestamp
E           - drugstoptimestamp
E           output_data_cols:
E           - medicationid
E           - drugivadmixture
E           - drugname
E           - drughiclseqno
E           - dosage
E           - routeadmin
E           - frequency
E           - loadingdose
E           - prn
E           - gtc
E           warning_items:
E           - We **IGNORE** the `drugOrderCancelled` column -- this may be a mistake!
E           
E           [2026-05-13 15:51:00,927][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for nurseAssessment:
E           offset_col:
E           - nurseassessoffset
E           - nurseassessentryoffset
E           pseudotime_col:
E           - nurseAssessPerformedTimestamp
E           - nurseAssessEnteredTimestamp
E           output_data_cols:
E           - nurseassessid
E           - celllabel
E           - cellattribute
E           - cellattributevalue
E           warning_items:
E           - Should we be using `cellAttributePath` instead of `cellAttribute`?
E           - SOME MAY BE LISTS
E           
E           [2026-05-13 15:51:00,928][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for nurseCare:
E           offset_col:
E           - nursecareoffset
E           - nursecareentryoffset
E           pseudotime_col:
E           - nurseCarePerformedTimestamp
E           - nurseCareEnteredTimestamp
E           output_data_cols:
E           - nursecareid
E           - celllabel
E           - cellattribute
E           - cellattributevalue
E           warning_items:
E           - Should we be using `cellAttributePath` instead of `cellAttribute`?
E           - SOME MAY BE LISTS
E           
E           [2026-05-13 15:51:00,928][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for nurseCharting:
E           offset_col:
E           - nursingchartoffset
E           - nursingchartentryoffset
E           pseudotime_col:
E           - nursingChartPerformedTimestamp
E           - nursingChartEnteredTimestamp
E           output_data_cols:
E           - nursingchartid
E           - nursingchartcelltypecat
E           - nursingchartcelltypevalname
E           - nursingchartcelltypevallabel
E           - nursingchartvalue
E           warning_items:
E           - SOME MAY BE LISTS
E           
E           [2026-05-13 15:51:00,929][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for pastHistory:
E           offset_col:
E           - pasthistoryoffset
E           - pasthistoryenteredoffset
E           pseudotime_col:
E           - pastHistoryTakenTimestamp
E           - pastHistoryEnteredTimestamp
E           output_data_cols:
E           - pasthistoryid
E           - pasthistorynotetype
E           - pasthistorypath
E           - pasthistoryvalue
E           - pasthistoryvaluetext
E           warning_items:
E           - SOME MAY BE LISTS
E           - How should we use `pastHistoryPath` vs. `pastHistoryNoteType`?
E           - How should we use `pastHistoryValue` vs. `pastHistoryValueText`?
E           
E           [2026-05-13 15:51:00,929][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for physicalExam:
E           offset_col: physicalexamoffset
E           pseudotime_col: physicalExamEnteredTimestamp
E           output_data_cols:
E           - physicalexamid
E           - physicalexamtext
E           - physicalexampath
E           - physicalexamvalue
E           warning_items:
E           - How should we use `physicalExamValue` vs. `physicalExamText`?
E           - I believe the `physicalExamValue` is a **LIST**. This must be processed specially.
E           
E           [2026-05-13 15:51:00,930][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for respiratoryCharting:
E           offset_col:
E           - respchartoffset
E           - respchartentryoffset
E           pseudotime_col:
E           - respChartPerformedTimestamp
E           - respChartEnteredTimestamp
E           output_data_cols:
E           - respchartid
E           - respcharttypecat
E           - respchartvaluelabel
E           - respchartvalue
E           warning_items:
E           - SOME MAY BE LISTS
E           
E           [2026-05-13 15:51:00,930][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for treatment:
E           offset_col: treatmentoffset
E           pseudotime_col: treatmentEnteredTimestamp
E           output_data_cols:
E           - treatmentid
E           - treatmentstring
E           warning_items:
E           - Absence of entries in table do not indicate absence of treatments
E           
E           [2026-05-13 15:51:00,931][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for vitalAperiodic:
E           offset_col: observationoffset
E           pseudotime_col: observationEnteredTimestamp
E           output_data_cols:
E           - vitalaperiodicid
E           - noninvasivesystolic
E           - noninvasivediastolic
E           - noninvasivemean
E           - paop
E           - cardiacoutput
E           - cardiacinput
E           - svr
E           - svri
E           - pvr
E           - pvri
E           
E           [2026-05-13 15:51:00,932][eICU_MEDS.pre_MEDS][INFO] -   Adding preprocessor for vitalPeriodic:
E           offset_col: observationoffset
E           pseudotime_col: observationEnteredTimestamp
E           output_data_cols:
E           - vitalperiodicid
E           - temperature
E           - sao2
E           - heartrate
E           - respiration
E           - cvp
E           - etco2
E           - systemicsystolic
E           - systemicdiastolic
E           - systemicmean
E           - pasystolic
E           - padiastolic
E           - pamean
E           - st1
E           - st2
E           - st3
E           - icp
E           warning_items:
E           - These are 5-minute median values. There are going to be a *lot* of events.
E           
E           [2026-05-13 15:51:00,932][eICU_MEDS.pre_MEDS][INFO] - Processing patient table first...
E           [2026-05-13 15:51:00,932][eICU_MEDS.pre_MEDS][INFO] - Loading ........./tmpclsvtm7s/raw/hospital.csv.gz...
E           [2026-05-13 15:51:00,934][eICU_MEDS.pre_MEDS][INFO] - Loading ........./tmpclsvtm7s/raw/patient.csv.gz...
E           [2026-05-13 15:51:00,946][eICU_MEDS.pre_MEDS][INFO] - Processing patient table...
E           [2026-05-13 15:51:00,947][eICU_MEDS.pre_MEDS][INFO] - Checking that the 24h times are consistent. If this is extremely slow, consider refactoring to have only one `.collect()` call.
E           [2026-05-13 15:51:00,947][eICU_MEDS.pre_MEDS][INFO] - Checking that stated 24h times are consistent given offsets between {pseudotime_col.name} and hospitaldischargetime24...
E           [2026-05-13 15:51:00,949][eICU_MEDS.pre_MEDS][INFO] - Checking that stated 24h times are consistent given offsets between {pseudotime_col.name} and hospitaladmittime24...
E           [2026-05-13 15:51:00,950][eICU_MEDS.pre_MEDS][INFO] - Checking that stated 24h times are consistent given offsets between {pseudotime_col.name} and unitadmittime24...
E           [2026-05-13 15:51:00,952][eICU_MEDS.pre_MEDS][INFO] - Checking that stated 24h times are consistent given offsets between {pseudotime_col.name} and unitdischargetime24...
E           [2026-05-13 15:51:00,953][eICU_MEDS.pre_MEDS][INFO] - Validated 24h times in 0:00:00.006168
E           [2026-05-13 15:51:00,954][eICU_MEDS.pre_MEDS][WARNING] - NOT validating the `unitVisitNumber` column as that isn't implemented yet.
E           [2026-05-13 15:51:00,954][eICU_MEDS.pre_MEDS][WARNING] - NOT SURE ABOUT THE FOLLOWING. Check with the eICU team:
E             - `apacheAdmissionDx` is not selected from the patients table as we grab it from `admissiondx`. Is this right?
E             - `admissionHeight` and `admissionWeight` are interpreted as **unit** admission height/weight, not hospital admission height/weight. Is this right?
E             - `age` is interpreted as the age at the time of the unit stay, not the hospital stay. Is this right?
E             - `What is the actual mean age for those > 89? Here we assume 90.
E             - Note that all the column names appear to be all in lowercase for the csv versions, vs. the docs
E           [2026-05-13 15:51:00,977][eICU_MEDS.pre_MEDS][INFO] - Processing medication...
E           [2026-05-13 15:51:01,126][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/medication.csv.gz in 0:00:00.148892
E           [2026-05-13 15:51:01,126][eICU_MEDS.pre_MEDS][WARNING] - NOT SURE ABOUT THE FOLLOWING for medication table. Check with the eICU team:
E             - We **IGNORE** the `drugOrderCancelled` column -- this may be a mistake!
E           [2026-05-13 15:51:01,182][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/medication.parquet in 0:00:00.205688
E           [2026-05-13 15:51:01,183][eICU_MEDS.pre_MEDS][INFO] - Processing treatment...
E           [2026-05-13 15:51:01,217][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/treatment.csv.gz in 0:00:00.034472
E           [2026-05-13 15:51:01,217][eICU_MEDS.pre_MEDS][WARNING] - NOT SURE ABOUT THE FOLLOWING for treatment table. Check with the eICU team:
E             - Absence of entries in table do not indicate absence of treatments
E           [2026-05-13 15:51:01,233][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/treatment.parquet in 0:00:00.050640
E           [2026-05-13 15:51:01,234][eICU_MEDS.pre_MEDS][INFO] - Processing careplangoal...
E           [2026-05-13 15:51:01,238][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/carePlanGoal.csv.gz in 0:00:00.004140
E           [2026-05-13 15:51:01,244][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/careplangoal.parquet in 0:00:00.010307
E           [2026-05-13 15:51:01,244][eICU_MEDS.pre_MEDS][INFO] - Processing pasthistory...
E           [2026-05-13 15:51:01,259][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/pastHistory.csv.gz in 0:00:00.015386
E           [2026-05-13 15:51:01,260][eICU_MEDS.pre_MEDS][WARNING] - NOT SURE ABOUT THE FOLLOWING for pastHistory table. Check with the eICU team:
E             - SOME MAY BE LISTS
E             - How should we use `pastHistoryPath` vs. `pastHistoryNoteType`?
E             - How should we use `pastHistoryValue` vs. `pastHistoryValueText`?
E           [2026-05-13 15:51:01,270][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/pasthistory.parquet in 0:00:00.026274
E           [2026-05-13 15:51:01,271][eICU_MEDS.pre_MEDS][WARNING] - No function needed for apachepredvar. For eICU, THIS IS UNEXPECTED
E           [2026-05-13 15:51:01,271][eICU_MEDS.pre_MEDS][INFO] - Processing careplaneol...
E           [2026-05-13 15:51:01,271][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/carePlanEOL.csv.gz in 0:00:00.000402
E           [2026-05-13 15:51:01,271][eICU_MEDS.pre_MEDS][WARNING] - NOT SURE ABOUT THE FOLLOWING for carePlanEOL table. Check with the eICU team:
E             - Is the DiscussionOffset time actually reliable? Should we fall back on the SaveOffset time?
E           [2026-05-13 15:51:01,274][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/careplaneol.parquet in 0:00:00.003797
E           [2026-05-13 15:51:01,275][eICU_MEDS.pre_MEDS][INFO] - Processing infusiondrug...
E           [2026-05-13 15:51:01,315][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/infusiondrug.csv.gz in 0:00:00.040114
E           [2026-05-13 15:51:01,336][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/infusiondrug.parquet in 0:00:00.061489
E           [2026-05-13 15:51:01,336][eICU_MEDS.pre_MEDS][INFO] - Processing admissiondx...
E           [2026-05-13 15:51:01,345][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/admissionDx.csv.gz in 0:00:00.008518
E           [2026-05-13 15:51:01,345][eICU_MEDS.pre_MEDS][WARNING] - NOT SURE ABOUT THE FOLLOWING for admissionDx table. Check with the eICU team:
E             - How should we use `admitdxtest`?
E             - How should we use `admitdxpath`?
E           [2026-05-13 15:51:01,352][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/admissiondx.parquet in 0:00:00.015328
E           [2026-05-13 15:51:01,352][eICU_MEDS.pre_MEDS][WARNING] - No function needed for respiratorycare. For eICU, THIS IS UNEXPECTED
E           [2026-05-13 15:51:01,352][eICU_MEDS.pre_MEDS][INFO] - Processing lab...
E           [2026-05-13 15:51:01,938][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/lab.csv.gz in 0:00:00.586340
E           [2026-05-13 15:51:01,939][eICU_MEDS.pre_MEDS][WARNING] - NOT SURE ABOUT THE FOLLOWING for lab table. Check with the eICU team:
E             - Is this the time the lab was drawn? Entered? The time the result came in?
E             - We **IGNORE** the `labResultRevisedOffset` column -- this may be a mistake!
E           [2026-05-13 15:51:02,094][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/lab.parquet in 0:00:00.741849
E           [2026-05-13 15:51:02,094][eICU_MEDS.pre_MEDS][WARNING] - No function needed for careplancareprovider. For eICU, THIS IS UNEXPECTED
E           [2026-05-13 15:51:02,094][eICU_MEDS.pre_MEDS][INFO] - Processing careplangeneral...
E           [2026-05-13 15:51:02,123][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/carePlanGeneral.csv.gz in 0:00:00.029179
E           [2026-05-13 15:51:02,137][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/careplangeneral.parquet in 0:00:00.042994
E           [2026-05-13 15:51:02,137][eICU_MEDS.pre_MEDS][WARNING] - No function needed for intakeoutput. For eICU, THIS IS UNEXPECTED
E           [2026-05-13 15:51:02,137][eICU_MEDS.pre_MEDS][INFO] - Processing vitalaperiodic...
E           [2026-05-13 15:51:02,387][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/vitalAperiodic.csv.gz in 0:00:00.249142
E           [2026-05-13 15:51:02,468][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/vitalaperiodic.parquet in 0:00:00.330487
E           [2026-05-13 15:51:02,468][eICU_MEDS.pre_MEDS][WARNING] - No function needed for apacheapsvar. For eICU, THIS IS UNEXPECTED
E           [2026-05-13 15:51:02,468][eICU_MEDS.pre_MEDS][INFO] - Processing physicalexam...
E           [2026-05-13 15:51:02,550][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/physicalExam.csv.gz in 0:00:00.081345
E           [2026-05-13 15:51:02,550][eICU_MEDS.pre_MEDS][WARNING] - NOT SURE ABOUT THE FOLLOWING for physicalExam table. Check with the eICU team:
E             - How should we use `physicalExamValue` vs. `physicalExamText`?
E             - I believe the `physicalExamValue` is a **LIST**. This must be processed specially.
E           [2026-05-13 15:51:02,583][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/physicalexam.parquet in 0:00:00.114596
E           [2026-05-13 15:51:02,583][eICU_MEDS.pre_MEDS][INFO] - Processing careplaninfectiousdisease...
E           [2026-05-13 15:51:02,584][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/carePlanInfectiousDisease.csv.gz in 0:00:00.000633
E           [2026-05-13 15:51:02,588][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/careplaninfectiousdisease.parquet in 0:00:00.004840
E           [2026-05-13 15:51:02,588][eICU_MEDS.pre_MEDS][INFO] - Processing vitalperiodic...
E           [2026-05-13 15:51:04,571][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/vitalPeriodic.csv.gz in 0:00:01.983163
E           [2026-05-13 15:51:04,572][eICU_MEDS.pre_MEDS][WARNING] - NOT SURE ABOUT THE FOLLOWING for vitalPeriodic table. Check with the eICU team:
E             - These are 5-minute median values. There are going to be a *lot* of events.
E           [2026-05-13 15:51:05,169][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/vitalperiodic.parquet in 0:00:02.580848
E           [2026-05-13 15:51:05,169][eICU_MEDS.pre_MEDS][INFO] - Processing nurseassessment...
E           [2026-05-13 15:51:05,283][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/nurseAssessment.csv.gz in 0:00:00.113740
E           [2026-05-13 15:51:05,283][eICU_MEDS.pre_MEDS][WARNING] - NOT SURE ABOUT THE FOLLOWING for nurseAssessment table. Check with the eICU team:
E             - Should we be using `cellAttributePath` instead of `cellAttribute`?
E             - SOME MAY BE LISTS
E           [2026-05-13 15:51:05,322][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/nurseassessment.parquet in 0:00:00.152255
E           [2026-05-13 15:51:05,322][eICU_MEDS.pre_MEDS][WARNING] - No function needed for customlab. For eICU, THIS IS UNEXPECTED
E           [2026-05-13 15:51:05,322][eICU_MEDS.pre_MEDS][WARNING] - No function needed for microlab. For eICU, THIS IS UNEXPECTED
E           [2026-05-13 15:51:05,322][eICU_MEDS.pre_MEDS][INFO] - Processing respiratorycharting...
E           [2026-05-13 15:51:05,496][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/respiratoryCharting.csv.gz in 0:00:00.173767
E           [2026-05-13 15:51:05,496][eICU_MEDS.pre_MEDS][WARNING] - NOT SURE ABOUT THE FOLLOWING for respiratoryCharting table. Check with the eICU team:
E             - SOME MAY BE LISTS
E           [2026-05-13 15:51:05,566][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/respiratorycharting.parquet in 0:00:00.244616
E           [2026-05-13 15:51:05,567][eICU_MEDS.pre_MEDS][INFO] - Processing nursecare...
E           [2026-05-13 15:51:05,618][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/nurseCare.csv.gz in 0:00:00.051285
E           [2026-05-13 15:51:05,618][eICU_MEDS.pre_MEDS][WARNING] - NOT SURE ABOUT THE FOLLOWING for nurseCare table. Check with the eICU team:
E             - Should we be using `cellAttributePath` instead of `cellAttribute`?
E             - SOME MAY BE LISTS
E           [2026-05-13 15:51:05,639][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/nursecare.parquet in 0:00:00.072110
E           [2026-05-13 15:51:05,639][eICU_MEDS.pre_MEDS][WARNING] - No function needed for apachepatientresult. For eICU, THIS IS UNEXPECTED
E           [2026-05-13 15:51:05,639][eICU_MEDS.pre_MEDS][WARNING] - Skipping admissiondrug as it is not supported in this pipeline.
E           [2026-05-13 15:51:05,639][eICU_MEDS.pre_MEDS][WARNING] - Skipping note as it is not supported in this pipeline.
E           [2026-05-13 15:51:05,639][eICU_MEDS.pre_MEDS][INFO] - Processing allergy...
E           [2026-05-13 15:51:05,644][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/allergy.csv.gz in 0:00:00.004711
E           [2026-05-13 15:51:05,644][eICU_MEDS.pre_MEDS][WARNING] - NOT SURE ABOUT THE FOLLOWING for allergy table. Check with the eICU team:
E             - How should we use `allergyNoteType`?
E             - How should we use `specialtyType`?
E             - How should we use `userType`?
E             - Is `drugName` the name of the drug to which the patient is allergic or the drug given to the patient (docs say 'name of the selected admission drug')?
E           [2026-05-13 15:51:05,649][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/allergy.parquet in 0:00:00.009748
E           [2026-05-13 15:51:05,649][eICU_MEDS.pre_MEDS][INFO] - Processing nursecharting...
E           [2026-05-13 15:51:07,352][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/nurseCharting.csv.gz in 0:00:01.703019
E           [2026-05-13 15:51:07,352][eICU_MEDS.pre_MEDS][WARNING] - NOT SURE ABOUT THE FOLLOWING for nurseCharting table. Check with the eICU team:
E             - SOME MAY BE LISTS
E           [2026-05-13 15:51:07,896][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/nursecharting.parquet in 0:00:02.247009
E           [2026-05-13 15:51:07,896][eICU_MEDS.pre_MEDS][INFO] - Processing diagnosis...
E           [2026-05-13 15:51:07,924][eICU_MEDS.pre_MEDS][INFO] -   * Loaded raw ........./tmpclsvtm7s/raw/diagnosis.csv.gz in 0:00:00.027803
E           [2026-05-13 15:51:07,924][eICU_MEDS.pre_MEDS][WARNING] - NOT SURE ABOUT THE FOLLOWING for diagnosis table. Check with the eICU team:
E             - Though we use it, the `diagnosisString` field documentation is unclear -- by what is it separated?
E           [2026-05-13 15:51:07,937][eICU_MEDS.pre_MEDS][INFO] -   * Processed and wrote to ........./tmpclsvtm7s/pre_MEDS/diagnosis.parquet in 0:00:00.040397
E           [2026-05-13 15:51:07,937][eICU_MEDS.pre_MEDS][INFO] - Done! All dataframes processed and written to ......................../tmp/tmpclsvtm7s/pre_MEDS
E           [2026-05-13 15:51:07,938][eICU_MEDS.__main__][INFO] - Running in serial mode as N_WORKERS is not set.
E           [2026-05-13 15:51:07,939][eICU_MEDS.commands][INFO] - Running command: DATASET_NAME=eICU DATASET_VERSION=2.0:0.0.1 EVENT_CONVERSION_CONFIG_FP=....../venvs/datasets/eICU/lib/python3.11.../eICU_MEDS/configs/event_configs.yaml PRE_MEDS_DIR=......................../tmp/tmpclsvtm7s/pre_MEDS MEDS_COHORT_DIR=.........................../tmp/tmpcd9bov59/eICU MEDS_transform-runner --config-path=....../venvs/datasets/eICU/lib/python3.11........./site-packages/eICU_MEDS/configs --config-name=runner pipeline_config_fp=....../venvs/datasets/eICU/lib/python3.11....../eICU_MEDS/configs/ETL.yaml ~parallelize 'hydra.searchpath=[pkg://MEDS_transforms.configs]' ++do_overwrite=False
E           [2026-05-13 15:51:09,494][eICU_MEDS.commands][INFO] - Command output:
E           [2026-05-13 15:51:08,440][MEDS_transforms.runner][INFO] - Running stage: shard_events
E           [2026-05-13 15:51:08,584][MEDS_transforms.runner][INFO] - Running command: MEDS_extract-shard_events --config-dir=....../venvs/datasets/eICU/lib/python3.11........./site-packages/eICU_MEDS/configs --config-name=ETL 'hydra.searchpath=[pkg://MEDS_transforms.configs]' stage=shard_events
E           [2026-05-13 15:51:09,391][MEDS_transforms.runner][INFO] - Command output:
E           [2026-05-13 15:51:09,176][MEDS_transforms.extract.shard_events][INFO] - Running with config:
E           input_dir: ${oc.env:PRE_MEDS_DIR}
E           cohort_dir: ${oc.env:MEDS_COHORT_DIR}
E           _default_description: 'This is a MEDS pipeline ETL. Please set a more detailed description
E             at the top of your specific pipeline
E           
E             configuration file.'
E           log_dir: ${stage_cfg.output_dir}/.logs
E           do_overwrite: false
E           seed: 1
E           stages:
E           - shard_events
E           - split_and_shard_subjects
E           - convert_to_sharded_events
E           - merge_to_MEDS_cohort
E           - extract_code_metadata
E           - finalize_MEDS_metadata
E           - finalize_MEDS_data
E           stage_configs:
E             shard_events:
E               row_chunksize: 200000000
E               infer_schema_length: 999999999
E               data_input_dir: ${input_dir}
E             split_and_shard_subjects:
E               is_metadata: true
E               output_dir: ${cohort_dir}/metadata
E               n_subjects_per_shard: 1000
E               external_splits_json_fp: null
E               split_fracs:
E                 train: 0.8
E                 tuning: 0.1
E                 held_out: 0.1
E             convert_to_sharded_events:
E               do_dedup_text_and_numeric: true
E             merge_to_MEDS_cohort:
E               unique_by: '*'
E               additional_sort_by: null
E             extract_code_metadata:
E               is_metadata: true
E               description_separator: '
E           
E                 '
E             finalize_MEDS_metadata:
E               is_metadata: true
E               do_retype: true
E             finalize_MEDS_data:
E               do_retype: true
E           worker: 0
E           polling_time: 300
E           stage: shard_events
E           stage_cfg: ${oc.create:${populate_stage:${stage}, ${input_dir}, ${cohort_dir}, ${stages},
E             ${stage_configs}}}
E           etl_metadata:
E             pipeline_name: ???
E             dataset_name: ${oc.env:DATASET_NAME}
E             dataset_version: ${oc.env:DATASET_VERSION}
E             package_name: ${get_package_name:}
E             package_version: ${get_package_version:}
E           etl_metadata.pipeline_name: extract
E           description: "This pipeline extracts a dataset in longitudinal, sparse form from an\
E             \ input dataset meeting\nselect criteria and converts them to the MEDS format.\n\
E             \nThis pipeline is for the ${etl_metadata.dataset_name} dataset version ${etl_metadata.dataset_version}.\n\
E             \nKey variables for this pipeline are\n  - `input_dir=$PATH_TO_INPUT_DIR`\n  - `cohort_dir=$PATH_TO_OUTPUT_DIR`.\n\
E             \  - `event_conversion_config_fp=$PATH_TO_EVENT_CONVERSION_CONFIG_FP`\n\nSee the\
E             \ MEDS-Transforms Extract documentation for configuration file details."
E           event_conversion_config_fp: ${oc.env:EVENT_CONVERSION_CONFIG_FP}
E           shards_map_fp: ${cohort_dir}/metadata/.shards.json
E           parallelize:
E             n_workers: ${oc.env:N_WORKERS}
E             launcher: joblib
E           
E           Stage: shard_events
E           
E           Stage config:
E           row_chunksize: 200000000
E           infer_schema_length: 999999999
E           data_input_dir: ......................../tmp/tmpclsvtm7s/pre_MEDS
E           is_metadata: false
E           metadata_input_dir: ................../tmpclsvtm7s/pre_MEDS/metadata
E           output_dir: ......................../tmpcd9bov59/eICU/shard_events
E           reducer_output_dir: null
E           
E           [2026-05-13 15:51:09,182][MEDS_transforms.extract.shard_events][INFO] - Reading event conversion config from ....../venvs/datasets/eICU/lib/python3.11.../eICU_MEDS/configs/event_configs.yaml to identify needed columns.
E           [2026-05-13 15:51:09,234][MEDS_transforms.extract.shard_events][WARNING] - Skipping ........./tmpclsvtm7s/pre_MEDS/nurseassessment.parquet as it is not specified in the event conversion configuration.
E           [2026-05-13 15:51:09,234][MEDS_transforms.extract.shard_events][WARNING] - Skipping ........./tmpclsvtm7s/pre_MEDS/nursecharting.parquet as it is not specified in the event conversion configuration.
E           [2026-05-13 15:51:09,234][MEDS_transforms.extract.shard_events][WARNING] - Skipping ........./tmpclsvtm7s/pre_MEDS/respiratorycharting.parquet as it is not specified in the event conversion configuration.
E           [2026-05-13 15:51:09,234][MEDS_transforms.extract.shard_events][WARNING] - Skipping ........./tmpclsvtm7s/pre_MEDS/physicalexam.parquet as it is not specified in the event conversion configuration.
E           [2026-05-13 15:51:09,234][MEDS_transforms.extract.shard_events][WARNING] - Skipping ........./tmpclsvtm7s/pre_MEDS/nursecare.parquet as it is not specified in the event conversion configuration.
E           [2026-05-13 15:51:09,234][MEDS_transforms.extract.shard_events][WARNING] - Skipping ........./tmpclsvtm7s/pre_MEDS/pasthistory.parquet as it is not specified in the event conversion configuration.
E           [2026-05-13 15:51:09,235][MEDS_transforms.extract.shard_events][INFO] - Starting event sub-sharding. Sub-sharding 14 files:
E             * ........./tmpclsvtm7s/pre_MEDS/diagnosis.parquet
E             * ........./tmpclsvtm7s/pre_MEDS/lab.parquet
E             * ........./tmpclsvtm7s/pre_MEDS/careplangoal.parquet
E             * ................../tmpclsvtm7s/pre_MEDS/patient.parquet
E             * ........./tmpclsvtm7s/pre_MEDS/careplaneol.parquet
E             * ........./tmpclsvtm7s/pre_MEDS/admissiondx.parquet
E             * ........./tmpclsvtm7s/pre_MEDS/vitalaperiodic.parquet
E             * ........./tmpclsvtm7s/pre_MEDS/allergy.parquet
E             * ........./tmpclsvtm7s/pre_MEDS/careplangeneral.parquet
E             * ........./tmpclsvtm7s/pre_MEDS/treatment.parquet
E             * ........./tmpclsvtm7s/pre_MEDS/vitalperiodic.parquet
E             * ........./tmpclsvtm7s/pre_MEDS/careplaninfectiousdisease.parquet
E             * ........./tmpclsvtm7s/pre_MEDS/medication.parquet
E             * ........./tmpclsvtm7s/pre_MEDS/infusiondrug.parquet
E           [2026-05-13 15:51:09,239][MEDS_transforms.extract.shard_events][INFO] - Will read raw data from ......................../tmp/tmpclsvtm7s/pre_MEDS/$IN_FILE.parquet and write sub-sharded data to ......................../tmpcd9bov59/eICU/shard_events/$IN_FILE/$ROW_START-$ROW_END.parquet
E           [2026-05-13 15:51:09,242][MEDS_transforms.extract.shard_events][INFO] - Processing ........./tmpclsvtm7s/pre_MEDS/diagnosis.parquet to ......................../tmpcd9bov59/eICU/shard_events/diagnosis.
E           [2026-05-13 15:51:09,242][MEDS_transforms.extract.shard_events][INFO] - Performing preliminary read of ........./tmpclsvtm7s/pre_MEDS/diagnosis.parquet to determine row count.
E           [2026-05-13 15:51:09,245][MEDS_transforms.extract.shard_events][INFO] - Ignoring infer_schema_length=999999999 for Parquet files.
E           [2026-05-13 15:51:09,249][MEDS_transforms.extract.shard_events][INFO] - Read 24978 rows from ........./tmpclsvtm7s/pre_MEDS/diagnosis.parquet.
E           [2026-05-13 15:51:09,249][MEDS_transforms.extract.shard_events][INFO] - Splitting ........./tmpclsvtm7s/pre_MEDS/diagnosis.parquet into 1 row-chunks of size 200000000.
E           [2026-05-13 15:51:09,249][MEDS_transforms.extract.shard_events][INFO] - Writing file 1/1: ........./tmpclsvtm7s/pre_MEDS/diagnosis.parquet row-chunk [0-24978) to ......................../tmpcd9bov59/eICU/shard_events/diagnosis/[0-24978).parquet.
E           [2026-05-13 15:51:09,253][MEDS_transforms.mapreduce.utils][INFO] - Reading input dataframe from ........./tmpclsvtm7s/pre_MEDS/diagnosis.parquet
E           [2026-05-13 15:51:09,253][MEDS_transforms.extract.shard_events][INFO] - Ignoring infer_schema_length=999999999 for Parquet files.
E           [2026-05-13 15:51:09,254][MEDS_transforms.mapreduce.utils][INFO] - Read dataset
E           [2026-05-13 15:51:09,254][MEDS_transforms.mapreduce.utils][INFO] - Writing final output to ......................../tmpcd9bov59/eICU/shard_events/diagnosis/[0-24978).parquet
E           [2026-05-13 15:51:09,262][MEDS_transforms.mapreduce.utils][INFO] - Succeeded in 0:00:00.009267
E           
E           [2026-05-13 15:51:09,392][MEDS_transforms.runner][INFO] - Command error:
E           Error executing job with overrides: ['stage=shard_events']
E           Traceback (most recent call last):
E             File "....../venvs/datasets/eICU/lib/python3.11.../MEDS_transforms/extract/shard_events.py", line 411, in main
E               rwlock_wrap(
E             File "....../venvs/datasets/eICU/lib/python3.11.../MEDS_transforms/mapreduce/utils.py", line 173, in rwlock_wrap
E               lock_fp.unlink()
E             File ".../hostedtoolcache/Python/3.11.15.../x64/lib/python3.11/pathlib.py", line 1147, in unlink
E               os.unlink(self)
E           FileNotFoundError: [Errno 2] No such file or directory: '......................../tmpcd9bov59/eICU/shard_events/diagnosis/[0-24978).parquet.lock'
E           
E           Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
E           
E           
E           [2026-05-13 15:51:09,495][eICU_MEDS.commands][ERROR] - Command failed with return code 1.
E           [2026-05-13 15:51:09,495][eICU_MEDS.commands][ERROR] - Command stderr:
E           Error executing job with overrides: ['pipeline_config_fp=....../venvs/datasets/eICU/lib/python3.11....../eICU_MEDS/configs/ETL.yaml', '~parallelize', '++do_overwrite=False']
E           Traceback (most recent call last):
E             File "....../venvs/datasets/eICU/lib/python3.11....../site-packages/MEDS_transforms/runner.py", line 329, in main
E               run_stage(cfg, stage, default_parallelization_cfg=default_parallelization_cfg)
E             File "....../venvs/datasets/eICU/lib/python3.11....../site-packages/MEDS_transforms/runner.py", line 271, in run_stage
E               raise ValueError(
E           ValueError: Stage shard_events failed via MEDS_extract-shard_events --config-dir=....../venvs/datasets/eICU/lib/python3.11........./site-packages/eICU_MEDS/configs --config-name=ETL 'hydra.searchpath=[pkg://MEDS_transforms.configs]' stage=shard_events with return code 1.
E           
E           Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
E           
E           
E           
E           stdout:
E           [2026-05-13 15:37:23,640][MEDS_DEV.utils][INFO] - Installing requirements from .../datasets/eICU/requirements.txt into virtual environment.
E           [2026-05-13 15:37:25,060][MEDS_DEV.utils][INFO] - Installed requirements from .../datasets/eICU/requirements.txt into virtual environment.
E           [2026-05-13 15:37:25,061][MEDS_DEV.datasets.__main__][INFO] - Considering running build command: MEDS_extract-eICU do_demo=True raw_input_dir="............/tmp/tmpclsvtm7s/raw" pre_MEDS_dir="......................../tmp/tmpclsvtm7s/pre_MEDS" MEDS_cohort_dir=".........................../tmp/tmpcd9bov59/eICU" log_dir="................../tmpcd9bov59/eICU/.pipeline_logs"
E           [2026-05-13 15:37:25,061][MEDS_DEV.utils][INFO] - Running command in .........................../tmpcd9bov59/eICU/cmd.sh:
E           #!/bin/bash
E           set -e
E           MEDS_extract-eICU do_demo=True raw_input_dir="............/tmp/tmpclsvtm7s/raw" pre_MEDS_dir="......................../tmp/tmpclsvtm7s/pre_MEDS" MEDS_cohort_dir=".........................../tmp/tmpcd9bov59/eICU" log_dir="................../tmpcd9bov59/eICU/.pipeline_logs"

tests/utils.py:146: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

mmcdermott · 2026-05-13T16:08:05Z

Splitting this into one PR per dataset for easier review (per #299 discussion). The cross-cutting demo_available infrastructure goes with each (will collapse to a single inclusion once the first one merges). Branch will be deleted; new PRs opened separately.

mmcdermott force-pushed the feat/add-datasets branch from 73023a4 to e3c7a4a Compare May 13, 2026 15:35

mmcdermott closed this May 13, 2026

mmcdermott deleted the feat/add-datasets branch May 13, 2026 16:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add six new datasets (EHRShot, HIRID, INSPIRE, NWICU, SICdb, eICU) + complete AUMCdb#299

Add six new datasets (EHRShot, HIRID, INSPIRE, NWICU, SICdb, eICU) + complete AUMCdb#299
mmcdermott wants to merge 1 commit into
devfrom
feat/add-datasets

mmcdermott commented May 12, 2026

Uh oh!

codecov Bot commented May 13, 2026

Uh oh!

mmcdermott commented May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

mmcdermott commented May 12, 2026

Summary

What's added

Fixes applied during local review

Open items needing maintainer / contributor input

Test plan

Supersedes / refs

Uh oh!

codecov Bot commented May 13, 2026

❌ 1 Tests Failed:

Uh oh!

mmcdermott commented May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant