diff --git a/README.md b/README.md
index 92b51e9dd..6946fa37a 100644
--- a/README.md
+++ b/README.md
@@ -332,5 +332,156 @@ For detailed development status and known issues, please check the [issue tracke
---
-Made with ❤️ by PRAISELab Team at University of Naples Federico II
+Made with love by PRAISELab Team at University of Naples Federico II
+
+---
+
+## ETL Pipeline Extension (AY 2025/2026)
+
+This fork extends `bibliometrix-python` with a fully source-agnostic ETL pipeline that
+replicates `convert2df()` from the R Bibliometrix package. All output conforms to the
+**WoS Field Tag schema**, making it compatible with every analytical function in
+`www/services/` and `www/functions/`.
+
+**Submitted by: Qamar Javed**
+*Data Science exam, Prof. Vincenzo Moscato, Federico II / UNINA, AY 2025/2026*
+
+---
+
+### Supported Sources
+
+| Source | Format | Retrieval |
+|---|---|---|
+| PubMed | MEDLINE / E-utilities API | Automated (API) |
+| OpenAlex | REST API (JSON) | Automated (API) |
+| Scopus | CSV export | File upload |
+| Scopus | BibTeX export | File upload |
+| Web of Science | TXT / CIW plaintext | File upload |
+| Web of Science | BibTeX export | File upload |
+| Dimensions | CSV / XLSX export | File upload |
+| Lens.org | CSV export | File upload |
+| Cochrane CDSR | TXT plaintext | File upload |
+| PubMed | MEDLINE TXT file export | File upload |
+
+---
+
+### Architecture
+
+Each pipeline phase is a dedicated module — monolithic functions are strictly prohibited.
+
+| Module | Phase | Responsibility |
+|---|---|---|
+| `www/services/mapping_dicts.py` | Config | 10 source mapping dicts; `MANDATORY_COLUMNS`, `LIST_FIELDS`, `SCALAR_FIELDS` |
+| `www/services/api_retriever.py` | Extract (API) | `fetch_pubmed()`, `fetch_openalex()` — pagination, exponential-backoff retry |
+| `www/services/standardizer.py` | Extract (file) + Transform | `load_file()`, `detect_source()`, `rename_columns()`, `enforce_types()`, `handle_nulls()`, `add_calculated_fields()`, `run_pipeline()` |
+| `www/services/validator.py` | Validate | `validate(df)` — raises `ValidationError` naming the failing column; returns structured report |
+| `dashboard/app.py` | Present | Streamlit dashboard (five tabs: API Query, File Upload, Validation, Analysis, About) |
+| `tests/test_etl.py` | Test | pytest suite covering all pipeline phases and all 10 source types |
+
+---
+
+### Mapping Strategy
+
+Column names are **never hardcoded** anywhere in the pipeline code.
+`mapping_dicts.py` contains one dictionary per source that maps source-native field
+names to WoS Field Tags. The dispatcher in `standardizer.py` selects the correct
+dictionary automatically.
+
+Selected mappings:
+
+| Source | Example mapping |
+|---|---|
+| PubMed API | `FAU` → `AF`, `MH` → `ID`, `DP` → `PY`, `TA` → `JI` |
+| OpenAlex API | `display_name` → `TI`, `cited_by_count` → `TC`, `author_names` → `AU` |
+| Scopus CSV | `EID` → `UT`, `Cited by` → `TC`, `Author Keywords` → `DE` |
+| Scopus BibTeX | `author` → `AU`, `note` → `TC` (Cited by: N), `url` → `UT` (EID) |
+| WoS BibTeX | `ID` → `UT`, `keywords-plus` → `ID`, `times-cited` → `TC` |
+| Dimensions | `Publication ID` → `UT`, `PubYear` → `PY`, `MeSH terms` → `ID` |
+| Lens.org | `Lens ID` → `UT`, `Author/s` → `AU`, `Citing Works Count` → `TC` |
+| Cochrane | `ID` → `UT`, `YR` → `PY`, `KY` → `DE`, `DOI` → `DI` |
+| PubMed file | `IP` → `IS`, `IS` → `SN`, `LID` → `DI`, `PMID` → `PMID` |
+
+All 10 sources pass through the **same** downstream `enforce_types` / `handle_nulls` /
+`add_calculated_fields` pipeline — zero duplicated transform logic.
+
+---
+
+### Type Contracts
+
+| Field | Type | Rule |
+|---|---|---|
+| `AU`, `AF`, `C1`, `CR`, `DE`, `ID` | `list[str]` | Split on `;`; `[]` if missing |
+| All other scalar fields | `str` | `""` if missing |
+| `PY` | `str` | 4-digit year extracted from full date string |
+| `TC` | `int` | `0` if missing or unparseable |
+| `DB` | `str` | Set from `SOURCE_TO_DB` map (e.g. `SCOPUS_CSV` → `"SCOPUS"`) |
+| `SR` | `str` | Computed by existing `SR(M)` in `metatagextraction.py` |
+
+Mandatory output columns (24):
+`DB`, `UT`, `DI`, `PMID`, `TI`, `SO`, `JI`, `PY`, `DT`, `LA`, `TC`, `AU`, `AF`,
+`C1`, `RP`, `CR`, `DE`, `ID`, `AB`, `VL`, `IS`, `BP`, `EP`, `SR`
+
+---
+
+### SR Field
+
+The Short Reference field is computed by calling the existing `SR(M)` function
+from `www/services/metatagextraction.py`. It is **never reimplemented** — the
+ETL pipeline calls it directly from `add_calculated_fields()`, with a faithful
+fallback for environments where Shiny-specific deps are absent.
+
+---
+
+### Patches Applied to Existing Code
+
+Six patches were applied in place to accept all new DB values. Every patch is
+marked with a `# PATCHED:` comment giving the reason.
+
+| File | Change |
+|---|---|
+| `www/services/histnetwork.py` | Added `"WOS"` to the `Web_of_Science` branch; added `"SCOPUS"` to the Scopus branch; added `"DIMENSIONS"`, `"LENS"`, `"COCHRANE"` routed through WoS citation analysis |
+| `www/services/biblionetwork.py` | Fixed `db_name == "SCOPUS"` case-sensitivity bug (was never matching); added `"wos"` to WoS branch in `label_short()`; added `"dimensions"`, `"lens"`, `"cochrane"` alongside `"pubmed"` / `"openalex"` |
+| `www/services/metatagextraction.py` | Added `"SCOPUS"`, `"DIMENSIONS"`, `"LENS"`, `"COCHRANE"` to the `AU_UN` C3 institution-name override check |
+
+---
+
+### Running the ETL Dashboard
+
+```bash
+streamlit run dashboard/app.py
+```
+
+The dashboard has five tabs:
+
+1. **API Query** — enter a query, choose PubMed or OpenAlex, set result count, click Run Pipeline
+2. **File Upload** — upload a bibliographic file (auto-detected or format selected), click Process File
+3. **Validation** — per-check pass/fail status for the most recent pipeline run
+4. **Analysis** — summary metrics, publications-per-year chart, top authors, top keywords
+5. **About** — architecture description, supported sources, mandatory columns, patch table
+
+---
+
+### Running Tests
+
+```bash
+# Unit tests only (no network, no file I/O)
+pytest tests/test_etl.py -m "not integration and not file_sources"
+
+# File-source tests (requires sources/ directory)
+pytest tests/test_etl.py -m "file_sources"
+
+# Full suite including live API calls
+pytest tests/test_etl.py
+
+# Verbose output
+pytest tests/test_etl.py -v
+```
+
+---
+
+### Data Output
+
+Standardized CSVs are written to `data/outputs/`. Multi-value fields use `;`
+as delimiter, matching the format expected by all analytical functions in
+`www/services/` and `www/functions/`.
diff --git a/dashboard/app.py b/dashboard/app.py
new file mode 100644
index 000000000..f3d30c950
--- /dev/null
+++ b/dashboard/app.py
@@ -0,0 +1,1053 @@
+"""
+Bibliometrix Python — Streamlit Dashboard
+
+Five sections:
+ 1. API Query — run the ETL pipeline via PubMed/OpenAlex API
+ 2. File Upload — run the ETL pipeline from a local file
+ 3. Validation — per-check pass/fail status
+ 4. Analysis — summary metrics and Plotly charts
+ 5. About — architecture description
+
+Design rules:
+ - No emojis anywhere
+ - Custom CSS injected at startup (DM Sans, deep purple palette)
+ - All charts use plotly.graph_objects with custom template
+ - Sidebar for all controls
+
+Submission: Qamar Javed — Data Science exam, Prof. Vincenzo Moscato,
+ Federico II / UNINA, AY 2025/2026
+"""
+
+import io
+import logging
+import sys
+import tempfile
+import types
+from pathlib import Path
+
+import pandas as pd
+import plotly.graph_objects as go
+import streamlit as st
+
+# ---------------------------------------------------------------------------
+# Package stubbing — MUST run before any www.services import.
+#
+# www/services/__init__.py wildcard-imports Shiny-specific modules that pull
+# in prince, igraph, faicons, etc., which are not installed in this Python
+# 3.12 Streamlit environment. Registering lightweight stubs prevents Python
+# from running __init__.py; ETL-specific submodules are loaded directly.
+#
+# Additionally, parsers.py does `from .utils import *`, and utils.py imports
+# the same Shiny deps. We stub www.services.utils with just `re` (the only
+# name parsers.py actually uses from utils) so parsers.py loads cleanly.
+# ---------------------------------------------------------------------------
+_root = str(Path(__file__).resolve().parent.parent)
+if _root not in sys.path:
+ sys.path.insert(0, _root)
+
+
+def _stub_pkg(name: str, fs_path: str) -> None:
+ if name not in sys.modules:
+ mod = types.ModuleType(name)
+ mod.__path__ = [fs_path] # type: ignore[attr-defined]
+ mod.__package__ = name
+ sys.modules[name] = mod
+
+
+_stub_pkg("www", str(Path(_root) / "www"))
+_stub_pkg("www.services", str(Path(_root) / "www" / "services"))
+
+# Stub www.services.utils so parsers.py can do `from .utils import *`
+# without importing the Shiny-only dependencies in utils.py.
+if "www.services.utils" not in sys.modules:
+ import re as _re
+ _utils_mod = types.ModuleType("www.services.utils")
+ _utils_mod.re = _re # type: ignore[attr-defined]
+ sys.modules["www.services.utils"] = _utils_mod
+
+from www.services.api_retriever import fetch_openalex, fetch_pubmed # noqa: E402
+from www.services.mapping_dicts import LIST_FIELDS # noqa: E402
+from www.services.standardizer import ( # noqa: E402
+ export_to_csv,
+ load_file,
+ run_pipeline,
+)
+from www.services.validator import ValidationError, validate # noqa: E402
+
+logging.basicConfig(level=logging.INFO)
+
+# ---------------------------------------------------------------------------
+# CSS
+# ---------------------------------------------------------------------------
+
+CUSTOM_CSS = """
+
+"""
+
+# ---------------------------------------------------------------------------
+# Plotly custom template
+# ---------------------------------------------------------------------------
+
+PLOTLY_TEMPLATE = go.layout.Template(
+ layout=go.Layout(
+ font=dict(family="DM Sans, sans-serif", size=12, color="#374151"),
+ paper_bgcolor="white",
+ plot_bgcolor="white",
+ colorway=["#7c3aed", "#a855f7", "#c084fc", "#6d28d9", "#4c1d95"],
+ xaxis=dict(
+ gridcolor="#f5f3ff",
+ linecolor="#ede9fe",
+ tickfont=dict(size=11),
+ title_font=dict(size=11, color="#7c3aed"),
+ ),
+ yaxis=dict(
+ gridcolor="#f5f3ff",
+ linecolor="#ede9fe",
+ tickfont=dict(size=11),
+ title_font=dict(size=11, color="#7c3aed"),
+ ),
+ margin=dict(l=48, r=24, t=36, b=44),
+ hoverlabel=dict(
+ bgcolor="white",
+ bordercolor="#ede9fe",
+ font=dict(family="DM Sans, sans-serif", size=12, color="#2e1760"),
+ ),
+ )
+)
+
+# File format label → Source literal for load_file()
+_FILE_FORMAT_MAP: dict[str, str | None] = {
+ "Auto-detect": None,
+ "Scopus CSV": "SCOPUS_CSV",
+ "Scopus BibTeX": "SCOPUS_BIB",
+ "WoS TXT / CIW": "WOS_TXT",
+ "WoS BibTeX": "WOS_BIB",
+ "Dimensions CSV/XLSX":"DIMENSIONS",
+ "Lens.org CSV": "LENS",
+ "Cochrane TXT": "COCHRANE",
+ "PubMed TXT": "PUBMED_FILE",
+}
+
+
+# ---------------------------------------------------------------------------
+# Session state helpers
+# ---------------------------------------------------------------------------
+
+def _init_state() -> None:
+ defaults: dict = {
+ # API Query
+ "df": None,
+ "validation_report": None,
+ "pipeline_error": None,
+ "csv_bytes": None,
+ "source_used": None,
+ # File Upload
+ "file_df": None,
+ "file_report": None,
+ "file_error": None,
+ "file_csv_bytes": None,
+ "file_source": None,
+ }
+ for k, v in defaults.items():
+ if k not in st.session_state:
+ st.session_state[k] = v
+
+
+# ---------------------------------------------------------------------------
+# Pipeline runners
+# ---------------------------------------------------------------------------
+
+def _run_api_pipeline(query: str, platform: str, max_results: int) -> None:
+ """Execute Extract → Transform → Validate → Load for an API source."""
+ st.session_state.df = None
+ st.session_state.validation_report = None
+ st.session_state.pipeline_error = None
+ st.session_state.csv_bytes = None
+ st.session_state.source_used = platform
+
+ progress = st.progress(0, text="Starting pipeline...")
+ try:
+ progress.progress(10, text="Phase 1: Extracting from " + platform + "...")
+ if platform == "PubMed":
+ raw_df = fetch_pubmed(query, max_results)
+ else:
+ raw_df = fetch_openalex(query, max_results)
+
+ progress.progress(40, text="Phase 2: Transforming...")
+ std_df = run_pipeline(raw_df)
+
+ progress.progress(70, text="Phase 3: Validating...")
+ report = validate(std_df)
+ st.session_state.validation_report = report
+
+ progress.progress(90, text="Phase 4: Exporting CSV...")
+ export_to_csv(std_df, platform.upper())
+
+ export_df = std_df.copy()
+ for col in LIST_FIELDS:
+ if col in export_df.columns:
+ export_df[col] = export_df[col].apply(
+ lambda v: ";".join(v) if isinstance(v, list) else str(v)
+ )
+ buf = io.BytesIO()
+ export_df.to_csv(buf, index=False, encoding="utf-8")
+ st.session_state.csv_bytes = buf.getvalue()
+ st.session_state.df = std_df
+ progress.progress(100, text="Pipeline complete.")
+
+ except ValidationError as exc:
+ st.session_state.pipeline_error = f"Validation failed: {exc}"
+ progress.empty()
+ except Exception as exc:
+ st.session_state.pipeline_error = f"Pipeline error: {exc}"
+ progress.empty()
+
+
+def _run_file_pipeline(uploaded, format_hint: "str | None") -> None:
+ """Execute Extract (file) → Transform → Validate → Load for a file source."""
+ st.session_state.file_df = None
+ st.session_state.file_report = None
+ st.session_state.file_error = None
+ st.session_state.file_csv_bytes = None
+ st.session_state.file_source = None
+
+ suffix = Path(uploaded.name).suffix or ".tmp"
+ progress = st.progress(0, text="Reading file...")
+ tmp_path: "Path | None" = None
+
+ try:
+ # Save uploaded bytes to a temp file (parsers require file paths)
+ with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as tmp:
+ tmp.write(uploaded.getvalue())
+ tmp_path = Path(tmp.name)
+
+ progress.progress(15, text="Phase 1: Loading " + uploaded.name + "...")
+ raw_df, source = load_file(tmp_path, source=format_hint) # type: ignore[arg-type]
+
+ progress.progress(40, text="Phase 2: Transforming (" + source + ")...")
+ std_df = run_pipeline(raw_df, source=source)
+
+ progress.progress(70, text="Phase 3: Validating...")
+ report = validate(std_df)
+ st.session_state.file_report = report
+
+ progress.progress(90, text="Phase 4: Exporting CSV...")
+ export_to_csv(std_df, source)
+
+ export_df = std_df.copy()
+ for col in LIST_FIELDS:
+ if col in export_df.columns:
+ export_df[col] = export_df[col].apply(
+ lambda v: ";".join(v) if isinstance(v, list) else str(v)
+ )
+ buf = io.BytesIO()
+ export_df.to_csv(buf, index=False, encoding="utf-8")
+ st.session_state.file_csv_bytes = buf.getvalue()
+ st.session_state.file_df = std_df
+ st.session_state.file_source = source
+ progress.progress(100, text="Pipeline complete.")
+
+ except ValidationError as exc:
+ st.session_state.file_error = f"Validation failed: {exc}"
+ progress.empty()
+ except Exception as exc:
+ st.session_state.file_error = f"Pipeline error: {exc}"
+ progress.empty()
+ finally:
+ if tmp_path and tmp_path.exists():
+ tmp_path.unlink(missing_ok=True)
+
+
+# ---------------------------------------------------------------------------
+# Shared preview helper
+# ---------------------------------------------------------------------------
+
+def _preview_df(df: pd.DataFrame, source_label: str, csv_bytes: bytes) -> None:
+ """Render a success banner, download button, and 20-row preview table."""
+ col_l, col_r = st.columns([6, 2])
+ with col_l:
+ st.success(f"Retrieved {len(df)} records from {source_label}.")
+ with col_r:
+ if csv_bytes:
+ st.download_button(
+ label="Download CSV",
+ data=csv_bytes,
+ file_name=f"{source_label.lower()}_standardized.csv",
+ mime="text/csv",
+ use_container_width=True,
+ )
+
+ preview_cols = [c for c in ["TI", "AU", "PY", "SO", "TC", "DI", "SR"] if c in df.columns]
+ preview_df = df[preview_cols].head(20).copy()
+ for col in preview_cols:
+ if col in LIST_FIELDS:
+ preview_df[col] = preview_df[col].apply(
+ lambda v: "; ".join(v[:3]) + (" ..." if len(v) > 3 else "")
+ if isinstance(v, list) else str(v)
+ )
+ st.dataframe(preview_df, use_container_width=True, height=420)
+
+
+# ---------------------------------------------------------------------------
+# Section renderers
+# ---------------------------------------------------------------------------
+
+def _render_api_query() -> None:
+ st.markdown('API Query
', unsafe_allow_html=True)
+
+ if st.session_state.pipeline_error:
+ st.error(st.session_state.pipeline_error)
+
+ df = st.session_state.df
+ if df is not None:
+ _preview_df(df, st.session_state.source_used or "API", st.session_state.csv_bytes or b"")
+ else:
+ st.info("Configure your query in the sidebar and click Run Pipeline.")
+
+
+def _render_file_upload() -> None:
+ st.markdown('File Upload
', unsafe_allow_html=True)
+
+ if st.session_state.file_error:
+ st.error(st.session_state.file_error)
+
+ df = st.session_state.file_df
+ if df is not None:
+ src = st.session_state.file_source or "file"
+ _preview_df(df, src, st.session_state.file_csv_bytes or b"")
+ else:
+ st.markdown(
+ ''
+ 'Upload a bibliographic export file using the sidebar controls, '
+ 'select its format (or leave on Auto-detect), then click '
+ 'Process File. Supported formats: '
+ 'Scopus CSV, Scopus BibTeX, WoS TXT/CIW, WoS BibTeX, '
+ 'Dimensions CSV/XLSX, Lens.org CSV, Cochrane CDSR TXT, PubMed MEDLINE TXT.'
+ '
',
+ unsafe_allow_html=True,
+ )
+
+
+def _render_validation_report() -> None:
+ st.markdown('Validation Report
', unsafe_allow_html=True)
+
+ # Prefer file upload report when present; fall back to API report
+ report = st.session_state.file_report if st.session_state.file_report is not None else st.session_state.validation_report
+ if report is None:
+ st.info("Run a query or process a file first to see validation results.")
+ return
+
+ overall = report.get("passed", False)
+ oc = "pass" if overall else "fail"
+ ol = "All checks passed" if overall else "One or more checks failed"
+ st.markdown(
+ f''
+ f'{"PASS" if overall else "FAIL"}'
+ f'{ol}'
+ f'
',
+ unsafe_allow_html=True,
+ )
+
+ for check in report.get("checks", []):
+ passed = check.get("passed", False)
+ bc = "pass" if passed else "fail"
+ badge = "PASS" if passed else "FAIL"
+ problems = check.get("problem_columns", [])
+ detail = (
+ f'Problem columns: {", ".join(problems)}'
+ if problems else ""
+ )
+ st.markdown(
+ f''
+ f'{badge}'
+ f'{check.get("name", "")}'
+ f'{detail}'
+ f'
',
+ unsafe_allow_html=True,
+ )
+
+
+def _render_analysis() -> None:
+ st.markdown('Analysis
', unsafe_allow_html=True)
+
+ # Prefer file upload data; fall back to API data
+ df = st.session_state.file_df if st.session_state.file_df is not None else st.session_state.df
+ if df is None:
+ st.info("Run a query or process a file first to see analysis.")
+ return
+
+ # Summary metrics
+ total_records = len(df)
+ year_col = df["PY"].replace("", pd.NA).dropna()
+ year_range = f"{year_col.min()} - {year_col.max()}" if len(year_col) > 0 else "N/A"
+ unique_journals = df["SO"].replace("", pd.NA).dropna().nunique()
+
+ au_col = df["AU"].apply(lambda v: v if isinstance(v, list) else [])
+ all_authors = [a for sub in au_col for a in sub if a]
+ unique_authors = len(set(all_authors))
+
+ col1, col2, col3, col4 = st.columns(4)
+ cards = [
+ ("Total Records", str(total_records), False),
+ ("Year Range", year_range, True),
+ ("Unique Journals", str(unique_journals), False),
+ ("Unique Authors", str(unique_authors), False),
+ ]
+ for col_widget, (label, value, small) in zip([col1, col2, col3, col4], cards):
+ with col_widget:
+ cls = "small" if small else ""
+ st.markdown(
+ f''
+ f'
{label}
'
+ f'
{value}
'
+ f'
", unsafe_allow_html=True)
+
+ # Publications per year
+ py_counts = df["PY"].replace("", pd.NA).dropna().value_counts().sort_index()
+ if len(py_counts) > 0:
+ fig_year = go.Figure(
+ go.Bar(
+ x=py_counts.index.tolist(),
+ y=py_counts.values.tolist(),
+ marker=dict(
+ color=py_counts.values.tolist(),
+ colorscale=[[0, "#ddd6fe"], [1, "#4c1d95"]],
+ showscale=False,
+ ),
+ hovertemplate="%{x}
%{y} records",
+ )
+ )
+ fig_year.update_layout(
+ template=PLOTLY_TEMPLATE,
+ xaxis_title="Publication Year",
+ yaxis_title="Records",
+ bargap=0.3,
+ )
+ st.markdown(
+ ''
+ '
Publications per Year
'
+ '
Annual output of retrieved records
',
+ unsafe_allow_html=True,
+ )
+ st.plotly_chart(fig_year, use_container_width=True, config={"displayModeBar": False})
+ st.markdown("
%{x} records",
+ )
+ )
+ fig_au.update_layout(
+ template=PLOTLY_TEMPLATE,
+ xaxis_title="Records",
+ margin=dict(l=160, r=16, t=28, b=36),
+ height=340,
+ bargap=0.25,
+ )
+ st.markdown(
+ ''
+ '
Top 10 Authors
'
+ '
By record count
',
+ unsafe_allow_html=True,
+ )
+ st.plotly_chart(fig_au, use_container_width=True, config={"displayModeBar": False})
+ st.markdown("
%{x} occurrences",
+ )
+ )
+ fig_kw.update_layout(
+ template=PLOTLY_TEMPLATE,
+ xaxis_title="Frequency",
+ margin=dict(l=200, r=16, t=28, b=36),
+ height=340,
+ bargap=0.25,
+ )
+ st.markdown(
+ ''
+ '
Top 15 Author Keywords
'
+ '
DE field — author-supplied terms
',
+ unsafe_allow_html=True,
+ )
+ st.plotly_chart(fig_kw, use_container_width=True, config={"displayModeBar": False})
+ st.markdown("
About
', unsafe_allow_html=True)
+ st.markdown(
+ """
+
+
+ A source-agnostic bibliometric ETL pipeline that replicates
+ convert2df() from the R Bibliometrix package.
+ All output conforms to the WoS Field Tag schema so every analytical
+ function in www/services/ runs without modification.
+ Submitted by Qamar Javed.
+
+
+
Sources — API
+
+ - PubMed — NCBI E-utilities (esearch + efetch, MEDLINE format) with exponential-backoff retry
+ - OpenAlex — REST API with cursor pagination and rate-limit handling
+
+
+
Sources — File Upload
+
+ - Scopus CSV — standard CSV export from the Scopus interface
+ - Scopus BibTeX — BibTeX export; EID extracted from URL, TC from note field
+ - Web of Science TXT / CIW — MEDLINE-style plaintext; parsed by
parse_wos_data()
+ - Web of Science BibTeX — BibTeX export; WoS accession number used as UT
+ - Dimensions CSV / XLSX — export with header row offset (skiprows=1)
+ - Lens.org CSV — standard CSV export from the Lens interface
+ - Cochrane CDSR TXT — KEY: value format; parsed by
parse_cochrane_data()
+ - PubMed TXT — MEDLINE plaintext file export; LID used for DOI
+
+
+
Pipeline Phases
+
+ 1
+ Extract — API retrieval or file loading via load_file(); auto-detects format from extension and content fingerprint
+
+
+ 2
+ Transform — rename via mapping_dicts.py, enforce type contracts (AU/AF as list[str], TC as int, PY as 4-digit string), eliminate nulls, compute SR
+
+
+ 3
+ Validate — mandatory columns, zero-null guarantee, list[str] type contract; raises ValidationError naming the failing column
+
+
+ 4
+ Load — CSV export to data/outputs/ with ;-delimited multi-value fields for compatibility with all analytical functions
+
+
+
Mandatory Output Columns (24)
+
+ DBUTDI
+ PMIDTISO
+ JIPYDT
+ LATCAU
+ AFC1RP
+ CRDEID
+ ABVLIS
+ BPEPSR
+
+
+
Modules
+
+ mapping_dicts.py — 10 source mapping dicts; schema constants (MANDATORY_COLUMNS, LIST_FIELDS, SCALAR_FIELDS)api_retriever.py — fetch_pubmed(), fetch_openalex()standardizer.py — load_file(), detect_source(), rename_columns(), enforce_types(), handle_nulls(), add_calculated_fields(), run_pipeline()validator.py — validate(), ValidationErrordashboard/app.py — this Streamlit interface (five sections)tests/test_etl.py — pytest suite covering all pipeline phases and all file sources
+
+
Patches Applied to Existing Code
+
+ histnetwork.py — added DIMENSIONS, LENS, COCHRANE, SCOPUS, WOS routing to WoS/Scopus citation analysis pathsbiblionetwork.py — fixed case-insensitive SCOPUS comparison; added new sources to label_short()metatagextraction.py — added SCOPUS, DIMENSIONS, LENS, COCHRANE to AU_UN C3 override check
+
+
+
'
+ '
Bibliometrix Python
'
+ '
Source-agnostic bibliometric ETL pipeline — WoS Field Tag schema output
'
+ '
'
+ 'PubMed'
+ 'OpenAlex'
+ 'Scopus'
+ 'WoS'
+ 'Dimensions'
+ 'Lens'
+ 'Cochrane'
+ '
'
+ '