Architecture Review: Handling Schema Evolution in a Netflix ETL Pipeline

[Thiago-code-lab]

Hello community!

I just released v1.0 of my Netflix Data Pipeline, and I'm looking for some feedback on the architectural decisions I made.

Project Context:
It's a Python-based ETL that ingests raw CSVs, performs heavy cleaning/feature engineering (Pandas), and loads normalized data into PostgreSQL via Docker.

🤔 Specific Questions for you:

1. Data Quality: Currently, I'm dropping rows that fail validation logic. For a production environment, would you recommend a "Dead Letter Queue" approach instead?
2. Orchestration: I wrote a custom Python script to trigger the steps (pipeline.py). At what point does it make sense to migrate this to Airflow or Prefect? Is it overkill for a dataset of this size (<10k rows)?
3. Docker Networking: I'm using docker-compose to bridge the app and the DB. Are there any security best practices I might be missing for the DB container?

Repo for context: https://github.com/Thiago-code-lab/data-engineering-netflix

Thanks in advance for any code reviews or architectural tips! I'm trying to adhere to clean code principles as much as possible.

[Leonardo-cyber-vale]

Great project! Here are some simplified insights for your architecture:
Data Quality: Definitely move to a Dead Letter Queue (DLQ). Dropping rows makes you lose visibility into why data is failing. Storing them separately allows for later auditing and reprocessing.
Orchestration: For <10k rows, Airflow is likely overkill. Stick to your script or use Prefect/Dagster if you need better UI and retry logic without the heavy infrastructure of Airflow.
Docker Networking: Ensure your DB container is not exposing ports to the public internet (remove the ports mapping in docker-compose if only the app needs access). Use a private Docker network and environment variables (secret files) for credentials instead of hardcoding them.
If your dataset grows or you need complex dependencies, that's the moment to migrate to a formal orchestrator!