diff --git a/CHANGELOG.md b/CHANGELOG.md index 11a9262..ab5dc9f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,17 @@ All notable changes to this project will be documented in this file. Format follows [Keep a Changelog](https://keepachangelog.com/). Versioning follows [CalVer](https://calver.org/) (`YYYY.M.DD`). +## [Unreleased] + +### Added +- `docs/DEVELOPING_PLUGINS.md` and `docs/DEVELOPING_PLUGINS.pl.md` with a full + broker-plugin development guide, cookiecutter flow, tax checklist, and + worked Revolut/Binance examples. + +### Changed +- Linked the new plugin-development guides from `README.md`, + `CONTRIBUTING.md`, and `pit38/plugins/README.md`. + ## [2026.4.20] — 2026-04-20 ### Added diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0e45df2..a5b63b1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -163,9 +163,12 @@ General code style: Broker plugins live under `pit38/plugins/` and transform a broker-specific CSV export into the standardized format that `pit38` consumes. Each -plugin is self-contained. See -[`pit38/plugins/README.md`](pit38/plugins/README.md) for the plugin -architecture overview (PL/EN). +plugin is self-contained. + +Primary plugin guides: +- [`docs/DEVELOPING_PLUGINS.md`](docs/DEVELOPING_PLUGINS.md) (English) +- [`docs/DEVELOPING_PLUGINS.pl.md`](docs/DEVELOPING_PLUGINS.pl.md) (Polish) +- [`pit38/plugins/README.md`](pit38/plugins/README.md) (quick index) **Reference implementations** (read these first): diff --git a/README.md b/README.md index 9332bae..4b0025e 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,9 @@ A command-line tool for calculating Polish income tax on **stocks** and **crypto | Manual CSV | Yes | Yes | For broker-specific quirks see [`docs/BROKERS.md`](docs/BROKERS.md). +For plugin development see: +- [`docs/DEVELOPING_PLUGINS.md`](docs/DEVELOPING_PLUGINS.md) (English) +- [`docs/DEVELOPING_PLUGINS.pl.md`](docs/DEVELOPING_PLUGINS.pl.md) (Polish) ## Quick Start @@ -90,7 +93,10 @@ For a detailed description of the rules, see: ## Contributing -Contributions are welcome — including first-time open-source PRs and new broker plugins. See **[CONTRIBUTING.md](CONTRIBUTING.md)** for dev setup, test conventions, PR guidelines, and a walkthrough for adding a new broker plugin. +Contributions are welcome — including first-time open-source PRs and new broker plugins. See **[CONTRIBUTING.md](CONTRIBUTING.md)** for dev setup, test conventions, and PR guidelines. +For plugin implementation details, use: +- [`docs/DEVELOPING_PLUGINS.md`](docs/DEVELOPING_PLUGINS.md) +- [`docs/DEVELOPING_PLUGINS.pl.md`](docs/DEVELOPING_PLUGINS.pl.md) Quick dev setup: diff --git a/docs/DEVELOPING_PLUGINS.md b/docs/DEVELOPING_PLUGINS.md new file mode 100644 index 0000000..8ac42b1 --- /dev/null +++ b/docs/DEVELOPING_PLUGINS.md @@ -0,0 +1,263 @@ +# Developing Broker Plugins + +This guide is for contributors who want to add support for a new broker import +in `pit-38`. + +Related docs: +- [`CONTRIBUTING.md`](../CONTRIBUTING.md) for setup and PR rules +- [`docs/BROKERS.md`](BROKERS.md) for broker-specific notes +- [`docs/DEVELOPING_PLUGINS.pl.md`](DEVELOPING_PLUGINS.pl.md) for Polish version + +## 1. What is a broker plugin? + +A broker plugin converts a broker export (CSV, PDF, etc.) into pit-38's +canonical transaction shape: + +- `date` +- `operation` +- `amount` +- `symbol` +- `fiat_value` +- `currency` + +After conversion, normal loaders and tax calculators process the output. + +## 2. Broker plugin contract (current code) + +There is no single `BrokerPlugin` class in the repo right now. The contract is +split into small pieces: + +1. A CLI import command in [`pit38/cli.py`](../pit38/cli.py) +2. A parser/reader that maps broker rows to domain objects +3. A saver that writes canonical CSV + +For stock plugins, formatting is driven by `BaseFormatter`: + +```python +class BaseFormatter(ABC): + @abstractmethod + def format(self, item: Any) -> Dict[str, Any]: + pass + + @abstractmethod + def item_type(self) -> Type: + pass +``` + +Reference files: +- [`pit38/plugins/stock/formatters.py`](../pit38/plugins/stock/formatters.py) +- [`pit38/plugins/stock/generic_saver.py`](../pit38/plugins/stock/generic_saver.py) +- [`pit38/plugins/crypto/generic_saver.py`](../pit38/plugins/crypto/generic_saver.py) + +## 3. Step by step: add a new broker (cookiecutter flow) + +### 3.1 Scaffold + +```bash +pip install cookiecutter +cookiecutter gh:pbialon/pit-38-broker-template +``` + +### 3.2 Inspect generated files + +At minimum, confirm: +- parser module +- csv reader/service module +- CLI entry module or function +- tests for parser/service +- README for the plugin + +### 3.3 Get a real export sample + +Use real broker exports with sanitized personal data: +- remove account ids +- remove names +- keep numeric and date format exactly as in source + +### 3.4 Implement parsing + +Map broker-specific operation names to domain operations: +- `BUY` +- `SELL` +- `DIVIDEND` +- `SERVICE_FEE` +- `STOCK_SPLIT` (stocks only) + +Write plugin output in canonical columns only. + +### 3.5 Wire into CLI + +Register command under `pit38 import` in [`pit38/cli.py`](../pit38/cli.py): +- input path option(s) +- output path option +- optional `--log-level` +- short success summary + +### 3.6 Run tests + +```bash +pytest tests/ +pytest tests/ --cov=pit38 --cov-branch +``` + +### 3.7 Open PR + +- link issue (`Closes #...`) +- include changelog line +- mention tax impact in PR template + +## 4. Common patterns + +### Date parsing + +- Prefer `pendulum.parse(...)` when source is ISO-like +- Use explicit `datetime.strptime(...)` for custom formats + +Example: Binance parser uses `%y-%m-%d %H:%M:%S` in +[`pit38/plugins/crypto/binance/csv.py`](../pit38/plugins/crypto/binance/csv.py). + +### Currency and amount normalization + +Use shared helpers instead of new regex per plugin: +- `normalize_currency_layout` +- `parse_amount` + +From [`pit38/plugins/normalization.py`](../pit38/plugins/normalization.py). + +### Missing columns and malformed rows + +- Fail early on missing required columns +- Skip unknown non-tax-relevant rows with clear warning +- Do not crash whole import on one bad row if behavior should be tolerant + +### Signed amounts and fee handling + +Keep semantics explicit: +- service fee is cost +- dividend is income +- do not silently flip signs without a test that documents why + +## 5. Testing your plugin + +Use helpers from [`tests/utils.py`](../tests/utils.py): +- `buy()`, `sell()`, `apple()`, `usd()`, `zl()` +- `StubExchanger` + +Typical test layers: +- parser unit tests (row -> domain object) +- csv service tests (file -> list of objects) +- e2e test with sanitized fixture in `tests/e2e/fixtures/` + +Minimal parser test sketch: + +```python +from unittest import TestCase +from tests.utils import buy, apple, usd + +class TestMyBrokerParser(TestCase): + def test_buy_row(self): + row = {"date": "2025-01-02 10:00:00", "type": "BUY", "qty": "1", "ticker": "AAPL", "value": "USD 200.00"} + tx = parse_row(row) + self.assertEqual(tx, buy(apple(1), usd(200.0), "2025-01-02 10:00:00")) +``` + +## 6. Tax-correctness checklist + +Before opening PR, verify: + +- Fee treatment: + - broker fee included in gross row? + - broker fee emitted as separate operation? +- Gross vs net: + - are proceeds net of fee or gross? + - did you avoid double-counting fee? +- Dividend withholding: + - broker reports gross dividend, net dividend, or both? + - which amount is canonical for PIT-38 flow here? +- Currency: + - original currency preserved before NBP conversion? +- Stock split: + - ratio stored without fake fiat value? + +If in doubt, open a Discussion with anonymized sample rows. + +## 7. Worked examples (before scaffold -> after plugin) + +### 7.1 Revolut stock example + +```diff +# scaffold (generated) +class RowParser: + def parse(self, row): + raise NotImplementedError + +# after implementation +class TransactionRowParser(RowParser): + OPERATIONS_HANDLED = {OperationType.BUY, OperationType.SELL} + + @classmethod + def parse(cls, row): + operation_type = cls._operation_type(row) + if operation_type not in cls.OPERATIONS_HANDLED: + return None + return Transaction( + asset=cls._asset(row), + fiat_value=cls._fiat_value(row), + action=operation_type, + date=cls._date(row), + ) +``` + +Related files: +- [`pit38/plugins/stock/revolut/transaction_row_parser.py`](../pit38/plugins/stock/revolut/transaction_row_parser.py) +- [`pit38/plugins/stock/revolut/operation_row_parser.py`](../pit38/plugins/stock/revolut/operation_row_parser.py) + +### 7.2 Binance crypto example + +```diff +# scaffold (generated) +def read(path): + return [] + +# after implementation +class BinanceTransactionProcessor: + def read(self, file_path: str) -> List[Transaction]: + convert_operations = [] + transaction_operations = [] + with open_csv_reader(file_path) as reader: + for row in reader: + tx = BinanceTransaction(row) + if tx.operation_type() == BinanceOperationType.DEPOSIT: + continue + if tx.operation_type() == BinanceOperationType.CONVERT: + convert_operations.append(tx) + else: + transaction_operations.append(tx) + all_transactions = ( + self._process_convert_transactions(convert_operations) + + self._process_transaction_operations(transaction_operations) + ) + return sorted(all_transactions, key=lambda x: x.date) +``` + +Related file: +- [`pit38/plugins/crypto/binance/csv.py`](../pit38/plugins/crypto/binance/csv.py) + +## 8. Transaction flow + +```mermaid +graph LR + A[Broker CSV] --> B[RowParser] + B --> C[Canonical CSV] + C --> D[GenericCsvLoader] + D --> E[Transaction objects] + E --> F[Tax Calculator] +``` + +## 9. Getting help + +- Discussions: +- Issues: +- Tax rules docs: + - [`docs/TAX_RULES.md`](TAX_RULES.md) + - [`docs/TAX_RULES.pl.md`](TAX_RULES.pl.md) diff --git a/docs/DEVELOPING_PLUGINS.pl.md b/docs/DEVELOPING_PLUGINS.pl.md new file mode 100644 index 0000000..d934dbc --- /dev/null +++ b/docs/DEVELOPING_PLUGINS.pl.md @@ -0,0 +1,261 @@ +# Tworzenie pluginow brokerskich + +Ten dokument jest dla osob, ktore chca dodac nowy import brokera do `pit-38`. + +Powiazane dokumenty: +- [`CONTRIBUTING.md`](../CONTRIBUTING.md) - setup i zasady PR +- [`docs/BROKERS.md`](BROKERS.md) - notatki per broker +- [`docs/DEVELOPING_PLUGINS.md`](DEVELOPING_PLUGINS.md) - wersja English + +## 1. Czym jest plugin brokera? + +Plugin brokera konwertuje eksport brokera (CSV, PDF itp.) do kanonicznego +formatu transakcji pit-38: + +- `date` +- `operation` +- `amount` +- `symbol` +- `fiat_value` +- `currency` + +Po konwersji standardowe loadery i kalkulatory podatku robia reszte. + +## 2. Kontrakt pluginu (aktualny kod) + +W repo nie ma teraz jednej klasy `BrokerPlugin`. Kontrakt jest podzielony: + +1. Komenda CLI importu w [`pit38/cli.py`](../pit38/cli.py) +2. Parser/czytnik mapujacy wiersze brokera na obiekty domenowe +3. Saver zapisujacy kanoniczny CSV + +Dla pluginow stock formatowanie opiera sie o `BaseFormatter`: + +```python +class BaseFormatter(ABC): + @abstractmethod + def format(self, item: Any) -> Dict[str, Any]: + pass + + @abstractmethod + def item_type(self) -> Type: + pass +``` + +Pliki referencyjne: +- [`pit38/plugins/stock/formatters.py`](../pit38/plugins/stock/formatters.py) +- [`pit38/plugins/stock/generic_saver.py`](../pit38/plugins/stock/generic_saver.py) +- [`pit38/plugins/crypto/generic_saver.py`](../pit38/plugins/crypto/generic_saver.py) + +## 3. Krok po kroku: nowy broker (cookiecutter) + +### 3.1 Scaffold + +```bash +pip install cookiecutter +cookiecutter gh:pbialon/pit-38-broker-template +``` + +### 3.2 Sprawdz wygenerowane pliki + +Minimum: +- modul parsera +- modul csv reader/service +- punkt wejscia CLI +- testy parsera/service +- README pluginu + +### 3.3 Zdobadz realny eksport + +Uzyj prawdziwego eksportu brokera, ale zanonimizuj dane: +- usun id kont +- usun imiona/nazwiska +- zostaw oryginalny format liczb i dat + +### 3.4 Napisz parser + +Mapuj typy operacji brokera na domenowe: +- `BUY` +- `SELL` +- `DIVIDEND` +- `SERVICE_FEE` +- `STOCK_SPLIT` (tylko stock) + +Wynik pluginu ma byc tylko w kanonicznych kolumnach CSV. + +### 3.5 Podepnij komende do CLI + +Zarejestruj pod `pit38 import` w [`pit38/cli.py`](../pit38/cli.py): +- opcja input path +- opcja output path +- opcjonalnie `--log-level` +- krotkie podsumowanie po imporcie + +### 3.6 Uruchom testy + +```bash +pytest tests/ +pytest tests/ --cov=pit38 --cov-branch +``` + +### 3.7 Otworz PR + +- podlacz issue (`Closes #...`) +- dodaj wpis do changelog +- opisz impact podatkowy w template PR + +## 4. Typowe wzorce + +### Parsowanie dat + +- `pendulum.parse(...)` dla formatow ISO-like +- `datetime.strptime(...)` dla niestandardowych formatow + +Przyklad: Binance parser uzywa `%y-%m-%d %H:%M:%S` w +[`pit38/plugins/crypto/binance/csv.py`](../pit38/plugins/crypto/binance/csv.py). + +### Normalizacja walut i kwot + +Uzywaj wspolnych helperow zamiast nowego regexa per plugin: +- `normalize_currency_layout` +- `parse_amount` + +z [`pit38/plugins/normalization.py`](../pit38/plugins/normalization.py). + +### Brakujace kolumny i uszkodzone wiersze + +- fail fast przy brakujacych wymaganych kolumnach +- nieznane nietaxowe operacje pomijaj z czytelnym warningiem +- nie wywalaj calego importu przez jeden zly wiersz, jesli flow ma byc tolerancyjny + +### Znaki kwot i fee + +Jawnie trzymaj semantyke: +- service fee to koszt +- dividend to przychod +- nie odwroc znaku "po cichu" bez testu, ktory to dokumentuje + +## 5. Testowanie pluginu + +Helpery z [`tests/utils.py`](../tests/utils.py): +- `buy()`, `sell()`, `apple()`, `usd()`, `zl()` +- `StubExchanger` + +Warstwy testow: +- unit test parsera (row -> obiekt domenowy) +- test csv service (plik -> lista obiektow) +- e2e z fixture po anonimizacji w `tests/e2e/fixtures/` + +Minimalny szkic testu parsera: + +```python +from unittest import TestCase +from tests.utils import buy, apple, usd + +class TestMyBrokerParser(TestCase): + def test_buy_row(self): + row = {"date": "2025-01-02 10:00:00", "type": "BUY", "qty": "1", "ticker": "AAPL", "value": "USD 200.00"} + tx = parse_row(row) + self.assertEqual(tx, buy(apple(1), usd(200.0), "2025-01-02 10:00:00")) +``` + +## 6. Checklista poprawnosci podatkowej + +Przed PR sprawdz: + +- Traktowanie fee: + - fee jest juz w kwocie gross? + - fee idzie osobna operacja? +- Gross vs net: + - przychod jest net czy gross? + - czy fee nie jest policzone podwojnie? +- Potracenie podatku od dywidendy: + - broker daje gross, net czy oba? + - ktora kwota jest kanoniczna dla PIT-38? +- Waluta: + - oryginalna waluta jest zachowana przed konwersja NBP? +- Stock split: + - ratio zapisane bez sztucznego fiat value? + +Przy watpliwosciach wrzuc sample (zanonimizowany) do Discussions. + +## 7. Przyklady (od scaffold do gotowego pluginu) + +### 7.1 Revolut stock + +```diff +# scaffold (generated) +class RowParser: + def parse(self, row): + raise NotImplementedError + +# after implementation +class TransactionRowParser(RowParser): + OPERATIONS_HANDLED = {OperationType.BUY, OperationType.SELL} + + @classmethod + def parse(cls, row): + operation_type = cls._operation_type(row) + if operation_type not in cls.OPERATIONS_HANDLED: + return None + return Transaction( + asset=cls._asset(row), + fiat_value=cls._fiat_value(row), + action=operation_type, + date=cls._date(row), + ) +``` + +Pliki: +- [`pit38/plugins/stock/revolut/transaction_row_parser.py`](../pit38/plugins/stock/revolut/transaction_row_parser.py) +- [`pit38/plugins/stock/revolut/operation_row_parser.py`](../pit38/plugins/stock/revolut/operation_row_parser.py) + +### 7.2 Binance crypto + +```diff +# scaffold (generated) +def read(path): + return [] + +# after implementation +class BinanceTransactionProcessor: + def read(self, file_path: str) -> List[Transaction]: + convert_operations = [] + transaction_operations = [] + with open_csv_reader(file_path) as reader: + for row in reader: + tx = BinanceTransaction(row) + if tx.operation_type() == BinanceOperationType.DEPOSIT: + continue + if tx.operation_type() == BinanceOperationType.CONVERT: + convert_operations.append(tx) + else: + transaction_operations.append(tx) + all_transactions = ( + self._process_convert_transactions(convert_operations) + + self._process_transaction_operations(transaction_operations) + ) + return sorted(all_transactions, key=lambda x: x.date) +``` + +Plik: +- [`pit38/plugins/crypto/binance/csv.py`](../pit38/plugins/crypto/binance/csv.py) + +## 8. Przeplyw transakcji + +```mermaid +graph LR + A[Broker CSV] --> B[RowParser] + B --> C[Canonical CSV] + C --> D[GenericCsvLoader] + D --> E[Transaction objects] + E --> F[Tax Calculator] +``` + +## 9. Gdzie pytac + +- Discussions: +- Issues: +- Zasady podatkowe: + - [`docs/TAX_RULES.md`](TAX_RULES.md) + - [`docs/TAX_RULES.pl.md`](TAX_RULES.pl.md) diff --git a/pit38/plugins/README.md b/pit38/plugins/README.md index dee6bf2..dbab83e 100644 --- a/pit38/plugins/README.md +++ b/pit38/plugins/README.md @@ -1,67 +1,11 @@ # Plugins -## English +This file is now a short index. -Plugins are designed to transform transaction data from various brokers into a standardized format that can be processed by this repository. Each plugin handles a specific broker's export file format and converts it to a unified CSV structure. +Full guides: +- English: [`docs/DEVELOPING_PLUGINS.md`](../../docs/DEVELOPING_PLUGINS.md) +- Polish: [`docs/DEVELOPING_PLUGINS.pl.md`](../../docs/DEVELOPING_PLUGINS.pl.md) -### Purpose - -The main purpose of plugins is to: - -1. Read transaction data from broker-specific export files -2. Transform this data into a standardized format -3. Save the transformed data as a CSV file that can be loaded by the generic loader and processed further - -### Usage - -Each plugin can be run from the command line with specific parameters. Typically, plugins require: - -- Input path: Path to the broker's export file -- Output path: Where to save the standardized CSV file -- Optional parameters: Such as logging level - -### Available Plugins - -The repository includes plugins for various brokers. Check individual plugin directories for specific documentation. - -### Adding New Broker Support - -If your broker is not supported: - -1. You can create a new plugin following the existing patterns -2. Request support by opening a GitHub issue -3. Submit a pull request with your implementation - ---- - -## Polski - -Wtyczki służą do przekształcania danych transakcyjnych z różnych brokerów w ustandaryzowany format, który może być przetwarzany przez to repozytorium. Każda wtyczka obsługuje specyficzny format pliku eksportu danego brokera i konwertuje go do ujednoliconej struktury CSV. - -### Cel - -Głównym celem wtyczek jest: - -1. Odczytanie danych transakcyjnych z plików eksportu specyficznych dla brokera -2. Przekształcenie tych danych w ustandaryzowany format -3. Zapisanie przekształconych danych jako plik CSV, który może być wczytany przez generyczny loader i dalej procesowany - -### Użycie - -Każda wtyczka może być uruchomiona z linii poleceń z określonymi parametrami. Zazwyczaj wtyczki wymagają: - -- Ścieżki wejściowej: Ścieżka do pliku eksportu brokera -- Ścieżki wyjściowej: Gdzie zapisać ustandaryzowany plik CSV -- Opcjonalnych parametrów: Takich jak poziom logowania - -### Dostępne wtyczki - -Repozytorium zawiera wtyczki dla różnych brokerów. Sprawdź dokumentację w katalogach poszczególnych wtyczek, aby uzyskać szczegółowe informacje. - -### Dodawanie obsługi nowych brokerów - -Jeśli Twój broker nie jest obsługiwany: - -1. Możesz utworzyć nową wtyczkę, wzorując się na istniejących -2. Poprosić o wsparcie, otwierając zgłoszenie (issue) na GitHubie -3. Przesłać pull request ze swoją implementacją +Quick references: +- Supported broker notes: [`docs/BROKERS.md`](../../docs/BROKERS.md) +- Contribution workflow: [`CONTRIBUTING.md`](../../CONTRIBUTING.md)