Skip to content

AMOS team contributions: forecasting, anomaly detection, decomposition, visualization #950

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
simonselbig wants to merge 22 commits into
base: develop
Choose a base branch
from
Open

AMOS team contributions: forecasting, anomaly detection, decomposition, visualization #950

simonselbig wants to merge 22 commits into from

Conversation

@simonselbig
Copy link

@simonselbig simonselbig commented Jan 25, 2026

Adds the contributions from the AMOS WS2025 team (amos2025ws03-rtdip-timeseries-forecasting) to the RTDIP SDK:

  • Data Manipulation (pandas & spark): Chronological sort, cyclical encoding, datetime features, lag features, MAD outlier detection, rolling statistics, and more
  • Forecasting Models (spark): Prophet, LSTM, XGBoost, CatBoost, AutoGluon time series models + prediction evaluation utilities
  • Decomposition (pandas & spark): Classical, STL, and MSTL decomposition methods
  • Anomaly Detection (spark): IQR-based and MAD-based anomaly detection
  • Visualization (matplotlib & plotly): Forecasting, decomposition, anomaly detection, and model comparison plots
  • Data Sources: Azure Blob storage source

All components include tests and documentation.

Environment Changes

Added ML/forecasting dependencies: tensorflow, xgboost, plotly, prophet, sktime, catboost, autogluon.timeseries

@simonselbig
Commit all AMOS changes to upstream repo for PR
35eccce 
Signed-off-by: simonselbig <simon.selbig@gmx.de>
@simonselbig
remove duplicate anomaly detection file
18dd377 
Signed-off-by: simonselbig <simon.selbig@gmx.de>
@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Jan 25, 2026 •
edited
Loading

CLA Signed

The committers listed above are authorized under a signed CLA.

@Amber-Rigg Amber-Rigg added enhancement New feature or request pipelines Pipeline components and ingestion framework labels Jan 26, 2026
@simonselbig
last changes to mkdocs & environment
8c6a601 
Signed-off-by: simonselbig <simon.selbig@gmx.de>
@zenzeii zenzeii mentioned this pull request Jan 27, 2026
Open
@Amber-Rigg Amber-Rigg added this to RTDIP Feb 2, 2026
@Amber-Rigg
Refactor logging in forecasting and data quality modules to use loggi…
2549681 
…ng instead of print statements

Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
@Amber-Rigg
Update copyright year from 2025 to 2026 in all relevant files
7049d4c 
Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
@Amber-Rigg
Refactor tests to use NumPy's default_rng for random number generation
37e2cee 
- Updated test files in the decomposition, forecasting, and visualization modules to replace np.random.seed with np.random.default_rng for improved randomness control.
- Ensured consistent random number generation across multiple test cases by initializing the random generator with a fixed seed.
- Adjusted assertions to use np.isclose for floating-point comparisons to enhance numerical stability in tests.
- Removed deprecated or commented-out code related to Prophet tests due to compatibility issues with Polars.

Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
@Amber-Rigg
Black Formatting
91ccdd1 
Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
@Amber-Rigg
Refactor imports and class inheritance in IQR anomaly detection; upda…
710ee7d 
…te evaluate method to return None for invalid samples in CatBoost; enhance logging in LSTM predictions; modify test data generation for KNN and LSTM tests.

Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
@Amber-Rigg
Update spark sessions for delta-spark
f7d8fe1 
Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
@Amber-Rigg
Refactor Spark session creation in tests; update import paths for LST…
43246d1 
…MTimeSeries and remove redundant tests

Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
@Amber-Rigg
Update tests for error TypeError: Casting to unit-less dtype 'datetim…
94d6f3a 
…e64' is not supported. Pass e.g. 'datetime64[ns]' instead.

Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
@Amber-Rigg
Add _prepare_pandas_to_convert_to_spark utility in anomaly detection …
85902f9 
…and decomposition classes for improved DataFrame compatibility

Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
@Amber-Rigg
Enable Arrow optimization in Spark testing configuration for improved…
839a4d3 
… performance

Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
@Amber-Rigg
Enhance SelectColumnsByCorrelation for better datetime handling and a…
ab8fe28 
…dd fixture for pandas compatibility with PySpark

Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
@Amber-Rigg
Add normalize_datetime_precision function to ensure consistent dateti…
a32d4e4 
…me column handling in DataFrames

Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
@Amber-Rigg
Refactor code for improved readability and consistency
0ec7ab7 
- Removed unnecessary line breaks and adjusted formatting in multiple files to enhance code clarity.
- Simplified tuple unpacking in function calls across various modules.
- Cleaned up imports by removing unused blank lines.
- Standardized the formatting of dictionary assignments for better readability.

Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
@Amber-Rigg
Update to free disc space
eb5ad65 
Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
@Amber-Rigg
Update for micromamba
48835b6 
Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
@Amber-Rigg
Add post-cleanup option to workflows for improved environment management
fd1e67d 
Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
@Amber-Rigg
Add cache-environment-key to workflows for improved caching efficiency
d57c7b9 
Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
@Amber-Rigg
fix: add setuptools install step and fix micromamba config for CI
bdfd88b
@Amber-Rigg
Add step to ensure setuptools is installed in workflows
24eaf61 
Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Amber-Rigg Amber-Rigg Awaiting requested review from Amber-Rigg

@cching95 cching95 Awaiting requested review from cching95

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

enhancement New feature or request pipelines Pipeline components and ingestion framework

Projects

RTDIP
Status: No status

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@simonselbig @Amber-Rigg