Add Python 3.11 to GHA by gliptak · Pull Request #1090 · capitalone/DataProfiler

* add downloads tile (capitalone#1085) * Replace snappy with cramjam * Delete test_no_snappy --------- Co-authored-by: Taylor Turner <taylorfturner@gmail.com>

taylorfturner · 2024-03-14T13:14:53Z

@gliptak #1091 merged ... rebase this and we'll take a look. Thanks for the contribution! 🎉

gliptak · 2024-03-14T14:48:32Z

dask packaging changed? https://pypi.org/project/dask/#history

https://github.com/capitalone/DataProfiler/actions/runs/8282560184/job/22663633408?pr=1090

____ ERROR collecting dataprofiler/tests/validators/test_base_validators.py ____
/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/dask/dataframe/__init__.py:22: in _dask_expr_enabled
    import dask_expr  # noqa: F401
E   ModuleNotFoundError: No module named 'dask_expr'

During handling of the above exception, another exception occurred:
dataprofiler/tests/validators/test_base_validators.py:4: in <module>
    from dask import dataframe as dd
/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/dask/dataframe/__init__.py:87: in <module>
    if _dask_expr_enabled():
/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/dask/dataframe/__init__.py:24: in _dask_expr_enabled
    raise ValueError("Must install dask-expr to activate query planning.")
E   ValueError: Must install dask-expr to activate query planning.
=============================== warnings summary ===============================
dataprofiler/tests/profilers/test_histogram_utils.py:35
  /home/runner/work/DataProfiler/DataProfiler/dataprofiler/tests/profilers/test_histogram_utils.py:35: PytestCollectionWarning: cannot collect test class 'TestColumn' because it has a __init__ constructor (from: dataprofiler/tests/profilers/test_histogram_utils.py)
    class TestColumn(NumericStatsMixin):

dataprofiler/tests/profilers/test_numeric_stats_mixin_profile.py:[21](https://github.com/capitalone/DataProfiler/actions/runs/8282560184/job/22663633408?pr=1090#step:6:22)
  /home/runner/work/DataProfiler/DataProfiler/dataprofiler/tests/profilers/test_numeric_stats_mixin_profile.py:21: PytestCollectionWarning: cannot collect test class 'TestColumn' because it has a __init__ constructor (from: dataprofiler/tests/profilers/test_numeric_stats_mixin_profile.py)
    class TestColumn(NumericStatsMixin):

dataprofiler/tests/profilers/test_numeric_stats_mixin_profile.py:[34](https://github.com/capitalone/DataProfiler/actions/runs/8282560184/job/22663633408?pr=1090#step:6:35)
  /home/runner/work/DataProfiler/DataProfiler/dataprofiler/tests/profilers/test_numeric_stats_mixin_profile.py:34: PytestCollectionWarning: cannot collect test class 'TestColumnWProps' because it has a __init__ constructor (from: dataprofiler/tests/profilers/test_numeric_stats_mixin_profile.py)
    class TestColumnWProps(TestColumn):

taylorfturner · 2024-03-14T14:58:47Z

@gliptak yeah I just started seeing this yesterday due to the package change by dask on the 12th. Haven't had the bandwidth to research why. I'd imagine a simple tag to not allow for this version would be a temporary fix to unblock

taylorfturner · 2024-03-14T15:02:57Z

seeing on #1115 too from @carlsonp

gliptak · 2024-03-14T15:13:26Z

https://github.com/capitalone/DataProfiler/actions/runs/8282934358/job/22664948501

2024-03-14T15:03:11.8884631Z WARNING: dask 2024.3.0 does not provide the extra 'dask-expr'

dask/dask-expr#968
dask/dask#10917
https://docs.dask.org/en/stable/changelog.html#v2024-3-0

gliptak · 2024-03-14T15:36:41Z

@taylorfturner corrected dask modules install

now there is a Keras(?) error https://github.com/capitalone/DataProfiler/actions/runs/8283138450/job/22665615645?pr=1090

gliptak · 2024-06-09T19:13:36Z

https://github.com/capitalone/DataProfiler/actions/runs/9438503984/job/25995558872?pr=1090

___________________ TestEvaluateAccuracy.test_save_conf_mat ____________________
self = <dataprofiler.tests.labelers.test_labeler_utils.TestEvaluateAccuracy testMethod=test_save_conf_mat>
mock_dataframe = <MagicMock name='DataFrame' id='140514062390544'>
mock_report = <MagicMock name='classification_report' id='[140](https://github.com/capitalone/DataProfiler/actions/runs/9438503984/job/25995558872?pr=1090#step:6:141)514060067088'>
    @mock.patch("dataprofiler.labelers.labeler_utils.classification_report")
    @mock.patch("pandas.DataFrame")
    def test_save_conf_mat(self, mock_dataframe, mock_report):
    
        # ideally mock out the actual contents written to file, but
        # would be difficult to get this completely worked out.
        expected_conf_mat = np.array(
            [
                [1, 0, 1],
                [1, 0, 0],
                [0, 1, 2],
            ]
        )
        expected_row_col_names = dict(
            columns=["pred:PAD", "pred:UNKNOWN", "pred:OTHER"],
            index=["true:PAD", "true:UNKNOWN", "true:OTHER"],
        )
>       mock_instance_df = mock.Mock(spec=pd.DataFrame)()
dataprofiler/tests/labelers/test_labeler_utils.py:255: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/unittest/mock.py:1106: in __init__
    _safe_super(CallableMixin, self).__init__(
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/unittest/mock.py:457: in __init__
    self._mock_add_spec(spec, spec_set, _spec_as_instance, _eat_self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <[AttributeError('_mock_methods') raised in repr()] Mock object at 0x7fcc6aa22410>
spec = <MagicMock name='DataFrame' id='140514062390544'>, spec_set = None
_spec_as_instance = False, _eat_self = False
    def _mock_add_spec(self, spec, spec_set, _spec_as_instance=False,
                       _eat_self=False):
        if _is_instance_mock(spec):
>           raise InvalidSpecError(f'Cannot spec a Mock object. [object={spec!r}]')
E           unittest.mock.InvalidSpecError: Cannot spec a Mock object. [object=<MagicMock name='DataFrame' id='140514062390544'>]
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/unittest/mock.py:508: InvalidSpecError

taylorfturner · 2024-06-10T14:33:34Z

.pre-commit-config.yaml

            chardet>=3.0.4,
            fastavro>=1.0.0.post1,
-            python-snappy>=0.5.4,
+            cramjam>=2.7.0,


I think this should be reverted 🤔

should match requirements.txt

python-snappy>=0.7.1 supports Python 3.11 by switching over to cramjam and removing libsnappy-dev dependency (I contributed intake/python-snappy#130)

I will work dependencies in a minute

we reverted these cramjam changes due to failing tests locally though in #1122.

do we have to include cramjam + 3.11 in same pr?

I'm game for the 3.11 add

is the test suite running locally for you using DATAPROFILER_SEED=0 python3 -m unittest discover -p "test*.py" and tox -e py311?

dataprofiler/__init__.py

gliptak · 2024-06-10T16:42:59Z

this is the above outstanding test fail python/cpython#87644

DataProfiler/dataprofiler/tests/labelers/test_labeler_utils.py

Line 255 in a448694

mock_instance_df = mock.Mock(spec=pd.DataFrame)()

taylorfturner · 2024-06-10T20:15:15Z

this is the above outstanding test fail python/cpython#87644

DataProfiler/dataprofiler/tests/labelers/test_labeler_utils.py

Line 255 in a448694

mock_instance_df = mock.Mock(spec=pd.DataFrame)()

I see -- you are welcome to propose a fix for this as part of this PR (instead of a separate PR). If you get something operational, we can include this in the 0.12.0 release; otherwise, I will need to deploy without. Thanks, @gliptak!

gliptak · 2024-06-10T21:08:06Z

@taylorfturner I rewrote the test and it ran green locally. please review

also let me know if separate bump PRs would work better (and guide on how you would like to split)

taylorfturner · 2024-06-11T12:36:39Z

.pre-commit-config.yaml

            chardet>=3.0.4,
            fastavro>=1.0.0.post1,
-            python-snappy>=0.5.4,
+            cramjam>=2.7.0,


we reverted these cramjam changes due to failing tests locally though in #1122.

do we have to include cramjam + 3.11 in same pr?

I'm game for the 3.11 add

is the test suite running locally for you using DATAPROFILER_SEED=0 python3 -m unittest discover -p "test*.py" and tox -e py311?

taylorfturner · 2024-06-11T12:40:18Z

.pre-commit-config.yaml

            pyarrow>=1.0.1,
            chardet>=3.0.4,
            fastavro>=1.0.0.post1,
-            python-snappy>=0.5.4,


why remove here? it is kept in requirements.txt file

synced with requirements

taylorfturner · 2024-06-11T12:42:05Z

@taylorfturner I rewrote the test and it ran green locally. please review

also let me know if separate bump PRs would work better (and guide on how you would like to split)

Thanks, @gliptak !

keep this PR but only do 3.11 add and the test update
remove the cramjam / snappy(?)

gliptak · 2024-06-11T13:43:33Z

python-snappy>=0.7.1 bump is required for Python 3.11

taylorfturner · 2024-06-11T14:50:09Z

.pre-commit-config.yaml

            pyarrow>=1.0.1,
            chardet>=3.0.4,
            fastavro>=1.0.0.post1,
-            python-snappy>=0.5.4,


dataprofiler/__init__.py

dataprofiler/tests/test_data_profiler.py

* refactor: Upgrade the models to use keras 3.0 (#1138) * Replace snappy with cramjam (#1091) * add downloads tile (#1085) * Replace snappy with cramjam * Delete test_no_snappy --------- Co-authored-by: Taylor Turner <taylorfturner@gmail.com> * pre-commit fix (#1122) * Bug fix for float precision calculation using categorical data with trailing zeros. (#1125) * Revert "Bug fix for float precision calculation using categorical data with t…" (#1133) This reverts commit d3159bd. * refactor: move layers outside of class * refactor: update model to keras 3.0 * fix: manifest * fix: bugs in compile and train * fix: bug in load_from_library * fix: bugs in CharCNN * refactor: loading tf model labeler * fix: bug in data_labeler identification * fix: update model to use proper softmax layer names * fix: formatting * fix: remove unused line * refactor: drop support for 3.8 * fix: comments * fix: comment --------- Co-authored-by: Gábor Lipták <gliptak@gmail.com> Co-authored-by: Taylor Turner <taylorfturner@gmail.com> Co-authored-by: James Schadt <jamesrschadt@gmail.com> * Fix Tox (#1143) * tox new * update * update * update * update * update * update * update * update tox.ini * update * update * remove docs * empty retrigger * update (#1146) * Add Python 3.11 to GHA (#1090) * add downloads tile (#1085) * Add Python 3.11 to GHA * Replace snappy with cramjam (#1091) * add downloads tile (#1085) * Replace snappy with cramjam * Delete test_no_snappy --------- Co-authored-by: Taylor Turner <taylorfturner@gmail.com> * Update dask modules * Install dask dataframe * Update dask modules in precommit * Correct copy/paste error * Try again to clear Unicode * Rolled back pre-commit dask version * Add py311 to tox * Bump dask to 2024.4.1 * Bump python-snappy 0.7.1 * Rewrite labeler test * Correct isort * Satisfy black * And flake8 * Synced with requirements --------- Co-authored-by: Taylor Turner <taylorfturner@gmail.com> * [Vuln Fix]: Resolve mend vulnerabilities related to requests. (#1162) * resolved check-manifest issue * updating keras version pin to <=3.4.0 * adding comment in requirements.txt to trigger mend check --------- Co-authored-by: Armaan <armaan.dhillon@capitalone.com> --------- Co-authored-by: JGSweets <JGSweets@users.noreply.github.com> Co-authored-by: Gábor Lipták <gliptak@gmail.com> Co-authored-by: Taylor Turner <taylorfturner@gmail.com> Co-authored-by: James Schadt <jamesrschadt@gmail.com> Co-authored-by: Michael Davis <36012613+micdavis@users.noreply.github.com>

* refactor: Upgrade the models to use keras 3.0 (#1138) * Replace snappy with cramjam (#1091) * add downloads tile (#1085) * Replace snappy with cramjam * Delete test_no_snappy --------- * pre-commit fix (#1122) * Bug fix for float precision calculation using categorical data with trailing zeros. (#1125) * Revert "Bug fix for float precision calculation using categorical data with t…" (#1133) This reverts commit d3159bd. * refactor: move layers outside of class * refactor: update model to keras 3.0 * fix: manifest * fix: bugs in compile and train * fix: bug in load_from_library * fix: bugs in CharCNN * refactor: loading tf model labeler * fix: bug in data_labeler identification * fix: update model to use proper softmax layer names * fix: formatting * fix: remove unused line * refactor: drop support for 3.8 * fix: comments * fix: comment --------- * Fix Tox (#1143) * tox new * update * update * update * update * update * update * update * update tox.ini * update * update * remove docs * empty retrigger * update (#1146) * Add Python 3.11 to GHA (#1090) * add downloads tile (#1085) * Add Python 3.11 to GHA * Replace snappy with cramjam (#1091) * add downloads tile (#1085) * Replace snappy with cramjam * Delete test_no_snappy --------- * Update dask modules * Install dask dataframe * Update dask modules in precommit * Correct copy/paste error * Try again to clear Unicode * Rolled back pre-commit dask version * Add py311 to tox * Bump dask to 2024.4.1 * Bump python-snappy 0.7.1 * Rewrite labeler test * Correct isort * Satisfy black * And flake8 * Synced with requirements --------- * [Vuln Fix]: Resolve mend vulnerabilities related to requests. (#1162) * resolved check-manifest issue * updating keras version pin to <=3.4.0 * adding comment in requirements.txt to trigger mend check --------- --------- Co-authored-by: JGSweets <JGSweets@users.noreply.github.com> Co-authored-by: Gábor Lipták <gliptak@gmail.com> Co-authored-by: Taylor Turner <taylorfturner@gmail.com> Co-authored-by: James Schadt <jamesrschadt@gmail.com> Co-authored-by: Michael Davis <36012613+micdavis@users.noreply.github.com>

…ure (#1168) * Fix Codeowners List (#1043) * fix codeowners * fix codeowners * dash * lower case * Add null ratio to column stats (#1052) * staging/main/0.12.0 (#1145) * refactor: Upgrade the models to use keras 3.0 (#1138) * Replace snappy with cramjam (#1091) * add downloads tile (#1085) * Replace snappy with cramjam * Delete test_no_snappy --------- Co-authored-by: Taylor Turner <taylorfturner@gmail.com> * pre-commit fix (#1122) * Bug fix for float precision calculation using categorical data with trailing zeros. (#1125) * Revert "Bug fix for float precision calculation using categorical data with t…" (#1133) This reverts commit d3159bd. * refactor: move layers outside of class * refactor: update model to keras 3.0 * fix: manifest * fix: bugs in compile and train * fix: bug in load_from_library * fix: bugs in CharCNN * refactor: loading tf model labeler * fix: bug in data_labeler identification * fix: update model to use proper softmax layer names * fix: formatting * fix: remove unused line * refactor: drop support for 3.8 * fix: comments * fix: comment --------- Co-authored-by: Gábor Lipták <gliptak@gmail.com> Co-authored-by: Taylor Turner <taylorfturner@gmail.com> Co-authored-by: James Schadt <jamesrschadt@gmail.com> * Fix Tox (#1143) * tox new * update * update * update * update * update * update * update * update tox.ini * update * update * remove docs * empty retrigger * update (#1146) * bump version * update 3.11 * remove dist/ --------- Co-authored-by: JGSweets <JGSweets@users.noreply.github.com> Co-authored-by: Gábor Lipták <gliptak@gmail.com> Co-authored-by: James Schadt <jamesrschadt@gmail.com> * Staging release 0.13.0 (#1165) (#1166) * refactor: Upgrade the models to use keras 3.0 (#1138) * Replace snappy with cramjam (#1091) * add downloads tile (#1085) * Replace snappy with cramjam * Delete test_no_snappy --------- * pre-commit fix (#1122) * Bug fix for float precision calculation using categorical data with trailing zeros. (#1125) * Revert "Bug fix for float precision calculation using categorical data with t…" (#1133) This reverts commit d3159bd. * refactor: move layers outside of class * refactor: update model to keras 3.0 * fix: manifest * fix: bugs in compile and train * fix: bug in load_from_library * fix: bugs in CharCNN * refactor: loading tf model labeler * fix: bug in data_labeler identification * fix: update model to use proper softmax layer names * fix: formatting * fix: remove unused line * refactor: drop support for 3.8 * fix: comments * fix: comment --------- * Fix Tox (#1143) * tox new * update * update * update * update * update * update * update * update tox.ini * update * update * remove docs * empty retrigger * update (#1146) * Add Python 3.11 to GHA (#1090) * add downloads tile (#1085) * Add Python 3.11 to GHA * Replace snappy with cramjam (#1091) * add downloads tile (#1085) * Replace snappy with cramjam * Delete test_no_snappy --------- * Update dask modules * Install dask dataframe * Update dask modules in precommit * Correct copy/paste error * Try again to clear Unicode * Rolled back pre-commit dask version * Add py311 to tox * Bump dask to 2024.4.1 * Bump python-snappy 0.7.1 * Rewrite labeler test * Correct isort * Satisfy black * And flake8 * Synced with requirements --------- * [Vuln Fix]: Resolve mend vulnerabilities related to requests. (#1162) * resolved check-manifest issue * updating keras version pin to <=3.4.0 * adding comment in requirements.txt to trigger mend check --------- --------- Co-authored-by: JGSweets <JGSweets@users.noreply.github.com> Co-authored-by: Gábor Lipták <gliptak@gmail.com> Co-authored-by: Taylor Turner <taylorfturner@gmail.com> Co-authored-by: James Schadt <jamesrschadt@gmail.com> Co-authored-by: Michael Davis <36012613+micdavis@users.noreply.github.com> * updated version.py * updated the order of install requirements * revert back gitignore to the initial config * refactored Makefile making sure setting up venv with right python version * Fixed pre-commit hook errors and finalize configs:mypy, blakc, flake8, check-manifest, autoflake --------- Co-authored-by: Taylor Turner <taylorfturner@gmail.com> Co-authored-by: Suprabhat Gurrala <supragurrala@gmail.com> Co-authored-by: JGSweets <JGSweets@users.noreply.github.com> Co-authored-by: Gábor Lipták <gliptak@gmail.com> Co-authored-by: James Schadt <jamesrschadt@gmail.com> Co-authored-by: armaan-dhillon <dhillon.armaan11@gmail.com> Co-authored-by: Michael Davis <36012613+micdavis@users.noreply.github.com>

taylorfturner and others added 2 commits January 16, 2024 16:19

add downloads tile (capitalone#1085)

3d9b5ff

Add Python 3.11 to GHA

516c6f5

gliptak requested a review from a team as a code owner February 2, 2024 18:11

taylorfturner changed the base branch from main to dev February 2, 2024 19:15

Merge branch 'dev' into python1

276861c

taylorfturner assigned gliptak Feb 2, 2024

taylorfturner requested review from ksneab7, micdavis and taylorfturner February 2, 2024 19:19

taylorfturner changed the base branch from old-old-dev to dev March 7, 2024 12:29

Replace snappy with cramjam (capitalone#1091)

f411110

* add downloads tile (capitalone#1085) * Replace snappy with cramjam * Delete test_no_snappy --------- Co-authored-by: Taylor Turner <taylorfturner@gmail.com>

Merge branch 'dev' into python1

5b234fb

Update dask modules

f814ab7

Install dask dataframe

eb9d89e

gliptak added 3 commits March 14, 2024 11:39

Update dask modules in precommit

d23c485

Correct copy/paste error

63acf45

Try again to clear Unicode

bd1874e

Bump dask to 2024.4.1

c507ade

gliptak mentioned this pull request Jun 9, 2024

Dask Max Version Tag #1121

Open

taylorfturner reviewed Jun 10, 2024

View reviewed changes

Bump python-snappy 0.7.1

56b8b2c

taylorfturner enabled auto-merge (squash) June 10, 2024 14:59

Rewrite labeler test

e118ee8

auto-merge was automatically disabled June 10, 2024 20:51
Head branch was pushed to by a user without write access

Correct isort

9021a7e

gliptak added 2 commits June 10, 2024 17:09

Satisfy black

c594034

And flake8

94236da

taylorfturner reviewed Jun 11, 2024

View reviewed changes

taylorfturner enabled auto-merge (squash) June 11, 2024 12:40

Synced with requirements

6ca2c87

auto-merge was automatically disabled June 11, 2024 13:39
Head branch was pushed to by a user without write access

taylorfturner suggested changes Jun 11, 2024

View reviewed changes

taylorfturner enabled auto-merge (squash) June 11, 2024 17:54

taylorfturner approved these changes Jun 11, 2024

View reviewed changes

micdavis approved these changes Jun 12, 2024

View reviewed changes

taylorfturner merged commit 4e4450a into capitalone:dev Jun 12, 2024

gliptak mentioned this pull request Jun 12, 2024

Add Python 3.11 to GHA capitalone/synthetic-data#346

Open

armaan-dhillon mentioned this pull request Jan 13, 2025

Release 0.13.0 #1166

Merged

Conversation

gliptak commented Feb 2, 2024

Uh oh!

taylorfturner commented Feb 2, 2024

Uh oh!

gliptak commented Feb 2, 2024

Uh oh!

gliptak commented Feb 2, 2024

Uh oh!

gliptak commented Feb 2, 2024

Uh oh!

gliptak commented Feb 26, 2024

Uh oh!

taylorfturner commented Mar 7, 2024

Uh oh!

gliptak commented Mar 7, 2024

Uh oh!

taylorfturner commented Mar 14, 2024

Uh oh!

gliptak commented Mar 14, 2024

Uh oh!

taylorfturner commented Mar 14, 2024

Uh oh!

taylorfturner commented Mar 14, 2024

Uh oh!

gliptak commented Mar 14, 2024

Uh oh!

gliptak commented Mar 14, 2024

Uh oh!

gliptak commented Jun 9, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gliptak commented Jun 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

taylorfturner commented Jun 10, 2024

Uh oh!

gliptak commented Jun 10, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

taylorfturner commented Jun 11, 2024

Uh oh!

gliptak commented Jun 11, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gliptak commented Jun 10, 2024 •

edited

Loading