Sourcery Starbot ⭐ refactored chip-felton-montage/sfguide-data-engineering-with-snowpark-python by SourceryAI · Pull Request #1 · chip-felton/sfguide-data-engineering-with-snowpark-python

SourceryAI · 2023-06-10T08:29:10Z

Thanks for starring sourcery-ai/sourcery ✨ 🌟 ✨

Here's your pull request refactoring your most popular Python repo.

If you want Sourcery to refactor all your Python repos and incoming pull requests install our bot.

Review changes via command line

To manually merge these changes, make sure you're on the main branch, then run:

git fetch https://github.com/sourcery-ai-bot/sfguide-data-engineering-with-snowpark-python main
git merge --ff-only FETCH_HEAD
git reset HEAD^

SourceryAI · 2023-06-10T08:29:12Z

-    if not "app.toml" in file_names:
+    if "app.toml" not in file_names:
 #        print(f"Skipping non-app folder {directory_path}")
        continue

    # Next determine what type of app it is
    app_type = "unknown"
-    if "local_connection.py" in file_names:
-        app_type = "procedure"
-    else:
-        app_type = "function"
-
+    app_type = "procedure" if "local_connection.py" in file_names else "function"


Lines 25-35 refactored with the following changes:

Simplify logical expression using De Morgan identities (de-morgan)

Replace if statement with if expression (assign-if-exp)

SourceryAI · 2023-06-10T08:29:12Z

-        location = "@external.frostbyte_raw_stage/{}/{}".format(s3dir, tname)
+        location = f"@external.frostbyte_raw_stage/{s3dir}/{tname}"
    else:
-        print('\tLoading year {}'.format(year)) 
-        location = "@external.frostbyte_raw_stage/{}/{}/year={}".format(s3dir, tname, year)
-    
+        print(f'\tLoading year {year}')
+        location = f"@external.frostbyte_raw_stage/{s3dir}/{tname}/year={year}"
+
    # we can infer schema using the parquet read option
    df = session.read.option("compression", "snappy") \
                            .parquet(location)
-    df.copy_into_table("{}".format(tname))
+    df.copy_into_table(f"{tname}")


Function load_raw_table refactored with the following changes:

Replace call to format with f-string [×4] (use-fstring-for-formatting)

SourceryAI · 2023-06-10T08:29:12Z

        schema = data['schema']
        for tname in tnames:
-            print("Loading {}".format(tname))
+            print(f"Loading {tname}")


Function load_all_raw_tables refactored with the following changes:

Replace call to format with f-string (use-fstring-for-formatting)

SourceryAI · 2023-06-10T08:29:13Z

-        print('{}: \n\t{}\n'.format(tname, session.table('RAW_POS.{}'.format(tname)).columns))
+        print(f"{tname}: \n\t{session.table(f'RAW_POS.{tname}').columns}\n")

    for tname in CUSTOMER_TABLES:
-        print('{}: \n\t{}\n'.format(tname, session.table('RAW_CUSTOMER.{}'.format(tname)).columns))
+        print(f"{tname}: \n\t{session.table(f'RAW_CUSTOMER.{tname}').columns}\n")


Function validate_raw_tables refactored with the following changes:

Replace call to format with f-string [×4] (use-fstring-for-formatting)

SourceryAI · 2023-06-10T08:29:13Z


 def main(temp_f: float) -> float:
-    return (float(temp_f) - 32) * (5/9)
+    return (temp_f - 32) * (5/9)


Function main refactored with the following changes:

Remove unnecessary casts to int, str, float or bool (remove-unnecessary-cast)

SourceryAI · 2023-06-10T08:29:14Z

-    exists = session.sql("SELECT EXISTS (SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = '{}' AND TABLE_NAME = '{}') AS TABLE_EXISTS".format(schema, name)).collect()[0]['TABLE_EXISTS']
-    return exists
+    return session.sql(
+        f"SELECT EXISTS (SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = '{schema}' AND TABLE_NAME = '{name}') AS TABLE_EXISTS"
+    ).collect()[0]['TABLE_EXISTS']


Function table_exists refactored with the following changes:

Inline variable that is immediately returned (inline-immediately-returned-variable)

Replace call to format with f-string (use-fstring-for-formatting)

SourceryAI · 2023-06-10T08:29:14Z

-    print("{} records in stream".format(session.table('HARMONIZED.ORDERS_STREAM').count()))
+    print(f"{session.table('HARMONIZED.ORDERS_STREAM').count()} records in stream")


Function merge_daily_city_metrics refactored with the following changes:

Replace call to format with f-string (use-fstring-for-formatting)

Applies the dictionary union operator where applicable (use-dictionary-union)

SourceryAI · 2023-06-10T08:29:14Z

-    
+
    merge_daily_city_metrics(session)
 #    session.table('ANALYTICS.DAILY_CITY_METRICS').limit(5).show()

-    return f"Successfully processed DAILY_CITY_METRICS"
+    return "Successfully processed DAILY_CITY_METRICS"


Function main refactored with the following changes:

Replace f-string with no interpolated values with string (remove-redundant-fstring)

SourceryAI · 2023-06-10T08:29:14Z

            for k, v in session_config.items()
        }
-        session_config_dict.update(app_config.get(environment))  # type: ignore
+        session_config_dict |= app_config.get(environment)


Function get_dev_config refactored with the following changes:

Merge dictionary updates via the union operator (dict-assign-update-to-union)

This removes the following comments ( why? ):

# type: ignore

SourceryAI · 2023-06-10T08:29:15Z

-        connection_path = 'connections.' + connection_name
+        connection_path = f'connections.{connection_name}'

        config.read(config_file_path)
        session_config = config[connection_path]
-        # Convert snowsql connection variable names to snowcli ones
-        session_config_dict = {
+        return {
            snowsql_to_snowpark_config_mapping[k]: v.strip('"')
            for k, v in session_config.items()
        }
-        return session_config_dict


Function get_snowsql_config refactored with the following changes:

Use f-string instead of string concatenation (use-fstring-for-concatenation)

Inline variable that is immediately returned (inline-immediately-returned-variable)

This removes the following comments ( why? ):

# Convert snowsql connection variable names to snowcli ones

'Refactored by Sourcery'

e0f1087

SourceryAI commented Jun 10, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sourcery Starbot ⭐ refactored chip-felton-montage/sfguide-data-engineering-with-snowpark-python#1

Sourcery Starbot ⭐ refactored chip-felton-montage/sfguide-data-engineering-with-snowpark-python#1
SourceryAI wants to merge 1 commit into
chip-felton:mainfrom
SourceryAI:main

SourceryAI commented Jun 10, 2023

Uh oh!

SourceryAI Jun 10, 2023

Uh oh!

SourceryAI Jun 10, 2023

Uh oh!

SourceryAI Jun 10, 2023

Uh oh!

SourceryAI Jun 10, 2023

Uh oh!

SourceryAI Jun 10, 2023

Uh oh!

SourceryAI Jun 10, 2023

Uh oh!

SourceryAI Jun 10, 2023

Uh oh!

SourceryAI Jun 10, 2023

Uh oh!

SourceryAI Jun 10, 2023

Uh oh!

SourceryAI Jun 10, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		print("{} records in stream".format(session.table('HARMONIZED.ORDERS_STREAM').count()))
		print(f"{session.table('HARMONIZED.ORDERS_STREAM').count()} records in stream")

Conversation

SourceryAI commented Jun 10, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant