Integrate Snowflake Migration (via snowflake connector) #43
Draft
Integrate Snowflake Migration (via snowflake connector) #43
Conversation
* add snowflake-connector-proxy code * typo fix * remove unused comment lines * lint fix * remove snowflake-proxy tests * add dag code for snowflake proxy integeration * add sleep log * Add base doc for snowflake migration * fix doc
…n endpoint (#28) * add snowflake-connector-proxy code * typo fix * remove unused comment lines * lint fix * remove snowflake-proxy tests * add dag code for snowflake proxy integeration * add sleep log * Add base doc for snowflake migration * fix doc * Add authorization header in cloud run endpoint calls * env var rename * doc: add pre-req step * readme merge conflict fix * readme merge conflict fix
* Adding translation and validation for snowflake * Making linting changes * Making linting changes * lint fix * import order fix --------- Co-authored-by: Sumit Banerjee <sumitba@google.com>
* add ./extract-ddl route to snowflake connector proxy * move snowflake connector proxy code to translation dir * move sf_connector_utils to common * move sf_connector_utils to common: dag changes * lint fix * Handling for non-json response * import order fix * isort for black profile * comment fix
* Adding translation and validation for snowflake * Making linting changes * Making linting changes * lint fix * import order fix * Adding ddl extraction,translation and validation for snowflake * Adding ddl extraction,translation and validation for snowflake * Resolving syntax errors * Making lint changes * Fixing linting changes ---------
* Adding translation and validation for snowflake * Making linting changes * Making linting changes * lint fix * import order fix * Adding ddl extraction,translation and validation for snowflake * Adding ddl extraction,translation and validation for snowflake * Resolving syntax errors * Making lint changes * Fixing linting changes * Adding support for using secret manager for password ---------
…#38) * Adding translation and validation for snowflake * Making linting changes * Making linting changes * lint fix * import order fix * Adding ddl extraction,translation and validation for snowflake * Adding ddl extraction,translation and validation for snowflake * Resolving syntax errors * Making lint changes * Fixing linting changes * Adding support for using secret manager for password * Resolving issues due to translated files * Making changes for validation error due to lineage folder * Correcting linting changes * Correcting linting changes --------- Co-authored-by: Sumit Banerjee <sumitba@google.com>
pcigoor
reviewed
May 16, 2024
| ### Tool Features | ||
|
|
||
| * Initial release will support bulk load data transfer only. | ||
| * Initial release will support Snowflake to BigQuery translation using BQ DTS, row and column validation of the data in source and target. |
Collaborator
There was a problem hiding this comment.
says "Snowflake to BigQuery translation using BQ DTS", this is incorrect. Should be "BQ Translation service"
pcigoor
reviewed
May 16, 2024
| * Snowflake Account URL | ||
| * Snowflake JDBC URL | ||
| * Snowflake Warehouse name with access to required snowflaketables/schemas to migrate | ||
| * Snowflake Storage Integeration with access to staging gcs buckets you wish to use for migration ([Steps](https://docs.snowflake.com/en/user-guide/data-load-gcs-config#configuring-a-snowflake-storage-integration) - Check Step 1, 2, 3) |
Collaborator
There was a problem hiding this comment.
"Snowflake Storage Integeration" - the word "integration" has a typo.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add snowflake as a new supported data source for migrations
Note: Migration DAGs in this PR are using snowflake connector via snowflake connector proxy which is a temporary stopgap solution until snowflake connector is released with run-time setup feature