Merge pull request #303 from Snowflake-Labs/dev

fix: incorporating forked changes along with github actions updates
Snowflake-Labs · Nov 22, 2024 · 99ff357 · 99ff357
2 parents 871d7c7 + 02e101e
commit 99ff357
Show file tree

Hide file tree

Showing 71 changed files with 1,950 additions and 1,543 deletions.
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
@@ -7,7 +7,8 @@ operating environment, schemachange version and python version. Whenever possibl
 also include a brief, self-contained code example that demonstrates the problem.
 
 We have
-included [issue templates](https://github.com/Snowflake-Labs/schemachange/issues/new/choose) for reporting bugs, requesting features and seeking clarifications. Choose the appropriate issue template to contribute to the repository.
+included [issue templates](https://github.com/Snowflake-Labs/schemachange/issues/new/choose) for reporting bugs,
+requesting features and seeking clarifications. Choose the appropriate issue template to contribute to the repository.
 
 ## Contributing code
 
@@ -22,8 +23,6 @@ Thank you for your interest in contributing code to schemachange!
 
 ### Guide to contributing to schemachange
 
-> **IMPORTANT** : You will need to follow the [provisioning and schemachange setup instructions](../demo/README.MD) to ensure you can run GitHub actions against your Snowflake account before placing a PR with main schemachange repository so that your PR can be merged into schemachange master branch.
-
 1. If you are a first-time contributor
     + Go to [Snowflake-Labs/Schemachange](https://github.com/Snowflake-Labs/schemachange) and click the "fork" button to
       create your own copy of the project.
@@ -53,8 +52,8 @@ Thank you for your interest in contributing code to schemachange!
     + [Pull](https://github.com/git-guides/git-pull) the latest changes from upstream, including tags:
 
     ```shell
-    git checkout main
-    git pull upstream main --tags
+    git checkout master
+    git pull upstream master --tags
     ```
 
 2. Create and Activate a Virtual Environment
@@ -98,24 +97,32 @@ Thank you for your interest in contributing code to schemachange!
 
     + Commit locally as you progress ( [git add](https://github.com/git-guides/git-add)
       and [git commit](https://github.com/git-guides/git-commit) ). Use a properly formatted commit message. Be sure to
-      document any changed behavior in the [CHANGELOG.md](../CHANGELOG.md) file to help us collate the changes for a specific release.
+      document any changed behavior in the [CHANGELOG.md](../CHANGELOG.md) file to help us collate the changes for a
+      specific release.
 
 4. Test your contribution locally
 
    ```bash
    python -m pytest
    ```
-   PS: Please add test cases to the features you are developing so that over time, we can capture any lapse in functionality changes.
+   PS: Please add test cases to the features you are developing so that over time, we can capture any lapse in
+   functionality changes.
+
+5. Perform integration tests on your branch from your fork
+    - Follow the [provisioning and schemachange setup instructions](../demo/README.MD) to configure your Snowflake
+      account for testing.
+    - Follow [these](https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-workflow-runs/manually-running-a-workflow)
+      instructions to manually run the `master-pytest` workflow on your fork of the repo, targeting your feature branch.
 
-5. Push your contribution to GitHub
+6. Push your contribution to GitHub
 
-    [Push](https://github.com/git-guides/git-push) your changes back to your fork on GitHub
+   [Push](https://github.com/git-guides/git-push) your changes back to your fork on GitHub
 
     ```shell
     git push origin update-build-library-dependencies
     ```
 
-6. Raise a Pull Request to merge your contribution into the a Schemachange Release
+7. Raise a Pull Request to merge your contribution into the a Schemachange Release
     + Go to GitHub. The new branch will show up with a
       green [Pull Request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests#initiating-the-pull-request)
       button. Make sure the title and message are clear, concise and self-explanatory. Then click the button to submit

diff --git a/.github/workflows/dev-pytest.yml b/.github/workflows/dev-pytest.yml
@@ -54,35 +54,22 @@ jobs:
           flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
           # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
           flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+      - name: Create and populate connections.toml
+        run: |
+          echo "Current Directory: ${PWD}"
+          echo "GITHUB WORKSPACE: ${GITHUB_WORKSPACE}"
+          chmod +x populateConnection.sh
+          bash populateConnection.sh
+        working-directory: .
       - name: Test with pytest
         id: pytest
         run: |
           pytest
       # Testing Schemachange demo projects
       - name: Test Schemachange on ${{ matrix.os }} targeting ${{ env.SNOWFLAKE_DATABASE }}.${{ env.MY_TARGET_SCHEMA }} schema
         run: |
-          echo "::group::Setting up ${MY_TARGET_SCHEMA}"
-          schemachange deploy --config-folder ./demo/setup/${SCENARIO_NAME}
-          echo "::endgroup::"
-          echo "::group::Testing Rendering to ${MY_TARGET_SCHEMA}"
-          schemachange render --config-folder ./demo/${SCENARIO_NAME} ./demo/${SCENARIO_NAME}/A__render.sql
-          schemachange render --config-folder ./demo/${SCENARIO_NAME} ./demo/${SCENARIO_NAME}/R__render.sql
-          schemachange render --config-folder ./demo/${SCENARIO_NAME} ./demo/${SCENARIO_NAME}/V1.0.0__render.sql
-          echo "::endgroup::"
-          echo "::group::Testing Deployment using ${MY_TARGET_SCHEMA}"
-          set +e
-          schemachange deploy --config-folder ./demo/${SCENARIO_NAME}
-          RESULT=$?
-          if [ $RESULT -eq 0 ]; then
-            echo "Deployment Completed!"
-          else
-            echo "Deployment Failed. Proceeding to Teardown."
-          fi
-          echo "::endgroup::"
-          set -e
-          echo "::group::Tearing down up ${MY_TARGET_SCHEMA}"
-          schemachange deploy --config-folder ./demo/teardown/${SCENARIO_NAME}
-          echo "::endgroup::"
-          if [ $RESULT -ne 0 ]; then
-            exit 1
-          fi
+          echo "Current Directory: ${PWD}"
+          echo "GITHUB WORKSPACE: ${GITHUB_WORKSPACE}"
+          chmod +x testSchemachange.sh
+          bash testSchemachange.sh
+        working-directory: .
diff --git a/.github/workflows/master-pytest.yml b/.github/workflows/master-pytest.yml
@@ -29,13 +29,13 @@ jobs:
     runs-on: ${{ matrix.os }}
     if: ${{ github.event.label.name == 'ci-run-tests' || github.event_name == 'push' || github.event_name == 'workflow_dispatch' }}
     env:
-      SNOWFLAKE_PASSWORD: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_PASSWORD }}
-      SNOWFLAKE_USER: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_USER }}
       SNOWFLAKE_ACCOUNT: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_ACCOUNT }}
-      SNOWFLAKE_DATABASE: SCHEMACHANGE_DEMO
-      SNOWFLAKE_WAREHOUSE: SCHEMACHANGE_DEMO_WH
+      SNOWFLAKE_USER: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_USER }}
       SNOWFLAKE_ROLE: SCHEMACHANGE_DEMO-DEPLOY
+      SNOWFLAKE_WAREHOUSE: SCHEMACHANGE_DEMO_WH
+      SNOWFLAKE_DATABASE: SCHEMACHANGE_DEMO
       MY_TARGET_SCHEMA: ${{ matrix.scenario-name }}_${{ github.run_number }}_${{ strategy.job-index }}
+      SNOWFLAKE_PASSWORD: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_PASSWORD }}
       SCENARIO_NAME: ${{ matrix.scenario-name }}
     steps:
       - uses: actions/checkout@v4
@@ -58,35 +58,22 @@ jobs:
           flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
           # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
           flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+      - name: Create and populate connections.toml
+        run: |
+          echo "Current Directory: ${PWD}"
+          echo "GITHUB WORKSPACE: ${GITHUB_WORKSPACE}"
+          chmod +x populateConnection.sh
+          bash populateConnection.sh
+        working-directory: .
       - name: Test with pytest
         id: pytest
         run: |
           pytest
       # Testing Schemachange demo projects
       - name: Test Schemachange on ${{ matrix.os }} targeting ${{ env.SNOWFLAKE_DATABASE }}.${{ env.MY_TARGET_SCHEMA }} schema
         run: |
-          echo "::group::Setting up ${MY_TARGET_SCHEMA}"
-          schemachange deploy --config-folder ./demo/setup/${SCENARIO_NAME}
-          echo "::endgroup::"
-          echo "::group::Testing Rendering to ${MY_TARGET_SCHEMA}"
-          schemachange render --config-folder ./demo/${SCENARIO_NAME} ./demo/${SCENARIO_NAME}/A__render.sql
-          schemachange render --config-folder ./demo/${SCENARIO_NAME} ./demo/${SCENARIO_NAME}/R__render.sql
-          schemachange render --config-folder ./demo/${SCENARIO_NAME} ./demo/${SCENARIO_NAME}/V1.0.0__render.sql
-          echo "::endgroup::"
-          echo "::group::Testing Deployment using ${MY_TARGET_SCHEMA}"
-          set +e
-          schemachange deploy --config-folder ./demo/${SCENARIO_NAME}
-          RESULT=$?
-          if [ $RESULT -eq 0 ]; then
-            echo "Deployment Completed!"
-          else
-            echo "Deployment Failed. Proceeding to Teardown."
-          fi
-          echo "::endgroup::"
-          set -e
-          echo "::group::Tearing down up ${MY_TARGET_SCHEMA}"
-          schemachange deploy --config-folder ./demo/teardown/${SCENARIO_NAME}
-          echo "::endgroup::"
-          if [ $RESULT -ne 0 ]; then
-            exit 1
-          fi
+          echo "Current Directory: ${PWD}"
+          echo "GITHUB WORKSPACE: ${GITHUB_WORKSPACE}"
+          chmod +x testSchemachange.sh
+          bash testSchemachange.sh
+        working-directory: .
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,12 +7,14 @@ All notable changes to this project will be documented in this file.
 ### Added
 - Use of `structlog~=24.1.0` for standard log outputs
 - Verified Schemachange against Python 3.12
+- Support for connections.toml configurations
+- Support for supplying the authenticator, private key path, token path, connections file path, and connection name via the YAML and command-line configurations.
 
 ### Changed
 - Refactored the main cli.py into multiple modules - config, session.
 - Updated contributing guidelines and demo readme content to help contributors setup local snowflake account to run the github actions in their fork before pushing the PR to upstream repository.
 - Removed tests against Python 3.8 [End of Life on 2024-10-07](https://devguide.python.org/versions/#supported-versions)
-
+- Command-line vars are now merged into YAML vars instead of overwriting them entirely
 
 ## [3.7.0] - 2024-07-22
 ### Added