Skip to content

Commit

Permalink
Add documentation for Snowflake Streaming
Browse files Browse the repository at this point in the history
  • Loading branch information
sali-a committed Apr 16, 2024
1 parent 2673360 commit 18f8c1a
Show file tree
Hide file tree
Showing 3 changed files with 32 additions and 8 deletions.
9 changes: 1 addition & 8 deletions docs/content/_data/config/pipes/snowflake/common.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,5 @@ fields:
description: Snowflake options to specify such as key = value pairs. These options are then passed as option to the Spark connector for Snowflake
example: Options { Database = "example_database"
Schema = "example_schema"
Table = "example_table"
Mode = "append"
User = "username"
Url = "account_address"
pem_private_key = "private_key"
sfWarehouse = "warehouse"
sfRole = "role"
column_mapping = "name"
...
}
21 changes: 21 additions & 0 deletions docs/content/_data/config/pipes/snowflake/streaming.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
output:
type: com.amadeus.dataio.pipes.snowflake.streaming.SnowflakeOutput
fields:
- name: Duration
mandatory: "No"
description: Sets the trigger for the stream query. Controls the trigger() Spark function.
example: Duration = "60 seconds"
- name: Timeout
mandatory: "Yes"
description: Controls the amount of time before returning from the streaming query, in hours. It can be a String or an Int.
example: Timeout = 24
- name: Mode
mandatory: "Yes"
description: The Spark Structured Streaming output mode.
example: Mode = "complete"
default: append
- name: AddTimestampOnInsert
mandatory: "No"
description: Add a column named "timestamp" containing the current timestamp at the start of query evaluation
example: AddTimestampOnInsert = "true"
default: false
10 changes: 10 additions & 0 deletions docs/content/configuration/pipes/snowflake.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
title: Snowflake
layout: default
grand_parent: Configuration
parent: Pipes
---

# Snowflake
{% include pipe_description.md pipe=site.data.config.pipes.snowflake %}

0 comments on commit 18f8c1a

Please sign in to comment.