Add GitHub Action (#97)

Dockerfile

@@ -0,0 +1,6 @@
+FROM tiangolo/docker-with-compose
+
+COPY entrypoint.sh /entrypoint.sh
+RUN chmod +x /entrypoint.sh
+
+ENTRYPOINT ["/entrypoint.sh"]

README.md

@@ -1,5 +1,52 @@
 ![image](images/confluent-logo-300-2.png)
-
-# Documentation
 
-You can find the documentation and instructions for this repo at [https://docs.confluent.io/platform/current/tutorials/build-your-own-demos.html](https://docs.confluent.io/platform/current/tutorials/build-your-own-demos.html?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.cp-all-in-one)
+# cp-all-in-one
+
+This repo runs [cp-all-in-one](https://github.com/confluentinc/cp-all-in-one/tree/latest/cp-all-in-one), a Docker Compose for Confluent Platform.
+
+# Standalone Usage
+
+See [Confluent documentation](https://docs.confluent.io/platform/current/tutorials/build-your-own-demos.html?utm_source=github&utm_medium=demo&utm_campaign=ch.examples_type.community_content.cp-all-in-one) for details.
+
+# Usage as a GitHub Action
+
+- `service`: up to which service in the docker-compose.yml file to run.  Default is none, so all services are run
+- `github-branch-version`: which GitHub branch of [cp-all-in-one](https://github.com/confluentinc/cp-all-in-one) to run.  Default is `latest`.
+- `type`: cp-all-in-one (based on Confluent Server) or cp-all-in-one-community (based on Apache Kafka)
+
+Example to run ZooKeeper and Confluent Server on Confluent Platform `7.1.0`:
+
+```yaml
+
+    steps:
+
+      - name: Run Confluent Platform (Confluent Server)
+        uses: confluentinc/cp-all-in-one@v0.1
+        with:
+          service: broker
+          github-branch-version: 7.1.1-post
+```
+
+Example to run all Apache Kafka services on `latest`:
+
+```yaml
+
+    steps:
+
+      - name: Run Confluent Platform (Confluent Server)
+        uses: confluentinc/cp-all-in-one@v0.1
+          type: cp-all-in-one-community
+```
+
+# Ports
+
+To connect to services in Docker, refer to the following ports:
+
+- ZooKeeper: 2181
+- Kafka broker: 9092
+- Kafka broker JMX: 9101
+- Confluent Schema Registry: 8081
+- Kafka Connect: 8083
+- Confluent Control Center: 9021
+- ksqlDB: 8088
+- Confluent REST Proxy: 8082

action.yml

@@ -0,0 +1,19 @@
+name: 'Run Kafka with Confluent cp-all-in-one'
+description: 'Setup Confluent cp-all-in-one'
+author: 'confluentinc'
+inputs:
+  github branch version:
+    description: 'GitHub branch of cp-all-in-one to use'
+    required: false
+    default: 'latest'
+  service:
+    description: 'Define service up to which to start'
+    required: false
+    default: ''
+  type:
+    description: 'Which cp-all-in-one to deploy (one of: cp-all-in-one, cp-all-in-one-community)'
+    required: false
+    default: 'cp-all-in-one'
+runs:
+  using: 'docker'
+  image: 'Dockerfile'

entrypoint.sh

@@ -0,0 +1,5 @@
+#!/bin/sh
+
+cp $INPUT_TYPE/docker-compose.yml docker-compose.yml
+docker-compose up -d $INPUT_SERVICE
+sleep 20

github-actions-example/Dockerfile

@@ -0,0 +1,7 @@
+FROM python:3.7-slim
+
+COPY requirements.txt /tmp/requirements.txt
+RUN pip3 install -U -r /tmp/requirements.txt
+
+COPY src/*.py /usr/bin/.
+COPY schemas/* /etc/.

github-actions-example/README.md

@@ -0,0 +1,50 @@
+## README
+
+* [Overview](#overview)
+* [Instructions](#instructions)
+* [Resources](#resources)
+
+## Overview
+
+This demo uses GitHub Actions for developing and valiating Apache Kafka client application, focusing on a Python producer and consumer using Kafka Python Client, plus examples for testing [schema](schemas/) compatibility using both Python and Java.
+
+The code defines an GitHub Actions workflow [kafka-app](.github/workflows/kafka-app.yml) with jobs to demonstrate a few capabilities:
+
+- [build](.github/workflows/kafka-app.yml#L20-L44): Build a Docker image for the [Kafka Python client application](src/)
+- [sanity](.github/workflows/kafka-app.yml#L46-L62): Sanity test the app with [rdkafka mock](https://github.com/edenhill/librdkafka/blob/master/src/rdkafka_mock.h)
+- [local](.github/workflows/kafka-app.yml#L64-L95): Locally test the app using an action called [cp-all-in-one-action](https://github.com/marketplace/actions/run-kafka-with-confluent-cp-all-in-one) to run Confluent Platform from a [Docker Compose file](https://docs.confluent.io/platform/current/tutorials/build-your-own-demos.html#cp-all-in-one)
+- [local-avro](.github/workflows/kafka-app.yml#L97-L128): Locally test the app with Schema Registry and Avro using an action called [cp-all-in-one-action](https://github.com/marketplace/actions/run-kafka-with-confluent-cp-all-in-one) to run Confluent Platform from a [Docker Compose file](https://docs.confluent.io/platform/current/tutorials/build-your-own-demos.html#cp-all-in-one)
+- [ccloud](.github/workflows/kafka-app.yml#L130-L160): Test the app to [Confluent Cloud](https://www.confluent.io/confluent-cloud/tryfree-v1/)
+- [ccloud-avro](.github/workflows/kafka-app.yml#L162-L192): Test the app to [Confluent Cloud](https://www.confluent.io/confluent-cloud/tryfree-v1/) with Confluent Cloud Schema Registry and Avro
+- [ccloud-python-schema-compatibility](.github/workflows/kafka-app.yml#L194-L222): Use [SchemaRegistryClient.test_compatibility()](https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/_modules/confluent_kafka/schema_registry/schema_registry_client.html) as shown in [src/test_schema_compatibility.py](src/test_schema_compatibility.py) to check a new schema's compatibility to the existing schema in Confluent Cloud Schema Registry (Python)
+- [ccloud-java-schema-compatibility](.github/workflows/kafka-app.yml#L224-L253): Use [kafka-schema-registry-maven-plugin](https://docs.confluent.io/platform/current/schema-registry/develop/maven-plugin.html) as shown in the [pom.xml](pom.xml) to check a new schema's compatibility to the existing schema in Confluent Cloud Schema Registry (Java)
+
+![image](images/jobs-graphic.png)
+
+## Instructions
+
+1. Provision a Kafka cluster, enable Schema Registry and create an API key and secret on [Confluent Cloud](https://www.confluent.io/confluent-cloud/tryfree-v1/). Use the [quickstart](https://developer.confluent.io/quickstart/kafka-on-confluent-cloud/), [Confluent Terraform Provider](https://registry.terraform.io/providers/confluentinc/confluent/0.9.0), or [ccloud-stack](https://docs.confluent.io/platform/current/tutorials/examples/ccloud/docs/ccloud-stack.html).
+
+2. In your GitHub account, create [Encrypted Secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets) for your Confluent Cloud cluster and credentials:
+
+```shell
+# Confluent Cloud Kafka cluster
+CONFLUENT_BOOTSTRAP_SERVERS
+CONFLUENT_API_KEY
+CONFLUENT_API_SECRET
+# Confluent Cloud Schema Registry
+CONFLUENT_SCHEMA_REGISTRY_URL
+CONFLUENT_BASIC_AUTH_USER_INFO
+```
+
+3. Fork this repository and push a change to trigger the [workflow](.github/workflows/kafka-app.yml).
+
+![image](images/jobs-text.png)
+
+## Resources
+
+- [Confluent Cloud](https://www.confluent.io/confluent-cloud/tryfree-v1/): Cloud-native service for Apache Kafka
+- [GitHub Actions for ksqlDB](https://github.com/jzaralim/ksqldb-migrations-action): demonstrates schema migrations upon a ksqlDB cluster running in Confluent Cloud
+- [Kafka Python Client](https://docs.confluent.io/kafka-clients/python/current/overview.html)
+- [GitHub Actions](https://docs.github.com/en/actions)
+

github-actions-example/checkstyle/suppressions.xml

@@ -0,0 +1,7 @@
+<?xml version="1.0"?>
+<!DOCTYPE suppressions PUBLIC
+        "-//Puppy Crawl//DTD Suppressions 1.1//EN"
+        "http://www.puppycrawl.com/dtds/suppressions_1_1.dtd">
+
+<suppressions>
+</suppressions>

github-actions-example/configs/librdkafka.ccloud-avro.config

@@ -0,0 +1,8 @@
+bootstrap.servers=${CONFLUENT_BOOTSTRAP_SERVERS}
+security.protocol=SASL_SSL
+sasl.mechanisms=PLAIN
+sasl.username=${CONFLUENT_API_KEY}
+sasl.password=${CONFLUENT_API_SECRET}
+schema.registry.url=${CONFLUENT_SCHEMA_REGISTRY_URL}
+basic.auth.credentials.source=USER_INFO
+basic.auth.user.info=${CONFLUENT_BASIC_AUTH_USER_INFO}

github-actions-example/configs/librdkafka.ccloud.config

@@ -0,0 +1,5 @@
+bootstrap.servers=${CONFLUENT_BOOTSTRAP_SERVERS}
+security.protocol=SASL_SSL
+sasl.mechanisms=PLAIN
+sasl.username=${CONFLUENT_API_KEY}
+sasl.password=${CONFLUENT_API_SECRET}

github-actions-example/configs/librdkafka.local-avro.config

@@ -0,0 +1,2 @@
+bootstrap.servers=broker:29092
+schema.registry.url=http://schema-registry:8081

github-actions-example/configs/librdkafka.local.config

@@ -0,0 +1 @@
+bootstrap.servers=broker:29092

github-actions-example/configs/librdkafka.sanity.config

@@ -0,0 +1 @@
+test.mock.num.brokers=1

github-actions-example/pom.xml

@@ -0,0 +1,60 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+
+    <parent>
+        <groupId>io.confluent</groupId>
+        <artifactId>rest-utils-parent</artifactId>
+        <version>7.1.1</version>
+    </parent>
+
+    <groupId>io.confluent</groupId>
+    <artifactId>GitHub-Actions-Schema-Registry-Test-Compatibility</artifactId>
+    <version>1.0</version>
+
+    <properties>
+        <schemaRegistryUrl></schemaRegistryUrl>
+        <schemaRegistryBasicAuthUserInfo></schemaRegistryBasicAuthUserInfo>
+        <newSchema>schemas/Count.avsc</newSchema>
+        <confluent.version>7.1.0</confluent.version>
+    </properties>
+
+    <repositories>
+      <repository>
+        <id>confluent</id>
+        <name>Confluent</name>
+        <url>https://packages.confluent.io/maven/</url>
+      </repository>
+    </repositories>
+
+    <build>
+        <plugins>
+            <plugin>
+                <groupId>io.confluent</groupId>
+                <artifactId>kafka-schema-registry-maven-plugin</artifactId>
+                <version>${confluent.version}</version>
+                <configuration>
+                    <schemaRegistryUrls>
+                        <param>${schemaRegistryUrl}</param>
+                    </schemaRegistryUrls>
+                    <userInfoConfig>${schemaRegistryBasicAuthUserInfo}</userInfoConfig>
+                    <subjects>
+                        <t3-avro-value>${newSchema}</t3-avro-value>
+                    </subjects>
+                </configuration>
+                <executions>
+                    <execution>
+                        <id>test-compatibility</id>
+                        <phase>validate</phase>
+                        <goals>
+                            <goal>test-compatibility</goal>
+                        </goals>
+                    </execution>
+                </executions>
+            </plugin>
+
+        </plugins>
+    </build>
+</project>

github-actions-example/requirements.txt

@@ -0,0 +1,3 @@
+requests
+certifi
+confluent-kafka[avro,json,protobuf]>=1.4.2

github-actions-example/schemas/Count-fail.avsc

@@ -0,0 +1,15 @@
+{
+  "fields": [
+    {
+      "name": "count",
+      "type": "int"
+    },
+    {
+      "name": "id",
+      "type": ["int", "null"]
+    }
+  ],
+  "name": "Count",
+  "namespace": "io.confluent.examples.clients.cloud",
+  "type": "record"
+}

github-actions-example/schemas/Count-pass.avsc

@@ -0,0 +1,16 @@
+{
+  "fields": [
+    {
+      "name": "count",
+      "type": "int"
+    },
+    {
+      "name": "id",
+      "type": ["int", "null"],
+      "default": null
+    }
+  ],
+  "name": "Count",
+  "namespace": "io.confluent.examples.clients.cloud",
+  "type": "record"
+}

github-actions-example/schemas/Count.avsc

@@ -0,0 +1,11 @@
+{
+  "fields": [
+    {
+      "name": "count",
+      "type": "int"
+    }
+  ],
+  "name": "Count",
+  "namespace": "io.confluent.examples.clients.cloud",
+  "type": "record"
+}