Skip to content

Commit

Permalink
Update README.MD
Browse files Browse the repository at this point in the history
  • Loading branch information
TylerJSimpson authored Mar 11, 2023
1 parent d325509 commit c0bb4f6
Showing 1 changed file with 38 additions and 0 deletions.
38 changes: 38 additions & 0 deletions week_6/README.MD
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# Week 6: Stream Processing - Kafka
Overview of Kafka ... Placeholder

## Setup and Examples
Setting up Kafka using Docker and running producer/consumer examples in JSON and AVRO format and Pyspark.

### Docker Setup

Expand Down Expand Up @@ -51,6 +53,42 @@ docker ps
```

### JSON Example
#### Kafka producer and consumer example using JSON format
Navigate to [python/json_example/](https://github.com/TylerJSimpson/data_engineering_zoomcamp/tree/main/week_6/python/json_example)
Execute [producer.py](https://github.com/TylerJSimpson/data_engineering_zoomcamp/blob/main/week_6/python/json_example/producer.py) to produce messages:
```bash
python producer.py
```
Execute [consumer.py](https://github.com/TylerJSimpson/data_engineering_zoomcamp/blob/main/week_6/python/json_example/consumer.py) to consume messages:
```bash
python consumer.py
```
Notice the output matches the print statement in the consume_from_kafka function:
```python
print(msg_val.key, msg_val.value)
```
The consumer will continue running until exited.

### AVRO Example
#### Kafka producer and consumer example using AVRO format
Navigate to [python/avro_example/](https://github.com/TylerJSimpson/data_engineering_zoomcamp/tree/main/week_6/python/avro_example)
Execute [producer.py](https://github.com/TylerJSimpson/data_engineering_zoomcamp/blob/main/week_6/python/avro_example/producer.py) to produce messages:
```bash
python producer.py
```
Execute [consumer.py](https://github.com/TylerJSimpson/data_engineering_zoomcamp/blob/main/week_6/python/avro_example/consumer.py) to consume messages:
```bash
python consumer.py
```

### Pyspark Example
#### Kafka producer and consumer example using Pyspark
Navigate to [python/streams-example/pyspark/](https://github.com/TylerJSimpson/data_engineering_zoomcamp/tree/main/week_6/python/streams-example/pyspark)
Execute [producer.py](https://github.com/TylerJSimpson/data_engineering_zoomcamp/blob/main/week_6/python/streams-example/pyspark/producer.py) to produce messages:
```bash
python producer.py
```
Execute [consumer.py](https://github.com/TylerJSimpson/data_engineering_zoomcamp/blob/main/week_6/python/streams-example/pyspark/consumer.py) to consume messages:
```bash
python consumer.py
```

0 comments on commit c0bb4f6

Please sign in to comment.