-
Launch CloudFormation stack
-
Update msk security group to allow inbound traffic from Cloud9 security group
- Navigate to Security Group page in the AWS console
- Select
msk security group
- Add inbound bound rule allowing all traffic from the aws-cloud9 security group
-
Create Kafka topic
- Navigate to Cloud9 page in the AWS console
- Open IDE for the msk-workshop-cloud9 enviorment
- Follow the instructions in Kafka/1_create_topic.py
-
Create OpenSearch index
- via. Cloud9 update the required section(s) and run OpenSearch/1_create_index.py
-
Configure Lambda
-
Navigate to lambda function page in the AWS console
-
Create a lambda function
- Funcation name =
msk-os-lambda
- Runtime =
python 3.7
- Architecture =
x86_64
- Permissions, Execution role = Use an existing role
Lambda-MSK-OpenSearch-Role
- Funcation name =
-
Add MSK trigger
- MSK cluster =
msk-cluster-workshop
- Batch size =
500
- Batch window =
30
- Topic name =
ApplicationMetricTopic
- Starting position =
Latest
- MSK cluster =
-
Add code
- Copy and past the code from Lambda/1_lambda_function_code_batch.py into the code section of the lambda function
- Update the
os_url
variable in the lambda code with the domain endpoint of the OpenSearch cluster deployed by the CloudFormation stack - Deploy the lambda function
-
-
Send data to OpenSearch
-
Navigate to Cloud9 page in the AWS console
-
Send base data via. Cloud9. Update the required section(s) and run Kafka/2_base_data.py
-
Send anomoly data via. Cloud9. Update the required section(s) and run Kafka/3_anomoly_data.py
-
-
Create + run OpenSearch anomaly detector
- via. Cloud9 update the required section(s) and run OpenSearch/3_create_anomoly_detector.py
-
Login to the OpenSeach dashboard, navigate to the anomoly detection section. Explore the anomolies OpenSearch detected
- Automate more of the set up ie. try to minimize the number of steps in the instructions