This lab is a condensed version of these workshop instructions.
In this lab, we will create and experiment an end-to-end ML Pipeline. This ML pipeline is supported by an automated infrastructure for training, testing, deploying and integrating ML Models.
Furthermore, the model can be deployed in two different environments: DEV environment for QA/Development endpoint and PRD environment for Production endpoint. In addition, we will apply some stress tests to check the ML system scalability and stability.
You'll need an AWS Account with access to the services above. There are resources required by this workshop that are eligible for the AWS Free Tier if your account is less than 12 months old.
WARNING: if your account is more than 12 months old, you may get a bill.
You should have some basic experience with:
- Train/test a ML model
- Python (scikit-learn)
- Jupyter Notebook
- AWS CodePipeline
- AWS CodeCommit
- AWS CodeBuild
- Amazon ECR
- Amazon SageMaker
- AWS CloudFormation
Some experience working with the AWS console is helpful as well.
The following image gives us a high level view of the architecture.
- An ETL process or the ML Developer, prepares a new dataset for training the model and copies it into an S3 Bucket;
- CodePipeline listens to this S3 Bucket, calls a Lambda function for starting training for a job in Sagemaker;
- The lambda function sends a training job request to Sagemaker;
- When the training is finished, CodePipeline gets its status and goes to the next stage if there is no error;
- CodePipeline calls CloudFormation to deploy a model in a Development/QA environment into Sagemaker;
- After finishing the deployment in DEV/QA, CodePipeline awaits for a manual approval
- An approver approves or rejects the deployment. If rejected the pipeline stops here; If approved it goes to the next stage;
- CodePipeline calls CloudFormation to deploy a model into production. This time, the endpoint will count with an AutoScaling policy for HA and Elasticity.
- Done.
This sample code is made available under a modified MIT license. See the LICENSE file.