Merge branch 'main' of github.com:evidentlyai/ml_observability_course

docs/book/README.md

@@ -12,15 +12,19 @@ Welcome to the Open-source ML observability course!
 The course starts on **October 16, 2023**. \
 [Sign up](https://www.evidentlyai.com/ml-observability-course) to save your seat and receive weekly course updates.
 
+# How to participate?
+* **Join the course**. [Sign up](https://www.evidentlyai.com/ml-observability-course) to receive weekly updates with course materials and information about office hours.
+* **Course platform [OPTIONAL]**. If you want to receive a course certificate, you should **also** [register](https://evidentlyai.thinkific.com/courses/ml-observability-course) on the platform and complete all the assignments before **December 1, 2023**.
+
+The course starts on **October 16, 2023**. The videos and course notes for the new modules will be released during the course cohort. 
+
 # Links
-* **Newsletter**. [Sign up](https://www.evidentlyai.com/ml-observability-course) to receive weekly updates with the course materials.
+
 * **Discord community**. Join the [community](https://discord.gg/PyAJuUD5mB) to ask questions and chat with others.
-* **Course platform**. [Register](https://evidentlyai.thinkific.com/courses/ml-observability-course) if you want to submit assignments and receive the certificate. This is optional.
 * **Code examples**. Will be published in this GitHub [repository](https://github.com/evidentlyai/ml_observability_course) throughout the course.
-* **Enjoying the course?** [Star](https://github.com/evidentlyai/evidently) Evidently on GitHub to contribute back! This helps us create free, open-source tools and content for the community.
+* **YouTube playlist**. [Subscribe](https://www.youtube.com/playlist?list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF) to the course YouTube playlist to keep tabs on video updates.
 
-
-The course starts on **October 16, 2023**. The videos and course notes for the new modules will be released during the course cohort. 
+**Enjoying the course?** [Star](https://github.com/evidentlyai/evidently) Evidently on GitHub to contribute back! This helps us create free, open-source tools and content for the community.
 
 # What the course is about
 This course is a deep dive into ML model observability and monitoring. 
@@ -38,8 +42,8 @@ ML observability course is organized into six modules. You can follow the comple
 [Module 2. ML monitoring metrics: model quality, data quality, data drift](ml-observability-course/module-2-ml-monitoring-metrics/readme.md). 
 {% endcontent-ref %}
 
-{% content-ref url="ml-observability-course/module-3-ml-monitoring-for-unstructured-data.md" %}
-[Module 3. ML monitoring for unstructured data: NLP, LLM, and embeddings](ml-observability-course/module-3-ml-monitoring-for-unstructured-data.md). 
+{% content-ref url="ml-observability-course/module-3-ml-monitoring-for-unstructured-data/readme.md" %}
+[Module 3. ML monitoring for unstructured data: NLP, LLM, and embeddings](ml-observability-course/module-3-ml-monitoring-for-unstructured-data/readme.md). 
 {% endcontent-ref %}
 
 {% content-ref url="ml-observability-course/module-4-designing-effective-ml-monitoring.md" %}
@@ -54,6 +58,20 @@ ML observability course is organized into six modules. You can follow the comple
 [Module 6. Deploying an ML monitoring dashboard](ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard.md). 
 {% endcontent-ref %}
 
+# Course calendar and deadlines
+
+We will publish new materials throughout the course. 
+
+| Module                                                                   | Week                                                          |
+|--------------------------------------------------------------------------|---------------------------------------------------------------|
+| [Module 1: Introduction to ML monitoring and observability](https://learn.evidentlyai.com/ml-observability-course/module-1-introduction)                | October 16, 2023                                              |
+| [Module 2: ML monitoring metrics: model quality, data quality, data drift](https://learn.evidentlyai.com/ml-observability-course/module-2-ml-monitoring-metrics) | October 23, 2023                                              |
+| [Module 3: ML monitoring for unstructured data: NLP, LLM and embeddings](https://learn.evidentlyai.com/ml-observability-course/module-3-ml-monitoring-for-unstructured-data)   | October 30, 2023                                              |
+| [Module 4: Designing effective ML monitoring](https://learn.evidentlyai.com/ml-observability-course/module-4-designing-effective-ml-monitoring)                              | November 6, 2023                                              |
+| [Module 5: ML pipelines validation and testing](https://learn.evidentlyai.com/ml-observability-course/module-5-ml-pipelines-validation-and-testing)                            | November 13, 2023                                             |
+| [Module 6: Deploying an ML monitoring dashboard](https://learn.evidentlyai.com/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard)                           | November 20, 2023                                             |
+| Final assignment                                                         | November 27, 2023 <br><br> Quizzes and assignment due December 1, 2023 |
+
 # Our approach
 * **Blend of theory and practice**. The course combines key concepts of ML observability and monitoring with practice-oriented tasks.
 * **Practical code examples**. We provide end-to-end deployment blueprints and walk you through the code examples.

docs/book/SUMMARY.md

@@ -17,8 +17,15 @@
   * [2.4. Data quality in machine learning](ml-observability-course/module-2-ml-monitoring-metrics/data-quality-in-ml.md)
   * [2.5. Data quality in ML [CODE PRACTICE]](ml-observability-course/module-2-ml-monitoring-metrics/data-quality-code-practice.md)
   * [2.6. Data and prediction drift in ML](ml-observability-course/module-2-ml-monitoring-metrics/data-prediction-drift-in-ml.md)
+  * [2.7. Deep dive into data drift detection [OPTIONAL]](ml-observability-course/module-2-ml-monitoring-metrics/data-drift-deep-dive.md)
   * [2.8. Data and prediction drift in ML [CODE PRACTICE]](ml-observability-course/module-2-ml-monitoring-metrics/data-prediction-drift-code-practice.md)
-* [Module 3: ML monitoring for unstructured data](ml-observability-course/module-3-ml-monitoring-for-unstructured-data.md)
+* [Module 3: ML monitoring for unstructured data](ml-observability-course/module-3-ml-monitoring-for-unstructured-data/readme.md)
+  * [3.1. Introduction to NLP and LLM monitoring](ml-observability-course/module-3-ml-monitoring-for-unstructured-data/introduction-nlp-llm-monitoring.md)
+  * [3.2. Monitoring data drift on raw text data](ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-data-drift-on-raw-text-data.md)
+  * [3.3. Monitoring text data quality and data drift with descriptors](ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-data-drift-with-descriptors.md)
+  * [3.4. Monitoring embeddings drift](ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-embeddings-drift.md)
+  * [3.5. Monitoring text data [CODE PRACTICE]](ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-text-data-code-practice.md)
+  * [3.6. Monitoring multimodal datasets](ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-multimodal-datasets.md)
 * [Module 4: Designing effective ML monitoring](ml-observability-course/module-4-designing-effective-ml-monitoring.md)
 * [Module 5: ML pipelines validation and testing](ml-observability-course/module-5-ml-pipelines-validation-and-testing.md)
 * [Module 6: Deploying an ML monitoring dashboard](ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard.md)

docs/book/ml-observability-course/module-1-introduction/ml-lifecycle.md

@@ -17,11 +17,11 @@ You can perform different types of evaluations at each of these stages. For exam
 * During data preparation, exploratory data analysis (EDA) helps to understand the dataset and validate the problem statement.
 * At the experiment stage, performing cross-validation and holdout testing helps validate and test if ML models are useful.
 
-![](<../../../images/2023109\_course\_module1\_fin\_images.005.png>)
+![](<../../../images/2023109\_course\_module1\_fin\_images.005-min.png>)
 
 However, the work does not stop here! Once the best model is deployed to production and starts bringing business value, every erroneous prediction has its costs. It is crucial to ensure that this model functions stably and reliably. To do that, one must continuously monitor the production ML model and data.
 
-![](<../../../images/2023109\_course\_module1\_fin\_images.008.png>)
+![](<../../../images/2023109\_course\_module1\_fin\_images.008-min.png>)
 
 ## What can go wrong in production?
 
@@ -34,21 +34,21 @@ Many things can go wrong once you deploy an ML model to the real world. Here are
 * Data schema changes in the upstream system, third-party APIs, or catalogs.
 * Data loss at source when dealing with broken sensors, logging errors, database outages, etc.
 
-![](<../../../images/2023109\_course\_module1\_fin\_images.011.png>)
+![](<../../../images/2023109\_course\_module1\_fin\_images.011-min.png>)
 
 **Broken upstream model**. Often, not one model but a chain of ML models operates in production. If one model gives wrong outputs, it can affect downstream models.
 
-![](<../../../images/2023109\_course\_module1\_fin\_images.012.png>)
+![](<../../../images/2023109\_course\_module1\_fin\_images.012-min.png>)
 
 **Concept drift**. Gradual concept drift occurs when the target function continuously changes over time, leading to model degradation. If the change is sudden – like the recent pandemic – you’re dealing with sudden concept drift. 
 
 **Data drift**. Distribution changes in the input features may signal data drift and potentially cause ML model performance degradation. For example, a significant number of users coming from a new acquisition channel can negatively affect the model trained on user data. Chances are that users from different channels behave differently. To get back on track, the model needs to learn new patterns. 
 
-![](<../../../images/2023109\_course\_module1\_fin\_images.015.png>)
+![](<../../../images/2023109\_course\_module1\_fin\_images.015-min.png>)
 
 **Underperforming segments**. A model might perform differently on diverse data segments. It is crucial to monitor performance across all segments.
 
-![](<../../../images/2023109\_course\_module1\_fin\_images.016.png>)
+![](<../../../images/2023109\_course\_module1\_fin\_images.016-min.png>)
 
 **Adversarial adaptation**. In the era of neural networks, models might face adversarial attacks. Monitoring helps detect these issues on time.
 

docs/book/ml-observability-course/module-1-introduction/ml-monitoring-architectures.md

@@ -14,7 +14,7 @@ It is essential to start monitoring ML models as soon as you deploy them to prod
 
 **Ad hoc reporting** is a great alternative when your resources are limited. You can use Python scripts to calculate and analyze metrics in your notebook. This is a good first step in logging model performance and data quality.
 
-![](<../../../images/2023109\_course\_module1\_fin\_images.061.png>)
+![](<../../../images/2023109\_course\_module1\_fin\_images.061-min.png>)
 
 ## Monitoring frontend
 
@@ -24,13 +24,13 @@ When it comes to visualizing the results of monitoring, you also have options.
 
 **One-off reports**. You can also generate reports as needed and create visualizations or specific one-off analyses based on the model logs. You can create your own reports in Python/R or use different BI visualization tools.
 
-![](<../../../images/2023109\_course\_module1\_fin\_images.065.png>)
+![](<../../../images/2023109\_course\_module1\_fin\_images.065-min.png>)
 
 **BI Systems**. If you want to create a dashboard to track ML monitoring metrics over time, you can also reuse existing business intelligence or software monitoring systems. In this scenario, you must connect existing tools to the ML metric database and add panels or plots to the dashboard.
 
 **Dedicated ML monitoring**. As a more sophisticated approach, you can set up a separate visualization system that gives you an overview of all your ML models and datasets and provides an ongoing, updated view of metrics.
 
-![](<../../../images/2023109\_course\_module1\_fin\_images.066.png>)
+![](<../../../images/2023109\_course\_module1\_fin\_images.066-min.png>)
 
 ## Summing up
 

docs/book/ml-observability-course/module-1-introduction/ml-monitoring-metrics.md

@@ -25,6 +25,6 @@ ML model performance metrics help to ensure that ML models work as expected:
 
 The ultimate measure of the model quality is its impact on the business. Depending on business needs, you may want to monitor clicks, purchases, loan approval rates, cost savings, etc. This is typically custom to the use case and might involve collaborating with product managers or business teams to determine the right business KPIs. 
 
-![](<../../../images/2023109\_course\_module1\_fin\_images.034.png>)
+![](<../../../images/2023109\_course\_module1\_fin\_images.034-min.png>)
 
 For a deeper dive into **ML model quality and relevance** and **data quality and integrity** metrics, head to [Module 2](../module-2-ml-monitoring-metrics/readme.md).

docs/book/ml-observability-course/module-1-introduction/ml-monitoring-observability.md

@@ -31,7 +31,7 @@ Accordingly, you also need to track data quality and model performance metrics.
 
 **Ground truth is not available immediately** to calculate ML model performance metrics. In this case, you can use proxy metrics like data quality to monitor for early warning signs. 
 
-![](<../../../images/2023109\_course\_module1\_fin\_images.024.png>)
+![](<../../../images/2023109\_course\_module1\_fin\_images.024-min.png>)
 
 ## ML monitoring vs ML observability
 
@@ -59,7 +59,7 @@ ML monitoring and observability help:
 * **Trigger actions**. Based on the calculated data and model health metrics, you can trigger fallback, model switching, or automatic retraining.
 * **Document ML model performance** to provide information to the stakeholders.
 
-![](<../../../images/2023109\_course\_module1\_fin\_images.030.png>)
+![](<../../../images/2023109\_course\_module1\_fin\_images.030-min.png>)
 
 ## Who should care about ML monitoring and observability?
 
@@ -72,7 +72,7 @@ The short answer: everyone who cares about the model's impact on business. At th
 
 Other stakeholders include model users, business stakeholders, support, and compliance teams.
 
-![](<../../../images/2023109\_course\_module1\_fin\_images.031.png>)
+![](<../../../images/2023109\_course\_module1\_fin\_images.031-min.png>)
 
 ## Summing up
 

docs/book/ml-observability-course/module-1-introduction/ml-monitoring-setup.md

@@ -17,7 +17,7 @@ While setting up an ML monitoring system, it makes sense to align the complexity
 * **Feedback loop and environmental stability**. Both influence the cadence of metrics calculations and the choice of specific metrics.
 * **Service criticality**. What is the business cost of model quality drops? What risks should we monitor for? More critical models might require a more complex monitoring setup.
 
-![](<../../../images/2023109\_course\_module1\_fin\_images.050.png>)
+![](<../../../images/2023109\_course\_module1\_fin\_images.050-min.png>)
 
 ## Model retraining cadence
 
@@ -26,7 +26,7 @@ ML monitoring and retraining are closely connected. Some retraining factors to k
 * How you implement the retraining: whether you want to monitor the metrics and retrain on a trigger or set up a predefined retraining schedule (for example, weekly).
 * Issues that prevent updating the model too often, e.g., complex approval processes, regulations, need for manual testing.
 
-![](<../../../images/2023109\_course\_module1\_fin\_images.052.png>)
+![](<../../../images/2023109\_course\_module1\_fin\_images.052-min.png>)
 
 ## Reference dataset