diff --git a/docs/book/README.md b/docs/book/README.md index c5ff495..a70a26f 100644 --- a/docs/book/README.md +++ b/docs/book/README.md @@ -12,15 +12,19 @@ Welcome to the Open-source ML observability course! The course starts on **October 16, 2023**. \ [Sign up](https://www.evidentlyai.com/ml-observability-course) to save your seat and receive weekly course updates. +# How to participate? +* **Join the course**. [Sign up](https://www.evidentlyai.com/ml-observability-course) to receive weekly updates with course materials and information about office hours. +* **Course platform [OPTIONAL]**. If you want to receive a course certificate, you should **also** [register](https://evidentlyai.thinkific.com/courses/ml-observability-course) on the platform and complete all the assignments before **December 1, 2023**. + +The course starts on **October 16, 2023**. The videos and course notes for the new modules will be released during the course cohort. + # Links -* **Newsletter**. [Sign up](https://www.evidentlyai.com/ml-observability-course) to receive weekly updates with the course materials. + * **Discord community**. Join the [community](https://discord.gg/PyAJuUD5mB) to ask questions and chat with others. -* **Course platform**. [Register](https://evidentlyai.thinkific.com/courses/ml-observability-course) if you want to submit assignments and receive the certificate. This is optional. * **Code examples**. Will be published in this GitHub [repository](https://github.com/evidentlyai/ml_observability_course) throughout the course. -* **Enjoying the course?** [Star](https://github.com/evidentlyai/evidently) Evidently on GitHub to contribute back! This helps us create free, open-source tools and content for the community. +* **YouTube playlist**. [Subscribe](https://www.youtube.com/playlist?list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF) to the course YouTube playlist to keep tabs on video updates. - -The course starts on **October 16, 2023**. The videos and course notes for the new modules will be released during the course cohort. +**Enjoying the course?** [Star](https://github.com/evidentlyai/evidently) Evidently on GitHub to contribute back! This helps us create free, open-source tools and content for the community. # What the course is about This course is a deep dive into ML model observability and monitoring. @@ -38,8 +42,8 @@ ML observability course is organized into six modules. You can follow the comple [Module 2. ML monitoring metrics: model quality, data quality, data drift](ml-observability-course/module-2-ml-monitoring-metrics/readme.md). {% endcontent-ref %} -{% content-ref url="ml-observability-course/module-3-ml-monitoring-for-unstructured-data.md" %} -[Module 3. ML monitoring for unstructured data: NLP, LLM, and embeddings](ml-observability-course/module-3-ml-monitoring-for-unstructured-data.md). +{% content-ref url="ml-observability-course/module-3-ml-monitoring-for-unstructured-data/readme.md" %} +[Module 3. ML monitoring for unstructured data: NLP, LLM, and embeddings](ml-observability-course/module-3-ml-monitoring-for-unstructured-data/readme.md). {% endcontent-ref %} {% content-ref url="ml-observability-course/module-4-designing-effective-ml-monitoring.md" %} @@ -54,6 +58,20 @@ ML observability course is organized into six modules. You can follow the comple [Module 6. Deploying an ML monitoring dashboard](ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard.md). {% endcontent-ref %} +# Course calendar and deadlines + +We will publish new materials throughout the course. + +| Module | Week | +|--------------------------------------------------------------------------|---------------------------------------------------------------| +| [Module 1: Introduction to ML monitoring and observability](https://learn.evidentlyai.com/ml-observability-course/module-1-introduction) | October 16, 2023 | +| [Module 2: ML monitoring metrics: model quality, data quality, data drift](https://learn.evidentlyai.com/ml-observability-course/module-2-ml-monitoring-metrics) | October 23, 2023 | +| [Module 3: ML monitoring for unstructured data: NLP, LLM and embeddings](https://learn.evidentlyai.com/ml-observability-course/module-3-ml-monitoring-for-unstructured-data) | October 30, 2023 | +| [Module 4: Designing effective ML monitoring](https://learn.evidentlyai.com/ml-observability-course/module-4-designing-effective-ml-monitoring) | November 6, 2023 | +| [Module 5: ML pipelines validation and testing](https://learn.evidentlyai.com/ml-observability-course/module-5-ml-pipelines-validation-and-testing) | November 13, 2023 | +| [Module 6: Deploying an ML monitoring dashboard](https://learn.evidentlyai.com/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard) | November 20, 2023 | +| Final assignment | November 27, 2023

Quizzes and assignment due December 1, 2023 | + # Our approach * **Blend of theory and practice**. The course combines key concepts of ML observability and monitoring with practice-oriented tasks. * **Practical code examples**. We provide end-to-end deployment blueprints and walk you through the code examples. diff --git a/docs/book/SUMMARY.md b/docs/book/SUMMARY.md index a96ba4a..5548c1a 100644 --- a/docs/book/SUMMARY.md +++ b/docs/book/SUMMARY.md @@ -17,8 +17,15 @@ * [2.4. Data quality in machine learning](ml-observability-course/module-2-ml-monitoring-metrics/data-quality-in-ml.md) * [2.5. Data quality in ML [CODE PRACTICE]](ml-observability-course/module-2-ml-monitoring-metrics/data-quality-code-practice.md) * [2.6. Data and prediction drift in ML](ml-observability-course/module-2-ml-monitoring-metrics/data-prediction-drift-in-ml.md) + * [2.7. Deep dive into data drift detection [OPTIONAL]](ml-observability-course/module-2-ml-monitoring-metrics/data-drift-deep-dive.md) * [2.8. Data and prediction drift in ML [CODE PRACTICE]](ml-observability-course/module-2-ml-monitoring-metrics/data-prediction-drift-code-practice.md) -* [Module 3: ML monitoring for unstructured data](ml-observability-course/module-3-ml-monitoring-for-unstructured-data.md) +* [Module 3: ML monitoring for unstructured data](ml-observability-course/module-3-ml-monitoring-for-unstructured-data/readme.md) + * [3.1. Introduction to NLP and LLM monitoring](ml-observability-course/module-3-ml-monitoring-for-unstructured-data/introduction-nlp-llm-monitoring.md) + * [3.2. Monitoring data drift on raw text data](ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-data-drift-on-raw-text-data.md) + * [3.3. Monitoring text data quality and data drift with descriptors](ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-data-drift-with-descriptors.md) + * [3.4. Monitoring embeddings drift](ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-embeddings-drift.md) + * [3.5. Monitoring text data [CODE PRACTICE]](ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-text-data-code-practice.md) + * [3.6. Monitoring multimodal datasets](ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-multimodal-datasets.md) * [Module 4: Designing effective ML monitoring](ml-observability-course/module-4-designing-effective-ml-monitoring.md) * [Module 5: ML pipelines validation and testing](ml-observability-course/module-5-ml-pipelines-validation-and-testing.md) * [Module 6: Deploying an ML monitoring dashboard](ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard.md) diff --git a/docs/book/ml-observability-course/module-1-introduction/ml-lifecycle.md b/docs/book/ml-observability-course/module-1-introduction/ml-lifecycle.md index 3845dd7..984653e 100644 --- a/docs/book/ml-observability-course/module-1-introduction/ml-lifecycle.md +++ b/docs/book/ml-observability-course/module-1-introduction/ml-lifecycle.md @@ -17,11 +17,11 @@ You can perform different types of evaluations at each of these stages. For exam * During data preparation, exploratory data analysis (EDA) helps to understand the dataset and validate the problem statement. * At the experiment stage, performing cross-validation and holdout testing helps validate and test if ML models are useful. -![](<../../../images/2023109\_course\_module1\_fin\_images.005.png>) +![](<../../../images/2023109\_course\_module1\_fin\_images.005-min.png>) However, the work does not stop here! Once the best model is deployed to production and starts bringing business value, every erroneous prediction has its costs. It is crucial to ensure that this model functions stably and reliably. To do that, one must continuously monitor the production ML model and data. -![](<../../../images/2023109\_course\_module1\_fin\_images.008.png>) +![](<../../../images/2023109\_course\_module1\_fin\_images.008-min.png>) ## What can go wrong in production? @@ -34,21 +34,21 @@ Many things can go wrong once you deploy an ML model to the real world. Here are * Data schema changes in the upstream system, third-party APIs, or catalogs. * Data loss at source when dealing with broken sensors, logging errors, database outages, etc. -![](<../../../images/2023109\_course\_module1\_fin\_images.011.png>) +![](<../../../images/2023109\_course\_module1\_fin\_images.011-min.png>) **Broken upstream model**. Often, not one model but a chain of ML models operates in production. If one model gives wrong outputs, it can affect downstream models. -![](<../../../images/2023109\_course\_module1\_fin\_images.012.png>) +![](<../../../images/2023109\_course\_module1\_fin\_images.012-min.png>) **Concept drift**. Gradual concept drift occurs when the target function continuously changes over time, leading to model degradation. If the change is sudden – like the recent pandemic – you’re dealing with sudden concept drift. **Data drift**. Distribution changes in the input features may signal data drift and potentially cause ML model performance degradation. For example, a significant number of users coming from a new acquisition channel can negatively affect the model trained on user data. Chances are that users from different channels behave differently. To get back on track, the model needs to learn new patterns. -![](<../../../images/2023109\_course\_module1\_fin\_images.015.png>) +![](<../../../images/2023109\_course\_module1\_fin\_images.015-min.png>) **Underperforming segments**. A model might perform differently on diverse data segments. It is crucial to monitor performance across all segments. -![](<../../../images/2023109\_course\_module1\_fin\_images.016.png>) +![](<../../../images/2023109\_course\_module1\_fin\_images.016-min.png>) **Adversarial adaptation**. In the era of neural networks, models might face adversarial attacks. Monitoring helps detect these issues on time. diff --git a/docs/book/ml-observability-course/module-1-introduction/ml-monitoring-architectures.md b/docs/book/ml-observability-course/module-1-introduction/ml-monitoring-architectures.md index 6592e8e..8b454a5 100644 --- a/docs/book/ml-observability-course/module-1-introduction/ml-monitoring-architectures.md +++ b/docs/book/ml-observability-course/module-1-introduction/ml-monitoring-architectures.md @@ -14,7 +14,7 @@ It is essential to start monitoring ML models as soon as you deploy them to prod **Ad hoc reporting** is a great alternative when your resources are limited. You can use Python scripts to calculate and analyze metrics in your notebook. This is a good first step in logging model performance and data quality. -![](<../../../images/2023109\_course\_module1\_fin\_images.061.png>) +![](<../../../images/2023109\_course\_module1\_fin\_images.061-min.png>) ## Monitoring frontend @@ -24,13 +24,13 @@ When it comes to visualizing the results of monitoring, you also have options. **One-off reports**. You can also generate reports as needed and create visualizations or specific one-off analyses based on the model logs. You can create your own reports in Python/R or use different BI visualization tools. -![](<../../../images/2023109\_course\_module1\_fin\_images.065.png>) +![](<../../../images/2023109\_course\_module1\_fin\_images.065-min.png>) **BI Systems**. If you want to create a dashboard to track ML monitoring metrics over time, you can also reuse existing business intelligence or software monitoring systems. In this scenario, you must connect existing tools to the ML metric database and add panels or plots to the dashboard. **Dedicated ML monitoring**. As a more sophisticated approach, you can set up a separate visualization system that gives you an overview of all your ML models and datasets and provides an ongoing, updated view of metrics. -![](<../../../images/2023109\_course\_module1\_fin\_images.066.png>) +![](<../../../images/2023109\_course\_module1\_fin\_images.066-min.png>) ## Summing up diff --git a/docs/book/ml-observability-course/module-1-introduction/ml-monitoring-metrics.md b/docs/book/ml-observability-course/module-1-introduction/ml-monitoring-metrics.md index 39d31fb..3fd0a43 100644 --- a/docs/book/ml-observability-course/module-1-introduction/ml-monitoring-metrics.md +++ b/docs/book/ml-observability-course/module-1-introduction/ml-monitoring-metrics.md @@ -25,6 +25,6 @@ ML model performance metrics help to ensure that ML models work as expected: The ultimate measure of the model quality is its impact on the business. Depending on business needs, you may want to monitor clicks, purchases, loan approval rates, cost savings, etc. This is typically custom to the use case and might involve collaborating with product managers or business teams to determine the right business KPIs. -![](<../../../images/2023109\_course\_module1\_fin\_images.034.png>) +![](<../../../images/2023109\_course\_module1\_fin\_images.034-min.png>) For a deeper dive into **ML model quality and relevance** and **data quality and integrity** metrics, head to [Module 2](../module-2-ml-monitoring-metrics/readme.md). diff --git a/docs/book/ml-observability-course/module-1-introduction/ml-monitoring-observability.md b/docs/book/ml-observability-course/module-1-introduction/ml-monitoring-observability.md index 1441c6c..adcc69d 100644 --- a/docs/book/ml-observability-course/module-1-introduction/ml-monitoring-observability.md +++ b/docs/book/ml-observability-course/module-1-introduction/ml-monitoring-observability.md @@ -31,7 +31,7 @@ Accordingly, you also need to track data quality and model performance metrics. **Ground truth is not available immediately** to calculate ML model performance metrics. In this case, you can use proxy metrics like data quality to monitor for early warning signs. -![](<../../../images/2023109\_course\_module1\_fin\_images.024.png>) +![](<../../../images/2023109\_course\_module1\_fin\_images.024-min.png>) ## ML monitoring vs ML observability @@ -59,7 +59,7 @@ ML monitoring and observability help: * **Trigger actions**. Based on the calculated data and model health metrics, you can trigger fallback, model switching, or automatic retraining. * **Document ML model performance** to provide information to the stakeholders. -![](<../../../images/2023109\_course\_module1\_fin\_images.030.png>) +![](<../../../images/2023109\_course\_module1\_fin\_images.030-min.png>) ## Who should care about ML monitoring and observability? @@ -72,7 +72,7 @@ The short answer: everyone who cares about the model's impact on business. At th Other stakeholders include model users, business stakeholders, support, and compliance teams. -![](<../../../images/2023109\_course\_module1\_fin\_images.031.png>) +![](<../../../images/2023109\_course\_module1\_fin\_images.031-min.png>) ## Summing up diff --git a/docs/book/ml-observability-course/module-1-introduction/ml-monitoring-setup.md b/docs/book/ml-observability-course/module-1-introduction/ml-monitoring-setup.md index 609a4ba..57544aa 100644 --- a/docs/book/ml-observability-course/module-1-introduction/ml-monitoring-setup.md +++ b/docs/book/ml-observability-course/module-1-introduction/ml-monitoring-setup.md @@ -17,7 +17,7 @@ While setting up an ML monitoring system, it makes sense to align the complexity * **Feedback loop and environmental stability**. Both influence the cadence of metrics calculations and the choice of specific metrics. * **Service criticality**. What is the business cost of model quality drops? What risks should we monitor for? More critical models might require a more complex monitoring setup. -![](<../../../images/2023109\_course\_module1\_fin\_images.050.png>) +![](<../../../images/2023109\_course\_module1\_fin\_images.050-min.png>) ## Model retraining cadence @@ -26,7 +26,7 @@ ML monitoring and retraining are closely connected. Some retraining factors to k * How you implement the retraining: whether you want to monitor the metrics and retrain on a trigger or set up a predefined retraining schedule (for example, weekly). * Issues that prevent updating the model too often, e.g., complex approval processes, regulations, need for manual testing. -![](<../../../images/2023109\_course\_module1\_fin\_images.052.png>) +![](<../../../images/2023109\_course\_module1\_fin\_images.052-min.png>) ## Reference dataset diff --git a/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/data-drift-deep-dive.md b/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/data-drift-deep-dive.md new file mode 100644 index 0000000..6e16a84 --- /dev/null +++ b/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/data-drift-deep-dive.md @@ -0,0 +1,204 @@ +# 2.7. Deep dive into data drift detection [OPTIONAL] + +{% embed url="https://www.youtube.com/watch?v=N47SHSP6RuY&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=13" %} + +**Video 7**. [Data and prediction drift in ML](https://www.youtube.com/watch?v=N47SHSP6RuY&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=13), by Emeli Dral + +Welcome to the deep dive into data drift detection! We will cover the following topics: + +**Data drift detection methods:** +* More on methods to detect data drift +* Strategies for choosing drift detection approach + +**Special cases:** +* Detecting drift for large datasets +* Detecting drift for real-time models +* Using drift as a retraining trigger + +**Useful tips:** +* How to interpret prediction and data drift together? +* What to do after drift is detected + +## Drift detection methods + +Let’s have a closer look at the commonly used approaches to drift detection. + +**Parametric statistical tests** +There are both **one-sample** and **two-sample** parametric tests.: +* **If you only have current data** and no reference data is available, you can use **Z-test and T-test for mean** (m = m0) or **one-proportion Z-test** (p = p0) to detect data drift. These methods can work if you have interpretable features – e.g., salary or age – as you need to develop the hypotheses on distribution values. +* **If reference data is available**, you can use two-sample parametric tests to compare distributions: for example, **two-proportions Z-test** or **two-sample Z-test and T-test for means** (for normally distributed samples). + +Some considerations to keep in mind when using parametric tests: +* **They require different tests for different features.** For example, some tests assume that your data are normally distributed. +* **They are more sensitive to drift than non-parametric tests.** If you work on a problem where you have a small dataset and want to react to even minor deviations – it makes sense to use parametric tests. +* **Hard to fine-tune if you have a lot of features.** If you have many features with different feature types, you need to invest a lot of time in choosing the right test for each feature. + +It makes sense to use parametric tests if you have a small number of interpretable features and work on critical use cases (e.g., in healthcare). + +![](<../../../images/2023109\_course\_module2.081-min.png>) + +**Non-parametric statistical tests** +Non-parametric tests are less demanding to the properties of data samples and thus are widely used. Examples include **Kolmogorov-Smirnov** test, K-sample **Anderson-Darling** test, **Pearson’s chi-squared** test, **Fisher’s/Barnard’s** exact test for small samples, etc. + +When using non-parametric tests, consider the following: +* **Feature type.** You can use heuristics to choose suitable tests based on the feature type, e.g., numerical, categorical, or binary. +* **Sensitivity.** Non-parametric tests are less sensitive to drift than parametric tests. +* **Data volumes.** It makes sense to use non-parametric tests for low-volume datasets or samples (e.g., less than 1000 objects). + +![](<../../../images/2023109\_course\_module2.082-min.png>) + +**Distance-based approaches** +Distance-based methods measure how far two distributions are from each other and thus are easy to interpret. For example, you can calculate **Wasserstein distance**, **Jensen-Shannon divergence**, or **Population Stability Index** (PSI). + +Some considerations to keep in mind when using distance-based methods to detect data drift: +* **Variety of metrics is available.** Roughly any metric that shows difference/similarity between distributions can be used as a drift detection method. +* **High interpretability compared to statistical tests.** Often, it makes more sense to pick an interpretable metric rather than a statistical test. +* **Data volume.** It makes sense to use distance-based methods for larger datasets (e.g., > 1000 objects). + +![](<../../../images/2023109\_course\_module2.083-min.png>) + +**Domain classification** +This approach uses binary classifiers to distinguish between reference and current data. It can be used to detect data drift in different data types, including embeddings, unstructured data (such as texts), and multimodal data. + +{% hint style="info" %} +**Further reading:** [Which test is the best? We compared 5 methods to detect data drift on large datasets](https://www.evidentlyai.com/blog/data-drift-detection-large-datasets). +{% endhint %} + +## How to choose a drift detection approach? + +**Data drift detection is a heuristic.** There is no strict law. This is why it is important that you consider why you want to detect drift, and what method might make sense for you. + +Consider your problem statement and dataset properties. For example: +* If your use case is sensitive, you might want to use parametric tests. +* If interpretability is important, consider distance-based methods. +* Domain classification can be a good choice if you work with various data types – text, videos, tabular data. + +To choose the right drift detection approach for your particular problem statement, you can consider two options: + +**1. Go with defaults.** + +In this scenario, you pick some reasonable defaults to start and adjust the sensitivity as you proceed with monitoring. +* Start with basic assumptions. Do you want to detect drift for the whole dataset or only consider drift in important features? +* Pick reasonable metrics and thresholds. For example, for numerical features, you can pick Wasserstein Distance at 0.1 threshold. +* Start monitoring. +* Visualize results. +* Adjust based on false alarms, sensitivity, and drift interpretations. + +**2. Experiment.** + +In this scenario, you use historical data to tweak detection parameters using past known drifts. Here is an example of an experiment: +* Take data for a stable period +* Take data with known drift or simulate drift using synthetic data. +* Apply different drift detection approaches. Experiment with tests, thresholds, window size and/or bucketing parameters. +* Choose the optimal approach that detects known drifts and minimizes false alarms. + +## Special cases +There are some special cases to keep in mind when detecting data drift: + +**Large datasets.** +Statistics was made to work with samples. Having many objects and/or features in a dataset can lead to some tests being “too sensitive” or taking too long to compute. If this is the case, you can use **sampling** to pick representative observations and apply tests on top of them. Alternatively, you can try **bucketing** to aggregate observations and reduce the amount of data. For example, you can detect drift on top of hourly data instead of minutely. + +![](<../../../images/2023109\_course\_module2.089-min.png>) + +**Non-batch models.** +While some metrics can be calculated in real-time, we need to generate a batch of data to detect data drift. + +The solution is to use **window functions** to perform tests on continuous data streams. You can pick a window function (i.e., moving windows with/without moving reference), choose the window and step size to create batches for comparison, and “compare” the windows. + +![](<../../../images/2023109\_course\_module2.090-min.png>) + +**Feature drift as a retraining trigger.** +There are both pros and cons of using drift detection as the retraining trigger. + +Generally, we do not recommend retraining a model every time the drift is detected because: +* **Data might be low-quality.** Retraining the model on corrupted data will be useless if data drift occurs due to data processing issues. +* **Data might be insufficient.** Sometimes, we just don’t have enough data for new model training. +* **Data might be non-representative.** Look out for unstable periods, e.g., pandemic, seasonal spikes, etc. + +Instead, try to understand data drift first: +* **Data drift as an investigation trigger.** Try to figure out the root cause of the detected drift. +* **Data drift as a labeling trigger.** You can use a data drift signal to start the labeling process to be able to compute the actual model quality metrics. + +If you use data drift as a retraining trigger, it is critical to implement a solid evaluation process before roll-out to make sure the new model performs well. + +![](<../../../images/2023109\_course\_module2.091-min.png>) + +## How to interpret data and prediction drift together? + +It often makes sense to monitor both prediction drift (change in the model outputs) and data drift (change in the model features). + +However, data and prediction drift do not necessarily mean that something is wrong. Let’s look at two examples of data and prediction drift detected together or independently. + +**Scenario 1. Data drift: detected. Prediction drift: not detected.** +There are both positive and negative ways to interpret it. + +**Positive interpretation:** +* Important features did not change. +* Model is robust enough to survive drift. +* No need to intervene. + +**Negative interpretation:** +* Important features changed. +* Model should have reacted but did not. It does not extrapolate well. +* We need to intervene. + +**Scenario 2. Data drift: detected. Prediction drift: detected.** +Again, there are positive and negative ways of interpreting it. + +**Positive interpretation:** +* Important features changed. +* Model reacts and extrapolates well (e.g., prices lower -> higher sales) +* No need to intervene. + +**Negative interpretation:** +* Important features changed. +* Model behavior is unreasonable. +* We need to intervene. + +## What to do if drift is detected? + +Here are some possible steps to take if the drift is detected: + +**1. Check the data quality.** +Make sure the drift is “real” and try to interpret where the drift is coming from. Data entry errors, stale features, and lost data are data quality issues disguised as data drift. If this is the case, fix the data first. + +![](<../../../images/2023109\_course\_module2.098-min.png>) + +**2. Investigate the drift.** +Analyze which features have changed and how much. To understand the shift, you can: +* Visualize distributions +* Analyze correlation changes +* Check descriptive stats +* Evaluate segments +* Seek real-world explanations (e.g., a new marketing campaign) +* Team up with domain experts + +![](<../../../images/2023109\_course\_module2.100-min.png>) + +**3. Doing nothing is also an option.** +You might treat the drift as a false alarm, be satisfied with how the model handles drift, or simply decide to wait. + +![](<../../../images/2023109\_course\_module2.101-min.png>) + +**4. Actively reacting to drift.** +However, often, we need to react when the drift is detected: +* **Retrain the model.** Get new labels and actual values and re-fit the same model on the latest data. +* **Rebuild the model.** If the change is significant, you might need to rebuild the training pipeline and test new model architectures. +* **Tune the model.** For example, you can change a threshold for drift detection. +* **Use a fallback strategy.** Decide without ML: switch to manual processing, heuristics, or non-ML models. + +![](<../../../images/2023109\_course\_module2.102-min.png>) + +{% hint style="info" %} +**Further reading:** ["My data drifted. What's next?" How to handle ML model drift in production.](https://www.evidentlyai.com/blog/ml-monitoring-data-drift-how-to-handle). +{% endhint %} + +## Summing up + +We discussed different drift detection methods and how to choose the optimal approach for your dataset and problem statement. We covered special cases like handling large datasets, calculating drift for real-time models, and using drift as a retraining trigger. We also learned how to interpret data and prediction drift and what to do if drift is detected. + +Further reading: +* [Which test is the best? We compared 5 methods to detect data drift on large datasets](https://www.evidentlyai.com/blog/data-drift-detection-large-datasets) +* ["My data drifted. What's next?" How to handle ML model drift in production.](https://www.evidentlyai.com/blog/ml-monitoring-data-drift-how-to-handle) + +Up next: code practice on how to detect data drift using the open-source [Evidently](https://github.com/evidentlyai/evidently) Python library. diff --git a/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/data-prediction-drift-in-ml.md b/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/data-prediction-drift-in-ml.md index ab996d6..aea501f 100644 --- a/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/data-prediction-drift-in-ml.md +++ b/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/data-prediction-drift-in-ml.md @@ -10,15 +10,15 @@ When ground truth is unavailable or delayed, we cannot calculate ML model qualit **Prediction drift** shows changes in the distribution of **model outputs** over time. Without target values, this is the best proxy of the model behavior. Detected changes in the model outputs may be an early signal of changes in the model environment, data quality bugs, pipeline errors, etc. -![](<../../../images/2023109\_course\_module2.058.png>) +![](<../../../images/2023109\_course\_module2.058-min.png>) **Feature drift** demonstrates changes in the distribution of **input features** over time. When we train the model, we assume that if the input data remains reasonably similar, we can expect similar model quality. Thus, data distribution drift can be an early warning about model quality decay, important changes in the model environment or user behavior, unannounced changes to the modeled process, etc. -![](<../../../images/2023109\_course\_module2.060.png>) +![](<../../../images/2023109\_course\_module2.060-min.png>) Prediction and feature drift can serve as early warning signs for model quality issues. They can also help pinpoint a root cause when the model decay is already observed. -![](<../../../images/2023109\_course\_module2.065.png>) +![](<../../../images/2023109\_course\_module2.065-min.png>) Some key considerations about data drift to keep in mind: * **Prediction drift is usually more important than feature drift**. If you monitor one thing, look at the outputs. @@ -51,11 +51,11 @@ Here is how the defaults are implemented in the Evidently open-source library. **For small datasets (<=1000)**, you can use Kolmogorov-Smirnov test for numerical features, Chi-squared test for categorical features, and proportion difference test for independent samples based on Z-score for binary categorical features. -![](<../../../images/2023109\_course\_module2.070.png>) +![](<../../../images/2023109\_course\_module2.070-min.png>) **For large datasets (>1000)**, you might use Wasserstein Distance for numerical features and Jensen-Shannon divergence for categorical features. -![](<../../../images/2023109\_course\_module2.071.png>) +![](<../../../images/2023109\_course\_module2.071-min.png>) ## Univariate vs. multivariate drift diff --git a/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/data-quality-in-ml.md b/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/data-quality-in-ml.md index 1fdd9df..d0c3bf7 100644 --- a/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/data-quality-in-ml.md +++ b/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/data-quality-in-ml.md @@ -17,7 +17,7 @@ Some common data processing issues are: Issues can also arise if the data schema changes or data is lost at the source (e.g., broken in-app logging or frozen sensor values). If you have several models interacting with each other, broken upstream models can affect downstream models. -![](<../../../images/2023109\_course\_module2.041.png>) +![](<../../../images/2023109\_course\_module2.041-min.png>) ## Data quality metrics and analysis @@ -30,7 +30,7 @@ Issues can also arise if the data schema changes or data is lost at the source ( Then, you can visualize and compare statistics and data distributions of the current data batch and reference data to ensure data stability. -![](<../../../images/2023109\_course\_module2.047.png>) +![](<../../../images/2023109\_course\_module2.047-min.png>) When it comes to monitoring data quality, you must define the conditions for alerting. diff --git a/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/evaluate-ml-model-quality.md b/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/evaluate-ml-model-quality.md index 207a40b..f5f6d1a 100644 --- a/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/evaluate-ml-model-quality.md +++ b/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/evaluate-ml-model-quality.md @@ -16,7 +16,7 @@ When it comes to standard ML monitoring, we usually start by measuring ML model * **Many segments with different quality**. Aggregated metrics might not provide insights for diverse user/object groups. In this case, we need to monitor quality metrics for each segment separately. * **The target function is volatile**. Volatile target function can lead to fluctuating performance metrics, making it difficult to differentiate between local quality drops and major performance issues. -![](<../../../images/2023109\_course\_module2.005.png>) +![](<../../../images/2023109\_course\_module2.005-min.png>) ## Early monitoring metrics @@ -27,7 +27,7 @@ Early monitoring focuses on metrics derived from consistently available data: in * **Data drift** to monitor changes in the input feature distributions. * **Output drift** to observe shifts in model predictions. -![](<../../../images/2023109\_course\_module2.006.png>) +![](<../../../images/2023109\_course\_module2.006-min.png>) ## Module 2 structure @@ -46,7 +46,7 @@ This module includes both theoretical parts and code practice for each of the ev * [OPTIONAL] Theory: a deeper dive into data drift detection methods and strategies. * Practice: building a sample report in Python to detect data and prediction drift for various data type. -![](<../../../images/2023109\_course\_module2.007.png>) +![](<../../../images/2023109\_course\_module2.007-min.png>) ## Summing up diff --git a/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/ml-quality-metrics-classification-regression-ranking.md b/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/ml-quality-metrics-classification-regression-ranking.md index 1956e71..6b6d22d 100644 --- a/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/ml-quality-metrics-classification-regression-ranking.md +++ b/docs/book/ml-observability-course/module-2-ml-monitoring-metrics/ml-quality-metrics-classification-regression-ranking.md @@ -12,7 +12,7 @@ You need **monitoring** to be able to maintain the ML model's relevance by detec But there is a caveat: to calculate classification, regression, and ranking quality metrics, **you need labels**. If you can, consider labeling at least part of the data to be able to compute them. -![](<../../../images/2023109\_course\_module2.009.png>) +![](<../../../images/2023109\_course\_module2.009-min.png>) ## Classification quality metrics @@ -24,7 +24,7 @@ A classification problem in ML is a task of assigning predefined categories or c * [**ROC-AUC**](https://www.evidentlyai.com/classification-metrics/explain-roc-curve) works for probabilistic classification and evaluates the model's ability to rank correctly. * **Logarithmic loss** demonstrates how close the prediction probability is to the actual value. It is a good metric for probabilistic problem statement. -![](<../../../images/2023109\_course\_module2.012.png>) +![](<../../../images/2023109\_course\_module2.012-min.png>) Methods to help visualize and understand classification quality metrics include: * [**Confusion matrix**](https://www.evidentlyai.com/classification-metrics/confusion-matrix) shows the number of correct predictions – true positives (TP) and true negatives (TN) – and the number of errors – false positives (FP) and false negatives (FN). You can calculate precision, recall, and F1-score based on these values. @@ -32,7 +32,7 @@ Methods to help visualize and understand classification quality metrics include: * **Class separation quality** helps visualize correct and incorrect predictions for each class. * **Error analysis**. You can also map predicted probabilities or model errors alongside feature values and explore if a specific type of misclassification is connected to the particular feature values. -![](<../../../images/2023109\_course\_module2.016.png>) +![](<../../../images/2023109\_course\_module2.016-min.png>) {% hint style="info" %} **Further reading:** [What is your model hiding? A tutorial on evaluating ML models](https://www.evidentlyai.com/blog/tutorial-2-model-evaluation-hr-attrition). @@ -47,7 +47,7 @@ Regression models provide numerical output which is compared against actual valu * **Mean Absolute Percentage Error (MAPE)** averages all absolute errors in %. Works well for datasets with objects of different scale (i.e., tens, thousands, or millions). * **Symmetric MAPE** provides different penalty for over- or underestimation. -![](<../../../images/2023109\_course\_module2.020.png>) +![](<../../../images/2023109\_course\_module2.020-min.png>) Some of the methods to analyze and visualize regression model quality are: * **Predicted vs. Actual** value plots and Error over time plots help derive patterns in model predictions and behavior (e.g., Does the model tend to have bigger errors during weekends or hours of peak demand?). @@ -55,7 +55,7 @@ Some of the methods to analyze and visualize regression model quality are: You can also map extreme errors alongside feature values and explore if a specific type of error is connected to the particular feature values. -![](<../../../images/2023109\_course\_module2.025.png>) +![](<../../../images/2023109\_course\_module2.025-min.png>) ## Ranking quality metrics @@ -69,7 +69,7 @@ We need to estimate the order of objects to measure quality in ranking tasks. So * **Recall @k** is a coverage of all relevant objects in top-K results. * **Lift @k** reflects an improvement over random ranking. -![](<../../../images/2023109\_course\_module2.028.png>) +![](<../../../images/2023109\_course\_module2.028-min.png>) If you work on a recommender system, you might want to consider additional – “beyond accuracy” – metrics that reflect RecSys behavior. Some examples are: * Serendipity diff --git a/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data.md b/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data.md deleted file mode 100644 index 4f49890..0000000 --- a/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -description: Monitoring NLP, LLM and embeddings. ---- - -# Module 3: ML monitoring for unstructured data - -Course content coming soon! diff --git a/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/introduction-nlp-llm-monitoring.md b/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/introduction-nlp-llm-monitoring.md new file mode 100644 index 0000000..98cb025 --- /dev/null +++ b/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/introduction-nlp-llm-monitoring.md @@ -0,0 +1,60 @@ +# 3.1. Introduction to NLP and LLM monitoring + +{% embed url="https://www.youtube.com/watch?v=tYA0h3mPeZE&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=15" %} + +**Video 1**. [Introduction to NLP and LLM monitoring](https://www.youtube.com/watch?v=tYA0h3mPeZE&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=15), by Emeli Dral + +## Use cases for NLP and LLM models + +NLP and LLM models are widely used for both predictive and generative tasks. + +**Predictive applications** include: +* **Classification tasks**, e.g., spam detection, classification of support tickets, evaluating text sentiment, etc. +* **Search (ranking) tasks** are common in e-commerce website search, content recommendations, etc. +* **Information extraction**, like extracting structured information (names, dates, etc.) from text, etc. + +**Generative applications** cover such tasks as translation, summarization, and text generation, e.g., conversational interfaces, article generation, code generation, etc. + +Modern ML systems are often a combination of those: e.g., a support chatbot includes a classifier to detect intent and a generative model to produce a text response. + +## What can go wrong with NLP and LLMs? + +All production ML models need monitoring. NLP and LLM models are no exception. To build our monitoring strategy, we need to understand what can go wrong with the models that run on the unstructured data. + +There are standard issues that apply to both unstructured and tabular data: +* **Technical errors** such as wrong encoding or data processing bugs. +* **Data shifts** like new topics or unexpected usage scenarios. + +However, there are also issues specific to working with unstructured data. For example: +* **Model attacks,** e.g., prompt injection and adversarial usage. +* **Model behavior shift.** For example, if you rely on third-party models, you can face changes in their properties due to retraining. +* **Hallucinations** arise when the model starts to generate factually incorrect or unrelated answers. + +![](<../../../images/2023109\_course\_module3.004-min.png>) + +## ML monitoring metrics for NLP and LLMs + +So, what can you monitor to detect the discussed issues? There are two groups of signals to keep tabs on: + +**1. Direct signals.** This group includes standard model quality metrics such as accuracy, precision, and recall – and requires labeled data. + +When dealing with **predictive applications** – e.g., classification or ranking tasks – we typically have a "right" answer. In this case, we can label the data and then compare labels and the model’s outputs to calculate direct quality metrics. + +However, for **generative applications** (e.g., translation, text summarization), there is often no single correct answer and many possible "good" responses. If this is the case, relying on labels and standard quality metrics is not always possible. Instead, you can use feedback, human labeling, LLM-based grading, and response validation as proxy signals. + +![](<../../../images/2023109\_course\_module3.006-min.png>) + +**2. Proxy metrics** help evaluate the model properties. You can look out for data quality, data and prediction drift, or user feedback. + +Here are three proxy strategies for unstructured data you can use _(disclaimer: we use text as the main type of unstructured data)_: +* **Monitoring raw text data.** Using analytical methods that process raw texts allows you to catch an interpretable signal and use it for hypothesis formulation and debugging. +* **Monitoring text descriptors.** You can extract signals from raw data and compute metrics based on these structured signals. +* **Monitoring embeddings.** When raw data is not available, the embeddings can be used. You can monitor for changes in the distributions of input and output embeddings. + +![](<../../../images/2023109\_course\_module3.007-min.png>) + +## Summing up + +Monitoring NLP and LLM models differs from monitoring models built on structured data. Both predictive and generative applications have their own monitoring needs, and understanding these differences is crucial for effective model observability. You can use both direct quality metrics and proxy signals to keep tabs on production NLP- and LLM-based systems. + +Up next: delving deeper into detecting raw text data drift. diff --git a/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-data-drift-on-raw-text-data.md b/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-data-drift-on-raw-text-data.md new file mode 100644 index 0000000..7b6f5dd --- /dev/null +++ b/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-data-drift-on-raw-text-data.md @@ -0,0 +1,58 @@ +# 3.2. Monitoring data drift on raw text data + +{% embed url="https://www.youtube.com/watch?v=wHyXSyVg5Ag&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=16" %} + +**Video 2**. [Monitoring data drift on raw text data](https://www.youtube.com/watch?v=wHyXSyVg5Ag&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=16), by Emeli Dral + +## Challenges of monitoring raw text data + +Handling raw text data is more complex than dealing with structured tabular data. With structured data, you can usually define “good” or “expected” data, e.g., particular feature distributions or statistical values can signal the data quality. For unstructured data, there is no straightforward way to define data quality or extract a signal for raw text data. + +When it comes to data drift detection, you can use two strategies that rely on raw text data: **domain classifier** and **topic modeling**. + +![](<../../../images/2023109\_course\_module3.011-min.png>) + +## Domain classifier + +**Domain classifier** method or **model-based drift detection** uses a classifier model to compare distributions of reference and current datasets by training a model that predicts to which dataset a specific text belongs. If the model can confidently identify which text samples belong to the current or reference dataset, the two datasets are probably sufficiently different. + +{% hint style="info" %} +**Further reading:** this approach is described in more detail in the paper ["Failing loudly: An Empirical Study of Methods for Detecting Dataset Shift"](https://arxiv.org/abs/1810.11953). +{% endhint %} + +![](<../../../images/2023109\_course\_module3.013-min.png>) + +You can directly use the ROC AUC of the binary classifier as the “drift score” when you deal with **large datasets**. If you work with **smaller datasets** (< 1000), you can compare the model ROC AUC against a random classifier. + +The benefit of using model-based drift detection on raw data is its **interpretability**. In this case, you can identify top words and text examples that were easy to classify to explain the drift and debug the model. + +![](<../../../images/2023109\_course\_module3.014-min.png>) + +{% hint style="info" %} +**Further reading:** [Monitoring NLP models in production: a tutorial on detecting drift in text data](https://www.evidentlyai.com/blog/tutorial-detecting-drift-in-text-data). +{% endhint %} + +## Topic modeling + +Another strategy for evaluating raw data quality is **topic modeling**. The goal here is to categorize text into interpretable topic clusters, so instead of a binary classification model, we use a **clustering model**. + +How it works: +* Apply the clustering model to new batches of data. +* Monitor the size and share of different topics over time. +* Changes in topics can indicate data drift. + +Using this method can be challenging due to difficulties in building a good clustering model: +* There is no ideal structure in clustering. +* Typically, it requires manual tuning to build accurate and interpretable clusters. + +![](<../../../images/2023109\_course\_module3.018-min.png>) + +## Summing up + +Defining data quality and tracking data drift for text data can be challenging. However, we can extract interpretable signals from text data to detect drift. You can use such methods as domain classifier and topic modeling to monitor for drift and evaluate the quality of raw text data. + +Further reading: +* [Failing loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://arxiv.org/abs/1810.11953) +* [Monitoring NLP models in production: a tutorial on detecting drift in text data](https://www.evidentlyai.com/blog/tutorial-detecting-drift-in-text-data) + +Up next: an exploration of alternative text drift detection methods that use descriptors. diff --git a/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-data-drift-with-descriptors.md b/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-data-drift-with-descriptors.md new file mode 100644 index 0000000..e5ff6a1 --- /dev/null +++ b/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-data-drift-with-descriptors.md @@ -0,0 +1,47 @@ +# 3.3. Monitoring text data quality and data drift with descriptors + +{% embed url="https://www.youtube.com/watch?v=UwWGxyCHQSw&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=17" %} + +**Video 3**. [Monitoring text data quality and data drift with descriptors](https://www.youtube.com/watch?v=UwWGxyCHQSw&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=17), by Emeli Dral + +## What is a text descriptor? + +A **text descriptor** is any feature you can derive or calculate from raw text. Text descriptor transforms unstructured data (e.g., text) into structured data (e.g., numeric or categorical descriptors). + +Depending on your goal and problem statement, there are various groups of descriptors you can calculate: +* **Text data quality.** Example descriptors are text length, the share of out-of-vocabulary words, the share of non-letter characters, regular expressions match, etc. + +![](<../../../images/2023109\_course\_module3.024-min.png>) + +* **Text contents.** For instance, you can monitor the presence of trigger words like mentions of specific brands or competitors or text sentiment. Collaborating with product managers or business teams is recommended to determine the meaningful text descriptors for your problem statement and domain area. + +![](<../../../images/2023109\_course\_module3.025-min.png>) + +You can also use **LLM-based grading** that replicates the evaluation that human assessors can do. In this scenario, you automate evaluations by using an LLM to “judge” the generated outputs. + +For example, it can assign a score, check whether the text answers a specific question, define the tone of the output, etc. This method has its caveats: it is not always reliable, can be costly, and requires another set of prompts and outputs to monitor. + +![](<../../../images/2023109\_course\_module3.026-min.png>) + +{% hint style="info" %} +**Further reading:** [Monitoring unstructured data for LLM and NLP with text descriptors](https://www.evidentlyai.com/blog/unstructured-data-monitoring). +{% endhint %} + +## Monitoring text descriptors + +Once you have a structured representation of the unstructured data – in our case, text descriptors – you can use monitoring techniques applied to structured data. For example, you can: +* Measure **descriptor distribution drift**. +* Run **rule-based checks**, such as min-max ranges for text length, the expected share of non-letter symbols, or responses that match a regular expression. +* Track any **statistics calculable for tabular data**, e.g., correlation changes between descriptors and model target. + +![](<../../../images/2023109\_course\_module3.029-min.png>) + +## Summing up + +We introduced basic concepts and examples of text descriptors. You can calculate myriad descriptors, and we recommend collaborating with someone from your business team to determine text descriptors that fit your problem statement best. + +While text descriptors are a powerful tool to analyze and monitor unstructured text data, we can use this approach only if the raw text is available. Other monitoring strategies may be required for scenarios where only embeddings are available. + +Further reading: [Monitoring unstructured data for LLM and NLP with text descriptors](https://www.evidentlyai.com/blog/unstructured-data-monitoring) + +Up next: monitoring embeddings drift. diff --git a/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-embeddings-drift.md b/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-embeddings-drift.md new file mode 100644 index 0000000..13e817f --- /dev/null +++ b/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-embeddings-drift.md @@ -0,0 +1,48 @@ +# 3.4. Monitoring embeddings drift + +{% embed url="https://www.youtube.com/watch?v=0XtABbNYU7U&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=18" %} + +**Video 4**. [Monitoring embeddings drift](https://www.youtube.com/watch?v=0XtABbNYU7U&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=18), by Emeli Dral + +## What are embeddings? + +Embeddings are numerical representations of the input data. They transform raw data like text, images, videos, or music into numerical vectors in high-dimensional space. Embeddings are frequently used in machine learning for classification, regression, and ranking tasks with unstructured data. + +![](<../../../images/2023109\_course\_module3.032-min.png>) + +## Embedding drift detection methods + +Since embeddings are numerical values, we can use many methods to monitor embedding distribution drift. + +**Distance metrics** +Each object in the dataset, represented as an embedding, is a numerical vector. Distance metrics allow calculating distances between these vectors. For example, you can use Euclidean distance or Cosine distance (to assess the angle between vectors). We can detect shifts in datasets by measuring the distance between centroids of reference data and current data. + +![](<../../../images/2023109\_course\_module3.035-min.png>) + +**Model-based drift detection** +This approach uses embeddings to build a domain classifier that distinguishes between reference and current data. The idea is similar to model-based drift detection on raw text data: you get an estimation of the model's ability to distinguish between datasets. However, with embeddings, this approach has a limitation: you cannot use the information about the strongest features or best objects to determine the root cause/source of drift. + +![](<../../../images/2023109\_course\_module3.036-min.png>) + +**Share of drifted components** +You can also use the share of drifted components to monitor embedding drift. This approach treats each embedding component independently and uses drift detection methods that can be applied to numerical values. For each component, drift size or score is assessed. You can then aggregate these individual scores into the number of drifted components or share of drifted components. + +![](<../../../images/2023109\_course\_module3.037-min.png>) + +{% hint style="info" %} +**Further reading:** [Shift happens: we compared 5 methods to detect drift in ML embeddings](https://www.evidentlyai.com/blog/embedding-drift-detection). +{% endhint %} + +The suggested blog might be especially interesting to those who work a lot of embeddings. It compares various methods to detect drift in embeddings – Euclidean distance, Cosine distance, Domain classifier, Share of drifted components, and Maximum mean discrepancy (MMD) – and evaluates each method against such criteria as computational speed, thresholds, PCA, and embedding model behavior. + +Domain classifier can be a good default: it is comparably fast, PCA-agnostic, agnostic to the embedding model, and easy to interpret. However, we suggest you check out the blog to choose the right drift detection method for your specific use case. Here is a blog summary sneak peek: + +![](<../../../images/2023109\_course\_module3.039-min.png>) + +## Summing up + +We discussed different strategies for monitoring embedding drift, including distance metrics, model-based drift detection, and share of drifted components. While using a domain classifier to detect embedding drift is a good default strategy, we suggest you evaluate other methods to choose the right one for your use case. + +Further reading: [Shift happens: we compared 5 methods to detect drift in ML embeddings](https://www.evidentlyai.com/blog/embedding-drift-detection) + +Up-next: code practice! We will apply ML monitoring strategies for unstructured data on real data to derive actionable metrics. diff --git a/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-multimodal-datasets.md b/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-multimodal-datasets.md new file mode 100644 index 0000000..ea389db --- /dev/null +++ b/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-multimodal-datasets.md @@ -0,0 +1,51 @@ +# 3.6. Monitoring multimodal datasets + +{% embed url="https://www.youtube.com/watch?v=b0a1iMlHgEs&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=20" %} + +**Video 6**. [Monitoring multimodal datasets](https://www.youtube.com/watch?v=b0a1iMlHgEs&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=20), by Emeli Dral + +## What is a multimodal dataset? + +Often, we don't only work with structured or unstructured data but a combination of both. Some common examples include product reviews, chats, support tickets, and emails. These applications may include unstructured data, e.g., text, and structured metadata like the region, device, product type, user category, etc. + +Both structured and unstructured data provide valuable signals. Considering signals from both data types is essential to build comprehensive ML models. + +![](<../../../images/2023109\_course\_module3.044-min.png>) + +## Monitoring strategies for multi-modal data + +We will cover three widely used strategies for monitoring multi-modal datasets. + +**Strategy 1. Split and monitor independently.** +The approach is straightforward – split the dataset by data type and monitor structured and unstructured data independently: +* Monitor structured data using descriptive statistics, share of missing values, distribution drift, correlation changes, etc. +* Use raw text data analysis or embedding monitoring for unstructured data. +* Combine monitoring results into a unified dashboard. + +![](<../../../images/2023109\_course\_module3.046-min.png>) + +**Strategy 2. A joint structured dataset.** +This approach is based on turning unstructured data into structured by using descriptors: +* Generate descriptors for unstructured data (e.g., text properties) to represent it in a structured form. +* Combine these structured descriptors with existing metadata. +* Perform a comprehensive analysis of the combined structured data. You can check for missing values, distribution drift, correlation changes, outliers, etc. + +![](<../../../images/2023109\_course\_module3.047-min.png>) + +**Strategy 3. Generate embeddings.** +As embeddings represent data as vectors in high-dimensional space, you can combine structured features with embeddings to create an expanded feature space. For instance, if you have 64 embeddings and three structured features, the combined space would be 67-dimensional. You can then apply various methods like share of drifted components, domain classifier, or distance-based metrics to this combined data. + +![](<../../../images/2023109\_course\_module3.048-min.png>) + +## Summing up + +We discussed three strategies for monitoring data quality and data drift in multi-modal datasets. This concludes our module on ML monitoring for unstructured data. Here are some considerations to keep in mind: +* **If you have access to raw text data, do not ignore it.** Interpretability wins! Evaluating metrics on raw text can provide a deep understanding of changes and potential issues with text data. +* **If working with embeddings,** numerous methods are also available to detect drift. +* **When dealing with multimodal datasets,** you can split data by type, leverage text descriptors, or generate a joint embedding dataset, depending on the specific use case and available data. + +## Enjoyed the content? + +Star Evidently on GitHub to contribute back! This helps us create free, open-source tools and content for the community. + +⭐️ [Star](https://github.com/evidentlyai/evidently) on GitHub! diff --git a/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-text-data-code-practice.md b/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-text-data-code-practice.md new file mode 100644 index 0000000..18e599e --- /dev/null +++ b/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-text-data-code-practice.md @@ -0,0 +1,20 @@ +# 3.5. Monitoring text data [CODE PRACTICE] + +{% embed url="https://www.youtube.com/watch?v=RIultWCjYXo&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=19" %} + +**Video 5**. [Monitoring text data [CODE PRACTICE]](https://www.youtube.com/watch?v=RIultWCjYXo&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=19), by Emeli Dral + +In this video, we walk you through the code example of evaluations for unstructured data using the open-source [Evidently](https://github.com/evidentlyai/evidently) Python library. + +**Want to go straight to code?** Here is the [example notebook](https://github.com/evidentlyai/ml_observability_course/blob/main/module3/unstructured_data_code_practice.ipynb) to follow along. + +**Outline:**\ +[00:00](https://youtu.be/RIultWCjYXo?si=5s0_-fMduGKorqci) Import libraries and datasets \ +[02:26](https://youtu.be/RIultWCjYXo?si=Vyrnq26avImqSUB6&t=146) Prepare a multimodal dataset with raw text \ +[08:15](https://youtu.be/RIultWCjYXo?si=hKrfvOBPZ3kFeisC&t=495) Text data overview report \ +[11:18](https://youtu.be/RIultWCjYXo?si=nws_RxLC2YsoiD1C&t=678) Model-based text data drift detection \ +[13:49](https://youtu.be/RIultWCjYXo?si=WlLgpbHHt2Bi-UIH&t=829) Adding custom text descriptors to report \ +[17:50](https://youtu.be/RIultWCjYXo?si=9KWXxjYqW4eaE97n&t=1070) Custom report with descriptor drift detection \ +[23:55](https://youtu.be/RIultWCjYXo?si=9lNBhLuipZrDK8zi&t=1435) Embedding drift detection + +That’s it! We covered three use cases for evaluating unstructured data – multimodal data with raw text, detecting drift in text data with descriptors, and embedding drift detection – and learned to derive actionable metrics. diff --git a/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/readme.md b/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/readme.md new file mode 100644 index 0000000..271fa8d --- /dev/null +++ b/docs/book/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/readme.md @@ -0,0 +1,16 @@ +--- +description: Monitoring NLP, LLM and embeddings. +--- + +# Module 3: ML monitoring for unstructured data + +This module covers evaluating and monitoring the production performance for models that use unstructured data, including LLM-based systems. + +We will cover: +* Why monitoring unstructured data is difficult; +* How to measure text data quality; +* What are text descriptors and how to use them; +* How to deal with embeddings; +* How to deal with multimodal data. + +This module includes both a **theoretical part and code practice**. At the end of this module, you will understand the possible approaches to monitoring ML models that work with texts and other unstructured data. diff --git a/docs/images/2023109_course_module1_fin_images.005-min.png b/docs/images/2023109_course_module1_fin_images.005-min.png new file mode 100644 index 0000000..a2e44d0 Binary files /dev/null and b/docs/images/2023109_course_module1_fin_images.005-min.png differ diff --git a/docs/images/2023109_course_module1_fin_images.005.png b/docs/images/2023109_course_module1_fin_images.005.png deleted file mode 100644 index 4d12db4..0000000 Binary files a/docs/images/2023109_course_module1_fin_images.005.png and /dev/null differ diff --git a/docs/images/2023109_course_module1_fin_images.008-min.png b/docs/images/2023109_course_module1_fin_images.008-min.png new file mode 100644 index 0000000..c198e9d Binary files /dev/null and b/docs/images/2023109_course_module1_fin_images.008-min.png differ diff --git a/docs/images/2023109_course_module1_fin_images.008.png b/docs/images/2023109_course_module1_fin_images.008.png deleted file mode 100644 index 615c721..0000000 Binary files a/docs/images/2023109_course_module1_fin_images.008.png and /dev/null differ diff --git a/docs/images/2023109_course_module1_fin_images.011-min.png b/docs/images/2023109_course_module1_fin_images.011-min.png new file mode 100644 index 0000000..a29dc42 Binary files /dev/null and b/docs/images/2023109_course_module1_fin_images.011-min.png differ diff --git a/docs/images/2023109_course_module1_fin_images.011.png b/docs/images/2023109_course_module1_fin_images.011.png deleted file mode 100644 index ddc7b52..0000000 Binary files a/docs/images/2023109_course_module1_fin_images.011.png and /dev/null differ diff --git a/docs/images/2023109_course_module1_fin_images.012-min.png b/docs/images/2023109_course_module1_fin_images.012-min.png new file mode 100644 index 0000000..37c04ae Binary files /dev/null and b/docs/images/2023109_course_module1_fin_images.012-min.png differ diff --git a/docs/images/2023109_course_module1_fin_images.012.png b/docs/images/2023109_course_module1_fin_images.012.png deleted file mode 100644 index e2fd6d3..0000000 Binary files a/docs/images/2023109_course_module1_fin_images.012.png and /dev/null differ diff --git a/docs/images/2023109_course_module1_fin_images.014-min.png b/docs/images/2023109_course_module1_fin_images.014-min.png new file mode 100644 index 0000000..16299b7 Binary files /dev/null and b/docs/images/2023109_course_module1_fin_images.014-min.png differ diff --git a/docs/images/2023109_course_module1_fin_images.015-min.png b/docs/images/2023109_course_module1_fin_images.015-min.png new file mode 100644 index 0000000..72df0b9 Binary files /dev/null and b/docs/images/2023109_course_module1_fin_images.015-min.png differ diff --git a/docs/images/2023109_course_module1_fin_images.015.png b/docs/images/2023109_course_module1_fin_images.015.png deleted file mode 100644 index 56e9507..0000000 Binary files a/docs/images/2023109_course_module1_fin_images.015.png and /dev/null differ diff --git a/docs/images/2023109_course_module1_fin_images.016-min.png b/docs/images/2023109_course_module1_fin_images.016-min.png new file mode 100644 index 0000000..bdf3d9a Binary files /dev/null and b/docs/images/2023109_course_module1_fin_images.016-min.png differ diff --git a/docs/images/2023109_course_module1_fin_images.016.png b/docs/images/2023109_course_module1_fin_images.016.png deleted file mode 100644 index 7da4bb8..0000000 Binary files a/docs/images/2023109_course_module1_fin_images.016.png and /dev/null differ diff --git a/docs/images/2023109_course_module1_fin_images.024-min.png b/docs/images/2023109_course_module1_fin_images.024-min.png new file mode 100644 index 0000000..05c77b6 Binary files /dev/null and b/docs/images/2023109_course_module1_fin_images.024-min.png differ diff --git a/docs/images/2023109_course_module1_fin_images.024.png b/docs/images/2023109_course_module1_fin_images.024.png deleted file mode 100644 index 27a13db..0000000 Binary files a/docs/images/2023109_course_module1_fin_images.024.png and /dev/null differ diff --git a/docs/images/2023109_course_module1_fin_images.028-min.png b/docs/images/2023109_course_module1_fin_images.028-min.png new file mode 100644 index 0000000..bff8fcb Binary files /dev/null and b/docs/images/2023109_course_module1_fin_images.028-min.png differ diff --git a/docs/images/2023109_course_module1_fin_images.030-min.png b/docs/images/2023109_course_module1_fin_images.030-min.png new file mode 100644 index 0000000..382b72e Binary files /dev/null and b/docs/images/2023109_course_module1_fin_images.030-min.png differ diff --git a/docs/images/2023109_course_module1_fin_images.030.png b/docs/images/2023109_course_module1_fin_images.030.png deleted file mode 100644 index fc0b228..0000000 Binary files a/docs/images/2023109_course_module1_fin_images.030.png and /dev/null differ diff --git a/docs/images/2023109_course_module1_fin_images.031-min.png b/docs/images/2023109_course_module1_fin_images.031-min.png new file mode 100644 index 0000000..1825a5b Binary files /dev/null and b/docs/images/2023109_course_module1_fin_images.031-min.png differ diff --git a/docs/images/2023109_course_module1_fin_images.031.png b/docs/images/2023109_course_module1_fin_images.031.png deleted file mode 100644 index 1266d83..0000000 Binary files a/docs/images/2023109_course_module1_fin_images.031.png and /dev/null differ diff --git a/docs/images/2023109_course_module1_fin_images.034-min.png b/docs/images/2023109_course_module1_fin_images.034-min.png new file mode 100644 index 0000000..a7546a1 Binary files /dev/null and b/docs/images/2023109_course_module1_fin_images.034-min.png differ diff --git a/docs/images/2023109_course_module1_fin_images.034.png b/docs/images/2023109_course_module1_fin_images.034.png deleted file mode 100644 index ec69872..0000000 Binary files a/docs/images/2023109_course_module1_fin_images.034.png and /dev/null differ diff --git a/docs/images/2023109_course_module1_fin_images.050-min.png b/docs/images/2023109_course_module1_fin_images.050-min.png new file mode 100644 index 0000000..f9eeba2 Binary files /dev/null and b/docs/images/2023109_course_module1_fin_images.050-min.png differ diff --git a/docs/images/2023109_course_module1_fin_images.050.png b/docs/images/2023109_course_module1_fin_images.050.png deleted file mode 100644 index 336407e..0000000 Binary files a/docs/images/2023109_course_module1_fin_images.050.png and /dev/null differ diff --git a/docs/images/2023109_course_module1_fin_images.052-min.png b/docs/images/2023109_course_module1_fin_images.052-min.png new file mode 100644 index 0000000..c78631b Binary files /dev/null and b/docs/images/2023109_course_module1_fin_images.052-min.png differ diff --git a/docs/images/2023109_course_module1_fin_images.052.png b/docs/images/2023109_course_module1_fin_images.052.png deleted file mode 100644 index f747fdf..0000000 Binary files a/docs/images/2023109_course_module1_fin_images.052.png and /dev/null differ diff --git a/docs/images/2023109_course_module1_fin_images.061-min.png b/docs/images/2023109_course_module1_fin_images.061-min.png new file mode 100644 index 0000000..c8ec10a Binary files /dev/null and b/docs/images/2023109_course_module1_fin_images.061-min.png differ diff --git a/docs/images/2023109_course_module1_fin_images.061.png b/docs/images/2023109_course_module1_fin_images.061.png deleted file mode 100644 index 2878351..0000000 Binary files a/docs/images/2023109_course_module1_fin_images.061.png and /dev/null differ diff --git a/docs/images/2023109_course_module1_fin_images.065-min.png b/docs/images/2023109_course_module1_fin_images.065-min.png new file mode 100644 index 0000000..ef8292f Binary files /dev/null and b/docs/images/2023109_course_module1_fin_images.065-min.png differ diff --git a/docs/images/2023109_course_module1_fin_images.065.png b/docs/images/2023109_course_module1_fin_images.065.png deleted file mode 100644 index 1cd59b7..0000000 Binary files a/docs/images/2023109_course_module1_fin_images.065.png and /dev/null differ diff --git a/docs/images/2023109_course_module1_fin_images.066-min.png b/docs/images/2023109_course_module1_fin_images.066-min.png new file mode 100644 index 0000000..c251cda Binary files /dev/null and b/docs/images/2023109_course_module1_fin_images.066-min.png differ diff --git a/docs/images/2023109_course_module1_fin_images.066.png b/docs/images/2023109_course_module1_fin_images.066.png deleted file mode 100644 index 7f6bde4..0000000 Binary files a/docs/images/2023109_course_module1_fin_images.066.png and /dev/null differ diff --git a/docs/images/2023109_course_module2.005-min.png b/docs/images/2023109_course_module2.005-min.png new file mode 100644 index 0000000..535ca4b Binary files /dev/null and b/docs/images/2023109_course_module2.005-min.png differ diff --git a/docs/images/2023109_course_module2.005.png b/docs/images/2023109_course_module2.005.png deleted file mode 100644 index d193652..0000000 Binary files a/docs/images/2023109_course_module2.005.png and /dev/null differ diff --git a/docs/images/2023109_course_module2.006-min.png b/docs/images/2023109_course_module2.006-min.png new file mode 100644 index 0000000..a209f78 Binary files /dev/null and b/docs/images/2023109_course_module2.006-min.png differ diff --git a/docs/images/2023109_course_module2.006.png b/docs/images/2023109_course_module2.006.png deleted file mode 100644 index b6ac49b..0000000 Binary files a/docs/images/2023109_course_module2.006.png and /dev/null differ diff --git a/docs/images/2023109_course_module2.007-min.png b/docs/images/2023109_course_module2.007-min.png new file mode 100644 index 0000000..bfe31ba Binary files /dev/null and b/docs/images/2023109_course_module2.007-min.png differ diff --git a/docs/images/2023109_course_module2.007.png b/docs/images/2023109_course_module2.007.png deleted file mode 100644 index 090d036..0000000 Binary files a/docs/images/2023109_course_module2.007.png and /dev/null differ diff --git a/docs/images/2023109_course_module2.009-min.png b/docs/images/2023109_course_module2.009-min.png new file mode 100644 index 0000000..691ddc7 Binary files /dev/null and b/docs/images/2023109_course_module2.009-min.png differ diff --git a/docs/images/2023109_course_module2.009.png b/docs/images/2023109_course_module2.009.png deleted file mode 100644 index 0c771a1..0000000 Binary files a/docs/images/2023109_course_module2.009.png and /dev/null differ diff --git a/docs/images/2023109_course_module2.012-min.png b/docs/images/2023109_course_module2.012-min.png new file mode 100644 index 0000000..e4b8c20 Binary files /dev/null and b/docs/images/2023109_course_module2.012-min.png differ diff --git a/docs/images/2023109_course_module2.012.png b/docs/images/2023109_course_module2.012.png deleted file mode 100644 index fdddf66..0000000 Binary files a/docs/images/2023109_course_module2.012.png and /dev/null differ diff --git a/docs/images/2023109_course_module2.016-min.png b/docs/images/2023109_course_module2.016-min.png new file mode 100644 index 0000000..b3757cd Binary files /dev/null and b/docs/images/2023109_course_module2.016-min.png differ diff --git a/docs/images/2023109_course_module2.016.png b/docs/images/2023109_course_module2.016.png deleted file mode 100644 index 05cadba..0000000 Binary files a/docs/images/2023109_course_module2.016.png and /dev/null differ diff --git a/docs/images/2023109_course_module2.020-min.png b/docs/images/2023109_course_module2.020-min.png new file mode 100644 index 0000000..ccfb475 Binary files /dev/null and b/docs/images/2023109_course_module2.020-min.png differ diff --git a/docs/images/2023109_course_module2.020.png b/docs/images/2023109_course_module2.020.png deleted file mode 100644 index a6a57ad..0000000 Binary files a/docs/images/2023109_course_module2.020.png and /dev/null differ diff --git a/docs/images/2023109_course_module2.025-min.png b/docs/images/2023109_course_module2.025-min.png new file mode 100644 index 0000000..abe416f Binary files /dev/null and b/docs/images/2023109_course_module2.025-min.png differ diff --git a/docs/images/2023109_course_module2.025.png b/docs/images/2023109_course_module2.025.png deleted file mode 100644 index 48ba4da..0000000 Binary files a/docs/images/2023109_course_module2.025.png and /dev/null differ diff --git a/docs/images/2023109_course_module2.028-min.png b/docs/images/2023109_course_module2.028-min.png new file mode 100644 index 0000000..a1a18b7 Binary files /dev/null and b/docs/images/2023109_course_module2.028-min.png differ diff --git a/docs/images/2023109_course_module2.028.png b/docs/images/2023109_course_module2.028.png deleted file mode 100644 index 6986f8f..0000000 Binary files a/docs/images/2023109_course_module2.028.png and /dev/null differ diff --git a/docs/images/2023109_course_module2.041-min.png b/docs/images/2023109_course_module2.041-min.png new file mode 100644 index 0000000..4032751 Binary files /dev/null and b/docs/images/2023109_course_module2.041-min.png differ diff --git a/docs/images/2023109_course_module2.041.png b/docs/images/2023109_course_module2.041.png deleted file mode 100644 index 0afed0a..0000000 Binary files a/docs/images/2023109_course_module2.041.png and /dev/null differ diff --git a/docs/images/2023109_course_module2.047-min.png b/docs/images/2023109_course_module2.047-min.png new file mode 100644 index 0000000..d2953dd Binary files /dev/null and b/docs/images/2023109_course_module2.047-min.png differ diff --git a/docs/images/2023109_course_module2.047.png b/docs/images/2023109_course_module2.047.png deleted file mode 100644 index 55033cd..0000000 Binary files a/docs/images/2023109_course_module2.047.png and /dev/null differ diff --git a/docs/images/2023109_course_module2.058-min.png b/docs/images/2023109_course_module2.058-min.png new file mode 100644 index 0000000..e097574 Binary files /dev/null and b/docs/images/2023109_course_module2.058-min.png differ diff --git a/docs/images/2023109_course_module2.058.png b/docs/images/2023109_course_module2.058.png deleted file mode 100644 index ab9d337..0000000 Binary files a/docs/images/2023109_course_module2.058.png and /dev/null differ diff --git a/docs/images/2023109_course_module2.060-min.png b/docs/images/2023109_course_module2.060-min.png new file mode 100644 index 0000000..11b7269 Binary files /dev/null and b/docs/images/2023109_course_module2.060-min.png differ diff --git a/docs/images/2023109_course_module2.060.png b/docs/images/2023109_course_module2.060.png deleted file mode 100644 index 1800100..0000000 Binary files a/docs/images/2023109_course_module2.060.png and /dev/null differ diff --git a/docs/images/2023109_course_module2.065-min.png b/docs/images/2023109_course_module2.065-min.png new file mode 100644 index 0000000..adf6edb Binary files /dev/null and b/docs/images/2023109_course_module2.065-min.png differ diff --git a/docs/images/2023109_course_module2.065.png b/docs/images/2023109_course_module2.065.png deleted file mode 100644 index 4c0eefb..0000000 Binary files a/docs/images/2023109_course_module2.065.png and /dev/null differ diff --git a/docs/images/2023109_course_module2.070-min.png b/docs/images/2023109_course_module2.070-min.png new file mode 100644 index 0000000..526ee28 Binary files /dev/null and b/docs/images/2023109_course_module2.070-min.png differ diff --git a/docs/images/2023109_course_module2.070.png b/docs/images/2023109_course_module2.070.png deleted file mode 100644 index 9fc8fc0..0000000 Binary files a/docs/images/2023109_course_module2.070.png and /dev/null differ diff --git a/docs/images/2023109_course_module2.071-min.png b/docs/images/2023109_course_module2.071-min.png new file mode 100644 index 0000000..866f4f6 Binary files /dev/null and b/docs/images/2023109_course_module2.071-min.png differ diff --git a/docs/images/2023109_course_module2.071.png b/docs/images/2023109_course_module2.071.png deleted file mode 100644 index c2e3ed5..0000000 Binary files a/docs/images/2023109_course_module2.071.png and /dev/null differ diff --git a/docs/images/2023109_course_module2.081-min.png b/docs/images/2023109_course_module2.081-min.png new file mode 100644 index 0000000..129ef10 Binary files /dev/null and b/docs/images/2023109_course_module2.081-min.png differ diff --git a/docs/images/2023109_course_module2.082-min.png b/docs/images/2023109_course_module2.082-min.png new file mode 100644 index 0000000..13c86cb Binary files /dev/null and b/docs/images/2023109_course_module2.082-min.png differ diff --git a/docs/images/2023109_course_module2.083-min.png b/docs/images/2023109_course_module2.083-min.png new file mode 100644 index 0000000..eddce83 Binary files /dev/null and b/docs/images/2023109_course_module2.083-min.png differ diff --git a/docs/images/2023109_course_module2.089-min.png b/docs/images/2023109_course_module2.089-min.png new file mode 100644 index 0000000..5dbbc3d Binary files /dev/null and b/docs/images/2023109_course_module2.089-min.png differ diff --git a/docs/images/2023109_course_module2.090-min.png b/docs/images/2023109_course_module2.090-min.png new file mode 100644 index 0000000..00bf5be Binary files /dev/null and b/docs/images/2023109_course_module2.090-min.png differ diff --git a/docs/images/2023109_course_module2.091-min.png b/docs/images/2023109_course_module2.091-min.png new file mode 100644 index 0000000..0bc8a93 Binary files /dev/null and b/docs/images/2023109_course_module2.091-min.png differ diff --git a/docs/images/2023109_course_module2.098-min.png b/docs/images/2023109_course_module2.098-min.png new file mode 100644 index 0000000..80ec18e Binary files /dev/null and b/docs/images/2023109_course_module2.098-min.png differ diff --git a/docs/images/2023109_course_module2.100-min.png b/docs/images/2023109_course_module2.100-min.png new file mode 100644 index 0000000..a39fe07 Binary files /dev/null and b/docs/images/2023109_course_module2.100-min.png differ diff --git a/docs/images/2023109_course_module2.101-min.png b/docs/images/2023109_course_module2.101-min.png new file mode 100644 index 0000000..af8c50c Binary files /dev/null and b/docs/images/2023109_course_module2.101-min.png differ diff --git a/docs/images/2023109_course_module2.102-min.png b/docs/images/2023109_course_module2.102-min.png new file mode 100644 index 0000000..065d6d8 Binary files /dev/null and b/docs/images/2023109_course_module2.102-min.png differ diff --git a/docs/images/2023109_course_module3.003-min.png b/docs/images/2023109_course_module3.003-min.png new file mode 100644 index 0000000..c1a1d2a Binary files /dev/null and b/docs/images/2023109_course_module3.003-min.png differ diff --git a/docs/images/2023109_course_module3.004-min.png b/docs/images/2023109_course_module3.004-min.png new file mode 100644 index 0000000..296e7fb Binary files /dev/null and b/docs/images/2023109_course_module3.004-min.png differ diff --git a/docs/images/2023109_course_module3.005-min.png b/docs/images/2023109_course_module3.005-min.png new file mode 100644 index 0000000..46a0c27 Binary files /dev/null and b/docs/images/2023109_course_module3.005-min.png differ diff --git a/docs/images/2023109_course_module3.006-min.png b/docs/images/2023109_course_module3.006-min.png new file mode 100644 index 0000000..697418e Binary files /dev/null and b/docs/images/2023109_course_module3.006-min.png differ diff --git a/docs/images/2023109_course_module3.007-min.png b/docs/images/2023109_course_module3.007-min.png new file mode 100644 index 0000000..6d8e799 Binary files /dev/null and b/docs/images/2023109_course_module3.007-min.png differ diff --git a/docs/images/2023109_course_module3.011-min.png b/docs/images/2023109_course_module3.011-min.png new file mode 100644 index 0000000..1d44368 Binary files /dev/null and b/docs/images/2023109_course_module3.011-min.png differ diff --git a/docs/images/2023109_course_module3.013-min.png b/docs/images/2023109_course_module3.013-min.png new file mode 100644 index 0000000..acd6509 Binary files /dev/null and b/docs/images/2023109_course_module3.013-min.png differ diff --git a/docs/images/2023109_course_module3.014-min.png b/docs/images/2023109_course_module3.014-min.png new file mode 100644 index 0000000..0330a5a Binary files /dev/null and b/docs/images/2023109_course_module3.014-min.png differ diff --git a/docs/images/2023109_course_module3.018-min.png b/docs/images/2023109_course_module3.018-min.png new file mode 100644 index 0000000..5aa4676 Binary files /dev/null and b/docs/images/2023109_course_module3.018-min.png differ diff --git a/docs/images/2023109_course_module3.024-min.png b/docs/images/2023109_course_module3.024-min.png new file mode 100644 index 0000000..29b8647 Binary files /dev/null and b/docs/images/2023109_course_module3.024-min.png differ diff --git a/docs/images/2023109_course_module3.025-min.png b/docs/images/2023109_course_module3.025-min.png new file mode 100644 index 0000000..41e30a5 Binary files /dev/null and b/docs/images/2023109_course_module3.025-min.png differ diff --git a/docs/images/2023109_course_module3.026-min.png b/docs/images/2023109_course_module3.026-min.png new file mode 100644 index 0000000..d195688 Binary files /dev/null and b/docs/images/2023109_course_module3.026-min.png differ diff --git a/docs/images/2023109_course_module3.029-min.png b/docs/images/2023109_course_module3.029-min.png new file mode 100644 index 0000000..e999dc2 Binary files /dev/null and b/docs/images/2023109_course_module3.029-min.png differ diff --git a/docs/images/2023109_course_module3.032-min.png b/docs/images/2023109_course_module3.032-min.png new file mode 100644 index 0000000..ff980de Binary files /dev/null and b/docs/images/2023109_course_module3.032-min.png differ diff --git a/docs/images/2023109_course_module3.035-min.png b/docs/images/2023109_course_module3.035-min.png new file mode 100644 index 0000000..a80babd Binary files /dev/null and b/docs/images/2023109_course_module3.035-min.png differ diff --git a/docs/images/2023109_course_module3.036-min.png b/docs/images/2023109_course_module3.036-min.png new file mode 100644 index 0000000..23ad55a Binary files /dev/null and b/docs/images/2023109_course_module3.036-min.png differ diff --git a/docs/images/2023109_course_module3.037-min.png b/docs/images/2023109_course_module3.037-min.png new file mode 100644 index 0000000..da0704f Binary files /dev/null and b/docs/images/2023109_course_module3.037-min.png differ diff --git a/docs/images/2023109_course_module3.039-min.png b/docs/images/2023109_course_module3.039-min.png new file mode 100644 index 0000000..17b2c66 Binary files /dev/null and b/docs/images/2023109_course_module3.039-min.png differ diff --git a/docs/images/2023109_course_module3.044-min.png b/docs/images/2023109_course_module3.044-min.png new file mode 100644 index 0000000..0ec6f16 Binary files /dev/null and b/docs/images/2023109_course_module3.044-min.png differ diff --git a/docs/images/2023109_course_module3.046-min.png b/docs/images/2023109_course_module3.046-min.png new file mode 100644 index 0000000..05eb571 Binary files /dev/null and b/docs/images/2023109_course_module3.046-min.png differ diff --git a/docs/images/2023109_course_module3.047-min.png b/docs/images/2023109_course_module3.047-min.png new file mode 100644 index 0000000..425cb46 Binary files /dev/null and b/docs/images/2023109_course_module3.047-min.png differ diff --git a/docs/images/2023109_course_module3.048-min.png b/docs/images/2023109_course_module3.048-min.png new file mode 100644 index 0000000..f9cb0ca Binary files /dev/null and b/docs/images/2023109_course_module3.048-min.png differ diff --git a/module2/requirements.txt b/module2/requirements.txt new file mode 100644 index 0000000..d1dc440 --- /dev/null +++ b/module2/requirements.txt @@ -0,0 +1,6 @@ +scikit-learn>=0.24.0 +pandas>=1.3.5 +numpy>=1.19.5 +jupyter>=1.0.0 + +evidently==0.4.7 diff --git a/module3/requirements.txt b/module3/requirements.txt new file mode 100644 index 0000000..c09203d --- /dev/null +++ b/module3/requirements.txt @@ -0,0 +1,7 @@ +scikit-learn>=0.24.0 +pandas>=1.3.5 +numpy>=1.19.5 +nltk>=3.6.7 +jupyter>=1.0.0 + +evidently==0.4.7