up:: [[DBA806 - Applied AI Innovation]] tags:: #source/course #on/AI #on/ML people:: [[Praphul Chandra]] # DBA806 M10 - Data Flywheel and CT [Session 10 - Dr Praphul Chandra - AI Paradigms Models, APIs and Agents - YouTube](https://www.youtube.com/watch?v=-vtikHUYjsY) [Slides: DataFlywheelAndContinuousTraining.pdf](https://cdn.upgrad.com/uploads/production/d4ffa2fa-8d97-4f63-917d-07e54a3f7cab/DataFlywheelAndContinuousTraining.pdf) The presentation provides a comprehensive overview of the practical considerations involved in deploying and managing AI/ML solutions in real-world settings, covering various aspects from *data management to post-deployment operations*. The text discusses the importance of continued monitoring and measurement of AI models post-deployment to ensure sustained performance. It emphasizes the need for ongoing evaluation and adjustment, rather than simply relying on initial success. Here are the key points summarized: - **Post-deployment monitoring**: After deploying an AI model, it's crucial to continue *measuring its performance over time* to ensure it remains effective. - **Spot checking predictions**: The common approach involves periodically checking some predictions made by the model, but this isn't comprehensive due to the scale of predictions made daily. - **Reactive problem-solving**: Often, issues with the model's performance are only addressed reactively, after problems arise, leading to potential revenue loss or customer complaints. - **Contrast with traditional software**: Unlike traditional software, AI/ML models require *ongoing attention* due to the dynamic nature of data and the model's performance. - **Expectation of model decay**: Models are expected to degrade in performance over time due to *changing data distributions* and other factors. - **Continuous learning**: Recognizing that models need to adapt, the focus shifts towards continual monitoring, retraining, and adaptation. - **Techniques for monitoring**: Third-party tools for detecting [[Model Drift]], small-scale pilots, and user feedback mechanisms are discussed as ways to monitor model performance. - **Addressing model degradation**: Rather than solely reacting to complaints, the text suggests proactive strategies, such as continuous monitoring and retraining, to address potential issues. ![[Pasted image 20240401004859.png]] The text stresses the need for a proactive approach to AI model management, advocating for continuous monitoring, retraining, and adaptation to ensure sustained performance in dynamic environments. The common practice in deploying machine learning models are discussed, highlighting its shortcomings and proposing the concept of a "[[Data Flywheel]]" as a solution. It emphasizes the repetitive nature of the deployment process, where issues persist despite fixes, resulting in business impacts such as downtime, revenue loss, and increased costs. The manual intervention required during such incidents contributes to inefficiencies. It involves continuously retraining the model using new data to adapt to evolving patterns and maintain optimal performance. The process of *retraining is outlined in six steps*, starting with determining the data to retrain on and involving human labeling of data points. **Key Points:** - Deploying machine learning models often leads to a repetitive cycle of fixing issues without addressing the underlying problem. - Business impacts, including downtime, revenue loss, and increased costs, occur when models fail to perform effectively. - The manual steps involved in fixing model issues contribute to inefficiencies in the deployment process. - The "data flywheel" concept suggests continuously retraining models using new data to adapt to changing patterns. - Retraining involves six steps, including selecting the data to retrain on and human labeling of data points. Challenges associated with retraining models periodically are discussed, such as the potential for generating excessive data that cannot be logged and the limitations of uniform sampling methods. Additionally, the cost of labeling data and the expense of retraining large models are highlighted. **Key Points:** - Challenges include managing excessive data generation, limitations of uniform sampling, and the cost of labeling data. - Retraining large models can be expensive, and justification for the frequency of retraining is necessary to optimize resources. The text discusses various aspects of training machine learning models, particularly focusing on the considerations for retraining models periodically. It emphasizes that not all models are deep-layered neural networks and that models are often treated as black boxes in practical applications. The discussion highlights the importance of data diversity and size in training models, as well as the relevance of historical data in predicting future outcomes. The decision to train on a specific time window of data *depends on the context and the rate of change in the environment the model operates* in, such as considering events like the COVID-19 pandemic. **Key Points:** - Models are often treated as black boxes, irrespective of their internal structure, in practical applications. - The decision to train a model on a specific time window of data depends on factors like data diversity, size, and the rate of change in the environment. - Historical data's relevance in predicting future outcomes varies depending on the context, such as significant events like the COVID-19 pandemic. - Models should ideally be retrained when needed rather than on a fixed schedule, to adapt to changing conditions effectively. - Metrics for determining the need for retraining include user feedback, model performance metrics, proxy metrics from user behavior, data quality checks, system metrics (e.g., CPU usage), and distribution shift analysis. - Distribution shift, where the new data in production significantly differs from the data the model was trained on, is a common reason for model performance deterioration. The text also addresses questions and discussions related to retraining models, such as the feasibility of retraining in scenarios like fraud detection and the potential risks associated with periodic retraining. It suggests an intelligent approach to retraining based on defined metrics and the need to adapt models to changing conditions effectively. **Key Points:** - Retraining models based on defined metrics and adapting to changing conditions is essential for maintaining model performance. - Feasibility of retraining depends on the specific domain and the availability of labeled data. - Risks associated with periodic retraining include introducing new risks with new models and the potential for model performance deterioration due to distribution shifts. - An adaptive approach to retraining, based on defined metrics and monitoring, is recommended to ensure optimal model performance over time. The participants explore various aspects of model retraining and optimization for machine learning models. Here are the key points: - **Model Optimization:** The conversation begins with a question about optimizing data requirements for a model. The participants discuss the concept of finding the minimum data requirement to achieve a generalized model. - **Active Learning:** They delve into Active Learning, a field of machine learning focused on selecting subsets of data to maximize model performance with minimal data usage. This involves choosing data subsets that yield the same performance as using the entire dataset. - **Model Adaptation to Data Changes:** The discussion moves on to real-life scenarios where data distribution shifts occur, affecting model performance. They discuss the importance of models adapting to changes in data over time. - **Quality Assurance for Retrained Models:** The participants discuss ensuring the quality of retrained models, emphasizing the importance of testing new models before deployment using offline testing on hold-out data. - **Types of Distribution Shifts:** Different types of distribution shifts are highlighted, including instantaneous, gradual, periodic, and temporary shifts. These shifts affect model performance and necessitate retraining. - **Monitoring Model Performance:** Various metrics are discussed for monitoring model performance, such as system metrics, model outcomes, and user feedback. These metrics help determine when to retrain the model. - **Logging and Data Curation:** The process of logging relevant data points and curating datasets for model retraining is explained. Curation involves selecting data points based on their relevance and accuracy. - **Retraining Triggers and Deployment:** Triggers for retraining models are identified based on collected data points, and the process of retraining the model is discussed. Once retrained, the model undergoes offline testing before deployment. - **Continuous Monitoring and Optimization:** The conversation emphasizes the cyclical nature of model optimization, where models are continuously monitored online for performance metrics. If certain conditions are met, the process of retraining and deployment is repeated. - **Real-Life Example:** A hypothetical scenario involving a social media app like Facebook is requested to illustrate the concepts discussed. The logging process would involve capturing user interactions and behaviors, while data curation would select relevant data points for model optimization. The text discusses the importance of continual learning and retraining in machine learning systems, using the example of an e-commerce platform like Amazon and its recommendation engine. It emphasizes the need to measure outcomes such as average price point per customer checkout and recommendation accuracy, alongside monitoring metrics like data quality and distribution shift. The process involves logging data, curating datasets, offline and online testing, and retraining triggers based on model performance. **Key points:** - Continuous learning in machine learning systems is crucial for adapting to changes and improving performance over time. - Metrics such as average price point per customer checkout and recommendation accuracy are vital for measuring outcomes. - Monitoring data quality and distribution shift helps ensure the reliability of the model. - The process involves logging data, curating datasets, testing offline and online, and retraining triggers based on performance. The text also addresses questions and discussions regarding the potential *limitations of continual learning*, such as scenarios where retraining may not be sufficient due to distribution shift or declining model performance. It highlights the need to determine when a model becomes obsolete and requires a new approach. **Key points:** - There are scenarios where continual retraining may not be sufficient, necessitating a reassessment of the model's viability. - Distribution shifts and declining model performance are factors that may indicate the need for a new model. - Continuous monitoring and evaluation are essential to identify when a model needs to be updated or replaced. Furthermore, the text discusses the concept of the "flywheel effect" in machine learning, where improved models lead to better user experiences, attracting more users and generating more data for further improvement. This positive feedback loop can lead to significant advantages for platforms like e-commerce websites. **Key points:** - The "flywheel effect" describes how improved models lead to better user experiences, attracting more users and generating more data for further improvement. - Platforms like e-commerce websites leverage this effect to gain a competitive advantage and capture a larger market share. - Continuous learning and improvement are key components of maintaining this positive feedback loop and staying competitive in the market. The text discusses various dimensions to consider when evaluating machine learning models beyond just model accuracy. Here are the key points summarized: - **Model Performance Dimensions:** - Model accuracy is important but not the only factor to consider. - Other dimensions include prediction speed and model explainability. - The choice of the best model depends on specific constraints and requirements. - **Performance Metrics:** - The most common metric is validation score, indicating performance on hold-out test data. - However, prediction speed is also crucial, especially for real-time applications. - Explainability matters, as simpler models with fewer features may be preferable for interpretability. - **Domain-specific Considerations:** - The importance of accuracy vs. speed vs. explainability varies based on the application domain. - For tasks like medical diagnosis, accuracy may be paramount, while for real-time recommendations, speed might be more critical. - **Model Versioning:** - Treating machine learning models as software products, version control becomes crucial. - Versioning allows tracking changes, parameters, datasets, and performance metrics across different iterations. - Tools like MLflow aid in maintaining a repository of model versions, akin to version control systems like GitHub. - **Experimentation and Deployment Workflow:** - Experimentation is ongoing, with data scientists continually trying out different models. - Deployment engineers ensure stability in production systems, requiring coordination with data scientists. - The deployment process is not linear but an iterative loop involving continual data capture, curation, training, and deployment. - **Cost-Benefit Analysis:** - There's no one-size-fits-all approach; the cost-benefit analysis depends on the specific business context and impact. - No general thumb rules or frameworks exist, necessitating a case-by-case evaluation. - **Data Pipeline Importance:** - Over 80% of the effort in machine learning goes into data-related tasks, emphasizing the importance of data pipelines. - Data engineers play a crucial role in ensuring data quality, transformation, and availability for model training.