up:: [[DBA806 - Applied AI Innovation]] tags:: #source/course #on/AI #on/fintech #on/finance people:: [[Anand Jayaraman]] # DBA806 M6 - AI in Finance The text provides insights into Dr. Anand Jayaraman's role as the Chief Scientist at Soothsayer Analytics and his expertise in *financial markets and algorithmic trading analytics*. Dr. Jayaraman holds a Ph.D. in Physics from the University of Pittsburgh and a B.Tech. in Engineering Physics from IIT Bombay. The focus of the discussion is on AI and machine learning (ML) methods, specifically supervised and unsupervised learning, regression, classification, optimization, linear algebra, and calculus. #### Key Discussions in Lecture [Session 6 - Dr Aand Jayaraman - AI in Finance - YouTube](https://www.youtube.com/watch?v=cW9IS0nUAD0) **Key Points:** - Machine learning involves algorithms that learn from data to make predictions or [[Classification Problems|Classification]]. - [[Supervised Learning]] involves predictions based on labeled data, while [[Unsupervised Learning]] focuses on analyzing patterns and trends in unlabeled data. Duration: The speaker introduces the example of using [[Machine Learning (ML)]] in financial markets to evaluate stock investments. - The speaker presents the traditional method of valuing companies through financial analysis. - The [[P/E Ratio]] (price-to-earnings ratio) is explained as the current stock price divided by earnings per share. - High P/E ratios may indicate an overpriced company, while low ratios may suggest an underpriced opportunity. The speaker introduces rules-based investing, where predefined rules guide stock market decisions based on various metrics. Whereas the goal of machine learning is to *understand the relationship between features and outcomes, allowing predictions for future scenarios*. Learning in machine learning involves identifying patterns from historical data to make predictions. **Highlights** - The speaker discusses the idea of *making machine learning algorithms more general by adding data from multiple companies* instead of focusing solely on Apple. - The approach involves creating a larger data frame by copying the process for other companies like IBM. - The goal is to train the algorithm on a more diverse dataset, leading to a more general rule for stock selection. - There's a question about whether this approach may make the algorithm less specific for individual stocks and potentially less accurate. - The speaker emphasizes the importance of understanding the user's intent – *whether they seek a general rule for stock selection or focus on specific stocks like Apple*. - The possibility of using multiple algorithms and evaluating their performance is discussed for better results. - *A question is raised about using AI to generate new algorithms, and the speaker explains that people are indeed doing it, but there are limits to what can be achieved.* - The discussion touches on the challenge of accuracy and the potential discrepancy between training results and real-world performance. - The speaker briefly addresses the issue of [[Overfitting]], where an algorithm may perform exceptionally well on training data but poorly in production. - The discussion concludes with an overview of supervised learning, emphasizing the role of the supervisor in guiding the algorithm to minimize the loss function and find the best possible function. - The specific type of learning discussed is [[Classification Problems|Classification]], where the *goal is to predict whether a stock is a buy or sell based on the learned function*. **Summary:** The speaker discusses different aspects of [[Machine Learning (ML)]] in the context of financial markets. They start by explaining the two main types of problems in ML: [[Classification Problems|Classification]] and [[Regression]]. Classification involves predicting categories, while regression involves predicting numerical outcomes. The speaker mentions the importance of choosing the right algorithm based on the problem at hand. They then introduce [[Unsupervised Learning]], specifically [[Clustering (ML)]], which aims to identify patterns in data without predefined outcomes. The example given is identifying similar behaviors in financial data to understand market patterns. The speaker emphasizes the *significance of clustering in various business domains, such as customer segmentation in retail*. The discussion touches on the *challenges of unsupervised learning, as it lacks a target to predict*. The speaker addresses questions about changing algorithms and functions in supervised learning and highlights the necessity of trying different algorithms in machine learning. **Applications** The lecture delves into the challenges of applying machine learning to financial markets, *contrasting it with more straightforward problems like predicting mileage per gallon in cars*. The dynamic nature of financial markets is highlighted, pointing out the complexity arising from the diverse and evolving reasons behind stock movements. A comparison is drawn between a regression problem in predicting car mileage based on known features and a similar problem in predicting stock market movements. The speaker explains that while the *former relies on physics-based relationships, the latter involves understanding the ever-changing dynamics influenced by individual beliefs and market conditions*. The discussion shifts towards practical examples, specifically introducing a product called "smart allocator" designed for investment in the stock market. The concept of staying invested for the long term is emphasized, with the S&P 500 Index serving as a benchmark for tracking the performance of top 500 companies. The speaker introduces the concept of the [[60/40 Portfolio]], a standard advice involving 60% investment in stocks (such as S&P 500) and 40% in bonds. The benefits of a balanced portfolio in reducing volatility and providing a smoother investment journey are highlighted. **Key Points:** - Financial markets pose challenges due to the dynamic and diverse reasons behind stock movements. - Contrast between regression problems in predicting car mileage and stock market movements. - The unpredictability of stock markets stems from individual beliefs and changing market conditions. - Long-term investment advice using the S&P 500 Index as a benchmark. - The 60/40 portfolio is a standard recommendation for a balanced investment strategy. **Summary:** The speaker discusses a machine learning (ML) approach used for predicting [[Market Volatility]]. Instead of predicting the stock market's direction (up or down), the *focus is on classifying whether the market will be volatile, normal, or benign* in the next quarter. The problem is framed as a *classification problem*. The predictors considered include macroeconomic indicators, such as treasury bond yields and market demand for bonds. The data used is sourced from the Federal Reserve. The ML algorithm is trained using historical data up to 2010 and is tested by predicting market conditions for subsequent quarters. The algorithm's predictions are used to make investment decisions, adjusting the portfolio's tilt based on the forecast. The portfolio adjustments involve allocating percentages between stocks and bonds depending on the predicted market conditions. The speaker introduces two examples: a *classification problem predicting market conditions* and a *regression problem predicting ideal sector allocations*. The latter involves determining the optimal allocation of assets to various sectors, considering technology, energy, and gold. The approach aims to maximize returns and improve the [[Sharpe Ratio]], resulting in a well-performing portfolio. **Key Points:** 1. Classification problem is used to classify market conditions as volatile, normal, or benign. 2. Predictors include macroeconomic indicators sourced from the Federal Reserve. 3. ML algorithm is trained on historical data up to 2010 and tested for subsequent quarters. 4. Portfolio adjustments are made based on the algorithm's predictions, altering the tilt between stocks and bonds. 5. Two examples are presented: classification problem for overall market conditions and regression problem for sector allocations. 6. Sector allocation considers technology, energy, and gold, aiming to optimize portfolio performance. 7. The ML model is designed for wealth managers managing retirement accounts, targeting investors with a risk-tolerant profile. 8. The portfolio adjustments are made quarterly, not through day trading. 9. Consideration of *geopolitical factors is excluded due to difficulty in quantification*, but alternative data like interest rates and money supply is utilized. **Summary:** The speaker discusses the various decisions involved in building a machine learning model, *emphasizing that these choices are domain-dependent*. The key decisions include selecting the model, choosing relevant features, deciding on the amount of data, determining retraining frequency, and defining the loss function. The speaker highlights the importance of considering domain expertise and mentions their *preference for creating custom loss functions tailored to the specific problem*. During the Q&A session, a participant asks about handling situations with a scarcity of data. The speaker shares their experience, citing an example related to stock market data and the [[VIX Index]]. They used *regression and imputation algorithms to fill in historical VIX data*. The speaker emphasizes that there's no one-size-fits-all solution for handling data scarcity and suggests adapting approaches based on the situation. **Key Points:** 1. Decisions in building a machine learning model are domain-dependent. 2. Key decisions include selecting the model, choosing features, determining the amount of data, retraining frequency, and defining the loss function. 3. Custom loss functions were preferred over standard ones for a specific problem. 4. Handling data scarcity involves adapting solutions based on the situation. 5. An example involves using regression and imputation algorithms for historical VIX data. 7. *No high computational power was needed for the solution, as it wasn't high-frequency data.* #### Summary of Slide Deck [AI in Finance](https://cdn.upgrad.com/uploads/production/ccdae87b-98b2-4127-82b4-530e2c4f3758/Lect6_Finance.pdf) Dr. Jayaraman introduces the SmartAllocator, a financial portfolio management system, with a primary allocation strategy of 60:40 between S&P500 and TLT (bonds). The system advocates *dynamic weight adjustments* based on predictions of [[Market Volatility]], utilizing [[Technical Indicators]] of SP500 and [[FED Indicators]]. The SmartAllocator extends its functionality to sectoral allocations and introduces gold as an additional asset class. Performance metrics for the SmartAllocator with gold included show a [[Sharpe Ratio]] of 1.22 and an [[AAR (Average Annual Return)]] of 14.1%. **SmartAllocator:** - Primary allocation: 60:40 between S&P500 and TLT (bonds). - Rebalancing every 3 months. - Utilizes technical indicators of SP500 and FED indicators. - Introduces dynamic weight adjustments based on volatility predictions. **SmartAllocator Performance:** - Sharpe ratio: 1.22. - AAR: 14.1%. **Sectoral Allocations:** - Indicators used for smart allocation between sectors. **Gold in SmartAllocator:** - Generates signals for gold. - Improves SmartAllocator performance. **Key Points:** - The application of machine learning in financial markets was illustrated through examples like predicting stock market movements using the price-to-earnings (P/E) ratio. - The speaker highlighted the *challenge of choosing the right algorithm in financial markets* due to the complexity of data and varying hypotheses. - Decision-making in AI involves *selecting features, considering domain-specific factors, and creating customized loss functions*. See [[Loss Functions]]. - The importance of understanding macro indicators, such as interest rates set by the Federal Reserve, in predicting financial market movements was discussed. - The session concluded with an acknowledgment of the diverse choices and decisions made in AI and machine learning, emphasizing the need for thoughtful considerations.