simulating-the-multi-armed-bandit.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[],"dockerImageVersionId":30786,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# <span style=\"color:#4CAF50; font-size:30px;\">📘 What We Are Doing in This Notebook</span>\n\n---\n\n## <span style=\"color:#2196F3; font-weight:bold;\">1. Simulating the 10-Armed Bandit Testbed:</span>\nWe are implementing a **bandit problem** with **10 arms**, where:\n- Each arm will have a true value **$q^∗(a)$** sampled from a Gaussian distribution with **mean 0** and **variance 1**.\n- Rewards for selecting an arm **a** will be sampled from a Gaussian distribution with **mean q(a)** and **variance 1**.\n\n---\n\n## <span style=\"color:#FFC107; font-weight:bold;\">2. Setting Up the Testbed:</span>\n### Steps:\n1. **Generate 2000 independent bandit problems.**  \n2. **Use the sample-average method** (with incremental updates) to learn action-value estimates.  \n3. Simulate learning over **1000 timesteps** and analyze the performance of the following algorithms:\n   - **Greedy**  \n   - **ϵ-greedy** (ϵ = 0.1 and ϵ = 0.01)  \n\n---\n\n## <span style=\"color:#FF5722; font-weight:bold;\">3. Running the UCB Algorithm:</span>\n### Steps:\n1. **Run the UCB algorithm** on the **ten-armed testbed**.  \n2. Compare its performance with **ϵ-greedy (ϵ = 0.1)**.  \n3. Plot and reproduce the following **learning curves**, averaged over **2000 bandits** for **1000 timesteps**:\n   - **Average Reward vs. Time**\n   - **% Optimal Actions vs. Time**\n\n---\n\n## <span style=\"color:#9C27B0; font-weight:bold;\">4. Learning Curves:</span>\nFor each algorithm, we will generate the following plots:\n- **Average Reward vs. Time:**  \n  Tracks the **average reward** obtained over time.  \n- **% Optimal Actions vs. Time:**  \n  Tracks the **percentage of times the optimal action** is chosen over time.\n\n---","metadata":{}},{"cell_type":"markdown","source":"# <span style=\"color:brown; font-weight:bold;\">📌Imports</span>","metadata":{}},{"cell_type":"code","source":"# import necessary libraries here\nimport numpy as np\nimport matplotlib.pyplot as plt\nnp.random.seed(63)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2024-11-25T16:04:33.420811Z","iopub.execute_input":"2024-11-25T16:04:33.421578Z","iopub.status.idle":"2024-11-25T16:04:33.453635Z","shell.execute_reply.started":"2024-11-25T16:04:33.421506Z","shell.execute_reply":"2024-11-25T16:04:33.452197Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# <span style=\"color:#3F51B5; font-weight:bold;\">🎯 Objective:</span>\nPlotting the following results:\n1. **Average Reward vs. Number of Steps**  \n2. **% Optimal Action vs. Number of Steps**\n\n---","metadata":{}},{"cell_type":"markdown","source":"# <span style=\"color:#FF6347; font-size:30px;\">🔨 Building the Bandit Object</span>\n---\n\nWe start by constructing the **Bandit Object**, representing the individual arms of the bandit.\n\n---","metadata":{}},{"cell_type":"code","source":"# Defining bandit object\nclass Bandit(object):\n    def __init__(self, arms=10):\n        self.arms = arms\n        self.q_star = np.random.normal(0, 1, self.arms)  # true action values\n\n    def get_reward(self, action):\n        return np.random.normal(self.q_star[action], 1)  # rewards sampled from N(q*(a), 1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-25T16:04:33.456144Z","iopub.execute_input":"2024-11-25T16:04:33.456612Z","iopub.status.idle":"2024-11-25T16:04:33.464024Z","shell.execute_reply.started":"2024-11-25T16:04:33.456562Z","shell.execute_reply":"2024-11-25T16:04:33.462111Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# <span style=\"color:#4682B4; font-size:30px;\">🤖 Defining the Multiarm Bandit for Greedy Approach</span>\n\nThis implementation simulates the **multi-arm bandit problem** using the **epsilon-greedy (ϵ-greedy)** method. Here's a step-by-step explanation:\n\n---\n\n### **🔍 Key Parameters**\n1. **`bandit`**: The multi-arm bandit environment containing:\n   - `arms`: Number of arms.\n   - $q^*(a)$: True reward distribution for each arm.\n   - `get_reward(action)`: Method to simulate the reward for a chosen arm.\n   \n2. **`timesteps`**: The total number of plays or time steps.\n\n3. **`epsilon (ϵ)`**: The probability of choosing a random (exploratory) action. \n   - With probability $1 - \\epsilon$, we select the arm with the **highest estimated value** (exploitation).\n   - With probability $\\epsilon$, we select a **random arm** (exploration).\n\n---\n\n### **📐 Algorithm**\n1. **Initialization**:\n   - `q_estimates`: An array of zeros to store the estimated reward for each arm.\n   - `action_counts`: An array of zeros to track the number of times each arm is selected.\n   - `rewards`: An array to record rewards received at each time step.\n   - `optimal_actions`: An array to record whether the **optimal arm** was selected at each time step.\n\n2. **Identifying the Optimal Arm**:\n   - $$ \\text{optimal\\_action} = \\arg\\max(q^*(a)) $$\n     This finds the arm with the highest true reward.\n\n3. **Game Simulation (for `t` in `timesteps`)**:\n   - **Exploration vs. Exploitation**:\n     - Generate a random number $p \\sim U(0, 1)$.\n     - If $p < \\epsilon$: **Exploration** → Choose a random arm.\n     - Otherwise: **Exploitation** → Choose the arm with the highest estimated reward:\n       $$ \\text{action} = \\arg\\max(q_{\\text{estimates}}) $$\n\n   - **Receive Reward**:\n     - Simulate the reward for the selected arm using:\n       $$ \\text{reward} = \\text{bandit.get\\_reward(action)} $$\n\n   - **Track Optimal Action**:\n     - If the chosen action matches the optimal arm:\n       $$ \\text{optimal\\_actions}[t] = 1 $$\n     - Otherwise, set it to 0.\n\n   - **Update Counts and Estimates**:\n     - Increment the count for the chosen arm:\n       $$ \\text{action\\_counts}[a] = \\text{action\\_counts}[a] + 1 $$\n     - Update the reward estimate using the incremental formula:\n       $$ q_{\\text{estimates}}[a] = q_{\\text{estimates}}[a] + \\frac{\\text{reward} - q_{\\text{estimates}}[a]}{\\text{action\\_counts}[a]} $$\n\n4. **Return Results**:\n   - **`rewards`**: A time series of rewards at each timestep.\n   - **`optimal_actions`**: A time series indicating whether the optimal action was selected.\n\n---\n\n### **🧮 Mathematical Formulation**\n\n#### **Incremental Update Formula**:\nThe estimated reward for arm $a$ is updated as:\n$$ q_{\\text{estimates}}[a] = q_{\\text{estimates}}[a] + \\frac{\\text{reward} - q_{\\text{estimates}}[a]}{\\text{action\\_counts}[a]} $$\n\n#### **Action Selection**:\n1. Exploitation:\n   $$ \\text{action} = \\arg\\max(q_{\\text{estimates}}) $$\n2. Exploration:\n   $$ \\text{action} = \\text{randomly chosen arm} $$\n\n---","metadata":{}},{"cell_type":"code","source":"# Defining Multiarm bandit game using epsilon greedy approach\ndef BanditGame(bandit, timesteps, epsilon=0):\n    arms = bandit.arms\n    q_estimates = np.zeros(arms)\n    action_counts = np.zeros(arms)\n    rewards = np.zeros(timesteps)\n    optimal_actions = np.zeros(timesteps)\n\n    optimal_action = np.argmax(bandit.q_star)\n\n    for t in range(timesteps):\n        p = np.random.rand()\n        if p < epsilon:\n            action = np.random.choice(arms)\n        else:\n            action = np.argmax(q_estimates)\n\n        reward = bandit.get_reward(action)\n        rewards[t] = reward\n\n        if action == optimal_action:\n            optimal_actions[t] = 1\n\n        action_counts[action] += 1\n        q_estimates[action] += (reward - q_estimates[action]) / action_counts[action]\n\n    return rewards, optimal_actions","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-25T16:04:33.465656Z","iopub.execute_input":"2024-11-25T16:04:33.466146Z","iopub.status.idle":"2024-11-25T16:04:33.477524Z","shell.execute_reply.started":"2024-11-25T16:04:33.466095Z","shell.execute_reply":"2024-11-25T16:04:33.476254Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n## <span style=\"color:#FFD700; font-size:30px;\">⚙️ Simulating the Game with Parameters</span>\n---\n\nThis section explores the simulation with different **parameters**, such as exploration strategies.\n\n---\n\n### **🎯 Insights**\n\n- **Balancing Exploration and Exploitation**:\n  - The parameter $ϵ$ controls the trade-off:\n    - High $ϵ$: Encourages exploration.\n    - Low $ϵ$: Favors exploitation.\n\n- **Optimal Action Tracking**:\n  - Tracks how often the best action (with the highest $q^*(a)$) is selected.\n\nThis approach helps evaluate how well the ϵ-greedy algorithm performs over multiple time steps and compares exploration vs. exploitation strategies.\n","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"# Simulating game with different parameters\ndef SimulateBanditGame(n_games, arms, timesteps, epsilons):\n    avg_rewards = {epsilon: np.zeros(timesteps) for epsilon in epsilons}\n    avg_optimal_actions = {epsilon: np.zeros(timesteps) for epsilon in epsilons}\n\n    for _ in range(n_games):\n        bandit = Bandit(arms)\n        for epsilon in epsilons:\n            rewards, optimal_actions = BanditGame(bandit, timesteps, epsilon)\n            avg_rewards[epsilon] += rewards\n            avg_optimal_actions[epsilon] += optimal_actions\n\n    for epsilon in epsilons:\n        avg_rewards[epsilon] /= n_games\n        avg_optimal_actions[epsilon] /= n_games\n\n    return avg_rewards, avg_optimal_actions\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-25T16:04:33.478936Z","iopub.execute_input":"2024-11-25T16:04:33.479289Z","iopub.status.idle":"2024-11-25T16:04:33.493506Z","shell.execute_reply.started":"2024-11-25T16:04:33.479255Z","shell.execute_reply":"2024-11-25T16:04:33.492065Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## <span style=\"color:#8A2BE2; font-size:30px;\">🎮 Playing Game for Greedy Approach and 📊 Analyzing Output</span>\n---\n\nRun the simulation with the **Greedy algorithm**, and analyze results like average rewards and optimal actions.\n\n---\n","metadata":{}},{"cell_type":"code","source":"# Playing the Game\ndef PlayBanditGame(n_games, arms, timesteps, epsilons):\n    avg_rewards, avg_optimal_actions = SimulateBanditGame(n_games, arms, timesteps, epsilons)\n    steps = np.arange(timesteps)\n    # Adjusting figure size for better clarity\n    plt.figure(figsize=(10, 10))\n    \n    # Defining colors and line styles for a visually appealing distinction\n    colors = {0: '#2E8B57', 0.01: '#FF4500', 0.1: '#1E90FF'}\n    linestyles = {0: 'dotted', 0.01: 'dashed', 0.1: 'solid'}\n    \n    # Plotting Average Rewards\n    plt.subplot(2, 1, 1)\n    for epsilon, rewards in avg_rewards.items():\n        label = 'Greedy' if epsilon == 0 else f'ε = {epsilon}'\n        plt.plot(\n            steps,\n            rewards,\n            label=label,\n            color=colors[epsilon],\n            linestyle=linestyles[epsilon],\n            linewidth=2\n        )\n    plt.xlabel('Steps', fontsize=14, fontweight='bold', color='#333')\n    plt.ylabel('Average Reward', fontsize=14, fontweight='bold', color='#333')\n    plt.legend(fontsize=12)\n    plt.title('Average Rewards vs. Steps', fontsize=16, fontweight='bold', color='#111')\n    plt.grid(color='gray', linestyle='--', linewidth=0.5, alpha=0.7)\n    \n    # Plotting % Optimal Actions\n    plt.subplot(2, 1, 2)\n    for epsilon, optimal_action in avg_optimal_actions.items():\n        label = 'Greedy' if epsilon == 0 else f'ε = {epsilon}'\n        plt.plot(\n            steps,\n            optimal_action * 100,\n            label=label,\n            color=colors[epsilon],\n            linestyle=linestyles[epsilon],\n            linewidth=2\n        )\n    plt.xlabel('Steps', fontsize=14, fontweight='bold', color='#333')\n    plt.ylabel('% Optimal Actions', fontsize=14, fontweight='bold', color='#333')\n    plt.legend(fontsize=12)\n    plt.title('% Optimal Actions vs. Steps', fontsize=16, fontweight='bold', color='#111')\n    plt.grid(color='gray', linestyle='--', linewidth=0.5, alpha=0.7)\n    \n    # Tight layout and display\n    plt.tight_layout(pad=3)\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-25T16:33:11.745512Z","iopub.execute_input":"2024-11-25T16:33:11.745987Z","iopub.status.idle":"2024-11-25T16:33:11.758512Z","shell.execute_reply.started":"2024-11-25T16:33:11.745945Z","shell.execute_reply":"2024-11-25T16:33:11.756974Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#GamePlay function call\nepsilons = [0,0.01,0.1]\nn_games = 2000\narms = 10\ntimesteps=1000\n\nPlayBanditGame(n_games,arms,timesteps,epsilons)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-25T16:33:21.794638Z","iopub.execute_input":"2024-11-25T16:33:21.795503Z","iopub.status.idle":"2024-11-25T16:33:57.671158Z","shell.execute_reply.started":"2024-11-25T16:33:21.795462Z","shell.execute_reply":"2024-11-25T16:33:57.669933Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# <span style=\"color:#FF6347; font-size:30px;\">📈 Analysis of Performance for Greedy and ϵ-Greedy Methods</span>\n---\n\n### **🟢 Upper Graph: Expected Reward vs. Experience**\n\n- The greedy method starts strong, improving faster initially compared to other methods.\n- **Key Observation**: Despite the early advantage, the greedy method plateaus at a **lower performance level** over time.\n- **Why?** The greedy method struggles in the long term because it often gets stuck with **suboptimal actions**, failing to explore other possibilities.\n\n---\n\n### **🔵 Lower Graph: % Optimal Actions vs. Experience**\n\n- The greedy method identifies the optimal action in only **one-third** of the tasks.\n- In the remaining two-thirds, it prematurely dismisses the optimal action due to **initial suboptimal samples** and never revisits it.\n\n---\n\n### **🌟 Advantages of ϵ-Greedy Methods**\n\n1. The **ϵ = 0.1** method:\n   - Explores more effectively, finding the optimal action earlier.\n   - **Limitation**: It selects the optimal action only about **91% of the time**.\n\n2. The **ϵ = 0.01** method:\n   - Improves slower but performs **better in the long run** than ϵ = 0.1 on both measures.\n   - Strikes a balance between exploration and exploitation.\n\n---","metadata":{}},{"cell_type":"markdown","source":"# <span style=\"color:#20B2AA; font-size:30px;\">🔨 Creating Multiarm Bandit for UCB</span>\n\nThis implementation simulates the **multi-arm bandit problem** using the **Upper Confidence Bound (UCB)** algorithm. It dynamically balances exploration and exploitation by incorporating confidence intervals in action selection.\n\n---\n\n### **🔍 Key Parameters**\n1. **`bandit`**: The multi-arm bandit environment containing:\n   - `arms`: Number of arms.\n   - $q^*(a)$: True reward distribution for each arm.\n   - `get_reward(action)`: Method to simulate the reward for a chosen arm.\n   \n2. **`timesteps`**: The total number of plays or time steps.\n\n3. **`c`**: A parameter controlling the degree of exploration. \n   - Larger $c$ values encourage exploration by widening the confidence bounds.\n   - Smaller $c$ values focus on exploitation.\n\n---\n\n### **📐 Algorithm**\n1. **Initialization**:\n   - `Q_estimates`: An array of zeros to store the estimated reward for each arm.\n   - `action_counts`: An array of zeros to track the number of times each arm is selected.\n   - `rewards`: An array to record rewards received at each time step.\n\n2. **Game Simulation (for `t` in `timesteps`)**:\n   - **Arm Selection**:\n     - In the first `arms` steps, each arm is played once to gather initial information:\n       $$ \\text{action} = t \\quad \\text{for } t < \\text{arms} $$\n     - Afterward, the UCB formula is used to select the arm with the highest **Upper Confidence Bound**:\n \n      $$ Q_{\\text{estimates}}[a] = Q_{\\text{estimates}}[a] + \\frac{\\text{reward} - Q_{\\text{estimates}}[a]}{\\text{action\\_counts}[a]} \n$$\n\n\n   - **Receive Reward**:\n     - Simulate the reward for the selected arm:\n       $$ \\text{reward} = \\text{bandit.get\\_reward(action)} $$\n\n   - **Update Counts and Estimates**:\n     - Increment the count for the chosen arm:\n       $$ \\text{action\\_counts}[a] = \\text{action\\_counts}[a] + 1 $$\n     - Update the reward estimate using the incremental formula:\n       $$ Q_{\\text{estimates}}[a] = Q_{\\text{estimates}}[a] + \\frac{\\text{reward} - Q_{\\text{estimates}}[a]}{\\text{action\\_counts}[a]} $$\n\n3. **Return Results**:\n   - **`rewards`**: A time series of rewards at each timestep.\n\n---\n\n### **🧮 Mathematical Formulation**\n\n#### **UCB Formula**:\nThe **Upper Confidence Bound** is calculated as:\n$$ Q_{\\text{estimates}}[a] + c \\cdot \\sqrt{\\frac{\\log(t+1)}{\\text{action\\_counts}[a]}} $$\n\n#### **Incremental Update**:\nThe estimated reward for arm $a$ is updated as:\n$$ Q_{\\text{estimates}}[a] = Q_{\\text{estimates}}[a] + \\frac{\\text{reward} - Q_{\\text{estimates}}[a]}{\\text{action\\_counts}[a]} $$\n\n---\n\n### **🎯 Insights**\n\n- **Exploration vs. Exploitation**:\n  - The UCB formula adds a **confidence interval** term to the reward estimate.\n  - Arms with lower play counts are assigned wider confidence intervals, encouraging exploration.\n\n- **Parameter `c`**:\n  - A higher $c$ value encourages more exploration by widening the confidence bounds.\n  - A lower $c$ value focuses on exploitation by narrowing the bounds.\n\n- **Performance**:\n  - UCB typically performs better than $\\epsilon$-greedy as it dynamically adjusts exploration based on the number of plays and observed rewards.\n\nThis approach helps evaluate how well the UCB algorithm performs over multiple time steps by efficiently balancing exploration and exploitation.\n","metadata":{}},{"cell_type":"code","source":"#Defining UCB Bandit game\ndef UCBBanditGame(bandit,timesteps,c=2):\n\n  arms=bandit.arms\n  Q_estimates= np.zeros(arms)\n  action_counts=np.zeros(arms)\n  rewards= np.zeros(timesteps)\n\n  for t in range (timesteps):\n    if t < arms:\n      action =t\n    else:\n      action = np.argmax(Q_estimates+ c * np.sqrt(np.log(t+1)/action_counts))\n\n    reward = bandit.get_reward(action)\n    rewards[t]=reward\n\n    action_counts[action]+=1\n    Q_estimates[action] += (reward - Q_estimates[action])/ action_counts[action]\n\n  return rewards","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-25T16:05:08.020822Z","iopub.execute_input":"2024-11-25T16:05:08.021177Z","iopub.status.idle":"2024-11-25T16:05:08.027662Z","shell.execute_reply.started":"2024-11-25T16:05:08.021144Z","shell.execute_reply":"2024-11-25T16:05:08.026640Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n## <span style=\"color:#FF4500; font-size:30px;\">⚙️ Running UCB Simulations and 📊 Analyzing Output</span>\n---\n\nSimulate the **UCB algorithm** and compare its performance with other methods like **ϵ-greedy**.\n\n---\n","metadata":{}},{"cell_type":"code","source":"#play the UCBEpislon game\ndef PlayUCBEpsilon(n_games,arms,timesteps,c,epsilon):\n    avg_rewards = { epsilon : np.zeros(timesteps) }\n    avg_rewards['UCB']=np.zeros(timesteps)\n    \n    for _ in range (n_games):\n    \n        bandit=Bandit(arms)\n        rewards,_ = BanditGame(bandit,timesteps,epsilon)\n        avg_rewards[epsilon]+= rewards\n        \n        rewardsUCB = UCBBanditGame(bandit,timesteps,c)\n        avg_rewards['UCB']+= rewardsUCB\n    \n    for key in avg_rewards.keys():\n        avg_rewards[key] /= n_games\n    \n        steps= np.arange(timesteps)\n\n    # Setting up the figure\n    plt.figure(figsize=(10, 8))\n    \n    # Plotting Average Rewards\n    plt.subplot(2, 1, 1)\n    for key, rewards in avg_rewards.items():\n        # Distinguishing UCB with a bold color and other methods with grey\n        label = f'ε = {key}' if key != 'UCB' else 'UCB (c=2)'\n        color = 'grey' if key != 'UCB' else '#1E90FF'\n        linestyle = 'solid' if key == 'UCB' else 'dashed'\n        plt.plot(\n            steps,\n            rewards,\n            label=label,\n            color=color,\n            linestyle=linestyle,\n            linewidth=2 if key == 'UCB' else 1.5\n        )\n    \n    # Adding labels, legend, and title\n    plt.xlabel('Steps', fontsize=14, fontweight='bold', color='#333')\n    plt.ylabel('Average Reward', fontsize=14, fontweight='bold', color='#333')\n    plt.legend(fontsize=12, loc='lower right')\n    plt.title('Average Rewards vs. Steps', fontsize=16, fontweight='bold', color='#111')\n    \n    # Adding grid for better readability\n    plt.grid(color='gray', linestyle='--', linewidth=0.5, alpha=0.7)\n    \n    # Adjust layout for spacing\n    plt.tight_layout()\n    plt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-25T16:42:19.833348Z","iopub.execute_input":"2024-11-25T16:42:19.833756Z","iopub.status.idle":"2024-11-25T16:42:19.845165Z","shell.execute_reply.started":"2024-11-25T16:42:19.833723Z","shell.execute_reply":"2024-11-25T16:42:19.843772Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#GamePlay\nPlayUCBEpsilon(n_games=2000,arms=10,timesteps=1000,c=2,epsilon=0.1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-25T16:42:24.592789Z","iopub.execute_input":"2024-11-25T16:42:24.593846Z","iopub.status.idle":"2024-11-25T16:43:00.905612Z","shell.execute_reply.started":"2024-11-25T16:42:24.593799Z","shell.execute_reply":"2024-11-25T16:43:00.904513Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n\n## <span style=\"color:#4682B4; font-size:30px;\">🚀 Insights on UCB (c=2) vs. ϵ-Greedy (ϵ=0.1)</span>\n---\n\n### **🔎 Initial Phase (First 10 Plays)**\n\n- **UCB (c=2)** focuses on **exploring all arms equally**, gathering information on each arm's rewards.\n- During this phase, the **average reward remains close to zero** for UCB.\n- Conversely, **ϵ-greedy (ϵ=0.1)** starts **outperforming UCB**, leveraging its early exploitation advantage.\n\n---\n\n### **📊 Mid-Phase (Around 11th Play)**\n\n- **Performance Spike**: \n   - UCB, having gathered sufficient information, shifts towards **exploitation**.\n   - It begins favoring the arm with the **highest estimated reward** based on confidence bounds.\n\n---\n\n### **🌟 Long-Term Advantage of UCB**\n\n- UCB eventually surpasses ϵ-greedy in **average reward** because it:\n  1. **Reduces exploration** over time.\n  2. Focuses more effectively on the **optimal arm**, leveraging its confidence-based strategy.\n- Compared to ϵ-greedy, UCB achieves **higher long-term rewards** while maintaining efficient exploration.\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"<span style=\"color:#4CAF50; font-size:22px; font-weight:bold;\">💬 Thank You for Reading!</span>  \n\n<p style=\"font-size:18px; color:#23fc;\">\nIf you enjoyed exploring this notebook or found it helpful, feel free to share your thoughts, suggestions, and feedback in the comments section. Your input is greatly valued!  \n</p>\n\n<p style=\"font-size:18px; color:#FF5722; font-weight:bold;\">\n⭐ Don’t forget to upvote if you liked this notebook!\n</p>\n\n<p style=\"font-size:16px; color:#888;\">\nHappy Learning and Best Wishes! 🚀\n</p>\n","metadata":{}}]}