Description
1. Tabular RL [100 points] In this problem, you will compare the performance of SARSA and expected SARSA on the Frozen Lake domain from the Gym environment suite: https://gymnasium.farama.org/environments/toy_text/frozen_lake/ Use a tabular representation of the state space. Exploration should be softmax (Boltzmann). You will do 10 independent runs. Each run consists of 500 segments, in each segment there are 10 episodes of training, followed by 1 episode in which you simply run the optimal policy so far (i.e. you pick actions greedily based on the current value estimates). Pick 3 settings of the temperature parameter used in the exploration and 3 settings of the learning rate. You need to plot: • One u-shaped graph that shows the effect of the parameters on the final training performance, expressed as the return of the agent (averaged over the last 10 training episodes and the 10 runs); note that this will typically end up as an upside-down u. • One u-shaped graph that shows the effect of the parameters on the final testing performance, expressed as the return of the agent (during the final testing episode, averaged over the 10 runs) • Learning curves (mean and standard deviation computed based on the 10 runs) for what you pick as the best parameter setting for each algorithm Write a small report that describes your experiment, your choices of parameters, and the conclusions you draw from the graphs. 2. Function approximation in RL [100 points] Implement and compare empirically Q-learning and actor-critic with linear function approximation on the cart-pole domain from the Gym environment suite: https://gymnasium.farama.org/environments/classic_control/cart_pole/ For this experiment, you should use a function approximator in which you discretize the state variables into 10 bins each; weights start initialized randomly between −0.001 and 0.001. You will need to use the same seed for this initialization for all parameters settings, but will have 10 different seeds (for the different runs). Use 3 settings of the learning rate parameter α = 1/4, 1/8, 1/16. Perform 10 independent runs, each of 1000 episodes. Each episode should start at a random initial state. The exploration policy should be -greedy and you should use 3 values of of your choice. Plot for each algorithm 3 graphs, one for each , containing the average and standard error of the learning curves for each value of α (each graph will have 3 curves). Make sure all graphs are on 1 the same scale. Based on these graphs, pick what you consider the best parameter choice for both and α and show the best learning curve for Q-learning and actor-critic on the same graph. Write a small report that describes your experiment, your choices of parameters, and the conclusions you draw from this experimentation. 2
