Lunar Lander

1. What is Lunar Lander v3?

The Lunar Lander v3 environment, part of the OpenAI Gym (now Gymnasium), simulates the challenge of controlling a lunar landing module to land safely on a designated platform. It provides a robust environment for training and evaluating reinforcement learning (RL) algorithms.

The environment is physics-based, capturing the dynamics of the lander under gravitational forces and thruster actions, making it a popular choice for RL practitioners to test their models on continuous or discrete control tasks.

1a. Goal of the Environment

The primary goal in Lunar Lander v3 is to land the lunar module:

• Safely: The module should land upright on the designated platform.
• Efficiently: Fuel usage should be minimized.

Points are awarded based on landing accuracy, and penalties are given for crashing or using excessive fuel.

1b. Action and Observation Space

1. Action Space:

   • Discrete Mode: The environment has 4 discrete actions:

     1. Do nothing.
     2. Fire left orientation engine.
     3. Fire right orientation engine.
     4. Fire the main engine (upwards thrust).

   • Continuous Mode: In continuous control, actions correspond to continuous values for the main engine and the side thrusters.

2. Observation Space: The observation is an 8-dimensional vector that includes:

   • x: Horizontal position of the lander (meters).
   • y: Vertical position of the lander (meters).
   • vx: Horizontal velocity (meters/second).
   • vy: Vertical velocity (meters/second).
   • θ: Angle of the lander (radians).
   • ω: Angular velocity (radians/second).
   • leg1_contact: Boolean indicating whether the left leg has touched the ground.
   • leg2_contact: Boolean indicating whether the right leg has touched the ground.

1c. Rewards

The reward system in Lunar Lander v3 is designed to encourage safe and accurate landings:

• Positive Rewards:
  • Landing close to the target platform.
  • Coming to rest with minimal velocity.
  • Keeping the module upright.
• Negative Rewards:
  • Crashing the lander.
  • Drifting too far from the platform.
  • Excessive use of fuel (penalized for each action).

Landing successfully within a small target zone gives a significant reward boost.

1d. Parameters Affecting the Lander

The behavior of the lander is influenced by:

1. Gravity: Constant downward force simulating the Moon's gravitational pull.
2. Thruster Force: Determines how much acceleration the engines produce.
3. Fuel Consumption: Actions consume fuel, penalizing inefficient strategies.
4. Wind (Optional): Random lateral forces can be added for additional complexity.

1e. Solving the Environment

The environment is considered "solved" when an RL agent consistently achieves an average score of 200 over 100 episodes.

2. Use in Reinforcement Learning

Lunar Lander v3 is an excellent platform for testing RL techniques like:

• Policy Gradient Methods (e.g., PPO, A3C).
• Value-Based Methods (e.g., DQN).
• Actor-Critic Algorithms (e.g., SAC, DDPG).

The discrete and continuous action spaces make it versatile for both categories of RL algorithms.

2a. Getting Started

import gymnasium as gym

# Initialize the environment
env = gym.make("LunarLander-v3", continuous=False, render_mode="rgb_array")

# Reset the environment
state, info = env.reset()

print(f"Initial State: {state}")
print(f"Action Space: {env.action_space}")
print(f"Observation Space: {env.observation_space}")

Initial State: [ 0.00619869  1.4208851   0.62784475  0.44288316 -0.00717595 -0.14221632 0.          0.        ]

Action Space: Discrete(4)

Observation Space: Box(
[ -2.5     -2.5     -10.     -10.     -6.2831855     -10.     -0.     -0. ],
[  2.5      2.5      10.      10.       6.2831855     10.      1.      1. ],
(8,),
float32)