Multimodal
Setting the Environment
The multimodal option provides visual features to the agent in an identical manner to the visual only feature set, but it also includes data from the vehicle’s IMU sensor along with a few other pieces of data. We expect that the performance of agents with multimodal sensory data to be better than that of visual only agents. To set the environment to multimodal, simply modify the multimodal
parameter to True
in the configs/params.yaml
file:
env_kwargs:
multimodal: True # when True, both images and pose data are provided to agent
max_timesteps: 5000
...
Environment Observations
Setting this parameter to True
will change the return type of the step()
method of the RacingEnv class to return a spaces.Dict
containing:
- Track ID
a numeric identifier of the current track, relevant for multi-track training
- Camera Images
a numpy array of shape (image_width, image_height, 3)
- Additional Data
a numpy array of shape (30,) with the following data:
Array Indicies
Data
0
steering request
1
gear request
2
mode
3,4,5
direction velocity in m/s
6,7,8
directional acceleration in m/s^2
9,10,11
directional angular velocity
12,13,14
vehicle yaw, pitch, and roll, respectively
15,16,17
center of vehicle coordinates in the format (y, x, z)
18,19,20,21
wheel revolutions per minute (per wheel)
22,23,24,25
wheel braking (per wheel)
26,27,28,29
wheel torque (per wheel)