Skip to content

RL Walking and Kicking

For learned walks and kicks the WalkingEngine uses neuronal networks for generating motion patterns. The currently learned walks were trained with the pipeline of Booster Robotics1.

Note

DO NOT lift the robot up when the walk or kick policy is active.

Network Structure Input

For walking and kicking we use two different policies with different observations, but both run at 50 Hz, meaning the policy is only called every 20 ms and the joint requests does not change inbetween, only once after inference. The walking uses a history over the last 200 ms, spaced 20 ms apart. It was intended to be also used for retraining for playing the ball, resulting in having many unused fields as dummy inputs. For kicking we use a policy without history, resulting in a more straight forward observation space.

Walking Policy

The expected neuronal network input is expected to be as follows:

  • History data from old to new. Last 10 inputs, 20 ms apart
    • 3D Vektor (0,0,-1) rotated around the current torso orientation
      • Sequence of x, y, z
      • 3 Inputs
    • Gyro values of the IMU
      • Sequence of x, y, z
      • 3 Inputs
    • The measured joint positions
      • Sequence for Booster robots is (waistYaw), lHipPitch, lHipRoll, lHipYaw, lKneePitch, lAnklePitch, lAnkleRoll, rHipPitch, rHipRoll, rHipYaw, rKneePitch, rAnklePitch, rAnkleRoll
      • A position offset is subtracted beforehand
      • [12|13] Inputs, depending on K1 or T1
    • The last policy joint output
      • The requested joint positions
        • Sequence the same as joint positions
      • [12|13] Inputs, depending on K1 or T1
    • Dummy fields with value 0
      • 2 Inputs
  • Walking speed
    • Sequence of x, y, yaw
    • When standing, these values should be 0
    • 3 Inputs
  • Phase input
    • Sequence cos, sin
    • The trained frequency is in the range of [1,2]. The policy itself controls the frequency.
    • 2 Inputs
  • The measured joint velocities
    • Sequence the same as joint positions
    • Raw values calculated based on the last and current joint positions
    • Multiplied by factor 0.1
    • [12|13] Inputs, depending on K1 or T1
  • The last frequency offset
    • 1 Input
  • Empty fields with value 0
    • 6 inputs

The walking speed is learned for the following ranges:

  • x: [-1, 2] (in m/s)
  • y: [-1, 1] (in m/s)
  • yaw: [-1.5, 1.5] (in rad/s)

For the actual usage, the y-speed can be set up to [-1.5, 1.5] and the yaw-speed to [-2.1, 2.1]. Combinations can be freely set, but the accurate execution starts to suffer, one the request changes too often by a large margin or the combinations are too large.

Kicking Policy

We use two different kick policies. One for softer kicks of up to 5 meters durings free kicks or the kick-off. All other kicks are done by the generic kick policy. Both use the same inputs, the softer kick policy is just more restricted. The expected neuronal network input is expected to be as follows:

  • 3D Vektor (0,0,-1) rotated around the current torso orientation
    • Sequence of x, y, z
    • 3 Inputs
  • Gyro values of the IMU
    • Sequence of x, y, z
    • 3 Inputs
  • 3D Ball positions
    • Sequence x, y, z
    • In meter
    • Multiplied by factor 0.1
    • z-value set to 0
    • 3 Inputs
  • Phase input
    • Sequence cos, sin
    • The trained frequency is in the range of [1,2]. The policy itself controls the frequency.
    • 2 Inputs
  • The measured joint positions
    • Sequence for Booster robots is (waistYaw), lHipPitch, lHipRoll, lHipYaw, lKneePitch, lAnklePitch, lAnkleRoll, rHipPitch, rHipRoll, rHipYaw, rKneePitch, rAnklePitch, rAnkleRoll
    • A position offset is subtracted beforehand
    • [12|13] Inputs, depending on K1 or T1
  • The measured joint velocities
    • Sequence the same as joint positions
    • Raw values calculated based on the last and current joint positions
    • Multiplied by factor 0.1
    • [12|13] Inputs, depending on K1 or T1
  • The last policy joint output
    • The requested joint positions
      • Sequence the same as joint positions
    • [12|13] Inputs, depending on K1 or T1
  • The last frequency offset
    • 1 Input
  • Special behavior flags
    • Values can be 1 or 0, nothing inbetween
    • First Flag: Scaled strength (0) or kick as strong as possible (1)
      • Set to 0 for soft kick policy
    • Second Flag: Always 0
    • Third Flag: Take some time (0) or execute as fast as possible (1)
      • Set to 0 for soft kick policy
    • 3 Inputs
  • 3D Ball velocity
    • Sequence x, y, z
    • In m/s
    • Multiplied by factor 0.1
    • 3 Inputs
  • Kick direction
    • sin(direction) + cos(direction)
    • 2 inputs
  • Kick strength
    • Target ball speed in m/s, multiplied by 0.1
    • 1 Input
  • 2D Old Ball positions
    • Sequence x, y
    • In meter
    • Multiplied by factor 0.1
    • 2 Inputs

The ball coordinates are relative to the yaw-orientation of the torso, and in the x- and y-axis frame between both sole origins. This also means, if the torso stays perfectly still in the world and only one leg is moving forward towards the ball, the relative ball position would move torwards the robot.

Neuronal Network output

The expected neuronal network output is as follows:

  • The requested joint positions
    • Same sequence as joint position Input
    • The basis offset needs to be added
  • Frequency offset
    • Used to update the phase gait

The output joint positions are expected to be clipped and a position offset added. The raw values are used as input for the next inference of the policy. The frequency offset is also clipped and relative to the frequency 1.5. The raw offset (clipped into a wider range) is used as the input in the next cycle.

Learnings From Training

Below a short list of our current learnings and observations from training neuronal networks and testing them on the real robot:

  • Learning an offset for the frequency results in a more stable walk with more stretched out legs.
  • Using a history helps learning a walk with more stretched out legs.
  • Setting simulation parameters as armature, damping and frictionloss result in a less stomping walk.
  • Ensuring the entropy loss does not become too negative during training helps the policy to not get stuck in local minima.
  • Implementing a symmetrie loss helps in more symmetric motions
  • The first initial training can be done with a lower number of robots (like 1024)

Current TODOs

- The `WalkingEngine` currently decides whether the walk or kick policy shall be called. This should be moved out, as this is **NOT** the task of the `WalkingEngine`.

  1. Booster Robotics - https://github.com/BoosterRobotics/booster_gym/tree/main ↩

Last update: September 1, 2026