Test Framework¶

Overview¶

The test framework provides a comprehensive solution for automated evaluation of robotic soccer simulations. It allows running complete games or individual game situations repeatedly, with configurable robot starting positions and the ability to introduce controlled variations to test robustness. Object positions and rotations can be randomized and predefined zones on the field enable easy evaluation of whether balls or robots occupy specific areas, independent of field size or scenario.

Test objectives can be defined flexibly and combined using logical expressions, including negation and nested conditions. Each objective is continuously monitored throughout a test, allowing for an early detection of failures while ensuring correct evaluation of complex scenarios. The framework also supports realistic full-game simulations, preserving state across halves, tracking team performance, and monitoring communication budgets to ensure adherence to limits.

Architecture¶

The test framework is built on a modular architecture that supports fully isolated, state-based, and efficient test execution. It manages the entire lifecycle of a test session — from command parsing and initialization to execution, persistence, and result generation.

1. Execution Flow¶

Each test session is executed as an independent test instance. When a test starts, a new instance is initialized that manages its own files and data independently of other sessions. Tests are executed in multiple rounds, during which defined checks are performed to validate various conditions. After each round, runtime data is stored, and statistics are updated. Once all rounds are completed, the results are generated, and temporary files are automatically cleaned up.

2. Instance Isolation¶

Each test instance operates in complete isolation from others. A dedicated subdirectory is created in the system’s temporary folder (e.g., %AppData%\Local\Temp on Windows, /tmp on Linux, or /var/folders on macOS) to store all data related to that instance.

Each instance is uniquely identified by a session ID, generated by combining the scene file name and the process ID, and converting the result into a CRC32 hash. This mechanism ensures that

each instance has its own data space
parallel executions do not interfere
all related files can be reliably located

3. State Management¶

The framework controls the execution process through an internal state system that defines four main states:

initial – default state before a test starts
running – test is currently active
resetting – end of a test round and preparation for the next
finished – all rounds completed and test finalized

The state system governs how the framework handles existing session data and determines the operating mode when a scene is loaded. It also allows the framework to reuse scene and script files efficiently without creating redundant copies.

4. Command Parsing¶

Test commands are processed using a structured top-down parser. The parser performs both syntactic and semantic validation of input, ensuring that all parameters are valid and consistent.

The parser supports flexible argument ordering:

Mandatory arguments must appear first.
Optional arguments can be specified in any order using unique keywords.

5. Data Organization¶

All test data is stored in logically separated files within the instance directory. The framework distinguishes between static and dynamic data:

Parameter file – contains static input parameters that remain unchanged during the test.
State file – contains runtime data that changes during execution.

Data is kept in memory throughout a test round and written to disk only once after the round ends.

6. Test Execution¶

Tests are executed automatically based on the loaded scene and script files. During each round, all defined checks are performed at specified intervals. Each check validates conditions and updates the current state as needed.

After the final round:

Test results are aggregated and written to a result file.
A summary of key statistics is printed to the console.
All temporary data and files are automatically deleted.

Manual¶

Automated tests provide two main ways to evaluate robot behavior in the simulator:

Running any number of full games between two teams
Testing specific game situations with defined target conditions

The following sections explain the available options and how to operate the tests. The guide is divided into a section for Fast Scenes (all robots have a prefect world model) and one for PerceptOracle Scenes (robots do not perform any image processing but receive simulated perceptions of their environment). While both share many features, there are also some important differences in usage.

Testing in Fast Scenes¶

Preparation: Start the simulator and select the desired Fast Scene.

Testing Full Games¶

test game <numRuns> [con <conFile>] [rt] [q]

Parameter Explanation:

game: Indicates that full games, i.e., two halves each, should be played.
numRuns: The number of games to play. Required parameter.
conFile: Optionally, a .con file can be specified to, for example, disable certain modules of a team.
rt: If set, games run in real-time; otherwise, as fast as the hardware allows.
q: If set, the application automatically exits after the last game.

Examples:

test game 2 con file.con rt q

Two games (four halves) are played.
The games run in real-time.
The application automatically closes after the last game.
The specified .con file is loaded once before each game.

test game 4

Four games (eight halves) are played.
The games run faster than real-time.
The application remains open after the last game.

Testing Game Situations¶

test situation <numRuns> <runTimeout> <listOfTargets>* [con <conFile>] [fuz <params>*] [rt] [q]

Parameter Explanation:

situation: Indicates that this is a situation test.
numRuns: Number of test iterations. Required parameter.
runTimeout: Maximum duration of a situation in seconds. If targets are not met within this time, the test iteration is considered failed. Required parameter.
conFile: Optionally, a .con file specifying robot starting positions. This file can be previously created with the command sv fast.
fuz: Specifies whether the positions from the .con file should be slightly varied in each iteration to introduce additional randomness. Only valid in combination with a conFile. If a test fails, the modified .con file is saved for later analysis. To enable fuzzing, one or more modifiers must follow the keyword fuz, defining the maximum deviations in mm per axis. Options include (multiple can be combined in any order):
all <deviation>: Uniform deviation for position and rotation (equivalent to pos <deviation> rot <deviation>).
pos <deviation>: Deviation only for position (equivalent to x <d> y <d> z <d>).
rot <deviation>: Deviation only for rotation (equivalent to rotx <d> roty <d> rotz <d>).
x | y | z <deviation>: Deviation only for the specified position axis.
rotx | roty | rotz <deviation>: Deviation only for the specified rotation axis.
rt: If set, tests run in real-time; otherwise, as fast as the hardware allows.
q: If set, the application automatically exits after all iterations.
listOfTargets: A list of conditions combined with logical operators that must be satisfied for a test iteration to succeed. Examples include:
goalTeamA: The left team must score a goal.
goalTeamB: The right team must score a goal.
ballInZone (<x1> <y1> <x2> <y2> | (fieldHalf | centerCircle | penaltyArea | goalArea)[A | B]): The ball must be inside the specified zone at least once. Either define a rectangle (top-left corner: x1, y1; bottom-right corner: x2, y2) or a predefined zone. Optionally, suffix A or B specifies the field side (e.g., centerCircleA for the center circle on the left team’s side).
robotTouch <robotX>: The specified robot must touch the ball at least once (e.g., robot1, robot21).
robotInZone (<x1> <y1> <x2> <y2> | (fieldHalf | centerCircle | penaltyArea | goalArea)[A | B]) <robotX>: The specified robot must enter the specified zone at least once.
teamPossession <team>: The specified team (teamA or teamB) must possess the ball at least once.
penalized <robotX>: The specified robot must be penalized.
robotInSkill <skillX> <robotX>: The specified robot must enter the given skill state at least once. Valid skills: none, stand, walk, shoot, pass, dribble, block, mark, observe, clear.
budget (teamA | teamB) (< | > | >= | <=) <value>: Defines a message budget rule for a team, checking whether the number of messages sent by the team over a defined period exceeds or falls below a specified threshold.

Logical Combination of Targets:

Target conditions can be combined using logical operators:

not for negation
& for logical AND
| for logical OR

Parentheses can be used for grouping.

Example:

not (penalized robot1 | penalized robot2) & (goalTeamA | goalTeamB)

Neither robot1 nor robot2 may be penalized and at least one team must score a goal.

Command Examples:

test situation 3 60 not goalTeamA & goalTeamB con Saved.con fuz all 1000 rt q

3 test iterations, each lasting max 60 seconds.
Saved.con is loaded before each test.
Robot positions are randomly varied up to 1000 mm in each axis (position + rotation).
Test runs in real-time.
Application closes after last iteration.
Left team must not score, right team must score.

test situation 3 60 goalTeamA | goalTeamB con Saved.con fuz rot 1000

3 iterations of max 60 seconds.
Saved.con used.
Only rotations are varied up to 1000 mm.
At least one team must score.

test situation 2 90 ballInZone 3000 0 0 3000 not goalTeamB

2 iterations, 90 seconds each.
Ball must enter the rectangle (3000, 0) to (0, 3000).
Right team may not score.

test situation 2 30 robotTouch robot1

2 iterations, 30 seconds each.
robot1 must touch the ball at least once.

test situation 2 30 robotInZone 3000 0 0 3000 robot1 & not robotTouch robot1

2 iterations, 30 seconds each.
robot1 must enter the rectangle once and must not touch the ball.

test situation 2 30 robotInZone not goalAreaA robot1 con Saved.con fuz rotx 1000 y 500

2 iterations, 30 seconds each.
Positions from Saved.con are varied:
X rotation: max 1000 mm
Y position: max 500 mm
robot1 must not be in goal area A.

test situation 2 30 teamPossession teamA

2 iterations, 30 seconds each.
Team A must possess the ball at least once.

test situation 2 30 not penalized robot21

2 iterations, 30 seconds each.
robot21 must not be penalized.

test situation 2 30 robotInSkill clear robot2

2 iterations, 30 seconds each.
robot2 must enter clear skill at least once.

test situation 5 60 not (penalized robot1 | penalized robot) & goalRightTeam con Saved.con fuz pos 300 rot 100

5 iterations, 60 seconds each.
Positions (max 300 mm) and rotations (max 100 mm) randomly varied.
robot1 and robot2 must not be penalized.
Right team must score.

test situation 4 30 robotInZone goalAreaB robot3 & robotTouch robot3 & teamPossession teamA

4 iterations, 30 seconds each.
robot3 must:
Be in goal area B
Touch the ball
Ensure team A possesses the ball

test situation 3 90 (robotInSkill shoot robot1 | robotInSkill pass robot) & robotTouch robot1

3 iterations, 90 seconds each.
robot1 must touch the ball and enter shoot or pass skill at least once.

test situation 2 40 ballInZone centerCircleA & not robotInZone centerCircleA robot22

2 iterations, 40 seconds each.
Ball must be in center circle A.
robot22 must not be in center circle A.

test situation 20 60 (budget teamA > 1000 & budget teamB < 50)

20 iterations, 60 seconds each.
Checks that team A’s message budget exceeds 1000 and team B’s is below 50.

test situation 1 30 budget teamA >= 200

1 iteration, 30 seconds.
Checks if team A’s message budget is greater than or equal to 200.

Testing in PerceptOracle Scenes¶

Start the simulator and load the desired PerceptOracle Scene.
Custom starting positions require a .con file created with sv oracle. Without it, robots cannot determine positions.
Remaining steps are the same as for Fast Scenes.

Starting Tests via `SimRobot.exe` (e.g., CI/CD)¶

Copy the desired .ros2 and .con files under a new name.
Add the desired test command at the end of the .con file. The q parameter for automatic exit is mandatory.
Optionally, add a second .con file for alternative positions.
Run via console:

./Build/Windows/SimRobot/Release/SimRobot.exe Config/Scenes/Tests/TestOneTeamFast.ros2

Optionally use -noWindow for headless mode.
Exit codes:
Success: EXIT_SUCCESS
Failure: EXIT_FAILURE

Stopping Tests¶

test stop

Running Tests Efficiently¶

To optimize efficiency, tests can be executed in parallel, with automated management of test distribution, process monitoring, and aggregation of results in structured formats for analysis. A tool called TestRunner, located under B-Human\Util\TestRunner, provides this functionality.

Distributes tests across multiple CPU cores
Runs without GUI for faster execution
Aggregates and outputs results to the console

Important: Must be run from the B-Human root directory.

Arguments:

scene: Path to a .ros2 scene or folder containing scenes. Required.
--env: Simulator environment: debug, develop, release (default release)
--testcmd: Full test command in quotes (required for unmodified scenes)
--workers: Number of parallel processes (default: CPU cores)
--loglevel: Logging level: debug, info, warning, error, critical (default warning)
--gui: Run with SimRobot GUI (default: no GUI)