A Q-learning algorithm to generate shots for walking robots in soccer simulations

RoboCup, originally named the J-League, is an annual robotics and artificial intelligence (AI) competition organized by the International RoboCup Federation. During RoboCup, robots compete with other robots soccer tournaments.

The idea for the competition originated in 1992, when Professor Alan Mackworth at University of British Columbia in Canada wrote a paper entitled “On Seeing Robots.” In 1993, a research team in Japan drew inspiration from this paper to organize the first robot soccer competition.

While RoboCup can be highly entertaining, its main objective is to showcase advancements in robotics and AI in a real-world setting. The robotic systems participating in the competition are the result of intensive research efforts by many researchers worldwide.

In addition to the real-world competition, computer scientists and roboticists can test their computational tools for robot soccer at the the RoboCup 3D soccer simulation league. This is essentially a platform that replicates the RoboCup environment in simulation, serving as a virtual “gym” for AI techniques and robotic systems designed to play soccer.

Researchers at Yantai Institute of Technology in China and University of Rahjuyan Danesh Borazjan in Iran have recently developed a new technique that could enhance the ability of robots participating in soccer games to shoot the ball while walking. This technique, presented in a paper published in Springer Link’s Journal of Ambient Intelligence and Humanized Computing, is based on a computational approach known as the Q-learning algorithm.

“One of the most important goals of the teams participating in the RoboCup3D league is the ability to increase the number of shots,” Yun Lin, Yibin Song and Amin Rezaeipanah, the three researchers who developed the technique, wrote in their paper. “The reason for this importance is that superiority over the opponent requires a powerful and precise shot.”

Most techniques to generate shots in simulation are based on two approaches called inverse kinematics (IK) and point analysis. These are mathematical methods that can be used both to create computer animations and in robotics to predict the joint parameters required for a robot to attain a given position or complete an action.

“The assumption of these methods is that the positions of the robot and the ball are fixed,” the researchers explained in their paper. “However, this is not always the case for shooting.”

To overcome the limitations of previously proposed methods, Lin and his colleagues created a new shooting strategy based on a Q-learning algorithm, which can enhance the ability of robots to shoot the ball while walking. Q-learning algorithms are model-free computational approaches based on reinforcement learning. These algorithms are particularly useful in instances where agents are attempting to learn how to optimally navigate their environment or perform complex actions.

“A curved path is designed to move the robot towards the ball, so that it will eventually have an optimal position to shoot,” the researchers wrote in their paper. “In general, the vision preceptor in RoboCup3D has noise. Hence, robot movement paramenters such as speed and angle are more precisely adjusted by the Q-learning algorithm. Finally, when the robot is in the optimal position relative to the ball and the goal, the IK module is applied to the shooting strategy.”

Lin, Song and Rezaeipanah evaluated their Q-learning algorithm in a series of experiments and simulations. Remarkably, they found that it allowed robots to shoot the ball while walking much better than robots in most teams participating in the RoboCupSoccer league and in Iran’s RoboCup3D league. Ultimately, it could thus significantly improve the performance of robots during RoboCup soccer games.

© 2021 Science X Network