For several years, the best basketball league in the world, the NBA, has had complex systems of cameras installed in all sports halls to track all players on court. These data are processed by the Data Science teams of each club, and they analyse them thoroughly in order to improve the team’s performance by detecting individualized trends or behavioural patterns as a group (called “spacing”). These systems, supplied by companies such as STATS or Second Spectrum, are extremely expensive for the budgets available to European teams.

The complex systems of cameras installed in the sports halls of the NBA, to track all players on court, is too expensive for European teams

So, as Adrià Arbués, first author of the work explains, “European teams still don’t have this data analysis technology. The aim of our research was to design a player tracking system to in sequences taken from the broadcast camera (without involving any extra cost for materials)”. “It should be mentioned that no game in the last years was recorded under these conditions, so this method could potentially be used to obtain tracking data from matches already played”, he adds.

“The aim of our research was to design a player tracking system in sequences taken from the broadcast camera (without involving any extra cost for materials)”

A study involving the doctoral research by Adrià Arbués, which is being supervised by the researchers Coloma Ballester and Gloria Haro, members of the Image Processing Group (GPI) at the Department of Information and Communication Technologies (DTIC) at UPF. “In particular, the field of research for my thesis is the application of computer vision techniques and deep learning to analyse the content of sports videos (only basketball for now)”. The authors published their research at the 2019 ICAIS: International Conference on Artificial Intelligence in Sports, held in Paris in July, for which they received a “Best Paper Award”. 

Research applied to basketball based on deep learning techniques

Deep learning techniques as part of a broader set of machine learning methods based on data representation assimilation, can be used for countless things such as classification or prediction. Previously, in the world of image, there have been hundreds of databases that have been entered in an artificial neural network, or paradigm of that of machine learning and processing, to perform the task at hand. For this study, two existing models were used: an OpenPose network to detect people and their body parts, and a pre-trained VGG-19 network for image classification. 

For this study, two existing models of neural networks were used: OpenPose, to detect people and their body parts, and pre-trained VGG-19, for image classification

“The main problem for solving, once court segmentation has been performed and the different players have been detected, is to perform assignments between consecutive frames: in other words, which player is which as time passes”. “To do this, we have taken advantage again of the VGG-19 network, extracting the features learned by the model in an intermediate layer; the logical reasoning for this is the following: if the network is trained to classify, it means that the artificial intelligence model is learning discriminating features between classes, and these can be used to distinguish between different players of the same class. Once the similarity between players has been calculated using these features, an algorithm is used to minimize the cost of the assignments”, Arbués explains.

Following this methodology, the authors assert that “having annotated over 10,000 boxes on the original frames, results can be extracted by comparing the manually entered position and the position predicted by the model implemented. The success rate of detections stands at around 95%, while frame-to-frame assignations are approximately 70% accurate (according to the MOTA metric), a commendable figure given that up to 10 players and three referees must be properly tracked at a low frame rate (if resources are limited)”, Arbués continues.

The article also includes a study with various comparisons on what happens if visual features (colour) are added regarding the characteristics of deep learning, or what is the effect of stabilizing the video stream, and how accuracy changes depending on which intermediate layer is used to extract features, or which body parts are more discriminatory, etc.

As for the future of this field of research, despite player tracking being remarkably accurate, this work cannot yet serve teams in order to draw conclusions on performance. “First, we must improve the accuracy of the method, and we are currently working on re-training another network, changing the training sets and the weights of the different layers; then we will have to perform tracking conversion and transform the pixels we see on screen to real positions on the court. In the long term, it would be interesting to see if we can use the content of the different parts of the body to perform recognition of the players action or matchplay”, Arbués concludes.

This project has been possible thanks to the database of basketball games provided by FC Barcelona, the club for which Adrià Arbués has been collaborating since 2017: “I coached last year and this season I’ll start to specialize only in the field of data science”, he says.

Adrià Arbués-Sangüesa, Coloma Ballester, Gloria Haro (2019), “Single-Camera Basketball Tracker through Pose and Semantic Feature Fusion”, ICAIS 2019 : International Conference on Artificial Intelligence in Sports, Best Paper Award, Paris (France), juliy.