How to do a correct pitch homography for football?

The homography module is a core component of RiseUp. This article explains how we solved pitch homography within the constraints of our video-based data extraction pipeline, with the goal of providing a high-level overview without getting into the underlying (but rather cool) mathematics.

Homography (Comp. Vision): A mapping between two 2D planes represented by a non-singular 3×3 matrix H, such that for a point in homogeneous coordinates x, its correspondence x′ satisfies x′=Hx (up to scale).

Practically, we're trying to perform a projective transformation that maps points from the plane of the camera video (plane 1) to the corresponding sections in a canonical 2D pitch (plane 2), preserving lines but not necessarily distances or angles. This allows us to extract (x, y) coordinates of pitch objects from video, from which we derive metrics such as distance and speed.

Limitations

• In lower leagues, pitch dimensions, camera parameters, angle and placement vary
• No real-time processing; computation is triggered on demand upon match upload.
• Markings can be hard to distinguish
• Variable lighting conditions.
• Partial pitch visibility from the main camera angle.

Assumptions

• Clubs play both home and away matches
  • For home matches, we can calibrate homography since the camera is fixed and its specifications are known
  • The canonical pitch can use actual pitch dimensions
  • For away matches, we use known pitch dimensions when available; otherwise, we default to standard FIFA dimensions

Let Claude do it

We're in the great revolution of AI, so we might as well use it. We tried to let Claude do it. It suggested using RANSAC to solve homography, trained on the SoccerNet-2023 dataset. For two weeks, my approach was to the best prompt engineering skills I have acquired from hours of vibe coding to get Claude to do it right. Unfortunately, this attempt was unsuccessful, homography projections were going everywhere. After 2 weeks of trial, testing different RANSAC variations, Hough transforms, I found the problem. Claude didn't understand partial observability, and tried to map the entire canonical pitch from a partially visible pitch which makes the solution space underdetermined. RANSAC needs correct inliers, but there were no keypoints at this point to get correct estimations.

Do some research

Claude failed, so we have to take matters into our own hands.

A first implementation: regression-based homography

I found Homography Estimation using Deep Learning for Registering All-22 Football Video Frames, a 2017 masters degree project from Hampus Fristedt, a student in the university of StockHolm, proposing a Convolutional Neural Networks (CNNs) based homography estimation that we adapted as follows:

• Using ResNet18 with a 4-point parameterization (Kanade), flattened into a 1D vector
• We used RANSAC instead of least squares in our findHomography() function from OpenCV.
• We used Mean Squared Error loss, and the Adam W optimizer.
• We have a segmentation model, SegFormer, which runs well with 1-2m error, used for segmenting the pitch. We decided to use it as a spatial constraint to avoid homography projections outside of the pitch, as we do with our player detection modules.
• We kept RGB augmentation and cropping.

This method finally got us homography projections that are in the field instead of projecting points in the stands, but the model kept collapsing toward the dataset mean, predicting the center of the pitch each time. That could be due to insufficient diversity in our dataset for global mapping, a lack of geometric constraints, as CNN-based regression learns a global mapping prior instead of explicit geometric constraints, and a center bias in the regression loss calculations the loss function favors mean solutions to minimize error. I probably didn't implement it correctly either. We made some progress, but not enough to make it usable.

Homography is correspondence, not regression

Homography requires explicit correspondences, not implicit inference. If we can detect the lines and intersections, we can then compute the homography instead of letting a model learn it. We call these lines and intersections keypoints. We went from:

image → homography

to:

image → keypoints → homography

Reintroducing geometry into the pipeline

I did some additional research, and landed on Narya, a repository containing the implementation of the following paper: Evaluating Soccer Player: from Live Camera to Deep Reinforcement Learning which provides a full implementation for player detection and homography, forming a foundation for deriving insights from match video. We're building a custom pipeline with different limitations, but we're also going to take this as a basis. It did suggest using keypoint detection, which confirms our thinking. Digging into keypoint detection, I found another paper: Enhancing Soccer Camera Calibration Through Keypoint Exploitation. Theoretically, these papers should help us calibrate our camera on setup, and have robust homography estimations. On the subject of camera calibration, I also found another paper: TVCalib: Camera Calibration for Sports Field Registration in Soccer, which suggested calibrating the camera without keypoint correspondences. Although the paper highlights clear advantages, it would introduce additional system complexity for minimal gain. I thus decided to keep the keypoint-based approach for now.

A hybrid geometric-learning system

In practice, pitch homography is not about predicting a global transformation, but about recovering it from partial, reliable correspondences under strong geometric priors.

First, we need to work on the keypoint detection module using field annotations. We used HRNet, taking RGB images along with the segmentation mask as input, and producing Gaussian heatmaps as output. The idea is to give the keypoint projections that aren't within our mask a lighter weight to prompt the model into finding keypoints within the pitch.

For training, we used MSE for loss, with the AdamW optimizer, activation via a Sigmoid per channel, and subpixel refinement. Data augmentations are triggered randomly at a certain probability, including color augmentation, gaussian noise and horizontal flip, with the goal of enhancing the model's robustness.

As for the dataset, we used the datasets used in Narya, along with the SoccerNet-v3 dataset, which contains around 34,000 football images from 400 games. We then used a web-based tool to reannotate the images from both datasets to a normalized annotation scheme to map it effectively to our canonical pitch coordinates. We can then use RANSAC + DLT to estimate the homography, resulting in the following hierarchy:

1. High-confidence intersections -> RANSAC since they are high-information keypoints
2. Extended keypoints (tangents, etc.) since they have lower precision but increase robustness
3. Pitch priors as a regularization layer

Calibration

General calibration

We added a simple camera smoothing and frame-to-frame stabilization to enhance the camera's input for actual video uploads. It would be good to mention that frames are extracted using FFMPEG, and only a subset of frames is processed to reduce computational load.

For home matches

The idea of calibration for home matches is to precompute homography using aggregated keypoints to enhance the estimation's accuracy and lessen its computing demands. When processing a home match, or a match which played on a pitch on which our pipeline has already ran, we can store that homography and use it to refine the output of the pipeline.

Results

We obtain a robust, functional homography pipeline, using keypoints + RANSAC, Gaussian heatmaps, prioritizing intersections, and adding our calibration and refinement systems. We find ourselves with ~2m reprojection error (mean) after running the pipeline on a full 90 min, 1080p National 2 (French 4th tier of the football league system), with around 92% frames getting successfully calibrated.

mermaid diagram

graph TD

A[Frame] --> B[SegFormer]
A --> C[Keypoint Detector]

B --> E[Homography Module]
C --> E

E -->|Mode: Calibrated| F[Stored Homography]
E -->|Mode: Online Estimation| G[RANSAC + DLT]

F --> H[Projection]
G --> H

H --> J[Analytics Layer]

Current limitations

Our pipeline is effective, but it could be better. We only have parts of the field being visible in the images we focused on. Matches in lower divisions have extreme camera angles, and the pipeline has a strong reliance on keypoint detection quality. We also don't have any full 3D modeling yet.

I already found a paper treating the subject of 3D modeling: WorldPose: A World Cup Dataset for Global 3D Human Pose Estimation. We could also refine our keypoint detection system by adding:

• 3D planar extensions, using goal posts to construct a 3D plane
• a line detection module for detecting lines accurately
• a learned refinement layer for refining the data
• better temporal modeling.
• using multiple camera angles if available to refine homography

These additional modules weren't implemented for simplicity's sake. We will be evaluating the trade-offs of adding modules in our pipeline, and implementing those which seem most worthwhile.

In the end, the winning approach wasn't more learning, but better problem formulation, by explicitly separating perception from geometry rather than relying on end-to-end learning.