FIFA Quality Programme for
VAR
Test Manual
1 December 2023 | Version 2.0
Contents
1. Introduction .................................................................................................................................... 2
2. Background and approach ............................................................................................................. 2
3. Testing and evaluation................................................................................................................... 4
3.1. Synchronicity test .................................................................................................................... 4
3.1.1. Test setup.......................................................................................................................... 4
3.1.2. Methodology .................................................................................................................... 5
3.1.3. Results and evaluation ..................................................................................................... 5
3.2. Latency test ............................................................................................................................. 6
3.2.1. Test setup.......................................................................................................................... 6
3.2.2. Methodology .................................................................................................................... 6
3.2.3. Results and evaluation ..................................................................................................... 6
3.3. Video quality test.................................................................................................................... 7
3.3.1. Test setup.......................................................................................................................... 7
3.3.2. Methodology .................................................................................................................... 7
3.3.3. Results and evaluation ..................................................................................................... 8
FIFA Quality Programme for VAR | Test Manual | Version 2.0 1
1. Introduction
Video Assistant Referee (VAR) was approved by IFAB to be used, in accordance with the Laws of the
Game, to support referees in case of a clear and obvious error. VARs who have access to all broadcast
feeds may assist the referee team in the following cases:
• Goals
• Penalties
• Red card offences
• Mistaken identity
When it comes to implementing a VAR system, three key components must be considered: the system
setup, the basic performance requirements of the VAR system, and the training of VAR operators. The
FIFA Quality Programme has developed a testing and certification scheme which focuses specifically
on the basic performance requirements of a VAR system. This manual is solely designed to provide
technical information on the underlying VAR technology and should be understood as one part of the
comprehensive Implementation Assistance and Approval Programme (IAAP) processes.
2. Background and approach
To establish technical guidelines for the approval of VAR systems a research project was undertaken
to identify key areas of technical importance for VAR systems. The aim of this research was to establish
objective test methods to quantify the relevant technical factors and set appropriate limits that ensure
that an approved system is fit for use.
Initial discussions identified various challenges linked to coding, decoding, synchronizing, re-formatting
broadcast feeds and the processing of images. In this process, three measurement points (MP) were
identified, see Figure 1. MP 0 is where the camera signals leave the Outside Broadcast (OB)
van/broadcast provider before entering the Video Operating Room (VOR), MP 1 is located after the
video server in the VOR (i.e. the VAR system) and MP 2 is where the video is sent back to the OB
van/broadcast provider.
Figure 1: Schematic overview of the VAR setup. Three measurement points (MP) are indicated as MP0, MP1 and MP2, for
the evaluation of latency, synchronization, and video quality.
FIFA Quality Programme for VAR | Test Manual | Version 2.0 2
Various challenges were identified along this workflow:
• Measurement of time synchronicity of broadcast images between different cameras (immensely
important for offside decisions) at MP1.
• Conversion and integration of different formats (1080, ultra-motion cameras, varying frequencies
& formats) and image sources into a single system and the quality of the resulting output at MP1.
• Measurement of absolute latency of processed images vs. “live” feed at MP1.
• Measurement of the output video quality from a VAR system back to the broadcaster for
transmission on air at MP2.
Due to these challenges, the basic performance requirements have been identified as the following:
• Synchronicity of the video feeds.
• Latency of VAR system video feeds:
o VAR live feed vs. broadcasting video feeds.
o VAR live feed vs. VAR replay monitor feeds.
• Objective video quality assessment of the output of VAR systems.
Following these basic performance requirements, the test methods described in this handbook include:
1. Measurement of the synchronicity of feeds by analysis of the content in the video feeds.
2. Measurement of the latency of feeds by using a sample video which is ingested through the VAR
system.
3. Measurement of the video quality of VAR system feeds using sample videos of known and pristine
quality, with varying content.
FIFA Quality Programme for VAR | Test Manual | Version 2.0 3
3. Testing and evaluation
3.1. Synchronicity test
VAR providers must be able to ensure that the camera feeds they are displaying to the VAR are in
synchronisation. This is particularly important when checking offside situations as it is essential that
the feed which captures the ball being played and the feed where the offside analysis is conducted,
are using the same moment in time. This test uses a stroboscope with repetitive flashes and a technical
set-up to identify in which frame the flash is visible in each camera, therefore identifying existing
synchronicity issues.
3.1.1. Test setup
Figure 2: Synchronicity test at MP0 and MP1 [blue: broadcaster, green: VAR provider, orange: test institute].
The following equipment shall be used for the assessment of synchronicity between video feeds:
• 4 HD/3G SDI cameras capable of filming in both 1080i and 1080p
• An OB van that generates a quad split
• An SDI signal representing the MP0 interface
• VAR replay monitor with quad-split playback
• An SDI signal representing the MP1 interface
• An HD/3G SDI to analogue video converter
• An analogue frame synch extractor
• Two sets of HD/3G SDI grabbing equipment
• Two laptops for recording the SDI-signals
• A stroboscope, which is required to be adjustable in intensity and in delay between the trig signal
to the stroboscope and the flash.
FIFA Quality Programme for VAR | Test Manual | Version 2.0 4
3.1.2. Methodology
The test set-up (Figure 2) is made up of the stroboscope which is pitch side and 4 HD/3G SDI cameras
which have a clear line of sight to the stroboscope. Before the test starts, the VAR provider is given a
10-minute period in which they must use their own method of verifying whether the cameras are in
synch and adjust the feeds accordingly on their system. Once this has been done, a flash rate of 1/64
of the frame rate is selected, and flash intensity is adjusted for a suitable flash strength in all video
feeds. A 5-minute long video of the VAR quad-split feed is recorded with the video sampling
equipment. The recording is split into 2 different sections, in the first section (2.5 minutes) the
stroboscope has no flash delay whilst in the second section (2.5 minutes) the stroboscope has a delay
of half a frame. The section which has the best data will be used. The video analysis algorithm reads
the captured files and describes in which field/frame the flashes occur for the four video feeds in the
VAR quad-split recording. Camera feeds are provided with Gen-lock by the broadcaster.
3.1.3. Results and evaluation
The synchronicity is evaluated by identifying a reference camera for each flash and measuring the
difference in frames between the reference camera flash and the same flash in the other cameras. The
reference camera is the camera in which the flash occurs first, with the lowest frame number. The
individual synchronicity values for each flash are calculated by taking the difference in frames between
each camera and the reference camera. To pass the test no individual synchronicity value may not be
greater than 3.
An example table is included below which illustrates the reference camera for the specific flash, the
fields/frames in which the flash occurs in each camera, and the difference in frames between the
reference camera and the other cameras.
Camera 1 Camera 2 Camera 3 Camera 4 Pass / Fail
Flash 1 37
38 39 40 Pass
Field/Frame (reference)
Flash 1
Difference in frames 0 1 2 3 Pass
compared to reference
… … … … … …
Flash N 37
38 38 41 Fail
Field/Frame (reference)
Flash N
Difference in frames 1 0 1 4 Fail
compared to reference
Table 1: Example table for synchronicity analysis
VAR systems are expected to be able to adjust an imbalance in the synchronicity of received broadcast
feeds up to 7 frames. De-synchronised broadcast feeds with a difference greater than 7 frames should
be flagged to the broadcaster who needs to adjust the synchronicity.
FIFA Quality Programme for VAR | Test Manual | Version 2.0 5
The output of the camera feeds should be in the following order (Figure 3):
• Top left: main camera
• Top right: centre pitch camera
• Bottom left: 16-meter left
• Bottom right: 16-meter right
Figure 3: Synchronicity test | VAR replay
monitor
3.2. Latency test
As outlined in the IAAP technical requirements, the VAR live feed on the VAR replay monitor must
have a 3-second delay. Besides this, VARs have a live feed in the VOR. This is so that events can be
rewatched again without needing to rewind the footage.
This test is to ensure that the delay caused by the VAR system (ingesting, recording, and displaying
the feed) is taken into account when adding the 3 seconds to the VAR live feed as well as to make
sure that the delay between an event on the live broadcast feed and the VAR live feed in the VOR is
minimal.
3.2.1. Test setup
The following equipment shall be used for the assessment of the latency:
• A laptop with similar or higher specifications to the ASUS GX501 (with Intel i7-7700HQ@2,8GHz,
16 GB RAM), with Thunderbolt 3 connection.
• Software compatible with 3G SDI generator (1080p50) and sampling equipment for ingesting
(sending box) and recording (grabbing box) uncompressed video.
• 3G SDI generator (sending box) and sampling equipment (grabbing box).
• 3G SDI quad-split generator to generate a quad-split out of 4 incoming SDI-videos.
3.2.2. Methodology
The latency test is based on ingesting a known uncompressed video into the VAR system via SDI and
then record the live video and the 3-second delay video at the same time. This is done by feeding both
videos into to a quad-split generator together with two copies of the reference signal. The reference
video will have a counter on the top 50 and the bottom 50 lines of the picture, so these lines need to
be kept free from graphics from the VAR provider. A two-minute video will be recorded.
Figure 4: Latency test [green: VAR provider, orange: test institute].
3.2.3. Results and evaluation
The counters will be read by an analysis software tool and the time marks for the live feed and the 3-
second delay are then compared with the time marks for one of the reference channels, the second
reference is just for internal tests. All 6000 frames (p50) will be analysed.
FIFA Quality Programme for VAR | Test Manual | Version 2.0 6
• Both the live feed and the 3-second delay are recorded at the same time.
• No frames in the live feed are allowed to be outside the 0.5-second limit.
• No frames in the 3-second delay feed are allowed to be outside the 2.5 to 3.5 seconds limit,
and within the highest and the lowest measurement value of one system, a maximum
difference of 0.5 seconds is allowed.
• It remains at FIFA’s and the test institute’s discretion to assess whether the occurrence of
black, unreadable, dropped or repeated frames in the recording restricts the evaluation of
the test, resulting in a failed test.
3.3. Video quality test
The final test of the testing procedure ensures that the quality of the footage provided to the VAR is
of suitable quality. This ensures that the VAR has the necessary tools to assist the on-field referee.
3.3.1. Test setup
Figure 5: Set-up for ingesting test video into a VAR system (above) and set-up for recording test video from a VAR system
(below) [blue: broadcaster, green: VAR provider, orange: test institute].
The following equipment shall be used for the assessment video quality:
• A laptop with similar or higher specifications to the ASUS GX501 (with Intel i7-7700HQ@2,8GHz,
16 GB RAM), with Thunderbolt 3 connection
• Software compatible with HD/3G SDI generator and sampling equipment for ingesting (video
player) and recording (video recorder) uncompressed video
• HD/3G SDI generator and sampling equipment
3.3.2. Methodology
The video quality test is based on ingesting a known uncompressed video into the VAR system via SDI
and then replay this video from the VAR system and recording it when it is sent back via SDI or HDMI
(Figure 6). The videos of known quality are available in both 1080i50 and 1080p50 and should both
be tested. The test will not cover all manipulations that may occur when examining a VAR situation
e.g. zoom. It will compare the video quality degradation inflicted by the VAR system on video that has
been processed by the VAR system to be stored and played back, but have not changed in size, format,
scaling or resolution. Input of 1080i50 on SDI with deinterlaced output on HDMI will be compared
though.
• Ingesting
o Connect HD/3G SDI generator’s SDI output to SDI-input from VAR-system
o Send test video to the VAR-system using the video player.
FIFA Quality Programme for VAR | Test Manual | Version 2.0 7
• Playback and recording
o Connect HD/3G SDI sampling equipment’s SDI/HMDI input to the SDI/HDMI-output from
VAR-system
o Play test video in VAR-system
o Output of the recorded video in 1080p50 into a recording device provided by the test institute
• The output video is then assessed using the Video Multi-method Assessment Fusion (VMAF) a
perceptual video quality assessment algorithm developed by Netflix.
3.3.3. Results and evaluation
After a detailed exploration of possible analysis methods, the Video Multi-method Assessment Fusion
(VMAF) was selected as the base of the video quality test. The VMAF is a perceptual video quality
assessment algorithm developed by Netflix. VMAF follows a machine-learning-based approach to first
extract a number of quality-relevant features from a distorted video and its reference full-quality video,
followed by fusing them into a final quality score using non-linear regression.
The video quality will be evaluated on seven 14 s test videos. The evaluation will be done by comparing
the quality of each individual 14s video, before and after the ingestion. In order to avoid temporary
glitches affecting the results, the ingestion and grabbing will be repeated three times.
The requirements are to obtain:
• Average Mean Opinion Score (MOSi) ≥ 4 for each set of test videos
• Min (MOSi) ≥ 3 for each set of test videos
• The above two requirements should be fulfilled on two or more of the three sets
• The final measurement value will be given from the highest-scoring set
The MOS requirements will be verified by the following VMAF scores on the individual test videos.
• 1080p: Mean (VMAFi) ≥ 92 and Min (VMAFi) ≥ 85
• 1080i: Mean (VMAFi) ≥ 85 and Min (VMAFi) ≥ 75
• The above two requirements should be fulfilled on two or more of the three sets
• The final measurement value will be given from the highest-scoring set
FIFA Quality Programme for VAR | Test Manual | Version 2.0 8
