The umlaut connect Mobile Benchmark in the United Kingdom com­prises of the ­results of extensive voice and data drive ­tests and walk tests as well as a sophisticated crowdsourcing approach. 

Drive testS AND Walk testS

The drive tests and walk tests in the UK took place between ­October 25th and November 15th, 2021. All samples were collected during the day, between 8.00 a.m. and 10.00 p.m. The network tests covered inner-city areas, outer metropolitan and sub­urban areas. Measurements were also taken in smaller towns and ­cities along connecting highways. The connecting routes between the cities alone ­covered about 1,365 kilometres per car – 54,62 kilometres for all four cars. In total, the four vehicles together have covered about 10,022 kilometres.

The combination of test areas has been selected to provide re­pre­sentative test results across the UK‘s population. The areas selected for the 2021 test ­account for 17.3 million people, or roughly 26 percent of the total popula­­tion of the United Kingdom. The test routes are shown on page 1 of this ­report, all visited cities and towns are listed in the box below.

The four drive-test cars were equipped with arrays of ­Samsung Galaxy S21+ smartphones for the ­simultaneous measurement of voice and data services.

VOICE TESTING

One smartphone per operator in each car was used for the voice tests, setting up test calls from one car to another („mobile-to-mobile“).
The walk test team also carried one ­smartphone per operator for the voice tests. In this case, the smartphones called a stationary
(smartphone) counterpart. The ­audio quality of the transmitted speech samples was ­evalua­ted using the HD-voice capable and ITU standardised so-called ­POLQA ­wideband algorithm. All smartphones used for the voice tests were set to “5G preferred“ mode. In addtition, they were set to “VoLTE preferred“. As Voice over 5G/Voice over New Radio is not yet supported in current 5G networks, this means that the devices would perform a fallback from 5G to 4G in order to establish voice calls.

In the assessment of call setup ­times we also rate the so-called
P90 value. Such values specify the threshold in a statistical distri­bu­tion, below which 90 percent of the gathered values are ­ranging. For speech quality, we publish the P10 value (10 percent of the values are lower than the specified threshold), because in this case higher values are better.

In order to ­account for ­typical ­smartphone-use scenarios during
the voice tests, background data ­traffic was generated in a ­controlled way ­through ­injection of data ­traffic (HTTP ­downloads). In the process, we also recorded Multi­RAB connectivity – the use of several “radio access bearers“ for the background data connections.

The voice scores ­account for 30 percent of the total results.

DATA TESTING

Data performance was ­mea­sured by using four more ­Galaxy S21+ in each car – one per operator. Their ­radio access technology was also set to 5G preferred mode.

For the web tests, they accessed web ­pages ­according to the ­widely ­recognised Alexa ranking.

In addition, the ­static ­“Kepler” test web ­page as ­spe­cified by ETSI (Euro­pean ­Tele­commu­­ni­cat­ions Standards Insti­tute) was used. In order to test the data ­service performance, files of 10 MB for download and and 5 MB for up­load ­were transferred from or to a test ­server ­located in the cloud. In addition, the peak data ­per­formance was tested in uplink and downlink ­directions by assessing the amount of data that was transferred within a ­seven ­seconds time period.

The Youtube measurements take into account the “adaptive resolution“ of the video platform: Youtube dynamically adjusts the played resolution to the available bandwidth. The rating therefore considers the average image resolution or number of lines of the videos. In addition, the video rating is based on the success rate, the time ­until playback starts and the propor­tion of video playbacks that went ­through without interruption.

All the tests were conducted with the best-performing ­mobile plan available from each operator. Data scores ­account for 45 percent of the total results.

CROWDSOURCING

Additionally, umlaut conducted crowd-based analyses of the UK‘s networks which contribute 25 percent to the end result. They are based on data gathered between calendar week 21 (end of May) until calendar week 44 (early November), 2021. 

In the process, a total of 2.7 billion samples from more than 672,600 users were evaluated. The area of the UK covered by these crowdsourcing analyses covers approx. 200,300 square km and 99.2 percent of the UK‘s built-up areas, which correspond to approx. 99.8% of the population.

For the collection of crowd data, umlaut has integrated a back­ground diagnosis ­pro­cess into more than 1000 diverse Android apps. If one of these applications is ­installed on the end-user’s phone and the user ­authorizes the background analysis, data collection takes place 24/7, 365 days a year. ­Reports are generated for every hour and sent daily to umlaut‘s cloud servers. Such ­reports ­occupy just a small number of bytes per mes­sage and do not include any ­personal user data.

This unique ­crowd­sourcing ­technology ­allows umlaut to ­collect data about ­real-world ­experience where­ever and when­ever customers use their smartphones.

NETWORK COVERAGE

In order to ­assess the “Coverage Reach“, the test area is divided by a grid of 2x2 km tiles (“Evaluation Areas“ or EAs for short). A minimum number of users and measured values must be available for an EA to be considered in the analysis.

For the evaluation, umlaut awards one point per EA if the network under consideration ­offers 3G coverage. Three points are awarded if 4G or 5G is avail- able in the EA. The number of ­points achieved in this way is then divided by the total number of ­points that can be achieved (three points per EA in the “common footprint“ – i.e. the area of the ­country covered by all tested ­mobile network operators).

In addition, we look at the “Coverage Quality“. It puts the percentage of EAs in which a user had 4G or 5G coverage in relation to all EAs in the common footprint.

A third KPI for broadband quality is “Time on Broadband“. It tells us how often an individual user had 4G or 5G reception in the period under consideration – regardless of the EAs in which the samples were recorded. For this purpose, umlaut sets the samples that show 4G/5G coverage in relation to the total number of all samples. Important: The percentage values determined and published for all three paramaters reflect the respective degree of fulfilment – they do not correspond to the percentage of 4G/5G mobile coverage in an area or in relation to the overall population.

DATA RATES AND LATENCIES

The data rates determined are included in the crowd score at 30%, the latencies at 20%.

The investigation of these parameters is also carried out independently of the EAs and thus concentrates on the experience of each individual user. Samples that were recorded via WiFi or when flight mode was activated, for example, are filtered out before further analysis. 

In order to take into account the fact that many mobile phone
tariffs throttle the usable data ­throughput, umlaut has defined three application-related speed classes: “Basic internet“ requires a minimum of 2 Mbps, “HD video“ requires 5 Mbps and “UHD video“ requires 20 Mbps. For a sample to be valid, a minimum amount of data must also have flowed in a 15-minute period. 

Similarly, the latency of the data packets is also assigned to an application-related class: Roundtrip times up to 100 ms are sufficient for “OTT voice services“, less than 50 ms qualify a sample for “gaming“.

In the evaluation, umlaut assigns the speeds and latencies determined
in the samples to one of these classes. “Basic internet“ then accounts for 55% of the data rate rating, “HD video“ for 33.8% and “UHD ­video“ for 11.3%. “OTT voice“ services ­account for 55% of the latency ­rating and gaming for 45%.

PARTICIPATE IN OUR CROWDSOURCING

Everybody interested in being a part of our global crowdsourcing panel and obtaining insights into the reliability of the mobile network that her or his smartphone is logged into, can most easily participate by installing and using the
“U get“ app. This app exclusively concentrates on network analyses and is available under http://uget-app.com.

“U get“ checks and visualises the current mobile network performance and contributes the results to our crowdsourcing platform. Join the global community of users who understand their personal wireless performance, while contributing to the world’s most comprehensive picture of mobile customer experience.

CONCLUSION

EE wins for the seventh time. Vodafone maintains the second place. Three takes back the third rank from O2, showing massive score gains. But also O2 mana-ges to improve over our previous benchmark in the UK.

The overall winner of the 2021 umlaut connect Mobile Benchmark in the UK is EE – for the seventh time (in 2016, EE shared the first place with Vodafone). EE scores best in the Data and Crowd categories, ­while Vodafone has the lead in Voice. Still, the second-­ranking Vodafone shows also strong results in the other categories.

Three manages to distinctly advance its score in comparison to our previous UK benchmark, improving in all three test categories. In doing so, Three also wins back the third rank from O2, which had taken this position in our previous UK benchmark. Although ranking fourth this time, O2 still achieves a clear score gain and improves its results in the Voice and Data categories.

In London, EE leads at a gap of ten points ahead ofVodafone, while O2 scores a little higher than Three. In our city comparison, EE leads in seven out of ten considered larger UK cities. Vodafone is a local champion in Belfast and Liverpool, while O2 takes the lead in Bradford.

In terms of 5G rollout, all UK operators already show good coverage in the cities. EE offers the highest 5G coverage in all tested scenarios, while in the 5G high band, Three achieves the highest average and maxi­mum data rates in all scenarios except for the walk tests conducted in the big cities.