METHODOLOGY

The measurements in Germany took place from 22.10. to 4.11.2021, in Austria from 9.10. to 18.10.2021 and in Switzerland from 21.10. to 30.10.2021. connect‘s network test partner umlaut sent four measurement vehicles per country, each one equipped with six smartphones. 

Two Samsung Galaxy S21+ were used for each network operator: one took the voice measurements, another the data measurements. For the voice and data measurements, the smartphones were set to “5G preferred“ – so wherever supported by the network, data tests took place via 5G.

In addition to the drivetests, two walktest teams carried out measurements on foot in each country – in zones with heavy public traffic such as railway ­stations, airport terminals, cafés, museums and public transport. The walktest programme also included journeys on long-distance railway lines. Also for the walktests, two Samsung Galaxy S21+ were used per network operator – one for voice and one for data tests in “5G preferred“ mode. The walktest teams transported the smartphones in backpacks or trolleys equipped with powerful batteries. The firmware of the test smartphones corresponded to the respective original network operator versions. If ­there was no such version, we used the respective regional “open market“ firmware.

Logistics

The drivetests and walktests were conducted between 8 am and 10 pm. During the drive tests, two vehicles were in the same city, but not in the same place, so that one car would not falsify the measurements
of the other. On the connecting roads, two vehicles each drove the same routes, but one after the other with a small temporal and spatial distance. 

In Germany, the drivetests took place in 22 large cities and 25 small towns, the walk tests in 11 cities. Thus, about 14.8 million inhabitants were ­covered, which corresponds to about 17.8% of the German population. The drive tests covered about 13 196 km.

In Austria, the testers drove through 9 large and 15 small cities and covered about 5580 km. In addition, there were walktests in six cities. Thus, a total of about 3.2 million inha­bitants (approx. 35.8% of the population) were covered.

The drive tests in Switzerland led through 29 large and 21 small cities, the walktests took place in eight cities. The test route in Switzerland was about 6400 km long, the measurement campaign covered about 2.3 million inhabitants (26.3% of the population). 

For the selection of the test ­routes, umlaut created four ­different proposals for each country, from which connect blindly selected a route.

Voice connections 

The telephony measurements took place from vehicle to vehicle (“mobile-to-mobile“). The smartphones of the walktest teams made calls for the speech tests with a stationary (smartphone) counterpart.

To ensure realistic conditions, data traffic was handled simultaneously in the background. In the process, we also recorded MultiRAB connectivity: the use of several “radio access bearers“ provides data connections in the background of the voice calls.

The trans­mission quality of the transmitted voice samples was evaluated using the POLQA wideband ­method suitable for HD voice. “VoLTE preferred“ was confi­gured on all phones – from 5G, the phones thus fell back to ­telephony via LTE.

Data connectivity 

During the data measurements, several popular live pages according to the Alexa ranking were downloaded (dynamically). In addition, the ETSI (European Telecommunications Standards Institute) reference page known as the Kepler page was used (statically). In addition, 10 and 5 MB files were downloaded and uploaded, respectively, to determine the performance for smaller data transfers. In addition, we determined the data rate within a 7-second period when uploading and downloading large files.

The Youtube measurements take into account the “adaptive resolution“ of the video platform: Youtube dynamically adjusts the played resolution to the available bandwidth. The rating therefore takes into account the average image resolution or number of lines of the videos. In addition, the video rating is based on the success rate, the time until playback starts and the proportion of video playbacks that went through without interruption.

Crowdsourcing

In addition, the results of the crowdsourcing analyses carried out by umlaut constitute 25% of the overall rating. This not only checks which portion of the networks‘ maximum performance actually reaches each user – the end devices and tariffs selected by the users also have an effect.

For this purpose, the samples collected in all three countries from mid-May to the end of ­October 2021 were evaluated. 

Around 1.7 billion individual ­readings were analysed for Germany. This covers 99.8% of the population. For Austria, umlaut evaluated around 262 million samples, which also corresponds to 99.8% of the population. The figures for Switzerland: around 260 million samples correspond to a coverage of 99.9% of the Swiss population.

To obtain the data basis for these analyses, thousands of popular apps record in the background whether there is contact with the network, which network technologies are available and at what data rate and latency the transmission takes place – always provided that the user has ­consented to the completely ­anonymous data collection. These measured values are recorded in 15-minute intervals and transmitted to umlaut‘s servers once a day. Such reports contain only a few bytes, so they hardly burden the user‘s data volume.

Broadband Coverage

In order to assess the Coverage Reach, the test area is covered by a grid of 2 x 2 km tiles (“Evaluation Areas“ or EAs for short). A minimum number of users and measured values must be available for each EA. 

For the evaluation, umlaut awards one point per EA if the network under consideration ­offers 3G coverage. Three points are awarded if 4G or 5G is avail­able in the EA. The number of points achieved in this way is then divided by the total number of points that can be achieved (three points per EA in the “common footprint“ – i.e. the area of the respective country covered by all tested providers). 

In addition, we look at the ­Coverage Quality. It relates the percentage of EAs in which a user had 4G or 5G coverage to all EAs in the common footprint.

A third KPI for broadband quality is Time on Broadband . It tells us how often an individual user had 4G or 5G reception in the period under consideration – regardless of the EAs in which the samples were recorded. For this purpose, umlaut sets the samples that show 4G/5G coverage in relation to the total number of all samples. Important: The percentage values ­determined for all three parameters reflect the respective degree of fulfilment ­– they do not correspond to the percentage of 4G/5G mobile coverage in an area or in relation to the overall population.

Data rates and Latencies

The data rates determined are included in the crowd score at 30%, the latencies at 20%. The investigation of these para­meters is also carried out independently of the EAs and thus concentrates on the experience of each individual user. 

Samples that ­were recorded via WiFi or when flight mode was activated, for example, are filtered out by umlaut before further analysis. In order to take into account the fact that many mobile phone ­tariffs throttle the usable data throughput, ­umlaut has defined three application-related speed classes: Basic internet requires a minimum of 2 Mbps, HD video requires 5 Mbps and UHD video requires 20 Mbps. For a sample to be ­valid, a minimum amount of data must also have flowed in a 15-minute period. Similarly, the latency of the data packets is also assigned to an application-related class: Roundtrip times up to 100 ms are sufficient for OTT voice services, less than 50 ms qualify a sample for gaming. 

In the evaluation, umlaut assigns the speeds and latencies determined in the sample to one of these classes. Basic internet then accounts for 55% of the data rate rating, HD video for 33.8% and UHD video for 11.3%. OTT voice services account for 55% of the latency rating and gaming for 45%.

conclusion

Despite stormy times, the ranking in the three countries which we tested here remains stable again – for the fourth year in a row. However, this primarily proves that all providers are making an equally strong effort to improve further. Without constant work on the network infrastructure, even merely maintaining the performance level from the previous year would not be possible. Even the wide-scale phase-out of the previous 3G networks last year did not cause any major negative impact on mobile coverage.

In Germany, we congratulate Deutsche Telekom on its eleventh consecutive test victory. The Bonn-based company also shows top performance in their 5G roll-out.

However, Vodafone was also able to clearly improve since the previous year, achieving the largest increase in points in the comparison of competitors and its best result to date in our mobile network test. In addition, the Duesseldorf-based company has significantly increased its 5G coverage. Keep it up!

Telefónica also recorded a ­significant increase in points compared to the previous year, also achieved its best result to date and has now clearly arrived in the top league. If now what this provider is already showing in terms of performance in the big cities – especially in terms of 5G roll-out – also continues in the rural areas, this upward trend should also continue in the future.

Since Austria and Switzerland already ranked at top levels in previous years, increases are even more difficult to realise ­there. Accordingly, they happen only in single digits in Austria, and the Hutchison brand Three even loses a few points. 

In the Alpine republic, Magenta is the shining winner for the fourth time in a row – in all three test categories, the provider keeps its pursuers at distance and also shows respectable performance in terms of 5G. But A1 also achieves a very good result and is particularly convincing in the 5G roll-out. Three also performs very well and essentially maintains its performance level from the previous year.

In Switzerland, a very strong Swisscom takes the top spot – again for the fourth time in a row. The provider once again showed a significant increase compared to last year and is also stepping on the gas with 5G. But Sunrise also offers outstanding services and is also making particularly great progress in the roll-out of 5G. The smallest Swiss operator, Salt, maintains its previous year‘s level. And where there is already 5G available in the Salt network, it is also already delivering high performance.