The tests in Germany took place from October 1st to November 1st, 2019, the ones in ­Austria from October 3rd to 18th, and the ones in Switzerland from October 10th to 30th, 2019. For each country, connect‘s partner for the network measure­ments, umlaut, used two ve­hicles for drivetesting the cho­sen cities, towns and roads. Each car ­carried a total of six Samsung ­Galaxy S9 smartphones. One per operator was used for the voice tests and another one for the ­data measurements. In ­addition to the drivetest, in each country a walktest team took measurements by foot. For this effort, the teams visited ­so-called “areas of interest“ with a strong visitor frequency like train stations, airport terminals, coffee shops, museums and also local public transport. Travelling from city to city allowed the ­assessment of cellular network quality within long ­distance trains.

The walktest teams also ­utilised Samsung Galaxy S9 smartphones that were installed in trolleys and back­packs with additional strong batteries. The devices’ firmware was each operator’s current firm­ware version. If such software was not available, the most current firm­ware from Samsung was used.

Logistics 

All drivetests and walktests were ­done between 8 am and 10 pm. During the drivetests, two cars were present in the same cities, but on different routes to avoid any in­terference of one car’s measure­ment by the other car’s. On the connecting roads, both vehicles drove a given route, but followed each other at a small temporal and spatial distance.

In Germany, the measurements included 20 larger cities and 24 smaller towns, while the walktests frequented ten cities. The measurement cars drove a total of approx. 9,780 kilometers. With their drivetests, they covered about 14.4 million inhabitants which equals about 1715 percent of the German popula­tion.

In Austria, the drivetests covered 11 big cities and 20 smaller towns, the walktest team ­visited seven cities. Here, the vehicles covered about 6,010 km equalling 3.1 ­million inhabitants or 35.3 percent of the population.

In Switzerland, the drivetests covered 18 big cities and 33 smaller towns, the walktests took place in eight cities. The test route in Switzerland was about 6,590 km long, equalling about 2.2 million inhabitants or approximately 26.1 percent of the population. 

For the definition of the test routes umlaut generates four independent plans, from which ­connect then randomly chooses one.

Voice telephony

Voice services were measured with the smartphones performing calls alternating between the measurement cars (“mobile ­to­ mobile“). The walktest teams called a stationary counterpart for all voice tests. In the assessment of call setup ­times we also rate the so-called P90 value. Such values specify the thres­hold in a statistical distri­bu­tion, below which 90 percent of the gathered values are ­ranging. For speech quality, we publish the P10 value (10 percent of the values are lower than the specified threshold), because in this case higher values are better.

Background data traffic was transmitted by one of the smartphones simultaneously in order to reflect a realistic usage scenario. As a new KPI in 2019, we also evaluate the so-called Multi­RAB (Multi ­Radio ­Access Bearer) ­Connectivity. This value denominates whether data connectivity is available during the phone calls. Audio quality was assessed by using the HD-Voice capable POLQA wide band scoring. All devices were configured in “LTE preferred” mode.

Data connectivity 

To assess cellular data perfor­mance, top websites (according to the Alexa ranking) were dynamically downloaded. Additionally a static website was tested, the industry standard ETSI (European Telecommunications Standards In­stitute) Kepler reference page. HTTP downloads and uploads were conducted with 5 MB and 2.5 MB files, ­simulating small file transfers. The networks’ peak performance was tested with a seven second download and upload of a single, very large file. Youtube measure­ments considered the “adaptive reso­lution“ feature of this video platform. In order to offer a per­sis­tent video experience, You­tube adapts the video streams‘ reso­lution ­dynamically to the band­width that is currently avail­able. Our scoring therefore considers the success ratio, the time
until the playback starts as well as the ­videos‘ average resolution. 

Crowdsourcing

The results of the crowdsourcing analysis performed by umlaut contribute 20 percent to the total score. For this purpose, in all three countries samples collected from mid-May until end of October 2019 were evaluated.

For Germany, a total of 2.8 billion single measurement values from von 476,000 users was analyzed. This represents about 99.3 percent of the built-up area in urban surroun­dings and 91.5 percent of the rest of the country‘s bulit-up area.

For Austria, umlaut considered approx. 902 million values from a total of 99,800 users – representing a 100 percent of the built-up area in cities and 92.1 percent of the built-up area outside of them.

The figures for Switzerland: 41,400 users contributed 200 million samples. This covers a 100 percent of the built-up area in cities, and 88.4 percent outside of them.

The data base for these analysis is obtained by more than 800 popular apps. They log in the background whether there is a network connections, which mobile network technologies are available and what download and upload data rates can be achieved – provided that the user has before ­agreed to this completely anony­mous data collection. These values are gathered every 15 minutes and transmitted ­once a day to umlaut‘s servers. The reports only comprise of a small number of bytes so that they do not put a sub­stantial strain to the users‘ data volumes. Readers who wish to ­actively support our network test can do this by installing and using the connect app (see box below). However, it is only one of many apps con­tai­ning the described background functionality.

Network coverage

In order to determine the cove­rage with 4G, “data“ (3G and 4G) as well as telephony, umlaut ­applies a grid of 2 by 2 km tiles to the test area. These “evalua­tion areas“ are then sub-divided into 16 smaller tiles. In order to ensure the stastistical relevance of the results, each tile must deliver a minimum number of users and measurement data.

In our 2019 benchmark frame­work, we differentiate between a „Benchmark View“ and an „Own Network View“: For the Benchmark View, only those evaluation areas are considered for which we have determined valid results for all operators who are considered in the benchmark. In the Own Network View  an evaluation area will also be consi­dered if there
is none of the competitors present. Above that, we now distinguish urban and non-urban areas in our crowd evaluations – re­specting that the coverage with mobile ­services is usually higher in urban than in rural areas.

Data Throughputs

For each “evaluation area“ umlaut determines the maximum download and upload speeds reached by each participating user with 15 minutes time slices. Then, the maximum value of these slices with a 4 week grid is calculated, and of these eventually the P90 value (as explained earlier).

Network stability

Another aspect determined in the crowdsourcing are degra­dations or anomalies in the (data) network connectivity.
A sophisticated algorithm distinguishes simple loss of network coverage (such as in elevators, car parks or buildings with poor indoor reception) from actual degradations. Incidents that ­occur at night between 10 pm and 8 am are not considered.

Points will be deducted for the number of days and the number of hours with recognized service degradations.

conclusion

The fact that the rankings in all three tested countries did not change compared to the pre­vious year, proves again the very high level that the competition of the operators takes place on.

Because in order to make sure  that our network test stays the de-facto industry standard also in its 26th year, we regularly update or testing and evaluation methods to the advancing technological development. Part of this is that umlaut and connect raise our thresholds and requirements from year to year. If under these conditions really all operators manage to improve over their results from the previous year, as is was the case this time, we are particularly happy. For this proves that our demanding network test ultimately helps to improve the networks – to the advantage of all customers.

In Germany, a strong Telekom gains the overall victory for the nineth time in a row. The Bonn company even managed to once again improve its scores in the voice discipline and in the crowdsourcing. Also Vodafone which ranks second improved its results. But definitely the greatest leap forward in Germany is achieved by Telefónica/O2. ­Overall good measurement ­results but also good marks in the crowdsoucing show that the Munich-based operator made notable headway in the last year in integrating the formerly separate networks of E-Plus and O2. 

But the performance level which overall ranks below those from the two neighbouring countries and particularly the poor results of our measurement in the trains make quite clear that there is still much to do for the German operators.

In Austria, Magenta (the former T-Mobile Austria) takes the overall win again this year – and gains the top grade “outstanding“ for the first time. The effects of the very close run at the top are well felt by A1 Telekom – which also receives the grade “outstanding“ but ranks second at a distance ­of only four points to the test
winner. The smallest network operator in the Alp republic, Hutchison‘s brand Three, achieves the overall grade “very good“ which still designates the most distinct improvement over the previous year among the Aus­trian providers.

On a still somewhat higher level the neck-and-neck race in Switzerland is fought. But there, the result shows a somewhat ­larger distance. Swisscom defends its top position and gains the overall grade “outstanding“. Sunrise scores seven points ­behind but is also rated to be “outstanding“.  Salt, the smallest operator in Switzerland, achieves the grade “very good“ – but can be pleased about achieving the highest growth in points over the previous year.