METHODOLOGY
For the tests in Austria and Switzerland, connect’s partner for the network measurements, P3 communications, used two vehicles to test drive the chosen cities, towns and roads. In Germany, even four cars were used simultaneously. Each car carried a total of nine smartphones. For the voice tests, we used two Samsung Galaxy S7 per operator, accounting for six of the total of nine phones per vehicle. The remaining three smartphones per car were used for the data measurements which were deliberately based on a mix of devices: In one half of the cars (thus one car each in the alpine countries and two cars each in Germany) we also used Samsung Galaxy S7 for the data tests. In the other half, Sony Xperia XZ took over the job in order to reflect varying device technologies in our measurements and evaluations. This is especially applicable when it comes to data transmissions at the edge of the networks‘ performance such as using “3 Carrier Aggregation“ (the combination of three LTE carrier frequencies).
The walktest teams consistently utilized Samsung Galaxy S7 that were installed in trolleys and back packs with additional strong batteries.
The devices’ firmware was each operator’s current firmware version. If such software was not available, the most current firmware from the smartphone manufacturer was used.
Voice telephony
Voice services were measured with the smartphones performing calls alternating between the measurement cars (“mobile to mobile“). The walktest teams called a stationary counterpart for all voice tests.
Background data traffic was transmitted by one of the smartphones simultaneously to each call to reflect a realistic usage scenario. Audio quality was assessed by using the HD-Voice capable POLQA (Perceptual Objective Listening Quality Assessment) wide band scoring.
All devices were configured in “LTE preferred” mode. Thus, in the three German Networks as well as with A1 in Austria plus Swisscom and Sunrise in Switzerland, the modern Voice over LTE (VoLTE) service could be used. Within networks not yet supporting VoLTE, the smartphones were forced to switch to 3G or 2G technology, the so-called circuit switched fallback (CSFB).
Data connectivity
To assess cellular data performance a sequence of tests was executed. As a dynamic web-browsing test, each country’s top web sites (according to the Alexa ranking) were downloaded in the so-called live web-browsing test. Additionally a static web site was tested, the industry standard ETSI (European Telecommunications Standards Institute) “Kepler“ reference page. HTTP downloads and uploads were performed with 3 MB and 1 MB files, simulating small file transfers. The networks’ peak performance was tested with a seven second download and upload of a single, very large file.
The Youtube measurements performed on the smartphones considered the “adaptive resolution“ feature of this video platform. In order to offer a persistent video experience, Youtube adapts the video streams‘ resolution dynamically to the bandwidth that is currently available. Our scoring therefore considers the success ratio, the time until the playback starts, the percentage of video playouts that take place without interruptions as well as the videos‘ average resolution or line number count respectively.
Indoor and train measurements
The walktests consisted of the same routines combining voice and data measurements. For this effort, the teams measured in so-called “areas of interest“ with a distinctive visitor frequency like train stations, airport terminals, buildings such as coffee shops and museums, but also in public transport. Travelling from city to city allowed the assessment of cellular network quality within the long distance trains.
Logistics
The tests were performed in Austria, Germany and Switzerland around the same period of time (Germany: October 11 – 30, 2017; Austria: October 9 – 27; Switzerland: October 7 – 27). All drivetests and walktests were done between 8 AM and 10 PM. During the drivetests, two cars were present in the same cities, but on different routes to avoid any interference of one car’s measurement by the other car’s.
On the connecting roads, both vehicles drove a given route, but followed each other at a small temporal and spatial distance. At each location, the test smartphones had to share the networks‘ bandwidth with normal customers. This represents the usual competition for the limited resource of the networks‘ available radio frequencies.
In Germany, the measurements included 19 larger cities and 28 smaller towns, while the walktests frequented ten cities. In Austria, the drivetests covered 11 big cities and 20 smaller towns, the walktest team visited seven cities. In Switzerland, the test route included 18 big cities and 35 smaller towns with the walktests conducted in also seven cities. Travel between the cities mainly used highways, but smaller state and county roads were driven as well. For each connect test, P3 communications follows a well-defined process to generate four independent and representative city and route plans. The connect editors then choose randomly one of these four alternatives.
Test efforts and results
Overall, 23,000 km were driven for the connect P3 mobile network test in 2017. In Germany, the approximately 9,600 km of driven routes alongside the cities and areas visited represent 14 million inhabitants, which equals around 17 per cent of Germany’s population.
Austria was measured by driving 5,000 km covering about 3 million inhabitants (approximately 35 per cent of the Austrian population). In Switzerland, the test teams drove approximately 8,460 km, covering 2 million people representing around 25 per cent of the Swiss population.
Certainly, a huge effort but necessary to gain the required statistical relevance and confidence in the test results.
Scoring
The results of the voice tests contribute 40 per cent of the total score, those of the data tests make up 60 per cent. For the overall result we apply a 1000 point scheme in order to represent sufficiently detailed results. Moreover, this scheme allows us to better compare the results of network tests that we have conducted in different countrie.
Professional and critical: Bernd Theiss, head of test and technology at connect (on the left), and Hakan Ekmen, managing director of P3 communications (on the right).
Each box was housing four smartphones which allowed the simultaneous testing of four mobile operators.
Fairness and transparency
In the preparation of this year’s network test, an anonymous writer tried to cast doubt over our neutrality and methodology in a letter with pretended insider information. The accusations turned out to be baseless.
Once again, in the preparation of this year‘s network test, connect and P3 met in early 2017 in order to define the conditions and parameters for the test. We identified changes to our methodology and test criteria, adapting them to the current state of technology. We defined the timeframe as well as a preselection of smartphone models that we intended to use for the measurements. Then, we communicated these preliminary definitions in advance to the CTOs of the network operators. In this process, we appreciate feedback about aspects like suitable tariffs or the firmware versions of the smartphones used for the measurements.
For each suggestion, we carefully consider whether it is technically justified or whether it tries to enforce measurement conditions that would favour the contenders’ own network. We understand these typically intense discussions to prove the high relevance that the operators assign to our network test.
Anonymous allegations of manipulations
Some attempts to put the neutrality and transparency of our measurement methods into question, however, originated from unknown sources. In the forefront of this year‘s network test, connect, P3 and several operators in one of the tested countries received an anonymous letter, accusing the winner of last year‘s network test in the respective country of manipulation. The pretended insider asserted the operator in question to have illegally increased the transmitting power of its cellular base stations. This accusation may match how a non-tech-savvy person imagines a possible cheating. In fact, it soon turned out to be completely baseless as the insinuated procedure would massive interfere with the delicate interplay of neighbouring cellular base stations. Now there is a strong suspicion that a stranger tried to exploit our network test in order to impair one of the operators.
Hannes Rügheimer, connect author
conclusion
In the context of adapting our test methods and the scaling of our points to the ongoing technical development, we deliberately and regularly increase our thresholds and requirements. If already very good candidates then succeed in still improving, we are even more happy because this demonstrates that our demanding network test in fact contributes to improving the mobile networks – to the benefit of all customers.
Fierce fighting in Switzerland resulted in a tie that was a surprise for us as well. With “outstanding“ performances (according to our grading), both Swisscom and Sunrise share the first rank this time. This is good news, equally for Swiss customers as for both operators. Salt somewhat falls back compared to the previous year due to weaker results in the telephony tests – but still achieves the grade „good“.
For many years now, our network test is the de-facto industry standard thanks to its sophisticated methodology and continuous adaptions to the technological development. In the future, additional crowdsourcing will contribute to portray the networks’ quality from a customer perspective even more accurately, over longer periods and over a wider geographical area.
