TESTING METHODOLOGY

The P3 connect Mobile Benchmark in the Netherlands took place from February 16th until March 10th, 2018. All samples were collected between 8am and 10pm. The network tests covered 21 cities, of which 16 are counting more than 100 000 inhabitants, and 31 smaller towns. Furthermore, our test routes included 3,600 kilometres of connecting roads. Additionally, a walktest team visited seven larger cities. The team took the trains between the cities and also into more rural areas of the country, performing voice and data tests on the railways as well.

This combination of test areas has been carefully selected to provide a significant series of test results covering the Dutch population. The areas chosen for the 2018 test account for about 5.8 million people, or 34 per cent of the total Dutch population.

P3 conducted the tests with two drive-test cars, equipped with arrays of Samsung Galaxy S7 Cat 9 smartphones (voice) as well as a mixed allocation of Samsung Galaxy S7 and Sony Xperia XZ Cat 9 smartphones (data) for the simultaneous measurement of voice and data services.

Voice testing

Two smartphones per operator in each car were used for the voice tests, setting up test calls from one car to another. The walktest team carried one smartphone per operator for the voice tests. In this case, the smartphones called a stationary counterpart. The audio quality of the transmitted speech samples was evaluated using the HD-voice capable and ITU standardised so-called POLQA wideband algorithm. All Dutch operators offer 4G capable subscriptions. In total, two voice channels were in operation. Both smartphones establishing the first voice test channel were set to 4G preferred mode forcing the network to perform a fallback to UMTS in LTE areas. On the second channel, both smartphones were set to VoLTE-preferred to simulate customer behaviour with the latest technology. The smartphones used in the walktests were all set to VoLTE-preferred.

In order to account for typical smartphone-use scenarios during the voice tests, background data traffic was generated in a controlled way through random injection of small amounts of HTTP traffic. The voice test scores account for 40 per cent of the total benchmark results.

Data testing

Data performance was measured by using four smartphones in each car – one per operator. One car was equipped with four Samsung Galaxy S7 while the other car was carrying four Sony Xperia XZ in order to respect the variable data performance of different smart phones in different networks. In order to further reflect the customer experience, the radio access technology was set to LTE preferred mode. The web tests accessed web pages according to the widely recognised Alexa ranking. In addition, the static “Kepler” test web page as specified by ETSI (European Telecommunications Standards Institute) was used. 

In order to test the data service performance, files of 3 MB and 1 MB for download and upload were transferred from or to a test server located on the Internet. In addition, the peak data performance was tested in uplink and downlink directions by assessing the amount of data that was transferred within a seven seconds time period. Another discipline was the playback of YouTube videos. It took into account that YouTube dynamically adapts the video resolution to the available bandwidth. So, in addition to success ratios, start times and playouts without interruptions, YouTube measurements also determined the average video resolution.

All the tests were conducted with the best-performing mobile plan available from each operator. Data scores account for 60 per cent of the total results.

Routes and samples

The test routes are shown on page 1 of this report. In the selected 21 cities and 31 smaller towns, the cars had to follow predefined routes. Altogether, the two test cars covered more than 6,420 kilometres, of which approximately 2,820 km led through the cities and towns, while 3,600 km covered connecting roads.

Performance indicators and rating

The score weighting reflects both the geographical distribution of the Dutch population and the ranking of usage scenarios. Therefore, 600 of the total of 1000 maximum points were assigned to the cities – 450 maximum points for the drivetest results (voice: max. 180 p., data: max. 270 p.) and 150 for walktests results (voice: max. 60 p., data: max. 90 p.). For the towns, a maximum of 200 points is available (voice: max. 80 p., data: max. 120 p.) and for the connecting roads a maximum of 125 points (voice: max. 50 p., data: max. 75 p.). The tests conducted in trains account for another 75 points (voice: max. 30 p., data: max. 45 p.) The tables on page 2 and page 11 of this report show the percentage of maximum points that each operator has achieved in each discipline.

Additional Analysis of Four Carrier Aggregation 

All decisions concerning the setup and methodology of the test are jointly made by P3 and connect. However, we do this in close consultation with the operators about to be tested. This way, we can ensure that all procedures of the test are fair and none of the candidates suffers any disadvantages. 

This principle also applies to the smartphones that we use for the measurements. The two LTE Cat 9 smartphones that we chose fulfilled each operator’s requirements regarding software stability and network settings as well as supporting three carrier aggregation (3CA) – the combination of three LTE/4G carrier frequencies which the Dutch mobile operators market under the term "4G+“.

As KPN, Vodafone and T-Mobile already use four different LTE/4G carrier frequencies in the Netherlands (800, 1800, 2100 and 2600 MHz), there have been discussions whether the data tests should be performed with a Cat 16 smartphone that already supports “4CA“ – such as the Samsung Galaxy S8. But as the integration of the Galaxy S8 was completed too late for our regular preparation and adjustment timeline, we decided to stick with the Galaxy S7 and Sony Xperia XZ devices for the current benchmark season. Another reason was that this decision facilitated that all network tests of the current season were based on the same setup. However, as we wanted to make sure that this decision does not cause an unfair disadvantage to any of the candidates, we equipped a third drivetest car with Samsung Galaxy S8 instead of the Galaxy S7 devices. Subsequently, we had this car drive on the same routes and perform the same series of tests as our actual drivetest cars. 

The results of these comparison measurements verified the validity of our methodology: Interestingly, the measurement values delivered by the control devices in the data category would have resulted in slightly higher data scores for all four candidates – an addition of between two and five points to the actual results. But the overall ranking would not have changed had we used the Galaxy S8 instead of the Galaxy S7.

Conclusion

T-Mobile wins again. Vodafone shows also “outstanding“ results. A very good KPN ranks third, and Tele2 fourth – but still with a “very good“ score.

For the third time in a row, T-Mobile is the clear winner of the P3 connect Mobile Benchmark in the Netherlands. Its impressive 977 total points is the highest score ever achieved in any P3 connect benchmark. As the distinct leader in the data category and being on par with Vodafone in the voice discipline, T-Mobile‘s grade “outstanding“ is well deserved. If this operator succeeds in integrating the frequencies and base stations of its projected subsidiary Tele2, it will be excellently positioned for the future.

The pursuer Vodafone also shows an “outstanding“ performance in this year‘s benchmark. It achieves the second rank especially due to excellent voice results which are the same a those of test winner T-Mobile, while in the data discipline, Vodafone is almost at the same level as the third-ranking KPN.

KPN is strong in the data discipline, where it ranks second behind test winner T-Mobile. But the Dutch market leader loses some points in the voice category due to slightly lower speech quality and longer call setup times. However, all in all KPN still achieves a very good result.

Tele2, the smallest Dutch operator and destined to be owned by T-Mobile, achieves the fourth rank. Due to its concentration on LTE/4G, it achieves a decent score in the data discipline. Its score for data connectivity on the roads is particularly strong. However, Tele2‘s voice score shows some room for improvement. It will be interesting to see how the merger with T-Mobile will affect this operator‘s future results.