Testing Methodology
The methodology of the P3 connect Mobile Benchmark is the result of more than 15 years of testing mobile networks, today in 65 countries. It was carefully designed to evaluate and objectively compare the performance and service quality of The Netherlands’ mobile networks from the users’ perspective.
The P3 connect Mobile Benchmark in the Netherlands took place from February 27 to March 6, 2017. All samples were collected between 8am and 10pm. The network tests covered larger cities, smaller towns and connecting roads. The combination of test areas had been carefully selected to provide a significant series of test results cove-ring the Dutch population. The areas chosen for the 2017 test account for more than 5.1 million people, or about 30 per cent of the total population.
P3 conducted the tests with four drive-test cars, equipped with arrays of twelve Samsung Galaxy S7 Cat 9 smartphones each. Eight phones were used for the measurement of voice services, and four additional S7 captured the values for the data services.
Voice testing
Two smartphones per operator in each car were used for the voice tests, setting up test calls from one car to another. The audio quality of the transmitted speech samples was evaluated using the HD-voice capable and ITU (International Telecommunication Union) standardized so-called POLQA wideband algorithm (Perceptual Objective Listening Quality Assessment). All Dutch network operators offer 4G capable subscriptions.
To take the high share of LTE into account, speech samples were acquired partly in 4G preferred to 3G preferred mode and partly in 4G preferred to 4G preferred mode. As a consequence, the phones in most cases needed to switch (“fall back”) to 2G or 3G when they were logged in to the LTE (so called “circuit-switched fall back“ or CSFB).
In order to account for typical smartphone usage scenarios during the voice tests, background data traffic was generated in a controlled way through random injections of small amounts of HTTP traffic. The voice test scores account for 40 per cent of the total benchmark results.
Data testing
Data performance was measured using four smartphones in each car – one per operator. The radio access technology was set to LTE preferred mode in order to reflect the customer experience. The web tests accessed ten web pages according to the widely recognized Alexa ranking. In addition, the artificial (static) “Kepler” test web page as specified by ETSI (European Telecommunications Standards Institute) for testing purposes was used.
In order to test the data service performance, files of 3MB and 1MB for download and upload respectively were transferred from or to a test server located on the Internet. In addition, the peak data performance was tested in uplink and downlink directions by assessing the amount of data that was transferred within a 10 seconds time period. Another discipline was the playback of Youtube videos, taking into account that Youtube dynamically adapts the video resolution to the available bandwidth. So, in addition to the success ratios, start times and playouts without interruptions, the Youtube measurements also determined the average video resolution.
All the tests were conducted with the best-performing mobile plan available from each operator. Data scores account for 60 per cent of the total results.Routes and samplesThe test routes are shown on page 1 of this report. In the 17 big cities and 19 smaller towns indicated, the cars had to follow predefined routes. Altogether, the four test cars covered more than 7900 kilometres, of which approximately 3500 km led through the big cities, while 4400 km were covered in smaller towns and on connecting roads.
Performance indicators and rating
The score weighting reflects both the geographical distribution of the Dutch population and the ranking of usage scenarios. Therefore, 600 of 1000 maximum points achievable were assigned to the cities – divided into 240 points reflecting the voice results and 360 points reflecting the data results. For the towns and the roads, 200 points each are available. In both categories, those are divided into 80 points referring to the voice and 120 points referring to the data category. The charts on page 2 and page 10 of this report show the percentage of maximum points that each operator achieved in each discipline.
Conclusion
All four Dutch operators score well over 900 out of 1000 points. Overall winner T-Mobile even surpasses the 950 points threshold, achieving the rare grade „outstanding“.
The results of the P3 connect mobile benchmark in the Netherlands had already reached a very high level last year. In 2017, again, all four Dutch operators show a remarkably high level of performance and stability. In a year-on-year comparison, Tele2 was able to improve in all tested categories. Vodafone takes over the second rank from KPN, which mainly results from Vodafone‘s distinct improvement in the voice category. If the evaluation had considered only legacy voice connections, KPN would have ranked better.
When it comes to data connectivity in the cities, T-Mobile and KPN lead the field, with Tele2 and Vodafone trailing a little behind, but still scoring at a very high level. In the smaller Dutch towns, the lead of the stronger two operators T-Mobile and KPN is slightly more pronounced. On the connecting roads, the distance between all four candidates is so close that no operator is a bad choice.
When comparing the results of this year‘s mobile network benchmark in the Netherlands to those from recent tests in other European countries, Dutch operators are among the top tiers. Even the fourth-ranking Tele2 scores at a level that would make it a suitable candidate for a winning position in countries with overall weaker mobile networks like Spain or the UK.
As in 2016, the third largest Dutch operator also wins the 2017 benchmark with a lead in the voice and data categories and the overall grade “outstanding“. The fact that T-Mobile achieves the best result in the voice category without offering VoLTE shows that legacy voice services can also offer perfect calls, when they are handled well.
With a considerable improvement in its voice performance, Vodafoneis able to outrank the market leader KPN in this year’s benchmark. In the data category, this operator still achieves a third place. However, the differences are small, and especially on the roads and to some extent in smaller towns, Vodafone holds up well with its competitors.
In a tight competitive environment, the largest Dutch operator has to give way to the smaller operators, that rank on second and third position in terms of subscriber numbers. However, KPN is beaten by Vodafone with a gap of only one point. But the Dutch market leader still achieves very good results both in the voice and data categories.
The Netherlands’ smallest operator scores last in this benchmark. However, Tele2’s results are still very competitive and deserve the overall grade „very good“. The loss of points in the voice category can mainly be explained by the disadvantageous effects of national roaming for legacy voice services on the T-Mobile network.
Outlook
Objective testing will be essential to the ongoing evolution of the Dutch networks as to emerging technologies like autonomous driving and smart cities. P3 communications is preparing for these future challenges.
As in the years before, all four candidates scored really well in this year‘s mobile network test in the Netherlands. This, however, cannot be taken for granted. Upcoming improvements like the further extension of 4G coverage, the ongoing network evolution with technologies like “4G Advanced“ and the continuing rollout of Voice over LTE (VoLTE) may change this picture.
P3 communications is well prepared to accompany the future developments of the Dutch telecommunications market. In this context, we are already preparing to include new technologies and enhancements mentioned above in next year‘s testing. Furthermore, P3 is also preparing for future challenges starting with new communications applications like autonomous driving and smart cities (see below), through to completely new mobile communications technologies like the upcoming super-fast 5G networks. However, 5G‘s approach of assigning different network capabilities to different use cases will also require an extended approach to network testing.This will involve concepts like advanced analytics and big-data technology.
Extensions to the test routines
Readers interested in taking part in our continuous efforts to evaluate the performance of mobile networks can already do so today – by downloading the “U get“ wireless performance rating app. See details on the right.
This crowdsourcing approach will give us valuable additional insights into the user experience and operational performance of mobile networks in the near future.
Crowdsourced network rating
P3 communications is increasingly focussing on aspects like the retainability of voice services, the integrity of data services and “operational excellence“. An important instrument for this approach is the “U get“ app that is available under uget-app.com or via the adjoint QR code. The Android app checks and visualises current network performance. Join the global community of users who understand their personal wireless performance, while contributing to the world’s most comprehensive picture of mobile customer experience.
Autonomous Driving
The future of transport is quickly coming upon us – one kilometre loop at a time. With each new generation of car models, the automotive industry comes closer to its vision of highly automated driving cars. However, to ensure that driverless cars can maintain connectivity and thus optimal performance and safety, we need to ensure that the technological infrastructure can manage the increasing demand that machines will place upon it. Therefore, autonomous driving scenarios play an important role in P3 communications‘ concepts for the evolution of mobile network testing
Smart Cities
Today, 54 per cent of the world’s population lives in urban areas, a proportion that is expected to increase to 66 per cent by 2050. Mobile communications will be an essential component to delivering on smart city promises. To enable smart cities to thrive and host successful businesses in the digital era, their technological infrastructure must be capable of managing the increased demand on network usage. Therefore, it will be a future focus point of P3 communications to determine whether cities become truly smart by taking an even closer look at their advances in connectivity.