Dutch operators swap rankings in P3 network test

Thursday 20 April 2017 | 12:15 CET | Background
T-Mobile has again emerged as the best mobile network in the Netherlands in the annual P3 Connect Mobile Benchmark test, by P3 Communications. Both in terms of data and voice T-Mobile achieved the best scores of the four networks, at 956 points. 

Vodafone (931) and KPN (930) followed in respectively second and third place. Vodafone won the first edition of the benchmark and was in third place last year. This year, KPN scored better in mobile data, while Vodafone did better in calls. However, their total scores were nearly the same. 

T-Mobile came out on top again this year, but it's total score is lower than in 2016. KPN also lost points. P3 Managing Director Hakan Ekmen said this was due to performance in outlying areas weighing more in the tests this year. The main outline of the test is the same each year, but the benchmarks are set higher each year in order to reflect the ever-increasing expectations of mobile users.  

The Netherlands, along with Switzerland, still scores the best networks overall, Ekmen said. The operators get the most out of their networks and achieve the highest values with almost all KPIs. And if P3 changes its test, the operators follow its parameters, Elmen noted. 

T-Mobile and Vodafone scored the best in terms of voice this year, with only a minor difference in scores. For data, T-Mobile and KPN performed best. Tele2 and Vodafone followed, although even for them the scores were high in international comparison. Vodafone had the lowest score for data, at 553, just under Tele2. While Tele2 was still fourth in the overall rankings, its score improved compared to 2016. 

Four vehicles and 48 smartphones

This is the third time the German consulting group has measured the quality of the Dutch LTE networks. T-Mobile was not included in the first test in 2015, as it was still in the process of upgrading its network and did not think the tests would be representative. The latest tests were conducted between 27 February and 7 March 2017.

P3 covered 7,900 km with four cars for the tests, visiting 17 larger and 19 smaller cities. Each car was fitted with 12 Samsung Galaxy S7 smartphones, three for each network, for testing mobile calls and measuring downloads and uploads and YouTube performance. 

P3 takes measurements from within the car and uses the smartphones' own antennas. However, the smartphones are in a case that may influence their antennas. This makes the measurements closer to the indoor experience. 

The total potential score of 1,000 points includes 400 points for voice and 600 for data. The large cities account for 60 percent of the end-score, and the smaller towns and connecting roads for 20 percent each.


Prior to the tests, P3 selected with the operators a relevant subscription plan and if needed, the latest operator firmware for the phone. The results showed that the operators are still in a transition phase for VoLTE. In principle, calls over 4G offer better quality, both in set-up and sound quality. 

 Network  VoLTE on network  VoLTE in P3 test  score
 T-Mobile  no  no  381
 KPN  yes  no  378
 Vodafone  yes  yes  363
 Tele2  yes  yes  359

KPN, Tele2 and Vodafone have all activated VoLTE, but T-Mobile not yet. KPN started at the end of October 2016 with a phased roll-out, but that was not yet included in this test. P3 only tested voice over 3G at KPN, but with a phone on the 4G network. This results in a fallback to the 3G network when the phone makes a call, which takes more time for call set-up. 

KPN's network showed the longest call set-up time, losing the operator points. In other parameters the differences between the operators were much smaller. The call success rate in cities and small towns was 99 percent or more of all connections. This was slightly less for all four operators on roads. Call quality shows a similar picture. Vodafone notably scored somewhat better in smaller towns than in cities.  

Fall-back to 3G does not necessarily have to mean a poorer performance, as T-Mobile's scores show. Even without VoLTE, T-Mobile scored the best in voice of all four operators. Vodafone's VoLTE score is much lower. Tele2 also has VoLTE on its network, but outside its 4G coverage area relies on a roaming agreement with T-Mobile. The legacy service resulted in the lowest score for voice, of 359 out of 400.  

Nevertheless, the introduction of VoLTE appears to have gone well for the operators, said Ekmen. It's a technically challenging operation, which can lead to quality suffering. However in the Netherlands, no such decline in quality was found; if anything, it improved, he said. The German-speaking countries showed more problems with this, while the Dutch operators appear to have it working from day one. 

Data: very high peak speeds

On each network, P3 uses a single Galaxy S7 smartphone for a series of tests. This includes measuring the time to load a web page and the quality of YouTube streams. A full 100.0 percent of YouTube videos played without buffering, at an adaptive rate of around 700p. 

P3 also measures how long it takes to download or send a test a file. A longer download, of around 10 seconds, shows what the network can offer as average throughput. The speed achieved in 90 percent of cases is considered the benchmark for what most customers experience. In cities, this measured between 16 and 30 Mbps. 

The 10 percent fastest connections show the maximum possible speeds on the networks. T-Mobile reached 143 Mbps in large cities and even faster elsewhere, over 150 Mbps. P3 detected the use of tri-carrier aggregation during the tests, which explains the very high peak speeds. 

Typical in the Netherlands is that the differences between large cities, smaller towns and roads are small. Outside cities the networks perform in general less well, but the differences are tiny. KPN achieved in smaller towns the same or even higher percentage scores, while the others were the same or somewhat lower. According to P3, this shows that are four truly national LTE networks.

International top

P3 Communications conducts the same tests in a number of countries. Most recently it looked at Germany, Austria, Switzerland and Sweden, and Spain and the UK were done in Q4 2016. The Netherlands and Switzerland scored the best in these results. T-Mobile Netherlands was overall the best network, just ahead of Sunrise in Switzerland. Each of the four Dutch networks would have topped the overall rankings in Germany, Austria, Spain, the UK and Sweden. T-Mobile Netherlands and Sunrise this year were the only operators to receive an 'excellent' mention, while the other three Dutch operators were 'very good'. 

Operator reactions

Rachid El Hattachi, Chief Technology Officer at T-Mobile Netherlands, said he was very pleased with the test results. Despite the growing number of customers the KPIs for the data network improved, such as the highest available download speed. The results also show that a quick call set-up and good voice quality are possible also with legacy voice, the CTO said. T-Mobile is testing VoLTE and aims to have it activated before the next round of tests by P3. T-Mobile is also expanding its network capacity. 

A KPN spokesperson said the company is more focused on network availability, which it considers the most important benchmark, alongside NPS (Net Promoter Scores). At 99.3 percent, KPN claims the best coverage in the Netherlands. The operator said the P3 results are skewed by giving big cities a 60 percent weighting in the scores. 

KPN also criticised the fact that its VoLTE was not measured, as it was available at the time of the tests, including on the S7. The operator pointed to an alternative test by Omnitele, where it scored best in terms of VoLTE quality. 

Vodafone said it's also focused on NPS, which it tracks. The operator said customers consider 'app coverage', and for these to work only 6-9Mbps is needed. As for the voice results, Vodafone said its score reflects the investments it has made recently, especially in improving indoor coverage. The same as KPN and T-Mobile, Vodafone is working on further upgrades.

Tele2 said in response to the P3 tests that it was "super proud" of the progress it's made in the past two years and its results in international comparison. 

Free Headlines in your E-mail

Every day we send out a free e-mail with the most important headlines of the last 24 hours.

Subscribe now

::: add a comment