Most of the top VPN applications can leak data during day-to-day use despite claims to the contrary. Comparitech has leveraged a new suite of tools designed to test the “leakiness” of VPNs and their accompanying apps. Our results show the vast majority of VPNs leak DNS, IP, and WebRTC traffic, even if they claim to utilize leak protection and kill switches.
The current standard for testing commercial VPN services for leaks is substandard at best, relying on simple web-based tools that fail to encompass the wide range of scenarios under which a VPN can leak data outside of the encrypted tunnel. Comparitech spoke with ExpressVPN’s privacy research lab, which developed the tools, to gain access to the full suite of tests to date. The tools are free and open source. In this article, we put several VPNs through the gauntlet of tests to assess their overall leak protection.
Key findings:
- VPN apps for MacOS suffered slightly more leaks than Windows VPN apps, but Windows apps’ leaks were more severe on average
- VPN apps struggle with WebRTC IPv6 leaks
- Many VPNs leak DNS and IP traffic when a disruption occurs, such as a change in the network configuration
VPN companies, tech reviewers, and customers can and do test these two features for themselves. Simple web-based tools like this one test for leaks, and that’s just one of many. When a kill switch kicks in due to a dropped connection, most apps immediately notify the user. Most of us take it for granted that these features just work based on this limited evidence.
What is a leak?
What constitutes a leak isn’t set in stone across the industry, but a reasonable definition would be:
“When the user is connected to the VPN, no Personally Identifiable Information (PII) is exposed unless the user actively chooses to expose it.”
The above definition is a good watermark to guide us but is quite broad and easily broken. In practice, our current stipulations include:
- Don’t send unencrypted traffic from the device
- Don’t let a user’s public IP address be visible to third parties
- Don’t allow a user’s DNS requests to be seen by any DNS server other than that of the VPN
Leak protection prevents unencrypted traffic from traveling outside the VPN tunnel while connected to a server. A leak could result in the user’s online activity being visible to an unwanted third party. A kill switch halts internet traffic in the event that the VPN connection drops, stopping any data from passing over the unencrypted network while the connection is reestablished. Both of these features are vital to VPN users who value privacy and security.
The other important term to define is what it means to be “connected”. We chose a very strict definition:
“From the moment the user turns the VPN on, until the moment when the user consciously chooses to turn the VPN off.”
The user should always be protected unless they choose not to be. If, for example, the VPN connection unexpectedly drops, then they should remain protected.
In order to fairly interpret the test results, we must be aware of the types of leaks, the scenarios in which they occur, and how severe they are.
Types of leaks
The tool suite tests for five common types of VPN leaks.
- Plain IP leaks: User’s public IP addresses (IPv4 or IPv6) are leaking via the browser
- IP Traffic leaks: Traffic leaves device outside of VPN tunnel
- DNS leaks:
- DNS traffic leaves the device outside of the VPN tunnel
- DNS traffic leaves via the tunnel but goes to a DNS server not run by the VPN provider
- WebRTC leaks: User’s public IP addresses (IPv4 or IPv6) are leaking via the browser’s WebRTC functionality
Some types of leaks overlap. For example, if we can show that IP traffic leaks outside of the tunnel, then this also implies a DNS leak. However, DNS leaks might not necessarily imply traffic leaks, so it’s important to check for both.
WebRTC IPv6 leaks, which are very common, come in one of two classes: no permissions granted and permissions granted. In the former case, the user’s public IPv6 address is visible to web pages without ever granting permissions to use WebRTC. This is the more severe leak because web pages can insert javascript to monitor your IPv6 address and identify your device. The second case requires users actively grant the website permissions to use audio and video for WebRTC. Some providers only leak IPv6 in this scenario, but is still a leak nonetheless. Users should be able to safely use WebRTC-enabled sites when connected to VPNs without exposing their privacy.
While these leaks are a serious problem, the majority of people in the world still do not have IPv6 connectivity and thus will not be affected.
The tool suite is not necessary to test a VPN for WebRTC IPv6 leaks. Just go to this site, ensure no permissions have been granted for audio and video, and check whether an IPv6 address is listed. You can then add those permissions and test for the permissions-granted class of leak. (Note: Browserleaks.com is not affiliated with Comparitech or ExpressVPN.)
Leak scenarios
The test suite runs through several different simulations in which a leak might occur. These include different machine and network configurations. In its current state, the test suite is currently only equipped to run on Mac and Windows, but ExpressVPN says it will extend coverage to all major platforms.
Again, there is no official set of standards by which to test VPN leaks, so we have taken the liberty of defining two distinct categories:
Vanilla Scenarios: We obviously need to test for leaks during a run-of-the-mill VPN session. The user connects and everything is running fine. In such a scenario, we don’t expect most VPNs to leak, but the tests are always worth doing. The types of leaks detected by web-based testing tools typically occur in vanilla scenarios.
The exception to this is IPv6. Many VPNs still leak IPv6 traffic, specifically with WebRTC. WebRTC allows for real-time voice and video communication through a web browser. It can be disabled by changing a browser setting in Firefox, and via an extension for Chrome.
Disruption Scenarios: The more interesting leaks come when the network or VPN is somehow disrupted. What happens if, for example, a cable is yanked while the VPN is connected? These types of disruptions cause the network to destabilize, which can affect the VPN and thus the leak protection. It’s perfectly reasonable for a user to expect to be protected under these circumstances.
Assuming two network connections are available, e.g. Wi-Fi and Ethernet, the full list of disruptions we test is as follows:
- Yank the ethernet cable
- Turn off Wi-Fi
- Wi-Fi network goes down
- Router goes down
- Plug in an ethernet cable (after connect)
- Turn on Wi-Fi (after connect)
- Reorder the priority of network services
- Disrupt connection with the VPN server, which can happen in the real world if:
- The VPN server goes down, possibly for maintenance, a data center outage, or confiscation by authorities
- The connection to the server is blocked, e.g. by China’s Great Firewall
- Routing issues to the VPN server
- The VPN process crashes, e.g. crash openvpn
- Total connectivity loss followed by a subsequent network restore
- Disrupt the network configuration, such as changing the DNS servers from under the VPN
Leak severity
The importance of understanding the severity of leaks when they occur cannot be understated. We address some key points below, in descending order according to severity:
Persistent Vanilla Leak: The most severe. When you connect to the VPN, data constantly leaks outside the VPN tunnel behind the scenes. Most VPN providers have no problems here. WebRTC IPv6 leaks are usually the only issue, but of course that requires the user to have an IPv6 address. Most don’t.
Persistent Triggered Leak: Still pretty severe, but it depends on the trigger. Leaks that occur due to some change on the system, such as changing the network configuration. VPN apps should be able to deal with these changes without leaking. Once the trigger occurs, the leak persists. Both the VPN application and user are unaware. One example of this occurs when DNS traffic starts leaking out of the VPN tunnel and never stops.
Temporary Triggered Leaks: Leaks triggered by some event that only occur for a short period of time. In some cases, this period can actually be quite long (at least in terms of IP packets) and plenty of data can escape the tunnel. In other cases, the window can be very brief; perhaps only a handful of packets escape.
Temporary leaks might not sound severe, especially when the window is small. But if the trigger is something that can reasonably happen to a user on a regular basis, then over time an ISP (or other 3rd party) can still glean enough information to build a profile about the user.
Understanding the actual triggers is important when assessing the severity of a leak. Triggers range widely from unplugging an Ethernet cable to crashing the VPN process.
Unplugging an Ethernet cable is a very real world use case. Not only is it likely to affect a considerable number of users, it also reflects a class of different issues that might cause the same problem. For example, if the provider leaks when swapping a cable, then there’s a high probability they’ll leak in other scenarios such as temporary network loss. The current test scenarios are not exhaustive, so we use these types of failures to indicate general leakiness.
Crashing the VPN process is, in contrast, an edge case. One would generally expect a VPN application to be stable. However, bugs get shipped and crashes do happen in the real world. With millions of people using VPNs globally, the number of occurrences where this happens to is sure to be non-zero.
VPN leak testing results
Below are the results of our tests in alphabetical order. For each VPN, you can see what leaks were detected, how they are triggered, and their severity. We will continually expand upon this list as time goes on, adding more VPNs, configurations, and devices.
Disclaimer: While we’ve spent time running the test suite and checking the above tests all fail as indicated, we haven’t gone into a detailed analysis of the integrity of the tools themselves. Since the tools are in an alpha state, it’s possible that there may be inaccuracies in results.
To gain confidence in the test suite we corroborated the results by running manual tests where possible. These manual tests reproduced the test scenarios without using ExpressVPN’s tools. In all manually tested cases we observed the leak as reported by the test suite. We thus have a strong level of confidence that the test suite is a good indicator of the performance of VPN applications when it comes to leaks.
We aim to spend more time in the future getting a deeper understanding of the tools, and we hope that others in the industry will do the same, taking advantage of the tools’ open-source nature.
Avast SecureLine
Windows
- (Persistent vanilla) WebRTC IPv6 Leaks without granting permissions
- (Temporary triggered) IP traffic leaks when primary network service is disrupted, e.g pulling the ethernet cable or disabling the service
- (Temporary triggered) IP traffic leaks when VPN server is unreachable
- (Temporary triggered) IP traffic leaks when VPN process crashes
Mac
- (Persistent triggered) WebRTC IPv6 leaks when permissions are granted
- (Temporary triggered) IP traffic leaks when VPN server is unreachable
- (Temporary triggered) IP traffic leaks when VPN process crashes
- (Temporary triggered) IP traffic leaks when primary interface is disrupted, e.g. pulling the ethernet cable or disabling the service
CyberGhost
Windows
- (Persistent vanilla) WebRTC IPv6 Leaks without granting permissions
- (Persistent Vanilla) DNS goes to their DNS server BUT goes outside of the tunnel all the time. So it’s visible to ISPs and third parties
Mac
- (Persistent triggered) WebRTC IPv6 Leaks when permissions are granted
- (Persistent triggered) DNS leak to ISP when new network service appears, e.g. plugging in the ethernet cable while connected to the VPN. Requires local DNS IP address. If DNS IP is public then requests will go to that server and not CyberGhost’s server.
ExpressVPN
Windows
- No leaks detected
Mac
- No leaks detected
To be fair, ExpressVPN built the test tools and applied them to its own VPN app prior to publication of this article, so it has already patched leaks that it initially detected. Prior to running the tests and patching its own app, it suffered from DNS leaks that resulted from switching network connections.
HideMyAss
Windows
- (Persistent vanilla) WebRTC IPv6 leaks without granting permissions
- (Temporary triggered) IP Traffic leaks when VPN process crashes
- (Temporary triggered) IP Traffic leaks when primary network adapter is disrupted, e.g pulling the ethernet cable or disabling the adapter
- (Temporary triggered) IP Traffic leaks when VPN server is unreachable
Mac
- (Persistent triggered) WebRTC IPv6 leaks when permissions are granted.
- (Temporary triggered) IP Traffic leaks when VPN server is unreachable.
- (Temporary triggered) IP Traffic leaks when VPN process crashes.
- (Temporary triggered) IP Traffic leaks when primary network service is disrupted, e.g pulling the ethernet cable or disabling the service.
Note: the recovery period for all of the temporary triggered IP traffic leaks on HMA for Mac is relatively long, around 30 seconds.
Hotspot Shield
Windows
- (Persistent vanilla) WebRTC IPv6 leaks without granting permissions
Mac
- (Persistent triggered) WebRTC IPv6 leaks when permissions are granted
- (Temporary triggered) IP traffic leaks when VPN server is unreachable
- (Temporary triggered) IP traffic leaks when VPN process crashes
- (Temporary triggered) IP traffic leaks when primary network service is disrupted, e.g pulling the ethernet cable or disabling the service
- (Temporary triggered) IP traffic leaks when a new network appears, e.g. plugging in the ethernet cable while the VPN is connected
IPVanish
Windows
- (Persistent vanilla) WebRTC IPv6 leaks without granting permissions
- (Temporary triggered) IP traffic leaks when VPN process crashes
- (Temporary triggered) IP traffic leaks when primary network adapter is disrupted, e.g pulling the ethernet cable or disabling the adapter
Mac
- (Persistent triggered) DNS leak to ISP when new network service appears, e.g. plugging in the ethernet cable while connected to the VPN. Requires local DNS IP address. If DNS IP is public then requests go to that server and not IPVanish’s server.
- (Temporary triggered) IP traffic leaks when VPN server is unreachable
- (Temporary triggered) IP traffic leaks when VPN process crashes
- (Temporary triggered) IP traffic leaks when primary network service is disrupted, e.g by pulling the ethernet cable or disabling the service
NordVPN
Windows
(Persistent triggered) WebRTC IPv6 Leaks when permissions are granted.Fixed!
Mac
- (Temporary triggered) IP traffic leaks when VPN process crashes. The kill switch doesn’t get triggered, so the VPN just disconnects
It should be noted that NordVPN has two different apps for MacOS: one from the provider’s website and one on the Apple App Store. NordVPN have informed us that they recommend the App Store version, which uses the IKEv2 protocol in lieu of OpenVPN. This version of the app has fewer leaks, namely no DNS leaks. The OpenVPN version from the website has two persistent triggered DNS leaks.
NordVPN has updated its apps since initial testing, significantly improving results across both apps. Most recently, Nord patched the WebRTC leak present on the Windows app.
Private Internet Access
Windows
- (Persistent vanilla) WebRTC IPv6 leaks without granting permissions
- (Temporary triggered) IP traffic leaks when VPN process crashes
Mac
- (Persistent triggered) DNS leak to ISP when new network service appears, e.g. plugging in an ethernet cable while the VPN is connected. Requires local DNS IP address. If DNS IP is public then requests will go to that server and not PIA’s server.
- (Temporary triggered) A strange DNS leak when disabling the underlying interface on the device. It’s an edge case test as we haven’t found a clear way this would happen in the real world, but it triggers a leak. It requires root, though, so not very threatening.
Surfeasy
Windows
- (Persistent triggered) WebRTC IPv6 leaks when permissions are granted
- (Temporary triggered) IP traffic leaks when primary network adapter is disrupted, e.g pulling the ethernet cable or disabling the adapter
- (Temporary triggered) IP traffic leaks when VPN server is unreachable
- (Temporary triggered) IP traffic leaks when VPN process crashes
Mac
- (Persistent vanilla) IPv6 traffic leaks due to lack of protection
- (Persistent vanilla) WebRTC IPv6 leaks without granting permissions
- (Temporary triggered) IP traffic leaks when VPN process crashes
- (Temporary triggered) IP traffic leaks when VPN server is unreachable
- (Temporary triggered) IP traffic leaks when primary network service is disrupted, e.g pulling the ethernet cable or disabling the service
- (Temporary triggered) IP traffic leaks when a new network appears, e.g. plugging in the ethernet cable while the VPN is connected
- (Temporary triggered) IP traffic leaks when network service order changes
Note: Surfeasy is owned by Symantec and powers Norton’s Wi-Fi VPN service, so we could expect to see similar results there.
Torguard
Windows
We encountered technical issues with ExpressVPN’s testing suite when analyzing Torguard on Windows and will update results on this VPN once these issues are resolved. We excluded a number of results for this reason and only included results where the test ran smoothly.
Mac
- (Persistent triggered) WebRTC IPv6 leaks when permissions are granted
- (Temporary triggered) IP traffic leaks when VPN process crashes
Update: TorGuard inform us they are unable to replicate the Mac results and do not see leaks when using ExpressVPN’s tools. We can reproduce the results, however, and are confident the tests work as intended. Windows results for Torguard have never appeared in a published version of this article.
Tunnelbear
Windows
- (Persistent triggered) WebRTC IPv6 leaks when permissions are granted.
- (Temporary triggered) IP traffic leaks when primary network adapter is disrupted, e.g pulling the ethernet cable or disabling the adapter
Mac
- (Persistent triggered) WebRTC IPv6 leaks when permissions are granted.
- (Temporary triggered) IP traffic leaks when primary network service is disrupted, e.g pulling the ethernet cable or disabling the service
- (Temporary triggered) IP traffic leaks when a new network appears, e.g. plugging in the ethernet cable while the VPN is connected
VyprVPN
Windows:
- (Persistent vanilla) WebRTC IPv6 leaks without granting permissions
- (Temporary triggered) IP traffic leaks when primary interface is disrupted, e.g. pulling the ethernet cable or disabling the network adapter
- (Temporary triggered) IP traffic leaks when VPN server is unreachable
- (Temporary triggered) IP traffic leaks when VPN process crashes
Note: when disruptions happen, VyprVPN detects them and disconnects, leaving the app in a kill switch state with no VPN connection. The triggered leaks occur during the transition.
Mac
- (Temporary triggered) DNS leak to the ISP when a new network service appears,
e.g. plug in ethernet cable after connecting to the VPN. Requires a local DNS IP address. If DNS IP is public then requests will go to that server and not VyprVPN’s server (note: this actually causes a persistent triggered leak, which is more serious). - (Temporary triggered) IP traffic leaks when VPN server is unreachable
- (Temporary triggered) IP traffic leaks when VPN process crashes
A more private future
The implications of these leaks can be serious for users who require unfailing privacy when using a VPN. A leak could allow hackers, ISPs, and government authorities to track and record online activity.
With that in mind, some leaks are much worse than others, and some might not be a threat at all (e.g. users without IPv6 connectivity need not worry about IPv6 leaks). A VPN with a less severe leak or two is still better than no VPN at all.
We hope that this report will raise the bar for privacy protection across the industry. Comparitech plans to expand its analysis to several more VPNs as well as other configurations and operating systems. We will re-test and update the current results as more VPNs patch their vulnerabilities.
Why are you censoring Uncle Ted’s comments? Who is paying you for this?
Uncle Ted will take this to twitter.
Uncle Ted has screenshots.
Hi Ted,
Sorry for the delay, we get lots of comments which we approve and reply to manually so there can be a delay. I’m sure Paul will get around to your comment soon.
This has all been done before.
AIRVPN has the best all in one leak test tool available at ipleak.net
VPN.AC also checks for all leaks and even browser fingerprint issues at ipx.ac
Perfect Privacy also has various leaktest tools for webrtc, ip, DNS and Windows login leak.
Good for EXPRESSVPN and their research for this article, but these tools are nothing new. Perhaps you should update the article to include other leak test tools.
You need to publish all your data.
Screenshots, methodology, testing, everything.
These are some serious accusations.
Hi bro,
There aren’t really any screenshots. All the testing is performed via a command line interface. You can find more detailed methodology in the GitHub repo. All the test results we have are in this article. We are only divulging the type of leak so as not to encourage bad actors who might exploit them. However, several VPNs have approached us in private and we have disclosed more specific details about the exact tests that failed in past, what settings were used, and system configurations. The effort has been fruitful thus far and these VPNs have pledged to plug the leaks and get back to us.
I would like to see the test results for another 2 VPNs that I am using: ibvpn and windscribe. Thank you in advance!
Hi Asr,
We have plans to test Windscribe in the near future. I’ll look into testing ibvpn.
Best,
Paul
Can you please test Perfect Privacy VPN?
Hi J,
I’ll look into it. We have a long queue to get through, so it will take some time.
Thanks for understanding,
Paul
Let me get this right. You used ExpressVPN’s tools to determine that ExpressVPN passed all of the tests outlined by ExpressVPN in its own tests?
Come on, you can do better than that.
If you know of anyone else that makes leak testing tools as thorough as these, I’m all ears. The tests run on ExpressVPN are the same tests run on everyone else. And as I mentioned in the article, ExpressVPN did have some leaks before this article was published. Being the test’s creators, they did have the advantage of plugging the leaks first, but these tests allowed them to find those leaks in their own app.
there are other tools that are much more thorough. see ipleak.net and ipx.ac
Hi Ted,
Those are web-based tools, so while they are useful to find persistent leaks of HTTP traffic, they are not so useful for finding leaks that a triggered by disruptions and leaks that occur outside of the browser. That being said, we did use browserleaks.com to confirm some of the WebRTC IPv6 leaks.
Best,
Paul