“All of them claim to be the best”: A multi-perspective study of VPN users and VPN providers

Reethika Ramesh, Anjali Vyas, Roya Ensafi

This report presents findings from our latest paper that is accepted to appear at USENIX Security 2023.
August 9, 2022.

arXiv Link: https://arxiv.org/abs/2208.03505

Our previous related work (NDSS 2022): https://www.ndss-symposium.org/wp-content/uploads/2022-285-paper.pdf

Introduction

The use of Virtual Private Network (VPN) technologies has grown rapidly since their introduction over two decades ago. With commercialization, they have also found their way into a regular Internet user’s toolbox. In our experience working with Consumer Reports and engaging with users, we find exceptional amounts of interest from users on VPNs and VPN use related topics. Over 1,500 enthusiastic users attended our inaugural virtual workshop organized by Consumer Reports titled “Exploring VPNs”.

In this report, we specifically focus on the findings of our large scale study of 1,252 VPN users and qualitative interviews of 9 VPN providers. These are consolidated and described in detail in our paper “‘All of them claim to be the best’: Multi-perspective study of VPN users and VPN providers” which will be presented at USENIX Security 2023.

While limited prior work has delved into the human factors of VPN use [ 1 , 2 , 3 , 4 , 5 ], no previous study has combined both the users and VPN providers perspectives to answer fundamental questions about the VPN ecosystem. To gain a clearer picture of the inner workings of such a large consumer ecosystem, it is imperative to study both its users and its providers. To that end, we surveyed 1,252 users that have either used or currently using a VPN, and interview 9 VPN providers to get insights into our various lines of inquiry which we systematize into the following research questions:

Research Questions and Key Takeaways

Motivations

We find that protection from threats (82.1%) and to make public networks safer to use (58.4%) are the biggest reasons users use VPNs. On the other hand, ﬁle sharing such as torrenting (12.1%) is among the least popular reasons.
We categorize survey participants based on their self-reported expertise on security and privacy issues, high-expertise users (511 of 1252, 40.81%), moderate-expertise users (631 of 1252, 50.40%), and limited-expertise users (110 of 1252, 8.79%).

Needs and considerations

Speed (72.6%), price (55.4%), and an easy to use app (44.1%) are among the top three requirements in a VPN.
Price is an especially big criteria for users with limited-to-moderate expertise, suggesting that discounts, and marketing around these factors is bound to have a vast effect on these users.
On the other hand, high-expertise users value clear explanations of logging.
Interestingly, we find that users rely on search (61.1%) and recommendation sites (56.5%) rather than word of mouth (5.7%) to choose a VPN, and users also overwhelmingly rate these recommendation websites as trustworthy sources.

Emotional connection and threat model

Exploring reasons for why users use VPNs, we find that users indicate that they feel unsafe browsing the Internet without a VPN (62.6%).
Subsequently, users indicate they feel safer using a VPN while browsing (86.7%).

Mental model - Do users have an accurate understanding of how VPNs work and the data they collect?

Given the emotional attachments and user dependency on VPNs for security, alarmingly we find that almost 40% of the users have a flawed mental model of what VPNs provide them and what data they collect. These users believe that their ISP can see what they do while using a VPN. More worryingly, we do not see significant differences between users of different expertise having flawed mental models .
At least 40% of users indicate they are unsure what data is collected by VPNs, and worryingly, 13% of the remaining users believe thatthink VPNs can and do collect private information from them, such as private messages and audio/video recordings.

Perception and trust

Users express concern mostly about VPN providers selling their data (73.2%) and the VPN software containing malware (65.6%).
Regarding efforts geared at earning and increasing user trust, users consistently rate having security protocols and disclosures of breaches (62%) as extremely important, followed by a clear logging policy (46.7%) and independent security audits (41.6%).

(Mis)alignment between VPN users and providers

Users and providers acknowledge that pricing is among one of the highest priorities when it comes to choosing a VPN. But at the same time, VPN providers point out that this very fact is frequently exploited in the industry with malicious marketing gimmicks and fear mongering .
Alarmingly, we find the highest degree of misalignment in the user’s trust in the VPN recommendation and review ecosystem. Most providers agreed that the review ecosystem is far from reliable and largely motivated by money. However, users are completely unaware of this, and rely on them believing they are trustworthy.
Another key area of misalignment is in data collection. Over 40% of users in our study are not sure exactly what data is being collected about them by providers. But on the other hand, multiple providers say that they clearly communicate their logging practices or that they do not log and have audits to prove it.

Our actionable recommendations and the path forward for the VPN ecosystem:

Prioritizing user education, oversight on advertisements and marketing surrounding VPNs, coordinated efforts to bring attention to the flawed VPN recommendation ecosystem, and regulations to curb malicious marketing tactics that lead to false mental models and expectations within users.
We believe that our work will help security and privacy advocates such as EFF and CDT, technologists, and VPN providers alike, by calling attention to the key areas of issues within the VPN ecosystem.

User survey

In partnership with Consumer Reports, a leading consumer research and advocacy organization, we launched our user survey on March 1, 2021. Ours is the largest survey of VPN users to date: over 1,500 VPN users participated in our 20-minute survey. We had 1,374 users indicating they are from the US, and the rest were from 21 other countries. We decided to focus on the U.S.-based participants (and VPN providers popular in the US) for this study, but we are currently working on surveying users from different parts of the world, using an adapted survey instrument, as a follow-up research study. Zooming in on the US participants, 1,252 users passed our quality and attention check. All our survey respondents either currently use or have previously used a VPN. Figure 1 summarizes how we developed, refined and distributed the survey, and finally, how we analyzed it.

Figure 1: User survey development and analysis procedure

Qualitative interviews of VPN providers

Using the same parent themes as mentioned before in Fig 1, we created a questionnaire to interview VPN providers. We interviewed CEOs, CMOs, or researchers working in the following VPN providers (in alphabetical order): CalyxVPN, Hide.me, IVPN, Jigsaw Outline, Mullvad VPN, RiseupVPN, Surfshark, TunnelBear VPN, and Windscribe. Figure 2 illustrates our process.

Figure 2: VPN provider interview development and analysis procedure

We provide detailed information in our paper regarding our study methods, recruitment, ethics, and analysis methods. To facilitate and encourage future work, we have provided our entire user survey instrument, and VPN provider interview questionnaire in the appendix of our paper.

Selected Results

Research Question 2: Needs and Considerations

We present the results in-depth for one out of our six key research questions, please refer to our paper for the rest of our results. We explored the user’s needs and considerations when it comes to choosing a VPN provider to use. We find that Speed (72.6%), price (55.4%) and an easy to use app (44.1%) are among the top 3 requirements in a VPN. On the other hand, variety or number of servers (18.8%) and using a VPN to change location for media sites such as Netflix (12.4%) are amongst the lowest ranked requirements, as illustrated in Figure 3.

Figure 3: Percentage of importance levels users attach with criteria they look for in a VPN, presented along with the number of users who chose it in [brackets]. Ranked from 1-most important to 7-least.

Price is a big criteria for limited to moderate expertise users: Users of all expertise rank speed equally highly as a top three criteria (p=0.348, N=1067). But users with limited-to-moderate expertise are signiﬁcantly more likely to rank price higher (χ2-test, p=0.000150, N=1048); 71.1% of these users rank it in their top three, compared to 59.3% of high expertise users. This means that prices, discounts, and marketing around these factors is bound to have a vast effect on these users.

Interestingly, when we explored what resources users use to discover and choose the right VPN for their needs, we found that users increasingly rely on researching on the Internet (61.1%, 765 of 1252), recommendation websites (56.5%, 708 of 1252) and providers’ own websites (48.1%, 602 of 1252) rather than word of mouth (5.7%, 167 of 1252) to choose a VPN. Investigating further, we find that 93.9% of people who used recommendation websites rate them as moderately to extremely trustworthy, and the recommendation sites received a significantly higher degree of endorsement than the other two “top” resources, as illustrated in Figure 4.

Figure 4: Trustworthiness of each resource as rated by users with different security and privacy expertise. Bars for each resource (from left to right) indicate: No Opinion, Not-Trustworthy, Moderately-Trustworthy and Extremely-Trustworthy

On the other hand, in our VPN provider interviews, we find that one of the main themes is the issue of the VPN review ecosystem being inherently biased and dishonest. One provider calls it a “parasitic industry” and a majority of providers (6/9) remark that the review ecosystem mostly runs on money, e.g. paid reviews, and cost-per-action (CPA). They also explain that VPNs or their parent companies may own different review sites, many review sites even auction the #1 spot, and write reviews for money. Multiple providers also mention that Google search results are unreliable, and that there are few good reviewers left; one provider says:

"You honestly cannot find even one ranking site that is honest, if you just tell people that...so that people know”

Since commercial VPNs are a growing multi-billion dollar industry, our study reveals that it is becoming increasingly important to deter the exploitation of VPN users who are possibly unaware of such practices in the VPN recommendation ecosystem. As we highlight from the rest of our VPN providers' interviews, a lot of the malicious marketing preys on users' misunderstandings, many of which we are able to corroborate from our user survey as well. We find flawed mental models existing in the minds of users with all levels of security and privacy expertise.

Recommendations and the path forward:

Our own prior work with VPNalyzer (NDSS 2022) shows that the lack of regulation and standardization leads to VPNs offering varying levels of security and privacy. Through this work, we recommend that FTC and other government organizations exert oversight on VPN advertising and curb malicious tactics used by VPNs, because such aggressive and misleading ad campaigns could degrade users' mental models on what VPNs offer. An example of successful oversight is NordVPN's ad being banned in the UK for misleading users. We advocate for coordinated efforts from the industry, academia, and consumer protection organizations to bring attention to the flawed VPN recommendation ecosystem as well. We recently presented our ideas as a research proposal at the Workshop on Technology and Consumer Protection (ConPro’22) for a data-driven investigation into the VPN recommendation ecosystem to uncover patterns of biased recommendations [ 1 ].

Our study also shows that user-education campaigns regarding VPNs and the VPN ecosystem must be prioritized. We find that the areas that need the most improvement are users' mental model of what a VPN provides, what it can do, and the threat models for which VPNs can be most useful. We have undertaken efforts in partnership with Consumer Reports and featured in their white paper detailing their objective evaluation of popular VPNs, and articles aimed at user education [ 2 , 3 ]. Since the user population surveyed in our study is on average older and more educated, our results suggest that incomplete and flawed mental models may be even more prevalent among the general U.S. population. We urge security and privacy advocates such as the EFF and CDT, consumer protection agencies such as the FTC, and community initiatives such as IFF to devote their efforts towards VPN user-education and awareness, and advocate for VPN industry oversight.