“We’re on the Same Page”: A Usability Study of Secure Email Using Pairs of Novice Users

Abstract
Secure email is increasingly being touted as usable by novice, with a push for adoption based on recent concerns government surveillance. To determine whether secure email is ready for grassroots adoption, we employ a laboratory user study that recruits pairs of novice users to install and use several of the latest systems to exchange secure messages. We present both quantitative and qualitative results from 25 pairs of novice users as they use three types of systems: integrated (Pwm), depot (Tutanota), and hybrid integrated–depot (Virtru). Participants report being more at ease with this type of study and better able to cope with mistakes since both participants are “on the same page”. We find that users prefer integrated solutions over depot-based solutions, and that tutorials are important in helping first-time users. Hiding the details of how a secure email system provides security can lead to a lack of trust in the system. Participants expressed a desire to use secure email, but few wanted to use it regularly and most were unsure of when they might use it.

Paper
S. Ruoti, J. Andersen, S. Heidbrink, M. O’Neill, E. Vaziripour, J. Wu, D. Zappala, K. Seamons. “We’re on the Same Page”: A Usability Study of Secure Email Using Pairs of Novice Users, Proceedings of the 34th Annual ACM Conference on Human Factors in Computing Systems (CHI 2016). ACM, 2016.

Usage Policy
This data is intended to be used for usage in academic research. No attempt should be made to de-anonymize users.

Data
chi2016-data_sanitized.xlsx

Surveys
Johnny’s Survey
Jane’s Survey

Recruiting Poster
Poster

Screenshots

Pwm

Pwm's integrated tutorial.

Pwm’s integrated tutorial.

Pwm's secure composition interface.

Pwm’s secure composition interface.

Pwm's secure read interface.

Pwm’s secure read interface.


Tutanota

Tutanota's sign up page.

Tutanota’s sign up page.

Tutanota's secure compose interface.

Tutanota’s secure compose interface.

Tutanota's password-encrypted email.

Tutanota’s password-encrypted email.

Tutanota's password-encrypted email's password entry interface.

Tutanota’s password-encrypted email’s password entry interface.

Tutanota's secure read interface.

Tutanota’s secure read interface.


Virtru

Virtru's integrated tutorials.

Virtru’s integrated tutorials.

Virtru's secure composition interface.

Virtru’s secure composition interface.

Virtru's integrated secure read interface.

Virtru’s integrated secure read interface.

Virtru's depot-based encrypted email.

Virtru’s depot-based encrypted email.

Virtru's depot-based secure read interface.

Virtru’s depot-based secure read interface.


Mailvelope

Mailvelope's button to enable secure compose. Found in the upper right corner of the webmail provider's compose interface.

Mailvelope’s button to enable secure compose. Found in the upper right corner of the webmail provider’s compose interface.

Mailvelope's secure composition interface.

Mailvelope’s secure composition interface.

Mailvelope's public key selection interface.

Mailvelope’s public key selection interface.

Mailvelope's secure read interface.

Mailvelope’s secure read interface.

Mailvelope-encrypted message.

Mailvelope-encrypted message.

Mailvelope's private key password entry interface.

Mailvelope’s private key password entry interface.

Mailvelope's encrypted message in the webmail provider's compose interface.

Mailvelope’s encrypted message in the webmail provider’s compose interface.