A Tale of Two Studies: The Best and Worst of YubiKey Usability

Abstract
Two-factor authentication (2FA) significantly improves the security of password-based authentication.
Recently, there has been increased interest in Universal 2nd Factor (U2F) security keys—small hardware devices that require
users to press a button on the security key to authenticate. To examine the usability of security keys in non-enterprise usage,
we conducted two user studies of the YubiKey, a popular line of U2F security keys. The first study tasked 31 participants
with configuring a Windows, Google, and Facebook account to authenticate using a YubiKey. This study revealed problems
with setup instructions and workflow including users locking themselves out of their operating system or thinking they had
successfully enabled 2FA when they had not. In contrast, the second study had 25 participants use a YubiKey in their daily
lives over a period of four weeks, revealing that participants generally enjoyed the experience. Conducting both a laboratory
and longitudinal study yielded insights into the usability of security keys that would not have been evident from either
study in isolation. Based on our analysis, we recommend standardizing the setup process, enabling verification of success,
allowing shared accounts, integrating with operating systems, and preventing lockouts.

Paper

J. Reynolds, T. Smith, K. Reese, L. Dickinson, S. Ruoti, and K. Seamons. A Tale of Two Studies: The Best and Worst of YubiKey Usability, 39th IEEE Symposium on Security and Privacy (S&P 2018), May 2018.

Usage Policy
This data is intended to be used for usage in academic research. No attempt should be made to de-anonymize users.

Data

sp2018-data_sanitized

Laboratory Open Respose Questions Coding YubiKey

Longitudinal Codebook YubiKey

Surveys

YubiKey Lab Study Survey

YubiKey Long Study Survey

Recruiting Poster

YubiKey-Lab-Study-Poster

YubiKey-Longitudinal-Study-Poster

Instructions