A Usability Evaluation of Let’s Encrypt and Certbot: Usable Security Done Right

“A Usability Evaluation of Let’s Encrypt and Certbot: Usable Security Done Right” Tiefenau et al., Conference on Computers and Communication Security 2019

When looking through the CCS proceedings, I knew that I had to find out more about the work by Tiefenau et al., purely by the title alone. What stuck out about the title in particular was the “Usable Security Done Right” subtitle. There’s been a trend in the Usable Security space of extracting a provocative quote from a survey or study participant that shows their misunderstanding of a method, technology, protocol, etc. and using that as part of the title. This paper didn’t didn’t do that but rather highlighted a certain tool that was getting things right. While my curiosity was piqued by the title, the rest of the paper definitely didn’t disappoint.

TLS TLS TLS

Transport Layer Security (TLS) is one of the protocols that people interact with pretty much every day without thinking about it. With its importance to securing data and how frequently people interact with it, it has been a subject of a lot of usable security research. Most of that research has focused on how users perceive TLS warnings and how to potentially improve them. While understanding the user side of things is important, it leaves out the other essential and less-studied side: server configuration. That’s where Let’s Encrypt and Certbot come in. Certbot helps automates the acquisition of certificates from Let’s Encryption and the parts of the configuration on web servers.

What were they looking for?

In their work, Tiefenau and his colleagues tried to find out the advantages of using Let’s encrypt through a randomized controlled trial comparing Let’s Encrypt and Certbot to a traditional CA. More specifically they had the following research questions:

• Does Certbot support its users in fulfilling the task of en-abling TLS?
• How do participants perceive Certbot’s functionality and usability?
• How can the Certbot process be improved?
• How does task framing affect how participants behave in the study?

Let’s talk design

Overall Design

The study design that Tiefenau et al. used was based on a study conducted by Krombholz et al [1]. There were two differences between the designs of the two studies 1) in Tiefenau et al.’s study they had two treatment groups which were a conventional CA and Certbot + Let’s Encrypt CA and 2) they also examined the skill of the participants by making them also perform server related actions such as proving server ownership. Another element that they added to to the study was the concept of framing. As mentioned earlier, they tried to understand if framing had an effect on how participants performed. They tried to accomplish this by creating a role-playing scenario for half of the participants were they were to imagine that they were working for a company. They made this scenario more realistic by using tailoring the URLs, usernames and passwords. When first reading the paper, I was a bit confused about which part of the study was within-subjects and which part was between-subjects but the table below really clears things up.

Tasks

Each participant had to complete four tasks:

• Task 1: The user had to SSH into the server and move web pages to the www directory. This task was used to get a baseline of the participant’s Linux skill level.
• Task 2: The participant had to acquire the certificate from a traditional CA or they have to use certbot to acquire a cert and install it on the server
• Task 3: The participants in the traditional CA treatment had to install the certificate
• Task 4: Participants checked their server configuration using Qualys SSL Server Test

Participants had to complete each of the four tasks for both treatments and were given a time limit of 2 hours for the Certbot task and 3 hours for the traditional CA task. After the completion of each task they completed a questionnaire that asked them to assess their performance and contained questions about the tasks themselves. At the conclusion of both tasks they completed a final questionnaire which compared the two options.

Participants and Support Channel

To substitute for an expert population, Tiefenau et al.used a class of computer science students. As would be expected the demographics of the group skewed male and young. About 71% of the participants had experience as a sysadmin but over half had never configured TLS before.

The last and the most interesting part of the study design to me was the support channel that they set up. They used this channel two counter two common issues in user studies with complex tasks: 1) participants failing early and never getting to the task of interest and 2) participants forgetting the issues they ran into when trying to complete a long procedure. In terms of participants failing early, they were able to give appropriate hints or nudges in the case a participant was having trouble SSH’ing into the server. Also, they were able to get immediate feedback when a participant ran into a problem because they would send them a message. Tiefenau et al. used a playbook to decide what help to provide to participants and what type of delay to use in responding.

Configure or not, there is no try

Now, that we’ve detailed the study design, let’s talk about how the participants did.

The completion results between the traditional CA treatment and the Certbot treatment were pretty eye-opening. 28 (90%) of participants were able to deploy a certificate from Let’s Encrypt using certbot as compared to only 16 (52%) with the traditional approach. It’s pretty clear that certbot drastically improved participants ability to configure TLS on a web server. The table below shows the different points that participants failed at.

The results regarding the time to configure TLS followed the same trend as the completions results. The median time for Certbot task was 18 minutes compared to the 65 minutes for the traditional CA task.

A finding that the authors found surprising was that participants performed well with security and CA parts; however, they struggled with the tasks that were specific to configuring Apache which was used as the webserver. These problems ranged from being able to perform the ownership verification to not knowing enable the SSL module of Apache2.

In the comparison survey, the combination of Let’s Encrypt and Certbot won out in every category except transparency. The reason for this was most likely because of the automation that Certbot provides potentially masking steps in the process.

In terms of framing, they didn’t not find any difference in the behavior of participants.

Tell me what’s good (or bad)

The bad

One of the negative things called out in the post-task questionnaire and the comparison survey was that actions taken by Certbot weren’t transparent to the user. One participant in particular had the following remark:

everything was easy,but […] Certbot is not transparent to me. I do not know what it ac-tually did and the whole process inside, for me it is like (a) blackbox”

The authors point out that Certbot has a verbose option but none of the participants chose to use it. Nevertheless, it seems like the default verbosity should be increased.

The next thing that users commented on was that Certbot didn’t offer more advanced security configurations. The authors counterpoint to this was that Certbot has to be general to support different use cases and administrators and it is on the right path; however, they do suggest that future work should look into the tradeoff between security and generalizability.

The good

The authors pointed out that the simplicity of the Certbot is one of the key factors of it success and most importantly it didn’t require to have knowledge about the process.

Certbot applied the knowledge of its experts auto-matically with little need for specialized knowledge, by guiding the user through the process using a dialog-like approach instead of requiring multiple commands on the command line.

The other part of it’s usability as stated by the authors was that it used safe defaults and automated several steps while only introducing two new ones.

Wrap Up

The combination of Let’s Encrypt and Certbot show that when “administrator-centered engineering” is combined with simplicity problems that are long considered hard to solve at scale can be become significantly easier

References

[1] Katharina Krombholz, Wilfried Mayer, Martin Schmiedecker, and Edgar R. Weippl.”I Have No Idea What I’m Doing” - On the Usability of Deploying HTTPS. In 26th USENIX Security Symposium, USENIX Security 2017, 2017.