As I was collecting reliability data for several PKI systems, I included Let’s Encrypt as it’s by far the biggest public PKI system. It has been running for less than 2 years and is still subject to significant development. It also provides its status data and its history at https://letsencrypt.status.io and here’s an informal analysis of its production systems.
As it is not easy to understand the impact of partial incidents, the uptime results are expressed in two numbers:
- 99.9% when only full disruptions are taken into account; and
- 98% (or 98.8% for 2017 only) when partial disruptions are included.
We expected the reliability of Let’s Encrypt to significantly improve after first 6 months or so. It doesn’t seem to be the case and service disruptions seem to occur regularly over the whole analyzed period. Having said that, the uptime keeps improving, but it’s a long way from 99.99% or higher, which you expect from commercial certificate providers.
The data also indicate that Let’s Encrypt doesn’t really gain as much benefit as one would expect from its high-availability configuration. Incidents usually show that both data centers with the certificate issuing service are impacted at the same time. Related to this are also several incidents when a bug in the software was discovered either during updates or shortly after — possibly indicating deficiencies in testing.
The attached chart shows the number of disruptions (full and partial) in the days after latest updates. You can see that the probability is skewed with the highest number of disruptions happening on the day of an update. The remaining days between updates seem to have similar probabilities of disruptions, although the data set is too small for any more detailed analysis.
We believe Let's Encrypt should consider some changes to the way updates are executed and possibly if it is possible to strengthen the testing of new versions of the software, like:
- Installing a new version on a staging server (with test signing key) for 1–2 weeks first. This would receive “production” requests, with responses sent to a monitoring system, rather than back to users.
- Staggered updates — update only one of the production servers with the latest release and in effect basically running 2 subsequent releases in parallel.
A pleasant surprise was the reliability of the OCSP service. The downtimes were limited to infrastructure issues (networking, server time synchronization).
Having said that, there are many users who don’t benefit from this service as Chrome and Safari browsers don’t do OCSP checks. This may get better with OCSP stapling. However, there are several issues here:
- it adds some complexity to the server configuration;
- the configuration must be resistant to short OCSP service outages; and
- it increases requirements on the reliability of Let’s Encrypt OCSP service, where a few hours’ long disruption can cause a denial of service for your web server.
There were 2 significant OCSP outages in the last twelve months, they lasted 2+ and 9+ hours.
Initially, we expected that the back-end system should be re-configured, to further minimize common infrastructure, to increase the overall reliability of the service, but it is not clear whether it would have a significant impact. It seems that most disruptions can be prevented on the software level. This is an expected situation for a new system, which is still under significant development.
A complete text of the analysis is at our blog Enigma Shards on Medium.com.
Enigma Bridge builds a new certificate management system for your web, email, and IT infrastructure: Instant PKI, which puts business n charge of encryption keys.