Using ACME for Certificate Management

Purpose

This document describes the procedure used to enroll in the InCommon ACME service, install a Certbot client, and connect to the Sectigo (InCommon) ACME service.

Background

ACME, i.e. Automated Certificate Management Environment, is a protocol that makes it possible to automate the issuance and renewal of certificates.  It was initially designed by The Internet Security Research Group (ISRG) for its own certificate service, Let’s Encrypt but has been implemented by other CAs following RFC 8555.

As more system administrators across UW are managing increasing numbers of TLS certificates, the demand for an automated solution has grown.  Since Sectigo (the vendor providing InCommon CA certificates) has recently developed a stable service offering for the ACME protocol, we are now able to offer automated InCommon certificate management through their service.

With ACME, certificates can be scripted to request and renew a certificate which can cut down on the overhead for users with a large number of certificates. But utilizing the ACME protocol can be a technical challenge and InCommon's implementation is not conducive for only a small number of certificates.  

For an overview and explanation of the ACME lifecycle, please see: https://sectigo.com/resource-library/what-is-acme-protocol.

Process overview

Connecting to the Sectigo ACME Service consists of 3 basic steps:

1. Request access to ACME 

2. Install Certbot in your environment

3. Connect to the ACME Service

Request access to ACME

Prerequisites

• A selection of domains you wish to be able to request certificates via ACME protocol
  • If you only need a few certificates and have a limited number of domains, please request your certificates as detailed in Obtain a Certificate from the InCommon CA
• For each domain that will utilize the ACME protocol, you must be a domain contact
  • This uses the same standard as noted in the prerequisites of Obtain a Certificate from the InCommon CA
• A technical comfort to research how certbot (or other ACME clients) work in your environment

Process

Please provide your department name and the domains you plan to automate with ACME. 

1. Request access to ACME via the form at: https://uwconnect.uw.edu/sp?id=sc_cat_item&sys_id=2cd2500bdba595906ccf6a9ed3961957 
   
   1. The Identity and Access Management (IAM) team will verify from your request:
      1. Department name to use and relationship of requestor to that department
      2. Domains to delegate to their ACME account.
      3. Domain ownership.
2. You will be asked if you want to join the ACME mailing list:_ . 
3. Once an IAM Specialist has delegated your department and domains, they will Securely transmit the ACME URL, Key ID, and HMAC Key to you.
   1. These secrets have the same security requirements as a private key, and thus must be appropriately safe guarded. Please see our recommendations for securing your ACME secrets.
4. The ACME setup isn't complete until you connect to the InCommon/Sectigo ACME server with an ACME client.
   1. See the section "Connect to the Sectigo ACME Service" for more details
5. Configure any scripting you wish for your environment to request additional certificates, or regularly renew existing certificates.

Install Certbot in your environment

Sectigo recommends Certbot as their supported ACME client.  The Electronic Frontier Foundation hosts an excellent set of instructions for installing certbot on your server.  Please visit https://certbot.eff.org/instructions and follow the instructions based on your web server software and operating system.

Connect to the ACME Service

In order to connect to the Sectigo ACME Service you'll need to use the command line sequence appropriate for your operating system.

Credentials

ACME URL: https://acme.sectigo.com/v2/InCommonRSAOV
Key ID: insert HMAC key
HMAC Key: insert key
Certbot requires admin privileges. Here are example commands that will use certbot to connect to the InCommon ACME service, complete your account setup, and get your first certificate. 
Depending on your platform, web server, ACME plug-ins, etc, this may not be the way you obtain subsequent certs. See the certbot documentation at https://certbot.eff.org/. 

Linux

sudo certbot certonly --standalone --non-interactive --agree-tos --email {requestor@uw.edu} --server https://acme.sectigo.com/v2/InCommonRSAOV --eab-kid {Key ID} --eab-hmac-key {HMAC Key} --domain {domain name used as CN for the cert} --cert-name {short friendly name for cert}

Windows

Start a CMD window as an admin account, then:

certbot certonly --standalone --non-interactive --agree-tos --email {requestor@uw.edu} --server https://acme.sectigo.com/v2/InCommonRSAOV --eab-kid {Key ID} --eab-hmac-key {HMAC Key} --domain {domain name used as CN for the cert} --cert-name {short friendly name for cert}

Note: Any domains you reference in your ACME request must first be added to InCommon certificate services (if they don't already exist there), then they must be delegated to your department, and then delegated to your departmental ACME account. 
This isn't the way the Let's Encrypt service works, but it is the way the InCommon ACME service works.

Please see Certbot resources for InCommon ACME for a summary reference of Certbot commands.

For Windows environments

Certbot

Once you have requested your certificates with Certbot as detailed above, you still need to prepare it for the windows environment. Since ACME returns the certificate in a .pem format, some extra steps are required to prepare most certificates for use. Certificates need to be in a .pfx or other "key-included" form to be added to the windows certificate store or IIS. 

One way to address this is detailed below, but note that there are plenty of ways to solve this issue, including scripts if you would like to write one. 

1. Create a copy of your privkey.pem file generated by ACME and rename the copy fullchain.key
2. Open the command prompt and navigate to the ACME directory (normally  /certbot/live/yourdomainhere)
3. In the command prompt, use certutil to create a .pfx certificate from the named certificate and key of the same name  (ex. certultil -MergePFX fullchain.pem fullchain.pfx)
4. Add fullchain.pfx to IIS, the windows keystore or elsewhere as desired

Win-acme

Another approach is to use an unsupported client like win-acme, but for unsupported clients there will be more responsibility for technical troubleshooting. The following steps were created by members of the UW Community from the School of Environmental Sciences

1. Download Win-Acme 
   1. You can successfully run win-acme from a share but subsequent task scheduler runs will fail without additional work.
2. Edit Settings.json to point to the UW InCommon provider:

     "Acme": { 

       "DefaultBaseUri": "https://acme.sectigo.com/v2/InCommonRSAOV/", 

       "DefaultBaseUriTest": "https://acme.sectigo.com/v2/InCommonRSAOV", 

       "DefaultBaseUriImport": "https://acme.sectigo.com/v2/InCommonRSAOV", 

       "PostAsGet": true, 

       "RetryCount": 15, 

       "RetryInterval": 5, 

       "PreferredIssuer": null 

     }, 

   
   

3. Run WCAS.exe as administrator and follow the default settings (“N”) to setup automatic renewal for IIS certs.
   1. When prompted, enter your AccountID/KeyID and HMAC key.
   2. These will be cached for use in subsequent renewals. 
   3. Note that you’ll need to make sure each certificate in IIS Site Bindings is set to “Require Server Name Indication” for the automated renewal to work properly. 
4. Confirm your certificate installed correctly in IIS

Support Notes

ACME Clients

Certbot is the only "officially" supported ACME client for InCommon/Sectigo, but there are many other clients available. Certbot for Linux is fairly mature and supports plug-ins for automation with web servers, etc. Certbot for Windows is a more recent development and is very basic. Most Linux customers will be happy enough with certbot, but not so much with Windows. In particular, the current version of certbot for Windows can't integrate with the Windows certificate store, which is half the drudgery on that platform. Other clients like win-acme have community supported efforts to bring them into the InCommon environment, and resources like the UW ACME mailing list (acme-users@uw.edu) can help with troubleshooting unofficial clients.

The bottom line is that ACME is just a protocol and any client that supports the protocol should interoperate with the InCommon/Sectigo ACME service. The choice of clients is up to the customer, but just because a client supports the protocol doesn't mean we can support the client. This is kind of like our SAML support. Our IdP supports SAML and a customer can choose their SAML client software. However, outside of Shibboleth, we don't have much expertise on specifics of other SAML clients. Certbot is similar to our Shibboleth in the ACME space.

Protecting ACME secrets

Unlike Let's Encrypt implementation of ACME, Sectigo's implementation provides ACME secrets that cover all of domains, not a single domain. This means the ACME secrets need to be handled with exceptional care because if they are compromised they can be used to request a unlimited number of certificates. 

As a result, secrets should not be shared over insecure communication methods or stored in environments that are not expected to be secure. Access to systems that contain ACME secrets should be limited to only those who require it (usually only the domain contacts). One recommended way to do this is to configure a centralized server that will request and share out the certificates as needed.  

The UW CISO has also released notes on handling secrets that have great advice for ACME secrets that we highly suggest reviewing: https://ciso.uw.edu/education/risk-advisories/managing-secrets/ 

Centralized vs Decentralized Management

As previously noted, one major way to protect your ACME secrets is to limit the number of locations they are saved. ACME is designed to benefit users who spend time requesting a large number of certificates manually, and for such users it is highly suggested to invest the time to configure an environment made to support the ACME automation. 

A centralized management approach requires configuring a server/container to act as a central server that will request all certificates, and scripts or other tools to distribute and install certificates from the central server to other locations. The exact design depends heavily on your environment, but there are options to do so with IIS. Different departments around the UW have created their own solutions. At time of writing the IAM team is looking to implement a centralized solution themselves, and we highly suggest using the ACME listserv to see other's approaches. 

Revoking ACME secrets

If you have reason to believe that your ACME secrets have been compromised , please email us immediately at iam-support@uw.edu. IAM will work to confirm the situation with you and revoke the ACME connection so the secrets can no longer be used. Additionally, IAM will work with you to review the domains covered by your ACME secrets and revoke any certificates you did not request. 

If you believe a certificate private key has been compromised but not the ACME secrets, you can reach out at iam-support@uw.edu and IAM will work to revoke the certificate(s) in question.

Please note that "comprimised" does not apply to no longer needing your ACME connection or certificate. If you would like to no longer use ACME, please reach out to iam-support@uw.edu. If you no longer need your certificate, please delete the private key and the certificate can expire naturally. 

Additional Resources

• Certbot ACME Client Documentation
• Let's Encrypt ACME Service Documentation
• ACME specification (IETF RFC-8555)