IAM is upgrading the OpenLDAP software and replacing servers for eds.u.washington.edu (Person Directory Service - PDS).
We are targeting August 10th, 2016 as our migration date. We're asking PDS customers to complete testing by August 3rd.
- Improved access to community support
- Current PDS server software was last rebuilt in 2006
- New OS and OpenLDAP software supports modern TLS protocols
- Data replication strategy changes
- Using new faster mdb (lmdb) storage technology
- OS platform is near end of life (Centos 5 - 32 bit)
- Better ability to rebuild production server environment
- Old PDS servers don't support SHA-2 signed certs from UWCA
Who needs to take action?
Developers and support teams that are still using the older PDS LDAP API need to test their applications against the newer PDS eval environment eds-eval.u.washington.edu. If your application has not yet transitioned to the newer Person Web Service (PWS) API you'll need to test your application for compatibility. Some environments use a full Certificate Authority list, file, or store and will not need to make any changes to their environment.
What do I need to do?
The original instructions for PDS suggested including only the UWCA root cert in a OpenSSL based configuration. We've had to change the server certificate on PDS to an InCommon issued certificate however clients are still required to connect using a UWCA certificate as their credential.
- Test to your application and configuration to see if any changes are needed
- If your application isn't able to connect add the InCommon root cert to your certificate store. For OpenSSL based clients:
- Create a new "CACert" file with UWCA root and InCommon root certs in your test environment
- Configure your application to use the new cert file
- Re-test against eds-eval.u.washington.edu and eds.u.washington.edu
- If the application now works: deploy the certificate file changes and continue using eds.u.washington.edu in your production environment
What is changing?
Nearly all software supporting the new PDS cluster has been updated. This includes new Linux OS, new OpenSSL libraries, newer OpenLDAP version, new OpenLDAP storage engine, and a new replication strategy.
The PDS service is critical to many core functions of the University. Our strategy with PDS has been very cautious as we tested this new platform.
- New OS: New PDS servers replaced physical ones 3 years ago but we kept the OS level the same. Servers environment goes from Enterprise Linux 5 (32 bit) to Enterprise Linux 6.
- New OpenSSL libraries: Come with the OS platform but TLS 1.2 is now supported.
- New OpenLDAP software: OpenLDAP goes from 2.3.24 c.2006 to 2.4.40 c.2015. After this migration we'll schedule testing and in place upgrades for the most recent OpenLDAP release.
- Server Certificate: During initial testing we found that PWS was unable to connect to these new systems. We found that the default schannel library configuration used on that platform block the UWCA root cert (MD5 signature) included as part of the PDS session negotiation. The Service Team opted to change the CA for PDS rather than ask all schannel based windows clients to downgrade to weaker security protocols or change the schannel security settings to accept weak certificates.
- New OpenLDAP storage engine: OpenLDAP makes many storage options available. The previous recommendation from the community was to use Berkley Database (BDB) After Oracle made changes to the licensing for newer versions of BDB the OpenLDAP community helped create a new storage engine LMDB. The newer DB strategy removes three layers of complicated caching options needed with BDB and is now the recommended option from the OpenLDAP project.
- Replication strategy: All data inside PDS is sourced from the UW's Identity Registry. Improvements in the Identity Registry software allows updating multiple systems targets removing the need to have a dedicated PDS replication server. The data source (Identity Registry) will now update all PDS servers.
What if something goes wrong?
We've asked several different (and diverse) application teams to test the new eds-eval.u.washington.edu server. All but one of test found no problems nor changes needed. If an application isn't able to complete testing in time or finds problems using the new PDS server configuration they will be allowed to use a CNAME pointer we've created for the old environment which will remain in-sync with the new PDS environment.
The single problem application we found is running an ancient version of OpenSSL that doesn't support SHA-2 signed certs, a work around was found in that situation (AIX 6.1).
eds-eval results so far?
eds-eval.u.washington.edu was upgraded on 3/15/2016 We found only 3 customers that used the evaluation environment in the last 9 months and each verified their application worked without any changes. We expect some customers are using a "CACerts" file that includes only the UWCA and will need to be updated to include the InCommon "root" cert. Any changes needed to connect to eds-eval will also need to be deployed to your production environment.
LDAP design changes?
No, the PDS LDAP (OU) structure is identical between the old and new PDS environments.
Why was the internal testing period so long?
During testing we found PWS and the other Business Web Services were not able to connect to PDS. During the Summer of 2015 we found the TLS 1.2 protocol brought enhancements in the client's and server's ability to specify which hash and signature algorithms they accept. Microsoft took advantage of the new TLS 1.2 capabilities in their schannel software and marked MD5 as a not acceptable signature algorithm. The UWCA root cert is signed with MD5 and is included in the PDS SSL payload.
What is the change with the back-end LDAP database?
The OpenLDAP projects has been moving away from the traditional BDB (BerkleyDB) due to licensing changes and lack of innovation in that product. For several years they've been discouraging the use of BDB as a storage engine back-end and instead promoting a memory mapped database product that was developed by several people on the OpenLDAP opensource project. You can read more about Lightning Memory-Mapped Database on Wikipedia. The article includes on performance comparisons to other modern key value pair DB products and some additional narrative. With the groups directory services update in Spring 2015 our performance improvement was in the 40%-50% range.
Which environments have already been tested?
As of our change announcement date the following environments confirmed working connections and functionality:
- Kuali RICE
- Shibboleth IdP
- Parking services
- Office of Educational Assessment
- Microsoft Infrastructure (NETID domain)
Why is the service still using OpenLDAP software?
OpenLDAP still provides excellent performance and reliability. UW-IT is currently limiting our investment in PDS which includes testing other LDAP software. Additional factors include a custom patch we apply to OpenLDAP to support certificate authentication in Windows environments.
UW-IT may retire PDS in the future. We have been suggesting all new clients use the Person Web Service (PWS) instead and except for one application we've successfully directed all new requests to PWS over the last 4 years. This may be an opportunity for some application teams to make the switch to PWS. At this time we do not have plans to eliminate PDS but new capabilities will be added to PWS and not PDS.
Script based clients CAFile
If you are using a script based LDAP client and would like to limit the number of root certs your client utilizes you should update or repoint your "CA Certs file" configuration to use the UWCA and InCommon roots certificates. If you are in this situation you may also choose to point your "CA Certs" configuration to a Certificate authority bundle that is provided by your platform vendor however you'll either need to augment the file to include the UWCA root cert before we cutover or make a copy that your application will use temporarily until we complete the production change on or around August 10th. Or in otherwords after the production change the PDS server will reference the "AddTrust" root cert below where it utilizes the UWCA root cert today.
AddTrust External CA Root
UW CA Root (MD5)
Note: the comments in the above text (2 lines above "BEGIN") may need to be removed for Java based clients.