Skip to end of metadata
Go to start of metadata

What

IAM is upgrading the OpenLDAP software and replacing servers for eds.u.washington.edu (Person Directory Service - PDS).

When

We are targeting August 10th, 2016 as our migration date. We're asking PDS customers to complete testing by August 3rd.

Why

  • Improved access to community support
  • Current PDS server software was last rebuilt in 2006
  • New OS and OpenLDAP software supports modern TLS protocols
  • Data replication strategy changes
  • Using new faster mdb (lmdb) storage technology
  • OS platform is near end of life (Centos 5 - 32 bit)
  • Better ability to rebuild production server environment
  • Old PDS servers don't support SHA-2 signed certs from UWCA

Who needs to take action?

Developers and support teams that are still using the older PDS LDAP API need to test their applications against the newer PDS eval environment eds-eval.u.washington.edu.  If your application has not yet transitioned to the newer Person Web Service (PWS) API you'll need to test your application for compatibility.  Some environments use a full Certificate Authority list, file, or store and will not need to make any changes to their environment.

What do I need to do?

The original instructions for PDS suggested including only the UWCA root cert in a OpenSSL based configuration. We've had to change the server certificate on PDS to an InCommon issued certificate however clients are still required to connect using a UWCA certificate as their credential.

  1. Test to your application and configuration to see if any changes are needed
  2. If your application isn't able to connect add the InCommon root cert to your certificate store. For OpenSSL based clients:
    1. Create a new "CACert" file with UWCA root and InCommon root certs in your test environment
    2. Configure your application to use the new cert file
    3. Re-test against eds-eval.u.washington.edu and eds.u.washington.edu
    4. If the application now works: deploy the certificate file changes and continue using eds.u.washington.edu in your production environment

What is changing?

Nearly all software supporting the new PDS cluster has been updated.  This includes new Linux OS, new OpenSSL libraries, newer OpenLDAP version, new OpenLDAP storage engine, and a new replication strategy.

The PDS service is critical to many core functions of the University. Our strategy with PDS has been very cautious as we tested this new platform.

  • New OS: New PDS servers replaced physical ones 3 years ago but we kept the OS level the same.  Servers environment goes from Enterprise Linux 5 (32 bit) to Enterprise Linux 6.
  • New OpenSSL libraries: Come with the OS platform but TLS 1.2 is now supported.
  • New OpenLDAP software: OpenLDAP goes from 2.3.24 c.2006 to 2.4.40 c.2015. After this migration we'll schedule testing and in place upgrades for the most recent OpenLDAP release.
  • Server Certificate: During initial testing we found that PWS was unable to connect to these new systems. We found that the default schannel library configuration used on that platform block the UWCA root cert (MD5 signature) included as part of the PDS session negotiation. The Service Team opted to change the CA for PDS rather than ask all schannel based windows clients to downgrade to weaker security protocols or change the schannel security settings to accept weak certificates.
  • New OpenLDAP storage engine: OpenLDAP makes many storage options available. The previous recommendation from the community was to use Berkley Database (BDB)  After Oracle made changes to the licensing for newer versions of BDB the OpenLDAP community helped create a new storage engine LMDB. The newer DB strategy removes three layers of complicated caching options needed with BDB and is now the recommended option from the OpenLDAP project.
  • Replication strategy: All data inside PDS is sourced from the UW's Identity Registry.  Improvements in the Identity Registry software allows updating multiple systems targets removing the need to have a dedicated PDS replication server.   The data source (Identity Registry) will now update all PDS servers.

What if something goes wrong?

We've asked several different (and diverse) application teams to test the new eds-eval.u.washington.edu server.  All but one of test found no problems nor changes needed.  If an application isn't able to complete testing in time or finds problems using the new PDS server configuration they will be allowed to use a CNAME pointer we've created for the old environment which will remain in-sync with the new PDS environment.

The single problem application we found is running an ancient version of OpenSSL that doesn't support SHA-2 signed certs, a work around was found in that situation (AIX 6.1).

Additional questions..

eds-eval results so far?

eds-eval.u.washington.edu was upgraded on 3/15/2016 We found only 3 customers that used the evaluation environment in the last 9 months and each verified their application worked without any changes. We expect some customers are using a "CACerts" file that includes only the UWCA and will need to be updated to include the InCommon "root" cert.  Any changes needed to connect to eds-eval will also need to be deployed to your production environment.

LDAP design changes?

No, the PDS LDAP (OU) structure is identical between the old and new PDS environments.

Why was the internal testing period so long?

During testing we found PWS and the other Business Web Services were not able to connect to PDS.  During the Summer of 2015 we found the TLS 1.2 protocol brought enhancements in the client's and server's ability to specify which hash and signature algorithms they accept.  Microsoft took advantage of the new TLS 1.2 capabilities in their schannel software and marked MD5 as a not acceptable signature algorithm.  The UWCA root cert is signed with MD5 and is included in the PDS SSL payload.

What is the change with the back-end LDAP database?

The OpenLDAP projects has been moving away from the traditional BDB (BerkleyDB) due to licensing changes and lack of innovation in that product.  For several years they've been discouraging the use of BDB as a storage engine back-end and instead promoting a memory mapped database product that was developed by several people on the OpenLDAP opensource project.  You can read more about Lightning Memory-Mapped Database on Wikipedia.  The article includes on performance comparisons to other modern key value pair DB products and some additional narrative.  With the groups directory services update in Spring 2015 our performance improvement was in the 40%-50% range.

Which environments have already been tested?

As of our change announcement date the following environments confirmed working connections and functionality:

  • Kuali RICE
  • Shibboleth IdP
  • PWS
  • SWS
  • HRPWS
  • Parking services
  • Office of Educational Assessment
  • Microsoft Infrastructure (NETID domain)

Why is the service still using OpenLDAP software?

OpenLDAP still provides excellent performance and reliability.  UW-IT is currently limiting our investment in PDS which includes testing other LDAP software. Additional factors include a custom patch we apply to OpenLDAP to support certificate authentication in Windows environments.

UW-IT may retire PDS in the future.  We have been suggesting all new clients use the Person Web Service (PWS) instead and except for one application we've successfully directed all new requests to PWS over the last 4 years. This may be an opportunity for some application teams to make the switch to PWS.  At this time we do not have plans to eliminate PDS but new capabilities will be added to PWS and not PDS.

Script based clients CAFile

If you are using a script based LDAP client and would like to limit the number of root certs your client utilizes you should update or repoint your "CA Certs file" configuration to use the UWCA and InCommon roots certificates. If you are in this situation you may also choose to point your "CA Certs" configuration to a Certificate authority bundle that is provided by your platform vendor however you'll either need to augment the file to include the UWCA root cert before we cutover or make a copy that your application will use temporarily until we complete the production change on or around August 10th.  Or in otherwords after the production change the PDS server will reference the "AddTrust" root cert below where it utilizes the UWCA root cert today.

AddTrust External CA Root
=========================
-----BEGIN CERTIFICATE-----
MIIENjCCAx6gAwIBAgIBATANBgkqhkiG9w0BAQUFADBvMQswCQYDVQQGEwJTRTEU
MBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFkZFRydXN0IEV4dGVybmFs
IFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBFeHRlcm5hbCBDQSBSb290
MB4XDTAwMDUzMDEwNDgzOFoXDTIwMDUzMDEwNDgzOFowbzELMAkGA1UEBhMCU0Ux
FDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5h
bCBUVFAgTmV0d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9v
dDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALf3GjPm8gAELTngTlvt
H7xsD821+iO2zt6bETOXpClMfZOfvUq8k+0DGuOPz+VtUFrWlymUWoCwSXrbLpX9
uMq/NzgtHj6RQa1wVsfwTz/oMp50ysiQVOnGXw94nZpAPA6sYapeFI+eh6FqUNzX
mk6vBbOmcZSccbNQYArHE504B4YCqOmoaSYYkKtMsE8jqzpPhNjfzp/haW+710LX
a0Tkx63ubUFfclpxCDezeWWkWaCUN/cALw3CknLa0Dhy2xSoRcRdKn23tNbE7qzN
E0S3ySvdQwAl+mG5aWpYIxG3pzOPVnVZ9c0p10a3CitlttNCbxWyuHv77+ldU9U0
WicCAwEAAaOB3DCB2TAdBgNVHQ4EFgQUrb2YejS0Jvf6xCZU7wO94CTLVBowCwYD
VR0PBAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wgZkGA1UdIwSBkTCBjoAUrb2YejS0
Jvf6xCZU7wO94CTLVBqhc6RxMG8xCzAJBgNVBAYTAlNFMRQwEgYDVQQKEwtBZGRU
cnVzdCBBQjEmMCQGA1UECxMdQWRkVHJ1c3QgRXh0ZXJuYWwgVFRQIE5ldHdvcmsx
IjAgBgNVBAMTGUFkZFRydXN0IEV4dGVybmFsIENBIFJvb3SCAQEwDQYJKoZIhvcN
AQEFBQADggEBALCb4IUlwtYj4g+WBpKdQZic2YR5gdkeWxQHIzZlj7DYd7usQWxH
YINRsPkyPef89iYTx4AWpb9a/IfPeHmJIZriTAcKhjW88t5RxNKWt9x+Tu5w/Rw5
6wwCURQtjr0W4MHfRnXnJK3s9EK0hZNwEGe6nQY1ShjTK3rMUUKhemPR5ruhxSvC
Nr4TDea9Y355e6cJDUCrat2PisP29owaQgVR1EX1n6diIWgVIEM8med8vSTYqZEX
c4g/VhsxOBi0cQ+azcgOno4uG+GMmIPLHzHxREzGBHNJdmAPx/i9F4BrLunMTA5a
mnkPIAou1Z5jJh5VkpTYghdae9C8x49OhgQ=
-----END CERTIFICATE-----
UW CA Root (MD5)
=========================
-----BEGIN CERTIFICATE-----
MIIEBzCCA3CgAwIBAgIBADANBgkqhkiG9w0BAQQFADCBlDELMAkGA1UEBhMCVVMx
CzAJBgNVBAgTAldBMSEwHwYDVQQKExhVbml2ZXJzaXR5IG9mIFdhc2hpbmd0b24x
FDASBgNVBAsTC1VXIFNlcnZpY2VzMRcwFQYDVQQDEw5VVyBTZXJ2aWNlcyBDQTEm
MCQGCSqGSIb3DQEJARYXaGVscEBjYWMud2FzaGluZ3Rvbi5lZHUwHhcNMDMwMjI1
MTgyNTA5WhcNMzAwOTAzMTgyNTA5WjCBlDELMAkGA1UEBhMCVVMxCzAJBgNVBAgT
AldBMSEwHwYDVQQKExhVbml2ZXJzaXR5IG9mIFdhc2hpbmd0b24xFDASBgNVBAsT
C1VXIFNlcnZpY2VzMRcwFQYDVQQDEw5VVyBTZXJ2aWNlcyBDQTEmMCQGCSqGSIb3
DQEJARYXaGVscEBjYWMud2FzaGluZ3Rvbi5lZHUwgZ8wDQYJKoZIhvcNAQEBBQAD
gY0AMIGJAoGBALwCo6h4T44m+7ve+BrnEqflqBISFaZTXyJTjIVQ39ZWhE0B3Laf
bbZYju0imlQLG+MEVAtNDdiYICcBcKsapr2dxOi31Nv0moCkOj7iQueMVU4E1Tgh
YIR2I8hqixFCQIP/CMtSDail/POzFzzdVxI1pv2wRc5cL6zNwV25gbn3AgMBAAGj
ggFlMIIBYTAdBgNVHQ4EFgQUVdfBM8b6k/gnPcsgS/VajliXfXQwgcEGA1UdIwSB
uTCBtoAUVdfBM8b6k/gnPcsgS/VajliXfXShgZqkgZcwgZQxCzAJBgNVBAYTAlVT
MQswCQYDVQQIEwJXQTEhMB8GA1UEChMYVW5pdmVyc2l0eSBvZiBXYXNoaW5ndG9u
MRQwEgYDVQQLEwtVVyBTZXJ2aWNlczEXMBUGA1UEAxMOVVcgU2VydmljZXMgQ0Ex
JjAkBgkqhkiG9w0BCQEWF2hlbHBAY2FjLndhc2hpbmd0b24uZWR1ggEAMAwGA1Ud
EwQFMAMBAf8wKwYDVR0RBCQwIoYgaHR0cDovL2NlcnRzLmNhYy53YXNoaW5ndG9u
LmVkdS8wQQYDVR0fBDowODA2oDSgMoYwaHR0cDovL2NlcnRzLmNhYy53YXNoaW5n
dG9uLmVkdS9VV1NlcnZpY2VzQ0EuY3JsMA0GCSqGSIb3DQEBBAUAA4GBAIn0PNmI
JjT9bM5d++BtQ5UpccUBI9XVh1sCX/NdxPDZ0pPCw7HOOwILumpulT9hGZm9Rd+W
4GnNDAMV40wes8REptvOZObBBrjaaphDe1D/MwnrQythmoNKc33bFg9RotHrIfT4
EskaIXSx0PywbyfIR1wWxMpr8gbCjAEUHNF/
-----END CERTIFICATE-----

Note: the comments in the above text (2 lines above "BEGIN") may need to be removed for Java based clients.