Child pages
  • 2019-06-13 Meeting notes and agenda
Skip to end of metadata
Go to start of metadata


June 13, 2019


Managing secrets for systems


Many teams use a password manager, such as UW-IT's LastPass service, to manage secrets for desktop applications and even for systems.

As IT teams mature their DevOps practices and increase automation of their software environments, there is a demand for secrets management tools and services, such as Hashicorp Vault. These tools promote security practices such as key rotation through automated provisioning of short-lived, dynamic secrets.

This session will review existing key management practices and tools, gauge our organizational interest and readiness to pursue a centrally-managed key service, and consider the model for a new service.



UW Tower, T22 Boardroom



10:30 amIntro, agendaAnnouncement
  • Rupert Berk
10:35 am

Existing key management practices and solutions for systems at the UW, including:

  • REF
  • LastPass
  • EWS Key Service
  • Azure Key Vault
  • Google Vault
  • ...
Discussion, white-boarding
  • Kristina Taylor

  • Eric Horst
  • Jeff Franklin
  • Ethan Turner
  • Maxime Deravet
  • Paul Prestin
10:55 am

Better solutions:

  • What are preferred tools and solutions in this space?
  • What else is needed in the automated secrets ecosystem? e.g. certificate management APIs
Discussion, white-boarding
  • All
11:25 am

Service model (and the beginning of a business case)

  • Value proposition: Why? What would be the value of a centrally-managed service? 
  • Activities: What kinds of things would that service provide and do?
  • Customers: Who would use it?
  • Risks: Why wouldn't we want a central service?
  • Partners: Who would be involved?
Discussion, white-boarding
  • All
11:50 amNext steps and wrap-up
  • Rupert Berk


Secrets Management Solutions

Minimum Best Practices

  • Encrypt on disk
  • Avoid secrets in source control (and put in automated checks)
  • Avoid secrets in backups(and put in automated checks)
  • Rotate keys at least on departure of staff
  • Ensure access of least privilege
  • Manage master keys securely and redundantly e.g. root account

The following table describes some of the secrets management solutions being employed or considered by various UW teams. Teams will often use a hybrid of different solutions.

Automation opportunities increase by row (1-6).

SolutionBenefitsLimitationsExample Tools
1No master repo, encrypted on disk, manually copied between cluster members
  • No central attack vector (security by obscurity?)
  • Low cost
  • Risk of key loss if cluster destroyed
  • Difficult to distribute keys
  • Difficult to rotate keys
  • Users can mis-enter credentials
  • Risk of weak manual credentials
  • File system
  • SSH
2Master repo, mirrored on file system, encrypted on disk, manually copied between cluster members
  • Recovery from master
  • Encryption on disk
  • Difficult to distribute keys
  • Difficult to rotate keys
  • Users can mis-enter credentials
  • Risk of weak manual credentials
3Master repo, mirrored on file system, distributed via local custom, privileged automation  
  • (above plus ...)
  • Ease of automated key distribution
  • Difficult to rotate keys
  • Requires custom plug-ins and code to work across deployment ecosystem e.g. Ansible, Terraform.
  • UW LastPass (everyone)
  • LastPass CLI (ORIS)
  • PassBolt (ORIS)
  • Private SVN (AXDD)
  • git-crypt (AXDD, EDM)
  • Custom tools e.g. REF (UW-IT Managed Servers, UW-IT infra tools)
  • Ansible Vault (AXDD, EDM, EIP, MyPlan)
  • LetsEncrypt (EDM)
  • UW-IT certificate service APIs (prototype) (UW Managed Servers)

Managed cloud-native platform ("AWS IAM model")

  • Permissions are granted between resources using platform-specific IAM
  • Keys may be automatically provisioned
  • (above plus ...)
  • Credentials opaque to users
  • Built-in audit functions
  • High availability
  • Credentials opaque to users
  • May not work well for all tools and scenarios e.g. non-platform tools and integrations, CI/CD, application-specific keys.
5Managed key vault service
  • (above plus ...)
  • Short-lived, limited scope, dynamic, access tokens reduce attack surface
  • Flexible infrastructure and application-level key management
  • Cost of locally managed on-premise OSS service
  • Vendor lock-in, limited cross-vendor support.
6Security identity framework
  • (above plus ...)
  • Open-source framework reduces vendor lock-in
  • Not yet prime-time
  • SPIFFE (video), SPIRE (CNCF sandbox and incubation projects)

UW-IT Managed Secrets Service? Who would consider using it?

  • AXDD
  • EDM
  • Managed Servers
  • IAM
  • FSB
  • iSchool
  • Marketing
  • ORIS
  • UW Medicine


  • No labels