Skip to end of metadata
Go to start of metadata

Summary

On December 19, and again on January 16, several of us met over lunch with the goal of designing a Web service that represents, exposes, and allows others to build upon today's UW Groups Service as a RESTful resource. These are our workshop proceedings.

The first session began with introductions followed by an overview of the nine-step design methodology described in the O'Reilly book "RESTful Web Services" by Leonard Richardson and Sam Ruby [groups:R&R]. By the end of this session we had covered roughly half of the design process and emerged having a strengthened sense of partnership to reconvene after the Winter break.

The second session began with a review of our earlier decisions and then picked up the process where we had left off, about midway, whereupon we moved forward and back through the design steps in order to extend and refine our original design. Although we didn't complete all the remaining steps, we closed the session with the sense that any remaining choices wouldn't significantly alter our basic design.

Participants

The workshop participants represented of a cross-organizational group of stakeholders (see list below). Several participants attended the two-day REST-Easy Workshop in November with Peter Lacey, and many of us are involved in the emerging Web Services SIG.

We also represented, rather opportunistically, several interesting, near-term use cases that argue in favor of opening the Groups Service to new data sources, including ASTRA roles provisioning, Catalyst Group Manager integration, and Graduate School data provisioning.

The workshop participants were: Rupert Berk, Miles Crawford, Nathan Dors, Jim Fox, Andrew Gorohoff, Jim Laney, RL "Bob" Morgan, Tracy Stenvik.

Proceedings

Step 1: Figure Out the Data Set

Since our goal was, and is, to create a resource that represents, exposes, and allows others to build upon the current Groups Service, we weren't figuring out a new data set from scratch, but rather constraining ourselves, pragmatically, to an existing one.

With that constraint in mind, we set out to "find all the nouns" in our data set. We promptly wrote some on the whiteboard and refined them into the following agreed upon list:

  • Group
  • Member
  • Owner
  • Stem
  • Username
  • Common name (cn)
  • Description
  • Registry ID (regid)

Note: We needed some tangible basis to discuss the current Groups Service data set; so we used an existing LDAP representation, as provided by the Groups Directory Service:

dn: serialNumber=<regid>,ou=groups,dc=washington,dc=edu
objectClass:  uwDepartmentGroup
serialNumber: <regid>
uwRegID:      <regid>
cn:           <group name>
description:  <group description>
owner:        uwNetID=<uwnetid>
member:       uwNetID=<uwnetid>
member:       uwNetID=<uwnetid>
member:       uwNetID=<uwnetid>
memberGroup:  cn=<group cn>

Step 2: Split the Data Set into Resources

Once we determined all the "nouns" in the data set, we went about identifying which ones to expose as HTTP resources addressable by URIs. We kept in mind that a resource is "anything interesting enough to be the target of a hypertext link" [groups:R&R, 112].

The following list summarizes how we separated the resources from metadata attributes (bold denotes areas of focus; question marks denote areas of uncertainty):

  • Group - resource
  • Member - resource; or attribute of a group resource?
  • Owner - resource?; or attribute of a group resource?
  • Stem - resource?
  • Username - attribute of a member or owner resource
  • Common name - attribute of a group resource
  • Description - attribute of a group resource
  • Registry ID - attribute of a group resource

To summarize, we decided that although each group falls into an encompassing hierarchy and namespace based on group naming stems, we would table these concepts and instead focus on groups, members, and owners as the resources of most interest. (Our initial lunchtime session was only two hours, and pizza was a factor.)

We therefore classified cn, description, and regid as likely attributes of a group resource, and similarly decided to pull username as an attribute under the member and owner resources. We acknowledged this classification would have implications on resource addresses and on "payload" representations, but we could revisit our rationale later.

The rationale for classifying members as a resource is noteworthy. It's because group memberships can be big, really big in some cases (e.g. all current UW students). Someone commented that it would be ridiculous to retrieve such a large membership and then put it right back, just to modify an attribute like cn or description. Besides, memberships are interesting in their own right, interesting enough to be the target of a hypertext link, and therefore interesting enough, and important enough, to be resources. (Owners seemed similar enough to treat the same.)

Step 3: Name the Resources with URIs

Once we had decided to expose groups, members, and possibly owners as resources, we started discussing how to name these resources with URI addresses. Here we kept several principles in mind. First, that resource URIs should be descriptive, friendly to us humans, and opaque to non-humans. Also, they should encode hierarchy (which we had tabled, above) into path variables. And lastly, they should be cool: "Cool URIs don't change." (Peter Lacey)

On that last point, we knew that common names change all the time, and that a group's regid stays the same but isn't human-friendly. So although we knew our resource URIs would have a group identifier in them, we weren't sure yet whether the identifier would be based on the group's common name or, more likely, its registry ID.

The following table summarizes how we expressed our resources as URI Templates (assuming an appropriate base URI, of course, including version):

Resource

URI

group

/group/{identifier}

member

/group/{identifier}/member/{username}

owner

/group/{identifier}/owner/{username}

Seeing these resource URIs written on the whiteboard was invigorating and nearly bewildering, raising all kinds of questions we would confront later in the workshop.

For example, we briefly considered what resource would be represented by dropping the trailing identifier: that is, what kind of root resource is "/group" and what representation, if any, would it return? Someone suggested (and wisely so) that it would return a 405 ("Method Not Allowed") until a version of the resource exposes functionality through the uniform interface at that address. By deferring the answer, we were putting our trust in the design methodology. And indeed, we would return to this question again, shortly after the pizza arrived. (Hint: revised URI Templates lurk below.)

Step 4: Expose a Subset of the Uniform Interface

Next we needed to decide how much of the uniform interface of HTTP request methods to expose for our resources. As soon as we started, the discussion quickly and chaotically alternated between our resource URI design (step 3) and what is supposed to happen at each URI (step 8).

We were rudderless ("where are the business requirements," someone said) until we focused on answering these questions for our resources [groups:R&R, 149-150]:

  • Will clients be creating new resources of this type?
  • When the client creates a new resource of this type, who's in charge of determining the new resource's URI? Is it the client or the server?
  • Will clients be modifying resources of this type?
  • Will clients be deleting resources of this type?
  • Will clients be fetching representations of resources of this type?

Focusing on these questions shed new light on our earlier questions about addressing group resources, by either common name or registry ID, and about whether or not to expose functionality at the root address.

In particular, by considering the uniform interface needed to create new group resources and to rename existing ones, we recognized that the registry ID is the canonical, invariant group identifier, so the common name would more appropriately be described as a name of convenience (suggesting, perhaps, the creation of a human-friendly convenience URI for addressability), as summarized in these revised URI Templates:

Resource

URI

group

/group/{regid}            (canonical URI)

group

/group/{cn}           (convenience URI)

member

/group/{identifier}/member/{username}

owner

/group/{identifier}/owner/{username}

Now that we had refined our resource URI design, we could consider and apply a common design pattern for creating new resources: Clients use PUT when they determine the URI a new resource should have; and clients use POST when the server determines the URI.

In our case, the canonical URI of a new group resource depends on a registry ID, and current business practice allows clients to generate one. Therefore clients will use PUT to create new group resources.

Note: Had current business practice run the other way, such that only the server determines the registry ID, clients would have used POST to create new group resources, and the server would have responded with a 201 ("Created") to provide the location of the new group.

The following table describes our design decisions on the uniform interface exposed for each URI (assuming an appropriate base URI with version). We included a description the business operations and some of the likely HTTP response codes:

URI

Method

Business Operation

Response Codes

/group

All

Not defined

405 ("Method Not Allowed")

/group/{cn}

GET

Retrieve a representation this group resource

200 ("OK")
404 ("Not Found")

/group/{regid}

GET

Retrieve a representation this group resource

200 ("OK")
404 ("Not Found")

 

PUT

Create or modify this group resource with this representation

200 ("OK")
400 ("Bad Request")

 

DEL

Delete this group resource

200 ("OK")
404 ("Not Found")

/group/{regid}/member

GET

Retrieve a representation of the membership of this group resource

200 ("OK")
404 ("Not Found")

 

PUT

Modify the membership of this group with this representation

200 ("OK")
400 ("Bad Request")

/group/{regid}/member/{memberid}

GET

Retrieve a representation of this member of this group

200 ("OK")
404 ("Not Found")

 

PUT

Add this member to this group

200 ("OK")
404 ("Not Found")

 

DEL

Delete this member from this group

200 ("OK")
404 ("Not Found")

/group/{regid}/owner

GET

Retrieve a representation of the ownership of this group resource

200 ("OK")
404 ("Not Found")

 

PUT

Modify the ownership of this group with this representation

200 ("OK")
400 ("Bad Request")

/group/{regid}/owner/{username}

GET

Retrieve a representation of this owner of this group

200 ("OK")
404 ("Not Found")

 

PUT

Add this owner to this group

200 ("OK")
404 ("Not Found")

 

DEL

Delete this owner from this group

200 ("OK")
404 ("Not Found")

Note: We didn't discuss response codes exhaustively, just enough to get a sense of what functionality we were exposing through the uniform interface; so general success and failure conditions are represented above. Further along (steps 8 and 9), we'd consider HTTP responses, whether to support conditional HTTP requests, and how to handle error conditions (including 401 "Not Authorized"). That's something we could wait for.

Step 5: Design the Representation(s) Accepted from the Client

The next step in the design methodology is to specify representations of resource states that clients can send as entity-bodies, through the uniform interface, to create and modify our resources. That is, we needed to define representation formats for new group, member, and owner resources.

The objective here is RESTful representation: "To convey the current state of the resource, and to link to possible new application and resource states," as well as "Connectedness: the ability to get from one resource to another by following links." [groups:R&R, 123-124] We acknowledged these principles, silently, by glancing and nodding at each other knowingly, and then approached the whiteboard.

First of all, we knew we didn't want to define a new content type. Also, we decided our resource state would be too complex to represent with simple key-value pairs. Therefore, we considered the two most obvious and best understood formats: a custom XML vocabulary and XHTML. We chose XHTML for the usual reasons and wrote our first angle bracket, then another, and another.

Soon we identified a basic XHTML document structure and class attributes that represent a new incoming group resource, as shown in Example 1.

Example 1. An XHTML representation of a group resource

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
  "http://www.w3.org/TR/xhtml11/DTD/xhtml11/dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
  <meta http-equiv="Content-Type" content="application/xhtml+xml; charset=utf-8"/>
</head>
<body>
  <div class="group">
    <span class="regid">{regid}</span>
    <span class="description">{description}</span>
    <ul class="names">
      <li class="name">{cn}</li>
    </ul>
    <a rel="members" href="{root}/group/{regid}/member">members</a>
    <a rel="owners" href="{root}/group/{regid}/owner">owners</a>
  </div>
</body>
</html>

Next, we designed the client representation of a member resource, as shown in Example 2:

Example 2. An XHTML representation of a member resource

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
  "http://www.w3.org/TR/xhtml11/DTD/xhtml11/dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
  <meta http-equiv="Content-Type" content="application/xhtml+xml; charset=utf-8"/>
</head>
<body>
  <div class="group">
    <span class="regid">{regid}</span>
    <ul class="members">
      <li>
        <a rel="memberlink" class="member" type="{member_type}"
           href="{root}/group/{regid}/member/{member_id}">{member_id}</a>
      </li>
    </ul>
  </div>
</body>
</html>

Similarly, we designed the client representation of an owner resource, as shown in Example 3:

Example 3. An XHTML representation of a owner resource

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
  "http://www.w3.org/TR/xhtml11/DTD/xhtml11/dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
  <meta http-equiv="Content-Type" content="application/xhtml+xml; charset=utf-8"/>
</head>
<body>
  <div class="group">
    <span class="regid">{regid}</span>
    <ul class="owners">
      <li>
        <a rel="ownerlink" class="owner" type="{owner_type}"
          href="{root}/group/{regid}/owner/{owner_id}">{owner_id}</a>
      </li>
    </ul>
  </div>
</body>
</html>

Step 6: Design the Representation(s) Served to the Client

Although the design process differentiates incoming from outgoing representations (steps 5 and 6, respectively), we didn't expect ours would diverge during design. In fact, we wanted to ease development and use of our service by designing it to serve outgoing representations in the same format that it accepts as incoming representations. Additionally, we understood that "representations should be human-readable, but computer-oriented." [groups:R&R, 234]

To make representations served to clients human-readable, we had to look beyond the more computer-oriented class attributes (our microformat) and spend a little time ensuring our XHTML elements made sense (<ul> elements for unordered lists, for example; and <div> elements for structure). This discussion influenced the client representations as well (Examples 1-3 above).

At this point, we wanted to identify any additional representations that would be served to clients. To do so, we looked for locations where we were exposing the uniform interface in such a way that a 200 ("OK") response would result in a representation being sent to the client (see table under step 4).

Two similar cases were identified where additional representations would be served to the client: when a client retrieves a member of a group or an owner of a group, as one might do verify membership or ownership, respectively. For both cases, since the response code is the client's primary concern, we designed a simple server representation for successful requests, as shown in Example 4:

Example 4. An XHTML representation of a member of a group resource

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
  "http://www.w3.org/TR/xhtml11/DTD/xhtml11/dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
  <meta http-equiv="Content-Type" content="application/xhtml+xml; charset=utf-8"/>
  <title></title>
</head>
<body>
  {member_id} is in {regid}
  <div class="parent_group">
    <a rel="parent_group"
     class=""
     href="{root}/group/{regid}">{regid}</a>
  </div>
</body>
</html>

Lastly, we considered server states that might be represented to clients as lists, directories, or collections of group resources. These states hadn't been discussed much, since they weren't in the initial data set, but we anticipated lists of groups nonetheless. In particular, lists of groups produced by sending inputs to an algorithmic search resource or lists of groups subsumed under a group naming stem resource might require representation, as shown in Example 5:

Example 5. An XHTML representation of a list of group references

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
  "http://www.w3.org/TR/xhtml11/DTD/xhtml11/dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
  <meta http-equiv="Content-Type" content="application/xhtml+xml; charset=utf-8"/>
  <title></title>
</head>
<body>
  <div>
    <ul class="groupreferences">
      <li class="groupreference">
        <span class="regid">{regid}</span>
        <span class="description">{description}</span>
        <ul>
          <li><a class="name" href="{root}/group/{regid}">{cn}</a></li>
        </ul>
      </li>
      <!-- additional li element for each group in the list -->
   </ul>
  </div>
</body>
</html>

Step 7: Link This Resource to Existing Resources

Now it was time to link our resources together and to other resources. Since we had defined our resources in relation to each other, links were already present in our representations:

  • Each group linked to its membership and ownership resources.
  • Each membership and ownership linked to individual members and owners, respectively.
  • Each member and owner of a group linked back to the appropriate parent group.

Yet we couldn't say that links were plentiful, let alone abundant. And the links we did define may not have captured their semantic relationships as well as we would have liked through rel and rev attributes. In other words, a typical version "one dot uh-oh" design.

In fact, our design might have been described as being congenitally shy (serendipitously challenged, "doesn't play well with others", etc.) because we made no decisions to link to other less obvious application states or "nearby" resources. For example, we could have linked representations of individual members and owners to new lists of groups that they're a member of or that they own; all we'd need to do is link to a search resource and provide the individual as an input. But decisions around exposing these levers of application state would ultimately be deferred to implementation, as would hypermedia (embedded forms) that might suggest to clients how to create, modify, and search for our resources.

We also talked about what additional UW resources exist on today's human-oriented web that might make interesting targets of links: vCards, for example, or other representations available via the UW faculty/staff/student directory. But we didn't design for those either.

Step 8: What's Supposed to Happen?

Finally, a chance to explore everything that's supposed to happen. We'd come a long way during our two workshop sessions and we were eager to discuss what's really supposed to happen when clients use our service.

For instance, we wanted to explore how search, overloaded POST, and other levers might connect our resources to solving real business problems. We wanted to make something, anything, happen at our service's root URI. And we wanted especially to explore authentication and authorization. After all, this step is about "making sure that client requests are correctly turned into responses" [groups:R&R, 137]. And correctness involves everything from understanding business requirements to defining business practices and supporting business policies.

But this may not have been what Richardson & Ruby had in mind for us. Step 8 is in fact about HTTP responses in a more narrow sense. Besides, we had to admit that our eagerness to discuss broader functional requirements and our remaining time together weren't available in equal amounts. Therefore we rallied the discussion toward topics that we'd need at least some consensus around to ease implementation of our HTTP responses, including [groups:in part from R&R, 155]:

  • Assessing HTTP request methods and headers
  • Assessing incoming representations, if any
  • Deciding how the request will affect resource state, if at all
  • Deciding appropriate numeric HTTP response codes
  • Deciding appropriate HTTP response headers
  • Providing an entity-body, if necessary

We had already covered this ground a bit while exposing our resources to the uniform interface (see step 4). However, we had limited ourselves to the simplest response codes, 200 ("OK"), 400 ("Bad Request"), and 404 ("Not Found"). Those response codes assume the server either doesn't need to authorize the client request or has successfully done so. Now we needed to identify what other important responses needed consideration.

One interesting prior decision concerned responding to client requests to create group resources. We had decided clients could use PUT, but the responses could have gone a couple of ways: either return 200 ("OK") and an entity-body, or return 201 ("Created") and a location header. We decided 200 ("OK") and an entity-body would suffice.

We also considered support for conditional HTTP requests to be good design. By adding the appropriate response headers to group creations and modifications (namely, by adding Last-Modified and ETag response headers), we not only could make retrieving resources more efficient but also could ensure resource updates were handled correctly.

Next, we considered the importance of the Content-Type header to both the human web and programmable web. While our representations were decidedly "application/xhtml+xml", and while we'd like to be true to this format, Internet Explorer 7.0 doesn't display it, so we weren't sure how to advise what to implement. Feedback from customers would probably have to decide this one.

Lastly, we discussed adding and overloading POST operations in order to "tunnel" or simulate parts of the uniform interface for browsers that don't support the PUT and DELETE operations needed to create, modify, and delete our resources. This didn't seem like a version one requirement, so we punted the idea forward to some future version of the service.

Step 9: Consider error conditions: what might go wrong?

Perhaps because this was the last step in the design recipe, or maybe because we were pre-fatigued by the fine detail it involved, we spent very little effort identifying error conditions and appropriate responses for them. Conditional HTTP was the one exception.

Our use of conditional HTTP operations required some design to prevent clients from updating groups without providing the correct If-Match header upfront. Such requests should fail and most likely fail with a 409 ("Conflict") or 412 ("Precondition Failed") response.

Beyond that special consideration, we decided we could iterate over the following error conditions and suggested response codes during implementation [groups:derived from R&R 156, 165]:

  • Client representation is unintelligible; returns 415 ("Unsupported Media Type").
  • Client representation is missing, ill-formed; returns 400 ("Bad Request").
  • Client request induces inconsistent or impossible server state; returns 400 or 409 ("Conflict").
  • Client can't be authenticated; returns 401 ("Unauthorized").
  • Client request is unauthorized; returns 403 ("Forbidden") or, better yet, 404 ("Not Found")
  • Server has unspecified problem; returns 500 ("Internal Server Error" or 503 ("Service Unavailable")

Conclusions 

What can be concluded from our design workshop? First, that cross-organizational collaboration works, especially when the participants have a shared stake in the outputs, as we did. Next, that cross-organizational collaboration improves design choices by drawing on diverse perspectives and differing levels of technical expertise, as ours were. Lastly, that cross-organizational collaboration is supported by processes like the nine-step RESTful design methodology because they provide direction and focus attention on decision-making and outputs at each stage, as we needed.

It's also fair to conclude, in a more topical and historical sense, that in the future we'll be able to apply hindsight and added experience in RESTful design to the decisions we made during this workshop and therefore learn from our mistakes and assumptions and design improved future versions. Perhaps we should have explored formats like JSON, ATOM, and GData, but they were new and esoteric to us (and therefore not discussed). And although we were designing an XHTML microformat for group data (by adding machine-readable semantic through our class attributes), we didn't consider tieing our collaboration to active discussions of such microformats. Perhaps we should have.

But there is only so much we could do over two two-hour sessions spread over two months. Much was deferred to implementation discussions. Yet we believe we've contributed something of value to others in the UW community designing RESTful Web services. And so, on that final encouraging note, our workshop proceedings conclude.