A RESTless week

I spent about a week of my time at Acquia on improving Drupal 8’s REST support.

That time was spent fixing, reviewing, triaging and documenting.

Drupal 8’s REST works very well, we just have to make it more friendly & helpful, remove Drupalisms and support more REST capabilities.

Fixing, reviewing & triaging

I went through the entire issue queues of rest.module, serialization.module and hal.module. I was able to mark about a dozen bug reports as duplicates, fix a dozen or so support requests, have reviewed probably a few dozen patches, rerolled about a dozen patches, created at least another dozen patches and … triaged 100% of the open issues. I clarified titles of >30 issues.

Now the rest.module issue queue (the most important one) fits on a single page again!¹ I collaborated a lot with neclimdul, klausi, damiankloip, dawehner and others.

dawehner and I decided to tag the issues that were especially relevant using the RX (REST Experience) issue tag.

I felt it was important to get a comprehensive picture of Drupal 8’s REST support state, so I insisted on going through all open issues (and was given the time to do so). This enabled me to document the current state of things (and upcoming improvements).

Documenting

So, I spent several days doing nothing else but writing and improving documentation, just like I did for the modules and subsystems I co-maintain. The following drupal.org handbook pages have either received minor updates, received complete overhauls or were written from scratch:

• d.o/documentation/modules/serialization
• d.o/developing/api/8/serialization
• d.o/documentation/modules/rest
• d.o/developing/api/8/rest
• d.o/documentation/modules/hal
• d.o/documentation/modules/basic_auth
• d.o/developing/api/8/authentication
• d.o/documentation/modules/rest/start
• d.o/documentation/modules/rest/get
• d.o/documentation/modules/rest/post
• d.o/documentation/modules/rest/patch
• d.o/documentation/modules/rest/delete

So, there you go, documentation covering fundamentals, like that for the Serialization API:

Serializing & deserializing

Using the serializer service’s (\Symfony\Component\Serializer\SerializerInterface) serialize() and deserialize() methods:

$output = $serializer->serialize($entity, 'json');
$entity = $serializer->deserialize($output, \Drupal\node\Entity\Node::class, 'json');

Serialization format encoding/decoding (format → array → format

The encoder (\Symfony\Component\Serializer\Encoder\EncoderInterface) and decoder (\Symfony\Component\Serializer\Encoder\DecoderInterface, to add support for encoding to new serialization formats (i.e. for reading data) and decoding from them (i.e. for writing data).

Normalization (array → object → array)

The normalizer (\Symfony\Component\Serializer\Normalizer\NormalizerInterface) and denormalizer (\Symfony\Component\Serializer\Normalizer\DenormalizerInterface), to add support for normalizing to a new normalization format. The default format is as close to a 1:1 mapping of the object data as possible, but other formats may want to omit e.g. local IDs (for example node IDs are local, UUIDs are global) or add additional metadata (such as URIs linking to related data).

Entity resolvers

In a Drupal context, usually it will be (content) entities that end up being serialized. When given an entity to normalize (object → array) and then encode (array → format), that entity may have references to other entities. Those references may use either UUIDs (\Drupal\serialization\EntityResolver\UuidResolver) or local IDs (\Drupal\serialization\EntityResolver\TargetIdResolver). For advanced use cases, additional mechanisms for referring to other entities may exist; in that case, you would add an additional entity resolver.

… to more practical information, such as the Getting started: REST configuration & REST request fundamentals handbook page for the rest module:

1. Configuration

First read RESTful Web Services API — Practical.

Now you know how to:

1. expose data as REST resources
2. grant the necessary permissions
3. customize a REST resource’s formats (JSON, XML, HAL+JSON, CSV …)
4. customize a REST resource’s authentication mechanisms (cookie, OAuth, OAuth 2.0 Token Bearer, HTTP Basic Authentication …)

Armed with that knowledge, you can configure a Drupal 8 site to expose data to precisely match your needs.

2. REST request fundamentals

2.1 Safe vs. unsafe methods

REST uses HTTP, and uses the HTTP verbs. The HTTP verbs (also called request methods) are: GET, HEAD, POST, PUT, DELETE, TRACE, OPTIONS, CONNECT and PATCH.
Some of these methods are safe: they are read-only. Hence they can never cause harm to the stored data, because they can’t manipulate them. The safe methods are HEAD, GET, OPTIONS and TRACE.
All other methods are unsafe, because they perform writes, and can hence manipulate stored data.

Note: PUT is not supported for good reasons.

2.2 Unsafe methods & CSRF protection: X-CSRF-Token request header

Drupal 8 protects its REST resources from CSRF attacks by requiring a X-CSRF-Token request header to be sent when using a non-safe method. So, when performing non-read-only requests, that token is required.
Such a token can be retrieved at /rest/session/token.

2.3 Format

When performing REST requests, you must inform Drupal about the serialization format you are using (even if only one is supported for a given REST resource). So:

1. Always specify the ?_format query argument, e.g. http://example.com/node/1?_format=json.
2. When sending a request body containing data in that format, specify the Content-Type request header. This is the case for POST and PATCH.

Note: Accept-header based content negotiation was removed from Drupal 8 because browsers and proxies had poor support for it.

3. Next

Now you’re ready to look at concrete examples, which start on the next page.

If that particular handbook page had already existed, it would have saved me so much time! The next page then contains examples for how to do GET requests, using various tools:

cURL

curl http://example.com/node/1?_format=hal_json

Guzzle

$response = \Drupal::httpClient()
  ->get('http://example.com/node/1?_format=hal_json', [
    'auth' => ['username', 'password'],
  ]);

$json_string = (string) $response->getBody();

jQuery

jQuery.ajax({
  url: 'http://example.com/node/1?_format=hal_json',
  method: 'GET',
  success: function (comment) {
    console.log(comment);
  }
});

… and the following pages then provide concrete examples (in those same tools) for POST, PATCH and DELETE requests².

Enjoy!

Why I did all of the above

It took me about three days to successfully PATCH a Comment entity³.

Why days?

I first forgot to specify the Content-Type request header. Then it turned out I also forgot the X-CSRF-Token request header — which was not documented anywhere to be a thing. I eventually found out about that Drupal-specific request header by analyzing the REST PATCH test coverage. Why did I not find it sooner? Because Drupal 8 was giving utterly unhelpful, and actually downright nonsensical (and incorrect!) responses⁴. It doesn’t end there though. Turns out that if you try to update an entity using JSON (and not HAL+JSON, which works fine), you MUST specify the bundle (otherwise it’s impossible to denormalize the entity you’re sending), but you also MUST NOT specify the entity type’s bundle if it’s a Comment (because you’re not allowed to modify this by CommentAccessControlHandler). So … it literally was impossible to update a comment⁵!

I didn’t have any experience with/knowledge about Drupal 8’s REST API. But I’m deeply familiar with Drupal 8. And it still took me days. Of course I wanted to prevent anyone from ever having to go through that.

Today, anybody would start at d.o/documentation/modules/rest/start and then look at d.o/documentation/modules/rest/patch and would hence be able to avoid all these pitfalls. Soon, Drupal will provide more helpful responses⁴ and allow comments to be updated⁵ using JSON.