State of JSON API (July 2018)

Quite a few people in the Drupal community are looking forward to see the JSON API module ship with Drupal 8 core.

Because:

• they want to use it on their projects
• the Admin UI & JS Modernization Initiative needs it
• they want to see Drupal 8 ship with a more capable RESTful HTTP API
• then Drupal will have a non-NIH (Not Invented Here) API but one that follows a widely used spec
• it enables them to build progressively decoupled components
• …

So where are things at?

Timeline

Let’s start with a high-level timeline:

1. The plan (intent) to move the JSON API module into Drupal core was approved by Drupal’s product managers and a framework manager 4 months ago, on March 19, 2018!
2. A core patch was posted on March 29 (issue #2843147). My colleague Gabe and I had already been working full time for a few months at that point to make the JSON API modules more stable: several security releases, much test coverage and so on.
3. Some reviews followed, but mostly the issue (#2843147) just sat there. Anybody was free to provide feedback. We encouraged people to review, test and criticize the JSON API contrib module. People did: another 1000 sites started using JSON API! Rather than commenting on the core issue, they filed issues against the JSON API contrib module!
4. Since December 2017, Gabe and I were still working on it full time, and e0ipso whenever his day job/free time allowed. Thanks to the test coverage Gabe and I had been adding, bugs were being fixed much faster than new ones were reported — and more often than not we found (long-existing) bugs before they were reported.
5. Then 1.5 week ago, on June 28, we released JSON API 1.22, the final JSON API 1.x release. That same day, we branched the 2.x version. More about that below.
6. The next day, on June 29, an updated core patch was posted. All feedback had been addressed!

June 29

I wrote in my comment:

Time to get this going again. Since #55, here’s what happened:

1. Latest release at #55: JSON API 1.14
2. Latest release today: JSON API 1.22
3. 69 commits: ($ git log --oneline --since "March 30 2018 14:21 CET" | wc -l)
4. Comprehensive test coverage completed (#2953318: Comprehensive JSON API integration test coverage phase 4: collections, filtering and sorting + #2953321: Comprehensive JSON API integration test coverage phase 5: nested includes and sparse field sets + #2972808: Comprehensive JSON API integration test coverage phase 6: POST/PATCH/DELETE of relationships)
5. Getting the test coverage to that point revealed some security vulnerabilities (1.16), and many before it (1.14, 1.10 …)
6. Ported many of the core REST improvements in the past 1.5 years to JSON API (1.15)
7. Many, many, many bugfixes, and much, much clean-up for future maintainability (1.16, 1.17, 1.18, 1.19, 1.20, 1.21, 1.22)

That’s a lot, isn’t it? :)

But there’s more! All of the above happened on the 8.x-1.x branch. As described in #2952293: Branch next major: version 2, requiring Drupal core >=8.5 (and mentioned in #61), we have many reasons to start a 8.x-2.x branch. (That branch was created months ago, but we kept them identical for months.)
Why wait so long? Because we wanted all >6000 JSON API users to be able to gently migrate from JSON API 1.x (on Drupal ⇐8.5) to JSON API 2.x (on Drupal >=8.5). And what better way to do that than to write comprehensive test coverage, and fixing all known problems that that surfaced? That’s what we’ve been doing the past few months! This massively reduces the risk of adding JSON API to Drupal core. We outlined a plan of must-have issues before going into Drupal core: #2931785: The path for JSON API to core — and they’re all DONE as of today! Dozens of bugs have been flushed out and fixed before they ever entered core. Important: in the past 6–8 weeks we’ve noticed a steep drop in the number of bug reports and support requests that have been filed against the JSON API module!

After having been tasked with maturing core’s REST API, and finding the less-than-great state that was in when Drupal 8 shipped, and having experienced how hard it is to improve it or even just fix bugs, this was a hard requirement for me. I hope it gives core committers the same feeling of relief as it gives me, to see that JSON API will on day one be in much better shape.

The other reason why it’s in much better shape, is that the JSON API module now has no API surface other than the HTTP API! No PHP API (its sole API was dropped in the 2.x branch: #2982210: Move EntityToJsonApi service to JSON API Extras) at all, only the HTTP API as specified by http://jsonapi.org/format/.

TL;DR: JSON API in contrib today is more stable, more reliable, more feature-rich than core’s REST API. And it does so while strongly complying with the JSON API spec: it’s far less of a Drupalism than core’s REST API.

So, with pride, and with lots of sweat (no blood and no tears fortunately), @gabesullice, @e0ipso and I present you this massively improved core patch!

EDIT: P.S.: 668K bytes of the 1.0M of bytes that this patch contains are for test coverage. That’s 2/3rds!

To which e0ipso replied:

So, with pride, and with lots of sweat (no blood and no tears fortunately), @gabesullice, @e0ipso and I present you this massively improved core patch!

So much pride! This was a long journey, that I walked (almost) alone for a couple of years. Then @Wim Leers and @gabesullice joined and carried this to the finish line. Such a beautiful collaboration!

:)

July 9

Then, about 12 hours ago, core release manager xjm and core framework manager effulgentsia posted a comment:

(@effulgentsia and @xjm co-authored this comment.) It’s really awesome to see the progress here on JSON API! @xjm and @effulgentsia discussed this with other core committers (@webchick, @Dries, @larowlan, @catch) and with the JSON API module maintainers. Based on what we learned in these discussions, we’ve decided to target this issue for an early feature in 8.7 rather than 8.6. Therefore, we will will set it 8.7 in a few days when we branch 8.7. Reviews and comments are still welcome in the meantime, whether in this issue, or as individual issues in the jsonapi issue queue. Feel free to stop reading this comment here, or continue reading if you want to know why it’s being bumped to 8.7. First, we want to give a huge applause for everything that everyone working on the jsonapi contrib module has done. In the last 3-4 months alone (since 8.5.0 was released and #44 was written):

• Over 100 issues in the contrib project have been closed.
• There are currently only 36 open issues, only 7 of which are bug reports.
• Per #62, the remaining bug fixes require breaking backwards compatibility for users of the 1.x module, so a final 1.x release has been released, and new features and BC-breaking bug fixes are now happening in the 2.x branch.
• Also per #62, an amazing amount of test coverage has been written and correspondingly there’s been a drop in new bug reports and support requests getting filed.
• The module is now extremely well-documented, both in the API documentation and in the Drupal.org handbook.

Given all of the above, why not commit #70 to core now, prior to 8.6 alpha? Well,

1. We generally prefer to commit significant new core features early in the release cycle for the minor, rather than toward the end. This means that this month and the next couple are the best time to commit 8.7.x features.
2. To minimize the disruption to contrib, API consumers, and sites of moving a stable module from core to contrib, we’d like to have it as a stable module in 8.7.0, rather than an experimental module in 8.6.0.
3. Per above, we’re not yet done breaking BC. The mentioned spec compliance issues still need more work.
4. While we’re still potentially evolving the API, it’s helpful to continue having the module in contrib for faster iteration and feedback.
5. Since the 2.x branch of JSON API was just branched, there are virtually no sites using it yet (only 23 as compared with the 6000 using 1.x). An alpha release of JSON API 2.x once we’re ready will give us some quick real-world testing of the final API that we’re targeting for core.
6. As @lauriii pointed out, an additional advantge of allowing a bit more time for API changes is that it allows more time for the Javascript Modernization Initiative, which depends on JSON API, to help validate that JSON API includes everything we need to have a fully decoupled admin frontend within Drupal core itself. (We wouldn’t block the module addition on the other initiative, but it’s an added bonus given the other reasons to target 8.7.)
7. While the module has reached maturity in contrib, we still need the final reviews and signoffs for the core patch. Given the quality of the contrib module this should go well, but it is a 1 MB patch (with 668K of tests, but that still means 300K+ of code to review.) :) We want to give our review of this code the attention it deserves.

None of the above aside from the last point are hard blockers to adding an experimental module to core. Users who prefer the stability of the 1.x module could continue to use it from contrib, thereby overriding the one in core. However, in the case of jsonapi, I think there’s something odd about telling site builders to experiment with the one in core, but if they want to use it in production, to downgrade to the one in contrib. I think that people who are actually interested in using jsonapi on their sites would be better off going to the contrib project page and making an explicit 1.x or 2.x decision from there. Meanwhile, we see what issues, if any, people run into when upgrading from 1.x to 2.x. When we’re ready to commit it to core, we’ll consider it at least beta stability (rather than alpha). Once again, really fantastic work here.

Next

So there you have it. JSON API will not be shipping in Drupal 8.6 this fall.
The primary reason being that it’s preferred for significant new core features to land early in the release cycle, especially ones shipping as stable from the start. This also gives the Admin UI & JS Modernization Initiative more time to actually exercise many parts of JSON API’s capabilities, and in doing so validate that it’s sufficiently capable to power it.

For us as JSON API module maintainers, it keeps things easier for a little while longer: once it’s in core, it’ll be harder to iterate: more process, slower test runs, commits can only happen by core committers and not by JSON API maintainers. Ideally, we’d commit JSON API to Drupal core with zero remaining bugs and tasks, with only feature requests being left. Good news: we’re almost there already: most open issues are feature requests!

For you as JSON API users, not much changes. Just keep using https://www.drupal.org/project/jsonapi. The 2.x branch introduced some breaking changes to better comply with the JSON API spec, and also received a few new small features. But we worked hard to make sure that disruption is minimal (example 1 2 3).¹
Use it, try to break it, report bugs. I’m confident you’ll have to try hard to find bugs … and yes, that’s a challenge to y’all!