Representational State Transfer (REST) is an approach to designing a web service to allow a user to retrieve and update information about "resources". It’s become the dominant approach when designing APIs for use over the web.
We’ve built a working web app without needing an API so far. Why might we want one? One motivation might be to improve the user experience by making the site more dynamic. Rather than waiting for the page to refresh after each addition to a list, we can use JavaScript to fire off those requests asynchronously to our API, and give the user a more interactive feeling.
Perhaps more interestingly, once we’ve built an API, we can interact with our back-end application via other mechanisms than the browser. A mobile app might be one new candidate client application, another might be some sort of command-line application, or other developers might be able to build libraries and tools around your backend.
In this chapter we’ll see how to build an API "by hand". In the next, I’ll give an overview of how to use a popular tool from the Django ecosystem called Django-Rest-Framework.
I won’t convert the entirety of the app for now; we’ll start by assuming we have an existing list. REST defines a relationship between URLs and the HTTP methods (GET and POST, but also the more funky ones like PUT and DELETE) which will guide us in our design.
The Wikipedia entry on REST has a good overview. In brief:
-
Our new URL structure will be '/api/lists/{id}/'
-
GET will give you details of a list (including all its items) in JSON format
-
POST lets you add an item
We’ll take the code from its state at the end of [chapter_page_pattern].
If we were
building an API that was entirely agnostic about its clients, we might
want to think about what levels to test it at. The equivalent of functional
tests would perhaps spin up a real server (maybe using LiveServerTestCase)
and interact with it using the requests library. We’d have to think carefully
about how to set up fixtures (if we use the API itself, that introduces a lot
of dependencies between tests) and what additional layer of lower-level/unit
tests might be most useful to us. Or we might decide that a single layer of
tests using the Django Test Client would be enough.
As it is, we’re building an API in the context of a browser-based client side. We want to start using it on our production site, and have the app continue to provide the same functionality as it did before. So our functional tests will continue to serve the role of being the highest-level tests, and of checking the integration between our JavaScript and our API.
That leaves the Django Test Client as a natural place to site our lower-level tests. Let’s start there.
We start with a unit test that just checks that our new URL structure returns a 200 response to GET requests, and that it uses the JSON format (instead of HTML):
import json
from django.test import TestCase
from lists.models import List, Item
class ListAPITest(TestCase):
base_url = '/api/lists/{}/' #(1)
def test_get_returns_json_200(self):
list_ = List.objects.create()
response = self.client.get(self.base_url.format(list_.id))
self.assertEqual(response.status_code, 200)
self.assertEqual(response['content-type'], 'application/json')-
Using a class-level constant for the URL under test is a new pattern we’ll introduce for this appendix. It’ll help us to remove duplication of hardcoded URLs. You could even use a call to
reverseto reduce duplication even further.
First we wire up a couple of 'urls' files:
from django.conf.urls import include, url
from accounts import urls as accounts_urls
from lists import views as list_views
from lists import api_urls
from lists import urls as list_urls
urlpatterns = [
url(r'^$', list_views.home_page, name='home'),
url(r'^lists/', include(list_urls)),
url(r'^accounts/', include(accounts_urls)),
url(r'^api/', include(api_urls)),
]and:
from django.conf.urls import url
from lists import api
urlpatterns = [
url(r'^lists/(\d+)/$', api.list, name='api_list'),
]And the actual core of our API can live in a file called 'api.py'. Just three lines should be enough:
from django.http import HttpResponse
def list(request, list_id):
return HttpResponse(content_type='application/json')The tests should pass, and we have the basic piping together:
$ python manage.py test lists [...] .................................................. --------------------------------------------------------------------- Ran 50 tests in 0.177s OK
Our next step is to get our API to actually respond with some content—specifically, a JSON representation of our list items:
def test_get_returns_items_for_correct_list(self):
other_list = List.objects.create()
Item.objects.create(list=other_list, text='item 1')
our_list = List.objects.create()
item1 = Item.objects.create(list=our_list, text='item 1')
item2 = Item.objects.create(list=our_list, text='item 2')
response = self.client.get(self.base_url.format(our_list.id))
self.assertEqual(
json.loads(response.content.decode('utf8')), #(1)
[
{'id': item1.id, 'text': item1.text},
{'id': item2.id, 'text': item2.text},
]
)-
This is the main thing to notice about this test. We expect our response to be in JSON format; we use
json.loads()because testing Python objects is easier than messing about with raw JSON strings.
And the implementation, conversely, uses json.dumps():
import json
from django.http import HttpResponse
from lists.models import List, Item
def list(request, list_id):
list_ = List.objects.get(id=list_id)
item_dicts = [
{'id': item.id, 'text': item.text}
for item in list_.item_set.all()
]
return HttpResponse(
json.dumps(item_dicts),
content_type='application/json'
)A nice opportunity to use a list comprehension!
The second thing we need from our API is the ability to add new items to our list by using a POST request. We’ll start with the "happy path":
def test_POSTing_a_new_item(self):
list_ = List.objects.create()
response = self.client.post(
self.base_url.format(list_.id),
{'text': 'new item'},
)
self.assertEqual(response.status_code, 201)
new_item = list_.item_set.get()
self.assertEqual(new_item.text, 'new item')And the implementation is similarly simple—basically the same as what we do in our normal view, but we return a 201 rather than a redirect:
def list(request, list_id):
list_ = List.objects.get(id=list_id)
if request.method == 'POST':
Item.objects.create(list=list_, text=request.POST['text'])
return HttpResponse(status=201)
item_dicts = [
[...]And that should get us started:
$ python manage.py test lists [...] Ran 52 tests in 0.177s OK
|
Note
|
One of the fun things about building a REST API is that you get to use a few more of the full range of HTTP status codes. |
Don’t even 'think' of doing Ajax testing without a mocking library. Different test frameworks and tools have their own; 'Sinon' is generic. It also provides JavaScript mocks, as we’ll see…
Start by downloading it from its site, http://sinonjs.org/, and putting it into our 'lists/static/tests/' folder.
Then we can write our first Ajax test:
<div id="qunit-fixture">
<form>
<input name="text" />
<div class="has-error">Error text</div>
</form>
<table id="id_list_table"> (1)
</table>
</div>
<script src="../jquery-3.1.1.min.js"></script>
<script src="../list.js"></script>
<script src="qunit-2.0.1.js"></script>
<script src="sinon-1.17.6.js"></script> (2)
<script>
/* global sinon */
var server;
QUnit.testStart(function () {
server = sinon.fakeServer.create(); //(3)
});
QUnit.testDone(function () {
server.restore(); //(3)
});
QUnit.test("errors should be hidden on keypress", function (assert) {
[...]
QUnit.test("should get items by ajax on initialize", function (assert) {
var url = '/getitems/';
window.Superlists.initialize(url);
assert.equal(server.requests.length, 1); //(4)
var request = server.requests[0];
assert.equal(request.url, url);
assert.equal(request.method, 'GET');
});
</script>-
We add a new item to the fixture
divto represent our list table. -
We import 'sinon.js' (you’ll need to download it and put it in the right folder).
-
testStartandtestDoneare the QUnit equivalents ofsetUpandtearDown. We use them to tell Sinon to start up its Ajax testing tool, thefakeServer, and make it available via a globally scoped variable calledserver. -
That lets us make assertions about any Ajax requests that were made by our code. In this case, we test what URL the request went to, and what HTTP method it used.
To actually make our Ajax request, we’ll use the
jQuery Ajax helpers, which are 'much'
easier than trying to use the low-level browser standard XMLHttpRequest objects:
@@ -1,6 +1,10 @@
window.Superlists = {};
-window.Superlists.initialize = function () {
+window.Superlists.initialize = function (url) {
$('input[name="text"]').on('keypress', function () {
$('.has-error').hide();
});
+
+ $.get(url);
+
};
+That should get our test passing:
5 assertions of 5 passed, 0 failed. 1. errors should be hidden on keypress (1) 2. errors aren't hidden if there is no keypress (1) 3. should get items by ajax on initialize (3)
Well, we might be pinging out a GET request to the server, but what about actually 'doing' something? How do we test the actual "async" part, where we deal with the (eventual) response?
This is a major reason to love Sinon. server.respond() allows us to exactly
control the flow of the asynchronous code.
QUnit.test("should fill in lists table from ajax response", function (assert) {
var url = '/getitems/';
var responseData = [
{'id': 101, 'text': 'item 1 text'},
{'id': 102, 'text': 'item 2 text'},
];
server.respondWith('GET', url, [
200, {"Content-Type": "application/json"}, JSON.stringify(responseData) //(1)
]);
window.Superlists.initialize(url); //(2)
server.respond(); //(3)
var rows = $('#id_list_table tr'); //(4)
assert.equal(rows.length, 2);
var row1 = $('#id_list_table tr:first-child td');
assert.equal(row1.text(), '1: item 1 text');
var row2 = $('#id_list_table tr:last-child td');
assert.equal(row2.text(), '2: item 2 text');
});-
We set up some response data for Sinon to use, telling it what status code, headers, and importantly what kind of response JSON we want to simulate coming from the server.
-
Then we call the function under test.
-
Here’s the magic. 'Then' we can call
server.respond(), whenever we like, and that will kick off all the async part of the Ajax loop—that is, any callback we’ve assigned to deal with the response. -
Now we can quietly check whether our Ajax callback has actually populated our table with the new list rows…
The implementation might look something like this:
if (url) {
$.get(url).done(function (response) { //(1)
var rows = '';
for (var i=0; i<response.length; i++) { //(2)
var item = response[i];
rows += '\n<tr><td>' + (i+1) + ': ' + item.text + '</td></tr>';
}
$('#id_list_table').html(rows);
});
}|
Tip
|
We’re lucky because of the way jQuery registers its callbacks for Ajax when we use
the .done() function. If you want to switch to the more standard JavaScript Promise
.then() callback, we get one more "level" of async. QUnit does have a
way of dealing with that. Check out the docs for the
async function.
Other test frameworks have something similar.
|
We break it first, by removing the list table {% for %} loop from the
lists.html template:
@@ -6,9 +6,6 @@
{% block table %}
<table id="id_list_table" class="table">
- {% for item in list.item_set.all %}
- <tr><td>{{ forloop.counter }}: {{ item.text }}</td></tr>
- {% endfor %}
</table>
{% if list.owner %}|
Note
|
This will cause one of the unit tests to fail. It’s OK to delete that test at this point. |
By removing the non-Ajax version of the lists page, I’ve removed the option of graceful degradation—that is, keeping a version of the site that will still work without JavaScript.
This used to be an accessibility issue: "screen reader" browsers for visually impaired people used not to have JavaScript, so relying entirely on JS would exclude those users. That’s not so much of an issue any more, as I understand it. But some users will block JavaScript for security reasons.
Another common problem is differing levels of JavaScript support in different browsers. This is a particular issue if you start adventuring off in the direction of "modern" frontend development and ES2015.
In short, it’s always nice to have a non-JavaScript "backup". Particularly if you’ve built a site that works fine without it, don’t throw away your working "plain old" HTML version too hastily. I’m just doing it because it’s convenient for what I want to demonstrate.
That causes our basic FT to fail:
$ python manage.py test functional_tests.test_simple_list_creation
[...]
FAIL: test_can_start_a_list_for_one_user
[...]
File "...python-tdd-book/functional_tests/test_simple_list_creation.py", line
32, in test_can_start_a_list_for_one_user
self.wait_for_row_in_list_table('1: Buy peacock feathers')
[...]
AssertionError: '1: Buy peacock feathers' not found in []
[...]
FAIL: test_multiple_users_can_start_lists_at_different_urls
FAILED (failures=2)
Let’s add a block called {% scripts %} to the base template, which we
can selectively override later in our lists page:
<script src="/static/list.js"></script>
{% block scripts %}
<script>
$(document).ready(function () {
window.Superlists.initialize();
});
</script>
{% endblock scripts %}
</body>And now in 'list.html' we add a slightly different call to initialize, with
the correct URL:
{% block scripts %}
<script>
$(document).ready(function () {
var url = "{% url 'api_list' list.id %}";
window.Superlists.initialize(url);
});
</script>
{% endblock scripts %}And guess what? The test passes!
$ python manage.py test functional_tests.test_simple_list_creation [...] Ran 2 test in 11.730s OK
That’s a pretty good start!
Now if you run all the FTs you’ll see we’ve got some failures in other FTs, so we’ll have to deal with them. Also, we’re using an old-fashioned POST from the form, with page refresh, so we’re not at our trendy hipster single-page app yet. But we’ll get there!
First we give our list form an id so we can pick it up easily in our JS:
<h1>{% block header_text %}{% endblock %}</h1>
{% block list_form %}
<form id="id_item_form" method="POST" action="{% block form_action %}{% endblock %}">
{{ form.text }}
[...]Next tweak the fixture in our JS test to reflect that ID, as well as the CSRF token that’s currently on the page:
@@ -9,9 +9,14 @@
<body>
<div id="qunit"></div>
<div id="qunit-fixture">
- <form>
+ <form id="id_item_form">
<input name="text" />
- <div class="has-error">Error text</div>
+ <input type="hidden" name="csrfmiddlewaretoken" value="tokey" />
+ <div class="has-error">
+ <div class="help-block">
+ Error text
+ </div>
+ </div>
</form>And here’s our test:
QUnit.test("should intercept form submit and do ajax post", function (assert) {
var url = '/listitemsapi/';
window.Superlists.initialize(url);
$('#id_item_form input[name="text"]').val('user input'); //(1)
$('#id_item_form input[name="csrfmiddlewaretoken"]').val('tokeney'); //(1)
$('#id_item_form').submit(); //(1)
assert.equal(server.requests.length, 2); //(2)
var request = server.requests[1];
assert.equal(request.url, url);
assert.equal(request.method, "POST");
assert.equal(
request.requestBody,
'text=user+input&csrfmiddlewaretoken=tokeney' //(3)
);
});-
We simulate the user filling in the form and hitting Submit.
-
We now expect that there should be a second Ajax request (the first one is the GET for the list items table).
-
We check our POST
requestBody. As you can see, it’s URL-encoded, which isn’t the most easy value to test, but it’s still just about readable.
And here’s how we implement it:
[...]
$('#id_list_table').html(rows);
});
var form = $('#id_item_form');
form.on('submit', function(event) {
event.preventDefault();
$.post(url, {
'text': form.find('input[name="text"]').val(),
'csrfmiddlewaretoken': form.find('input[name="csrfmiddlewaretoken"]').val(),
});
});That gets our JS tests passing but it breaks our FTs, because, although we’re doing our POST all right, we’re not updating the page after the POST to show the new list item:
$ python manage.py test functional_tests.test_simple_list_creation [...] AssertionError: '2: Use peacock feathers to make a fly' not found in ['1: Buy peacock feathers']
We want our client side to update the table of items after the Ajax POST completes. Essentially it’ll do the same work as we do as soon as the page loads, retrieving the current list of items from the server, and filling in the item table.
Sounds like a helper function is in order!
window.Superlists = {};
window.Superlists.updateItems = function (url) {
$.get(url).done(function (response) {
var rows = '';
for (var i=0; i<response.length; i++) {
var item = response[i];
rows += '\n<tr><td>' + (i+1) + ': ' + item.text + '</td></tr>';
}
$('#id_list_table').html(rows);
});
};
window.Superlists.initialize = function (url) {
$('input[name="text"]').on('keypress', function () {
$('.has-error').hide();
});
if (url) {
window.Superlists.updateItems(url);
var form = $('#id_item_form');
[...]That was just a refactor; now we check that the JS tests all still pass:
12 assertions of 12 passed, 0 failed. 1. errors should be hidden on keypress (1) 2. errors aren't hidden if there is no keypress (1) 3. should get items by ajax on initialize (3) 4. should fill in lists table from ajax response (3) 5. should intercept form submit and do ajax post (4)
Now how to test that our Ajax POST calls updateItems on POST success? We
don’t want to dumbly duplicate the code that simulates a server response
and checks the items table manually…how about a mock?
First we set up a thing called a "sandbox". It will keep track of all the mocks we create, and make sure to un-monkeypatch all the things that have been mocked after each test:
var server, sandbox;
QUnit.testStart(function () {
server = sinon.fakeServer.create();
sandbox = sinon.sandbox.create();
});
QUnit.testDone(function () {
server.restore();
sandbox.restore(); //(1)
});-
This
.restore()is the important part; it undoes all the mocking we’ve done in each test.
QUnit.test("should call updateItems after successful post", function (assert) {
var url = '/listitemsapi/';
window.Superlists.initialize(url); //(1)
var response = [
201,
{"Content-Type": "application/json"},
JSON.stringify({}),
];
server.respondWith('POST', url, response); //(1)
$('#id_item_form input[name="text"]').val('user input');
$('#id_item_form input[name="csrfmiddlewaretoken"]').val('tokeney');
$('#id_item_form').submit();
sandbox.spy(window.Superlists, 'updateItems'); //(2)
server.respond(); //(2)
assert.equal(
window.Superlists.updateItems.lastCall.args, //(3)
url
);
});-
First important thing to notice: We only set up our server response 'after' we do the initialize. We want this to be the response to the POST request that happens on form submit, not the response to the initial GET request. (Remember our lesson from [chapter_javascript]? One of the most challenging things about JS testing is controlling the order of execution.)
-
Similarly, we only start mocking our helper function 'after' we know the first call for the initial GET has already happened. The
sandbox.spycall is what does the job thatpatchdoes in Python tests. It replaces the given object with a mock version. -
Our
updateItemsfunction has now grown some mocky extra attributes, likelastCallandlastCall.args, which are like the Python mock’scall_args.
To get it passing, we first make a deliberate mistake, to check that our tests really do test what we think they do:
$.post(url, {
'text': form.find('input[name="text"]').val(),
'csrfmiddlewaretoken': form.find('input[name="csrfmiddlewaretoken"]').val(),
}).done(function () {
window.Superlists.updateItems();
});Yep, we’re almost there but not quite:
12 assertions of 13 passed, 1 failed.
[...]
6. should call updateItems after successful post (1, 0, 1)
1. failed
Expected: "/listitemsapi/"
Result: []
Diff: "/listitemsapi/"[]
Source: file://...python-tdd-book/lists/static/tests/tests.html:124:15
And we fix it thusly:
}).done(function () {
window.Superlists.updateItems(url);
});And our FT passes! Or at least one of them does. The others have problems, and we’ll come back to them shortly.
First, I’m not happy until we’ve seen through this refactor, and made our unit tests match the code a little more:
@@ -50,9 +50,19 @@ QUnit.testDone(function () {
});
-QUnit.test("should get items by ajax on initialize", function (assert) {
+QUnit.test("should call updateItems on initialize", function (assert) {
var url = '/getitems/';
+ sandbox.spy(window.Superlists, 'updateItems');
window.Superlists.initialize(url);
+ assert.equal(
+ window.Superlists.updateItems.lastCall.args,
+ url
+ );
+});
+
+QUnit.test("updateItems should get correct url by ajax", function (assert) {
+ var url = '/getitems/';
+ window.Superlists.updateItems(url);
assert.equal(server.requests.length, 1);
var request = server.requests[0];
@@ -60,7 +70,7 @@ QUnit.test("should get items by ajax on initialize", function (assert) {
assert.equal(request.method, 'GET');
});
-QUnit.test("should fill in lists table from ajax response", function (assert) {
+QUnit.test("updateItems should fill in lists table from ajax response", function (assert) {
var url = '/getitems/';
var responseData = [
{'id': 101, 'text': 'item 1 text'},
@@ -69,7 +79,7 @@ QUnit.test("should fill in lists table from ajax response", function [...]
server.respondWith('GET', url, [
200, {"Content-Type": "application/json"}, JSON.stringify(responseData)
]);
- window.Superlists.initialize(url);
+ window.Superlists.updateItems(url);
server.respond();And that should give us a test run that looks like this instead:
14 assertions of 14 passed, 0 failed. 1. errors should be hidden on keypress (1) 2. errors aren't hidden if there is no keypress (1) 3. should call updateItems on initialize (1) 4. updateItems should get correct url by ajax (3) 5. updateItems should fill in lists table from ajax response (3) 6. should intercept form submit and do ajax post (4) 7. should call updateItems after successful post (1)
If you do a full test run, you should find two of the validation FTs are failing:
$ python manage.py test [...] ERROR: test_cannot_add_duplicate_items (functional_tests.test_list_item_validation.ItemValidationTest) [...] ERROR: test_error_messages_are_cleared_on_input (functional_tests.test_list_item_validation.ItemValidationTest) [...] selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: .has-error
I won’t spell this all out for you, but here’s at least the unit tests you’ll need:
from lists.forms import DUPLICATE_ITEM_ERROR, EMPTY_ITEM_ERROR
[...]
def post_empty_input(self):
list_ = List.objects.create()
return self.client.post(
self.base_url.format(list_.id),
data={'text': ''}
)
def test_for_invalid_input_nothing_saved_to_db(self):
self.post_empty_input()
self.assertEqual(Item.objects.count(), 0)
def test_for_invalid_input_returns_error_code(self):
response = self.post_empty_input()
self.assertEqual(response.status_code, 400)
self.assertEqual(
json.loads(response.content.decode('utf8')),
{'error': EMPTY_ITEM_ERROR}
)
def test_duplicate_items_error(self):
list_ = List.objects.create()
self.client.post(
self.base_url.format(list_.id), data={'text': 'thing'}
)
response = self.client.post(
self.base_url.format(list_.id), data={'text': 'thing'}
)
self.assertEqual(response.status_code, 400)
self.assertEqual(
json.loads(response.content.decode('utf8')),
{'error': DUPLICATE_ITEM_ERROR}
)And on the JS side:
QUnit.test("should display errors on post failure", function (assert) {
var url = '/listitemsapi/';
window.Superlists.initialize(url);
server.respondWith('POST', url, [
400,
{"Content-Type": "application/json"},
JSON.stringify({'error': 'something is amiss'})
]);
$('.has-error').hide();
$('#id_item_form').submit();
server.respond(); // post
assert.equal($('.has-error').is(':visible'), true);
assert.equal($('.has-error .help-block').text(), 'something is amiss');
});
QUnit.test("should hide errors on post success", function (assert) {
[...]You’ll also want some modifications to 'base.html' to make it compatible with both displaying Django errors (which the home page still uses for now) and errors from JavaScript:
@@ -51,17 +51,21 @@
<div class="col-md-6 col-md-offset-3 jumbotron">
<div class="text-center">
<h1>{% block header_text %}{% endblock %}</h1>
+
{% block list_form %}
<form id="id_item_form" method="POST" action="{% block [...]
{{ form.text }}
{% csrf_token %}
- {% if form.errors %}
- <div class="form-group has-error">
- <div class="help-block">{{ form.text.errors }}</div>
+ <div class="form-group has-error">
+ <div class="help-block">
+ {% if form.errors %}
+ {{ form.text.errors }}
+ {% endif %}
</div>
- {% endif %}
+ </div>
</form>
{% endblock %}
+
</div>
</div>
</div>By the end you should get to a JS test run a bit like this:
20 assertions of 20 passed, 0 failed. 1. errors should be hidden on keypress (1) 2. errors aren't hidden if there is no keypress (1) 3. should call updateItems on initialize (1) 4. updateItems should get correct url by ajax (3) 5. updateItems should fill in lists table from ajax response (3) 6. should intercept form submit and do ajax post (4) 7. should call updateItems after successful post (1) 8. should not intercept form submit if no api url passed in (1) 9. should display errors on post failure (2) 10. should hide errors on post success (1) 11. should display generic error if no error json (2)
And a full test run should pass, including all the FTs:
$ python manage.py test [...] Ran 81 tests in 62.029s OK
Laaaaaahvely.[1]
And there’s your hand-rolled REST API with Django. If you need a hint finishing it off yourself, check out the repo.
But I would never suggest building a REST API in Django without at least checking out 'Django-Rest-Framework'. Which is the topic of the next appendix! Read on, Macduff.
- Dedupe URLs
-
URLs are more important, in a way, to an API than they are to a browser-facing app. Try to reduce the amount of times you hardcode them in your tests.
- Don’t work with raw JSON strings
-
json.loadsandjson.dumpsare your friend. - Always use an Ajax mocking library for your JS tests
-
Sinon is fine. Jasmine has its own, as does Angular.
- Bear graceful degradation and progressive enhancement in mind
-
Especially if you’re moving from a static site to a more JavaScript-driven one, consider keeping at least the core of your site’s functionality working without JavaScript.