One of the first projects I like to do when I start learning a new language or a new technology is to find that language’s HTTP framework and try to write a simple CRUD (Create, Read, Update, Delete) JSON API. In my view, that’s a great starting point, because integrating a database and a HTTP layer is quite simple, but it is not trivial, especially as you start adding concepts like authentication or validation.
This approach, however, has been frustrated slightly in recent years with the wider and wider adoption of automatically generated code in many frameworks. It seems that many creators now prefer to give the user the ability to get to a working application quickly by using command-line tooling to generate various components of the system. This, though, comes at the expense of understanding of how the pieces fit together. Many guides will not offer the write-from-scratch experience anymore, and will just get the user to the quick-start point by using automated code generation.
I found that it was quite tough to find such resources for the premier Elixir HTTP framework—Phoenix. Hence, I decided to sit down and write a simple todo-list API without using the built-in JSON endpoint generator, in order to get a bird’s eye view of how the pieces fit together. I was guided by the code output by the mix phx.gen.json command, and the resulting code is pretty much the same thing as the code generated by the codegen.
Initial setup
We start with the barebones project bootstrap using mix phx.new, followed by the project name, followed by a bunch of flags that indicate what we don’t want in our app.
mix phx.new todos --no-html --no-assets --no-esbuild --no-gettext --no-mailer --no-tailwind --database sqlite3
Most of the files that Phoenix generates in this initial bootstrap are the kind of thing that in many other frameworks would be tucked away in the library code. In the case of Phoenix, we can change the macros that we use to load HTTP-specific DSLs (domain-specific languages), so we get a few bits of generated code that we are responsible for, but we will most likely not edit them at all.
Then we do a few initial tasks: create the database, initialise a Git repository and commit our initial changes.
cd todos
mix ecto.create
git init
git commit -am 'initial commit'
Database migration and schemas
The first step is to create the database integration: the migration and schema. We use the mix ecto.gen.migration followed by the migration name to create the migration file. This is the last time we use generated code in this guide. The migration codegen is quite trivial, it only creates a timestamped file with a module loading the Ecto migration DSL.
$ mix ecto.gen.migration create_todos
* creating priv/repo/migrations/20241020104740_create_todos.exs
defmodule Todos.Repo.Migrations.CreateTodos do
use Ecto.Migration
def change do
# We will define our migration here.
end
end
The details on how to write Ecto migrations can be found in this guide. Since we’re building a simple todo list, we will add two fields: name and done, as well as the timestamps/1 macro which adds “created at” and “updated at” fields. We add the schema inside change/0.
create table("todos") do
add :title, :string
add :done, :boolean, default: false, null: false
timestamps(type: :utc_datetime)
end
When run, this migration will add two columns to the “todos” table: title with type string and done with type boolean that is non-nullable and set to false by default. We also define the timestamps to be stored as utc_datetime.
We run the migrations and start the server.
mix ecto.migrate
iex -S mix phx.server
The next step is to create the Phoenix context and Ecto schema.
Phoenix uses DDD-inspired context modules to organise code. The contexts expose and group related functionality. In this project, we will create a Todolist context that is going to be the main point of interaction with our schema. If we had other entities associated with Todo, we might also add them there, to have a boundary of consistency for our entities. In our case, we will later add the functions that create, read, update and delete our Todos. For now, the context will be empty.
# /lib/todos/todolist.ex
defmodule Todos.Todolist do
# Context will go here.
end
We also add a Todo schema for our database entity.
# /lib/todos/todolist/todo.ex
defmodule Todos.Todolist.Todo do
use Ecto.Schema
schema "todos" do
# Fields go here.
end
end
Next we add fields to the schema. They will correspond one-to-one with the columns in the database (the syntax here is a little different than in migrations, see this guide for details).
schema "todos" do
field :title, :string
field :done, :boolean, default: false
timestamps(type: :utc_datetime)
end
Using IEx we can now add some Todos to the database and display them.
iex(2)> %Todos.Todolist.Todo{title: "Walk the dog", done: false} |> Todos.Repo.insert!()
%Todos.Todolist.Todo{
__meta__: #Ecto.Schema.Metadata<:loaded, "todos">,
id: 1,
title: "Walk the dog",
done: false,
inserted_at: ~U[2024-10-20 11:32:24Z],
updated_at: ~U[2024-10-20 11:32:24Z]
}
iex(3)> Todos.Todolist.Todo |> Todos.Repo.all()
[
%Todos.Todolist.Todo{
__meta__: #Ecto.Schema.Metadata<:loaded, "todos">,
id: 1,
title: "Walk the dog",
done: false,
inserted_at: ~U[2024-10-20 11:32:24Z],
updated_at: ~U[2024-10-20 11:32:24Z]
}
]
Read
While we aren’t going to allow the user to create Todos this way (by directly using the schema), we can already make a HTTP route that will display the added Todo. There are (generally) two ways Ecto lets us select data from the database:
- Schema queries, which are similar to queries that you might be familiar with from ORMs like Prisma.
- Schemaless queries, which more resemble SQL.
Under the hood schema queries are converted to schemaless queries, which then can be executed by the Repo module. In this case, we will use a schema query to get all the Todos. The Repo.all is a reasonably good entry point for this task. We add a function to our Todolist context:
defmodule Todos.Todolist do
alias Todos.Repo
alias Todos.Todolist.Todo
def list_todos, do: Repo.all(Todo)
end
We can check if it works in IEx:
iex(5)> Todos.Todolist.list_todos()
[
%Todos.Todolist.Todo{
__meta__: #Ecto.Schema.Metadata<:loaded, "todos">,
id: 1,
title: "Walk the dog",
done: false,
inserted_at: ~U[2024-10-20 11:32:24Z],
updated_at: ~U[2024-10-20 11:32:24Z]
}
]
The next step is adding a controller. The controller contains the route handlers that process the HTTP requests and return a response. In this case we attempt to directly return JSON using json/2. We create a route handler for the /api/todos route by creating a controller with an index/2 callback and include it in the /api scope inside the router file.
# /lib/todos_web/controllers/todo_controller.ex
defmodule TodosWeb.TodoController do
use TodosWeb, :controller
alias Todos.Todolist
def index(conn, _params) do
todos = Todolist.list_todos()
json(conn, todos)
end
end
Note that the TodoController here is not an alias.
# /lib/todos_web/router.ex
scope "/api", TodosWeb do
pipe_through :api
get "/todos", TodoController, :index
end
Unfortunately when we try to call this route, we get an error. This fails, because Jason (the library for encoding and parsing JSON used by Phoenix) doesn’t know how to encode structs. We have to add @derive to our schema.
# /lib/todos/todolist/todo.ex
@derive Jason.Encoder
If we try to call the endpoint again, it’s going to fail, this time because we try to send the Ecto __meta__ field to the client which is not allowed. We have to change the derivation to only include specific fields from our entity.
@derive {Jason.Encoder, only: [:id, :title, :done]}
This should now work, however, this is not the way that Phoenix uses in its generated routes.
Instead of the json/2 function in the controller, Phoenix codegen creates a JSON view that gives more fine-grained control over the serialisation of the JSON and the controller uses render/3 to render it. Another advantage of using the views is that Phoenix can decide which template to render based on the Accept header of the incoming request: if we had an API that served both JSON and HTML, we could use the same route to serve different formats. In my experience, this is quite uncommon.
On the other hand, deriving the Jason.Encoder protocol directly inside the schema and calling json/2 in the controller instead of using views colocates the serialisation logic in the schema file. I suppose that if the serialisation logic is quite involved, it might be better to use views. For simpler handlers, the simpler solution might be enough.
In any case, this time we will try to follow the codegen way of returning JSON data. To do this, we remove the @derive and create the view. The codegen creates a function data which extracts the fields that are supposed to be included in the response, we will follow this convention. The view module for SomethingController should be called SomethingJSON (pay attention to the casing).
# /lib/todos_web/controllers/todo_json.ex
defmodule TodosWeb.TodoJSON do
alias Todos.Todolist.Todo
def index(%{todos: todos}) do
todos |> Enum.map(&data/1)
end
defp data(%Todo{} = todo) do
%{
id: todo.id,
title: todo.title,
done: todo.done,
}
end
end
Note that this is slightly different than the default template created by the codegen. By default, valid JSON responses are wrapped in a { data: ... } object and error responses are wrapped in { error: ... }. I think the reason for this behaviour is that the view can later also add additional metadata to the response, like pagination information. In this case, we will not worry about this and just return the data “neat.”
We also replace the call to json/2 with a call to render/3.
# /lib/todos_web/controllers/todo_controller.ex
def index(conn, _params) do
todos = Todolist.list_todos()
render(conn, :index, todos: todos)
end
Now opening /api/todos yields:
[{"id":1,"done":false,"title":"Walk the dog"}]
While we are here, we can also add functions that retrieve a Todo by ID. We define the interface to our schema inside the Todolist context.
# /lib/todos/todolist.ex
def get_todo!(id), do: Repo.get!(Todo, id)
Next, we add the route handler inside the controller.
# /lib/todos_web/controllers/todo_controller.ex
def show(conn, %{"id" => id}) do
todo = Todolist.get_todo!(id)
render(conn, :show, todo: todo)
end
Next, we add another function inside the view module that will serialise a single Todo. Here we also deviate from the codegen way, since we do not wrap our data in an object.
# /lib/todos_web/controllers/todo_json.ex
def show(%{todo: todo}), do: data(todo)
Finally, we attach the route handler to the /todos/:id route.
# /lib/todos_web/router.ex
get "/todos/:id", TodoController, :show
Now visiting /api/todos/1 yields:
{"id":1,"done":false,"title":"Walk the dog"}
By default the codegen will use the banged (throwing) versions of Repo queries. In case we hit a non-existent route, in development we will get the standard Phoenix error screen. In production, the error_json.ex file will be used to determine the message, by default it will return something like this:
{"errors":{"detail":"Not Found"}}
Alternatively we could use the regular Repo.get/2 and handle the nil result.
Create
To create entities in our database via the API, we should first have a mechanism in place for validating the responses. Ecto recognises that selecting from a database is much different than inserting or updating records. Hence, instead of using queries directly (like we do with fetching data), we will instead use changesets.
Changesets are objects that track changes to a given entity. They can be piped into Repo functions (for example Repo.insert/1) to execute database operations like insert or update.
First, we add a changeset/2 function to our Todo schema.
# /lib/todos/todolist/todo.ex
import Ecto.Changeset
# schema "todos" ...
def changeset(todo, attrs) do
todo
|> cast(attrs, [:title, :done])
|> validate_required([:title])
|> validate_length(:title, min: 5)
end
Changeset functions typically take the entity that’s being modified, in this case a %Todo{} struct (it can be empty if we want to add a new entity) and attributes that are being changed. We take the todo entity, cast the attributes from the attrs map to create the changeset, and then validate using some provided validators from the Ecto.Changeset module.
It is worth noting that the validation only runs for changed fields. If the attrs map contains the done field, but not the title field, the title validations will not run.
This is what changesets look like in IEx (note the valid? and errors fields):
iex(6)> Todos.Todolist.Todo.changeset(%Todos.Todolist.Todo{}, %{title: "Walk the dog"})
#Ecto.Changeset<
action: nil,
changes: %{title: "Walk the dog"},
errors: [],
data: #Todos.Todolist.Todo<>,
valid?: true,
...
>
iex(7)> Todos.Todolist.Todo.changeset(%Todos.Todolist.Todo{}, %{title: "Wal"})
#Ecto.Changeset<
action: nil,
changes: %{title: "Wal"},
errors: [
title: {"should be at least %{count} character(s)",
[count: 5, validation: :length, kind: :min, type: :string]}
],
data: #Todos.Todolist.Todo<>,
valid?: false,
...
>
A valid changeset can be piped into a Repo function that will return {:ok, entity} if the operation succeeds, or {:error, changeset} if something goes wrong.
Phoenix uses changesets extensively in HTML and LiveView. For entities that have multiple reasons to change, it is common to have multiple changesets. For example, you might have different changesets for different levels of access to the application, or for different business events. In this case however, a single changeset is enough.
It’s common to add a change_entity/2 function to the context that returns an empty changeset and can be used in situations that require validation and change tracking (for example in LiveView forms). We don’t need this now, so we won’t add it, but the automated codegen would create that function for us.
We add a create_todo/1 function that applies attrs to a changeset and pipes it into Repo.insert/1, creating the entity.
# /lib/todos/todolist.ex
def create_todo(attrs \\ %{}) do
%Todo{}
|> Todo.changeset(attrs)
|> Repo.insert()
end
Now we add a route handler to our controller.
# /lib/todos_web/controllers/todo_controller.ex
def create(conn, %{"todo" => todo_params}) do
{:ok, todo} = Todolist.create_todo(todo_params)
render(conn, :show, todo: todo)
end
Note that the params here are wrapped in an object with key todo. Autogenerated code will expect you to pass the params wrapped in an object with the entity name, like {"todo": {"title: "Walk the dog"}} rather than just {"title": "Walk the dog"}. I couldn’t find much justification for this, and such a practice is uncommon in REST in general, however Phoenix mixes query, route and body params into one params map so expecting the body to be “namespaced” might be reasonable to avoid collisions.
Next, we attach the route handler to a POST route.
# /lib/todos_web/router.ex
scope "/api", TodosWeb do
# pipe_through ...
post "/todos", TodoController, :create
end
Now if we try to create a valid todo using curl:
curl -XPOST -H "Content-type: application/json" -d '{"todo": {"title": "Walk the dog"}}' 'http://localhost:4000/api/todos'
We get a response back and the Todo is created. The problem is, when we try to create an invalid Todo (for instance, missing the title or the title being to short) we receive an unexpected error and the HTTP 500 error code. In the case that validation fails, we would like to return the error.
The source of the error is that this line doesn’t match {:ok, todo} if validation fails:
# /lib/todos_web/controllers/todo_controller.ex
{:ok, todo} = Todolist.create_todo(todo_params)
To fix it, we will attempt to use with. It attempts a pattern match, executing the do block if it succeeds, and returning the failed pattern if it fails.
def create(conn, %{"todo" => todo_params}) do
with {:ok, todo} <- Todolist.create_todo(todo_params) do
render(conn, :show, todo: todo)
end
end
This doesn’t work either, but now we get a different error:
[error] ** (RuntimeError) expected action/2 to return a Plug.Conn, all plugs must receive a connection (conn) and return a connection
We haven’t really talked about Phoenix controllers and route handlers. They are plugs (similar to middleware in other frameworks) that should always return a conn. Conn is basically an object that describes the current request. Coming from JS, they’re kind of like Express’s req and res objects smooshed together. The Plug.Conn module (which we import from todos_web.ex via use TodosWeb, :controller) also has a number of useful functions that manipulate conns by adding statuses, redirecting them, and so on.
We could change the with to a case and try to handle the error in a way that returns a valid conn. However, the error message is lying to us a little. The plugs don’t always have to return conn - they can return something else, as long as we provide a fallback controller which will translate that something else back into a conn.
We create a new file with the fallback controller:
# /lib/todos_web/controllers/fallback_controller.ex
defmodule TodosWeb.FallbackController do
use TodosWeb, :controller
def call(conn, error) do
IO.inspect(error)
conn
end
end
We also need to add the action_fallback/1 macro to the TodoController:
# /lib/todos_web/controllers/todo_controller.ex
defmodule TodosWeb.TodoController do
# use TodosWeb...
action_fallback FallbackController
# def index...
end
Now calling the endpoint with incorrect data reveals, that the failed pattern match is in the form of {:error, changeset}. We can pattern match in the call/2 function of the fallback controller for that kind of error and convert it to a conn that returns an error message.
The automatic codegen would create another view module (changeset_json.ex) for handling changeset errors. This is useful, for example if we wanted to use Gettext during serialisation to get translated error messages, but in this case, instead of making use of a view module, we return the errors directly from the handler using json/2.
# /lib/todos_web/controllers/fallback_controller.ex
defmodule TodosWeb.FallbackController do
use TodosWeb, :controller
def call(conn, {:error, %Ecto.Changeset{} = changeset}) do
conn
|> put_status(:unprocessable_entity)
|> json(Ecto.Changeset.traverse_errors(changeset, &translate_error/1))
end
defp translate_error({msg, opts}) do
Enum.reduce(opts, msg, fn {key, value}, acc ->
String.replace(acc, "%{#{key}}", fn _ -> to_string(value) end)
end)
end
end
Note also the put_status/2 plug here, this is used to change the HTTP status of the response. We could add a specific status code here, but Plug also lets us use atoms that get translated to the status codes. We then use the built-in traverse_errors/2 function that turns the errors into a simple map that can be transformed to JSON.
Making a request to the endpoint with incorrect data now returns something more meaningful:
curl -XPOST -H "Content-type: application/json" -d '{"todo": {"title": "Wa"}}' 'http://localhost:4000/api/todos'
{"title":["should be at least 5 character(s)"]}
We also change the create/2 route handler to add additional information to the successful request:
# /lib/todos_web/controllers/todo_controller.ex
def create(conn, %{"todo" => todo_params}) do
with {:ok, todo} <- Todolist.create_todo(todo_params) do
conn
|> put_status(:created)
|> put_resp_header("location", ~p"/api/todos/#{todo}")
|> render(:show, todo: todo)
end
end
Update and Delete
Now that we have pretty much every bit of plumbing ready and we serve read and create requests, we can add the last two actions to our context:
# /lib/todos/todolist.ex
def update_todo(%Todo{} = todo, attrs) do
todo
|> Todo.changeset(attrs)
|> Repo.update()
end
def delete_todo(%Todo{} = todo), do: Repo.delete(todo)
Next, we add the route handlers to our controller:
# /lib/todos_web/controllers/todo_controller.ex
def update(conn, %{"id" => id, "todo" => todo_params}) do
todo = Todolist.get_todo!(id)
with {:ok, %Todolist.Todo{} = todo} <- Todolist.update_todo(todo, todo_params) do
render(conn, :show, todo: todo)
end
end
def delete(conn, %{"id" => id}) do
todo = Todolist.get_todo!(id)
with {:ok, %Todolist.Todo{}} <- Todolist.delete_todo(todo) do
send_resp(conn, :no_content, "")
end
end
Now we should add the routes. We can add them directly under the old ones. However, Phoenix Router gives us a resources/3 macro that we can use to define standard REST routes in this format:
# /lib/todos_web/router.ex
get "/todos", TodoController, :index
get "/todos/:id", TodoController, :show
post "/todos", TodoController, :create
put "/todos/:id", TodoController, :update
delete "/todos/:id", TodoController, :delete
# is equivalent to:
resources "/todos", TodoController, except: [:new, :edit]
Note the except: [:new, :edit] - these actions would be used for a HTML Phoenix app, where /todos/new and /todos/:id/edit routes would direct the user to HTML views with forms for adding and editing a Todo respectively. The forms would call the corresponding create and update routes. Since we are working with a JSON API, they are not necessary.
Testing
The automatic codegen also generates some tests for us. I’m not going to go through every test case, but there are a few basic patterns that we can use to test our business logic and our route handlers.
Business logic
First we’ll make a fixture that creates a todo with given parameters and inserts it into the database, which we’ll use in tests that expect some data to already be in the DB.
# /test/support/fixtures/todolist_fixtures.ex
defmodule Todos.TodolistFixtures do
@defaults %{done: false, title: "Walk the dog."}
def todo_fixture(attrs \\ %{}) do
{:ok, todo} =
attrs
|> Enum.into(@defaults)
|> Todos.Todolist.create_todo()
todo
end
end
Next, we’ll create the module for testing the business logic (note the _text.exs suffix of the test files that are supposed to be run by ExUnit).
# /test/todos/todolist_test.exs
defmodule Todos.TodolistTest do
use Todos.DataCase
alias Todos.Todolist
describe "todos" do
import Todos.TodolistFixtures
test "list_todos/0 returns all todos" do
todo = todo_fixture()
assert Todolist.list_todos() == [todo]
end
end
end
Now we can run mix test to see if everything is all right.
mix test
Running ExUnit with seed: 653502, max_cases: 32
...
Finished in 0.03 seconds (0.02s async, 0.01s sync)
3 tests, 0 failures
assert/1 is a very useful macro, we can use it to test not only for equality but also for pattern matching. We could, for example, write a test that verifies that we get an error with a changeset if we try to create an invalid Todo. We can do this by adding a pattern match inside assert, like this:
# /test/todos/todolist_test.exs
test "create_todo/1 with invalid data returns error changeset" do
invalid_attrs = %{done: nil, title: nil}
assert {:error, %Ecto.Changeset{}} = Todolist.create_todo(invalid_attrs)
end
Writing the remaining business logic tests is left as an exercise to the reader. As a pro tip: you probably shouldn’t use the todo_fixture/1 in the test for create_todo/1. If you get stuck, you can try to run the codegen and see what tests it spits out.
These tests use Ecto’s SQL Sandbox under the hood (loaded by use Todos.DataCase), so the database is cleared for each test automatically.
Route handlers
To test route handlers we make use of Phoenix.ConnTest which is already included in the TodosWeb.ConnCase module in our app that we can just use. The tests for route handlers are quite similar to the handlers themselves, except “in reverse”: we build the request conn and then dispatch it to the endpoint. At the end we use json_reponse/2 function to retrieve the response from the conn and pattern match it or compare to the expected values.
First we create a module for our route handler tests:
# /test/todos_web/todo_controller_test.exs
defmodule TodosWeb.TodoControllerTest do
use TodosWeb.ConnCase
setup %{conn: conn} do
{:ok, conn: put_req_header(conn, "accept", "application/json")}
end
describe "index" do
test "lists all todos", %{conn: conn} do
conn = get(conn, ~p"/api/todos")
assert json_response(conn, 200) == []
end
end
end
Note that the ~p sigil works in tests: we can use it to get compile-time information about correctness of our routes.
Running mix test shows that it passes:
mix test
Running ExUnit with seed: 899063, max_cases: 32
.....
Finished in 0.05 seconds (0.02s async, 0.03s sync)
5 tests, 0 failures
In case we want to rewrite the test to have some initial setup function that’s going to test against something different than an empty state, we can also include a setup function that’s going to add more data to our test context.
# /test/todos_web/todo_controller_test.exs
describe "index with setup" do
setup [:setup_todo]
test "lists all todos (setup)", %{conn: conn, todo: todo} do
conn = get(conn, ~p"/api/todos")
expected_todo = %{"id" => todo.id, "title" => todo.title, "done" => todo.done}
assert [^expected_todo] = json_response(conn, 200)
end
end
# ...
defp setup_todo(_) do
todo = todo_fixture()
%{todo: todo}
end
A final example of a more complicated integration test:
# /test/todos_web/todo_controller_test.exs
describe "create todo" do
test "responds with a todo when data is valid", %{conn: conn} do
# given
attrs = %{title: "Walk the dog.", done: false}
# when
conn = post(conn, ~p"/api/todos", todo: attrs)
# then
assert %{"id" => id} = json_response(conn, 201)
conn = get(conn, ~p"/api/todos/#{id}")
assert %{"id" => ^id, "done" => false, "title" => "Walk the dog."} = json_response(conn, 200)
end
end
And that’s pretty much it. You can find the code I wrote for this blog post in this GitHub repo.
