In this piece of article I’m going to go through how to write extensive integration tests for our Go applications. The purpose of this exercise is to be able to write integration tests without mocking too much dependencies but also avoid connecting to other services, ie. remote servers, remote databases, etc. Everything should exist locally on the host running the test, and the test should be able to setup testing environment without external tools.
Yet another post about integration tests?
Most of the articles discussing integration tests are either
Missing key elements, like setting up a database for testing, or http requests are going to a server running in a production or dev environment.
In case of the former, the article is assuming that we have a database setup somewhere but this was not discussed in the article or addressed in the test code either.
The problem with the latter is that integration tests should usually run segregated from any environment and should not depend on the state of a server or the network connection of the host. If we have unstable network we cannot run our tests and couldn’t finish our work?! Or even worse, someone else also running their tests against the same database in the dev environment could conflict with our testing eventually failing the tests.
Using external components to orchestrate the integration test environment. A typical example is using docker-compose to spin-up the containers where our application should connect to. This is usually done in a
Makefileso the test can be executed with one command. I don’t like this approach because the test should be self-contained, and shouldn’t depend on if I ran a command before starting the tests or not. I just wantgo test ./...and let the test do its job.
So how will all this be different? First, we will start the containers for our dependencies from the test code
using testcontainers. Our code will configure the
database and populate it with the necessary test data. Second, outgoing http requests will be handled by our
test server using the httptest package
Defining the problem scope
Let’s assume we have a REST API where a client can save their favorite book by it’s ISBN. Our API will
query said book from another API called books to verify its validity and then store it in the database
with a timestamp when it was added. Below the sequence diagram for that scenario.
For the sake of simplicity of this example, our API will return a HTTP 500 Internal Server Error if the
Books API returns an error or the insertion into the database fails for any reason. Our API will
return a HTTP 400 Bad Request if the the Books API returns non-200 status.
Implementing the the REST API
First, we need an http client that will call the Books API to validate an ISBN. You can see that the baseUrl
of the server is saved in a field of the BaseClient struct, so we can configure it in our testing as well.
func (bc *BookClient) Validate(isbn string) (bool, error) {
res, err := http.
Get(fmt.Sprintf("%s/%s?q=%s", bc.BaseUrl, BookValidationPath, isbn))
// handle error
return res.StatusCode == http.StatusOK, nil
}Next we need a database connection and a method to insert entries into the database.
func InitDatabase(connUrl string) (*Database, error) {
db, err := sql.Open("pgx", connUrl)
// handle error and check connection
return &Database{
db: db,
}, nil
}
const insertCmd string = `insert into "favorites"("isbn","created_at") values($1, $2)`
func (d *Database) SaveFavorite(favorite *FavoriteEntry) error {
_, err := d.db.Exec(insertCmd, favorite.ISBN, favorite.Created)
if err != nil {
return errors.New("failed to insert into db")
}
return nil
}Finally we need to implement our endpoint to handle the favorite requests. We need to validate the ISBN and then store it in the database
func (f *Favorite) AddFavorite(w http.ResponseWriter, r *http.Request) {
// handle request, deserialize request payload
valid, err := f.Bc.Validate(req.ISBN)
// handle result from `Books API`
err = f.Db.SaveFavorite(&FavoriteEntry{ISBN: req.ISBN, Created: time.Now()})
if err != nil {
w.WriteHeader(http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusCreated)
}Adding the integration test
The database
As I mentioned in the introduction we use testcontainers to spin up the test database to use in our integration test. With testcontainers we can use any docker image and completely customize how the container is started, what envrionment variables are set, and when we consider the container started.
cfg := &config{
name: "favorite",
username: "user",
password: "password",
}
port := "5432/tcp"
req := testcontainers.ContainerRequest{
Image: "postgres:alpine",
ExposedPorts: []string{port},
Env: map[string]string{
"POSTGRES_DB": cfg.name,
"POSTGRES_PASSWORD": cfg.password,
"POSTGRES_USER": cfg.username,
},
WaitingFor: wait.ForSQL(nat.Port(port), "pgx", func(port nat.Port) string {
return fmt.Sprintf("postgres://%s:%s@%s:%s/%s?sslmode=disable",
cfg.username,
cfg.password,
"localhost",
port.Port(),
cfg.name)
}).Timeout(time.Second * 60),
}
db, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{ContainerRequest: req})
require.NoError(t, err, "creating container")We inject the db name, the password, and the username. When the container starts, the postgres server read these variables and configures the database automatically. These values are coming from our config object so we can later use them to configure our database connection.
After creating the container we can start it and add the host and mapped port to our config object.
cfg.host, err = db.Host(ctx)
require.NoError(t, err, "getting host")
require.NoError(t, db.Start(ctx), "starting container")
p, err := db.MappedPort(ctx, nat.Port(port))
require.NoError(t, err, "getting port")
cfg.port = p.Port()Books API
Let’s setup the fake Books API now. The httptest.Server allows us to use a custom handler to handle the requests.
We will mimic a real Books API in this handler to test our application.
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodGet && r.RequestURI == fmt.Sprintf("/%s?q=%s", BookValidationPath, isbn) {
w.WriteHeader(http.StatusOK)
return
}
w.WriteHeader(http.StatusNotFound)
}))For this test we simply return HTTP 200 OK if the server was called with the expected method and requestURI, and return
HTTP 404 Not Found for any other case. Of course the handler could be more complex, or even providing a different one
for each of your test cases.
Putting it together
After setting everything up and creating the handler for our favorites endpoint the rest is very similar to a unit test.
connUrl := fmt.Sprintf("host=%s port=%s user=%s password=%s dbname=%s sslmode=disable",
cfg.host,
cfg.port,
cfg.username,
cfg.password,
cfg.name)
db, err := InitDatabase(connUrl)
require.NoError(t, err, "init database")
f := Favorite{
Bc: &BookClient{
BaseUrl: ts.URL,
},
Db: db,
}
favoriteRequest := FavoriteRequest{
ISBN: isbn,
}
body, err := json.Marshal(favoriteRequest)
require.NoError(t, err, "marshal favoriteRequest")
req := httptest.NewRequest(http.MethodPost, "/favorite", bytes.NewReader(body))
rec := httptest.NewRecorder()
f.AddFavorite(rec, req)
assert.Equal(t, http.StatusCreated, rec.Result().StatusCode)We initialize our db connection with the config based on the postgres container. Then, we set the base url
of the BookClient to the url of our test server. After everything is ready we create a new request and a
new recorder and call the handler method to test our endpoint. You can check the complete code here.
Wrapping it up
In this article we created integration test for a simple REST API using testcontainers
and a test server provided by the httptest package. We saw how to create and
configure a postgres database running in a container, and how to use that for
our tests. We also created a fake http server to mimic the behavior of an upstream service. We used no mocking in our tests,
every component was real and made network requests toward its dependencies.