This is a collection of subtle bugs I’ve encountered in Go that took time to understand. Each entry documents the issue, a reproducible example, and the root cause

1. HTTP call to 3rd party API responds with error from our local database

Diagnosis: I see a weird error in the logs, it looks like this:

error happened GET https://example.com, <service database error>

where you can see endpoint URL and denoted with angle brackets an error from our own database. What the heck? How come we get an error from our own database when we call an endpoint?!

Below you will find a completely reproducible example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

package main

import (
	"context"
	"fmt"
	"net/http"
	"net/http/httptest"
	"time"

	"golang.org/x/sync/errgroup"
)

func main() {
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		time.Sleep(2 * time.Second)
		w.WriteHeader(http.StatusOK)
	}))
	defer server.Close()

	g, ctx := errgroup.WithContext(context.Background())

	g.Go(func() error {
		time.Sleep(500 * time.Millisecond)
		return fmt.Errorf("mongodb_error_poison")
	})

	g.Go(func() error {
		req, _ := http.NewRequestWithContext(ctx, http.MethodGet, server.URL, nil)
		_, err := http.DefaultClient.Do(req)
		if err != nil {
			fmt.Printf("client.Do error: %v\n", err)
		}
		return err
	})

	g.Wait()
}

The output will be:

client.Do error: Get "http://127.0.0.1:41339": mongodb_error_poison

How come? What is going on? Any guesses?

It turns out the following is happening: errgroup.Wait() will cancel the context as soon as any of goroutines return an error. Internally, errgroup will use a special context.WithCancelCause() context which returns the context itself and a cancel function that accepts an error as a cause.

Now, as soon as the context is cancelled, http.DefaultClient.Do(req) with a request that has our context immediately returns an error. But the method Do() has this comment: // Any returned error will be of type [*url.Error]. and the url.Error struct looks like this:

// Error reports an error and the operation and URL that caused it.
type Error struct {
        Op  string
        URL string
        Err error
}

Long story short. The “cause” from the context, which is an error, will be assigned to the Err field in url.Error. This is how the database error bubbled up from the context and was returned from the HTTP call.