How to determine which goroutine is blocking execution?

[oshokin (Oleg Shokin)]

Hi! I’m new to Golang.
I have a small parser that writes found data to Postgres, as database framework I use https://github.com/jackc/pgx.
I write parsed data to an unbuffered channel from various goroutines.
I have special goroutine where I read data from this channel and write it to the database.
I’m debugging an application and it hangs forever sometime after (perhaps waiting for a free connection to a database in the pool).
How to determine which goroutine is blocking execution?
I’ve heard that there is a pprof, but I never used it.
Thanks.

Minimal example:
I’ve struct like this:

ParsingResults struct {
    parser  DataParser
    data []*common.Data
    err     error
}

in separate goroutine I initialize unbuffered channel like this:

results = make(chan *ParsingResults)

then I start various goroutines, where I run parsers:

go fetcher.Parse(results)

each parser gathers data and passes it to the channel like this:

var (
    results chan<- *ParsingResults
    pageResults *ParsingResults
)
results <- pageResults
if pageResults.err != nil {
    return
}

time.Sleep(p.provider.DelayBetweenPages)

and in a separate goroutine such a function is launched:

func (fetcher *Fetcher) waitForResults(ctx context.Context) {
    for {
        select {
        case results := <-fetcher.resultsChannel:
            provider := results.parser.GetProvider()
            if results.err != nil {
                common.Logger.Errorw("failed to fetch data from provider",
                    "provider", provider.Url,
                    "error", results.err)
                continue
            }
            data := fetcher.removeDuplicates(results.data)
            common.Logger.Infow("fetched some data",
                "provider", provider.Url,
                "rows_count", len(results.data),
                "unique_rows_count", len(data))
            _, err := fetcher.Repo.SaveFetchedData(ctx, data)
            if err != nil {
                common.Logger.Errorw("failed to save fetched data",
                    "provider", provider.Url,
                    "error", err)
                continue
            }
            common.Logger.Infow("fetched data were saved successfully",
                "provider", provider.Url,
                "rows_count", len(results.data),
                "unique_rows_count", len(data))
        case <-ctx.Done():
            return
        default:
            common.Logger.Infow("for debugging's sake! waiting for some data to arrive!")
        }
    }
}

the data is stored in the database in this function:

func (repo *Repository) SaveFetchedData(ctx context.Context, rows []*common.Data) (int64, error) {
    if len(rows) == 0 {
        return 0, nil
    }

    baseQB := sq.Insert(db.DataTableName).
        Columns(saveFetchedDataCols...).
        PlaceholderFormat(sq.Dollar)

    batch := &pgx.Batch{}
    for _, p := range rows {
        curQB := baseQB.Values(p.Row1, p.Row2, sq.Expr("NOW()"))
        curQuery, curArgs, err := curQB.ToSql()

        if err != nil {
            return 0, fmt.Errorf("failed to generate SQL query: %w", err)
        }
        batch.Queue(curQuery, curArgs...)
    }

    br := repo.pool.SendBatch(ctx, batch)
    ct, err := br.Exec()
    if err != nil {
        return 0, fmt.Errorf("failed to run SQL query batch: %w", err)
    }

    return ct.RowsAffected(), nil
}

[christophberger]

Hi @oshokin,

Indeed, pprof can be your friend.

Here is how to trivially add a pprof server to your code.

1. Start an HTTP server (unless your code already has one):

func main() {
    go func() {
        log.Println(http.ListenAndServe("localhost:8080", nil))
    }()

The goroutine is necessary because ListenAndServe blocks for the runtime of the HTTP server.

Your editor should automatically add the import; otherwise, add the import of “net/http” manually.

2. Add this import to your list of imports:

_ "net/http/pprof" 

Note the underscore (the “blank identifier” symbol). The package is imported purely for its side effects, hence you need to use the blank identifier here; otherwise Go complains about unused imports.

3. Start your app, and open

http://localhost:7070/debug/pprof

in a browser. Then, when you are sure your app has reached the deadlock, click “full goroutine dump”.

In the dump, search for a pair of goroutines where either both have the status [semacquire], or one is in [chan read] and the other in [chan write] status. (And I guess there are other combinations of blocking states as well.)

Consider that the stack dump is only a snapshot, hence you might want to refresh the page a few times. If you wait for more than a minute, the blocked states show the number of minutes with no change.

Inspect the call stack to find out what funcs are blocking each other there.

And that should be all that’s required for finding the blocking goroutines. Any errors are mine as I typed this within a few minutes at the breakfast table.

Tip: instead of using the browser, you can also run

curl http://localhost:7070/debug/pprof/goroutine?debug=2

in a terminal to get the goroutine stack dump directly.

[oshokin (Oleg Shokin)]

Thank you very much Chris, I will try your guide.