Runtime error in go routine

Getting Help
1

Hi,
I am getting an error while executing my go application. Go application calls a plugin method and crashed with following stack trash. Can someone help to fix this error.

runtime: unexpected return pc for testmethod called from 0x7fdda7006de0
stack: frame={sp:0xc000d09850, fp:0xc000d09858} stack=[0xc000d04000,0xc000d0c000)
000000c000d09750: 00007fddb3e17d2f <runtime.cgocallbackg+239> 00007fddb3e17ba5 <runtime.cgocall+101>
000000c000d09760: 000000c000d097e8 000000c000d097a8
000000c000d09770: 00007fddb3e67e56 <runtime.exitsyscallfast+214> 000000c000177000
000000c000d09780: 000000c000d097b8 00007fddb3e674de <runtime.reentersyscall+126>
000000c000d09790: 000000c0012ead80 0000000300000002
000000c000d097a0: 000000c0012ead80 000000c000d097d8
000000c000d097b0: 00007fddb3e67cc9 <runtime.exitsyscall+393> 000000c000d097d8
000000c000d097c0: 00007fddb3e676e6 <runtime.entersyscall+38> 00007fddb3e17ba5 <runtime.cgocall+101>
000000c000d097d0: 000000c000d097e8 00007fddb3e9c7ff <runtime.asmcgocall+63>
000000c000d097e0: 00007fddb3e17bc9 <runtime.cgocall+137> 00007fddb43c0070
000000c000d097f0: 000000c000d09850 000000c000000000
000000c000d09800: 0000000000d09830 <github.com/microcosm-cc/bluemonday.UGCPolicy+2448> 0000000001ee2620
000000c000d098d0: 000000c001801f88 00007fddac071698
000000c000d098e0: 000000c001801f00 0000000000000000
000000c000d098f0: 0000000000000000 0000000000000001
000000c000d09900: 0000000000000000 0000000000000000
000000c000d09910: 00007fddac350c28 000000c001801f00
000000c000d09920: 0000000000000005 0000000000000000
000000c000d09930: 0000000000000000 00007fddac350c28
000000c000d09940: 0000000000000001 0000000000000000
000000c000d09950: 0000000000000000
fatal error: unknown caller pc

runtime stack:
runtime.throw(0x12ebd52, 0x11)
/usr/local/go/src/runtime/panic.go:1116 +0x74
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc0012ead80, 0x0, 0x0, 0x7fffffff, 0x7fdd5e7fbbb8, 0x0, 0x0, …)
/usr/local/go/src/runtime/traceback.go:273 +0x1aac
runtime.scanstack(0xc0012ead80, 0xc000547e98)
/usr/local/go/src/runtime/mgcmark.go:844 +0x165
runtime.markroot.func1()
/usr/local/go/src/runtime/mgcmark.go:245 +0xc6
runtime.markroot(0xc000547e98, 0xea)
/usr/local/go/src/runtime/mgcmark.go:218 +0x345
runtime.gcDrain(0xc000547e98, 0x3)
/usr/local/go/src/runtime/mgcmark.go:1108 +0x146
runtime.gcBgMarkWorker.func2()
/usr/local/go/src/runtime/mgc.go:1958 +0x87
runtime.systemstack(0x1e7e050)
/usr/local/go/src/runtime/asm_amd64.s:370 +0x63
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1116

goroutine 25 [GC worker (idle)]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc000186f60 sp=0xc000186f58 pc=0x8d1be0
runtime.gcBgMarkWorker(0xc000546800)
/usr/local/go/src/runtime/mgc.go:1945 +0x1ea fp=0xc000186fd8 sp=0xc000186f60 pc=0x87f9ca
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000186fe0 sp=0xc000186fd8 pc=0x8d39c1
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1839 +0x85

goroutine 1 [IO wait]:
internal/poll.runtime_pollWait(0x7fddac3c5798, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:220 +0x65
internal/poll.(*pollDesc).wait(0xc00081e798, 0x72, 0x0, 0x0, 0x12dc0f0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(…)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc00081e780, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:394 +0x1fc
net.(*netFD).accept(0xc00081e780, 0x7667f3ba6dfc7c5a, 0x0, 0x0)
/usr/local/go/src/net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc00086f340, 0x60e5b630, 0xc0002f7c58, 0x8faa28)
/usr/local/go/src/net/tcpsock_posix.go:139 +0x34
net.(*TCPListener).Accept(0xc00086f340, 0xc0002f7ca8, 0x18, 0xc000000180, 0xb7d954)
/usr/local/go/src/net/tcpsock.go:261 +0x66
net/http.(*Server).Serve(0xc0002e8000, 0x192dc00, 0xc00086f340, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2937 +0x286
net/http.(*Server).ListenAndServe(0xc0002e8000, 0xc0002e8000, 0xc00086f320)
/usr/local/go/src/net/http/server.go:2866 +0xb9
net/http.ListenAndServe(…)
/usr/local/go/src/net/http/server.go:3120
main.main()

goroutine 22 [select, 50 minutes]:
database/sql.(*DB).connectionOpener(0xc0001bfad0, 0x1932fc0, 0xc0005a7b80)
/usr/local/go/src/database/sql/sql.go:1126 +0xe8
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:740 +0x13f

goroutine 11340 [IO wait]:
internal/poll.runtime_pollWait(0x7fddac3c4d18, 0x72, 0x191f800)
/usr/local/go/src/runtime/netpoll.go:220 +0x65
internal/poll.(*pollDesc).wait(0xc0007cc418, 0x72, 0xc000637d00, 0x1, 0x1)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(…)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0007cc400, 0xc000637db1, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1b1
net.(*netFD).Read(0xc0007cc400, 0xc000637db1, 0x1, 0x1, 0xbf5152, 0xc0001b4ae0, 0xc00070af80)
/usr/local/go/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc000d14018, 0xc000637db1, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:182 +0x90
net/http.(*connReader).backgroundRead(0xc000637da0)
/usr/local/go/src/net/http/server.go:690 +0x5a
created by net/http.(*connReader).startBackgroundRead
/usr/local/go/src/net/http/server.go:686 +0xd5

goroutine 10891 [select]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc000c16360, 0xc0007e45a0, 0xc0002a84e0)
/root/go/pkg/mod/github.com/go-sql-driver/mysql@v1.5.0/connection.go:621 +0xbf
created by github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher
/root/go/pkg/mod/github.com/go-sql-driver/mysql@v1.5.0/connection.go:618 +0xc8

goroutine 11381 [IO wait]:
internal/poll.runtime_pollWait(0x7fddac3c5178, 0x72, 0x191f800)
/usr/local/go/src/runtime/netpoll.go:220 +0x65
internal/poll.(*pollDesc).wait(0xc0007cc998, 0x72, 0xc00092e000, 0x1000, 0x1000)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(…)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0007cc980, 0xc00092e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1b1
net.(*netFD).Read(0xc0007cc980, 0xc00092e000, 0x1000, 0x1000, 0xc000e02f00, 0x300000002, 0xc000e02f00)
/usr/local/go/src/net/fd_posix.go:55 +0x51
net.(*conn).Read(0xc000d14048, 0xc00092e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:182 +0x90
net/http.(*connReader).Read(0xc000897d10, 0xc00092e000, 0x1000, 0x1000, 0x8, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:798 +0x1c2
bufio.(*Reader).fill(0xc000903ec0)
/usr/local/go/src/bufio/bufio.go:101 +0x105
bufio.(*Reader).ReadSlice(0xc000903ec0, 0xa, 0x1481128, 0xc001423988, 0x86e990, 0xc000d48a00, 0x100)
/usr/local/go/src/bufio/bufio.go:360 +0x3f
bufio.(*Reader).ReadLine(0xc000903ec0, 0xc001423990, 0xc0006d0000, 0x7fddacfd9c28, 0x0, 0x86f37a, 0x30)
/usr/local/go/src/bufio/bufio.go:389 +0x36
net/textproto.(*Reader).readLineSlice(0xc000c15ad0, 0xc000d48a00, 0x94404f, 0xc0007cc980, 0x0, 0xc0007cc980)
/usr/local/go/src/net/textproto/reader.go:58 +0x6e
net/textproto.(*Reader).ReadLine(…)
/usr/local/go/src/net/textproto/reader.go:39
net/http.readRequest(0xc000903ec0, 0x0, 0xc000d48a00, 0x0, 0x0)
/usr/local/go/src/net/http/request.go:1012 +0xac
net/http.(*conn).readRequest(0xc0003b94a0, 0x1932fc0, 0xc000082100, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:984 +0x19a
net/http.(*conn).serve(0xc0003b94a0, 0x1932fc0, 0xc000082100)
/usr/local/go/src/net/http/server.go:1851 +0x70d
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2969 +0x394

2

Can you provide steps to reproduce it?

1 Like
3

The crash does not come every time. It’s very rare. There is no step to reproduce the crash. I am calling a plugin method. The crash occurs randomly. Below line comes every time

/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000186fe0 sp=0xc000186fd8 pc=0x8d39c1
created by runtime.gcBgMarkStartWorkers

4

Hi @skillian,
I understand it’s difficult to find the cause without reproducing it. I’m not expert in golang so want to ask some common questions. I think i’m not following the basic rules of golang.

I created a plugin and write some go functions. Also used a c library in plugin and calling cgo functions from go functions and write some cgo functions in plugin.
Can I call multiple cgo functions at the same time from go functions (multithreading)?
Similarly can I call go functions from cgo functions at the same time?

In most cases i’m getting error in runtime.cgocall and proc.go:1116 (mstart()) and runtime.systemstack_switch()

5

Hi @skillian
could you please reply for the query which I asked

6

Sorry, yes, I’ll get back to you.

7

Yes.

That is interesting; is your code weaving between C and Go throughout the call stack? This might be a problem because Go stacks grow and can be moved. I’m not sure what would happen if there was some of the C stack in there. I don’t even know if the C stack is the same stack as the Go stack (I have almost zero experience with cgo).

8

Thanks

9

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.