I have the same experience while ago. The problem is that I used my own DNS cache. I thought this was the problem in Go resolver library, I even take time to look into Go source code, until I found out that the problem is in my DNS cache [1].
Go resolver library reject DNS UDP packet that is greater than 512 bytes, following the RFC 1035, meanwhile my program use dnscrypt as parent DNS server and forward the response to client without checking whether the UDP response packet size is greater than 512 bytes or not (at somepoint some dnscrypt server return UDP packet greater than 512 bytes).
So, if you have custom DNS server on your server, test whether it return valid packet size for UDP or not using tools like dig or drill. That is my thought on your problem, just want to share it.