Slow http requests

Getting Help
1

Part of the application I am currently working on is a web scraper that hits a bunch of api endpoints that I’m paying for access for.

All well and good, but for some reason the requests I send (using the http package) sometimes occur very slowly, around 3 minutes per request at the very bad end, but often as long as 3 seconds (fast requests at the fast end usually take about 50ms).

I’m just wondering if any of you guys have ever had anything like this happen before, or know of what might be causing this?

Additional things:

  • All requests are run sequentially. Never more than 1 request going at a time.
  • Memory usage is always stable at around 7mbs.
  • pprof shows nothing untoward

Cheers in advance.

2

Do you observe the same behaviour using a diffent HTTP client like curl?

1 Like
3

I would suspect to overloaded endpoints or networking issue, but not to the http client. If you need to speedup the client at some point, you may consider using fasthttp (no http2 support though).

Also check https://blog.golang.org/http-tracing

4

Hi. Do you know if the endpoints have som protection against ddos attacks or similar? We are using a webservice here which we needed to slow down access to. If we tried to get answers as quick as possible would it become slower and slower but if we waited a small amount of time would it be ok. You could do several things:

  1. Contact the api owners and ask them how often you can send a request and if the have some limits
  2. Measure the requests and see if there is some pattern. If it gets slower over time or some questions are slower than others
  3. Try put in a pause between each request.
1 Like
5

Thanks for the input guys.

I took @lutzhorn’s suggestion and tried a simpler version of my code, which was faster, so presumably there’s just something in my code that’s causing a lot of memory usage or something and slowing down the http requests.

In the end I managed to speed up my complex code by just running 10 versions of it at once in different go-routines. The requests still took far longer to return than seems reasonable, but sending 10 times as many made up for it.

1 Like
6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.