Part of the application I am currently working on is a web scraper that hits a bunch of api endpoints that I’m paying for access for.
All well and good, but for some reason the requests I send (using the
http package) sometimes occur very slowly, around 3 minutes per request at the very bad end, but often as long as 3 seconds (fast requests at the fast end usually take about 50ms).
I’m just wondering if any of you guys have ever had anything like this happen before, or know of what might be causing this?
- All requests are run sequentially. Never more than 1 request going at a time.
- Memory usage is always stable at around 7mbs.
- pprof shows nothing untoward
Cheers in advance.