I want to use Go for web scraping and web app automation.
If I want my Go program to do a job search on a website and the outcome is 3000 results spread out over 40 pages, would it be too much work for the program to enter the search term and location into the job search page, click on each job post and scrape the page for that job, do this for all jobs on a page, when it reaches the bottom of the page, then click the next page and repeat the same process until it gets to the last page in the result (in this case, page 40)?
Opening every job posting in 3000 job postings on 40 pages would take a person a while to do, I’m not sure how fast a web scraping and web automation program can do this. I think a web scraper would be fast and not use much memory, but for a web driver I don’t know if it will be too slow to do what I described above?