- They are simply inconvenient, as they require manual work for every result page
- They stop working after returning the search results of the first 3-5 search requests, because the service got blocked by the search engine
Why is it so hard to scrape search results?
The reason why it is harder than expected to find a working solution is fairly simple:
Search engines usually do not want you to scrape its search result pages programmatically.
Search Engines has put a lot of systems in place to detect, whether your search is run by a real human or an automated software script. Even when trying to mimic human behaviour by adding a long waiting period between your queued search requests, the search engine is very likely to confront your script or tool with a captcha it can not solve and thus stop the script.
Solutions that actually work
The solution to the previously described problem is the combination of a large proxy network and captcha solvers.
Zenserp.com is offering this infrastructure as a SERP API, with which you can reliably retrieve the TOP 100 search result pages for a specific search phrase.
Besides zenserp.com, here is a list of more providers, offering a similar service:
Click here to create a free Test-Account (50 search requests / month). Zenserp.com enables you to scrape search results hassle-free and without limiting your request volume. Scraping serps has never been easier.