Often, people want to build AI automation that requires going to a web site and searching in it for content. But there’s actually a much faster and better way to do that. Google has already crawled and indexed the public web. So why repeat that work. You can just call a web search to get the matching links to look at.
In a thunk, you can do a general search, a site restricted search, news search, image search, maps search, get place details. And starting tomorrow, shopping search to find products and prices.
Use AI to form the right queries to these search tools, and then to consume the results in a meaningful way. You don’t need AI to actually browse the sites and find things of interest.
This is great @praveen-Thunk.AI … I am thinking this could be used for help with pricing models. Perhaps an agent can monitor (on a schedule) top 20 skus/products for a business across vendors and see the fluctuations, upper/lower bounds, plus sentiment/reviews… I’m going to try this and report back if successful.
1 Like
Yes, it should work. There are others successfully using Google maps and places search at scale. This works wildly better than trying to scrape pages directly. The main problem with that is that many sites block site scrapers. Yet they have intentionally allowed Google’s web crawler.
And Google has pre-organized the content in a searchable, structured, and consumable way. So we should this information for AI decisions whenever it is relevant.