By default, the AI agents in your thunk have access to Google Search via the web_search “tool”. And they can restrict the web search to a specific site following the same syntax as is used in the Google Search page.
Let’s say you provide these instructions to your AI: “Search the web for Geodesic Domes and restrict to wikipedia.com”. When the agent runs, it will use the websearch tool with query:“Geodesic Domes” and site:“wikipedia.com”. The results that come back to the AI agent will be identical to what you’d see in a Google search page if you typed “Geodesic Domes site:wikipedia.com” into the search bar.
Most sites allow the Google browser to crawl and index their content, and that’s what lets Google search implement this feature called “site-specific search”. But a few (old-fashioned) sites don’t want to do that. They don’t want to let Google index their data. They want users to come to their site and type a query into their own custom search box. And then they produce a search result. If you want to use a site like this, you’d have to try it out and see what the search URL looks like. For exmple, it might be something like this …
https://www.troublesome.com/api/search?q=Geodesic+Domes
So what you might do is provide this information in the instructions. Instead of telling it to do a web search, tell your AI to visit https://www.troublesome.com/api/search?q= to find the desired information. This should work instead of the standard web search
@jrozen fyi