Use Bing Web Search with Power Automate to crawl Websites

[ursruee]

The Power Platform supports connectors (See list of connectors) out-of-the-box. The current provided Bing Search and Bing Maps connectors provides capability like «List news by query» or «Get location by address».

What if you want to go beyond the out-of-the box capability and accelerate additional functionality provided by the Azure Cognitive Web Search API - v7 like search for Webpages, Entity, Images etc.?

Microsoft Power Automate provides the flexibility to connect to any API utilizing the HTTP action within Power Automate and can even be triggered by ‘listening’ to a third party webhook that would conduct a HTTP Post request to your Flow.

With Azure Cognitive Services Web search, find what you’re looking for from the world-wide-web

• Bing Entity Search recognize and classify named entities and find search results based on them.
• Bing Image Search, add a variety of search image options to your app.
• Bing Video Search add advanced video search features to your app.
• Bing Visual Search enable users to search using images.
• Bing Web Search Enable safe, ad-free, location-aware search for your users, surfacing relevant information from web results, images, news, videos, and visuals.

and include this into your business application for example enrich your data with data from various sources.

The Azure Cognitive Web Search API - v7 provides a similar (but not exact) experience to Bing.com/Search by returning search results that Bing determines are relevant to the specified query.

Prerequisite to use it is an Azure Subscription and an Azure Cognitive Web Search key to be used to connect the services.

Business scenario

In a B2B2C (Brand-Channel Partner-Customer) scenario we want to gain insights about the share of brand of our actual and potential partners to evaluate their end customer communication. To evaluate our brand against our competitors we’re establishing a share of brand metric. See below sample market insights for domains and brands.

In this article, learn how to connect to Azure Cognitive Web Search API - v7 to search on specific websites about your brand occurrence and gathering the total matched results by following the step by step guide below.

Part 1 – Test and verify your Web Search query

Before you start creating the Power Automate flow, test and verify your web search query. Azure Cognitive Web Search provides an interactive webpage to test and verify it.

1. Go to https://azure.microsoft.com/en-us/services/cognitive-services/bing-web-search-api/
2. Scroll down to Learn more and Resources
3. Choose «API Reference»
4. Choose «Try it»
5. Add your query parameters and the Azure Cognitive Web Search key. See resource info below regarding search parameters and advanced search keyword.
6. Choose «Send» to execute the search query
7. Check the response status (Status = 200 means successfully) and response content. Keep the web page as you’ll use the response content JSON later when you parse the HTTP call response within our Power Automate flow.

Resources for query parameters are

• Web Search API v7 reference https://docs.microsoft.com/en-us/rest/api/cognitiveservices-bingsearch/bing-web-api-v7-reference
• Advanced search keywords https://help.bing.microsoft.com/#apex/18/en-US/10001/-1

Part 2 – Simple scenario to use Bing Web Search API with Power Automate

As we have tested and verified our web search query, we can start to automate the search using Power Automate. All Flows begin with a trigger. Flow provides hundreds of trigger options, including a HTTP webhook that is triggered by real time information. To keep our Flow simple, we use a «Recurrence» trigger.

1. Open Power Automate and create new flow and choose «Scheduled – from blank»
   
2. Triggering the Flow choosing the interval and frequency
3. Add new step and choose «HTTP» to make a HTTP call.
   

1. Choose method «GET»
2. Add Azure Cognitive Web Search API URI: https://api.cognitive.microsoft.com/bing/v7.0/search
3. Add in the Headers the «Ocp-Apim-Subscription-Key» and your Azure Cognitive Web search key
4. Add your tested and verified query parameters from part 1 to the «Queries» part
5. Add new step and choose «Parse JSON» to parse the HTTP response content. The HTTP action will send your request to the API endpoint and then return with a response in JSON format.
   
6. Within Content choose «Body» from the dynamics content
7. Once we have selected the content, we would need to provide a schema which will act as a key to translate the response into usable variables in the form of dynamic content. As you’ve tested and verified your web search query before using the Azure Cognitive «Try it» webpage, you can copy the response content part within the begin { and end bracket }
   
8. And go back you your Power Automate flow and choose in the «Parse JSON» step «Generate from sample».
9. Paste the response content to the Pop-up window. Flow will do the heavy lifting and do its best at converting the sample response into a schema.
10. At this point we can use the data that is parsed through to automate any business process we want. For this example, we are going to keep things simple and send the web search result as an email. Add new step and choose «Send an email».
    
11. Providing the «To» address, «Subject» and «Body» of the email, including our comments – again using the dynamic content selector, this time to select «totalEstimatedMatches» from the Parse JSON step.
12. Save and test the Flow to make sure that everything works.
13. And then check the results in Outlook!
    

Once we are happy with the Flow, make sure to give the Flow, triggers and actions meaningful names and leave comments where necessary!

Wrapping up

This is a very simple example of how to connect to a third-party API, make sense of the response and use the response in a subsequent action. With Microsoft Power Automate, we can build upon this to meet much more complex business needs with useful automation.