cancel
Showing results for 
Search instead for 
Did you mean: 

Use Bing Web Search with Power Automate to crawl Websites

The Power Platform supports connectors (See list of connectors) out-of-the-box. The current provided Bing Search and Bing Maps connectors provides capability like «List news by query» or «Get location by address».

 

What if you want to go beyond the out-of-the box capability and accelerate additional functionality provided by the Azure Cognitive Web Search API - v7 like search for Webpages, Entity, Images etc.?

 

Microsoft Power Automate provides the flexibility to connect to any API utilizing the HTTP action within Power Automate and can even be triggered by ‘listening’ to a third party webhook that would conduct a HTTP Post request to your Flow.

With Azure Cognitive Services Web search, find what you’re looking for from the world-wide-web

  • Bing Entity Search recognize and classify named entities and find search results based on them.
  • Bing Image Search, add a variety of search image options to your app.
  • Bing Video Search add advanced video search features to your app.
  • Bing Visual Search enable users to search using images.
  • Bing Web Search Enable safe, ad-free, location-aware search for your users, surfacing relevant information from web results, images, news, videos, and visuals.

and include this into your business application for example enrich your data with data from various sources.

The Azure Cognitive Web Search API - v7 provides a similar (but not exact) experience to Bing.com/Search by returning search results that Bing determines are relevant to the specified query.

Prerequisite to use it is an Azure Subscription and an Azure Cognitive Web Search key to be used to connect the services.

 

Business scenario

In a B2B2C (Brand-Channel Partner-Customer) scenario we want to gain insights about the share of brand of our actual and potential partners to evaluate their end customer communication. To evaluate our brand against our competitors we’re establishing a share of brand metric. See below sample market insights for domains and brands.

Bing Web Search API with Power Automate Market Insights.PNG

 

In this article, learn how to connect to Azure Cognitive Web Search API - v7 to search on specific websites about your brand occurrence and gathering the total matched results by following the step by step guide below.

 

Part 1 – Test and verify your Web Search query

Before you start creating the Power Automate flow, test and verify your web search query. Azure Cognitive Web Search provides an interactive webpage to test and verify it.

 
  1. Go to https://azure.microsoft.com/en-us/services/cognitive-services/bing-web-search-api/
  2. Scroll down to Learn more and Resources
  3. Choose «API Reference»
  4. Choose «Try it»
  5. Add your query parameters and the Azure Cognitive Web Search key. See resource info below regarding search parameters and advanced search keyword.
  6. Choose «Send» to execute the search query
  7. Check the response status (Status = 200 means successfully) and response content. Keep the web page as you’ll use the response content JSON later when you parse the HTTP call response within our Power Automate flow.

Resources for query parameters are

Bing Web Search API with Power Automate Part 1 - Bing Web Search - Try it.PNG

 

Part 2 – Simple scenario to use Bing Web Search API with Power Automate

As we have tested and verified our web search query, we can start to automate the search using Power Automate. All Flows begin with a trigger. Flow provides hundreds of trigger options, including a HTTP webhook that is triggered by real time information. To keep our Flow simple, we use a «Recurrence» trigger.

 
    1. Open Power Automate and create new flow and choose «Scheduled – from blank»
      Bing Web Search API with Power Automate Part 2 - Recurrence.PNG

       

    2. Triggering the Flow choosing the interval and frequency
    3. Add new step and choose «HTTP» to make a HTTP call.
      Bing Web Search API with Power Automate Part 2 - HTTP.PNG

 

  1. Choose method «GET»
  2. Add Azure Cognitive Web Search API URI: https://api.cognitive.microsoft.com/bing/v7.0/search
  3. Add in the Headers the «Ocp-Apim-Subscription-Key» and your Azure Cognitive Web search key
  4. Add your tested and verified query parameters from part 1 to the «Queries» part
  5. Add new step and choose «Parse JSON» to parse the HTTP response content. The HTTP action will send your request to the API endpoint and then return with a response in JSON format.
    Bing Web Search API with Power Automate Part 2 - Parse JSON.PNG

     

  6. Within Content choose «Body» from the dynamics content
  7. Once we have selected the content, we would need to provide a schema which will act as a key to translate the response into usable variables in the form of dynamic content. As you’ve tested and verified your web search query before using the Azure Cognitive «Try it» webpage, you can copy the response content part within the begin { and end bracket }
    Bing Web Search API with Power Automate Part 2 - Parse JSON - Schema.PNG

     

  8. And go back you your Power Automate flow and choose in the «Parse JSON» step «Generate from sample».
  9. Paste the response content to the Pop-up window. Flow will do the heavy lifting and do its best at converting the sample response into a schema.
  10. At this point we can use the data that is parsed through to automate any business process we want. For this example, we are going to keep things simple and send the web search result as an email. Add new step and choose «Send an email».
    Bing Web Search API with Power Automate Part 2 - Send an email.PNG

     

  11. Providing the «To» address, «Subject» and «Body» of the email, including our comments – again using the dynamic content selector, this time to select «totalEstimatedMatches» from the Parse JSON step.
  12. Save and test the Flow to make sure that everything works.
  13. And then check the results in Outlook!
    Bing Web Search API with Power Automate Part 2 - Outlook.PNG

     

Once we are happy with the Flow, make sure to give the Flow, triggers and actions meaningful names and leave comments where necessary!
Bing Web Search API with Power Automate Part 2 - Save and Test Flow.PNG

 

Wrapping up

This is a very simple example of how to connect to a third-party API, make sense of the response and use the response in a subsequent action. With Microsoft Power Automate, we can build upon this to meet much more complex business needs with useful automation.

Meet Our Blog Authors
  • Experienced Consultant with a demonstrated history of working in the information technology and services industry. Skilled in Office 365, Azure, SharePoint Online, PowerShell, Nintex, K2, SharePoint Designer workflow automation, PowerApps, Microsoft Flow, PowerShell, Active Directory, Operating Systems, Networking, and JavaScript. Strong consulting professional with a Bachelor of Engineering (B.E.) focused in Information Technology from Mumbai University.
  • Encodian Owner / Founder - Ex Microsoft Consulting Services - Architect / Developer - 20 years in SharePoint - PowerPlatform Fan
  • Cambridge UK Power Platform User Group Leader, Technical evangelist and speaker. Always says yes to coffee! #LetsGetCoffee
  • Passionate #Programmer #SharePoint #SPFx #Office365 #MSFlow | C-sharpCorner MVP | SharePoint StackOverflow, Github, PnP contributor
  • I am building business processes and applications that are easy for users' to stick to, so they can follow and understand them. In overall I transform processes to be more reliable and effortless. I am a proud co-organizer of SharePoint Saturday Warsaw and active community member, blogger and international speaker.