Showing results for 
Search instead for 
Did you mean: 
Frequent Visitor

Custom Connectors - My experience creating a FLOW with a custom connector, and what you should know

Hey Everyone,


I just wanted to share what my experience was when I created a custom connector for a FLOW I worked on.


FLOW Description


The FLOW we needed was an electronic document creation system. It required taking a document from a scanner, which was emailed via outlook to a particular mailbox. One the email arrived, we take the document, and send it to a Custom Connector which is connected to a REST API on a remote machine. Once the document is uploaded to the REST API, parse out some data from the document, rename the document, and upload the document into Sharepoint (TEAMS), with a row entered into a list with a link to the document, and the document saved in a folder.


Custom Connector (assuming your account has the access/authority to create a CC)


I won't get into the details of the FLOW so much, but I'll focus on the Custom Connector (CC). Creating the CC was very straightforward. Within FLOW editor, on the left side of the screen, click "Data", "Custom Connectors".




On the next screen, you click "+ New Custom Connector", "Create from blank".



You enter the name for the CC, and then proceed, on the next 4 screens (1. General, 2. Security, 3. Definition, 4. Test) in CC creation, entering in the host, protocol, action (for uploading a file, I'm using POST), request definition (multipart/form-data) and endpoint as well as security key was as easy as filling in the data on the CC creation screens. 




Testing on the 4th screen however is another topic for another day, but I was NEVER able to test successfully from the 4th CC screen, its useless.


* Lesson Learned - Create a new CC by importing an OpenAPI file


One thing you could do when you want to create a copy or clone, is that inside the CC screen, which lists all of your CCs, each CC has a download button;




When you click the download button, it creates a .swagger.json file which is also known as an OpenAPI file.  What you can do is download a .swagger.json file from an existing CC, and then open it in Notepad or some other app to edit text file, change the name, the endpoint, etc and then create a new one by importing the .swagger.json (OpenAPI file).






So what is a connection?  In order to get FLOW to connect to your REST API endpoint over a CC, you need to create a "Connection".

A connection is simply a place to store (in my case because I use the API key) the API Key for security. I normally create 2, one for test/development and another for production, this way I can switch out when I need to.  Switching out from production to test/dev is not actually this easy - you would need to create a clone or copy of your CC, because most likely your production REST API endpoint will be different from your test/development (see above for how to create a new CC by importing an OpenAPI file), and then create a copy/clone of your FLOW, and put in the test/dev CC step.


You can create a connection in one of 3 places. 


1. On the 4th screen of the CC Creation process




2. On the "Data", "Connections" screen ("+ New Connection")




3. Inside the FLOW Editor ("My Connections", "+Add new connection")




Incorporating the CC into your flow


So now you have your FLOW, at least started anyways, and now you have your custom connector, with a connection, that will hit your REST API.  Now add the CC to your FLOW by clicking the + button to "Insert a new action", and choose "Custom" to select your custom CC (the pic below actually has 4 CCs, you would choose whichever one you need);




* Lesson Learned - OFF TOPIC - How to structure the REQUEST data to upload a doc to a REST API from a FLOW/CC


I'm including this just to show you guys how to structure the REQUEST within the CC inside your flow for uploading a document.

There are other places online you can look it up, but there are really 2 parts to this;


1. BASE64 ENCODE THE DOCUMENT DATA - in my case, the document type we are dealing with is pdf.

To get the documents or really, attachments, first we need to setup a loop, so we can loop thru all of the attachments (See "Attachments" below in the pic inside "select an output from the previous steps" - this just means get the list of attachments from the email and loop over it) in the email.





What I need to do in my flow is, take each pdf attachment from the incoming email trigger (trigger is the first action of the FLOW), and upload it via the CC to my REST API. So I create a "Compose" step before the CC step, and load it with the "AttachmentsContent";




But the above pic is just to get and show the partial text i need for the pdf attachment contents, and if you hover over it, the "Compose" variable value looks like;




But this isn't all we need, like I said its just the partial text we need for this "Compose" element. What we really need to do is to take that partial text and wrap it in base64 encoding and enter it into the "Expression" editor which pops up every time you click inside a text field like below. We do this because we need to;


"...encode binary data into an ASCII character set known to pretty much every computer system, in order to transmit the data without loss or modification of the contents itself." What is the real purpose of Base64 encoding? - Stack Overflow -giorgio, Wolverine


Now Hold the blue "Ok" button under "Expression" for 2 seconds, and it will update the text field with the new base64 value.




which essentially looks like;




This is a super crucial pre-step for the actual CC step. From my experience, a CC to upload a file, WILL NOT work unless you base64 encode it.  


*Note: base64 decoding


Just as a reminder, because you are base64 encoding the binary file data in your FLOW prior to making the CC call, you'll need to base64 decode the file in somewhere in your REST API's endpoint function before you can do anything with the file (Early on in my endpoint function), I do;




Flask/python makes this easy, and I'm certain it will be just as easy in other languages/frameworks.


2. STRUCTURE THE REQUEST WITH multipart/form-data - now inside the CC step in our FLOW, we need to do pretty much the same thing when we created the CC on screen # 3. Definition, although here we are using variables for the values, whereas when we created the CC, we just typed in text as placeholders.




The first variable inside "$mulitipart-item 1" is just for the AttachmentName. The second is the output from the "Compose" step where we base64 encode the contentBytes of the pdf attachment.


Test it Out


Now we have our FLOW created along with our CC and inside it's REQUEST, we are sending the base64 encoded stream of our pdf file contentBytes. The first few times I tested this out, it worked great. Each attachment that was sent to the REST API via my CC contained only 1 page, hence 1 pdf file. But then I realized that part of the requirement was that users can scan multiple docs at a time within a stack of paper, and those pieces of paper in turn will all get combined into one pdf file attachment, and I would need to split the pdf doc into individual docs, and then upload each one to Sharepoint. Testing one page pdfs was fine, but these multi-page pdfs started taking a long time, and they were starting to fail, and I had no idea why.


Why was my CC (Custom Connector) failing?


One of the reasons was that the duration for the REQUESTS spawned from my CC could take varying amounts of time based on how many pages there were.  Inside my REST API I split the pdf, and for each page, convert each one to an image, convert the images to text, parse the text and look for specific fields, etc, etc, etc. Some took 10 seconds, while others took several minutes. So I looked in the "Settings" of the CC inside my FLOW, and saw a timeout setting.  I thought great, I'll just set this timeout to like 10 or 15 minutes or some amount that I know will never happen, and it will be fine!  Ummmmm..... no.


CC Settings Inside Flow


Lets take a look at the "Settings" for our CC in the flow;




Asynchronous Pattern


Setting this to "On", means that the CC will handle specific return values. If 202 "Accepted" is returned, then the CC will continue to call the REST API until it gets success 200 or anything else (Failures of 4xx, 5xx, etc).




This one can be confusing - here is the description;


"Limit the maximum duration an asynchronous pattern may take. Note: this does not alter the request timeout of a single request."


A single request timeout is 120 seconds or 2 minutes. And the above text is saying that no matter what you set this timeout value, and no matter how many calls are made async, it will not allow you to set it beyond the limit of a single request, which is again, 2 minutes.  And this is exactly what I saw when testing. I would set it to 15 minutes and if a REST API call took 3 minutes, it would always fail with a timeout failure after 2 minutes. And even worse, if a RESPONSE came back with a status code of 4xx, 5xx, etc, the whole FLOW would then go into a fail state, and you wouldn't have the opportunity to get the message or anything inside the response and then try to handle it based on what's in there.


How do we handle long running CC REQUESTS?


You can try to handle this by using this built-in Async handling in your CC settings, if you'd like. I ended up not doing that.  


For my process, I needed a little more control over what comes back inside the RESPONSE, as in I wanted to be able to read the error message or success message from the RESPONSE so I can put that in an email and send it to support or mgt.  The built-in Async handling did not seem as though it would let me do this, if a failure came back in the RESPONSE, the FLOW would just fail, and not give me chance to look inside the RESPONSE.  Keep in mind, you can keep the Asynchronous Pattern setting turned on for this CC as well as the Timeout setting, but in effect they won't matter because the 202 "Accepted" status comes back right away when we put the task on the queue inside flask.


To handle this the way I wanted, I had to do a couple few things, and if you don't care about my REST API specs, and what I had to change there, you don't need to read this section.


Change REST API endpoint to be asynchronous


Within my python flask REST API, I had to change how the endpoint function was structured;


1. Setup REDIS Queue - I had to install REDIS queue software on my dev and production machines.

2. Install and Configure Celery task queue manager - python has a task queue manager called celery which is a library that connects to and manages the REDIS (an many other) task queue(s).

3. Add a Status Check GET Request - I had to add another REST API endpoint to act as a task status check, that passes the task id, and asks the queue what the status of that task is. The Response json from both the CC that uploads the document and the CC that checks the status will be built inside flask and looks like when we put the task on the queue and its still pending and being worked on;




  { "status": 202,


    "msg": "",





And for our success...




  { "status": 200,


    "msg": "2 Files were uploaded to sharepoint",





And for our failures...




  { "status": 5xx,


    "msg": "An exception occurred trying to parse document text.",





Notice the HTTP Response status is always 200. We set that to 200 for ALL Responses we send back from the REST API, again, so we can control the FLOW and do what we want to do next, by reading the inner status, and then responding in the flow accordingly. We want all responses to come back, and to read the data from the response, FLOW will not allow you to control it very well if you send back something like;




  { "status": 5xx,


    "msg": "An exception occurred trying to parse document text.",





And sometimes it will just stop on the previous step or several steps above it and not allow you to see what is going on inside each step of the FLOW to debug it.


Also, the "Location" key in the response is really just the task_id from the task queue - we pull this value out of the RESPONSE to call the Status Check CC (see below).  In the built-in Async process its expected that it will contain 202 Accepted and a "Location" param on PENDING tasks - they suggest that the "Location" param should contain the full endpoint you need to call to get the status.  However, even though I kept the name of the param as "Location", I just ended up putting in the task_id ONLY, not the full endpoint.



Add a New CC (Custom Connector) for the Task Status Check


I also had to add a new CC;


1. Create a new CC - I created a new CC to call the REST API endpoint that will return the status of the task in the queue.




Creating the new CC was easier, its just setting it to GET on the # 3. Definition screen, and setting the endpoint with a parameter (localhost:5000 is just an example dev host);




And when we call it from the FLOW and enter the task_id in there, the URI endpoint would look like (for example);




CC Status Check Settings inside the FLOW


Within the Status Check CC, we had to change the settings a little bit;




1. Turn Asynchronous Pattern off - for this CC inside the "Do Until Loop", we don't want to be effected by the timeout or the built-in behavior of Async Pattern here. We want put this CC inside a "Do Until Loop", and then stop the loop when the status changes from 202 "Accepted" to 200 "success" (or failure);




As you can see as well, I set the Limit count to 20 and the timeout to 20 minutes, we should never eclipse either of those. We will break out of the loop on either success or failure from the RESPONSE. We set the FLOW variable called variables('CONTINUE STATUS CHECK') to either True to keep checking, or False to break out the loop based on the inner status code in the RESPONSE.


2. Do not set a Timeout value on CC - for this CC we will not set a Timeout value, we will rely on the "Do Until Loop" timeout limit we set.


2. Do not set a Retry Policy - for this CC we will not set a Retry Policy value, we will set it to None - this can just get confusing and mess up the FLOW and the timing of the CC REQUESTs - of something fails inside the critical functions inside flask, we already baked in retry logic in python, no real need for it here.


Paralell Processing tip - Previously in flask I would run a clean up function to delete the temp files i had created in a folder. But I had to set the initial Incoming Email Trigger to only allow 1 FLOW to be run at 1 time.  This was because if I allowed multiple FLOWS to be run, and the cleanup function was run, it might delete files that were needed later.  So I just named any files created with an additional hex key in flask during a FLOW call to the REST API from the CC, and then at the end of the REST API call, I delete any file with that key so it doesn't affect other FLOW runs. By doing that, I can now set the Concurrency Control value to 2, which is the number of cpu's i currently have on the machine. I set the uwsgi # of processes to 2 as well, and now have 2 celery workers, so running 2 FLOWS concurrently works fine. I may ramp it up to 3 if this continues to perform well.






Creating my own version of an Async CC for me was definitely a learning curve. I had never worked with REDIS or Celery before, and quite frankly after more research, there is a redis python lib out there you can use instead of celery, which I think is a little more straightforward. Once I put this into production, there were a configuration element in Celery that broke an API call I was making later in the process. Once I figured out which config value it should be (--pool=solo) , it was fine. Again, I was not able to get the built-in Async feature of the CC in the FLOW to do what I wanted, but at least with the above (convoluted) process, I was able to control the FLOW a bit more and set the timeout to whatever I want now (within the limitations of the Do Until Loop) for the Check Status REST API call and send notifications for failure, etc. I hope providing insight into the the things that tripped me up can help someone.

Power Automate
Power Automate

Excellent post! Thanks for sharing your experience and feedback. I am sure there are many things we can do to improve the overall experience - perhaps more of a guided topic with explanations rather than a series of do this, do that. 


Thanks for sharing this outstanding article. I would encourage you to try out the import from github option and the paconn cli, which is designed to accelerate the custom connector development cycle.


Helpful resources


Back to Basics: Tuesday Tip #1: All About YOUR Community Account

We are excited to kick off our new #TuesdayTIps series, "Back to Basics." This weekly series is our way of helping the amazing members of our community--both new members and seasoned veterans--learn and grow in how to best engage in the community! Each Tuesday, we will feature new areas of content that will help you best understand the community--from ranking and badges to profile avatars, from Super Users to blogging in the community. Our hope is that this information will help each of our community members grow in their experience with Power Platform, with the community, and with each other!     This Week's Tips: Account Support: Changing Passwords, Changing Email Addresses or Usernames, "Need Admin Approval," Etc.Wondering how to get support for your community account? Check out the details on these common questions and more. Just follow the link below for articles that explain it all.Community Account Support - Power Platform Community (   All About GDPR: How It Affects Closing Your Community Account (And Why You Should Think Twice Before You Do)GDPR, the General Data Protection Regulation (GDPR), took effect May 25th 2018. A European privacy law, GDPR imposes new rules on companies and other organizations offering goods and services to people in the European Union (EU), or that collect and analyze data tied to EU residents. GDPR applies no matter where you are located, and it affects what happens when you decide to close your account. Read the details here:All About GDPR - Power Platform Community (   Getting to Know You: Setting Up Your Community Profile, Customizing Your Profile, and More.Your community profile helps other members of the community get to know you as you begin to engage and interact. Your profile is a mirror of your activity in the community. Find out how to set it up, change your avatar, adjust your time zone, and more. Click on the link below to find out how:Community Profile, Time Zone, Picture (Avatar) & D... - Power Platform Community (   That's it for this week. Tune in for more Tuesday Tips next Tuesday and join the community as we get "Back to Basics."

Announcing the MPPC's Got Power Talent Show at #MPPC23

Are you attending the Microsoft Power Platform Conference 2023 in Las Vegas? If so, we invite you to join us for the MPPC's Got Power Talent Show!      Our talent show is more than a show—it's a grand celebration of connection, inspiration, and shared journeys. Through stories, skills, and collective experiences, we come together to uplift, inspire, and revel in the magic of our community's diverse talents. This year, our talent event promises to be an unforgettable experience, echoing louder and brighter than anything you've seen before.    We're casting a wider net with three captivating categories:  Demo Technical Solutions: Show us your Power Platform innovations, be it apps, flows, chatbots, websites or dashboards... Storytelling: Share tales of your journey with Power Platform. Hidden Talents: Unveil your creative side—be it dancing, singing, rapping, poetry, or comedy. Let your talent shine!    Got That Special Spark? A Story That Demands to Be Heard? Your moment is now!  Sign up to Showcase Your Brilliance:  Deadline for submissions: Thursday, Sept 28th    How It Works:  Submit this form to sign up:  We'll contact you if you're selected. Get ready to be onstage!  The Spotlight is Yours: Each participant has 3-5 minutes to shine, with insightful commentary from our panel of judges. We’re not just giving you a stage; we’re handing you the platform to make your mark.     Be the Story We Tell: Your talents and narratives will not just entertain but inspire, serving as the bedrock for our community’s future stories and successes.    Celebration, Surprises, and Connections: As the curtain falls, the excitement continues! Await surprise awards and seize the chance to mingle with industry experts, Microsoft Power Platform leaders, and community luminaries. It's not just a show; it's an opportunity to forge connections and celebrate shared successes.    Event Details:  Date and Time: Wed Oct 4th, 6:30-9:00PM   Location: MPPC23 at the MGM Grand, Las Vegas, NV, USA  

September User Group Success Story: Reading Dynamics 365 & Power Platform User Group

The Reading Dynamics 365 and Power Platform User Group is a community-driven initiative that started in September 2022. It has quickly earned recognition for its enthusiastic leadership and resilience in the face of challenges. With a focus on promoting learning and networking among professionals in the Dynamics 365 and Power Platform ecosystem, the group has grown steadily and gained a reputation for its commitment to its members!   The group, which had its inaugural event in January 2023 at the Microsoft UK Headquarters in Reading, has since organized three successful gatherings, including a recent social lunch. They maintain a regular schedule of four events per year, each attended by an average of 20-25 enthusiastic participants who enjoy engaging talks and, of course, pizza.   The Reading User Group's presence is primarily spread through LinkedIn and Meetup, with the support of the wider community. This thriving community is managed by a dedicated team consisting of Fraser Dear, Tim Leung, and Andrew Bibby, who serves as the main point of contact for the UK Dynamics 365 and Power Platform User Groups.   Andrew Bibby, an active figure in the Dynamics 365 and Power Platform community, nominated this group due to his admiration for the Reading UK User Group's efforts. He emphasized their remarkable enthusiasm and success in running the group, noting that they navigated challenges such as finding venues with resilience and smiles on their faces. Despite being a relatively new group with 20-30 members, they have managed to achieve high attendance at their meetings.   The group's journey began when Fraser Dear moved to the Reading area and realized the absence of a user group catering to professionals in the Dynamics 365 and Power Platform space. He reached out to Andrew, who provided valuable guidance and support, allowing the Reading User Group to officially join the UK Dynamics 365 and Power Platform User Groups community.   One of the group's notable achievements was overcoming the challenge of finding a suitable venue. Initially, their "home" was the Microsoft UK HQ in Reading. However, due to office closures, they had to seek a new location with limited time. Fortunately, a connection with Stephanie Stacey from Microsoft led them to Reading College and its Institute of Technology. The college generously offered them event space and support, forging a mutually beneficial partnership where the group promotes the Institute and encourages its members to support the next generation of IT professionals.   With the dedication of its leadership team, the Reading Dynamics 365 and Power Platform User Group is poised to continue growing and thriving! Their story exemplifies the power of community-driven initiatives and the positive impact they can have on professional development and networking in the tech industry. As they move forward with their upcoming events and collaborations with Reading College, the group is likely to remain a valuable resource for professionals in the Reading area and beyond.  

A Celebration of What We've Achieved--And Announcing Our Winners

As the sun sets on the #SummerofSolutions Challenge, it's time to reflect and celebrate! The journey we embarked upon together was not just about providing answers – it was about fostering a sense of community, encouraging collaboration, and unlocking the true potential of the Power Platform tools.   From the initial announcement to the final week's push, the Summer of Solutions Challenge has been a whirlwind of engagement and growth. It was a call to action for every member of our Power Platform community, urging them to contribute their expertise, engage in discussions, and elevate collective knowledge across the community as part of the low-code revolution.   Reflecting on the Impact As the challenge ends, it's essential to reflect on the impact it’s had across our Power Platform communities: Community Resilience: The challenge demonstrated the resilience of our community. Despite geographical distances and diverse backgrounds, we came together to contribute, learn, and collaborate. This resilience is the cornerstone of our collective strength.Diverse Expertise: The solutions shared during the challenge underscore the incredible expertise within our community. From intricate technical insights to creative problem-solving, our members showcased their diverse skill sets, enhancing our community's depth.Shared Learning: Solutions spurred shared learning. They provided opportunities for members to grasp new concepts, expand their horizons, and uncover the Power Platform tools' untapped potential. This learning ripple effect will continue to shape our growth. Empowerment: Solutions empowered community members. They validated their knowledge, boosted their confidence, and highlighted their contributions. Each solution shared was a step towards personal and communal empowerment. We are proud and thankful as we conclude the Summer of Solutions Challenge. The challenge showed the potential of teamwork, the benefit of knowledge-sharing, and the resilience of our Power Platform community. The solutions offered by each member are more than just answers; they are the expression of our shared commitment to innovation, growth, and progress!     Drum roll, Please... And now, without further ado, it's time to announce the winners who have risen above the rest in the Summer of Solutions Challenge!   These are the top community users and Super Users who have not only earned recognition but have become beacons of inspiration for us all.   Power Apps Community:  Community User Winner: @SpongYe Super User Winner: Pending Acceptance Power Automate Community:  Community User Winner: @trice602 Super User Winner: @Expiscornovus  Power Virtual Agents Community: Community User Winner: Pending AcceptanceSuper User: Pending Acceptance Power Pages Community: Community User Winner: @OOlashyn Super User Winner: @ChristianAbata   We are also pleased to announced two additional tickets that we are awarding to the Overall Top Solution providers in the following communities:    Power Apps: @LaurensM   Power Automate: @ManishSolanki    Thank you for making this challenge a resounding success. Your participation has reaffirmed the strength of our community and the boundless potential that lies within each of us. Let's keep the spirit of collaboration alive as we continue on this incredible journey in Power Platform together.Winners, we will see you in Vegas! Every other amazing solutions superstar, we will see you in the Community!Congratulations, everyone!

September featured user group leader

 Ayonija Shatakshi, a seasoned senior consultant at Improving, Ohio, is a passionate advocate for M365, SharePoint, Power Platform, and Azure, recognizing how they synergize to deliver top-notch solutions. Recently, we asked Ayonija to share her journey as a user group leader, shedding light on her motivations and the benefits she's reaped from her community involvement.      Ayonija embarked on her role as a user group leader in December 2022, driven by a desire to explore how the community leveraged various Power Platform components. When she couldn't find a suitable local group, she decided to create one herself!    Speaking about the impact of the community on her professional and personal growth, Ayonija says, "It's fascinating to witness how everyone navigates the world of Power Platform, dealing with license constraints and keeping up with new features. There's so much to learn from their experiences.:        Her favorite aspect of being a user group leader is the opportunity to network and engage in face-to-face discussions with fellow enthusiasts, fostering deeper connections within the community. Offering advice to budding user group leaders, Ayonija emphasized the importance of communication and consistency, two pillars that sustain any successful community initiative.      When asked why she encourages others to become user group leaders, Ayonija said, "Being part of a user group is one of the best ways to connect with experienced professionals in the same field and glean knowledge from them. If there isn't a local group, consider starting one; you'll soon find like-minded individuals."      Her highlight from the past year as a user group leader was witnessing consistent growth within the group, a testament to the thriving community she has nurtured. Advocating for user group participation, Ayonija stated, "It's the fastest route to learning from the community, gaining insights, and staying updated on industry trends."   Check out her group: Cleveland Power Platform User Group

An MPPC23 Invitation from Charles Lamanna, CVP of Microsoft Business Applications & Platform

Hear from Corporate Vice President for Microsoft Business Applications & Platform, Charles Lamanna, as he looks ahead to the second annual Microsoft Power Platform Conference from October 3rd-5th 2023 at the MGM Grand in Las Vegas.Have you got your tickets yet? Register today at  

Top Solution Authors
Users online (2,293)