Random Intermittent Failures with Power Automate Desktop and Gateway

[HEATFreight]

Attended RPA PAD flows triggered from the cloud have been working for us most of the time, but occasionally they fail for no apparent reason.

We get the green status bar icon that shows a flow is queued, but the flow never starts. Additionally, we have seen that the File System connector which requires a gateway instance is also breaking. It would seem that these failures coincide but I can't say for sure.

With the File System connector in a cloud flow, the error we get is as follows:

{
  "status": 404,
  "message": "Could not connect to your data gateway. Make sure your file system is active with the gateway running. The detailed error is 'Gateway on endpoint '...' is unreachable.'.\r\nclientRequestId: ...",
  "error": {
    "message": "Could not connect to your data gateway. Make sure your file system is active with the gateway running. The detailed error is 'Gateway on endpoint '...' is unreachable.'."
  },
  "source": "filesystem-eus.azconn-eus-01.p.azurewebsites.net"
}

Because we have a timeout set, the PAD flows triggered from the cloud fail with:

ActionTimedOut. The workflow action 'Run_a_flow_built_with_Power_Automate_Desktop' timed out while waiting for underlying operation to complete.

 
However the timeout is not the problem. If we did away with the timeout, the flow would apparently just never start at all.

This is extremely frustrating. 

For reference, our PAD flows run in a Hyper-V Windows 10 Pro VM. Not sure if there could be some setting in the VM that could cause intermittent issues, or possibly our router/firewall settings could have some effect that only causes errors occasionally. But since it works maybe 80 or 90 percent of the time, it would be hard to find some local reason for these issues.

We assume Microsoft just sucks. We won't keep paying for our subscription if it continues to suck.

[HEATFreight]

There is a chance that inadequate RAM allocated to the Hyper-V virtual machine that runs our PAD flows was the root cause of these intermittent issues. We've doubled the RAM for that VM. Hoping that fixes it.

[HEATFreight]

Edit: THIS IS STILL HAPPENING. RAM is at 8GB with 2.9GB in use. The rest of the RAM which is not "In Use", "Modified", "Standby", or "Free" is in the "Hardware Reserved" section of Physical Memory tab of Resource Monitor. There are about 5GB in "Hardware Reserved". Not sure, but that sounds right. I mean it adds up to the total 8GB amount that I've assigned in Hyper-V Manager settings for this VM.

I have now updated to the newest release of Power Automate Desktop and Gateway. Issue had returned and was persistent before and after updating and restarting multiple times.

Will try to run flow with Gateway connection instead of Machine connection, which is in preview right now so could be buggy...???

Edit: it was not the Machine connection. Switching the "Run a desktop flow" trigger to the Gateway connection instead of the Machine connection did not fix the problem. 

[HEATFreight]

"Save as" might be a workaround. I used "Save as" to copy the cloud flow that triggers the desktop flow, and after this both the original cloud flow and the copy were able to trigger the desktop flow without any problems.

Could just be a coincidence though. As far as I can tell these are random outages in the connection between a cloud flow and the desktop flow that it triggers.

[HEATFreight]

Here is the response I got from Microsoft Support today:

I have been informed today that there was an issue with desktop flows not executing, the issue started 2-3 days ago where all flows are failing and it was resolved last night, I suggest you try again today.

 

if the issue persists please send me the flow id, you can check on this clicking on my flows in power automate, click on the three dots and select details, also if you can send me the network trace that you did, it would be helpful.

[HEATFreight]

Do we get a discount on our paid subscription plans whenever the service sh!ts the bed?

[CParsons09]

@HEATFreight Did you find a resolution for this? I just started experiencing very similar behavior. It hits my 30 minute timeout but if I resubmit, it works fine the second time. This desktop automation has been in production for over 6 months without this issue. I'm hoping it's temporary and on MS' side. If it continues I will open a ticket and see what kind of response I get. 

[HEATFreight]

In our current configuration we do not experience this issue anymore.

I'm not entirely certain what, if anything, is different now compared to when I posted this.

It's possible that allocating more resources to the VM where our PAD flow runs fixed the issue. Currently we have a W10 Hyper-V VM with 8GB RAM and two cores which executes our PAD flow flawlessly and quickly, very little discernible latency from Power Automate cloud to desktop and back.

Works great! Never have any issues.

[HEATFreight]

I will add that allocating a larger volume for the VM might have also helped. 20GB is simply not enough for a Windows 10 VM!

My other suspicion is that queued runs were making each other time out. If one run got stuck or failed or ran extra slow, the error would cascade through to other queued runs since they all needed to run in less than the, I think, 5-minute timeout that we allocated to the PAD flow. We were at the time using a browser automation to fill data into legacy forms, but we have since moved on and now our PAD flows pretty much only run scripts, as well as doing some other orchestration of logic that we didn't feel like scripting.

The scripts run extremely reliably and do not get triggered often enough to have several runs queued up on our only PAD machine at any given time, and even if a queue forms, they typically run fast enough that user won't know the difference. I wish the initial connection to and response from PAD were faster because those seem to be the bottleneck, rather than the run time of the PAD flow itself.

tl;dr: probably due to queued flow runs interfering with each other's ability to succeed within timeout

[momlo]

As far as I remember, flows will wait up to 3h by default. To extend that, you need to edit the Power Automte action and set up custom timeouts. You can also build PA flow in a way it parses the output of the PAD action and takes action based on the exit code - repeats the action, for example in case of timeout.

[HEATFreight]

Why would I ever want to extend the PAD flow timeout beyond the default? I just made a PAD flow that runs fast enough that the timeout doesn't matter. If there are so many flows queued that it needs a longer timeout, then you would be best served by adding more machines running PAD, not a longer timeout.

My users aren't waiting hours for their data. They are waiting seconds.

[momlo]

Man.. chill out; You didn't write "flows need to execute in seconds" - You just wrote about flows getting stuck in a queue and timeout. If you do not want them to fail, one thing is to manage the schedule, timeout, etc. settings, and the other is to monitor the queue proactively for bottlenecks and remove stuck flows manually, or built additional routines on your Power Automate flow to automate this - simply - automate your automation. Over.

[HEATFreight]

I don't think you read the part where I said I've eliminated all the issues we've had with PAD flows:

The scripts run extremely reliably and do not get triggered often enough to have several runs queued up on our only PAD machine at any given time, and even if a queue forms, they typically run fast enough that user won't know the difference. I wish the initial connection to and response from PAD were faster because those seem to be the bottleneck, rather than the run time of the PAD flow itself.

You are replying to me as if I need support. I did start this thread, but that was a long time ago. The only reason I'm replying now is because someone else asked if I resolved my issue.

You seem to be offering unhelpful solutions for a problem I've already explained how I solved.

[asbpowerbi]

Hello Friends,

  I am very new to Power Automate and I too amd getting this same issue with the following message:

The workflow action 'Run_a_flow_built_with_Power_Automate_for_desktop_2' timed out while waiting for underlying operation to complete. How do I debug this issue? When I log into the remote system and execute the desktop flows manually, they work fine. I then go and trigger the Power Automate flow and it seems to go fine. What is the best way to find out what caused the time-out and how do I find out the instruction that it failed on?

Cheers!

Peter