Previously working DataFlow now throwing "We encountered an unexpected error."

[monstrator]

Hi!

I hope to get a pointer on the error that suddenly came up in all our DataFlows: "We encountered an unexpected error." 

The DataFlows fetch data from Excel files on Sharepoint. It worked well a month ago but now the above error occurs efter I went through the Edit steps of a DataFlow (even without editing it). Previews work fine, I can map all columns etc. 

The only change that was done recently is to change the Environment name. Even new DataFlows see this error.

Best regards and thanks!

// Denis

[EricRegnier]

Hi @monstrator, so if I understand correctly, you corrected this error? If not, do you get this error when editing a new dataflow or after a run? To see the error, click on the little warning icon next to the Last Refresh in make.powerapps.com --> Data --> Dataflows

 

Thanks!

[CharlesS]

Hello @monstrator !

All your dataflows have the same problem? Interesting. Have you tried creating a new dataflow, something very simple, importing nothing but the key value and a single text field? This would just validate if dataflows stopped working completely or if only your previously built dataflows are at issue.

If the new dataflow with nothing but one field works, I would try this:

1. If you still have access to one of your previously imported datasets, try importing a single record from the old dataset (one that had a successful import) using your existing dataflow. If it works, the issue may be with the data.

2. If it doesn't work, try unmapping all your fields but one. If that works, add another field and refresh again until you identify the problematic field. If that doesn't work, try rebuilding your dataflows from scratch (or copy and paste from the advanced editor). 

You will notice that the new datasets will be Standard V2. The differences between V1 and V2 are not clearly documented (to anyone: if I'm wrong and documentation exists, please reply below, I would love to read it). For sure, there have been back end changes lately. I've also had previously working dataflows "die" and I solved the issue by creating a new V2 dataflow, but some tweaking was required with dates and option sets (see here https://powerusers.microsoft.com/t5/Microsoft-Dataverse/Dataflow-Engine-V2-What-exactly-changed/m-p/...

Also, you said nothing changed except for the name of the environment. Did you consider if any new business rules were applied? I had another dataflow that failed because of a faulty business rules (see https://powerusers.microsoft.com/t5/Microsoft-Dataverse/Dataflow-error-Error-code-0x80040265-Message...)

Keep 'em flowin'

Charles

[monstrator]

Thanks for your comment @EricRegnier !

I tried and and it failed:

But the run status is not visible on the Refresh History overview:

BR, Denis 

[monstrator]

Thanks @CharlesS !

I have V2 Dataflows since before. 

We are pulling data from Excel files stored on Sharepoint. All M-Query conversions are tested in PowerBI before.

I tried to create a test dataFlow and only map one single field to an existing table. The same 'Unexpected error' occurs.

I also realized that the Overview page of all my DataFlows becomes awfully slow after I have seen that Unexpected Error - either by editing an existing DF or adding a new one, showing this screen forever:

I have the role 'System Administrator'

Br, Denis

[EricRegnier]

Did you click on the warning icon? What's the error?

[monstrator]

Hi @EricRegnier ,

The status of the failed refresh is not visible in the status:

When clicking the Warning symbol I only get old runs:

Seems like the logging is the culprit here and might have something to do with why the data flows don't work any more after the environment name changed.

// Denis

[CharlesS]

Hello @monstrator !

It is quite possible that changing the environment name may indeed be the root cause of the problem.

I have two suggestions for you:

1. Raise a support ticket with Microsoft and see what they have to say. If you choose this option, it would be really great if you could post their solution here for other users to see.
2. Just an idea... this sound like it could be a data integration problem. You can try to play around with the "Data Integration" tab in the Power Apps Admin Center. I have dabbled around here before (solved a problem with incorrectly mapped alternate keys for lookup fields here) but I am no expert with this part of Power Apps. Nevertheless, here is something you can try:
   1. Go to Data Integrations in the Power Platform admin center 
      1. 
   2. Select Connection Sets
      1. 
   3. Open one of your data integration projects and click on Organizations
      1. 
         1. If the name under Common Data Service is the old name of your environment, then for sure the name change is the problem.
         2. If this is the case, perhaps creating a new connection set or modifying the old connection set to change the environment name could be a solution. Unfortunately, I cannot be of much assistance here as my knowledge to this effect is limited.
         3. Start by checking it out and let me know what you find. We may be able to find someone who knows more about this subject and who can help.

Keep 'em flowin'!

Charles

[monstrator]

@CharlesS good ideas!

I had a look but there is no integration project with the old or the new environment name.

My thought is that the changed environment name caused some issues for logging of the Data Flow runs. That's why the latest failed run is not visible either.

The DataFlow tries to run and write the progress/results to the log location but cannot do that for some reason and fails with an unexpected error no matter of the run itself has issues or not...

We are exporting the solution now, testing it in our test environment. If it works there, we will recreate the DEv environment and import there again.

Thanks,  Denis

[monstrator · Accepted solution]

Hi all!

In the end it was a service request with Microsoft that was necessary. Now things are working again.

Moving the solution into another environment caused an error for the data flows as well so I had to recreate them manually.

I could use the classic interface but then the data flows were created as 'V1' dataflows. But that's a topic for another post.

Summary: Problem solved through Service Request!

BR, Denis