How to fetch file more than 30 MB from folder on server using on premise data gateway in MS Flow

[tidkeamol]

Hello Everyone,

I have created a flow to fetch files from folder on server and upload it to SharePoint document library.

I used on premise data gateway connector to access the files from remote server and I am able to access files and upload it to SharePoint using create file.

But I got below error for some of my files in "Get Files":

{

  "status": 413,

  "message": "The file contains 60.494 megabytes which exceeds the maximum 30 megabytes.\r\nclientRequestId: 4ec7642b-8a3e-46f7-a01c-7fd8c7eb64f8",

  "error": {

    "message": "The file contains 60.494 megabytes which exceeds the maximum 30 megabytes."

  },

  "source": "filesystem-we.azconn-we-01.p.azurewebsites.net"

}

Can someone help to access files more than 30MB or its 

Is there any way around to achieve this?

Thanks & Regards,

Amol Tidke

[v-litu-msft · Accepted solution]

Hi @tidkeamol,

Maximum file size for general operations is 30 MB;

Maximum file size for create file operation is 20 MB.

There is no way that could exceed the limits of the file system.

For more information about File system, please refer to this document:

File System - Connectors | Microsoft Docs

Best Regards,
Community Support Team _ Lin Tu
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[martinav]

I have been using this function for over two years.  It was not until today that I found this limit.  This is traffic on my own on-prem gateway.  I'm quite miffed that this limit exists.  This crashes my ENTIRE basis and file structure scheme I have developed over these years.  I had no reason to even think there was a file-size limit.  Yet another unclear limitation that screws me after the fact.  Wow, now what?  I have to reconsider and restructure everything due to this.  Thanks Microsoft ( @v-litu-msft )

Sincerely,

Rod Martin

[NorthernChuck]

It's possible you are maxing out the limitations of this construct. Have you checked to see if Azure Logic Apps has the same limitations?  Maybe Razor and a SWA is better suited to your application of these larger file transfers?

I find limits all the time with Flow. It's handy to learn more than one language like Javascript, Python, Powershell, C# - where one language fails in creating a workable construct, another might be better suited. 

[martinav]

@NorthernChuck ,

Well, I do not consider 30mb to be large at all.  I manipulate data in powerapps and flow in other ways that I'm sure exceeds 30mb of file transfer.  

I am unaware of Razor, SWA, and I am not using Azure in any way.  I'm sure all of these things have their own vast array of licensing schemes.  I am not positioned to add these to our licensing costs, just for the purpose of a puny 30mb file size limit.  Looks like more wasted time to come up with a work-around that will overall require more file transfer than if a larger size were allowed to begin with.

[NorthernChuck]

I mentioned SWA cause it sounded like you had some kind of external assets that you needed to trigger on. ... it's probably overkill and/or not applicable now that I read your other post. 
What is Azure Static Web Apps? | Microsoft Docs

... I'm not sure how to get-around your limitation.... i'm lucky enough that I can use PowerShell and copy-item and so I don't run into this large file problem... 
I know that you don't have Azure resources.. but should you ever get Azure resources this might help, Azure Cloud Shell overview | Microsoft Docs

[kevinli]

I am on the same boat. Not sure if you have found a workaround solution yet.

I have previous built many solutions to copy files from SharePoint to On-Prem server, never had issue as the files were smaller than 20mb. 

Discovered this limitation today while setting up a new Power Automate Flow to copy files with size between 50mb to 100mb from SharePoint to On-Prem server. 

My workaround is: First use "CREATE FILE" to create an empty file on the destination, then use a "DO UNTIL" loop with "Append file" to copy the file piece by piece from SharePoint to on-Prem.

This works, but with some issues. As the file is not locked between each Create file, or between each "Append File" actions, there is a risk that depending process may consume the file while its partially transferred. I also tried to transfer the file to staging folder than using "Copy File" function, then run into the 30MB limitation. 

It would be really appreciated if you can share your ideas to address the 20mb/30mb limitation.

One workaround is to 

[rsuau]

Hi All
I'm having the exact same scenario and problem. Getting native xlsx files from a sharepoint online and having to copy it to an internal fileshare through an on premise data gateway. Xlsx files are produced by a provider application, and some of them go over the 20MB. Connectors do not provide chunking, and there seems no gateway parameter allow to increase the limit, so i'm stuck
@kevinli workaround seems very promising but I'm quite new to power automate and could not find a way to split a binary file in subparts. Would you be keen to share a piece of code to help us?
Thanks in advance

[Thoman]

Couldn't you check out the file with power automate, do the loop, then check it back in?