cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Ale12
Frequent Visitor

Multiple Lists contents to Blob storage

Hi,

i'm trying to copy all contents from several lists stored on a single Sharepoint page to Blob storage.

Ale12_0-1620725314829.png

Solution 1.
Occurance everyday
get lists on the SharePoint

copy them on the blob storage

Problem sol. 1: on the blob storage I can only select as content Name and display name, but I need all information and values stored.

solution 2.
Get contents on Sharapoints
Create a Blob storage 


Problem sol. 2: I can only copy contents of a single list, the file contains around 20 lists, how can I copy the contents of all lists at the same time?

Do you have better solutions?

11 REPLIES 11
annajhaveri
Super User
Super User

@Ale12 few questions:

 

1. How do you want the details to be stored on blog, as list data would be rows and column of data, would json data of list work?

2. When you say you the file contains around 20 lists, would that be a dynamic file, and you need to read lists from file and get data and save it to blob?

3. The next time the flow runs in recurrence, you want to get all data from list and save to blob? If yes, what about the data saved in previous run that is saved to blob?

Regards,
Anna Jhaveri

If I have answered your question, please mark the post as Solved to help other users to identify the correct answer
If you like my response, please give it a Thumbs Up.

So FYI the goal would be than to connect Tableau to the data stored on Tableau.

1. I believe both solution works fine. I was thinking to make a .csv, other suggestions?

2. Correct the file contains several list. I was thinking to create for each list a "Get items".

3.All data should be saved on the blob and ideally the data stored in the blob will be updated automatically. on a daily basis.

 

Ale12_0-1620908481716.png

 

@Ale12 , see below responses

 

1. Yes you can create a CSV, refer to this template https://flow.microsoft.com/en-us/galleries/public/templates/79095f001ee911e79dc3835964ea3218/copy-sh... which has required actions to create CSV from SharePoint data

2. If you have fixed 20 lists every time to process yes you can create 20 Get Items action. How many rows of data would your lists have, if its more than 5000 rows, then you may face some limitations of getting the data, alternatively for faster performance of your flow, you can use Send HTTP Request to SharePoint action to use REST API to get data from SharePoint, let me know if you need instructions on how to do that.

 

3. Understood, so i understand you will have list name as unique identifier to replace the data on blob

Regards,
Anna Jhaveri

If I have answered your question, please mark the post as Solved to help other users to identify the correct answer
If you like my response, please give it a Thumbs Up.

1. The problem is that i have multiple lists, in this case, what should I do? Is it fine to have the apply each and then create the .csv table? Why I can not find the exact match for some items, besides APQR site owners?

2. What happens if the lists increase?

3. I need to have all lists and all data send out to a blob storage. 

Can we have a short call? 

 

Ale12_0-1620913392727.png

 

@Ale12 yes we can have a call, i sent you a private message.

Regards,
Anna Jhaveri

If I have answered your question, please mark the post as Solved to help other users to identify the correct answer
If you like my response, please give it a Thumbs Up.

@Ale12 

 

1. The problem is that i have multiple lists, in this case, what should I do? Is it fine to have the apply each and then create the .csv table? Why I can not find the exact match for some items, besides APQR site owners?

A: You dont need apply to each for creating the CSV, and it shouldnt be used as it will make your flow slow, as you will have more records to process

 

2. What happens if the lists increase?

A: What is expected number of records the list can have? You will need to implement some logic to get paged records. I think you should use REST API to get data from SharePoint using Send HTTP Request to SharePoint rather than using Get Items action

 

3. I need to have all lists and all data send out to a blob storage. 

A: So you need all lists data in one csv stored to blog storage?

Regards,
Anna Jhaveri

If I have answered your question, please mark the post as Solved to help other users to identify the correct answer
If you like my response, please give it a Thumbs Up.

Hi @annajhaveri , i've sent you a private message please check it.
Considering how important and insightful this post might be for the whole MS power automate community, may I ask you if you can post a picture of how the flow should be created? A step-by-step guide would be great!

1. if I don't use .csv, there is a better solution, which one?  
2. How can I implement the REST API? and HTTP request? Also I need to have data updated on a daily basis.
3. Well my goal is simply to have all data that are currently in SharePoint in blob storage. In the past, I've created a .csv to accomplish this result, but at the time I was working on a single list, here we have +20 lists.

@Ale12  you can use below flow as reference to start working on your flow

 

annajhaveri_0-1621268626608.png

 

 

It gets all the lists from SharePoint, then use Get Items to get data from the list in the apply to each loop which loop through all the lists. If you want to skip any list from not getting processed then add condition action under apply to each action and check if List Display is not equal to that specific display name, then in Yes branch add the get items action. 

 

Now for creating CSV currently i havent added logic to combine all list data, as each list can have its own set of columns, so if set of columns are different for each list, it doesnt makes sense to merge all list data. So let me know if all your lists has same set of columns, if not, then you can save each csv table to blob separately.

 

I saw that you shared link to process more than 5000 items, i will look into it, but it is not advisable to process so many records in flow, it affects flow performance, still i will look into it and try to implement in the current flow and share the updated screenshots.

Regards,
Anna Jhaveri

If I have answered your question, please mark the post as Solved to help other users to identify the correct answer
If you like my response, please give it a Thumbs Up.

Hi @annajhaveri 

some follow ups:

- for the get items when i run the model it runs for 25 times the same list ... the only one defined under "get items"

Should i add for each lists a "get items"?

- the total of items is more than 5000, we should include the workaround

- for the blob should I only have "create blob" > if the sharepoint is updated with a new item will the flow update the same file or will it create a new blob? My goal is to update the same file. @eliotcole 

Ale12_1-1621341054682.png

 

 

 

@Ale12 please see responses below:

 

- for the get items should I have a single or should I create for each list a "get items"

A: As we are getting all the lists using Get Lists, and then under apply to each loop we are looping through each list and getting items, using get items, you dont need to add multiple get items actions.

 

- the total of items is more than 5000, we should include the workaround

A: Yes i will provide you workaround for more than 5000 items, however it is not advisable to process more than 5000 items, still i will provide you steps

 

- for the blob should I only have "create blob" > if the SharePoint is updated with a new item will the flow update the same file or will it create a new blob? My goal is to update the same file.

A: I think you need to implement logic in the flow to check whether specific blob exist, if it exist then use Update Blob to update the same file and if it doesnt exist, then use Create Blob to create new blob file.

Regards,
Anna Jhaveri

If I have answered your question, please mark the post as Solved to help other users to identify the correct answer
If you like my response, please give it a Thumbs Up.

Hi Anna:

so i want to clarify something the sharepoint is made of 25 list, which are used for this dashboard. I'm wondering if I use "get lists", when I create the .csv the connection between data will be maintained?

Ale12_0-1621411173224.png



- for one of the list this error pops up: "The query cannot be completed because the number of lookup columns it contains exceeds the lookup column threshold.
would be possible to select only few lists? If yes can you show me how? I sent you a message with a possible solution 

- can you show me the flow on how to Create or Update blob?

Helpful resources

Announcements
Process Advisor

Introducing Process Advisor

Check out the new Process Advisor community forum board!

MPA User Group

Welcome to the User Group Public Preview

Check out new user group experience and if you are a leader please create your group

V3_PVA CAmpaign Carousel.png

Community Challenge - Giveaways!

Participate in the Power Virtual Agents Community Challenge

Carousel 2021 Release Wave 2 Plan 768x460.jpg

2021 Release Wave 2 Plan

Power Platform release plan for the 2021 release wave 2 describes all new features releasing from October 2021 through March 2022.

Users online (3,446)