cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Mo_Islam
Helper III
Helper III

Using flow to collect Sharepoint List (Almost there)

Hi,

 

I have a SP List which has over 4000 items. I found Paul's method of using flow to collect data very useful.

https://www.tachytelic.net/2020/04/many-ways-get-sharepoint-items-power-automate/

 

His method works very well, however I am having difficulties understanding how to use complex fields which I have in my list. It works for string and integer fields, straight forward but for people picker, choice and lookup I have no idea.

 

Below in the uri I have so far entered only string and integer, but please can someone help me with the rest.

 

Thank you,

Mo

Mo_Islam_0-1605800427180.png

These are all my fields I need:

Mo_Islam_1-1605800615436.png

 

--> Also, does anyone know how to generate JSON schema? I need it for the final part of this flow:

 

{
    "type": "array",
    "items": {
        "type": "object",
        "properties": {
            "Id": {
                "type": "integer"
            },
            "Title": {
                "type": "string"
            },
            "LessonNo": {
                "type": "integer"
            },
            "EventStart": {
                "type": "string"
            },
            "EventEnd": {
                "type": "string"
            },
            "CourseTeacher": {
                "type": "string"
            },
            "TeacherName": {
                "type": "string"
            },
            "Description": {
                "type": "string"
            },
            "Location": {
                "type": "string"
            }
        },
        "required": [
            "Id",
            "Title",
            "LessonNo",
            "EventStart",
            "EventEnd",
            "CourseTeacher",
            "TeacherName",
            "Description",
            "Location"
        ]
    }
}

 



 

1 ACCEPTED SOLUTION

Accepted Solutions
Paulie78
Super User
Super User

@Pstork1 there absolutely is another method, which is why I wrote the article above (which many people have implemented successfully). That article describes a method in which a query which originally took over two minutes was reduced to under 10 seconds. To collect 17,000 rows of data, and people have used it with many more than that.

 

Here you can see it running live:

https://www.tachytelic.net/wp-content/uploads/flowSharepointItemsDemo/jQuery-Grid-From-Power-Automat...

17,000 rows from SharePoint in no time at all.

 

@Mo_Islam you don't need to worry about generating the schema manually.

Just run the flow then check the run history, get a sample of your compiled data and then use the "Generate from Sample" tool to generate the schema for you, do you know what I mean?

View solution in original post

11 REPLIES 11
Pstork1
Multi Super User
Multi Super User

If you use the Pagination settings available in the Get Items action you don't need to resort to retrieving the results in two halves.  You can get the entire set of responses in one action.  The flow will automatically retrieve the items in pages until all the items are retrieved.  Then you can use a Select action to get only the columns you need.  This article explains how to turn on pagination.  I've used this for lists that are over 5,000 items without a problem.

https://alextofan.com/2019/08/22/how-to-get-more-than-5000-item-from-sharepoint-online-in-flow/ 



-------------------------------------------------------------------------
If I have answered your question, please mark your post as Solved.
If you like my response, please give it a Thumbs Up.

Thank you for your advice @Pstork1  Can I ask what is the response time? I need to retrieve the data in no more than 10 seconds at most.

 

Also, is there any way that I can limit the number of rows pulled based on the date selected?

e.g. If the date on my screen is 20/Nov/2020, I only want to bring data 1 month back and 1 month ahead, therefore from 20 Oct to 20 Dec.

If you can't get the response time down to what you want using pagination then there is no other method that will do it faster.

 

In terms of filtering you can use an OData query to filter the records.  Here's a brief article that discusses what an odata query for Get items looks like.



-------------------------------------------------------------------------
If I have answered your question, please mark your post as Solved.
If you like my response, please give it a Thumbs Up.
Paulie78
Super User
Super User

@Pstork1 there absolutely is another method, which is why I wrote the article above (which many people have implemented successfully). That article describes a method in which a query which originally took over two minutes was reduced to under 10 seconds. To collect 17,000 rows of data, and people have used it with many more than that.

 

Here you can see it running live:

https://www.tachytelic.net/wp-content/uploads/flowSharepointItemsDemo/jQuery-Grid-From-Power-Automat...

17,000 rows from SharePoint in no time at all.

 

@Mo_Islam you don't need to worry about generating the schema manually.

Just run the flow then check the run history, get a sample of your compiled data and then use the "Generate from Sample" tool to generate the schema for you, do you know what I mean?

I understand the other method, but using the pagination settings properly will provide as good or better performance in my experience than the method you are suggesting.



-------------------------------------------------------------------------
If I have answered your question, please mark your post as Solved.
If you like my response, please give it a Thumbs Up.

It simply won't, it wouldn't even be close. But ok 😁

Thank you @Paulie78 

How can I include the complicated field types as part of the uri as described above?

Also, is there a reason why you have 'ID' and 'Id'.

 

Many thanks

 

Paulie78
Super User
Super User

Hi @Mo_Islam 

 

I am not sure I understand what you mean by complicated field types, can you explain?

Hi @Paulie78 

Basically, in the part where you select the fields you want under the URI, because I have choice and lookup as well, i get an error. So far only if I choose the string and integer fields, it works but I need to also collect fields of other types.

Mo_Islam_0-1605921905231.png

 

 

Paulie78
Super User
Super User

I don't know how to do that via the API, sorry, I would assume you can still use the internal field name. But I guess you tried that. 

Thank you for prompt response @Paulie78 

Also, I downloaded your v3, as my list is still under 5000. I edited everything and when i check the flow results it shows the correct data, however in the PowerApps when I check the collection, it only has one item called 'Value' and not the actual table results. Any ideas?

 

Mo_Islam_0-1605923016088.png

 

Helpful resources

Announcements

Power Platform Connections - Episode 7 | March 30, 2023

Episode Seven of Power Platform Connections sees David Warner and Hugo Bernier talk to Microsoft MVP Dian Taylor, alongside the latest news, product reviews, and community blogs.     Use the hashtag #PowerPlatformConnects on social media for a chance to have your work featured on the show!      Show schedule in this episode:    0:00 Cold Open 00:30 Show Intro 01:02 Dian Taylor Interview 18:03 Blogs & Articles 26:55 Outro & Bloopers    Check out the blogs and articles featured in this week’s episode:    https://francomusso.com/create-a-drag-and-drop-experience-to-upload-case-attachments @crmbizcoach https://www.youtube.com/watch?v=G3522H834Ro​/  @pranavkhuranauk https://github.com/pnp/powerapps-designtoolkit/tree/main/materialdesign%20components @MMe2K​ https://2die4it.com/2023/03/27/populate-a-dynamic-microsoft-word-template-in-power-automate-flow/ @StefanS365 https://d365goddess.com/viva-sales-administrator-settings/ @D365Goddess https://marketplace.visualstudio.com/items?itemName=megel.mme2k-powerapps-helper#Visualize_Dataverse_Environments @MMe2K    Action requested:  Feel free to provide feedback on how we can make our community more inclusive and diverse.    This episode premiered live on our YouTube at 12pm PST on Thursday 30th March 2023.    Video series available at Power Platform Community YouTube channel.    Upcoming events:  Business Applications Launch – April 4th – Free and Virtual! M365 Conference - May 1-5th - Las Vegas Power Apps Developers Summit – May 19-20th - London European Power Platform conference – Jun. 20-22nd - Dublin Microsoft Power Platform Conference – Oct. 3-5th - Las Vegas    Join our Communities:  Power Apps Community Power Automate Community Power Virtual Agents Community Power Pages Community    If you’d like to hear from a specific community member in an upcoming recording and/or have specific questions for the Power Platform Connections team, please let us know. We will do our best to address all your requests or questions.       

Announcing | Super Users - 2023 Season 1

Super Users – 2023 Season 1    We are excited to kick off the Power Users Super User Program for 2023 - Season 1.  The Power Platform Super Users have done an amazing job in keeping the Power Platform communities helpful, accurate and responsive. We would like to send these amazing folks a big THANK YOU for their efforts.      Super User Season 1 | Contributions July 1, 2022 – December 31, 2022  Super User Season 2 | Contributions January 1, 2023 – June 30, 2023    Curious what a Super User is? Super Users are especially active community members who are eager to help others with their community questions. There are 2 Super User seasons in a year, and we monitor the community for new potential Super Users at the end of each season. Super Users are recognized in the community with both a rank name and icon next to their username, and a seasonal badge on their profile.  Power Apps  Power Automate  Power Virtual Agents  Power Pages  Pstork1*  Pstork1*  Pstork1*  OliverRodrigues  BCBuizer  Expiscornovus*  Expiscornovus*  ragavanrajan  AhmedSalih  grantjenkins  renatoromao    Mira_Ghaly*  Mira_Ghaly*      Sundeep_Malik*  Sundeep_Malik*      SudeepGhatakNZ*  SudeepGhatakNZ*      StretchFredrik*  StretchFredrik*      365-Assist*  365-Assist*      cha_cha  ekarim2020      timl  Hardesh15      iAm_ManCat  annajhaveri      SebS  Rhiassuring      LaurensM  abm      TheRobRush  Ankesh_49      WiZey  lbendlin      Nogueira1306  Kaif_Siddique      victorcp  RobElliott      dpoggemann  srduval      SBax  CFernandes      Roverandom  schwibach      Akser  CraigStewart      PowerRanger  MichaelAnnis      subsguts  David_MA      EricRegnier  edgonzales      zmansuri  GeorgiosG      ChrisPiasecki  ryule      AmDev  fchopo      phipps0218  tom_riha      theapurva  takolota     Akash17  momlo     BCLS776  Shuvam-rpa     rampprakash  ScottShearer     Rusk  ChristianAbata     cchannon  Koen5     a33ik  Heartholme     AaronKnox  okeks      Matren   David_MA     Alex_10        Jeff_Thorpe        poweractivate        Ramole        DianaBirkelbach        DavidZoon        AJ_Z        PriyankaGeethik        BrianS        StalinPonnusamy        HamidBee        CNT        Anonymous_Hippo        Anchov        KeithAtherton        alaabitar        Tolu_Victor        KRider        sperry1625        IPC_ahaas      zuurg    rubin_boer   cwebb365   Dorrinda   G1124   Gabibalaban   Manan-Malhotra   jcfDaniel   WarrenBelz   Waegemma   drrickryp   GuidoPreite    If an * is at the end of a user's name this means they are a Multi Super User, in more than one community. Please note this is not the final list, as we are pending a few acceptances.  Once they are received the list will be updated. 

Register now for the Business Applications Launch Event | Tuesday, April 4, 2023

Join us for an in-depth look into the latest updates across Microsoft Dynamics 365 and Microsoft Power Platform that are helping businesses overcome their biggest challenges today.   Find out about new features, capabilities, and best practices for connecting data to deliver exceptional customer experiences, collaborating, and creating using AI-powered capabilities, driving productivity with automation—and building towards future growth with today’s leading technology.   Microsoft leaders and experts will guide you through the full 2023 release wave 1 and how these advancements will help you: Expand visibility, reduce time, and enhance creativity in your departments and teams with unified, AI-powered capabilities.Empower your employees to focus on revenue-generating tasks while automating repetitive tasks.Connect people, data, and processes across your organization with modern collaboration tools.Innovate without limits using the latest in low-code development, including new GPT-powered capabilities.    Click Here to Register Today!    

Check out the new Power Platform Communities Front Door Experience!

We are excited to share the ‘Power Platform Communities Front Door’ experience with you!   Front Door brings together content from all the Power Platform communities into a single place for our community members, customers and low-code, no-code enthusiasts to learn, share and engage with peers, advocates, community program managers and our product team members. There are a host of features and new capabilities now available on Power Platform Communities Front Door to make content more discoverable for all power product community users which includes ForumsUser GroupsEventsCommunity highlightsCommunity by numbersLinks to all communities Users can see top discussions from across all the Power Platform communities and easily navigate to the latest or trending posts for further interaction. Additionally, they can filter to individual products as well.   Users can filter and browse the user group events from all power platform products with feature parity to existing community user group experience and added filtering capabilities.     Users can now explore user groups on the Power Platform Front Door landing page with capability to view all products in Power Platform.      Explore Power Platform Communities Front Door today. Visit Power Platform Community Front door to easily navigate to the different product communities, view a roll up of user groups, events and forums.

Microsoft Power Platform Conference | Registration Open | Oct. 3-5 2023

We are so excited to see you for the Microsoft Power Platform Conference in Las Vegas October 3-5 2023! But first, let's take a look back at some fun moments and the best community in tech from MPPC 2022 in Orlando, Florida.   Featuring guest speakers such as Charles Lamanna, Heather Cook, Julie Strauss, Nirav Shah, Ryan Cunningham, Sangya Singh, Stephen Siciliano, Hugo Bernier and many more.   Register today: https://www.powerplatformconf.com/   

Users online (2,958)