cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
GerardoPlass
Helper II
Helper II

read excel with more than 90000 records in powerapps

Hello everyone, I have a database that they gave me where I have more than 90,000 records and I want to read each one of them, is it possible with such amount of data? what would be the ideal way to be able to read that amount of information without using premium connectors, the question is can you insert that amount of information in sharepoint? what is the best way , your guidance please .

1 ACCEPTED SOLUTION

Accepted Solutions
WarrenBelz
Most Valuable Professional
Most Valuable Professional

@GerardoPlass ,
Adding to @RusselThomas 's detailed comments - I totally agree with his observation I haven't met an app yet that couldn't be built in SharePoint - I have never built in anything else and have never found anything I could not achieve with a bit of thought given to data structure and process.

What I can add is two things - I advise against using Import from Excel as it names the underlying fields Field_1, Field_2  . . .  and will cause you some grief in your Power Apps coding.

The good news is that you can bulk-import Excel once you have the SharePoint list set up with Power Automate. In the Flow below, I imported 75,000 records in batches of 10,000 - it took about 30 minutes for each batch. I put the Excel file in my OneDrive with eight tabs (eight tables) and simply changed the table number each run.

ExcelBulkUpload.png

It is important also that you set your Pagination on the Get Rows to the number you are importing per run.

WarrenBelz_1-1668327501872.png

 

Please click Accept as solution if my post helped you solve your issue. This will help others find it more readily. It also closes the item. If the content was useful in other ways, please consider giving it Thumbs Up.

Visit my blog Practical Power Apps

 

 

 

View solution in original post

6 REPLIES 6
RusselThomas
Community Champion
Community Champion

Hi @GerardoPlass ,

A few ways, depending on what you're wanting to do with it once you've loaded it.

If you just want to look at the data in PowerApps, you can simply embed the table into PowerApps directly and access it from there - but then the data will be static and will reset every time you reload the app.

 

If you want to build an app using this data, and you want to have multiple people access and potentially change the data, then you need a source.

If you can't use SQL as a source, then I'd suggest dataverse, dataverse for teams, or if you have to use SPO you can export it to a list as well - SPO lists can handle up to around 30mil rows - depending on how many columns you have (not that I've ever seen an SPO list with 30m rows mind you) - so sharepoint won't blink, but your app might pass out trying to load that many rows all at once 🙂

 

But if you put it in a list and then connect it to a gallery, you can scroll through the data 100 rows at a time pretty seamlessly inside PowerApps...

If you need to filter it though, you'll need to index your filtering columns in SPO.

 

Can you share more about what you're planning to do with, or how you need to work with the data?

Kind regards,

RT

Thank you very much for your answer

 

I start by showing you my data in an excel , there are approximately 90000 rows

 

GerardoPlass_0-1668255728635.png


First I must show basic information in the app, for example your name, what race is your age etc (this must match between the User().Email and the email column of the excel.

 

GerardoPlass_1-1668256568305.png

 

and it should show like this

 

GerardoPlass_2-1668256675908.png

 

Then the student should be shown some options according to what corresponds to him;  For example, if you select 50-50, it should show you the similar race 1 column and if you select 80-20 0 100-0, it should show you the similar race 2

 

GerardoPlass_3-1668256857740.png

 

 

GerardoPlass_4-1668257102197.png

after selecting this radio, the similar race that matches it appears and then this has to be sent to a sharepoint list to evaluate results. (and I need to attach files too)

 

 

 

 

 

 

 

Drrickryp
Super User
Super User

@GerardoPlass 

Given the operators and functions you want to use on your dataset, LookUp(), Filter(), Sort(), SortByColumns() and "=", SharePoint will work for you.  However, if you need to use "not", "in", "<>" or Search(), as these are not delegatable in SharePoint, then you will need to select either Sql or Dataverse,  I suggest that you watch a short video showing how to do it. https://www.youtube.com/watch?v=IvTiAanB3Cs before you decide which connector to use since it will be a tedious process to load 90,000 items from Excel to SharePoint.  

 

WarrenBelz
Most Valuable Professional
Most Valuable Professional

@GerardoPlass ,

Excel is not a good data source for queries with that size of records. If you have access to SharePoint, I would recommend you switch to that.

I have a blog on SharePoint Delegation that also may assist here.

RusselThomas
Community Champion
Community Champion

Hi @GerardoPlass ,

If that image has actual people's names and email addresses in it - I would suggest you blank it out or depersonalise it before posting it on a public forum, just to be safe and respect people's personal information. 

The important info is the column names, if anyone wants to offer you some usable formulas, and for data you can always just show one row of 'dummy' data so people can see the data types. 

 

Dataverse for teams, dataverse or SQL are always going to be better choices for data sources, but if you only have access to SharePoint then it's not too hard to get it in there - just note that no matter what source you use you will always have data row limits on your queries and as @Drrickryp said, you need to know what's delegable and what isn't between the data sources (Sharepoint, Dataverse, SQL)

That said, I haven't met an app yet that couldn't be built in SharePoint, you just have to think carefully about your indexing and how you query the data - but there are times when it may become really tedious to work around delegable queries, so if it becomes impossible then you'll have shift to another datasource.

 

Getting it into SharePoint.

You can Import from Excel when creating a list in SharePoint, or Export directly from Excel into a SharePoint site.  I haven't imported Excel from the SharePoint side for some time, as it always used to be much slower than Exporting to SharePoint from Excel for some reason - maybe it's better these days, but I always still export from Excel.  The only annoying thing with this process is that the default view for the list always ends up "Edit in Grid view" for some reason - but it's hardly a massive issue so I still use it.

 

  • If you want to import from SharePoint, it's pretty straight forward - Home page, New List, From Excel.
  • If you want to try export from Excel and you're unfamiliar with the export function in Excel, make sure the data is a table in Excel, select all of it and hit ctrl+T to make sure it's a table - if it isn't it'll prompt you to make it one, if so just check your headers are correct.  Then go to the Table Design tab (it only shows up for Tables) and hit "Export" and select "Export table to SharePoint list" and follow the prompts.

90k rows will still take a few minutes to create either direction, so be patient.  

Once it's done, immediately go the List Settings, then find "Indexed Columns" and open that.  Index simple text columns that are unique first - like the email address, assuming a person won't appear more than once in this column, and then one or two other (preferably unique) columns that you plan to filter on.

Note - the Index will only run at the next cycle which runs around every 15 minutes for SPO.

 

Then, connect from Power Apps and start playing 🙂

Hope this helps,

RT

WarrenBelz
Most Valuable Professional
Most Valuable Professional

@GerardoPlass ,
Adding to @RusselThomas 's detailed comments - I totally agree with his observation I haven't met an app yet that couldn't be built in SharePoint - I have never built in anything else and have never found anything I could not achieve with a bit of thought given to data structure and process.

What I can add is two things - I advise against using Import from Excel as it names the underlying fields Field_1, Field_2  . . .  and will cause you some grief in your Power Apps coding.

The good news is that you can bulk-import Excel once you have the SharePoint list set up with Power Automate. In the Flow below, I imported 75,000 records in batches of 10,000 - it took about 30 minutes for each batch. I put the Excel file in my OneDrive with eight tabs (eight tables) and simply changed the table number each run.

ExcelBulkUpload.png

It is important also that you set your Pagination on the Get Rows to the number you are importing per run.

WarrenBelz_1-1668327501872.png

 

Please click Accept as solution if my post helped you solve your issue. This will help others find it more readily. It also closes the item. If the content was useful in other ways, please consider giving it Thumbs Up.

Visit my blog Practical Power Apps

 

 

 

Helpful resources

Announcements

Update! June 13th, Community Ambassador Call for User Group Leaders and Super Users

Calling all Super Users & User Group Leaders     UPDATE:  We just wrapped up June's Community Ambassador monthly calls for Super Users and User Group Leaders. We had a fantastic call with lots of engagement. We are excited to share some highlights with you!    Big THANK YOU to our special guest Thomas Verhasselt, from the Copilot Studio Product Team for sharing how to use Power Platform Templates to achieve next generation growth.     A few key takeaways: Copilot Studio Cookbook Challenge:  Week 1 results are posted, Keep up the great work!Summer of Solutions:  Starting on Monday, June 17th. Just by providing solutions in the community, you can be entered to win tickets to Power Platform Community Conference.Super User Season 2: Coming SoonAll communities moving to the new platform end of July We also honored two different community members during the call, Mohamed Amine Mahmoudi and Markus Franz! We are thankful for both leaders' contributions and engagement with their respective communities. 🎉   Be sure to mark your calendars and register for the meeting on July 11th and stay up to date on all of the changes that are coming. Check out the Super User Forum boards for details.   We're excited to connect with you and continue building a stronger community together.   See you at the call!  

Win free tickets to the Power Platform Conference | Summer of Solutions

We are excited to announce the Summer of Solutions Challenge!    This challenge is kicking off on Monday, June 17th and will run for (4) weeks.  The challenge is open to all Power Platform (Power Apps, Power Automate, Copilot Studio & Power Pages) community members. We invite you to participate in a quest to provide solutions in the Forums to as many questions as you can. Answers can be provided in all the communities.    Entry Period: This Challenge will consist of four weekly Entry Periods as follows (each an “Entry Period”)   - 12:00 a.m. PT on June 17, 2024 – 11:59 p.m. PT on June 23, 2024 - 12:00 a.m. PT on June 24, 2024 – 11:59 p.m. PT on June 30, 2024 - 12:00 a.m. PT on July 1, 2024 – 11:59 p.m. PT on July 7, 2024 - 12:00 a.m. PT on July 8, 2024 – 11:59 p.m. PT on July 14, 2024   Entries will be eligible for the Entry Period in which they are received and will not carryover to subsequent weekly entry periods.  You must enter into each weekly Entry Period separately.   How to Enter: We invite you to participate in a quest to provide "Accepted Solutions" to as many questions as you can. Answers can be provided in all the communities. Users must provide a solution which can be an “Accepted Solution” in the Forums in all of the communities and there are no limits to the number of “Accepted Solutions” that a member can provide for entries in this challenge, but each entry must be substantially unique and different.    Winner Selection and Prizes: At the end of each week, we will list the top ten (10) Community users which will consist of: 5 Community Members & 5 Super Users and they will advance to the final drawing. We will post each week in the News & Announcements the top 10 Solution providers.  At the end of the challenge, we will add all of the top 10 weekly names and enter them into a random drawing.  Then we will randomly select ten (10) winners (5 Community Members & 5 Super Users) from among all eligible entrants received across all weekly Entry Periods to receive the prize listed below. If a winner declines, we will draw again at random for the next winner.  A user will only be able to win once overall. If they are drawn multiple times, another user will be drawn at random.  Individuals will be contacted before the announcement with the opportunity to claim or deny the prize.  Once all of the winners have been notified, we will post in the News & Announcements of each community with the list of winners.   Each winner will receive one (1) Pass to the Power Platform Conference in Las Vegas, Sep. 18-20, 2024 ($1800 value). NOTE: Prize is for conference attendance only and any other costs such as airfare, lodging, transportation, and food are the sole responsibility of the winner. Tickets are not transferable to any other party or to next year’s event.   ** PLEASE SEE THE ATTACHED RULES for this CHALLENGE**

Copilot Cookbook Challenge | Week 2 Results | Win Tickets to the Power Platform Conference

We are excited to announce the "The Copilot Cookbook Community Challenge is a great way to showcase your creativity and connect with others. Plus, you could win tickets to the Power Platform Community Conference in Las Vegas in September 2024 as an amazing bonus.   Two ways to enter: 1. Copilot Studio Cookbook Gallery: https://aka.ms/CS_Copilot_Cookbook_Challenge 2. Power Apps Copilot Cookbook Gallery: https://aka.ms/PA_Copilot_Cookbook_Challenge   There will be 5 chances to qualify for the final drawing: Early Bird Entries: March 1 - June 2Week 1: June 3 - June 9Week 2: June 10 - June 16Week 3: June 17 - June 23Week 4: June 24 - June 30     At the end of each week, we will draw 5 random names from every user who has posted a qualifying Copilot Studio template, sample or demo in the Copilot Studio Cookbook or a qualifying Power Apps Copilot sample or demo in the Power Apps Copilot Cookbook. Users who are not drawn in a given week will be added to the pool for the next week. Users can qualify more than once, but no more than once per week. Four winners will be drawn at random from the total qualifying entrants. If a winner declines, we will draw again at random for the next winner.  A user will only be able to win once. If they are drawn multiple times, another user will be drawn at random. Prizes:  One Pass to the Power Platform Conference in Las Vegas, Sep. 18-20, 2024 ($1800 value, does not include travel, lodging, or any other expenses) Winners are also eligible to do a 10-minute presentation of their demo or solution in a community solutions showcase at the event. To qualify for the drawing, templates, samples or demos must be related to Copilot Studio or a Copilot feature of Power Apps, Power Automate, or Power Pages, and must demonstrate or solve a complete unique and useful business or technical problem. Power Automate and Power Pagers posts should be added to the Power Apps Cookbook. Final determination of qualifying entries is at the sole discretion of Microsoft. Weekly updates and the Final random winners will be posted in the News & Announcements section in the communities on July 29th, 2024. Did you submit entries early?  Early Bird Entries March 1 - June 2:  If you posted something in the "early bird" time frame complete this form: https://aka.ms/Copilot_Challenge_EarlyBirds if you would like to be entered in the challenge.   Week 1 Results:  Congratulations to the Week 1 qualifiers, you are being entered in the random drawing that will take place at the end of the challenge. Copilot Cookbook Gallery:Power Apps Cookbook Gallery:1.  @Mathieu_Paris 1.   @SpongYe 2.  @Dhanush 2.   @Deenuji 3.  n/a3.   @Nived_Nambiar  4.  n/a4.   @ManishSolanki 5.  n/a5.    n/a   Week 2 Results:  Congratulations to the Week 2 qualifiers, you are being entered in the random drawing that will take place at the end of the challenge. Copilot Cookbook Gallery:Power Apps Cookbook Gallery:1. Kasun_Pathirana1. ManishSolanki2. cloudatica2. madlad3. n/a3. SpongYe4. n/a4. n/a5. n/a5. n/a

Celebrating the June Super User of the Month: Markus Franz

Markus Franz is a phenomenal contributor to the Power Apps Community. Super Users like Markus inspire others through their example, encouragement, and active participation.    The Why: "I do this to help others achieve what they are trying to do. As a total beginner back then without IT background I know how overwhelming things can be, so I decided to jump in and help others. I also do this to keep progressing and learning myself." Thank you, Markus Franz, for your outstanding work! Keep inspiring others and making a difference in the community! 🎉  Keep up the fantastic work! 👏👏   Markus Franz | LinkedIn  Power Apps: mmbr1606  

Your Moment to Shine: 2024 PPCC’s Got Power Awards Show

For the third year, we invite you, our talented community members, to participate in the grand 2024 Power Platform Community Conference's Got Power Awards. This event is your opportunity to showcase solutions that make a significant business impact, highlight extensive use of Power Platform products, demonstrate good governance, or tell an inspirational story. Share your success stories, inspire your peers, and show off some hidden talents.  This is your time to shine and bring your creations into the spotlight!  Make your mark, inspire others and leave a lasting impression. Sign up today for a chance to showcase your solution and win the coveted 2024 PPCC’s Got Power Award. This year we have three categories for you to participate in: Technical Solution Demo, Storytelling, and Hidden Talent.      The Technical solution demo category showcases your applications, automated workflows, copilot agentic experiences, web pages, AI capabilities, dashboards, and/or more. We want to see your most impactful Power Platform solutions!  The Storytelling category is where you can share your inspiring story, and the Hidden Talent category is where your talents (such as singing, dancing, jump roping, etc.) can shine! Submission Details:  Fill out the submission form https://aka.ms/PPCCGotPowerSignup by July 12th with details and a 2–5-minute video showcasing your Solution impact. (Please let us know you're coming to PPCC, too!)After review by a panel of Microsoft judges, the top storytellers will be invited to present a virtual demo presentation to the judges during early August. You’ll be notified soon after if you have been selected as a finalist to share your story live at PPCC’s Got Power!  The live show will feature the solution demos and storytelling talents of the top contestants, winner announcements, and the opportunity to network with your community.  It's not just a showcase for technical talent and storytelling showmanship, show it's a golden opportunity to make connections and celebrate our Community together! Let's make this a memorable event! See you there!   Mark your calendars! Date and Time: Thursday, Sept 19th Location: PPCC24 at the MGM Grand, Las Vegas, NV 

Tuesday Tip | Accepting Solutions

It's time for another TUESDAY TIPS, your weekly connection with the most insightful tips and tricks that empower both newcomers and veterans in the Power Platform Community! Every Tuesday, we bring you a curated selection of the finest advice, distilled from the resources and tools in the Community. Whether you’re a seasoned member or just getting started, Tuesday Tips are the perfect compass guiding you across the dynamic landscape of the Power Platform Community.   To enhance our collaborative environment, it's important to acknowledge when your question has been answered satisfactorily. Here's a quick guide on how to accept a solution to your questions: Find the Helpful Reply: Navigate to the reply that has effectively answered your question.Accept as Solution: Look for the "Accept as Solution" button or link, usually located at the bottom of the reply.Confirm Your Selection: Clicking this button may prompt you for confirmation. Go ahead and confirm that this is indeed the solution.Acknowledgment: Once accepted, the reply will be highlighted, and the original post will be marked as "Solved". This helps other community members find the same solution quickly. By marking a reply as an accepted solution, you not only thank the person who helped you but also make it easier for others with similar questions to find answers. Let's continue to support each other by recognizing helpful contributions. 

Top Solution Authors
Top Kudoed Authors
Users online (6,436)