Hello everyone, I have a database that they gave me where I have more than 90,000 records and I want to read each one of them, is it possible with such amount of data? what would be the ideal way to be able to read that amount of information without using premium connectors, the question is can you insert that amount of information in sharepoint? what is the best way , your guidance please .
Solved! Go to Solution.
@GerardoPlass ,
Adding to @RusselThomas 's detailed comments - I totally agree with his observation I haven't met an app yet that couldn't be built in SharePoint - I have never built in anything else and have never found anything I could not achieve with a bit of thought given to data structure and process.
What I can add is two things - I advise against using Import from Excel as it names the underlying fields Field_1, Field_2 . . . and will cause you some grief in your Power Apps coding.
The good news is that you can bulk-import Excel once you have the SharePoint list set up with Power Automate. In the Flow below, I imported 75,000 records in batches of 10,000 - it took about 30 minutes for each batch. I put the Excel file in my OneDrive with eight tabs (eight tables) and simply changed the table number each run.
It is important also that you set your Pagination on the Get Rows to the number you are importing per run.
Please click Accept as solution if my post helped you solve your issue. This will help others find it more readily. It also closes the item. If the content was useful in other ways, please consider giving it Thumbs Up.
Visit my blog Practical Power Apps
Hi @GerardoPlass ,
A few ways, depending on what you're wanting to do with it once you've loaded it.
If you just want to look at the data in PowerApps, you can simply embed the table into PowerApps directly and access it from there - but then the data will be static and will reset every time you reload the app.
If you want to build an app using this data, and you want to have multiple people access and potentially change the data, then you need a source.
If you can't use SQL as a source, then I'd suggest dataverse, dataverse for teams, or if you have to use SPO you can export it to a list as well - SPO lists can handle up to around 30mil rows - depending on how many columns you have (not that I've ever seen an SPO list with 30m rows mind you) - so sharepoint won't blink, but your app might pass out trying to load that many rows all at once 🙂
But if you put it in a list and then connect it to a gallery, you can scroll through the data 100 rows at a time pretty seamlessly inside PowerApps...
If you need to filter it though, you'll need to index your filtering columns in SPO.
Can you share more about what you're planning to do with, or how you need to work with the data?
Kind regards,
RT
Thank you very much for your answer
I start by showing you my data in an excel , there are approximately 90000 rows
First I must show basic information in the app, for example your name, what race is your age etc (this must match between the User().Email and the email column of the excel.
and it should show like this
Then the student should be shown some options according to what corresponds to him; For example, if you select 50-50, it should show you the similar race 1 column and if you select 80-20 0 100-0, it should show you the similar race 2
after selecting this radio, the similar race that matches it appears and then this has to be sent to a sharepoint list to evaluate results. (and I need to attach files too)
Given the operators and functions you want to use on your dataset, LookUp(), Filter(), Sort(), SortByColumns() and "=", SharePoint will work for you. However, if you need to use "not", "in", "<>" or Search(), as these are not delegatable in SharePoint, then you will need to select either Sql or Dataverse, I suggest that you watch a short video showing how to do it. https://www.youtube.com/watch?v=IvTiAanB3Cs before you decide which connector to use since it will be a tedious process to load 90,000 items from Excel to SharePoint.
Excel is not a good data source for queries with that size of records. If you have access to SharePoint, I would recommend you switch to that.
I have a blog on SharePoint Delegation that also may assist here.
Hi @GerardoPlass ,
If that image has actual people's names and email addresses in it - I would suggest you blank it out or depersonalise it before posting it on a public forum, just to be safe and respect people's personal information.
The important info is the column names, if anyone wants to offer you some usable formulas, and for data you can always just show one row of 'dummy' data so people can see the data types.
Dataverse for teams, dataverse or SQL are always going to be better choices for data sources, but if you only have access to SharePoint then it's not too hard to get it in there - just note that no matter what source you use you will always have data row limits on your queries and as @Drrickryp said, you need to know what's delegable and what isn't between the data sources (Sharepoint, Dataverse, SQL).
That said, I haven't met an app yet that couldn't be built in SharePoint, you just have to think carefully about your indexing and how you query the data - but there are times when it may become really tedious to work around delegable queries, so if it becomes impossible then you'll have shift to another datasource.
Getting it into SharePoint.
You can Import from Excel when creating a list in SharePoint, or Export directly from Excel into a SharePoint site. I haven't imported Excel from the SharePoint side for some time, as it always used to be much slower than Exporting to SharePoint from Excel for some reason - maybe it's better these days, but I always still export from Excel. The only annoying thing with this process is that the default view for the list always ends up "Edit in Grid view" for some reason - but it's hardly a massive issue so I still use it.
90k rows will still take a few minutes to create either direction, so be patient.
Once it's done, immediately go the List Settings, then find "Indexed Columns" and open that. Index simple text columns that are unique first - like the email address, assuming a person won't appear more than once in this column, and then one or two other (preferably unique) columns that you plan to filter on.
Note - the Index will only run at the next cycle which runs around every 15 minutes for SPO.
Then, connect from Power Apps and start playing 🙂
Hope this helps,
RT
@GerardoPlass ,
Adding to @RusselThomas 's detailed comments - I totally agree with his observation I haven't met an app yet that couldn't be built in SharePoint - I have never built in anything else and have never found anything I could not achieve with a bit of thought given to data structure and process.
What I can add is two things - I advise against using Import from Excel as it names the underlying fields Field_1, Field_2 . . . and will cause you some grief in your Power Apps coding.
The good news is that you can bulk-import Excel once you have the SharePoint list set up with Power Automate. In the Flow below, I imported 75,000 records in batches of 10,000 - it took about 30 minutes for each batch. I put the Excel file in my OneDrive with eight tabs (eight tables) and simply changed the table number each run.
It is important also that you set your Pagination on the Get Rows to the number you are importing per run.
Please click Accept as solution if my post helped you solve your issue. This will help others find it more readily. It also closes the item. If the content was useful in other ways, please consider giving it Thumbs Up.
Visit my blog Practical Power Apps
A new month means it's time to celebrate and welcome the new user groups that have joined our community. We are excited to announce that we have more than 8 New Groups, which is no surprise after the amazing Microsoft Power Platform Conference. This month, we are breaking them out by the different community categories. If your group is listed here, give this post a kudo so we can celebrate with you! Don't forget to take a look at the many events happening near you or virtually! It's a great time of year to connect and engage with User Groups both locally and online. Please Welcome Our NEW User Groups Power Platform: PowerIT User Group: Nottingham Power Platform User Group: Bergen Power Platform User Group: Danmark Nashville Power Platform User Group Microsoft Ambassador Program y Mujer Latina Technolochicas NCWIT Community Copilot Studio: Copilot User Group Italia Dynamics365: Dynamics User Group AdriaticDynamic 365 Azerbaijan December User Group Events 01 Dec 2023 Aprendiendo Desarrollo web, creando mi primer power app y power page. 01 Dec 2023 Q4 Hybrid Philadelphia Dynamics 365 & Power Platform User Group Meeting05 Dec 2023APAC Dynamics 365 FastTrack Bootcamp - BI and Analytics07 Dec 2023Bay Area Power Platform Meetup: Learn, Share, and Connect07 Dec 2023Indiana D365/AX December User Group Meeting07 Dec 2023Dynamics User Group Meeting: Houston09 Dec 2023 December '23 - NEW Power Apps User Group Meeting - Online 12 Dec 2023December Cleveland Power Platform User Group Meeting12 Dec 2023 RW2 Data Stewardship Open Forum Discussion 13 Dec 2023 Black Country Power Platform User Group - December 2023 - West Midlands
Our ongoing BACK TO BASICS: TUESDAY TIP series dedicated to helping both new members and seasoned veterans of our community learn and grow reached a milestone ten posts! We're excited to present this "one stop" post for each of our #TuesdayTips, making it easier to find what you're looking for and help you understand the community: from ranking and badges to profile avatars, from being a Super User to blogging in the community, and so much more. Thank you for your incredible support for this series--we are so glad it was able to help so many of you navigate your community experience. Back to Basics Tuesday Tip #1: All About Your Community Account Find out the basics of your community account. Whether it's changing your username, updating an email address, understanding GDPR, or customizing your profile, this is the place to begin. ○ Power Apps ○ Power Automate ○ Power Pages ○ Copilot Studio Back to Basics Tuesday Tip #2: All About Community Ranks Have you ever wondered how your fellow community members earn the different ranks available? What is the difference between an Advocate and a Helper, a Solution Sage and a Community Champion? In this #TuesdayTip, we share the secrets and tips to help YOU keep your ranking growing--and why it's so important to our communities. ○ Power Apps ○ Power Automate ○ Power Pages ○ Copilot Studio Back to Basics Tuesday Tip #3: Contributing to the Community If you need to understand how subscriptions or notifications work, how to use search to find the answers you're looking for, or editing your posts, this is the place to start. With these handy tips, you'll find what you're looking for, ask some great questions, and format your posts perfectly! ○ Power Apps ○ Power Automate ○ Power Pages ○ Copilot Studio Back to Basics Tuesday Tip #4: What is a Super User? Have you ever been exploring the community and come across a user with this unique icon next to their name? It means you have found the actual, real-life superheroes of the Power Platform Community! Super Users are our heroes because of the way they are consistently helpful with everything from solutions to flagging spam, offering insight on the community, and so much more! Find out more in this #TuesdayTip. ○ Power Apps ○ Power Automate ○ Power Pages ○ Copilot Studio Back to Basics Tuesday Tip #5: How to Become a Community Blog Author We want YOU to be part of the community blog! Sharing your knowledge of Power Platform is an essential part of our community! By sharing what you know and have learned with the community in the Power Platform in the community blog, you help us create a more engaged and informed community, better equipped to tackle complex challenges. To get started with blogging across the Power Platform communities, please visit the following links. ○ Power Apps ○ Power Automate ○ Power Pages ○ Copilot Studio Back to Basics Tuesday Tip #6 All About Community User Groups Being part of, starting, or leading a User Group can have many great benefits for our community members who want to learn, share, and connect with others who are interested in the Microsoft Power Platform and the low-code revolution. Don't wait. Get involved with (or maybe even start) a User Group today--just follow the tips below to get started. ○ Power Apps ○ Power Automate ○ Power Pages ○ Copilot Studio Back to Basics Tuesday Tip #7: Resources for User Groups Once you've launched your Community User Group, we are excited to have many resources available that can help you lead, engage, and grow your User Group! Whether it's access to the Microsoft Community Tenant for User Groups, help with finding speakers for your User Group meetings (both local and virtual speakers), and even finding spaces to have your meetings in--check out this #TuesdayTip to get what you need! ○ Power Apps ○ Power Automate ○ Power Pages ○ Copilot Studio Back to Basics Tuesday Tip #8: All About Subscriptions and Notifications Keeping track of what you want to know and how you want to find out about it may seem confusing. This #TuesdayTip is all about your community profile's subscriptions and notifications settings. Check out the links below for clear directions and how-to's. ○ Power Apps ○ Power Automate ○ Power Pages ○ Copilot Studio Back to Basics Tuesday Tip #9: All About the Community Galleries Have you checked out the library of content in our galleries? Whether you're looking for the latest info on an upcoming event, a helpful webinar, or tips and tricks from some of our most experienced community members, our galleries are full of the latest and greatest video content for the Power Platform communities. Find out more by following the links below. ○ Power Apps ○ Power Automate ○ Power Pages ○ Copilot Studio Back to Basics Tuesday Tip #10: Community Support Whether you're a seasoned community veteran or just getting started, you may need a bit of help from time to time! If you need to share feedback with the Community Engagement team about the community or are looking for ways we can assist you with user groups, events, or something else, Community Support is the place to start. Community Support is part of every one of our communities, accessible to all our community members, so find out what it means for your community with our last #TuesdayTip. ○ Power Apps ○ Power Automate ○ Power Pages ○ Copilot Studio Thank you for your support for our #TuesdayTip series. We look forward to bringing you more tips and tricks to help make your community experience the best it can be!
Welcome to our November Newsletter, where we highlight the latest news, product releases, upcoming events, and the amazing work of our outstanding Community members. If you're new to the Community, please make sure to follow the latest News & Announcements and check out the Community on LinkedIn as well! It's the best way to stay up-to-date with all the news from across the Power Platform and beyond. This month's highlights:- - Our most active community members- Microsoft Power Up Program- Microsoft Community Days website - The latest blogs and more COMMUNITY HIGHLIGHTS Check out the most active community members of the last month. These hardworking members are posting regularly, answering questions, kudos, and providing top solutions in their communities. We are so thankful for each of you--keep up the great work! If you hope to see your name here next month, just get active! FLMikePstork1Nived_NambiarWarrenBelzSprongYeManishSolankiLaurensMwskinnermlcAgniusExpiscornovuscreativeopinion KatieAUinzil2kHaressh2728hafizsultan242douicmccaughanwoLucas001domliu Power Up Program Click the image below to discover more about the amazing Microsoft Power Up Program, as Reem Omar, Abbas Godhrawala, Chahine Atallah, Ruby Ruiz Brown, Juan Francisco Sánchez Enciso, Joscelyne Andrade Arévalo, Eric G. and Paulina Pałczyńska share how non-tech professionals can successfully advance into a new career path using Microsoft #PowerPlatform. To find out more about this amazing initiative, click here to apply for the program and reboot your journey into low-code app development today! Community Days - Event Website Have you checked out the Community Days website yet? Dedicated to the volunteer community organizers around the world, Community Days is the perfect place to find an event near you or add an event for wider exposure. Many thanks to Thomas Daly, Sharon Weaver, Sedat Tum, Jonathan Weaver, Manpreet Singh, David Leveille, Jason Rivera, Mike Maadarani, Rob Windsor and the team for all their hard work. Anyone can host a Community Day on any topic relevant to our industry, just click the image below to find out more. EVENT NEWS Power Platform French Summit - Paris/Virtual - 6-7th Dec It's not long now until the Power Platform French Summit, which takes place both virtually and in-person at the Microsoft France conference center in Paris on 6-7th December 2023. If you can't make it in-person, all sessions will also be broadcast on virtual networks for better distribution and accessibility. There's a fantastic array of speakers, including Jérémy LAPLAINE, Amira Beldjilali, Rémi Chambard, Erika Beaumier, Makenson Frena, Assia Boutera, Elliott Pierret, Clothilde Facon, Gilles Pommier, Marie Aubert, Antoine Herbosa, Chloé Moreau, Raphaël Senis, Rym Ben Hamida, Loïc Cimon, Joséphine Salafia, David Zoonekyndt, Aïcha Charpentier, Henry Jammes, Milene Rochard, Mehdi EL YASSIR, and many more. Click the image below for more information. LATEST COMMUNITY BLOG ARTICLES Power Apps Community Blog Power Automate Community Blog Copilot Community Blog Power Pages Community Blog
In the bustling world of technology, two dynamic leaders, Geetha Sivasailam and Ben McMann, have been at the forefront, steering the ship of the Dallas Fort Worth Power Platform User Group since its inception in February 2019. As Practice Lead (Power Platform | Fusion Dev) at Lantern, Geetha brings a wealth of consulting experience, while Ben, a key member of the Studio Leadership team at Lantern, specializes in crafting strategies that leverage Microsoft digital technologies to transform business models. Empowering Through Community Leadership Geetha and Ben's journey as user group leaders began with a simple yet powerful goal: to create a space where individuals across the DFW area could connect, grow their skills, and add value to their businesses through the Power Platform. The platform, known for its versatility, allows users to achieve more with less code and foster creativity. The Power of Community Impact Reflecting on their experiences, Geetha and Ben emphasize the profound impact that community engagement has had on both their professional and personal lives. The Power Platform community, they note, is a wellspring of resources and opportunities, fostering continuous learning, skill enhancement, and networking with industry experts and peers. Favorite Moments and Words of Wisdom The duo's favorite aspect of leading the user group lies in witnessing the transformative projects and innovations community members create with the Power Platform. Their advice to aspiring user group leaders? "Encourage diverse perspectives, maintain an open space for idea-sharing, stay curious, and, most importantly, have fun building a vibrant community." Building Bridges, Breaking Barriers Geetha and Ben encourage others to step into the realm of user group leadership, citing the rewarding experience of creating and nurturing a community of like-minded individuals. They highlight the chance to influence, impact, and positively guide others, fostering connections that extend beyond mere technology discussions. Joining a User Group: A Gateway to Growth The leaders stress the importance of joining a user group, emphasizing exposure to diverse perspectives, solutions, and career growth opportunities within the Power Platform community. "Being part of such a group provides a supportive environment for seeking advice, sharing experiences, and navigating challenges." A Year of Milestones Looking back at the past year, Geetha and Ben express pride in the group's growth and global participation. They recount the enriching experience of meeting members in person at the Microsoft Power Platform conference, showcasing the diverse range of perspectives and guest speakers that enriched the community's overall experience. Continuous Learning on the Leadership Journey As user group leaders, Geetha and Ben recognize the continuous learning curve, blending interpersonal skills, adaptability, and dedication to foster a vibrant community. They highlight the importance of patience, persistence, and flexibility in achieving group goals, noting the significance of listening to the needs and suggestions of group members.They invite all tech enthusiasts to join the Dallas Fort Worth Power Platform User Group, a thriving hub where the power of community propels individuals to new heights in the dynamic realm of technology.
This is the TENTH post in our ongoing series dedicated to helping the amazing members of our community--both new members and seasoned veterans--learn and grow in how to best engage in the community! Each Tuesday, we feature new content that will help you best understand the community--from ranking and badges to profile avatars, from Super Users to blogging in the community. Our hope is that this information will help each of our community members grow in their experience with Power Platform, with the community, and with each other! This Week: All About Community Support Whether you're a seasoned community veteran or just getting started, you may need a bit of help from time to time! If you need to share feedback with the Community Engagement team about the community or are looking for ways we can assist you with user groups, events, or something else, Community Support is the place to start. Community Support is part of every one of our communities, accessible to all our community members. Power Apps: https://powerusers.microsoft.com/t5/Community-Support/ct-p/pa_community_support Power Automate: https://powerusers.microsoft.com/t5/Community-Support/ct-p/mpa_community_support Power Pages: https://powerusers.microsoft.com/t5/Community-Support/ct-p/mpp_community_support Copilot Studio: https://powerusers.microsoft.com/t5/Community-Support/ct-p/pva_community-support Within each community's Community Support page, you'll find three distinct areas, each with a different focus to help you when you need support from us most. Community Accounts & Registration is the go-to source for any and all information related to your account here in the community. It's full of great knowledge base articles that will help you manage your community account and know what steps to take if you wish to close your account. ● Power Apps ● Power Automate ● Power Pages, ● Copilot Studio Using the Community is your source for assistance with everything from Community User Groups to FAQ's and more. If you want to know what kudos are, how badges work, how to level up your User Group or something else, you will probably find the answers here. ● Power Apps ● Power Automate ● Power Pages ● Copilot Studio Community Feedback is where you can share opportunities, concerns, or get information from the Community Engagement team. It's your best place to post a question about an issue you're having in the community, a general question you need answered. Whatever it is, visit Community Feedback to get the answers you need right away. Our team is honored to partner with you and can't wait to help you! ● Power Apps ● Power Automate ● Power Pages ● Copilot Studio
What an amazing event we had this year, as Microsoft showcased the latest advancements in how AI has the potential to reshape how customers, partners and developers strategize the future of work. Check out below some of our handpicked videos and Ignite announcements to see how Microsoft is driving real change for users and businesses across the globe. Video Highlights Click the image below to check out a selection of Ignite 2023 videos, including the "Microsoft Cloud in the era of AI" keynote from Scott Guthrie, Charles Lamanna, Arun Ulag, Sarah Bird, Rani Borkar, Eric Boyd, Erin Chapple, Ali Ghodsi, and Seth Juarez. There's also a great breakdown of the amazing Microsoft Copilot Studio with Omar Aftab, Gary Pretty, and Kendra Springer, plus exciting sessions from Rajesh Jha, Jared Spataro, Ryan Jones, Zohar Raz, and many more. Blog Announcements Microsoft Copilot presents an opportunity to reimagine the way we work—turning natural language into the most powerful productivity tool on the planet. With AI, organizations can unearth value in data across productivity tools like business applications and Microsoft 365. Click the link below to find out more. Check out the latest features in Microsoft Power Apps that will help developers create AI-infused apps faster, give administrators more control over managing thousands of Microsoft Power Platform makers at scale, and deliver better experiences to users around the world. Click the image below to find out more. Click below to discover new ways to orchestrate business processes across your organization with Copilot in Power Automate. With its user-friendly interface that offers hundreds of prebuilt drag-and-drop actions, more customers have been able to benefit from the power of automation. Discover how Microsoft Power Platform and Microsoft Dataverse are activating the strength of your enterprise data using AI, the announcement of “plugins for Microsoft Copilot for Microsoft 365”, plus two new Power Apps creator experiences using Excel and natural language. Click below to find out more about the general availability of Microsoft Fabric and the public preview of Copilot in Microsoft Fabric. With the launch of these next-generation analytics tools, you can empower your data teams to easily scale the demand on your growing business. And for the rest of all the good stuff, click the link below to visit the Microsoft Ignite 2023 "Book of News", with over ONE HUNDRED announcements across infrastructure, data, security, new tools, AI, and everything else in-between!
User | Count |
---|---|
156 | |
77 | |
63 | |
49 | |
48 |