cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
mattyn07
Helper II
Helper II

Question regarding Power Query and CDS

Hi,

 

I have created a Power Query with subsequent dataflow which created the records in CDS fine except for one thing - it created duplicate records.

 

I have selected the "Do Not Delete records not in this query" however it just seems to create duplicate records in my CDS environment which isnt ideal when your Power Query has 9000+ records and your existing CDS entity you are using has 30k records.

 

What I need to occur is:

 

  • Nightly job runs and creates ONLY the new records that have entered that query
  • No duplicate records to be created
  • No existing data to be deleted

 

Is this something I need to amend within my Power Query to only show those with a created on date of today? Or do i need to bring my CDS table into Power Query (if possible) to ensure it knows what records already exist before running its nightly update?

 

Happy to provide more details to assist further.


Thanks

1 ACCEPTED SOLUTION

Accepted Solutions
EricRegnier
Most Valuable Professional
Most Valuable Professional

Hi @mattyn07,

Unfortunately this is a current limitation of Dataflows with the only workaround selecting the setting the "Delete rows that no longer exist". You can vote for this feature below though and hopefully Microsoft will be looking into this for a further release:

https://powerusers.microsoft.com/t5/Power-Apps-Ideas/Upsert-update-capabilities-in-Power-Apps-Datafl...

Cheers

 

View solution in original post

13 REPLIES 13
EricRegnier
Most Valuable Professional
Most Valuable Professional

Hi @mattyn07,

Unfortunately this is a current limitation of Dataflows with the only workaround selecting the setting the "Delete rows that no longer exist". You can vote for this feature below though and hopefully Microsoft will be looking into this for a further release:

https://powerusers.microsoft.com/t5/Power-Apps-Ideas/Upsert-update-capabilities-in-Power-Apps-Datafl...

Cheers

 

HSheild
Community Champion
Community Champion

Hi @mattyn07 

 

You can avoid duplicates by setting up a Key on the entity you are importing into and then setting this as the Alternate Key field on the Field Mapping in the dataflow.  You will then also need to have a unique identifier for each record in your source data and map that to the Key field that you set up on the entity.  This then tells the dataflow not to create a duplicate because a record with that key already exists in CDS.

Annotation 2020-06-19 0747031.png

This will resolve your duplicate issue, however, as @EricRegnier says dataflow is limited and cannot meet your requirement exactly for

  • Nightly job runs and creates ONLY the new records that have entered that query

The dataflow will "modify" all records in CDS that match your source file if they already exist in CDS, regardless of whether the data has changed or not.  However, this may be OK for you?

Hi @HSheild  and @EricRegnier ,

 

Thanks a lot for your feedback on this.

 

I assumed this was the case however I wanted to clarify before I looked at alternate solutions.

 

The key solution was something I was looking at I just need to ensure our Power App isn't reading a flag on our existing contact record. The alternate solution would be to create these records in a data import entity per see and use flow to create the new records however with what I am reading below the created on record will always be the same? What about if I used the "delete records if not in that query" in that seperate entity? Would this result in my created on date being the same date as my data came in via my query? Eg stages dates for each record that came in via the source?

EricRegnier
Most Valuable Professional
Most Valuable Professional

Unless Dataflows supports it now, but from what I've seen even with alternate keys, Dataflows tries to insert records, the only difference is you'll get a duplicate key exception for that row and thus won't be updated. @HSheild are you saying we can now update? Because that would be awesome!

 

@mattyn07, having some kind of staging entity will help with a flow that does the update so will keep your original created on in the "real" entity.

Cheers

 

 

Hi @EricRegnier , yep,  you can update existing data without getting a duplicate warning. Dataflows seem to have changed a lot over the past.  Just not a lot of documentation about it.  Most of my knowledge is from experimenting with it!

 

Here is an example, where dataflow has updated existing Contact records in CDS which were created a while ago

Annotation 2020-06-19 1118463.png

 

@mattyn07 , using a staging entity is interesting.  And you could take that approach.

To answer your other questions. 

If a record already exists in CDS and is matched by the Alternative Key then that record is updated.  In the past I think dataflow used to delete the records and then re-create them.

If you choose the option to "delete records" then this will delete all records in CDS that are not matched with your data source - so this should only be used if your data source is the master for this data and if users are not creating records for this entity directly in CDS.

 

Hope this helps

I 100% agree re lack of documentation.

 

It has been tricky to find any real answers apart from the discussions we are having now. So your example below I noticed the "Created On" date remained the same date as I assume the dataflow populated that record in CDS? Or was it when the record was created in your source data (Power Query connection).

 

My source of truth will remain my external SQL DB which is good for my situation so I think the solution proposed will work as long as the modified date only updates the fields I have mapped (which i am 99.99999% sure it will).

Il see how I go over the weekend and potentially do a blog post as I think the community needs to see more info regarding what a dataflow can and cannot do.

 

Thanks again @HSheild  @EricRegnier 

EricRegnier
Most Valuable Professional
Most Valuable Professional

Cheers @HSheild, is there something in particular you configured? I just tried form my side and getting duplicate key exceptions. Here's how my config looks like:

2020-06-19_15-16-34.png

Error message of the row:

Reason: Precondition Failed, Error code: 0x80060892, Message: A record that has the attribute values Column1.char_id already exists. The entity key Char requires that this set of attributes contains unique values. Select unique values and try again.

 

@mattyn07, good idea for the blog post. You can also amend the Dataflow documentation directly now that everything is hosted in GitHub and if your updates are valid, MS will approve your PR.  @HSheild you should do the same with all the Dataflow tricks you have!

Hi @mattyn07 

 

"So your example below I noticed the "Created On" date remained the same date as I assume the dataflow populated that record in CDS? Or was it when the record was created in your source data (Power Query connection)."  Created On is the date that the record was first created in CDS by dataflow.  There is a field in CDS called overriddencreatedon which you can normally use to set the Created On date to be from your source data but that field doesn't seem to be available with dataflow yet.

 

Great idea on writing a blog - I was also thinking about doing the same thing.  Let me know how you get on with yours.

 

@EricRegnier, I'm not sure what your issue could be.  Your mapping looks OK.  I'll write a blog with steps for how I have it working and maybe that will help you.  Any chance that Char ID already exists twice in your CDS? If you have created data before you created the key?

EricRegnier
Most Valuable Professional
Most Valuable Professional

Nah, if Char ID wasn't unique, the alt key wouldn't get activated. I think it's just not supported yet, I also tried with a brand new dataflow, with data before, without, but no luck...

Helpful resources

Announcements

Exclusive LIVE Community Event: Power Apps Copilot Coffee Chat with Copilot Studio Product Team

  It's time for the SECOND Power Apps Copilot Coffee Chat featuring the Copilot Studio product team, which will be held LIVE on April 3, 2024 at 9:30 AM Pacific Daylight Time (PDT).     This is an incredible opportunity to connect with members of the Copilot Studio product team and ask them anything about Copilot Studio. We'll share our special guests with you shortly--but we want to encourage to mark your calendars now because you will not want to miss the conversation.   This live event will give you the unique opportunity to learn more about Copilot Studio plans, where we’ll focus, and get insight into upcoming features. We’re looking forward to hearing from the community, so bring your questions!   TO GET ACCESS TO THIS EXCLUSIVE AMA: Kudo this post to reserve your spot! Reserve your spot now by kudoing this post.  Reservations will be prioritized on when your kudo for the post comes through, so don't wait! Click that "kudo button" today.   Invitations will be sent on April 2nd.Users posting Kudos after April 2nd. at 9AM PDT may not receive an invitation but will be able to view the session online after conclusion of the event. Give your "kudo" today and mark your calendars for April 3rd, 2024 at 9:30 AM PDT and join us for an engaging and informative session!

Tuesday Tip: Unlocking Community Achievements and Earning Badges

TUESDAY TIPS are our way of communicating helpful things we've learned or shared that have helped members of the Community. Whether you're just getting started or you're a seasoned pro, Tuesday Tips will help you know where to go, what to look for, and navigate your way through the ever-growing--and ever-changing--world of the Power Platform Community! We cover basics about the Community, provide a few "insider tips" to make your experience even better, and share best practices gleaned from our most active community members and Super Users.   With so many new Community members joining us each week, we'll also review a few of our "best practices" so you know just "how" the Community works, so make sure to watch the News & Announcements each week for the latest and greatest Tuesday Tips!     THIS WEEK'S TIP: Unlocking Achievements and Earning BadgesAcross the Communities, you'll see badges on users profile that recognize and reward their engagement and contributions. These badges each signify a different achievement--and all of those achievements are available to any Community member! If you're a seasoned pro or just getting started, you too can earn badges for the great work you do. Check out some details on Community badges below--and find out more in the detailed link at the end of the article!       A Diverse Range of Badges to Collect The badges you can earn in the Community cover a wide array of activities, including: Kudos Received: Acknowledges the number of times a user’s post has been appreciated with a “Kudo.”Kudos Given: Highlights the user’s generosity in recognizing others’ contributions.Topics Created: Tracks the number of discussions initiated by a user.Solutions Provided: Celebrates the instances where a user’s response is marked as the correct solution.Reply: Counts the number of times a user has engaged with community discussions.Blog Contributor: Honors those who contribute valuable content and are invited to write for the community blog.       A Community Evolving Together Badges are not only a great way to recognize outstanding contributions of our amazing Community members--they are also a way to continue fostering a collaborative and supportive environment. As you continue to share your knowledge and assist each other these badges serve as a visual representation of your valuable contributions.   Find out more about badges in these Community Support pages in each Community: All About Community Badges - Power Apps CommunityAll About Community Badges - Power Automate CommunityAll About Community Badges - Copilot Studio CommunityAll About Community Badges - Power Pages Community

Tuesday Tips: Powering Up Your Community Profile

TUESDAY TIPS are our way of communicating helpful things we've learned or shared that have helped members of the Community. Whether you're just getting started or you're a seasoned pro, Tuesday Tips will help you know where to go, what to look for, and navigate your way through the ever-growing--and ever-changing--world of the Power Platform Community! We cover basics about the Community, provide a few "insider tips" to make your experience even better, and share best practices gleaned from our most active community members and Super Users.   With so many new Community members joining us each week, we'll also review a few of our "best practices" so you know just "how" the Community works, so make sure to watch the News & Announcements each week for the latest and greatest Tuesday Tips!   This Week's Tip: Power Up Your Profile!  🚀 It's where every Community member gets their start, and it's essential that you keep it updated! Your Community User Profile is how you're able to get messages, post solutions, ask questions--and as you rank up, it's where your badges will appear and how you'll be known when you start blogging in the Community Blog. Your Community User Profile is how the Community knows you--so it's essential that it works the way you need it to! From changing your username to updating contact information, this Knowledge Base Article is your best resource for powering up your profile.     Password Puzzles? No Problem! Find out how to sync your Azure AD password with your community account, ensuring a seamless sign-in. No separate passwords to remember! Job Jumps & Email Swaps Changed jobs? Got a new email? Fear not! You'll find out how to link your shiny new email to your existing community account, keeping your contributions and connections intact. Username Uncertainties Unraveled Picking the perfect username is crucial--and sometimes the original choice you signed up with doesn't fit as well as you may have thought. There's a quick way to request an update here--but remember, your username is your community identity, so choose wisely. "Need Admin Approval" Warning Window? If you see this error message while using the community, don't worry. A simple process will help you get where you need to go. If you still need assistance, find out how to contact your Community Support team. Whatever you're looking for, when it comes to your profile, the Community Account Support Knowledge Base article is your treasure trove of tips as you navigate the nuances of your Community Profile. It’s the ultimate resource for keeping your digital identity in tip-top shape while engaging with the Power Platform Community. So, dive in and power up your profile today!  💪🚀   Community Account Support | Power Apps Community Account Support | Power AutomateCommunity Account Support | Copilot Studio  Community Account Support | Power Pages

Super User of the Month | Chris Piasecki

In our 2nd installment of this new ongoing feature in the Community, we're thrilled to announce that Chris Piasecki is our Super User of the Month for March 2024. If you've been in the Community for a while, we're sure you've seen a comment or marked one of Chris' helpful tips as a solution--he's been a Super User for SEVEN consecutive seasons!       Since authoring his first reply in April 2020 to his most recent achievement organizing the Canadian Power Platform Summit this month, Chris has helped countless Community members with his insights and expertise. In addition to being a Super User, Chris is also a User Group leader, Microsoft MVP, and a featured speaker at the Microsoft Power Platform Conference. His contributions to the new SUIT program, along with his joyous personality and willingness to jump in and help so many members has made Chris a fixture in the Power Platform Community.   When Chris isn't authoring solutions or organizing events, he's actively leading Piasecki Consulting, specializing in solution architecture, integration, DevOps, and more--helping clients discover how to strategize and implement Microsoft's technology platforms. We are grateful for Chris' insightful help in the Community and look forward to even more amazing milestones as he continues to assist so many with his great tips, solutions--always with a smile and a great sense of humor.You can find Chris in the Community and on LinkedIn. Thanks for being such a SUPER user, Chris! 💪🌠

Find Out What Makes Super Users So Super

We know many of you visit the Power Platform Communities to ask questions and receive answers. But do you know that many of our best answers and solutions come from Community members who are super active, helping anyone who needs a little help getting unstuck with Business Applications products? We call these dedicated Community members Super Users because they are the real heroes in the Community, willing to jump in whenever they can to help! Maybe you've encountered them yourself and they've solved some of your biggest questions. Have you ever wondered, "Why?"We interviewed several of our Super Users to understand what drives them to help in the Community--and discover the difference it has made in their lives as well! Take a look in our gallery today: What Motivates a Super User? - Power Platform Community (microsoft.com)

March User Group Update: New Groups and Upcoming Events!

  Welcome to this month’s celebration of our Community User Groups and exciting User Group events. We’re thrilled to introduce some brand-new user groups that have recently joined our vibrant community. Plus, we’ve got a lineup of engaging events you won’t want to miss. Let’s jump right in: New User Groups   Sacramento Power Platform GroupANZ Power Platform COE User GroupPower Platform MongoliaPower Platform User Group OmanPower Platform User Group Delta StateMid Michigan Power Platform Upcoming Events  DUG4MFG - Quarterly Meetup - Microsoft Demand PlanningDate: 19 Mar 2024 | 10:30 AM to 12:30 PM Central America Standard TimeDescription: Dive into the world of manufacturing with a focus on Demand Planning. Learn from industry experts and share your insights. Dynamics User Group HoustonDate: 07 Mar 2024 | 11:00 AM to 01:00 PM Central America Standard TimeDescription: Houston, get ready for an immersive session on Dynamics 365 and the Power Platform. Connect with fellow professionals and expand your knowledge. Reading Dynamics 365 & Power Platform User Group (Q1)Date: 05 Mar 2024 | 06:00 PM to 09:00 PM GMT Standard TimeDescription: Join our virtual meetup for insightful discussions, demos, and community updates. Let’s kick off Q1 with a bang! Leaders, Create Your Events!    Leaders of existing User Groups, don’t forget to create your events within the Community platform. By doing so, you’ll enable us to share them in future posts and newsletters. Let’s spread the word and make these gatherings even more impactful! Stay tuned for more updates, inspiring stories, and collaborative opportunities from and for our Community User Groups.   P.S. Have an event or success story to share? Reach out to us – we’d love to feature you!

Users online (6,814)