cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Anonymous
Not applicable

Archiving CDS data to Data Lake

I have a situation where I am receiving a high volume of data on a daily basis. Practically, we only need to have access to the previous month, but need to store the historical data somewhere for reporting purposes.

 

We have determined that Azure would be a good place to store it, and I have read a little on the data lake connections, dataflows and export to data lake, but I'm not certain how it works.

 

I need to expert records that are greater than one month old to the archive storage. I don't want to keep them in the CDS as it is obliterating our storage capacity.

 

What would be the recommended way to achieve this?

 

Would a dataflow add to the existing data in the data lake, or would it overwrite with whatever is currently in CDS? This is how it reads for the export to data lake option, but it's unclear for dataflows.

 

I think I could make a power automate flow to push the data from CDS to Azure then delete the record in CDS, but this doesn't feel like the best way to go about it.

 

Any suggestions or recommendations would be greatly appreciated.

1 ACCEPTED SOLUTION

Accepted Solutions
RishabBehl31
Resolver I
Resolver I

Hey @Anonymous 

 

According to my perspective,  it is definitely a play for Data Lake (gen 2). Keep data backed up there, then have a flow run that cleans up the CDS. You could even have the flow do checks to make sure the data is backed up prior to deleting. or add an intermediary step where you deactivate old data on a first run, then delete it on a second run at a later time.

 

Refer to this article: https://docs.microsoft.com/en-us/powerapps/maker/common-data-service/export-to-data-lake

 

Also, it might be worth looking to see if you can use data flows/analytical entity to export the data to the lake. It is almost near real-time.

 

Refer to this article: https://docs.microsoft.com/en-us/powerapps/maker/common-data-service/connect-azure-data-lake-storage-for-dataflow

 

If you like my thoughts. A thumbs up is highly appreciated!

 

Warmly 🙂

View solution in original post

3 REPLIES 3
RishabBehl31
Resolver I
Resolver I

Hey @Anonymous 

 

According to my perspective,  it is definitely a play for Data Lake (gen 2). Keep data backed up there, then have a flow run that cleans up the CDS. You could even have the flow do checks to make sure the data is backed up prior to deleting. or add an intermediary step where you deactivate old data on a first run, then delete it on a second run at a later time.

 

Refer to this article: https://docs.microsoft.com/en-us/powerapps/maker/common-data-service/export-to-data-lake

 

Also, it might be worth looking to see if you can use data flows/analytical entity to export the data to the lake. It is almost near real-time.

 

Refer to this article: https://docs.microsoft.com/en-us/powerapps/maker/common-data-service/connect-azure-data-lake-storage-for-dataflow

 

If you like my thoughts. A thumbs up is highly appreciated!

 

Warmly 🙂

View solution in original post

Anonymous
Not applicable

Hi @RishabBehl31 

 

Thanks for your reply!

 

What you're saying is to store in data lake and use a flow to push the data?

 

I don't need the data to be real time, and any deletes from CDS need to be ignored.

 

I'll have a look at the dataflows some more, but it might be easier to use a flow for familiarity.

 

Thanks for your input, I appreciate it 🙂

@Anonymous  Yes i was suggesting Data flows to push data to the lake. Several advantages it offer over using a flow is - Dataflow with Analytical entities feature utilizes the Export to data lake service, which may offer varying levels of compliance, privacy, security, and data location commitments.

 

I suggest you to look into Data flows. A wonderful way to go!

 

Thanks for accepting my response a solution 🙂

Helpful resources

Announcements
UG GA Amplification 768x460.png

Launching new user group features

Learn how to create your own user groups today!

Community Connections 768x460.jpg

Community & How To Videos

Check out the new Power Platform Community Connections gallery!

Welcome Super Users.jpg

Super User Season 2

Congratulations, the new Super User Season 2 for 2021 has started!

Carousel 2021 Release Wave 2 Plan 768x460.jpg

2021 Release Wave 2 Plan

Power Platform release plan for the 2021 release wave 2 describes all new features releasing from October 2021 through March 2022.

Users online (1,076)