cancel
Showing results for 
Search instead for 
Did you mean: 

Increase performance of bulk delete

The performance of bulk delete has to be increased.

 

Currently I have measured it to be somewhere around 1-3 records per second. Maybe a bit higher.

 

When using external tools via the API which utilize the multithreading and batching logic (Kingswaysoft) I often reach 2-300 records per second, and have even for simple records such as system jobs, gotten speeds of 7-800 records per second.

 

This is important for several reasons:

1. The price for storage is very high compared to for instance storage in ASQL or even SharePoint. Hence keeping it down is essential.

2. With the Power Platform push of getting large enterprises both internal and customer sides into Power Platform, the amount of data will be even larger and hence also the need to keep it in check.

3. GDPR and other privacy and compliance regulations all over the world require personal and other data to be removed when not being used any more. This is most pertinent for B2C organizations and they also have the largest databases and hence the largest problems maintaining the data.

4. Entitlements. As the built in functionality cannot be used, many organization HAVE TO use external tools like the excellent tool Kingswaysoft with SSIS to solve this. However, this generates massive amounts of API requests which Microsoft will start to charge for if they exceed 100 000, which they most likely will for larger organizations. Microsoft should provide decent tools to handle the data, decently performant bulk delete is one essential part of this. Not providing decent tools and then charging the customers when they manage what Microsoft weren't able to provide, is less than decent.

Status: New
Comments
GustafW
Regular Visitor

Update, latest measures show that it can often reach speeds of 25 records per second. This is a lot better, but still a lot less than what is required if the platform is to be able to handle tens of millions of records.