[Enhancement suggestion] - Enhancement for allowing users to set the frequency of COMMIT during importing bulk data in Dynamics CRM 4.0 by using import class in SDK
[Problem] - Unable to complete importing as daily batch process in the nighttime. This importing process actually updating 3.7 million existing records in Dynamics CRM 4.0. - Although the target processing time for the processing is 12 hours, the actual processing time we did was 40 hours even in the result of several turnings.
[Cause] - In our investigation, a main cause of this performance problem is that system job of importing submits COMMIT to DB in every 10 records. - COMMIT in every 10 records is too frequent for process 3.7 million records. This causes I/O bottleneck in a drive where transaction log file is stored. And this affects lowering performance of the whole importing process.
[Severity] - This problem would cause serious lowering of the service level of the system which needs to import or update bulk data.
[Background] - This customer would use Dynamics CRM working closely with back-office system which has dozens of million records as master data. - 3.7 million records of master data in back-office system is updated daily and should be synchronized to Dynamics CRM.