DataFlow does not bring all records to the final table in Dataverse

[calerof]

Hi,

I'm creating a DataFlow appending 3 tables from different sources:

1. Excel file in OneDrive for Business
2. SharePoint List
3. Imported table from SQL Server

After several attempts I got it working but when checking the actual data, I see some records missing, but in the DataFlow they seem to be correct.

Here's an screen-shot of one item with more than 15 rows in the DataFlow.

Here is a filter of the table in Dataverse for the same item, it only has 8 rows.

Any help would be extremely appreciated.

Thanks,

Fernando

[mahoneypat]

Just to clarify.  You are using a Dataflow to combine 3 sources which is working correctly, but when you view the Dataverse table in Excel rows are missing?

Regards,

Pat

[calerof]

Hi @mahoneypat 

The DataFlow creates an appended table Cardex. When I look at the data in the DataFlow everything looks fine, but when I look at the actual Cardex table it's incomplete. No, I don't use Flow.

[EricRegnier]

Hi @calerof,

Just to clarify, the 3 sources import into 3 different tables, correct? If so, which one is causing issues? Have you checked in advanced find without any filters to confirm that records are missing? Also, can you check that there are no errors/warnings after a dataflow run? To check go to make.powerapps.com --> Data --> Dataflows, click the little warning icon next to the Last Refresh column:

Cheers

[calerof]

Hi @EricRegnier ,

The Dataflow image above shows the original tables and the new appended one is called "cardex". In this new table within the Dataflow all records appear to be complete.

I didn't find the Advanced Find, this is a Canvas app, but I checked for example one record of date 13-feb-2021, which appears in the Dataflow image above with DocNum 1100 and Qty of 10 two times, and in the final appended table it does not exist.

Also, there are no errors nor warnings in the Dataflow refresh:

Still I don't know what's going on.

Thanks for the help.

Regards,

Fernando

[dpoggemann]

Hi @calerof,

Quick question, have you verified that the records showing in your dataflow were ever created in the Dataverse?  The 15 records you show have DocNum values where in your screenshot from the 8 rows in the dataverse only 5 of the 8 DocNum values match and if you look at these 5, none of the Fecha or CodigoTransaccion values match between the two.  I am guessing none of these records have been written to the cardex table.

Am I looking at something incorrectly?

Thanks,

Drew

[calerof]

Hi @dpoggemann ,

Thank you for your reply. Here are two screen shots of the complete Dataflow, all records from the appended table "Cardex" are in there:

Fernando

[dpoggemann]

Hi @calerof,

Sorry, I am a bit confused here.  You mentioned the records are not making it to the dataverse but in your last communication you are identifying they are "all there"?  Are you comfortable the data is in the dataverse and it is just a challenge with viewing the records?  If so, could this be simply a difference between running the dataflow as a user with admin rights (or elevated rights at BU or Oganization level) vs you querying dataverse as your own user that may have a role where you can view only your one records in the Cardex table?  Can you verify this from a security perspective with user roles assigned? 

Sorry if still off base, trying to understand exactly what the challenge is here.

Thanks,

Drew

[CharlesS · Accepted solution]

I'm thinking the table you extracted from Dataverse shows that the missing records are actually not in Dataverse. You can always check with a view as well to be sure they are not there.

If they are indeed missing, here is what I would do:

1. Filter your dataflow to include only the 15 rows you are trying to import.

2. Once the refresh is complete, check the refresh history (see@EricRegnier 's reply) and open the log to validate how many upserts you got. If someone else in your organization is running that dataflow, ask him or her to do the same. If the upserts number is not the same, then @dpoggemann idea about user rights to data would need to be investigated further.

3. What are you using as a primary key? Is there any way the data could be overwriting itself?

[calerof]

Hi @dpoggemann ,

The Dataflow records are complete, all the records in the appended table in Dataverse are included in the Dataflow, but not all the records in the Dataflow are included in the appended table.

I am System Administrator in the environment. 

Thanks,

Fernando