Hello,
I am having some difficulty with pulling data out of AWS with Power Automate using the MySQL connector and an on-premises data gateway.
I am pulling several tables with this flow, and the tables are much larger than the maximum record limit for the MySQL connector. We need as much data as we can possibly get out of these tables. So, the settings of the request are as follows:
- I have pagination turned on and set to 100,000, which increases the maximum pull to 100,352 records.
- I have my $orderby parameter set to "id desc" to pull just the latest data.
- The $filter parameter changes based on the table.
I am getting no errors on the flow it's executing successfully and pulling data from all tables in the database.
However, I have noticed that any time I pull more than 512 records from a table, my $orderby parameter seems to be failing after record 512 and suddenly the records start at the bottom and sort by ascending order. The id column in the resulting table, for example, will look like this (this is not a real example, just an illustration):
| Record | Id |
| . . . | . . . |
| 509 | 3664759 |
| 510 | 3664758 |
| 511 | 3664757 |
| 512 | 3664756 |
| 513 | 12339 |
| 514 | 12340 |
| 515 | 12341 |
| . . . | . . . |
The $filter does not seem to be affected by this same problem. All results appear to be properly filtered.
So, in short, I am missing a huge chunk of records in every table that exceeds 512 records because $orderby only seems to be working for the first 512 in a request.
Does anyone have any insight?
