"Get files (Properties only)" not fetching all "Identifiers"

[Anonymous]

Hi everyone!

I have a problem which has been bugging me for a while now.

We have a huge Sharepoint library we are getting files from so we can later sort them. The library has 27.000 files.

When we try to use the "Get files (properties only)" it successfully gets the "Name" and "ID" variable of all 27.00 files, however it only gets about 3000 "Identifier" variables. I thought every file in a sharepoint library should have a Name (filename) an associated ID and an associated source file identifier (Identifier), so how come it gets the 20k names and IDs but not 20k identifiers? It's not matching up.

Any help? Have we just hit a limit or something along those lines? Thank you very much PA community!

[AndreasN]

Hi,

Have you tried to enable Pagination on the "get files" action, and set the Threshold to what you need?

[Anonymous]

I set Pagination to 20k and 50k and ran two test, got the same amount of items back both times, 20k on name and ID but only 3k for identifier, so strange...

Thanks for the quick reply however! Much appreciated!

[Anonymous]

Tried again with pagination at 100k (max amount) and I got 6k Identifiers this time. However 100k is the max amount, maybe I need to limit it so that it only fetches Name, ID and Identifiers from the "Get files (properties only)" connector, so it isn't clogged up with a ton of other variables, any idea on how to do that? Thanks again for your help 🙂 

[AndreasN]

Hello,

I'm unsure how you can limit the data it fetches, but you can limit the amount of files it fetches by applying the filter quary. 

You mentioned that these documents are sorted later on. If you apply some sort of metadata or properties to the documents, when they are sent to the SharePoint libary, you can use those properties to limit the amounts of files by using the filter query.

E.g. assume you have three different kinds of documents, it could be:
Contracts
Technical documents
Invoices

You would create a property with the column name "Document type" as a choice column, and apply those three choices.

Then you apply those properties to the documents.

In the filter query, you would then write 
Document type eq 'contracts'

to limit your files to only those that are contracts. I think that would be the most feasible solution, given you have so many files, to filter away some of them.