cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
AudiJames
New Member

Scrape Website Doesn't Finish

I've got my PowerAutomate all finished and it goes through all 50 pages that it has found. But the problem is that the script doesn't get to the Excel step because it keeps clicking on the next page part. I'm lost on exactly how to tell it to "know" when it is finished. I used the "RECORD" function to scrape data from a site, and I clicked on the fields and then I clicked on the Pager field, and that's it. 

Can someone help? Image below of my script.

 

AudiJames_0-1646614777031.png

 

1 ACCEPTED SOLUTION

Accepted Solutions
Pavel_NaNoi
Resolver III
Resolver III

Found the issue to your problem and have made a fix for it, the problem was as follows:

-Your code is 100% correct, but the website you're on has a lot of very horrible selectors, namely the pager selector, it is not changing state when its clicked or reaches the end, i.e. the only thing that changes is its colour and I think hyperlink state? either way, its impossible for PAD at the moment to spot this, I tried all sorts of selectors for this, I'm sure there's someone out there that knows better than me on this topic. If the pager element either disappeared or changed state this issue wouldn't occur.

 

As for the fix, please email me at ploktionov@automationoutcomes.co.uk I'll send you over a text file with the automation, in there copy-paste the text into your PAD automation, literally just CTRL+V the text from the .txt file anywhere in your Main flow and it should paste the work-around that I've devised for you which looks something like this:

Pavel_NaNoi_0-1646669590543.png

I'd paste the txt file in here, but you can't attach files in here by the looks of it other than images for some reason and the text in the .txt file is far too large for this board.

View solution in original post

3 REPLIES 3
Guriga
Advocate II
Advocate II

Guriga_0-1646669036500.png

 

Guriga_1-1646669083633.png

 

I tested for the 1st page. It's working perfectly.

 

Pavel_NaNoi
Resolver III
Resolver III

Found the issue to your problem and have made a fix for it, the problem was as follows:

-Your code is 100% correct, but the website you're on has a lot of very horrible selectors, namely the pager selector, it is not changing state when its clicked or reaches the end, i.e. the only thing that changes is its colour and I think hyperlink state? either way, its impossible for PAD at the moment to spot this, I tried all sorts of selectors for this, I'm sure there's someone out there that knows better than me on this topic. If the pager element either disappeared or changed state this issue wouldn't occur.

 

As for the fix, please email me at ploktionov@automationoutcomes.co.uk I'll send you over a text file with the automation, in there copy-paste the text into your PAD automation, literally just CTRL+V the text from the .txt file anywhere in your Main flow and it should paste the work-around that I've devised for you which looks something like this:

Pavel_NaNoi_0-1646669590543.png

I'd paste the txt file in here, but you can't attach files in here by the looks of it other than images for some reason and the text in the .txt file is far too large for this board.

Thanks again for your help -- those pager settings can be rather tricky depending on the site!

Helpful resources

Announcements
Power Platform Conf 2022 768x460.jpg

Join us for Microsoft Power Platform Conference

The first Microsoft-sponsored Power Platform Conference is coming in September. 100+ speakers, 150+ sessions, and what's new and next for Power Platform.

MPA Virtual Workshop Carousel 768x460.png

Register for a Free Workshop

Learn to digitize and optimize business processes and connect all your applications to share data in real time.

Power Automate Designer Feedback_carousel.jpg

Help make Flow Design easier

Are you new to designing flows? What is your biggest struggle with Power Automate Designer? Help us make it more user friendly!

Top Solution Authors
Top Kudoed Authors
Users online (1,567)