Unanswered

LinkedIn web scraping not allowing to go to multiple pages

(0) Share

Report

Posted on by bplives

Hello,

I wanted to perform a "simple" task on Linked in. So lets say I want to get all the employees location of "company X" and they have 500 employees. So I used PAD to go to the linkedin site with that company name already loaded. I then used the recorder and selected the name, the job title and the city. PAD recognized the pattern so I scrolled to the bottom of the page and clicked on the "next" button and selected "set this element as pager" and then I wanted to save the data in Excel and then be able to sort by city of the employees I wanted to contact. The automation ran but it only did the the10 contacts on the page and would never go to the next page never mind all 500 pages.

Does anyone know why it wouldn't' work?

Categories:

Power Automate Desktop

I have the same question (0)

All responses (6)

Answers (0)

Sort by

PhilipTreacy 120 on at

Like (0)

Report
Copy link

Link copied!

Hi @bplives

Can you link to one of the pages that has this Next link on it?

Can you also show what your flow looks like?

You can try manually adding Click link on web page

Can you see that the flow is looping? Or does it just run through once? You may need to manually add a loop to see the other pages.

regards

Phil

Was this reply helpful? Yes No
bplives 6 on at

Like (0)

Report
Copy link

Link copied!

Hello....Thanks for the tips.

Here is an example of a LinkedIn link I was trying to scrub the names from.

"samtec inc" | Search | LinkedIn

Also here is a screen shot of the automation.

Not sure where to put a loop......if it is ONLY running the one page and gives me about 10 contacts. Even though I select the next button as "set this element as pager"

But I would love to hear of some solution.

Thank you

Was this reply helpful? Yes No
PhilipTreacy 120 on at

Like (0)

Report
Copy link

Link copied!

Hi @bplives

I discovered that you can access the different pages of employee data by using the URL parameter page so adding page=2 gives you the 2nd page of data, etc. No need to try to find links to click on the page, just load the appropriate URL.

With this flow you can load the company page then load each page of employees. I haven't got any steps for scraping data, you can add those in.

There are 2 variables. PageNum controls the page of search results to load. Continue determines if the last page of results has been reached.

Initially you load https://www.linkedin.com/search/results/people/?keywords=samtec inc&origin=CLUSTER_EXPANSION which is the first page of employee data.

You need to add your steps between my Steps 5 and 6 to scrape the data you want.

The PageNum is then increased and the next page loaded with this URL https://www.linkedin.com/search/results/people/?keywords=samtec inc&origin=CLUSTER_EXPANSION&page=%PageNum%

If this next page contains the text No results found then there's no more data and the variable Continue is set to false - this ends the scraping loop, and ends the flow.

Good luck.

Phil
is a that control

Was this reply helpful? Yes No
bplives 6 on at

Like (0)

Report
Copy link

Link copied!

Ohhh wow. First of thank you for the reply so quickly,. I will try to do this today and report back to here on if I was successful.

Was this reply helpful? Yes No
bplives 6 on at

Like (0)

Report
Copy link

Link copied!

OKay I wanted to do your steps above just to make sure I got that part right.

Here is a small video link I hope works to show that it is not changing pages.

https://netorgft3515962-my.sharepoint.com/:v:/g/personal/bparker_daventria-tech_com/EV6ZG2X4k0FKjL_TICuCAvoBupAs2r7Pf0_8Hmu0VbPy1A?e=Uolmw5

I also noticed something. The links I am using form your example is actually pulling any mention of SemTec. I was wanting to just use the current employees structure. It is current and a shorter list.

So it should be this type of link:

https://www.linkedin.com/search/results/people/?currentCompany=%5B%22516326%22%5D&origin=COMPANY_PAGE_CANNED_SEARCH&sid=S6k

This now makes the list form 4000 down to 2200 since it is current.

I believe you added the "&pages='PageNum to the end of the link. Can I do that with this link as well?

Also when I did the recorder to go actually scrap the names, position, and location. It looked odd and wasn't working so I am not sure if I had those steps inserted where they should have been. Then I went back to basics and just wanted to see if it would change pages without the data recorder.

So I think I need to fix this changing of the pages. But not sure what I did wrong in your steps since you got it to work

Was this reply helpful? Yes No
Thatdeveloper 2 on at

Like (0)

Report
Copy link

Link copied!

did you get it to work? Im facing the same problem..

Was this reply helpful? Yes No