Web Scraping With a Twist

(1) Share

Report

Posted on by frank933

Guys, I'm new to the community and new to power automate, and I love the possibilities it brings.

I am looking to scrape a website in the following manner:

I have 1000 different URLs of a website laid out in column A in excel spreadsheet starting with cell A2. Using power automate I need to go through each cell, A2, A3 so on to A1000 and one by one plug in each url in the web browser so I could scrap some information from the webpage. The scraped information is to be then brought back and stored in the excel sheet next to the corresponding cell that contains the URL that opened that webpage. (Hope that's clear).

I have understood how to select items from a page and retrieve the information, but I need some pointers on how to set it up so the information can be used from excel sheet to open the page one by one (a loop I think?) and be stored in excel sheet one row at a time.

Can someone please provide a flow of steps in terms of what would be needed in Power automate?

Any help would be appreciate.

Thanks

Categories:

Power Automate Desktop

I have the same question (0)

All responses (7)

Answers (0)

Sort by

trice602 16,348 Super User 2026 Season 1 on at

Like
a
(1)

Report
Copy link

Link copied!

Hi @frank933 ,

I think I can help you, what you described is exactly what I use for one of my current projects and I look up about 500 rows.

So, here's an overview:

In step #2 I have a list of URLs in column A starting in A2.
In step #3 I want to get the first free row in column A.
In step #4 I read all the URLs in column A from A2 to the first free row -1.
In step #5 - 13 I look through each URL and, in my example, extract a specific element on webpage and store those values in the related Excel row in column B, C, D.
In step #6 I get the first free row in column B so I know where to write this information.

Let me know if this framework does the job for you!

Was this reply helpful? Yes No
Agnius Bartninkas Most Valuable Professional on at

Like
a
(1)

Report
Copy link

Link copied!
I was going to suggest exactly the same as what @trice602 suggested, except for a few minor modifications:

You need to Launch Excel before you can attach to a running file, so you might want to use that right away,

I'd use a different loop, so I don't need to get the first free row on column B every time - it's slightly faster this way.

It would look like this:

And here's a snippet you can copy and paste directly into PAD:

WebAutomation.LaunchEdge.LaunchEdge Url: $'''www.google.com''' WindowState: WebAutomation.BrowserWindowState.Normal ClearCache: False ClearCookies: False WaitForPageToLoadTimeout: 60 Timeout: 60 BrowserInstance=> Browser Excel.LaunchExcel.LaunchAndOpenUnderExistingProcess Path: $'''Add Excel file path here''' Visible: True ReadOnly: False Instance=> ExcelInstance Excel.GetFirstFreeColumnRow Instance: ExcelInstance FirstFreeColumn=> FirstFreeColumn FirstFreeRow=> FirstFreeRow Excel.ReadFromExcel.ReadCells Instance: ExcelInstance StartColumn: $'''A''' StartRow: 2 EndColumn: $'''A''' EndRow: FirstFreeRow - 1 ReadAsText: False FirstLineIsHeader: False RangeValue=> ExcelData LOOP LoopIndex FROM 0 TO ExcelData.RowsCount - 1 STEP 1 WebAutomation.GoToWebPage.GoToWebPage BrowserInstance: Browser Url: ExcelData[LoopIndex][0] WaitForPageToLoadTimeout: 60 # Do your scraping here Excel.WriteToExcel.WriteCell Instance: ExcelInstance Value: $'''{whatever you scraped from the web}''' Column: $'''B''' Row: LoopIndex + 2 Excel.SaveExcel.Save Instance: ExcelInstance END Excel.CloseExcel.Close Instance: ExcelInstance WebAutomation.CloseWebBrowser BrowserInstance: Browser

You will need to change the path to the file in Launch Excel as well as the value you're writing to Excel in Write to Excel worksheet after you add your web scraping actions.

You can copy and paste as many Write to Excel worksheet actions as you need for each different value you scrape, where you will just need to change the column to write to.

The reason I use %LoopIndex+2% as the row to write to is because loop indexes are 0-based in PAD, and your data starts at row 2 in Excel.

-------------------------------------------------------------------------
If I have answered your question, please mark it as the preferred solution. If you like my response, please give it a Thumbs Up.

I also provide paid consultancy and development services using Power Automate. If you're interested, DM me and we can discuss it.

Was this reply helpful? Yes No
frank933 12 on at

Like
a
(2)

Report
Copy link

Link copied!

You guys are amazing. Seems like both solutions would work well. I will test it out and post feedback.

Thanks so much for such prompt and informative responses.

Was this reply helpful? Yes No
trice602 16,348 Super User 2026 Season 1 on at

Like
a
(1)

Report
Copy link

Link copied!

Nice! Always great to hear!

Was this reply helpful? Yes No
frank933 12 on at

Like
a
(1)

Report
Copy link

Link copied!

Guys, forgive my lack of knowledge on this, as I just getting started in PAD.

So the procedure that @trice602 and @Agnius laid out about, does it handle error or an exception case where the data that's being picked up from most pages is all of sudden missing on a page or two or more?

Can you kindly guide me if there is a command or step(s) I'd have to undertake to handle that case so that the flow doesn't stops but ignore missing elements on some pages?
Thanks

Was this reply helpful? Yes No
trice602 16,348 Super User 2026 Season 1 on at

Like
a
(0)

Report
Copy link

Link copied!

Hi @frank933 ,

Mine does not. In my example, the pages I am scraping all are written with the same template and in fact, all three elements will always exist.

Was this reply helpful? Yes No
Agnius Bartninkas Most Valuable Professional on at

Like
a
(0)

Report
Copy link

Link copied!

You could add an On block error action inside your loop that goes to the end of the block if any of the web automation actions fail.

If the actions do not fail, but simply return a blank value, you can use an If statement that checks the value before writing anything to Excel. Then use Next loop to skip the rest of the actions in the loop and move on to the next case if the value is empty.

-------------------------------------------------------------------------
If I have answered your question, please mark it as the preferred solution. If you like my response, please give it a Thumbs Up.

I also provide paid consultancy and development services using Power Automate. If you're interested, DM me and we can discuss it.

Was this reply helpful? Yes No