Notifications
Announcements
PAD n00b trying to start off with data extraction from web pages. This will only be my second PAD flow and I would really appreciate some detailed guidance.
I am trying to extract data from https://www.inecnigeria.org/elections/polling-units/. This page has multiple drop down lists from State-->Local Government-->Ward which then displays the polling units in the selected ward. I would like to iterate through the 8810 wards across 774 Local Governments in 37 states and store all of the data in a spreadsheet with format
STATE | LOCAL GOVERNMENT | WARD | Polling Unit ID | Polling Unit Name | Remark
This should generate 176,846 rows. And I am completely clueless how to automate the scraping. I would appreciate any assistance to get this done.
Thanks
Hi, @airsay, can you please edit into your original question some screenshots of your current flow (obfuscate where necessary) from the edit screen?
It helps with context, and it also helps folks not feel that they're just doing free contract work for you. 😉
Speaking generally, though ... what you're attempting here is called "scraping" and there's a little trick that many scrapers use to economise their data gathering, and that is to directly query the API that is making the information on the page. This saves you making multiple trips to the page and potentially run an extremely large flow that will probably never stop. 😅
First of all, after you've put some images of your current flow in the question ... I have a separate question:
Do you have a paid Power Automate licence, or are you running Desktop flows because you need to find a way around using HTTP calls?
I ask this because if you have the ability to query for this information, then you could send queries directly to this API, and you would get data back like in the following Spoiler:
{ "0": { "id": "1", "name": "ABIA POLY - ABIA POLY I", "registration_area_id": "1", "precise_location": null, "abbreviation": "005", "state": "01", "lga": "01", "ward": "01", "units": "005", "delimitation": "01/01/01/005", "remark": "EXISTING PU" }, "1": { "id": "2", "name": "ABIA POLY - ABIA POLY II", "registration_area_id": "1", "precise_location": null, "abbreviation": "006", "state": "01", "lga": "01", "ward": "01", "units": "006", "delimitation": "01/01/01/006", "remark": "EXISTING PU" }, "2": { "id": "3", "name": "ABIA POLY - ABIA POLY III", "registration_area_id": "1", "precise_location": null, "abbreviation": "007", "state": "01", "lga": "01", "ward": "01", "units": "007", "delimitation": "01/01/01/007", "remark": "EXISTING PU" }, "3": { "id": "4", "name": "ABIA POLY - ABIA POLY IV", "registration_area_id": "1", "precise_location": null, "abbreviation": "008", "state": "01", "lga": "01", "ward": "01", "units": "008", "delimitation": "01/01/01/008", "remark": "EXISTING PU" }, "4": { "id": "5", "name": "ABIA POLY - ABIA POLY V", "registration_area_id": "1", "precise_location": null, "abbreviation": "009", "state": "01", "lga": "01", "ward": "01", "units": "009", "delimitation": "01/01/01/009", "remark": "EXISTING PU" }, "5": { "id": "6", "name": "ABIA POLY - ABIA POLY VI", "registration_area_id": "1", "precise_location": null, "abbreviation": "010", "state": "01", "lga": "01", "ward": "01", "units": "010", "delimitation": "01/01/01/010", "remark": "EXISTING PU" }, "6": { "id": "7", "name": "EZIAMA HIGH SCHOOL - EZIAMA HIGH SCHOOL I", "registration_area_id": "1", "precise_location": null, "abbreviation": "011", "state": "01", "lga": "01", "ward": "01", "units": "011", "delimitation": "01/01/01/011", "remark": "EXISTING PU" }, "7": { "id": "8", "name": "EZIAMA HIGH SCHOOL - EZIAMA HIGH SCHOOL II", "registration_area_id": "1", "precise_location": null, "abbreviation": "012", "state": "01", "lga": "01", "ward": "01", "units": "012", "delimitation": "01/01/01/012", "remark": "EXISTING PU" }, "8": { "id": "9", "name": "EZIAMA HIGH SCHOOL - EZIAMA HIGH SCHOOL III", "registration_area_id": "1", "precise_location": null, "abbreviation": "013", "state": "01", "lga": "01", "ward": "01", "units": "013", "delimitation": "01/01/01/013", "remark": "EXISTING PU" }, "9": { "id": "10", "name": "EZIAMA HIGH SCHOOL - EZIAMA HIGH SCHOOL IV", "registration_area_id": "1", "precise_location": null, "abbreviation": "014", "state": "01", "lga": "01", "ward": "01", "units": "014", "delimitation": "01/01/01/014", "remark": "EXISTING PU" }, "10": { "id": "11", "name": "EZIAMA HIGH SCHOOL - EZIAMA HIGH SCHOOL V", "registration_area_id": "1", "precise_location": null, "abbreviation": "015", "state": "01", "lga": "01", "ward": "01", "units": "015", "delimitation": "01/01/01/015", "remark": "EXISTING PU" }, "11": { "id": "12", "name": "EZIAMA CENTRAL SCHOOL - EZIAMA CENTRAL SCHOOL I", "registration_area_id": "1", "precise_location": null, "abbreviation": "016", "state": "01", "lga": "01", "ward": "01", "units": "016", "delimitation": "01/01/01/016", "remark": "EXISTING PU" }, "12": { "id": "13", "name": "EZIAMA CENTRAL SCHOOL - EZIAMA CENTRAL SCHOOL II", "registration_area_id": "1", "precise_location": null, "abbreviation": "017", "state": "01", "lga": "01", "ward": "01", "units": "017", "delimitation": "01/01/01/017", "remark": "EXISTING PU" }, "13": { "id": "14", "name": "EZIAMA CENTRAL SCHOOL - EZIAMA CENTRAL SCHOOL III", "registration_area_id": "1", "precise_location": null, "abbreviation": "018", "state": "01", "lga": "01", "ward": "01", "units": "018", "delimitation": "01/01/01/018", "remark": "EXISTING PU" }, "14": { "id": "15", "name": "EZIAMA CENTRAL SCHOOL - EZIAMA CENTRAL SCHOOL IV", "registration_area_id": "1", "precise_location": null, "abbreviation": "019", "state": "01", "lga": "01", "ward": "01", "units": "019", "delimitation": "01/01/01/019", "remark": "EXISTING PU" }, "15": { "id": "16", "name": "EZIAMA CENTRAL SCHOOL - EZIAMA CENTRAL SCHOOL V", "registration_area_id": "1", "precise_location": null, "abbreviation": "020", "state": "01", "lga": "01", "ward": "01", "units": "020", "delimitation": "01/01/01/020", "remark": "EXISTING PU" }, "16": { "id": "17", "name": "EZIAMA CENTRAL SCHOOL - EZIAMA CENTRAL SCHOOL VI", "registration_area_id": "1", "precise_location": null, "abbreviation": "021", "state": "01", "lga": "01", "ward": "01", "units": "021", "delimitation": "01/01/01/021", "remark": "EXISTING PU" }, "17": { "id": "18", "name": "EZIAMA CENTRAL SCHOOL - EZIAMA CENTRAL SCHOOL VII", "registration_area_id": "1", "precise_location": null, "abbreviation": "022", "state": "01", "lga": "01", "ward": "01", "units": "022", "delimitation": "01/01/01/022", "remark": "EXISTING PU" }, "18": { "id": "19", "name": "EZIAMA CENTRAL SCHOOL - EZIAMA CENTRAL SCHOOL VIII", "registration_area_id": "1", "precise_location": null, "abbreviation": "023", "state": "01", "lga": "01", "ward": "01", "units": "023", "delimitation": "01/01/01/023", "remark": "EXISTING PU" }, "19": { "id": "20", "name": "ABIA POLY VII", "registration_area_id": "1", "precise_location": null, "abbreviation": "024", "state": "01", "lga": "01", "ward": "01", "units": "024", "delimitation": "01/01/01/024", "remark": "NEW PU" }, "20": { "id": "21", "name": "ABIA PLOY VIII", "registration_area_id": "1", "precise_location": null, "abbreviation": "025", "state": "01", "lga": "01", "ward": "01", "units": "025", "delimitation": "01/01/01/025", "remark": "NEW PU" }, "21": { "id": "22", "name": "ABIA PLOY IX", "registration_area_id": "1", "precise_location": null, "abbreviation": "026", "state": "01", "lga": "01", "ward": "01", "units": "026", "delimitation": "01/01/01/026", "remark": "NEW PU" }, "22": { "id": "23", "name": "ABIA POLY X", "registration_area_id": "1", "precise_location": null, "abbreviation": "027", "state": "01", "lga": "01", "ward": "01", "units": "027", "delimitation": "01/01/01/027", "remark": "NEW PU" }, "23": { "id": "24", "name": "ABIA PLOY XI", "registration_area_id": "1", "precise_location": null, "abbreviation": "028", "state": "01", "lga": "01", "ward": "01", "units": "028", "delimitation": "01/01/01/028", "remark": "NEW PU" }, "24": { "id": "25", "name": "ABIA POLY XII", "registration_area_id": "1", "precise_location": null, "abbreviation": "029", "state": "01", "lga": "01", "ward": "01", "units": "029", "delimitation": "01/01/01/029", "remark": "NEW PU" }, "25": { "id": "26", "name": "ABIA PLOY XIII", "registration_area_id": "1", "precise_location": null, "abbreviation": "030", "state": "01", "lga": "01", "ward": "01", "units": "030", "delimitation": "01/01/01/030", "remark": "NEW PU" }, "26": { "id": "27", "name": "ABIA PLOY XIV", "registration_area_id": "1", "precise_location": null, "abbreviation": "031", "state": "01", "lga": "01", "ward": "01", "units": "031", "delimitation": "01/01/01/031", "remark": "NEW PU" }, "27": { "id": "28", "name": "ABIA POLY XV", "registration_area_id": "1", "precise_location": null, "abbreviation": "032", "state": "01", "lga": "01", "ward": "01", "units": "032", "delimitation": "01/01/01/032", "remark": "NEW PU" }, "28": { "id": "29", "name": "ABIA PLOY XVI", "registration_area_id": "1", "precise_location": null, "abbreviation": "033", "state": "01", "lga": "01", "ward": "01", "units": "033", "delimitation": "01/01/01/033", "remark": "NEW PU" }, "29": { "id": "30", "name": "ABIA PLOY", "registration_area_id": "1", "precise_location": null, "abbreviation": "034", "state": "01", "lga": "01", "ward": "01", "units": "034", "delimitation": "01/01/01/034", "remark": "NEW PU" }, "30": { "id": "31", "name": "ABIA PLOY XVIII", "registration_area_id": "1", "precise_location": null, "abbreviation": "035", "state": "01", "lga": "01", "ward": "01", "units": "035", "delimitation": "01/01/01/035", "remark": "NEW PU" }, "31": { "id": "32", "name": "ABIA POLY XIX", "registration_area_id": "1", "precise_location": null, "abbreviation": "036", "state": "01", "lga": "01", "ward": "01", "units": "036", "delimitation": "01/01/01/036", "remark": "NEW PU" }, "32": { "id": "33", "name": "ABIA PLOY XX", "registration_area_id": "1", "precise_location": null, "abbreviation": "037", "state": "01", "lga": "01", "ward": "01", "units": "037", "delimitation": "01/01/01/037", "remark": "NEW PU" }, "33": { "id": "34", "name": "ABIA PLOY XXI", "registration_area_id": "1", "precise_location": null, "abbreviation": "038", "state": "01", "lga": "01", "ward": "01", "units": "038", "delimitation": "01/01/01/038", "remark": "NEW PU" }, "34": { "id": "35", "name": "ABIA PLOY XXII", "registration_area_id": "1", "precise_location": null, "abbreviation": "039", "state": "01", "lga": "01", "ward": "01", "units": "039", "delimitation": "01/01/01/039", "remark": "NEW PU" }, "35": { "id": "36", "name": "ABIA POLY XXIII", "registration_area_id": "1", "precise_location": null, "abbreviation": "040", "state": "01", "lga": "01", "ward": "01", "units": "040", "delimitation": "01/01/01/040", "remark": "NEW PU" }, "36": { "id": "37", "name": "ABIA PLOY XXIV", "registration_area_id": "1", "precise_location": null, "abbreviation": "041", "state": "01", "lga": "01", "ward": "01", "units": "041", "delimitation": "01/01/01/041", "remark": "NEW PU" }, "37": { "id": "38", "name": "ABIA PLOY XXV", "registration_area_id": "1", "precise_location": null, "abbreviation": "042", "state": "01", "lga": "01", "ward": "01", "units": "042", "delimitation": "01/01/01/042", "remark": "NEW PU" }, "38": { "id": "39", "name": "ABIA POLY VI", "registration_area_id": "1", "precise_location": null, "abbreviation": "043", "state": "01", "lga": "01", "ward": "01", "units": "043", "delimitation": "01/01/01/043", "remark": "NEW PU" }, "39": { "id": "40", "name": "ABIA PLOY XXIV", "registration_area_id": "1", "precise_location": null, "abbreviation": "044", "state": "01", "lga": "01", "ward": "01", "units": "044", "delimitation": "01/01/01/044", "remark": "NEW PU" }, "40": { "id": "41", "name": "EZIAMA HIGH SCHOOL II", "registration_area_id": "1", "precise_location": null, "abbreviation": "045", "state": "01", "lga": "01", "ward": "01", "units": "045", "delimitation": "01/01/01/045", "remark": "NEW PU" }, "41": { "id": "42", "name": "EZIAMA HIGH SCHOOL II", "registration_area_id": "1", "precise_location": null, "abbreviation": "046", "state": "01", "lga": "01", "ward": "01", "units": "046", "delimitation": "01/01/01/046", "remark": "NEW PU" }, "42": { "id": "43", "name": "EZIAMA HIGH SCH. III", "registration_area_id": "1", "precise_location": null, "abbreviation": "047", "state": "01", "lga": "01", "ward": "01", "units": "047", "delimitation": "01/01/01/047", "remark": "NEW PU" }, "43": { "id": "44", "name": "EZIAMA HIGH SCH. XI", "registration_area_id": "1", "precise_location": null, "abbreviation": "048", "state": "01", "lga": "01", "ward": "01", "units": "048", "delimitation": "01/01/01/048", "remark": "NEW PU" }, "44": { "id": "45", "name": "EZIAMA HIGH SCH. XII", "registration_area_id": "1", "precise_location": null, "abbreviation": "049", "state": "01", "lga": "01", "ward": "01", "units": "049", "delimitation": "01/01/01/049", "remark": "NEW PU" }, "45": { "id": "46", "name": "EZIAMA HIGH SCH.", "registration_area_id": "1", "precise_location": null, "abbreviation": "050", "state": "01", "lga": "01", "ward": "01", "units": "050", "delimitation": "01/01/01/050", "remark": "NEW PU" }, "46": { "id": "47", "name": "EZIAMA HIGH SCH. X", "registration_area_id": "1", "precise_location": null, "abbreviation": "051", "state": "01", "lga": "01", "ward": "01", "units": "051", "delimitation": "01/01/01/051", "remark": "NEW PU" }, "47": { "id": "48", "name": "EZIAMA HIGH SCH. XI", "registration_area_id": "1", "precise_location": null, "abbreviation": "052", "state": "01", "lga": "01", "ward": "01", "units": "052", "delimitation": "01/01/01/052", "remark": "NEW PU" }, "48": { "id": "49", "name": "RAILWAY QUARTERS V", "registration_area_id": "1", "precise_location": null, "abbreviation": "053", "state": "01", "lga": "01", "ward": "01", "units": "053", "delimitation": "01/01/01/053", "remark": "NEW PU" }, "49": { "id": "50", "name": "RAILWAY QUARTERS VI", "registration_area_id": "1", "precise_location": null, "abbreviation": "054", "state": "01", "lga": "01", "ward": "01", "units": "054", "delimitation": "01/01/01/054", "remark": "NEW PU" }, "50": { "id": "51", "name": "RAILWAY QUARTERS VII", "registration_area_id": "1", "precise_location": null, "abbreviation": "055", "state": "01", "lga": "01", "ward": "01", "units": "055", "delimitation": "01/01/01/055", "remark": "NEW PU" }, "51": { "id": "52", "name": "RAILWAY QUARTERS VIII", "registration_area_id": "1", "precise_location": null, "abbreviation": "056", "state": "01", "lga": "01", "ward": "01", "units": "056", "delimitation": "01/01/01/056", "remark": "NEW PU" }, "52": false }
That would be really easy for you to then work with. Although I would recommend not using Excel for this, as having it in your own SharePoint, DataVerse, or other, data store will allow you much more ability to use the information well. 👍
Hello @eliotcole thank you for the response. So I literally stumbled on Power Automate over the weekend from this video https://youtu.be/DgBZiBIgh3w. Prior to that I had been copying and pasting to Excel and knew there had to be a more efficient way to get this done. I was previously working with data from https://www.integrity.ng/index.php/units/browse. But with 1198 pages, it was hopeless. After watching referenced video, I was able to come up with the below workflow.
After harvesting that data, I found out that new polling units had been created and the provided data was off by 56k rows. The link I provided in original post hosts the up to date data and manual copying is not going to cut it for me hence the request. I am starting off into Data Analysis and with local elections due in February 2023, I thought analyzing this kind of data (polling units and ultimately number of registered voters in each polling unit) would be a good project to analyze real life election data. I am a complete n00b to PAD, but I am somewhat technically savvy. I did some programming courses over 20 years ago and my gut feeling says I would need three nested "for-loops" to be able to traverse what I believe is an array. But like I said I have absolutely zero experience with PAD (and automation in general) and would really appreciate a helping hand
Thanks for this. How and where would I place this to ensure it is able to scrape through al of the Polling Units and save the scrapped data into excel. I see you do not recommend excel for this. I know there are more efficient solutions to analyzing this data. But I am veering off into Data Analysis and using Excel as an initial building block.
Like I say, to really get much more of a fix from folks, @airsay, you will need to show what you've tried so far.
I'm not too clued in on PAD, so won't be able to help much. However, if you have a paid account then you can use HTTP get actions to get all the data that you need. There are other methods of doing it, but I won't speak of them here.
Under review
Thank you for your reply! To ensure a great experience for everyone, your content is awaiting approval by our Community Managers. Please check back later.
In our never-ending quest to improve we are simplifying the forum hierarchy…
We are honored to recognize Ajay Kumar Gannamaneni as our Community Spotlight for December…
These are the community rock stars!
Stay up to date on forum activity by subscribing.
Michael E. Gernaey 525 Super User 2025 Season 2
Tomac 324 Moderator
abm abm 232 Most Valuable Professional
