OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

Octoparse: Scraping when needing to first interact with duplicate controls

  • Thread starter Thread starter DanB
  • Start date Start date
D

DanB

Guest
I need to scrape a page where:

  • [Loop:
  • There's a long list with rows of items
  • Final col in each row is a button which I must click
  • [Loop:
  • Clicking the button displays another control with additional data
  • I scrape data within that new control
  • I close the control
  • :End Loop]
  • I click 'next page'
  • :End Loop]

The buttons on each row seem indistinguishable, so the xpath to every button is: //div[@class="btn generic-btn"]

But that means every time I click a button, I'm just getting data for the first row. It is not recursing through rows.

Any ideas how to approach this? I need to click on each button as I work down the page, but Octoparse doesn't seem happy to iterate through them and just keeps clicking the first one.

This is the page, if helpful (it takes 20 secs to load then please click the first blue button) https://agent-counsellor-ukhub.britishcouncil.org/gal

Thanks in advance Dan

<p>I need to scrape a page where:</p>
<ul>
<li>[Loop:</li>
<li>There's a long list with rows of items</li>
<li>Final col in each row is a button which I must click</li>
<li>[Loop:</li>
<li>Clicking the button displays another control with additional data</li>
<li>I scrape data within that new control</li>
<li>I close the control</li>
<li>:End Loop]</li>
<li>I click 'next page'</li>
<li>:End Loop]</li>
</ul>
<p>The buttons on each row seem indistinguishable, so the xpath to every button is:
//div[@class="btn generic-btn"]</p>
<p>But that means every time I click a button, I'm just getting data for the first row. It is not recursing through rows.</p>
<p>Any ideas how to approach this? I need to click on each button as I work down the page, but Octoparse doesn't seem happy to iterate through them and just keeps clicking the first one.</p>
<p>This is the page, if helpful (it takes 20 secs to load then please click the first blue button)
<a href="https://agent-counsellor-ukhub.britishcouncil.org/gal" rel="nofollow noreferrer">https://agent-counsellor-ukhub.britishcouncil.org/gal</a></p>
<p>Thanks in advance
Dan</p>
 

Latest posts

Top