OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

Can't fetch all the section elements using find_elements with Selenium

  • Thread starter Thread starter luis torres
  • Start date Start date
L

luis torres

Guest
I've been trying to use the find_elements() method of Selenium to extract the element of a webpage, there are 3 elements but the code is only fetching the first one.

This is the code I'm using:

Code:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager

url = f"https://www.latamairlines.com/us/en/flight-offers?origin=BOG&inbound=2024-07-15T12%3A00%3A00.000Z&outbound=2024-07-01T12%3A00%3A00.000Z&destination=MDE&adt=1&chd=0&inf=0&trip=RT&cabin=Economy&redemption=false&sort=RECOMMENDED"

options = webdriver.ChromeOptions()
options.add_argument('--incognito')

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options = options)
driver.get(url)

WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, './/button[@class="sc-dAbbOL dxkJUk sc-fHjqPf sc-bHvAfQ lmJCPj jshqAF"]'))).click()

driver.implicitly_wait(10)

flights = driver.find_elements(by = By.XPATH, value = '/html/body/div[1]/div[1]/main/div/div/div/div/ol/li')

WebDriverWait(flights[18], 10).until(EC.element_to_be_clickable((By.XPATH, f'/html/body/div[1]/div[1]/main/div/div/div/div/ol/li[19]/div/div/div[2]/div[1]/a'))).click()
WebDriverWait(flights[18], 10).until(EC.presence_of_all_elements_located((By.XPATH, '/html/body/div[1]/div[1]/main/div/div/div/div/ol/li[19]/div/div/div[2]/div[1]/dialog/div/div[1]/article/div')))
itineraries = flights[18].find_elements(by = By.XPATH, value = '/html/body/div[1]/div[1]/main/div/div/div/div/ol/li[19]/div/div/div[2]/div[1]/dialog/div/div[1]/article/div')
print(itineraries)

This image is a portion of the webpage that I'm scrapping.

The URL of the webpage is webpage

I used the same code to fetch all the <li> elements of the same webpage and it worked, I don't understand what I'm doing wrong with the <section> elements, so how can I do to fetch the 3 section elements?
<p>I've been trying to use the find_elements() method of Selenium to extract the element of a webpage, there are 3 elements but the code is only fetching the first one.</p>
<p>This is the code I'm using:</p>
<pre><code>from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager

url = f"https://www.latamairlines.com/us/en...bin=Economy&redemption=false&sort=RECOMMENDED"

options = webdriver.ChromeOptions()
options.add_argument('--incognito')

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options = options)
driver.get(url)

WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, './/button[@class="sc-dAbbOL dxkJUk sc-fHjqPf sc-bHvAfQ lmJCPj jshqAF"]'))).click()

driver.implicitly_wait(10)

flights = driver.find_elements(by = By.XPATH, value = '/html/body/div[1]/div[1]/main/div/div/div/div/ol/li')

WebDriverWait(flights[18], 10).until(EC.element_to_be_clickable((By.XPATH, f'/html/body/div[1]/div[1]/main/div/div/div/div/ol/li[19]/div/div/div[2]/div[1]/a'))).click()
WebDriverWait(flights[18], 10).until(EC.presence_of_all_elements_located((By.XPATH, '/html/body/div[1]/div[1]/main/div/div/div/div/ol/li[19]/div/div/div[2]/div[1]/dialog/div/div[1]/article/div')))
itineraries = flights[18].find_elements(by = By.XPATH, value = '/html/body/div[1]/div[1]/main/div/div/div/div/ol/li[19]/div/div/div[2]/div[1]/dialog/div/div[1]/article/div')
print(itineraries)
</code></pre>
<p>This <a href="https://i.sstatic.net/oII7VPA4.png" rel="nofollow noreferrer">image</a> is a portion of the webpage that I'm scrapping.</p>
<p>The URL of the webpage is <a href="https://www.latamairlines.com/us/en...bin=Economy&redemption=false&sort=RECOMMENDED" rel="nofollow noreferrer">webpage</a></p>
<p>I used the same code to fetch all the <code><li></code> elements of the same webpage and it worked, I don't understand what I'm doing wrong with the <code><section></code> elements, so how can I do to fetch the 3 section elements?</p>
 

Latest posts

Top