OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

Scraping content using selenium from a website using for loop in python

  • Thread starter Thread starter cooler gamer
  • Start date Start date
C

cooler gamer

Guest
I am new at python and selenium. I am trying to webscrape text from the website audible.in/search using for loop. Below is the code written.

I was running the following code

Code:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

web = "https://www.audible.in/search"
path = "C:/Users/vikas/Downloads/chromedriver-win64/chromedriver-win64/chromedriver"
driver = webdriver.Chrome(path)
driver.get(web)

container = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CLASS_NAME, "adbl-    impression-container "))
products = WebDriverWait(container, 5).until(EC.presence_of_all_elements_located((By.XPATH, "./div/span/ul/li")))

book_title = []
book_author = []
book_length = []

for product in products:
    book_title.append(product.find_element("xpath",'//h3[contains(@class, "bc-heading")]').text)
    book_author.append(product.find_element("xpath",'//a[contains(@href, "author")]').text)
    book_length.append(product.find_element("xpath",'//li[contains(@class, "runtimeLabel")]').text)

df = pd.DataFrame({'title':book_title, 'author':book_author, 'length':book_length})
df.to_csv('books.csv')

When I run the code I am expecting to append all the h3, a and li in the loop to the list. There are total 20 elements. What I am getting is the first element 20 times. What is that I am doing wrong here. Kindly help.
<p>I am new at python and selenium. I am trying to webscrape text from the website <a href="https://audible.in/search" rel="nofollow noreferrer">audible.in/search</a> using for loop. Below is the code written.</p>
<p>I was running the following code</p>
<pre><code>from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

web = "https://www.audible.in/search"
path = "C:/Users/vikas/Downloads/chromedriver-win64/chromedriver-win64/chromedriver"
driver = webdriver.Chrome(path)
driver.get(web)

container = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CLASS_NAME, "adbl- impression-container "))
products = WebDriverWait(container, 5).until(EC.presence_of_all_elements_located((By.XPATH, "./div/span/ul/li")))

book_title = []
book_author = []
book_length = []

for product in products:
book_title.append(product.find_element("xpath",'//h3[contains(@class, "bc-heading")]').text)
book_author.append(product.find_element("xpath",'//a[contains(@href, "author")]').text)
book_length.append(product.find_element("xpath",'//li[contains(@class, "runtimeLabel")]').text)

df = pd.DataFrame({'title':book_title, 'author':book_author, 'length':book_length})
df.to_csv('books.csv')
</code></pre>
<p>When I run the code I am expecting to append all the <code>h3</code>, <code>a</code> and <code>li</code> in the loop to the list. There are total 20 elements. What I am getting is the first element 20 times. What is that I am doing wrong here. Kindly help.</p>
 
Top