OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

How to web scrape thomasnet website to get suppliers information in python

  • Thread starter Thread starter user3642360
  • Start date Start date
U

user3642360

Guest
I want to extract supplier information like supplier name, location, annual revenue, year founded, number of employees, product description etc from https://www.thomasnet.com/ for a particular location and category. For example, I want to extract all 201 suppliers information for category "Battery" and location "Southern California".

I am copying the url of each page for category "Battery" and location "Southern California" and getting the supplier information. But is there any way to automate the process such that I will get all the suppliers information if I put the category and location (irrespective of the number of pages for that search)?

This is what I am doing right now.

Code:
import requests
import ssl

from bs4 import BeautifulSoup, SoupStrainer

url = 'https://www.thomasnet.com/southern-california/batteries-3510203-1.html'
html_content = requests.get(url).text

# Parse the html content
soup = BeautifulSoup(html_content, "lxml")

supp_lst = soup.find_all( class_ = "profile-card__title" )
for data in supp_lst:
    # Get text from each tag
    print(data.text)
    
supp_location_lst = soup.find_all( class_ = "profile-card__location")
for data in supp_location_lst:
    # Get text from each tag
    print(data.text)

supp_content_lst = soup.find_all( class_ = "profile-card__body profile-card__mobile-view read-more-wrap")
for data in supp_content_lst:
    # Get text from each tag
    print(data.text)

supp_lst = soup.find_all(class_ = "profile-card__supplier-data")
for data in supp_lst:
    # Get text from each tag
    print(data.text)

I am very much new in web scraping. Any help and suggestion will be highly appreciated. TIA.,
<p>I want to extract supplier information like supplier name, location, annual revenue, year founded, number of employees, product description etc from <a href="https://www.thomasnet.com/" rel="nofollow noreferrer">https://www.thomasnet.com/</a> for a particular location and category. For example, I want to extract all 201 suppliers information for category "Battery" and location "Southern California".</p>
<p>I am copying the url of each page for category "Battery" and location "Southern California" and getting the supplier information. But is there any way to automate the process such that I will get all the suppliers information if I put the category and location (irrespective of the number of pages for that search)?</p>
<p>This is what I am doing right now.</p>
<pre><code>import requests
import ssl

from bs4 import BeautifulSoup, SoupStrainer

url = 'https://www.thomasnet.com/southern-california/batteries-3510203-1.html'
html_content = requests.get(url).text

# Parse the html content
soup = BeautifulSoup(html_content, "lxml")

supp_lst = soup.find_all( class_ = "profile-card__title" )
for data in supp_lst:
# Get text from each tag
print(data.text)

supp_location_lst = soup.find_all( class_ = "profile-card__location")
for data in supp_location_lst:
# Get text from each tag
print(data.text)

supp_content_lst = soup.find_all( class_ = "profile-card__body profile-card__mobile-view read-more-wrap")
for data in supp_content_lst:
# Get text from each tag
print(data.text)

supp_lst = soup.find_all(class_ = "profile-card__supplier-data")
for data in supp_lst:
# Get text from each tag
print(data.text)
</code></pre>
<p>I am very much new in web scraping. Any help and suggestion will be highly appreciated. TIA.,</p>
 

Latest posts

Online statistics

Members online
0
Guests online
2
Total visitors
2
Ads by Eonads
Top