Following this tutorial https://www.scrapingdog.com/blog/scrape-indeed-using-python/, and running into this error:
Traceback (most recent call last):
File "C:/Users/det-lab/Documents/PycharmProjects/Indeed_webscrape/Indeed_job_data.py", line 23, in <module>
alllitags = allData.find_all("li", {"class":"eu4oa1w0"})
AttributeError: 'NoneType' object has no attribute 'find_all'
Here is my version of the program:
import requests
from bs4 import BeautifulSoup
l = []
o = {}
#declare the target URL and make an HTTP connection to that website.
target_url="https://www.indeed.com/jobs?q=data+analyst&l=New+York%2C+NY&from=searchOnHP&vjk=7cfb06a1924a00ef&advn\
=8756188910781422"
head = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0",
"Accept-Encoding": "gzip, deflate, br",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"Connection": "keep-alive",
"Accept-Language": "en-US,en;q=0.9,lt;q=0.8,et;q=0.7,de;q=0.6",
}
resp = requests.get(target_url, headers=head)
soup = BeautifulSoup(resp.text, 'html.parser')
allData = soup.find("div", {"class": "mosaic-provider-jobcards"})
#iterate over each of these li tags and extract all the data one by one using a for loop.
alllitags = allData.find_all("li", {"class":"eu4oa1w0"})
for i in range(0,len(alllitags)):
try:
o["name-of-the-job"]=alllitags[i].find("a").find("span").text
except:
o["name-of-the-job"] = None
try:
o["name-of-the-company"] = alllitags[i].find("span", {"data-testid":"company-name"}).text
except:
o["name-of-the-company"] = None
try:
o["job-location"] = alllitags[i].find("div", {"data-testid":"text-location"}).text
except:
o["job-location"] = None
try:
o["job-details"] = alllitags[i].find("div", {"class":"jobMetaDataGroup"}).text
except:
o["job-details"] = None
try:
o["pay-range"] = alllitags[i].find("div", {"class":"metadata salary-snippet-container"}).text
except:
o["pay-range"] = None
l.append(o)
o={}
print(l)
I’m aware this is a common error when dealing with webscraping but id like a specific explanation.
You need to sign in to view this answers
Leave feedback about this