python - Get xpath of a link in a webpage containing "sometext" -
i'm using scrapy (web crawling framework). there way can xpath of element (containing "sometext") in web page can extract elements similar xpaths? don't want xpaths hardcoded because crawling multiple websites.
i'm new scrapy , have been searching days , can't find :(
you have explicitly specify element want scrape either use xpath
or regular expression
or library beautifulsoup
. 1 way of not explicitly specifying xpath
traverse dom
, extracting elements need. in case need kind of mechanism identifying elements want scrape. should write different spiders scraping different websites. scraping multiple website single spider make task harder , not practice either.
for deploying , running spiders can scrapyd
Comments
Post a Comment