使用pycharm批量爬取小说

爬取小说的思路：

1.获取小说地址

本文以搜书网一小说为例《嘘，梁上有王妃！》
目录网址：https:///XuLiangShangYouWangFei'chapter_html=requests.get(base_url)chapter_page=ds(chapter_html.content,'lxml')chapter_novel=chapter_page.find(id="novel108799")#print(chapter_novel)chapter_novel_str=str(chapter_novel)regx = '<dd><a href="/XuLiangShangYouWangFei(.*?)"'chapter_href_list = re.findall(regx, chapter_novel_str)#print(chapter_href_list)chapter_url_list = []for i in chapter_href_list: url=base_url+i chapter_url_list.append(url)#print(chapter_url_list)for u in chapter_url_list: body_html=requests.get(u) body_page=ds(body_html.content,'lxml') body = body_page.find(class_='content') body_content=str(body) # print(body_content) body_regx='<br/> (.*?)\n' content_list=re.findall(body_regx,body_content) #print(content_list) title_regx = '<h1>(.*?)</h1>' title = re.findall(title_regx, body_html.text) #print(title) with open('1.txt', 'a+') as f: f.write('\n\n') f.write(title[0] + '\n') f.write('\n\n') for e in content_list: f.write(e + '\n') print('{} 爬取完毕'.format(title[0]))

到此这篇关于使用PyCharm批量爬取小说的文章就介绍到这了,更多相关PyCharm批量爬取小说内容请搜索以前的文章或继续浏览下面的相关文章希望大家以后多多支持！