一、Jsoup爬虫

jsoup 是一款Java 的HTML解析器，可直接解析某个URL地址、HTML文本内容。它提供了一套非常省力的API，可通过DOM，CSS以及类似于jQuery的操作方法来取出和操作数据。

以博客园首页为例

1、idea新建maven工程

pom.xml导入jsoup依赖

<dependency> <groupId>org.jsoup</groupId> <artifactId>jsoup</artifactId> <version>1.12.1</version></dependency>

jsoup代码

package com.blb;import org.jsoup.Jsoup;import org.jsoup.nodes.Document;import org.jsoup.nodes.Element;import org.jsoup.select.Elements;import java.io.IOException;public class jsoup { public static void main(String[] args) { //博客园首页url final String url= "https://";public static void main(String[] args) {System.setProperty("webdriver.chrome.driver","D:\\idea_workspace\\Jsoup\\src\\main\\chromedriver.exe");WebDriver driver = new ChromeDriver();driver.get(url);String pageSource = driver.getPageSource();Document jsoup = Jsoup.parse(pageSource);String rule1="body > div.xing_vb > ul> li > span.xing_vb4 > a";Elements select = jsoup.select(rule1);//遍历当前页的所有电影详情入口for (Element e:select){//获取电影详情页链接String href = e.attr("href");//进入每个电影详情页面driver.get(url+href);String pageSource2= driver.getPageSource();Document jsoup2 = Jsoup.parse(pageSource2);//定义获取电影信息元素的规则String mname="body > div.warp > div:nth-child(1) > div > div > div.vodInfo > div.vodh > h2";String mpic="body > div.warp > div:nth-child(1) > div > div > div.vodImg > img";String mdirector="body > div.warp > div:nth-child(1) > div > div > div.vodInfo > div.vodinfobox > ul > li:nth-child(2) > span";String mactor="body > div.warp > div:nth-child(1) > div > div > div.vodInfo > div.vodinfobox > ul > li:nth-child(3) > span";String marea="body > div.warp > div:nth-child(1) > div > div > div.vodInfo > div.vodinfobox > ul > li:nth-child(5) > span";String mlanguage="body > div.warp > div:nth-child(1) > div > div > div.vodInfo > div.vodinfobox > ul > li:nth-child(6) > span";String mshowtime="body > div.warp > div:nth-child(1) > div > div > div.vodInfo > div.vodinfobox > ul > li:nth-child(7) > span";String mscore="body > div.warp > div:nth-child(1) > div > div > div.vodInfo > div.vodh > label";String mtimelength="body > div.warp > div:nth-child(1) > div > div > div.vodInfo > div.vodinfobox > ul > li:nth-child(8) > span";String mlastmodifytime="body > div.warp > div:nth-child(1) > div > div > div.vodInfo > div.vodinfobox > ul > li:nth-child(9) > span";String minfo="body > div.warp > div:nth-child(1) > div > div > div.vodInfo > div.vodinfobox > ul > li.cont > div > span.more";String mplayaddress1="#play_1 > ul > li";String mplayaddress2="#play_2 > ul > li";String msv="body > div.warp > div:nth-child(1) > div > div > div.vodInfo > div.vodh > span";//获取元素信息String sv=jsoup2.select(msv).text();String name = jsoup2.select(mname).text();String pic = jsoup2.select(mpic).attr("src");String director=jsoup2.select(mdirector).text();String actor=jsoup2.select(mactor).text();String area=jsoup2.select(marea).text();String language=jsoup2.select(mlanguage).text();String showtime=jsoup2.select(mshowtime).text();String score=jsoup2.select(mscore).text();String timelength=jsoup2.select(mtimelength).text();String lastmodifytime=jsoup2.select(mlastmodifytime).text();String info=jsoup2.select(minfo).text();String playaddress1 = jsoup2.select(mplayaddress1).text();String playaddress2=jsoup2.select(mplayaddress2).text();//打印电影名System.out.println(name);}}}

为了不显示浏览器爬取过程，可以将chromedriver.exe 换成无头浏览器 phantomjs.exe

下载地址：https://phantomjs.org/download.html

到此这篇关于Java爬虫(Jsoup与WebDriver)的使用的文章就介绍到这了,更多相关Java Jsoup与WebDriver爬虫内容请搜索以前的文章或继续浏览下面的相关文章希望大家以后多多支持！