本文将带您了解关于用PhantomJS启动的RemoteWebdriver无法打开httpsurl的新内容,同时我们还将为您解释webdriver.phantomjs()的相关知识,另外,我们还将为您
本文将带您了解关于用PhantomJS启动的RemoteWebdriver无法打开https url的新内容,同时我们还将为您解释webdriver.phantomjs()的相关知识,另外,我们还将为您提供关于CasperJS / PhantomJS不加载https页面、Flutter webview无法打开http协议网页的办法、Java Selenium封装--RemoteWebDriver、org.openqa.selenium.phantomjs.PhantomJSDriverService的实例源码的实用信息。
本文目录一览:- 用PhantomJS启动的RemoteWebdriver无法打开https url(webdriver.phantomjs())
- CasperJS / PhantomJS不加载https页面
- Flutter webview无法打开http协议网页的办法
- Java Selenium封装--RemoteWebDriver
- org.openqa.selenium.phantomjs.PhantomJSDriverService的实例源码
用PhantomJS启动的RemoteWebdriver无法打开https url(webdriver.phantomjs())
我将硒与PhantomJs配合使用来抓取URL。我如下初始化了驱动程序
final DesiredCapabilities caps = DesiredCapabilities.chrome();caps.setCapability( PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY, "PhantomJsPath");caps.setCapability("page.settings.loadImages", false);caps.setCapability("trustAllSSLCertificates", true);RemoteWebDriver driver = new PhantomJSDriver(caps);driver.setLogLevel(Level.OFF);driver.get("https://.......")
从驱动程序获得的页面源为空
我有什么想念的吗?
答案1
小编典典最近,POODLE漏洞迫使网站删除SSLv3支持。由于PhantomJS<v1.9.8默认情况下使用SSLv3,因此无法加载页面。要解决此问题,您需要使用--ssl-protocol=tlsv1
或运行PhantomJS--ssl-protocol=any
。请参阅此答案以获取纯PhantomJS。
caps = DesiredCapabilities.phantomjs(); // or new DesiredCapabilities();caps.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, new String[] {"--ssl-protocol=tlsv1"});// other capabilitiesdriver = new PhantomJSDriver(caps);
如果这样不能解决问题,您还可以添加
"--web-security=false", "--ignore-ssl-errors=true"
CasperJS / PhantomJS不加载https页面
我知道某些网页PhantomJS /CasperJS无法打开,我想知道这是否是其中之一:给出错误:PhantomJS无法打开页面status = fail。
我尝试忽略ssl错误并更改了用户代理,但是我不确定如何确定要使用哪个代理。
我现在正在做的只是在casper.start(url, function () { ...})
哪里进行Casper的基本设置url=https://maizepages.umich.edu
;
答案1
小编典典该问题可能与最近发现的SSLv3漏洞(POODLE)有关。网站所有者被迫从其网站中删除SSLv3支持。由于PhantomJS <v1.9.8默认情况下使用SSLv3 ,因此您应该使用TLSv1:
casperjs --ssl-protocol=tlsv1 yourScript.js
全面解决方案将any
用于新的PhantomJS版本与其他SSL协议一起使用时。但是,这将使POODLE漏洞可在尚未禁用SSLv3的站点上被利用。
casperjs --ssl-protocol=any yourScript.js
替代方法:更新到PhantomJS 1.9.8或更高版本。请注意,更新到PhantomJS
1.9.8会导致一个新的错误,这对于CasperJS来说尤其烦人。
如何验证:resource.error
在脚本的开头添加一个这样的事件处理程序:
casper.on("resource.error", function(resourceError){ console.log(''Unable to load resource (#'' + resourceError.id + ''URL:'' + resourceError.url + '')''); console.log(''Error code: '' + resourceError.errorCode + ''. Description: '' + resourceError.errorString);});
如果确实是SSLv3存在问题,则错误将类似于:
错误代码:6。说明:SSL握手失败
--ignore-ssl-errors=true
顺便说一句,当证书有问题时,您可能还想使用命令行选项来运行。
Flutter webview无法打开http协议网页的办法
以下方法参考了网友的设置并且确认可以成功在webview中打开http协议的网页
仅在flutter_inappbrowser组件测试成功,其余组件未测试
1、编辑 /项目目录/android/app/src/main/AndroidManifest.xml
...
>
<activity
找到"<activity"这行后,在上行">"前面添加一个设置
android:networkSecurityConfig="@xml/network_security_config"
2、在/项目目录/android/app/src/res目录创建xml目录,新建network_security_config.xml文件,内容如下
<?xml version="1.0" encoding="utf-8"?>
<network-security-config>
<base-config cleartextTrafficPermitted="true">
<trust-anchors>
<certificates src="system" />
</trust-anchors>
</base-config>
</network-security-config>
重新运行就可以打开http网页了
Java Selenium封装--RemoteWebDriver
package com.selenium.driver;
import java.io.File;
import java.io.IOException;
import java.net.URL;
import java.util.HashMap;
import java.util.Map;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.commons.io.FileUtils;
import org.openqa.selenium.Alert;
import org.openqa.selenium.Capabilities;
import org.openqa.selenium.Cookie;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.NoSuchElementException;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.TakesScreenshot;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.remote.Augmenter;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.openqa.selenium.remote.RemoteWebElement;
public class JSWebDriver{
private RemoteWebDriver wd = null;
private JavascriptExecutor jse = null;
public JSWebDriver(URL remoteAddress, Capabilities desiredCapabilities) {
wd = new RemoteWebDriver(remoteAddress, desiredCapabilities);
}
///
///浏览器url导航
///
public void goTo(String url){
wd.get(url);
}
///
///浏览器退出
///
public void quit(){
wd.quit();
}
///
///浏览器后退
///
public void back(){
wd.navigate().back();
}
///
///浏览器前进
///
public void forward(){
wd.navigate().forward();
}
///
///浏览器刷新
///
public void refresh(){
wd.navigate().refresh();
}
///
///切换到新浏览器窗口;按照title、url、index;支持正则匹配
///
public void switchToWindow(String by, String value, String...match) throws Exception{
String currenthandle = wd.getWindowHandle();
Set<String> handles = wd.getWindowHandles();
int currentIndex = -1;
String searchString = "";
for(String handle : handles){
currentIndex += 1;
if(handle.equals(currenthandle)){
continue;
}else{
wd.switchTo().window(handle);
if (match.length == 1 && match[0].equals("regex")){
if (by.equals("title")){
searchString = wd.getTitle();
}else if (by.equals("url")){
searchString = wd.getCurrentUrl();
}
Pattern pattern = Pattern.compile(value);
Matcher matcher = pattern.matcher(searchString);
if(matcher.find()){
return;
}
}else{
if (by.equals("title")){
searchString = wd.getTitle();
}else if (by.equals("url")){
searchString = wd.getCurrentUrl();
}else if (by.equals("index")){
searchString = Integer.toString(currentIndex);
}
if(searchString.equals(value)){
return;
}
}
}
}
Exception e = new Exception("Swtich Window Failed, Please Make Sure The Locator Was Right.");
throw e;
}
///
///JS弹框确认
///
public void clickAlertSure(){
Alert alert = wd.switchTo().alert();
alert.accept();
}
///
///JS弹框取消
///
public void clickAlertDismiss()
{
Alert alert = wd.switchTo().alert();
alert.dismiss();
}
///
///设置prompt弹框内容
///
public void setPromptMessage(String parameter){
Alert alert = wd.switchTo().alert();
alert.sendKeys(parameter);
}
///
///获取JS弹框内容
///
public String getPromptMessage(){
Alert alert = wd.switchTo().alert();
return alert.getText();
}
///
///切换到Frame窗口;先定位到iframe元素
///
public void switchToFrame(JSWebElement jselement){
wd.switchTo().frame(jselement.getNativeWebElement());
}
///
///执行JS脚本
///
public void executeScript(String parameter){
JavascriptExecutor js = getJSE();
js.executeScript(parameter);
}
///
///获取指定cookie
///
public String getCookie(String name){
Cookie cookie=wd.manage().getCookieNamed(name);
if (cookie == null){ return "null"; }
return cookie.getValue();
}
///
///获取所有cookie
///
public Map<String, String> getCookies(){
Map<String, String> newCookies = new HashMap<String, String>();
Set<Cookie> cookies= wd.manage().getCookies();
for (Cookie cookie : cookies){
newCookies.put(cookie.getName(), cookie.getValue());
}
return newCookies;
}
///
///截取屏幕
///
public void getScreen(String filepath){
WebDriver augmentedDriver = new Augmenter().augment(this.wd);
TakesScreenshot ts = (TakesScreenshot) augmentedDriver;
File screenShotFile = ts.getScreenshotAs(OutputType.FILE);
try {
FileUtils.copyFile (screenShotFile, new File(filepath));
}catch (IOException e){
e.printStackTrace();
}
}
///
///获取title
///
public String getTitle(){
return wd.getTitle();
}
///
///获取url
///
public String getUrl(){
return wd.getCurrentUrl();
}
///
///获取HTML源码
///
public String getSource(){
try {
Thread.sleep(500);
} catch (InterruptedException e) {
e.printStackTrace();
}
return wd.getPageSource();
}
///
///滚动页面到指定位置
///
public void scroll(String x, String y){
if (x.equals("left")){
x = "0";
}else if (x.equals("right")){
x = "document.body.scrollWidth";
}else if (x.equals("middle")){
x = "document.body.scrollWidth/2";
}
if (y.equals("top")){
y = "0";
}else if (y.equals("buttom")){
y = "document.body.scrollHeight";
}else if (y.equals("middle")){
y = "document.body.scrollHeight/2";
}
this.executeScript(String.format("scroll(%s,%s);", x, y));
}
///
///最大化浏览器
///
public void maximize(){
wd.manage().window().maximize();
}
public JSWebElement findElementById(String using) {
try {
return new JSWebElement((RemoteWebElement)wd.findElementById(using));
}catch (NoSuchElementException e){
return new JSWebElement();
}
}
public JSWebElement findElementByCssSelector(String using) {
try {
return new JSWebElement((RemoteWebElement)wd.findElementByCssSelector(using));
}catch (NoSuchElementException e){
return new JSWebElement();
}
}
public JSWebElement findElementByXPath(String using) {
try {
return new JSWebElement((RemoteWebElement)wd.findElementByXPath(using));
}catch (NoSuchElementException e){
return new JSWebElement();
}
}
public JSWebElement findElementByLinkText(String using) {
try {
return new JSWebElement((RemoteWebElement)wd.findElementByLinkText(using));
}catch (NoSuchElementException e){
return new JSWebElement();
}
}
public JSWebElement findElementByDom(String using) {
try {
JavascriptExecutor js = this.getJSE();
WebElement we = (WebElement)js.executeScript(String.format("return %s", using));
return new JSWebElement((RemoteWebElement)we);
}catch (NoSuchElementException e){
return new JSWebElement();
}
}
///
///获取原生的RemoteWebdriver对象
///
public RemoteWebDriver getNativeWebDriver(){
return this.wd;
}
private JavascriptExecutor getJSE(){
if (this.jse == null){
this.jse = (JavascriptExecutor) this.wd;
}
return jse;
}
}
org.openqa.selenium.phantomjs.PhantomJSDriverService的实例源码
protected WebDriver getDriver() { if (this._driver == null) { DesiredCapabilities caps = new DesiredCapabilities(); caps.setJavascriptEnabled(true); caps.setCapability( PhantomJSDriverService.PHANTOMJS_PAGE_SETTINGS_PREFIX + "userAgent",spoofUserAgent); caps.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS,new String[] { "--web-security=false","--ssl-protocol=any","--ignore-ssl-errors=true","--webdriver-loglevel=INFO" }); PhantomJSDriverService service = new PhantomJSDriverService.Builder() .usingPort(8080) .usingPhantomJSExecutable(new File("/usr/local/bin/phantomjs")) .build(); this._driver = new PhantomJSDriver(service,caps); } return this._driver; }
public PhantomJSDriverService.Builder createPhantomJsDriverServiceBuilder(SiteConfig siteConfig,DriverConfig driverConfig,File driverFile) { PhantomJSDriverService.Builder serviceBuilder = new PhantomJSDriverService.Builder(); serviceBuilder.usingPhantomJSExecutable(driverFile); // serviceBuilder.usingAnyFreePort(); // serviceBuilder.withEnvironment(Map<String,String>); // serviceBuilder.withLogFile(File); // serviceBuilder.usingPort(int); // // serviceBuilder.withProxy(Proxy); // serviceBuilder.usingGhostDriver(File); // serviceBuilder.usingCommandLineArguments(String[]); // serviceBuilder.usingGhostDriverCommandLineArguments(String[]); return serviceBuilder; }
/** * Generates a phantomJs webdriver. * * @return * A phantomJs webdriver * @throws TechnicalException * if an error occured when Webdriver setExecutable to true. */ private WebDriver generatePhantomJsDriver() throws TechnicalException { final String pathWebdriver = DriverFactory.getPath(Driver.PHANTOMJS); if (!new File(pathWebdriver).setExecutable(true)) { throw new TechnicalException(Messages.getMessage(TechnicalException.TECHNICAL_ERROR_MESSAGE_WEBDRIVER_SET_EXECUTABLE)); } logger.info("Generating Phantomjs driver ({}) ...",pathWebdriver); final DesiredCapabilities capabilities = new DesiredCapabilities(); capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_PAGE_CUSTOMHEADERS_PREFIX + "Accept-Language","fr-FR"); capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,pathWebdriver); capabilities.setCapability(CapabilityType.UNEXPECTED_ALERT_BEHAVIoUR,UnexpecteDalertBehavIoUr.ACCEPT); capabilities.setJavascriptEnabled(true); setLoggingLevel(capabilities); // Proxy configuration String proxy = ""; if (Context.getProxy().getProxyType() != ProxyType.UNSPECIFIED && Context.getProxy().getProxyType() != ProxyType.AUTODETECT) { proxy = Context.getProxy().getHttpProxy(); } capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS,new String[] { "--proxy=" + proxy,"--web-security=no","--ssl-protocol=tlsv1","--webdriver-loglevel=NONE" }); return new PhantomJSDriver(capabilities); }
private static WebDriver createPhantomJSDriver(String driverPath) { DesiredCapabilities desiredCapabilities = DesiredCapabilities.phantomjs(); desiredCapabilities.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,driverPath); desiredCapabilities.setCapability(CapabilityType.ELEMENT_SCROLL_BEHAVIOR,true); desiredCapabilities.setCapability(CapabilityType.TAKES_SCREENSHOT,true); desiredCapabilities.setCapability(CapabilityType.ENABLE_PROFILING_CAPABILITY,true); desiredCapabilities.setCapability(CapabilityType.HAS_NATIVE_EVENTS,true); desiredCapabilities.setJavascriptEnabled(true); ArrayList<String> cliArgs = new ArrayList<String>(); cliArgs.add("--web-security=true"); cliArgs.add("--ignore-ssl-errors=true"); desiredCapabilities.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS,cliArgs); return new PhantomJSDriver(desiredCapabilities); }
PwnBackWebDriver(PwnBackMediator mediator) { this.mediator = mediator; PhantomJSDriverService driverService = new PhantomJSDriverService.Builder() .usingPhantomJSExecutable(new File(PwnBackSettings.phatomjsLocation)) .build(); DesiredCapabilities capability = new DesiredCapabilities(); capability.setCapability("takesScreenshot",false); String[] args = new String[1]; args[0] = ""; if (checkSSLCertPathDefined()) { args[0] = "--ssl-certificates-path=" + PwnBackSettings.caBundleLocation; } capability.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS,args); capability.setCapability("phantomjs.page.settings.userAgent","Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"); driver = new PhantomJSDriver(driverService,capability); }
@Override public void onTrigger(ProcessContext context,ProcessSession session) { DesiredCapabilities DesireCaps = new DesiredCapabilities(); DesireCaps.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,context.getProperty(DRIVER_LOCATION).getValue()); driver = new PhantomJSDriver(DesireCaps); FlowFile flowFile = session.create(); try { driver.get(url); (new webdriverwait(driver,timeout)).until( ExpectedConditions.visibilityOfElementLocated(getExpectedCondition(selectorType,selector)) ); final byte[] page = formatToXHtml(driver.getPageSource(),StandardCharsets.UTF_8); flowFile = session.write(flowFile,outputStream -> outputStream.write(page)); session.transfer(flowFile,REL_SUCCESS); } catch (Exception e) { flowFile = session.write(flowFile,outputStream -> outputStream.write(e.getMessage().getBytes())); session.transfer(flowFile,REL_FAILURE); } finally { driver.quit(); } session.getProvenanceReporter().create(flowFile); }
public static PhantomJSDriverService createDefaultService(Capabilities desiredCapabilities,Map<String,String> env) { Proxy proxy = null; if (desiredCapabilities != null) { proxy = Proxy.extractFrom(desiredCapabilities); } File phantomjsfile = findPhantomJS(desiredCapabilities,"https://github.com/ariya/phantomjs/wiki","http://phantomjs.org/download.html"); File ghostDriverfile = findGhostDriver(desiredCapabilities,"https://github.com/detro/ghostdriver/blob/master/README.md","https://github.com/detro/ghostdriver/downloads"); Builder builder = new Builder(); builder.usingPhantomJSExecutable(phantomjsfile) .usingGhostDriver(ghostDriverfile) .usingAnyFreePort() .withProxy(proxy) .withLogFile(new File("phantomjsdriver.log")) .usingCommandLineArguments(findCLIArgumentsFromCaps(desiredCapabilities,"phantomjs.cli.args")) .usingGhostDriverCommandLineArguments(findCLIArgumentsFromCaps(desiredCapabilities,"phantomjs.ghostdriver.cli.args")); if(null != env) builder.withEnvironment(env); return builder.build(); }
public PhantomJSDriverService createService(String... arguments) { List<String> argList = new ArrayList<>(); argList.add("--web-security=false"); argList.add("--webdriver-loglevel=TRACE"); argList.add("--load-images=false"); argList.add("--ignore-ssl-errors=true"); argList.add("--ssl-protocol=any"); argList.addAll(Arrays.asList(arguments)); PhantomJSDriverService service = new PhantomJSDriverService.Builder() .usingPhantomJSExecutable(EXECUTABLE) .usingCommandLineArguments(argList.toArray(new String[argList.size()])) .usingAnyFreePort() .build(); LoggingOutputStream loggingOutputStream = new LoggingOutputStream(log,LogLevel.INFO); service.sendOutputTo(loggingOutputStream); return service; }
@Test public void startAndStop() throws IOException { PhantomJSDriverService driverService = factory.createService(); driverService.start(); try { WebDriver webDriver = factory.createWebDriver(driverService); String url = "http://localhost:" + httpServer.getAddress().getPort() + "/webdriverTest"; webDriver.get(url); String currentUrl = webDriver.getcurrenturl(); assertthat(currentUrl).isEqualTo(url); } finally { driverService.stop(); } }
@Ignore @Test public void testProxyHTTP() throws IOException { String[] arguments = factory.createServiceArgumentsBuilder() .httpProxyServer("http://localhost:" + proxyPortProber.getPort()) .build(); PhantomJSDriverService service = factory.createService(arguments); service.start(); try { WebDriver webDriver = factory.createWebDriver(service); webDriver.get("http://www.lgohlke.de"); assertthat(webDriver.getPageSource().length()).isBetween(24000,26000); } finally { service.stop(); } }
@Ignore @Test public void testProxyHTTPS() throws IOException { String[] arguments = factory.createServiceArgumentsBuilder() .httpProxyServer("http://localhost:" + proxyPortProber.getPort()) .build(); PhantomJSDriverService service = factory.createService(arguments); service.start(); try { WebDriver webDriver = factory.createWebDriver(service); webDriver.get("https://www.google.de"); assertthat(webDriver.getPageSource().length()).isBetween(100000,110000); } finally { service.stop(); } }
@Test public void testSeleniumGhostDriver() { File phantomjs = Phanbedder.unpack(); DesiredCapabilities dcaps = new DesiredCapabilities(); dcaps.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,phantomjs.getAbsolutePath()); PhantomJSDriver driver = new PhantomJSDriver(dcaps); try { driver.get("https://www.google.com"); WebElement query = driver.findElement(By.name("q")); query.sendKeys("Phanbedder"); query.submit(); Assertions.assertthat(driver.getTitle()).contains("Phanbedder"); } finally { driver.quit(); } }
private static DesiredCapabilities initbrowserCapabilities() { DesiredCapabilities browserCapabilities = new DesiredCapabilities(); browserCapabilities.setJavascriptEnabled(true); if (StringUtils.isNotEmpty(PHANTOM_JS_PATH_PROP)) { System.out.printf("\n\nSetting Phantom JS path to %s\n\n%n",PHANTOM_JS_PATH_PROP); browserCapabilities.setCapability( PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,PHANTOM_JS_PATH_PROP); } browserCapabilities.setCapability("takesScreenshot",true); browserCapabilities.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS,buildPhantomJsCommandLineArguments()); browserCapabilities.setCapability(PhantomJSDriverService.PHANTOMJS_GHOSTDRIVER_CLI_ARGS,new String[]{ formatArgument(LOG_LEVEL_ARG,ERROR) }); return browserCapabilities; }
/** * PhantomJS bean * @param capabilities Desirercapabilities for WebDriver * @return WebDriver instance */ @Bean(name="phantomJS") @Lazy(true) @Scope(Configurablebeanfactory.ScopE_PROTOTYPE) public WebDriver phantomJS(DesiredCapabilities capabilities){ DesiredCapabilities phantomJSCap = new DesiredCapabilities(); phantomJSCap.setJavascriptEnabled(true); try { phantomJSCap.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,URLDecoder.decode(getClass() .getClassLoader().getResource(".").getPath(),"UTF-8")+System.getProperty("file.separator")+"downloads"+System.getProperty("file.separator")+"phantomjs-2.0.0-windows"+System.getProperty("file" + ".separator")+"phantomjs.exe"); } catch (UnsupportedEncodingException e) { log.error("Exception occured constructing path to executable: {}",e); } phantomJSCap.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS,new String[]{ "--ignore-ssl-errors=yes","--web-security=false","--ssl-protocol=any"}); phantomJSCap.setCapability(PhantomJSDriverService.PHANTOMJS_PAGE_SETTINGS_PREFIX,"Y"); phantomJSCap.setCapability("phantomjs.page.settings.userAgent","Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/41.0.2228.0 Safari/537.36"); phantomJSCap.setCapability("elementScrollBehavior",1); return new PhantomJSDriver(capabilities.merge(phantomJSCap)); }
public static void main(String[] args) { //设置相应的驱动程序的位置 System.setProperty(ChromeDriverService.CHROME_DRIVER_EXE_PROPERTY,"F:\\selenium\\chromedriver.exe"); System.setProperty(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,"F:\\selenium\\phantomjs.exe"); //任务执行线程池大小 int threadCount = 5; SpiderConfig spiderConfig = SpiderConfig.create("baidu",threadCount) .setEmptySleepMillis(1000) .setExitWhenComplete(true); SiteConfig siteConfig = SiteConfig.create() .setMaxConnTotal(200) .setMaxConnPerRoute(100); //定义WebDriver池 WebDriverPool webDriverPool = new WebDriverPool( new WebDriverFactory(),new DefaultWebDriverChooser(DriverType.CHROME),threadCount); //使用WebDriverDownloader请求任务下载器 Downloader downloader = new WebDriverDownloader(webDriverPool); Spider spider = Spider.create(spiderConfig,siteConfig,new BaiduPageProcessor()) .setDownloader(downloader) //设置请求任务下载器 .setPipeline(new BaiduPipeline()) .addStartRequests("https://www.baidu.com/s?wd=https"); //监控 SpiderMonitor.register(spider); //启动 spider.start(); }
public static DesiredCapabilities getDefaultDesiredCapabilities() { DesiredCapabilities capabilities = new DesiredCapabilities(); capabilities.setJavascriptEnabled(true); capabilities.setCapability("acceptSslCerts",true); capabilities.setCapability("takesScreenshot",true); capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS,asList("--ignore-ssl-errors=true")); LoggingPreferences loggingPreferences = new LoggingPreferences(); loggingPreferences.enable(LogType.broWSER,Level.ALL); capabilities.setCapability(CapabilityType.LOGGING_PREFS,loggingPreferences); return capabilities; }
/** * * @param poolsize * @param loadImg * 是否加载图片,默认不加载 */ public PhantomjsWebDriverPool(int poolsize,boolean loadImg,String phantomjsPath) { this.CAPACITY = poolsize; innerQueue = new LinkedBlockingDeque<WebDriver>(poolsize); PHANTOMJS_PATH = phantomjsPath; caps.setJavascriptEnabled(true); caps.setCapability("webStorageEnabled",true); caps.setCapability( PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,PHANTOMJS_PATH); // caps.setCapability("takesScreenshot",false); caps.setCapability( PhantomJSDriverService.PHANTOMJS_PAGE_CUSTOMHEADERS_PREFIX + "User-Agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/50.0.2661.102 Safari/537.36"); ArrayList<String> cliArgsCap = new ArrayList<String>(); // http://phantomjs.org/api/command-line.html cliArgsCap.add("--web-security=false"); cliArgsCap.add("--ssl-protocol=any"); cliArgsCap.add("--ignore-ssl-errors=true"); if (loadImg) { cliArgsCap.add("--load-images=true"); } else { cliArgsCap.add("--load-images=false"); } caps.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS,cliArgsCap); caps.setCapability( PhantomJSDriverService.PHANTOMJS_GHOSTDRIVER_CLI_ARGS,new String[] { "--logLevel=INFO" }); }
public PhantomJSDriverService getPhantomJsService() { return new PhantomJSDriverService.Builder() .usingAnyFreePort() .usingPhantomJSExecutable( new File(ResourceHelper .getResourcePath("driver/phantomjs.exe"))) .withLogFile( new File(ResourceHelper .getResourcePath("logs/phantomjslogs/") + "phantomjslogs" + DateTimeHelper.getCurrentDateTime() + ".log")) .build(); }
private static DesiredCapabilities getPhantomCapabilities(Locale lang) { DesiredCapabilities capabilities = DesiredCapabilities.phantomjs(); String value = lang.getLanguage().toLowerCase(); capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_PAGE_CUSTOMHEADERS_PREFIX + "Accept-Language",value); capabilities.setCapability(CapabilityType.UNEXPECTED_ALERT_BEHAVIoUR,UnexpecteDalertBehavIoUr.IGnorE); return capabilities; }
/** * Use phantomjs to fetch the web content. * @return */ protected WebDriver phantomJsDriver() { String phantomJsExecPath = System.getProperty("phantomjsExec"); if(phantomJsExecPath == null || "".equals(phantomJsExecPath)) { phantomJsExecPath = "/usr/local/bin/phantomjs"; } DesiredCapabilities dc = new DesiredCapabilities(); dc.setJavascriptEnabled(true); dc.setCapability( PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,phantomJsExecPath ); return new PhantomJSDriver(dc); }
private static WebDriver getPhantomJsDriver() { final DesiredCapabilities caps = DesiredCapabilities.phantomjs(); caps.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS,new String[] { "--ignore-ssl-errors=true","--web-security=false" }); final PhantomJSDriver phantomJSDriver = new PhantomJSDriver(caps); phantomJSDriver.manage().window().setSize(new Dimension(1280,800)); return phantomJSDriver; }
@BeforeClass public static void setupEnv() throws IOException { String phantomJSPath = System.getProperty("phantomjs.binary"); File phantomJSExec = new File(phantomJSPath); service = new PhantomJSDriverService.Builder().usingPhantomJSExecutable(phantomJSExec).build(); service.start(); }
@BeforeClass public static void setupEnv() throws IOException { String phantomJSPath = System.getProperty("phantomjs.binary"); File phantomJSExec = new File(phantomJSPath); service = new PhantomJSDriverService.Builder().usingPhantomJSExecutable(phantomJSExec).build(); service.start(); }
@BeforeClass public static void setupEnv() throws IOException { String phantomJSPath = System.getProperty("phantomjs.binary"); File phantomJSExec = new File(phantomJSPath); service = new PhantomJSDriverService.Builder().usingPhantomJSExecutable(phantomJSExec).build(); service.start(); }
@BeforeClass public static void setupEnv() throws IOException { String phantomJSPath = System.getProperty("phantomjs.binary"); File phantomJSExec = new File(phantomJSPath); service = new PhantomJSDriverService.Builder().usingPhantomJSExecutable(phantomJSExec).build(); service.start(); }
/** * Setting up PhantoJSDriver * * @return PhantomJSDriver */ public static WebDriver setUpPhantomJSDriver(String driverPath) { File phantomjs = new File(driverPath); DesiredCapabilities dcaps = new DesiredCapabilities(); dcaps.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS,new String[]{"--ignore-ssl-errors=yes","version=2.0","driverVersion=1.2.0"}); dcaps.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,phantomjs.getAbsolutePath()); return new PhantomJSDriver(dcaps); }
/** * Default implementation throws UnsupportedOperationException * @return WebDriver instance */ protected WebDriver getPhantomJsWebDriver() { String pathToBin = getosspecificBinaryPathFromProp(PHANTOM_JS_BIN_PROP,"phantomjs"); DesiredCapabilities capabilities = getPhantomJsCapabilities(); capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,pathToBin); return new PhantomJSDriver(capabilities); }
@Override public void before() { File phantomjs = Phanbedder.unpack(); DesiredCapabilities dcaps = new DesiredCapabilities(); dcaps.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,phantomjs.getAbsolutePath()); driver = new PhantomJSDriver(dcaps); }
public WebDriver get() { File phantomJsExe = new File("phantomjs-1.9.7-windows/phantomjs.exe"); Map<String,Object> caps = new HashMap<>(); caps.put(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,phantomJsExe.getAbsolutePath()); DesiredCapabilities capabilities = new DesiredCapabilities(caps); WebDriver driver = new PhantomJSDriver(capabilities); Dimension DEFAULT_WINDOW_SIZE = new Dimension(1024,768); driver.manage().window().setSize(DEFAULT_WINDOW_SIZE); return driver; }
@Override public WebDriver getWebDriver() { DesiredCapabilities caps = new DesiredCapabilities(); caps.setJavascriptEnabled(true); caps.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,"src/test/resources/phantomjs"); caps.setCapability("takesScreenshot",true); return new PhantomJSDriver(caps); }
private static Capabilities configWebDriver() { DesiredCapabilities dCaps = new DesiredCapabilities(); dCaps.setCapability("takesScreenshot",true); dCaps.setCapability( PhantomJSDriverService.PHANTOMJS_PAGE_SETTINGS_PREFIX + "userAgent","Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/53 (KHTML,like Gecko) Chrome/15.0.87" ); dCaps.setCapability( PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,"path_to_phantom_js" ); return dCaps; }
private WebDriver createWebDriver() { final DesiredCapabilities caps = new DesiredCapabilities(); caps.setJavascriptEnabled(true); caps.setCapability("takesScreenshot",true); caps.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,preparePhantomJSExecutable()); final WebDriver driver = new PhantomJSDriver(caps); return driver; }
@Override public DriverService createDriverService() { Builder builder = new PhantomJSDriverService.Builder(); if (driverExecutable != null) builder.usingPhantomJSExecutable(driverExecutable); if (ghostDriver != null) builder.usingGhostDriver(ghostDriver); if (port != null) builder.usingPort(port); if (environment != null) builder.withEnvironment(environment); if (logFile != null) builder.withLogFile(logFile); if (proxy != null) builder.withProxy(new Proxy(proxy)); if (commandLineArguments != null) builder.usingCommandLineArguments(commandLineArguments); if (ghostDriverCommandLineArguments != null) builder.usingGhostDriverCommandLineArguments(ghostDriverCommandLineArguments); return builder.build(); }
/** * * @param config * @return A FirefoxDriver. */ @Override public RemoteWebDriver make(HashMap<String,String> config) { DesiredCapabilities caps = new DesiredCapabilities(); caps.setJavascriptEnabled(true); if (System.getProperty(PHANTOMJS_PATH_PROPERTY) != null) { path = System.getProperty(PHANTOMJS_PATH_PROPERTY); } caps.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,path); return new PhantomJSDriver(caps); }
@Override public PhantomJSDriver initialize() { DesiredCapabilities capabilities = new DesiredCapabilities(); capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS,new String[] { "--webdriver-loglevel=NONE","--ignore-ssl-errors=true" }); return new PhantomJSDriver(capabilities); }
public WebDriverEx createPhantomJsWebDriver(SiteConfig siteConfig,DriverConfig driverConfig) throws IOException { File driverFile = createDriverFile(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY); DesiredCapabilities desiredCapabilities = createPhantomJsDesiredCapabilities(siteConfig,driverConfig); PhantomJSDriverService driverService = createPhantomJsDriverService(siteConfig,driverConfig,driverFile); return createWebDriver(driverService,desiredCapabilities,driverConfig); }
public PhantomJSDriverService createPhantomJsDriverService(SiteConfig siteConfig,File driverFile) { return createPhantomJsDriverServiceBuilder(siteConfig,driverFile).build(); }
/** * {@inheritDoc} */ @Override public String getPathproperty() { return PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY; }
public WebDriver getPhantomJsDriver(PhantomJSDriverService sev,Capabilities cap) { return new PhantomJSDriver(sev,cap); }
public WebDriver getPhantomJsDriver(String hubUrl,PhantomJSDriverService sev,Capabilities cap) throws MalformedURLException { return new RemoteWebDriver(new URL(hubUrl),cap); }
我们今天的关于用PhantomJS启动的RemoteWebdriver无法打开https url和webdriver.phantomjs()的分享就到这里,谢谢您的阅读,如果想了解更多关于CasperJS / PhantomJS不加载https页面、Flutter webview无法打开http协议网页的办法、Java Selenium封装--RemoteWebDriver、org.openqa.selenium.phantomjs.PhantomJSDriverService的实例源码的相关信息,可以在本站进行搜索。
本文标签: