Android平台下使用OkHttp采集数据
OkHttp是Android普遍应用的Http第三方库,可以直接使用OkHttp模拟客户端请求,配合爬虫代理IP进行APP或WAP数据采集。 OkHttp的优势:
套接字共享,连接池自动化管理减少请求延时。
支持GZIP压缩,减少了下载数据带宽。
响应缓存,提高了网络速度。
自动dns多IP主机解析,实行多机负载均衡
数据采集过程中,目标网站会限制ip请求频率,因此需要使用爬虫代理,示例如下:
import okhttp3.*;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Proxy;
import java.util.concurrent.TimeUnit;
public class OkHttp {
// 代理服务器(产品官网 www.16yun.cn)
final static String proxyHost = "t.16yun.cn";
final static Integer proxyPort = 31111;
// 代理验证信息
final static String proxyUser = "USERNAME";
final static String proxyPass = "PASSWORD";
static OkHttpClient client = null;
static {
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort));
Authenticator proxyAuthenticator = new Authenticator() {
public Request authenticate(Route route, Response response) {
String credential = Credentials.basic(proxyUser, proxyPass);
return response.request().newBuilder()
.header("Proxy-Authorization", credential)
.build();
}
};
client = new OkHttpClient().newBuilder()
.connectTimeout(5, TimeUnit.SECONDS)
.readTimeout(5, TimeUnit.SECONDS)
.proxy(proxy)
.proxyAuthenticator(proxyAuthenticator)
.connectionPool(new ConnectionPool(5, 1, TimeUnit.SECONDS))
.build();
}
public static Response doGet() throws IOException {
// 要访问的目标页面
String targetUrl = "http://httpbin.org/ip";
Request request = new Request.Builder()
.url(targetUrl)
.build();
Response response = client.newCall(request).execute();
return response;
}
public static void main(String[] args) throws IOException {
Response response1 = doGet();
System.out.println("GET请求返回结果:");
System.out.println(response1.body().string());
}
}
Android平台下使用OkHttp采集数据
xiaotaomi
会员积分:6520
OkHttp是Android普遍应用的Http第三方库,可以直接使用OkHttp模拟客户端请求,配合爬虫代理IP进行APP或WAP数据采集。 OkHttp的优势:
套接字共享,连接池自动化管理减少请求延时。
支持GZIP压缩,减少了下载数据带宽。
响应缓存,提高了网络速度。
自动dns多IP主机解析,实行多机负载均衡
数据采集过程中,目标网站会限制ip请求频率,因此需要使用爬虫代理,示例如下:
import okhttp3.*;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Proxy;
import java.util.concurrent.TimeUnit;
public class OkHttp {
// 代理服务器(产品官网 www.16yun.cn)
final static String proxyHost = "t.16yun.cn";
final static Integer proxyPort = 31111;
// 代理验证信息
final static String proxyUser = "USERNAME";
final static String proxyPass = "PASSWORD";
static OkHttpClient client = null;
static {
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort));
Authenticator proxyAuthenticator = new Authenticator() {
public Request authenticate(Route route, Response response) {
String credential = Credentials.basic(proxyUser, proxyPass);
return response.request().newBuilder()
.header("Proxy-Authorization", credential)
.build();
}
};
client = new OkHttpClient().newBuilder()
.connectTimeout(5, TimeUnit.SECONDS)
.readTimeout(5, TimeUnit.SECONDS)
.proxy(proxy)
.proxyAuthenticator(proxyAuthenticator)
.connectionPool(new ConnectionPool(5, 1, TimeUnit.SECONDS))
.build();
}
public static Response doGet() throws IOException {
// 要访问的目标页面
String targetUrl = "http://httpbin.org/ip";
Request request = new Request.Builder()
.url(targetUrl)
.build();
Response response = client.newCall(request).execute();
return response;
}
public static void main(String[] args) throws IOException {
Response response1 = doGet();
System.out.println("GET请求返回结果:");
System.out.println(response1.body().string());
}
}
20-10-09 15:05
725
0
回复
暂无评论