场景描述：
有时候我们会遇到需要抓取某些app的数据的需求，比如新闻客户端，或者微信的数据接口
这时候我们可以用wireshark来截取流量，或者fildder，或者charles。
但是如果数据接口是https的话，我们需要自己颁发ca证书，需要手机信任证书，下面是charles截取https流量的方法，写的很详细
https://www.cnblogs.com/weiming4219/p/7908668.html
这里android反倒不容易，因为android的http proxy不是全局代理，是仅限于浏览器的http proxy，需要额外安装软件达到全局代理的目的，而且还需要root，现在的手机厂商已经很少开放root权限了，我前几年还会折腾root，现在大家也越来越少折腾root了，所以不方便。
而ios的wifi设置下面的http proxy是全局的，所有app都生效了。所以这里如果有iphone的话，会比较方便。
ios只需要安装ca证书，并且信任即可。
但是这里要介绍的不是通过charles来达成流量监听目的的做法，而是用编程，我们自己借助littleproxy的http proxy代理服务器达成我们的目的
介绍一下littleproxy是一款基于java的http代理工具，包括了中间人攻击模块。
https://github.com/adamfisk/LittleProxy
具体可以点进去看看。
接下来我们实操
新建一个java工程，maven项目
把依赖添加进去
<dependency>
<groupId>org.littleshoot</groupId>
<artifactId>littleproxy</artifactId>
<version>1.1.2</version>
</dependency>
<dependency>
<groupId>com.github.ganskef</groupId>
<artifactId>littleproxy-mitm</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>net.lightbody.bmp</groupId>
<artifactId>mitm</artifactId>
<version>2.1.4</version>
</dependency>核心代码就80行
public static void way1(){
try {
HttpProxyServer server =
DefaultHttpProxyServer.bootstrap()
.withAddress(new InetSocketAddress("192.168.55.100",9090))
//.withPort(9090) // for both HTTP and HTTPS
.withManInTheMiddle(new CertificateSniffingMitmManager())
.withFiltersSource(new HttpFiltersSourceAdapter() {
public HttpFilters filterRequest(HttpRequest originalRequest, ChannelHandlerContext ctx) {
return new HttpFiltersAdapter(originalRequest) {
@Override
public HttpResponse clientToProxyRequest(HttpObject httpObject) {
// TODO: implement your filtering here
//System.out.println(httpObject);
// if(httpObject instanceof DefaultHttpRequest){
// System.out.println("req:"+((DefaultHttpRequest) httpObject).getUri());
// System.out.println(originalRequest.getUri());
// }
return null;
}
@Override
public HttpObject serverToProxyResponse(HttpObject httpObject) {
try{
responseMap.putIfAbsent(originalRequest,new MyResponse());
MyResponse myResponse = responseMap.get(originalRequest);
if (httpObject instanceof HttpResponse) {
myResponse.setHttpHeaders(((HttpResponse) httpObject).headers());
} else if (httpObject instanceof HttpContent) {
HttpHeaders httpHeaders = myResponse.getHttpHeaders();
if(httpHeaders!=null){
String contentType = httpHeaders.get("Content-Type");
if(contentType!=null && !contentType.contains("image") && !contentType.contains("audio")
&&!contentType.contains("zip") && !contentType.contains("application/octet-stream")
){
ByteBuf buf = ((HttpContent) httpObject).content();
buf.markReaderIndex();
byte[] array = new byte[buf.readableBytes()];
buf.readBytes(array);
buf.resetReaderIndex();
myResponse.appendByte(array);
}
}
}
if(httpObject instanceof LastHttpContent){
if(myResponse.getContent()!=null){
System.out.println(originalRequest.getUri());
HttpHeaders httpHeaders = myResponse.getHttpHeaders();
myResponse.printHeader();
String ce = httpHeaders.get("Content-Encoding");
if(ce!=null&&ce.contains("gzip")){
if(myResponse.getContent()!=null){
ByteArrayInputStream bais = new ByteArrayInputStream(myResponse.getContent());
GZIPInputStream gzis = new GZIPInputStream(bais);
byte[] decompressedData = IOUtils.toByteArray(gzis);
System.out.println(new String(decompressedData,"utf-8"));
}
}else {
if(myResponse.getContent()!=null){
System.out.println(new String(myResponse.getContent(),"utf-8"));
}
}
}
}
}catch (Exception e){
e.printStackTrace();
}
return httpObject;
}
};
}
})
.start();
} catch (Exception e) {
e.printStackTrace();
}
}
public static void main(String[] args){
way1();
}public class MyResponse {
private StringBuffer body = new StringBuffer();
private HttpHeaders httpHeaders;
private byte [] content;
public byte[] getContent() {
return content;
}
public synchronized void appendByte(byte [] array){
if(content==null){
content = array;
}else {
byte []temp = new byte[content.length+array.length];
System.arraycopy(content,0,temp,0,content.length);
System.arraycopy(array,0,temp,content.length,array.length);
content = temp;
}
}
public StringBuffer getBody() {
return body;
}
public void printHeader(){
if(httpHeaders!=null){
httpHeaders.forEach(stringStringEntry -> {
System.out.println(stringStringEntry.getKey()+":"+stringStringEntry.getValue());
});
}
}
public void setBody(StringBuffer body) {
this.body = body;
}
public HttpHeaders getHttpHeaders() {
return httpHeaders;
}
public void setHttpHeaders(HttpHeaders httpHeaders) {
this.httpHeaders = httpHeaders;
}
}
首先我们要绑定ip到本地机器，设置http代理的端口，然后设置中间人攻击模块
DefaultHttpProxyServer.bootstrap()
.withAddress(new InetSocketAddress("192.168.55.100",9090))
//.withPort(9090) // for both HTTP and HTTPS
.withManInTheMiddle(new CertificateSniffingMitmManager())如果你用默认的withPort的话，它是绑定在127.0.0.1的，那么局域网的其它设备，手机等是没法设置代理的。
然后他会生成两个文件，请搭建一个静态文件服务器，让ios的safari浏览器访问pem文件，然后safari会提示你是否安装该文件描述，请安装
接下来设置我们的httpfilter
.withFiltersSource(new HttpFiltersSourceAdapter() {
public HttpFilters filterRequest(HttpRequest originalRequest, ChannelHandlerContext ctx) {
return new HttpFiltersAdapter(originalRequest) {
@Override
public HttpResponse clientToProxyRequest(HttpObject httpObject) {
// TODO: implement your filtering here
//System.out.println(httpObject);
// if(httpObject instanceof DefaultHttpRequest){
// System.out.println("req:"+((DefaultHttpRequest) httpObject).getUri());
// System.out.println(originalRequest.getUri());
// }
return null;
}
@Override
public HttpObject serverToProxyResponse(HttpObject httpObject) {
try{
responseMap.putIfAbsent(originalRequest,new MyResponse());
MyResponse myResponse = responseMap.get(originalRequest);
if (httpObject instanceof HttpResponse) {
myResponse.setHttpHeaders(((HttpResponse) httpObject).headers());
} else if (httpObject instanceof HttpContent) {
HttpHeaders httpHeaders = myResponse.getHttpHeaders();
if(httpHeaders!=null){
String contentType = httpHeaders.get("Content-Type");
if(contentType!=null && !contentType.contains("image") && !contentType.contains("audio")
&&!contentType.contains("zip") && !contentType.contains("application/octet-stream")
){
ByteBuf buf = ((HttpContent) httpObject).content();
buf.markReaderIndex();
byte[] array = new byte[buf.readableBytes()];
buf.readBytes(array);
buf.resetReaderIndex();
myResponse.appendByte(array);
}
}
}
if(httpObject instanceof LastHttpContent){
if(myResponse.getContent()!=null){
System.out.println(originalRequest.getUri());
HttpHeaders httpHeaders = myResponse.getHttpHeaders();
myResponse.printHeader();
String ce = httpHeaders.get("Content-Encoding");
if(ce!=null&&ce.contains("gzip")){
if(myResponse.getContent()!=null){
ByteArrayInputStream bais = new ByteArrayInputStream(myResponse.getContent());
GZIPInputStream gzis = new GZIPInputStream(bais);
byte[] decompressedData = IOUtils.toByteArray(gzis);
System.out.println(new String(decompressedData,"utf-8"));
}
}else {
if(myResponse.getContent()!=null){
System.out.println(new String(myResponse.getContent(),"utf-8"));
}
}
}
}
}catch (Exception e){
e.printStackTrace();
}
return httpObject;
}
};
}
})
.start();代码可以看出我们过滤了image，zip等二进制包，同样对于gzip压缩的json格式，也做了很好的处理。
可以看看具体效果：
这里就是返回的百度的内容，可以看出https又如何？还是被截获流量，其它app里面的内容同理可以做到。
开源中国app的接口：