1. 现象
HttpClient请求后，会在sender端报如下错误：
2018-04-28T10:08:46,144 ERROR [d78a9e9c-eac5-44ad-bf1b-4c0e8d387540 HiveServer2-Handler-Pool: Thread-90] common.HookHelper: failureHookRequest fail, url:http://url/func
com.foo.analytics.auth.org.apache.http.ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected
at com.foo.analytics.auth.org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:266) ~[foo-analytics-auth-1.0.0.jar:?]
at com.foo.analytics.auth.org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225) ~[foo-analytics-auth-1.0.0.jar:?]
at com.foo.analytics.auth.org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184) ~[foo-analytics-auth-1.0.0.jar:?]
at com.foo.analytics.auth.org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) ~[foo-analytics-auth-1.0.0.jar:?]
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) ~[?:1.8.0_144]
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) ~[?:1.8.0_144]
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) ~[?:1.8.0_144]
at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[?:1.8.0_144]
at java.io.Reader.read(Reader.java:140) ~[?:1.8.0_144]
at com.foo.analytics.auth.org.apache.http.util.EntityUtils.toString(EntityUtils.java:227) ~[foo-analytics-auth-1.0.0.jar:?]
at com.foo.analytics.auth.org.apache.http.util.EntityUtils.toString(EntityUtils.java:308) ~[foo-analytics-auth-1.0.0.jar:?]
at com.foo.analytics.auth.hive.common.HookHelper.smartEngineRequest(HookHelper.java:219) [foo-analytics-auth-1.0.0.jar:?]
at com.foo.analytics.auth.hive.Hook.HivePreSemanticAnalyzerHook.func(HivePreSemanticAnalyzerHook.java:44) [foo-analytics-auth-1.0.0.jar:?]
2. 排查思路
- 首先确定了在webserver端的日志里根本找不到此请求对应的unique id, 也就是说，根本没有进入controller
- 经google，发现HttpClient和nginx都有对请求的timeout设置，分别做如下改动：
HttpClient端把timeout从60s改为10min：
PoolingHttpClientConnectionManager connManager = new PoolingHttpClientConnectionManager();
// Create socket configuration
SocketConfig socketConfig = SocketConfig.custom().setSoKeepAlive(true)
.setSoTimeout(180000).setSoReuseAddress(true).setTcpNoDelay(true).build();
// Configure the connection manager to use socket configuration either
// by default or for a specific host.
connManager.setDefaultSocketConfig(socketConfig);
// Validate connections after 1 minute of inactivity
connManager.setValidateAfterInactivity(180000);
connManager.setMaxTotal(100);
connManager.setDefaultMaxPerRoute(20);
// Create global request configuration
RequestConfig defaultRequestConfig = RequestConfig.custom()
.setCookieSpec(CookieSpecs.DEFAULT).setExpectContinueEnabled(true)
.setConnectTimeout(180000).setSocketTimeout(180000)
.setConnectionRequestTimeout(180000).build();
CloseableHttpClient hc = HttpClients.custom().setConnectionManager(connManager)
.setDefaultRequestConfig(defaultRequestConfig).setDefaultSocketConfig(socketConfig)
.build();
String encodedMsg = new String(Files.readAllBytes(Paths.get(args[0])), UTF_8);
UrlEncodedFormEntity entity = new UrlEncodedFormEntity(Lists.newArrayList(new BasicNameValuePair("message", encodedMsg)), UTF_8);
HttpUriRequest httpPost = HttpUtils.post("http://url/func", entity);
try (CloseableHttpResponse response = hc.execute(httpPost)) {
// ...
}
在nginx对应的upstream设置：
proxy_connect_timeout 300s;
proxy_read_timeout 300s;
proxy_send_timeout 300s;
client_body_timeout 300s;
send_timeout 300s;
消息体过大问题
1. 现象
通过HttpClient发送POST请求到SpringBoot（Embedded Tomcat）时，如果send的data较小，则一切正常；如果data较大（~3MB），则controller端接收到的String为空。
2. 排查思路
- 因为请求是先经过nginx，再redirect到两台webserver。所以先看了下nginx的access log，发现请求会经过nginx并不报错。排除。
- 直接把sending的message保存到文件，模拟一个test case，直接将请求发送到一台具体的webserver。在controller接收端打印接收到的message和对应的size，为空。
- google了下，发现nginx、SpringBoot和tomcat都有对request size的限制：
在application.properties里添加如下参数：
spring.servlet.multipart.max-file-size=30MB
spring.servlet.multipart.max-request-size=30MB
server.tomcat.max-http-post-size=-1
在nginx对应的upstream里添加：
client_max_body_size 10M;
REFERENCE:
