POST请求超时问题和消息体过大问题

超时问题

1. 现象

HttpClient请求后，会在sender端报如下错误：

2018-04-28T10:08:46,144 ERROR [d78a9e9c-eac5-44ad-bf1b-4c0e8d387540 HiveServer2-Handler-Pool: Thread-90] common.HookHelper: failureHookRequest fail, url:http://url/func

com.foo.analytics.auth.org.apache.http.ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected

    at com.foo.analytics.auth.org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:266) ~[foo-analytics-auth-1.0.0.jar:?]

    at com.foo.analytics.auth.org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225) ~[foo-analytics-auth-1.0.0.jar:?]

    at com.foo.analytics.auth.org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184) ~[foo-analytics-auth-1.0.0.jar:?]

    at com.foo.analytics.auth.org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) ~[foo-analytics-auth-1.0.0.jar:?]

    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) ~[?:1.8.0_144]

    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) ~[?:1.8.0_144]

    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) ~[?:1.8.0_144]

    at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[?:1.8.0_144]

    at java.io.Reader.read(Reader.java:140) ~[?:1.8.0_144]

    at com.foo.analytics.auth.org.apache.http.util.EntityUtils.toString(EntityUtils.java:227) ~[foo-analytics-auth-1.0.0.jar:?]

    at com.foo.analytics.auth.org.apache.http.util.EntityUtils.toString(EntityUtils.java:308) ~[foo-analytics-auth-1.0.0.jar:?]

    at com.foo.analytics.auth.hive.common.HookHelper.smartEngineRequest(HookHelper.java:219) [foo-analytics-auth-1.0.0.jar:?]

    at com.foo.analytics.auth.hive.Hook.HivePreSemanticAnalyzerHook.func(HivePreSemanticAnalyzerHook.java:44) [foo-analytics-auth-1.0.0.jar:?]

2. 排查思路

1. 首先确定了在webserver端的日志里根本找不到此请求对应的unique id, 也就是说，根本没有进入controller
2. 经google，发现HttpClient和nginx都有对请求的timeout设置，分别做如下改动：

HttpClient端把timeout从60s改为10min：

PoolingHttpClientConnectionManager connManager = new PoolingHttpClientConnectionManager();

// Create socket configuration

SocketConfig socketConfig = SocketConfig.custom().setSoKeepAlive(true)

        .setSoTimeout(180000).setSoReuseAddress(true).setTcpNoDelay(true).build();




// Configure the connection manager to use socket configuration either

// by default or for a specific host.

connManager.setDefaultSocketConfig(socketConfig);

// Validate connections after 1 minute of inactivity

connManager.setValidateAfterInactivity(180000);

connManager.setMaxTotal(100);

connManager.setDefaultMaxPerRoute(20);




// Create global request configuration

RequestConfig defaultRequestConfig = RequestConfig.custom()

        .setCookieSpec(CookieSpecs.DEFAULT).setExpectContinueEnabled(true)

        .setConnectTimeout(180000).setSocketTimeout(180000)

        .setConnectionRequestTimeout(180000).build();




CloseableHttpClient hc = HttpClients.custom().setConnectionManager(connManager)

        .setDefaultRequestConfig(defaultRequestConfig).setDefaultSocketConfig(socketConfig)

        .build();







String encodedMsg = new String(Files.readAllBytes(Paths.get(args[0])), UTF_8);

UrlEncodedFormEntity entity = new UrlEncodedFormEntity(Lists.newArrayList(new BasicNameValuePair("message", encodedMsg)), UTF_8);

HttpUriRequest httpPost = HttpUtils.post("http://url/func", entity);




try (CloseableHttpResponse response = hc.execute(httpPost)) {

    // ...

}

在nginx对应的upstream设置：

proxy_connect_timeout 300s;

proxy_read_timeout 300s;

proxy_send_timeout 300s;

client_body_timeout 300s;

send_timeout 300s;

---

消息体过大问题

1. 现象

通过HttpClient发送POST请求到SpringBoot（Embedded Tomcat）时，如果send的data较小，则一切正常；如果data较大（~3MB），则controller端接收到的String为空。

2. 排查思路

1. 因为请求是先经过nginx，再redirect到两台webserver。所以先看了下nginx的access log，发现请求会经过nginx并不报错。排除。
2. 直接把sending的message保存到文件，模拟一个test case，直接将请求发送到一台具体的webserver。在controller接收端打印接收到的message和对应的size，为空。
3. google了下，发现nginx、SpringBoot和tomcat都有对request size的限制：

在application.properties里添加如下参数：

spring.servlet.multipart.max-file-size=30MB

spring.servlet.multipart.max-request-size=30MB

server.tomcat.max-http-post-size=-1

在nginx对应的upstream里添加：

client_max_body_size 10M;

REFERENCE: 

1. https://stackoverflow.com/questions/50074707/received-field-is-empty-when-the-size-is-big-in-spring-boot-post-method/50074885#50074885 

2. https://blog.csdn.net/LZJLZJLZJLZJLZJLZJ/article/details/79179279 

3. https://docs.spring.io/spring-boot/docs/current/reference/html/common-application-properties.html