Hive throws Socket Timeout Exception: Read time out

Hive on our gateway applies thrift-server for the purpose of connecting to metadata at remote MySQL database. Yesterday, there was an error complaining about 'MetaException(message:Got exception: org.apache.thrift.transport.TTransportException java.net.SocketTimeoutException: Read timed out)', and it just emerged out of the void.

Firstly, I checked which remote thrift-server current Hive is connecting to via netstat command:

# On terminal_window_1
$ nohup hive -e "use target_db;" &

# On another terminal_window_2
$ netstat -anp | grep 26232

Invoke the second command on terminal_window_2 right after executing the first command, in which, 26232 is the first command's PID. This will end up showing all network connections the specific process is opening:

Proto Recv-Q Send-Q Local Address               Foreign Address                State       PID/Program name
tcp        0      0 127.0.0.1:21786             192.168.7.11:9083              ESTABLISHED 26232/java

Obviously, current hive is connecting to a thrift-server residing at 192.168.7.11:9083. Execute command `telnet 192.168.7.11 9083` to verify whether it is accessible to the target IP and port. In our case, it is accessible.

96:/root>telnet 192.168.7.11 9083
Trying 192.168.7.11...
Connected to undefine.inidc.com.cn (192.168.7.11).
Escape character is '^]'.

Then I switch to that thrift-server, find the process via `ps aux | grep HiveMetaStore`, kill it and then start it again by `hive --service metastore`, it complains 'Access denied for user 'Diablo'@'d82.hide.cn''. Apparently, it is related with MySQL privileges. run the following command in MySQL as user root:

GRANT ALL PRIVILEGES ON Diablo.* TO 'Diablo'@'d82.hide.cn' IDENTIFIED BY 'Diablo'

Restart thrift-server again, and eventually, everything's back to normal.