Wednesday, January 21, 2015

Commonly-Used Commands For Hadoop


YARN Service (ResourceManager + NodeManager):
$HADOOP_HOME/sbin/stop-yarn.sh
$HADOOP_HOME/sbin/start-yarn.sh

HDFS Service (NameNode + DataNode):
$HADOOP_HOME/sbin/stop-dfs.sh
$HADOOP_HOME/sbin/start-dfs.sh

Balancer Service (Do Balance):
$HADOOP_HOME/sbin/stop-balancer.sh
$HADOOP_HOME/sbin/start-balancer.sh

Start DataNode, NameNode, NodeManager, ResourceManager Service respectively:
$HADOOP_HOME/sbin/hadoop-daemon.sh start datanode
$HADOOP_HOME/sbin/hadoop-daemon.sh start namenode
$HADOOP_HOME/sbin/yarn-daemon.sh start nodemanager
$HADOOP_HOME/sbin/yarn-daemon.sh start resourcemanager

Start zkfc(DFSZKFailoverController) Service:
./sbin/hadoop-daemon.sh start zkfc
./bin/hdfs zkfc   (which will show detailed launching information for debugging)

ZooKeeper(QuorumPeerMain) Service:
$ZK_HOME/bin/zkServer.sh stop
$ZK_HOME/bin/zkServer.sh start
$ZK_HOME/bin/zkServer.sh restart
$ZK_HOME/bin/zkServer.sh status

Start JournalNode Service:
$HADOOP_HOME/sbin/hadoop-daemon.sh start journalnode

NameNode-HA-Related Operation:
Check NameNode Status(active/standby):    hdfs haadmin -getServiceState <serviceId>
Transfer from standby to active manually:  hdfs haadmin -transitionToActive <serviceId>

List Current Undone MapReduce Task:
mapred job -list

Kill MapReduce Task:
mapred job -kill <jobid>

List Current Undone YARN Task:
yarn application -list

Kill YARN Task:
yarn application -kill <YARN_Application_id>



© 2014-2017 jason4zhu.blogspot.com All Rights Reserved 
If transfering, please annotate the origin: Jason4Zhu

Note On NameNode HA

The overall procedure is well-explained in the references listed at the bottom. Here's just some essential points that I want to emphasize.

Here's the configurations for our Hadoop cluster, which is all related with NameNode HA:
core-site.xml:
<property>
    <name>fs.defaultFS</name>
    <value>hdfs://ns1</value>
</property>

<property>
    <name>ha.zookeeper.quorum</name>
    <value>644v4.mzhen.cn:2181,644v5.mzhen.cn:2181,644v6.mzhen.cn:2181</value>
</property>

hdfs-site.xml:
<property>
    <name>dfs.nameservices</name>
    <value>ns1</value>
</property>

<property>
     <name>dfs.ha.namenodes.ns1</name>
     <value>nn1,nn2</value>
</property>

<property>
     <name>dfs.namenode.rpc-address.ns1.nn1</name>
     <value>644v1.mzhen.cn:9000</value>
</property>

<property>
     <name>dfs.namenode.rpc-address.ns1.nn2</name>
     <value>644v2.mzhen.cn:9000</value>
</property>

<property>
     <name>dfs.namenode.http-address.ns1.nn1</name>
     <value>644v1.mzhen.cn:10001</value>
</property>

<property>
     <name>dfs.namenode.http-address.ns1.nn2</name>
     <value>644v2.mzhen.cn:10001</value>
</property>

<property>
     <name>dfs.namenode.shared.edits.dir</name>
     <value>qjournal://644v4.mzhen.cn:8485;644v5.mzhen.cn:8485;644v6.mzhen.cn:8485/ns1</value>
</property>

<property>
     <name>dfs.journalnode.edits.dir</name>
     <value>/home/data/hdfsdir/journal</value>
</property>

<property>
     <name>dfs.client.failover.proxy.provider.ns1</name>
     <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>

<property>
     <name>dfs.ha.fencing.methods</name>
     <value>sshfence</value>
</property>

<property>
     <name>dfs.ha.fencing.ssh.private-key-files</name>
     <value>/home/supertool/.ssh/id_rsa</value>
</property>

<property>
     <name>dfs.ha.automatic-failover.enabled</name>
     <value>true</value>
</property>


After Hadoop cluster is fully started, there are some checkpoints that should be verified to make sure NameNode HA is fully applied:

1. Nodes corresponding to "dfs.ha.namenodes.ns1" argument in hdfs-site.xml should have processes named "DFSZKFailoverController", "NameNode".
2. Nodes corresponding to "dfs.namenode.shared.edits.dir" argument in hdfs-site.xml should have process named "JournalNode".
3. Nodes corresponding to "ha.zookeeper.quorum" argument in core-site.xml should have process named "QuorumPeerMain".

The way to launch all the processes above, if needed to do so respectively, is listed as below:

QuorumPeerMain: The service for ZooKeeper.
        bin/zkServer.sh start
        bin/zkServer.sh status
        bin/zkServer.sh stop
        bin/zkServer.sh restart

JournalNode: In order for the Standby node to keep its state synchronized with the Active node, both nodes communicate with a group of separate daemons called "JournalNodes" (JNs)
        ./sbin/hadoop-daemon.sh stop journalnode
        ./sbin/hadoop-daemon.sh start journalnode

NameNode:
        ./sbin/hadoop-daemon.sh stop namenode
        ./sbin/hadoop-daemon.sh start namenode

DFSZKFailoverController:
        ./sbin/hadoop-daemon.sh stop zkfc
        ./sbin/hadoop-daemon.sh start zkfc
        Attention that if the above command fails to start with no explicit errors, you could try executing command `./bin/hdfs zkfc` so as to retrieve detailed information.

Lastly, Some common commands relevant with NameNode HA is listed here:
## Get the status of NameNode, active or standby.
hdfs haadmin -getServiceState nn1

## Transfer a NameNode to active manually, which requires 'dfs.ha.automatic-failover.enabled' be set to 'false'.
hdfs haadmin -transitionToActive nn1



Reference:
1. High Availability for Hadoop - Hortonworks
2. HDFS High Availability Using the Quorum Journal Manager - Apache Hadoop



Monday, January 19, 2015

One Simple Example To Illustrate The Difference Between $@ and $* In Shell

The following code is well self-explained:
bash test.sh 123 "456 7" 8

#!/bin/bash

echo "--\"\$*\"--"
for args in "$*"
do
    echo "#"$args"#"
done

echo "--\$*--"
for args in $*
do
    echo "#"$args"#"
done

echo "--\"\$@\"--"
for args in "$@"
do
    echo "#"$args"#"
done

echo "--\$@--"
for args in $@
do
    echo "#"$args"#"
done

Output:
--"$*"--
#123 456 7 8#
--$*--
#123#
#456#
#7#
#8#
--"$@"--
#123#
#456 7#
#8#
--$@--
#123#
#456#
#7#
#8#


Reference:
1. Difference between $@ and $* - CSDN



© 2014-2017 jason4zhu.blogspot.com All Rights Reserved 
If transfering, please annotate the origin: Jason4Zhu

Tuesday, January 13, 2015

Deal With Error "A semaphore set has to be created but the system limit for the maximum number of semaphore sets has been exceeded"

There are some time that a specific node in our Hadoop cluster will be stuck at ssh connection procedure, whereas ping command is traversable. When this node recovers by itself, I look through the system messages in '/var/log/messages', where all of the content is:
A semaphore set has to be created but the system limit for the maximum number of semaphore sets has been exceeded

There are some commands that we could apply to check on semaphore status. You can run ipcs -s to list all of the semaphores currently allocated on your system and then use ipcrm -s <id> to remove a semaphore (if you're reasonably sure it's no longer needed). You might also want to track down the program that created them (using information from ipcs -s -i <id>) to make sure it's not leaking semaphores.

Alternatively, you could check the semaphore setting in sysctl(8) via issuing command `sysctl -a | grep kernel.sem`, it is equivalent to invoke command `ipcs -ls`, which would show descriptions to all values:
$ /sbin/sysctl -a | grep kernel.sem
kernel.sem = 250 32000 32 128

$ ipcs -ls

------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767

The preceding error is related with the 'max semaphores system wide', thus we could increase this value by increasing either 'max number of arrays' or 'max semaphores per array'. However, there's something should be noticed, that when revising, we should always validate the equation: (max number of arrays)*(max semaphores per array)=(max semaphores system wide).

In my case, I changed the 'max number of arrays' to 1280, and 'max semaphores system wide' to 320000 correspondingly by appending "kernel.sem = 250 320000 32 1280" to '/etc/sysctl.conf' and execute `/sbin/sysctl -p`.



Reference:
1. Answer from ServerFault
2. Power Panel process fails to start - Parallels
3. 8.5. Setting Semaphore Parameters - RedHat
4. Setting the Kernel Parameters - SYBASE



© 2014-2017 jason4zhu.blogspot.com All Rights Reserved 
If transfering, please annotate the origin: Jason4Zhu

Essential Keypoint Of Hive Metastore And Notes On Configuring Remote Metastore

MARK.

It is well-explained in the following references.

Reference:
1. Hive: AdminManual+MetastoreAdmin
2. Configuring the Hive Metastore - Cloudera
3. Starting the Metastore - Cloudera
4. Hive Ports - Hortonworks



Sunday, January 11, 2015

Way To Set Up Replication In MySQL

Rationale

When it comes to talking about replication in MySQL, there should be one master node (in the sense that one slave could have only one master, although you may set multiple master nodes globally) and multiple slaves nodes.

Master node will write replication events to a special log called binary log, which will be read by slave nodes via IO thread and stores in a file called relay log. Eventually, SQL thread in slave nodes will reads events from relay log and then applies them to current slave MySQL server. As shown below:


As we can see, though multiple nodes in MySQL cluster there are, it is still single point of failure (SPOF) given its semantics that "when master node breaks down, the service supplied by MySQL is down", which is not virtually the same as a decentralized distributed system.

Configuration Process

Find the valid my.ini or my.cnf used by MySQL server:
[mysql@644v3 mysql]$ $MYSQL_HOME/libexec/mysqld --verbose --help | grep -B 1 -E "my.(cnf|ini)" --color
Default options are read from the following files in the given order:
/etc/my.cnf ~/.my.cnf /usr/local/mysql/etc/my.cnf 

Edit one of the configuration files listed above (in my case, "/usr/local/mysql/etc/my.cnf" is used) on master node. The following configuration enables binary logging using a log file name prefix of mysql-bin, and configure a server ID of 1:
[mysqld]
log-bin=mysql-bin
server-id=1

Likewise, append the following configuration to my.cnf on slave nodes, in which, server-id should be unique in each slave node:
[mysqld]
server-id=2

read_only=1

After setting, we need to restart master and slave node respectively.
shell> $MYSQL_HOME/share/mysql/mysql.server stop
shell> $MYSQL_HOME/share/mysql/mysql.server start

As we can see from the rationale section, only operations via SQL on master node from the time binary log is on will be synchronized to slave nodes, thus we have to sync existing MySQL data to slave nodes ourselves.

Before exporting current data, we have to make the whole process as a transaction for consistency and integrity. Therefore, the following command is executed on master node to disable statements like insert/update/delete/replace/alter.
mysql> FLUSH TABLES WITH READ LOCK;

Then we can dump current data out to local filesystem and record current binary log's name and position:
# Syntax of mysqldump
# shell> mysqldump [options] db_name [tbl_name ...]
# shell> mysqldump [options] --databases db_name ...
# shell> mysqldump [options] --all-databases

shell> $MYSQL_HOME/bin/mysqldump -u repl -p test > ./mysql_snapshot.sql


#show current binary log's name and position
mysql> show master status;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000001 |     2114 |              |                  |
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec)

When the above oprations done, remember to release the read lock via `UNLOCK TABLES;`.

Scp the dump file "mysql_snapshot.sql" to slave nodes and restore it via command, in which, test is the name of database:
mysql -uroot -p test < mysql_snapshot.sql

Next, we are going to add a MySQL user on master node especially for slave node connection with the only replication privilege.
mysql> CREATE USER 'repl'@'%' IDENTIFIED BY 'slavepass';
mysql> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%';

There's something beside the main topic: When specifying '%', it stands for accepting login sessions from any host, EXCEPT from 'localhost'. Actually, 'localhost' is special in mysql, it means a connection over a unix socket (or named pipes on windows I believe) as opposed to a TCP/IP socket. using '%' as the host does not include 'localhost'. Thus, create a user named 'repl'@'localhost' if necessary.

Eventually, we have to specify the start position of synchronizing from master node, which is recorded above (in my case, the position is 2114 and filename is mysql-bin.000001), and start slave threads for replication:
mysql> CHANGE MASTER TO
        MASTER_HOST='master_host',
        MASTER_USER='repl',
        MASTER_PASSWORD='slavepass',
        MASTER_LOG_FILE='mysql-bin.000001',
        MASTER_LOG_POS=2114;

mysql> START SLAVE;

If we intend to see the slave's status, we could simply invoke `show slave status;` in MySQL prompt, which will show information like the progress of synchronization, etc.

We could verify the replication in MySQL by executing an insert statement in master node, and the newly-added record appears in slave node synchronously.


Reference
1. MySQL Official Documentation - Replication Configuration
2. MySQL Replication Configuration - CSDN
3. what-exactly-does-flush-tables-with-read-lock-do
4. creating-a-mysql-user-without-host-specified
5. MySQL Official Documentation - mysqldump



© 2014-2017 jason4zhu.blogspot.com All Rights Reserved 
If transfering, please annotate the origin: Jason4Zhu

Sunday, January 4, 2015

Deal With "java.lang.OutOfMemoryError: unable to create new native thread" Error in Java

Upon upgrading CentOS from release 5.3 to 6.4, "java.lang.OutOfMemoryError: unable to create new native thread" is thrown after starting NodeManager in Hadoop for a while. This is because in CentOS 6.x, there is limit to the number of created threads / processes, whose default setting is 1024. In CentOS 5.x, it can be more than 40000 or no limit.
//---- CentOS 5.3 ----
$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 530432
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 530432        //Comparison 1
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

//---- CentOS 6.4 ----
$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 514891
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024          //Comparison 2
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Thus, we have to enlarge the maximum number of user processes by increasing the number of the following parameter in "/etc/security/limits.d/90-nproc.conf":
*          soft    nproc     1024

Eventually, start a new session for the change to take effect.