How to Deploy a Hadoop Cluster (Part 6)

🌐 中文 English

This article mainly introduces the final steps of Hadoop deployment: initialization, startup, shutdown, and deletion.

Introduction

After the previous steps of virtual machine deployment and Hadoop cluster installation and configuration with its dependencies, we now need to complete the final step of the Hadoop cluster—initialization, startup, shutdown, and deletion.

Configuring the workers Configuration File

First, let’s add a configuration file to specify all nodes in the Hadoop cluster.

cd /path/to/hadoopfile/etc/hadoop/
vim workers

Note: /path/to/hadoopfile/ is the Hadoop installation directory. Modify it as needed. Add the hostnames of all nodes, one per line, for example:
```
1
2
hadoopname01
hadoopname02
```

Initializing HDFS

cd /path/to/hadoopfile/ # Enter the Hadoop installation directory

Note: /path/to/hadoopfile/ is the Hadoop installation directory. Modify it as needed. Before starting HDFS for the first time, you need to format the NameNode:
```
1
hdfs namenode -format 
```
Note: Formatting the NameNode should be executed on the NameNode node. The 2NN (Secondary NameNode) node does not need to be formatted.

Starting the Cluster

Starting HDFS

First, switch to a non-root account, as explained in a previous article.

su - hadoopName # Switch to the hadoop user

Note: hadoopName is a non-root user. Modify it as needed.

start-dfs.sh # Start HDFS

Note: Starting HDFS should be executed on the NameNode node. Other nodes like the 2NN (Secondary NameNode) do not need to execute the startup command.

Starting YARN

start-yarn.sh # Start YARN

Note: Starting YARN should be executed on the ResourceManager node. Other nodes do not need to execute the startup command.

Starting the History Server

/path/to/hadoopfile/bin/mapred --daemon start historyserver

Note: /path/to/hadoopfile/ is the Hadoop installation directory. Modify it as needed. Starting the history server should be executed on the JobHistoryServer node. Other nodes do not need to execute the startup command.
Checking Cluster Status
```
1
jps
```
If you see processes other than jps running on the corresponding nodes, it indicates the cluster has started successfully.

Shutting Down the Cluster

Stopping the History Server

/path/to/hadoopfile/bin/mapred --daemon stop historyserver

Note: /path/to/hadoopfile/ is the Hadoop installation directory. Modify it as needed. Stopping the history server should be executed on the JobHistoryServer node. Other nodes do not need to execute the shutdown command.
Stopping YARN
```
1
stop-yarn.sh # Stop YARN
```
Note: Stopping YARN should be executed on the ResourceManager node. Other nodes do not need to execute the shutdown command.
Stopping HDFS
```
1
stop-dfs.sh 
```
Note: Stopping HDFS should be executed on the NameNode node. Other nodes like the 2NN (Secondary NameNode) do not need to execute the shutdown command.
Checking Cluster Status
```
1
jps
```
If you do not see any processes other than jps, it indicates the cluster has been shut down successfully.

Deleting the Cluster

If configuration issues arise, you can delete the cluster and redeploy it.

rm -rf /path/to/data

Note: /path/to/data is the Hadoop data directory. Modify it as needed. This deletes the Hadoop data directory.
```
1
rm -rf /path/to/logs
```
Note: /path/to/logs is the Hadoop logs directory. Modify it as needed. This deletes the Hadoop logs directory.

Then, reinitialize and format HDFS.

Note: Each DataNode node needs to delete its corresponding Hadoop data directory, and the Hadoop logs directory on each node.

FAQ

Why does the cluster report an error after configuring workers?

1️⃣ Ensure each node’s hostname is correctly configured in the workers file, and also ensure the corresponding hostname and IP address are correctly configured in the /etc/hosts file. 2️⃣ The workers file can only contain one node’s hostname per line; multiple hostnames cannot be on the same line. 3️⃣ There should be no spaces or comment lines.

Why format the NameNode?

The main purpose of formatting the NameNode is to initialize or reinitialize HDFS metadata, including the file system directory structure, file block information, permissions, etc.

Correct startup and shutdown order and why

Startup Order

1️⃣ Start NameNode (HDFS metadata management) 2️⃣ Start Secondary NameNode (merge metadata logs, optional) 3️⃣ Start DataNode (store HDFS data blocks) 4️⃣ Start ResourceManager (YARN task scheduling) 5️⃣ Start NodeManager (worker nodes that execute computing tasks) 6️⃣ Start JobHistory Server (store historical records of completed jobs)

Shutdown Order

1️⃣ Stop JobHistory Server (ensure historical data is completely written) 2️⃣ Stop NodeManager (prevent running tasks from being forcibly interrupted) 3️⃣ Stop ResourceManager (prevent YARN from accepting new tasks) 4️⃣ Stop DataNode (avoid HDFS misjudging data loss) 5️⃣ Stop Secondary NameNode (avoid log merge errors after NameNode shutdown) 6️⃣ Stop NameNode (shut down last to ensure metadata integrity)

Why follow this order?

1️⃣ HDFS must start first because YARN and JobHistory Server depend on HDFS. 2️⃣ DataNode must start after NameNode; otherwise, it may trigger safe mode (if some DataNodes have not responded or if HDFS data is inconsistent, the NameNode will enter Safe Mode to prevent erroneous data from being written to the system). 3️⃣ JobHistory Server needs to start after YARN; otherwise, job history cannot be saved. 4️⃣ The shutdown order is reversed to ensure metadata integrity and prevent the NameNode from misjudging data loss.

Why delete the data and logs directories?

1️⃣ The old data directory stores previous block data, while formatting the NameNode reinitializes the metadata. If old data still exists, it may cause mismatches between the new metadata and actual storage, leading to errors or data inconsistencies. 2️⃣ Log confusion: Old logs in the logs directory may mix with new system debugging information, causing trouble for troubleshooting.