How to Deploy a Hadoop Cluster (Part 3)

🌐 中文

English

This article mainly introduces how to clone VMware virtual machines and configure the cloned virtual machines.

Preface

In How to Deploy a Hadoop Cluster (Part 2), I have already configured the JDK dependencies and installed Hadoop on one virtual machine. Next, we need to clone enough VMware virtual machines to simulate the scenario of a large number of servers in a cluster.

How Many Servers Should Be Set Up

Although I recommend setting up 7 servers when building a Hadoop cluster:

• 3 servers for setting up the Hadoop DataNode cluster
• 1 server for setting up the NameNode node of the Hadoop cluster
• 1 server for setting up the SecondaryNameNode node of the Hadoop cluster
• 1 server for setting up the ResourceManager node of the Hadoop cluster
• 1 server for setting up the JobHistory node of the Hadoop cluster This can better simulate a production cluster scenario. However, due to the limited performance of personal computers, when learning about Hadoop clusters, we can set up 3 servers like this:
• One server for setting up the DataNode, NameNode, and JobHistory nodes of the Hadoop cluster
• One server for setting up the DataNode and SecondaryNameNode nodes of the Hadoop cluster
• One server for setting up the DataNode and ResourceManager nodes of the Hadoop cluster To achieve the goal of learning about Hadoop clusters to the greatest extent.

Note: It is essential to place the NameNode, SecondaryNameNode, and ResourceManager nodes on separate servers. This better simulates a production cluster scenario.

How to Correctly Clone a Virtual Machine (This step requires shutting down the virtual machine to be cloned first)

We can use VMware’s cloning feature to clone virtual machines, which allows for faster cluster setup. Right-click on the virtual machine, select Manage -> Clone to enter the cloning interface. In the cloning method, choose Full Clone, then click Next, select the name and location for the cloned virtual machine, and click Finish.

Then, open the settings of the newly cloned virtual machine -> Network Adapter -> Advanced, and click Generate next to the MAC address (MAC addresses must be different and unique within the same network to prevent network connection issues), then click OK.

How to Configure Virtual Machines

If you have cloned at least three virtual machines, the next step is to power on the newly created virtual machines and configure them.

• Note: Please do not power on more than one newly created virtual machine at the same time, as this may cause network connection issues. Complete the configuration on one virtual machine before powering on another for configuration.

  Configure Hostname and Static IP Address

  Configure Hostname

  1
  vim /etc/hostname
  

  Modify the content in the file to your desired hostname (it must be different from other hosts), then save and exit.

  Configure Static IP Address

  1
  vim /etc/sysconfig/network-scripts/ifcfg-ens33
  

  Change the IPADDR to your desired IP address (it must be in the same subnet as other Hadoop hosts), then save and exit.

After modifying the hostnames and IP addresses for all hosts, the configuration for this step is complete.

Configure the hosts File

To facilitate subsequent cluster configuration and prevent the need for extensive configuration changes if virtual machine IPs are modified, we need to configure the hosts file. This allows us to use hostnames to access other hosts in the cluster.

Configure the Linux hosts File

1
vim /etc/hosts

Add hostnames and IP addresses at the end in the following format: ip hostname For example:

1
2
3
192.168.138.128 hadoop101    # ip hostname
192.168.138.129 hadoop103
192.168.138.130 hadoop104

• Note: Replace the IP addresses and hostnames with those of your Hadoop hosts, and ensure each IP address corresponds to a hostname.

After configuring all hosts as described above, the Linux hosts file configuration is complete.

Configure the Windows hosts File

Open the hosts file located at C:\Windows\System32\drivers\etc\, and add hostnames and IP addresses at the end in the following format: ip hostname For example:

192.168.138.128 hadoop101
192.168.138.129 hadoop103
192.168.138.130 hadoop104

• Note: Replace the IP addresses and hostnames with those of your Hadoop hosts, and ensure each IP address corresponds to a hostname.

  Verify Successful Configuration

  On a Hadoop host, use the ping hostname command to test if the configuration is successful, for example:

  1
  ping hadoop103
  

  If it can ping successfully, the configuration is successful.

Configure SSH Passwordless Login

In Hadoop, when one server uses SSH to control other servers, it requires entering passwords. Therefore, we need to configure SSH passwordless login to save a lot of time entering passwords. The code is as follows:

1
ssh-keygen -t rsa

This is used to generate a key pair. Press Enter all the way. The generated keys are in the ~/.ssh/ directory, where id_rsa is the private key and id_rsa.pub is the public key. Then use:

1
ssh-copy-id hostname

To copy the public key to other hosts, where hostname is the hostname of the other host, for example:

1
ssh-copy-id hadoop103

• Note:
  1. When copying the public key, you will be prompted to enter the password of the other host. Enter it to copy successfully.
  2. When copying the public key, you need to copy it to all hosts, including itself, via the command; otherwise, login failures may occur.

After configuring SSH passwordless login for each host, the SSH passwordless login configuration for Hadoop hosts is complete.

####

FAQ

Why Shut Down Before Cloning

Because when cloning a virtual machine, all files of the virtual machine are copied, including some cache and temporary files. These files are automatically cleared after shutdown. If you clone without shutting down, these files may cause the cloned virtual machine to fail to start properly.

What Is the Purpose of the hosts File

When an operating system (such as Windows, Linux, macOS, etc.) performs domain name resolution, it first checks the entries in the hosts file. If there is a matching entry in the file, the system will directly use that IP address without querying an external DNS server.

Why Copying the Public Key to Other Hosts Enables Passwordless Login

When using SSH for remote login, the client decrypts the encrypted data sent by the server using its private key to prove it possesses the private key matching the public key in authorized_keys, thereby completing identity authentication and allowing login. The copied public key is added to the authorized_keys file, so in subsequent login processes, the client will automatically use this public key for verification.