How to Deploy a Hadoop Cluster (Part 1)

🌐 中文

English

This article mainly introduces the process of setting up the virtual machines required for a Hadoop 3.x cluster.

What is Hadoop

Hadoop is an open-source distributed computing framework developed by the Apache Foundation for storing and analyzing large-scale datasets. It is a reliable, scalable distributed computing ecosystem that allows users to process massive amounts of data on clusters of multiple computers using a simple programming model. It is one of the core components of big data.

Why Deploy a Hadoop Cluster

In the era of big data, data volume is growing exponentially, and traditional single-machine processing methods can no longer meet the demand. A Hadoop cluster can distribute data storage across multiple computers and process massive datasets through parallel computing, thereby improving the efficiency and speed of data processing.

What is Needed to Deploy a Hadoop Cluster

1. VMware 17.5 or higher. It is not recommended to install other 17.x versions because VMware has issued warnings about USB security vulnerabilities in those versions.
2. JDK and a compatible Hadoop 3.x version.
3. CentOS 7 ISO image file.
4. Since this blog will build a cluster with 4 virtual machines, it is recommended to have at least 16GB of RAM, an SSD with at least 160GB of free space, and a 16-core CPU (each VM should have over 4GB RAM, over 40GB storage, and over 4 cores).
5. Some basic Linux knowledge.

Deploying the Hadoop Cluster

The following describes how to deploy a Hadoop 3.x cluster in VMware virtual machines (please ensure you have VMware 17.5 or higher installed first):

1. Configure the CentOS System

Open VMware, click “Create a New Virtual Machine,” select Custom (advanced) mode -> Corresponding VMware version -> “I will install the operating system later” -> Select Linux, version choose Red Hat Enterprise Linux 7 64-bit (CentOS 7.x) -> Set the subsequent installation name, location, core count, memory size, network type NAT, storage size, and choose multi-file storage and its location (for the hard disk, select SCSI controller and SCSI disk), then click Finish. Then open the virtual machine settings, select CD/DVD, choose “Use ISO image file,” select your downloaded CentOS 7 ISO image file, ensure the network adapter’s connection mode is set to NAT, and click OK.

2. Install the CentOS System

Power on the virtual machine, select language and keyboard layout, then enter the installation interface. For software selection, choose Minimal Install (no GUI) or GNOME Desktop (with GUI, recommended for beginners), then click OK.

• Note: If choosing GNOME Desktop, you can select Compatibility Libraries, Legacy X Window System Compatibility, Development Tools, etc.

Click on Installation Destination, select “I will configure partitioning” and click Done, then add partitions and set their sizes, click Done.

• Note: The /boot partition size should be set to over 1GB, filesystem choose ext4. The swap partition size should be set to over 2GB, filesystem choose swap. The / partition size should be set to the remaining space, filesystem choose ext4.

Configure hostname and other information, then click Begin Installation.

• Note: If your memory is relatively small, you can disable kdump during the learning process to save some memory. Set the root password, create a user and its password, wait for the installation to complete and reboot, then accept the license to finish the CentOS system installation.

3. Configure VMware’s NAT Network Mode

Click the Edit menu in VMware, click Virtual Network Editor, click Change Settings -> Add Network, select VMnet8, choose NAT mode. You can modify the NAT gateway, then click OK.

4. Configure Hostname and Static IP Address

4.1 Configure Hostname

1
vim /etc/hostname

Modify the content in the file to your desired hostname, then save and exit.

4.2 Configure Static IP Address

1
vim /etc/sysconfig/network-scripts/ifcfg-ens33

Change the BOOTPROTO value in the file to static. Then modify IPADDR to your desired IP address, add the subnet mask NETMASK and set it to 255.255.255.0, add the gateway GATEWAY and set it to the NAT gateway from the VMware Virtual Network Editor, add DNS1 and set it to your desired DNS server (e.g., Google’s server: 8.8.8.8). Keep other settings unchanged, then save and exit.

• Note: The IP addresses for IPADDR and GATEWAY must be within the subnet of the NAT gateway IP from the VMware Virtual Network Editor, and they must not conflict with IP addresses in the Windows host’s network segment.

FAQ

Why Choose NAT as the Network Type

NAT stands for NETWORK ADDRESS TRANSLATION. It is a technology that converts private IP addresses into public IP addresses, enabling communication between different networks. In the current situation of IPv4 address exhaustion, NAT technology can effectively address the shortage of IP addresses and provide users with more stable network connections. In the context of virtual machine usage, NAT can translate the IP addresses of Windows and the virtual machines to be on the same subnet, enabling access from Windows and communication between virtual machines. It also avoids conflicts with IP addresses in the Windows host’s network segment and does not affect the Windows network.

How to Install and Use the Vim Editor

Vim is a powerful text editor. You can install it using the following command:

1
yum install vim

In command mode, press i to enter insert mode, press Esc to exit insert mode, and type :wq to save and exit the Vim editor.

Summary

This article mainly introduced the process of setting up the virtual machines required for a Hadoop 3.x cluster. We hope it is helpful to you.