🌐 中文 English
This article mainly introduces how to set up the JDK and Hadoop dependencies for a Hadoop 3.x cluster.
Preface
In How to Deploy a Hadoop Cluster (Part 1), I have already set up a virtual machine. Next, we need to install the JDK and Hadoop dependencies on this virtual machine to prepare for our subsequent Hadoop cluster deployment.
What is JDK
JDK stands for Java Development Kit. It is a software development kit used by Java developers, containing the Java Runtime Environment (JRE), Java compiler (javac), Java debugger (jdb), and other tools. The JDK is the foundation of Java development, providing all the necessary tools and environment for Java program development. Hadoop 3.x is developed based on Java, so installing the JDK is required to deploy a Hadoop 3.x cluster.
JDK Version Dependency for Hadoop 3.x
I recommend deploying JDK 1.8 (Java 8) as the dependency version for Hadoop 3.x. Many Hadoop-related tools and libraries have been optimized for JDK 1.8, so Java 8 offers the best compatibility and performance. Java 9+ may cause some issues because it introduced the module system (Jigsaw), which can lead to compatibility problems with some dependencies or libraries related to Hadoop 3.x. If you have already deployed Java 9+, I will provide a solution in the FAQ.
How to Deploy Hadoop and JDK Dependencies
1. Download JDK 1.8
Go to the official website, select the tar.gz file for Linux x64, and click download. After downloading, upload the file from Windows to the virtual machine (the FAQ provides upload methods). Use the following command to extract the file:
1
tar -zxvf jdk-8u301-linux-x64.tar.gz -C /dest/
Note: /dest/ is the directory where you want to extract the files. Replace it with your desired directory, such as /opt/.
2. Download Hadoop 3.x
Open the Hadoop official website, select a version, click on binary under Binary download, then click the download link under Http to download the Hadoop 3.x tar.gz file. After downloading, upload the file from Windows to the virtual machine (the FAQ provides upload methods). Use the following command to extract the file:
1
tar -zxvf hadoop-3.3.3.tar.gz -C /dest/
Note: /dest/ is the directory where you want to extract the files. Replace it with your desired directory, such as /opt/.
3. Configure and Start JDK
To clearly organize environment variables for different applications and avoid cluttering /etc/profile, we can create a new file, such as /etc/profile.d/env.sh.
1
vim /etc/profile.d/env.sh
Add the following content to the file:
1
2
3
#JAVA_HOME
export JAVA_HOME=/opt/jdkfilename
export PATH=$PATH:$JAVA_HOME/bin
Note: /opt/jdkfilename is the directory where you extracted the JDK. Replace it with your JDK extraction directory (the previous /dest/), e.g., /opt/jdk-8u202-linux-x64. Start the JDK:
1
source /etc/profile
Type java -version. If the Java version number is returned, the installation is successful.
4. Configure and Start Hadoop
In the previous steps, we have already extracted the Hadoop 3.x tar.gz file. Now we need to configure the Hadoop environment variables. Open the /etc/profile.d/env.sh file and add the following content:
1
2
3
4
#HADOOP_HOME
export HADOOP_HOME=/opt/hadoopfilename
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
Note: /opt/hadoopfilename is the directory where you extracted Hadoop 3.x. Replace it with your Hadoop extraction directory (the previous /dest/), e.g., /opt/hadoop-3.4.0. Start Hadoop:
1
source /etc/profile
Type hadoop version. If the Hadoop version number is returned, the installation is successful.
FAQ
What to do if the official website cannot be opened
Please enable a VPN (“magic”) and reopen the official website.
How to upload files to the virtual machine
There are many software options to upload files from Windows to the virtual machine, such as Xftp, Tabby, WinSCP, etc. After downloading, configure the address, username, and password to upload files.
How to resolve issues when deploying Hadoop 3.x with Java 9+
If you have already deployed Java 9+, you can follow these steps to resolve issues when deploying the Hadoop 3.x cluster:
First, enter the following command:
1
vim /opt/hadoopfilename/etc/hadoop/hadoop-env.sh
Note: /opt/hadoopfilename is the directory where you extracted Hadoop 3.x. Replace it with your Hadoop extraction directory (the previous /dest/), e.g., /opt/hadoop-3.4.0
Then, in the file, below:
1
2
3
4
5
# Extra Java runtime options for all Hadoop commands. We don't support
# IPv6 yet/still, so by default the preference is set to IPv4.
# export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true"
# For Kerberos debugging, an extended option set logs more information
# export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug"
Add export HADOOP_OPTS="$HADOOP_OPTS --add-opens java.base/java.lang=ALL-UNNAMED" to resolve the issue.
Summary
This article mainly introduced how to set up the Hadoop and JDK dependencies for a Hadoop 3.x cluster, preparing for our subsequent Hadoop cluster deployment.
