linux下对Hadoop3.1.4源码编译（详细步骤，附所有资源下载地址）
Hadoop源码编译环境准备1.系统环境虚拟机安装 centos7已配置好网卡，能联网root用户安装2.软件安装包下载jdk8下载地址https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.htmlhadoop-3.1.4-src.tar.gz 源码下载https://hadoop.apach
·
Hadoop源码编译
环境准备
1.系统环境
- 虚拟机安装 centos7
- 已配置好网卡，能联网
- root用户安装
2.软件安装包下载
-
jdk8下载地址
https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
-
hadoop-3.1.4-src.tar.gz 源码下载
https://hadoop.apache.org/releases.html
-
apache-maven-3.6.3-bin.tar.gz下载地址
https://maven.apache.org/download.cgi
-
apache-ant-1.10.9-bin.tar.gz下载地址
https://www.apache.org/dist/ant/binaries/
-
protobuf-2.5.0.tar.gz下载地址
https://github.com/protocolbuffers/protobuf/releases/tag/v2.5.0
jar包安装
1.JDK1.8安装，配置环境变量
略，请自行查询安装
2.Maven配置安装
2.1 解压到指定文件夹（目标文件根据自己的情况而定）
[root@hdp16 hadoop-src]# ls
apache-ant-1.10.9-bin.tar.gz apache-maven-3.6.3-bin.tar.gz hadoop-3.1.4-src.tar.gz protobuf-2.5.0.tar.gz
[root@hdp16 hadoop-src]# pwd
/opt/resource/hadoop-src
[root@hdp16 hadoop-src]# tar -zxvf apache-maven-3.6.3-bin.tar.gz -C /opt/bigdata/
apache-maven-3.6.3/README.txt
apache-maven-3.6.3/LICENSE
apache-maven-3.6.3/NOTICE
apache-maven-3.6.3/lib/
apache-maven-3.6.3/lib/cdi-api.license
apache-maven-3.6.3/lib/commons-cli.license
......
2.2 修改maven配置文件settings.xml，添加阿里镜像
[root@hdp16 conf]# pwd
/opt/bigdata/apache-maven-3.6.3/conf
[root@hdp16 conf]# vim settings.xml
找到，在其内添加如下配置：
<mirror>
<id>nexus-aliyun</id>
<mirrorOf>central</mirrorOf>
<name>Nexus aliyun</name>
<url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>
2.3 配置maven环境变量
[root@hdp16 apache-maven-3.6.3]# vi /etc/profile
#在文件末尾添加如下内容，注意自己maven的安装路径
#MAVEN_HOME
export MAVEN_HOME=/opt/bigdata/apache-maven-3.6.3
export PATH=$PATH:$MAVEN_HOME/bin
[root@hdp16 apache-maven-3.6.3]# source /etc/profile
3.ant 安装配置
3.1 解压到指定目录
[root@hdp16 hadoop-src]# pwd
/opt/resource/hadoop-src
[root@hdp16 hadoop-src]# tar -zxvf apache-ant-1.10.9-bin.tar.gz -C /opt/bigdata/
3.2 配置环境变量
vi /etc/profile
#在文件末尾添加如下内容
#ANT_HOME
export ANT_HOME=/opt/bigdata/apache-ant-1.10.9
export PATH=$PATH:$ANT_HOME/bin
[root@hdp16 apache-maven-3.6.3]# source /etc/profile
4.验证jdk，maven，ant是否配置正确
[root@hdp16 hadoop-src]# java -version
java version "1.8.0_212"
Java(TM) SE Runtime Environment (build 1.8.0_212-b10)
Java HotSpot(TM) 64-Bit Server VM (build 25.212-b10, mixed mode)
[root@hdp16 hadoop-src]# mvn -version
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /opt/bigdata/apache-maven-3.6.3
Java version: 1.8.0_181, vendor: Oracle Corporation, runtime: /usr/java/jdk1.8.0_181-cloudera/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-1160.el7.x86_64", arch: "amd64", family: "unix"
[root@hdp16 hadoop-src]# ant -version
Apache Ant(TM) version 1.10.9 compiled on September 27 2020
5.yum安装依赖包
yum install gcc gcc-c++
yum install make cmake
yum install autoconf automake libtool curl
yum install lzo-devel zlib-devel openssl openssl-devel ncurses-devel
yum install snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop libXtst
6.安装Protobuf
6.1解压安装包到指定目录
[root@hdp16 hadoop-src]# pwd
/opt/resource/hadoop-src
[root@hdp16 hadoop-src]# tar -zxvf protobuf-2.5.0.tar.gz -C /opt/bigdata/
6.2 安装配置
[root@hdp16 protobuf-2.5.0]# pwd
/opt/bigdata/protobuf-2.5.0
[root@hdp16 protobuf-2.5.0]# ls
aclocal.m4 config.guess configure COPYING.txt examples install-sh ltmain.sh Makefile.in protobuf.pc.in src
autogen.sh config.h.in configure.ac depcomp generate_descriptor_proto.sh INSTALL.txt m4 missing python vsprojects
CHANGES.txt config.sub CONTRIBUTORS.txt editors gtest java Makefile.am protobuf-lite.pc.in README.txt
[root@hdp16 protobuf-2.5.0]# ./configure
......
[root@hdp16 protobuf-2.5.0]# make check
......
[root@hdp16 protobuf-2.5.0]# make install
......
[root@hdp16 protobuf-2.5.0]# ldconfig
......
#配置环境变量
[root@hdp16 protobuf-2.5.0]# pwd
/opt/bigdata/protobuf-2.5.0
[root@hdp16 protobuf-2.5.0]# vi /etc/profile
#在文件末尾添加如下内容：
#LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/opt/bigdata/protobuf-2.5.0
export PATH=$PATH:$LD_LIBRARY_PATH
[root@hdp16 protobuf-2.5.0]# source /etc/profile
编译源码
1.解压源码
[root@hdp16 hadoop-src]# ls
apache-ant-1.10.9-bin.tar.gz apache-maven-3.6.3-bin.tar.gz hadoop-3.1.4-src.tar.gz protobuf-2.5.0.tar.gz
[root@hdp16 hadoop-src]# pwd
/opt/resource/hadoop-src
[root@hdp16 hadoop-src]# tar -zxvf hadoop-3.1.4-src.tar.gz -C /opt/
......
[root@hdp16 hadoop-src]# cd /opt/
[root@hdp16 opt]# ls
bigdata cloudera cloudera-manager hadoop-3.1.4-src module resource software
[root@hdp16 opt]# cd hadoop-3.1.4-src/
[root@hdp16 hadoop-3.1.4-src]# ls
BUILDING.txt hadoop-build-tools hadoop-common-project hadoop-mapreduce-project hadoop-project hadoop-yarn-project NOTICE.txt README.txt
dev-support hadoop-client-modules hadoop-dist hadoop-maven-plugins hadoop-project-dist Jenkinsfile patchprocess start-build-env.sh
hadoop-assemblies hadoop-cloud-storage-project hadoop-hdfs-project hadoop-minicluster hadoop-tools LICENSE.txt pom.xml
2.编译
源码目录执行如下命令，等待
[root@hdp16 hadoop-3.1.4-src]# mvn package -Pdist,native -DskipTests -Dtar
遇到如下错误
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:43 min
[INFO] Finished at: 2021-04-30T09:16:35+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.1.4:cmake-compile (cmake-compile) on project hadoop-common: CMake failed with error code 1 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <args> -rf :hadoop-common
卸载 cmake，重新安装
[root@hdp16 hadoop-3.1.4-src]# yum erase cmake
下载安装包
cmake-3.13.5.tar.gz，手动安装：
https://cmake.org/files/v3.13/
安装教程参考 https://www.cnblogs.com/liudw-0215/p/9877290.html
[root@hdp16 hadoop-src]# ls
apache-ant-1.10.9-bin.tar.gz apache-maven-3.6.3-bin.tar.gz cmake-3.13.5.tar.gz hadoop-3.1.4-src.tar.gz protobuf-2.5.0.tar.gz
[root@hdp16 hadoop-src]# pwd
/opt/resource/hadoop-src
#解压
[root@hdp16 hadoop-src]# tar -zxvf cmake-3.13.5.tar.gz -C /opt/bigdata/
...
#编译
[root@hdp16 cmake-3.13.5] ./configure
...
#安装
[root@hdp16 cmake-3.13.5]# sudo make && make install
#验证
[root@hdp16 cmake-3.13.5]# cmake -version
cmake version 3.13.5
CMake suite maintained and supported by Kitware (kitware.com/cmake).
重新进去源码目录，重新执行编译命令
经过漫长的等待，编译终于成功
Reactor Summary for Apache Hadoop Main 3.1.4:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 6.136 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 5.946 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 2.541 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 3.955 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.321 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.979 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 7.253 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 4.419 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 8.949 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 4.395 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:50 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 10.181 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 6.701 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.089 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [01:03 min]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:43 min]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 3.546 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 19.263 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 5.768 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 49.629 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.076 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.092 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 20.100 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [01:03 min]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 7.901 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.050 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 18.634 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 20.205 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 7.330 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 7.996 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 6.143 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 35.662 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 0.962 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 6.662 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 4.611 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 3.948 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 0.056 s]
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [ 6.872 s]
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [ 6.143 s]
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [ 0.043 s]
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SUCCESS [ 6.142 s]
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [ 3.106 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 5.439 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.040 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 4.889 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 3.747 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.323 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 28.221 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 20.595 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 5.583 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 10.253 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 9.863 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 8.261 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.927 s]
[INFO] Apache Hadoop YARN Services ........................ SUCCESS [ 0.036 s]
[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [ 1.588 s]
[INFO] Apache Hadoop YARN Services API .................... SUCCESS [ 1.521 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.055 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 0.064 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 10.699 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 3.906 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [ 9.688 s]
[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [ 2.401 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 6.423 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 5.309 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 9.735 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 6.077 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 2.907 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 2.749 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 9.198 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 7.328 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 4.882 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 3.831 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 0.451 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 5.952 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 22.555 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [ 32.827 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 32.073 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 43.862 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [ 2.023 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 7.329 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 21.866 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 32.568 s]
[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [ 13.138 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 14.721 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.028 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [03:23 min]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [02:09 min]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [ 11.921 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [04:12 min]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [ 0.201 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [ 0.236 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 59.964 s]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [ 0.064 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.760 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.041 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 28:44 min
[INFO] Finished at: 2021-05-06T16:15:17+08:00
[INFO] ------------------------------------------------------------------------
查看一下，编译成果：
[root@hdp16 hadoop-3.1.4-src]# ls
BUILDING.txt hadoop-cloud-storage-project hadoop-maven-plugins hadoop-yarn-project pom.xml
dev-support hadoop-common-project hadoop-minicluster Jenkinsfile README.txt
hadoop-assemblies hadoop-dist hadoop-project LICENSE.txt start-build-env.sh
hadoop-build-tools hadoop-hdfs-project hadoop-project-dist NOTICE.txt
hadoop-client-modules hadoop-mapreduce-project hadoop-tools patchprocess
[root@hdp16 hadoop-3.1.4-src]# cd hadoop-dist/target/
[root@hdp16 target]# ls
antrun hadoop-3.1.4 hadoop-tools-deps test-classes
classes hadoop-3.1.4.tar.gz maven-shared-archive-resources test-dir
华为开发者空间，是为全球开发者打造的专属开发空间，汇聚了华为优质开发资源及工具，致力于让每一位开发者拥有一台云主机，基于华为根生态开发、创新。
更多推荐
2
0- 0
已为社区贡献3条内容
所有评论(0)