hadoop2.7汇总:新增功能最新编译64位安装、源码包、API、eclipse插件下载

本帖最后由 nettman 于 2015-4-24 01:36 编辑

hadoop2.7发布(目前还未发布文档,后期补充)

版本更新说明:

http://zh.hortonworks.com/blog/apache-hadoop-2-7-0-released/

下载地址

http://apache.fayea.com/hadoop/common/hadoop-2.7.0/

官网下载安装包

hadoop-2.7.0.tar.gz

链接: http://pan.baidu.com/s/1eQ1sUjS 密码:0h97

maven包

hadoop-2.7.0-src.tar

链接: http://pan.baidu.com/s/1pJ7gmlt 密码:xqyw

64位编译包

hadoop-2.7.0-aboutyun-64bit.tar.gz

链接: http://pan.baidu.com/s/1o6kNDGY

密码:

游客,如果您要查看本帖隐藏内容请回复

hadoop2.7-api文档

链接: http://pan.baidu.com/s/1kTDZYPD 密码:j1r9

eclipse插件 Linux及window操作系统都不太一样,导致插件使用不了,

最好自己制作适合自己的插件

如果真不会制作也有办法开发,可查看

hadoop开发方式总结及操作指导

############################################

编译64位包

1.编译按照这篇帖子即可成功:

从零教你在Linux环境下(ubuntu)如何编译hadoop2.4

最好在root下编译,否则可能会遇到权限问题

2.总结编译中遇到的两大问题:

1.插件下载不完全

表现:Could not resolve dependencies

例如:

Could not resolve dependencies for project org.apache.hadoop:hadoop-minikdc:jar:2.7.0:

解决办法:

这种情况很常见,而且很多都碰到了,他们也是完全按照文档来配置的,但是就不成功,这就是因为插件没有下载完毕造成的。所以尽量多执行几次下面命令

  1. mvn package -Pdist,native -DskipTests -Dtar
复制代码

2.权限问题

权限一般报找不到文件

表现

No such file or directory

例如:

Error executing ant tasks: /home/aboutyun/hadoop-2.7.0-src/hadoop-project/target/antrun/build-main.xml ( No such file or directory ) -> [Help 1]

解决办法:

在root下编译。然后确保所属权限属于root。

剩下的就是多编译几次

3.相关工具:

1.vmware10

链接: http://pan.baidu.com/s/1bIBuA 密码:3ss9

2.winscp516setup

链接: http://pan.baidu.com/s/1nt64rzj 密码:2109

3.jdk-7u67-linux-x64.tar

链接: http://pan.baidu.com/s/1pJ60syf 密码:9kee

4.protobuf-2.5.0.tar.gz

链接: http://pan.baidu.com/s/1dDgWe6P 密码:xgiu

5.apache-ant-1.9.4-bin.tar.gz

http://pan.baidu.com/s/1c0vjhBy

4.编译遇到问题

1.权限问题

本来已经改变了文件夹的权限,可是文件夹里面的还是没有改变

[INFO] ------------------------------------------------------------------------

[INFO] Reactor Summary:

[INFO]

[INFO] Apache Hadoop Main ................................ SUCCESS [4:34.701s]

[INFO] Apache Hadoop Project POM ......................... FAILURE [29.710s]

[INFO] Apache Hadoop Annotations ......................... SKIPPED

[INFO] Apache Hadoop Assemblies .......................... SKIPPED

[INFO] Apache Hadoop Project Dist POM .................... SKIPPED

[INFO] Apache Hadoop Maven Plugins ....................... SKIPPED

[INFO] Apache Hadoop MiniKDC ............................. SKIPPED

[INFO] Apache Hadoop Auth ................................ SKIPPED

[INFO] Apache Hadoop Auth Examples ....................... SKIPPED

[INFO] Apache Hadoop Common .............................. SKIPPED

[INFO] Apache Hadoop NFS ................................. SKIPPED

[INFO] Apache Hadoop KMS ................................. SKIPPED

[INFO] Apache Hadoop Common Project ...................... SKIPPED

[INFO] Apache Hadoop HDFS ................................ SKIPPED

[INFO] Apache Hadoop HttpFS .............................. SKIPPED

[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SKIPPED

[INFO] Apache Hadoop HDFS-NFS ............................ SKIPPED

[INFO] Apache Hadoop HDFS Project ........................ SKIPPED

[INFO] hadoop-yarn ....................................... SKIPPED

[INFO] hadoop-yarn-api ................................... SKIPPED

[INFO] hadoop-yarn-common ................................ SKIPPED

[INFO] hadoop-yarn-server ................................ SKIPPED

[INFO] hadoop-yarn-server-common ......................... SKIPPED

[INFO] hadoop-yarn-server-nodemanager .................... SKIPPED

[INFO] hadoop-yarn-server-web-proxy ...................... SKIPPED

[INFO] hadoop-yarn-server-applicationhistoryservice ...... SKIPPED

[INFO] hadoop-yarn-server-resourcemanager ................ SKIPPED

[INFO] hadoop-yarn-server-tests .......................... SKIPPED

[INFO] hadoop-yarn-client ................................ SKIPPED

[INFO] hadoop-yarn-server-sharedcachemanager ............. SKIPPED

[INFO] hadoop-yarn-applications .......................... SKIPPED

[INFO] hadoop-yarn-applications-distributedshell ......... SKIPPED

[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SKIPPED

[INFO] hadoop-yarn-site .................................. SKIPPED

[INFO] hadoop-yarn-registry .............................. SKIPPED

[INFO] hadoop-yarn-project ............................... SKIPPED

[INFO] hadoop-mapreduce-client ........................... SKIPPED

[INFO] hadoop-mapreduce-client-core ...................... SKIPPED

[INFO] hadoop-mapreduce-client-common .................... SKIPPED

[INFO] hadoop-mapreduce-client-shuffle ................... SKIPPED

[INFO] hadoop-mapreduce-client-app ....................... SKIPPED

[INFO] hadoop-mapreduce-client-hs ........................ SKIPPED

[INFO] hadoop-mapreduce-client-jobclient ................. SKIPPED

[INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED

[INFO] Apache Hadoop MapReduce Examples .................. SKIPPED

[INFO] hadoop-mapreduce .................................. SKIPPED

[INFO] Apache Hadoop MapReduce Streaming ................. SKIPPED

[INFO] Apache Hadoop Distributed Copy .................... SKIPPED

[INFO] Apache Hadoop Archives ............................ SKIPPED

[INFO] Apache Hadoop Rumen ............................... SKIPPED

[INFO] Apache Hadoop Gridmix ............................. SKIPPED

[INFO] Apache Hadoop Data Join ........................... SKIPPED

[INFO] Apache Hadoop Ant Tasks ........................... SKIPPED

[INFO] Apache Hadoop Extras .............................. SKIPPED

[INFO] Apache Hadoop Pipes ............................... SKIPPED

[INFO] Apache Hadoop OpenStack support ................... SKIPPED

[INFO] Apache Hadoop Amazon Web Services support ......... SKIPPED

[INFO] Apache Hadoop Azure support ....................... SKIPPED

[INFO] Apache Hadoop Client .............................. SKIPPED

[INFO] Apache Hadoop Mini-Cluster ........................ SKIPPED

[INFO] Apache Hadoop Scheduler Load Simulator ............ SKIPPED

[INFO] Apache Hadoop Tools Dist .......................... SKIPPED

[INFO] Apache Hadoop Tools ............................... SKIPPED

[INFO] Apache Hadoop Distribution ........................ SKIPPED

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 8:34.724s

[INFO] Finished at: Wed Apr 22 03:40:41 PDT 2015

[INFO] Final Memory: 28M/67M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (create-testdirs) on project hadoop-project: Error executing ant tasks: /home/aboutyun/hadoop-2.7.0-src/hadoop-project/target/antrun/build-main.xml (No such file or directory) -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluen ... oExecutionException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn <goals> -rf :hadoop-project

查看下权限,都是root

  1. aboutyun@ubuntu:~/hadoop-2.7.0-src$ ll
复制代码
  1. total 120
  2. drwxr-xr-x 15 aboutyun aboutyun  4096 Apr 10 11:39 ./
  3. drwxr-xr-x 24 aboutyun aboutyun  4096 Apr 22 03:32 ../
  4. -rw-r--r--  1 root     root     13110 Apr 10 11:35 BUILDING.txt
  5. drwxr-xr-x  2 root     root      4096 Apr 22 03:31 dev-support/
  6. drwxr-xr-x  3 root     root      4096 Apr 22 03:31 hadoop-assemblies/
  7. drwxr-xr-x  2 root     root      4096 Apr 22 03:31 hadoop-client/
  8. drwxr-xr-x 10 root     root      4096 Apr 22 03:31 hadoop-common-project/
  9. drwxr-xr-x  2 root     root      4096 Apr 22 03:31 hadoop-dist/
  10. drwxr-xr-x  6 root     root      4096 Apr 22 03:31 hadoop-hdfs-project/
  11. drwxr-xr-x  9 root     root      4096 Apr 22 03:31 hadoop-mapreduce-project/
  12. drwxr-xr-x  3 root     root      4096 Apr 22 03:31 hadoop-maven-plugins/
  13. drwxr-xr-x  2 root     root      4096 Apr 22 03:31 hadoop-minicluster/
  14. drwxr-xr-x  3 root     root      4096 Apr 22 03:31 hadoop-project/
  15. drwxr-xr-x  2 root     root      4096 Apr 22 03:31 hadoop-project-dist/
  16. drwxr-xr-x 16 root     root      4096 Apr 22 03:31 hadoop-tools/
  17. drwxr-xr-x  3 root     root      4096 Apr 22 03:31 hadoop-yarn-project/
  18. -rw-r--r--  1 root     root     15429 Apr 10 11:35 LICENSE.txt
  19. -rw-r--r--  1 root     root       101 Apr 10 11:35 NOTICE.txt
  20. -rw-r--r--  1 root     root     18893 Apr 10 11:35 pom.xml
  21. -rw-r--r--  1 root     root      1366 Apr 10 11:35 README.txt
复制代码

修改为aboutyun.

执行下面命令:

  1. sudo chown aboutyun:aboutyun hadoop-2.7.0-src/*
复制代码

修改完毕

错误2

  1. Downloaded: http://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-archiver/2.4.4/plexus-archiver-2.4.4.jar (161 KB at 34.7 KB/sec)
  2. [INFO] ------------------------------------------------------------------------
  3. [INFO] Reactor Summary:
  4. [INFO]
  5. [INFO] Apache Hadoop Main ................................ SUCCESS [2.569s]
  6. [INFO] Apache Hadoop Project POM ......................... FAILURE [2:16.305s]
  7. [INFO] Apache Hadoop Annotations ......................... SKIPPED
  8. [INFO] Apache Hadoop Assemblies .......................... SKIPPED
  9. [INFO] Apache Hadoop Project Dist POM .................... SKIPPED
  10. [INFO] Apache Hadoop Maven Plugins ....................... SKIPPED
  11. [INFO] Apache Hadoop MiniKDC ............................. SKIPPED
  12. [INFO] Apache Hadoop Auth ................................ SKIPPED
  13. [INFO] Apache Hadoop Auth Examples ....................... SKIPPED
  14. [INFO] Apache Hadoop Common .............................. SKIPPED
  15. [INFO] Apache Hadoop NFS ................................. SKIPPED
  16. [INFO] Apache Hadoop KMS ................................. SKIPPED
  17. [INFO] Apache Hadoop Common Project ...................... SKIPPED
  18. [INFO] Apache Hadoop HDFS ................................ SKIPPED
  19. [INFO] Apache Hadoop HttpFS .............................. SKIPPED
  20. [INFO] Apache Hadoop HDFS BookKeeper Journal ............. SKIPPED
  21. [INFO] Apache Hadoop HDFS-NFS ............................ SKIPPED
  22. [INFO] Apache Hadoop HDFS Project ........................ SKIPPED
  23. [INFO] hadoop-yarn ....................................... SKIPPED
  24. [INFO] hadoop-yarn-api ................................... SKIPPED
  25. [INFO] hadoop-yarn-common ................................ SKIPPED
  26. [INFO] hadoop-yarn-server ................................ SKIPPED
  27. [INFO] hadoop-yarn-server-common ......................... SKIPPED
  28. [INFO] hadoop-yarn-server-nodemanager .................... SKIPPED
  29. [INFO] hadoop-yarn-server-web-proxy ...................... SKIPPED
  30. [INFO] hadoop-yarn-server-applicationhistoryservice ...... SKIPPED
  31. [INFO] hadoop-yarn-server-resourcemanager ................ SKIPPED
  32. [INFO] hadoop-yarn-server-tests .......................... SKIPPED
  33. [INFO] hadoop-yarn-client ................................ SKIPPED
  34. [INFO] hadoop-yarn-server-sharedcachemanager ............. SKIPPED
  35. [INFO] hadoop-yarn-applications .......................... SKIPPED
  36. [INFO] hadoop-yarn-applications-distributedshell ......... SKIPPED
  37. [INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SKIPPED
  38. [INFO] hadoop-yarn-site .................................. SKIPPED
  39. [INFO] hadoop-yarn-registry .............................. SKIPPED
  40. [INFO] hadoop-yarn-project ............................... SKIPPED
  41. [INFO] hadoop-mapreduce-client ........................... SKIPPED
  42. [INFO] hadoop-mapreduce-client-core ...................... SKIPPED
  43. [INFO] hadoop-mapreduce-client-common .................... SKIPPED
  44. [INFO] hadoop-mapreduce-client-shuffle ................... SKIPPED
  45. [INFO] hadoop-mapreduce-client-app ....................... SKIPPED
  46. [INFO] hadoop-mapreduce-client-hs ........................ SKIPPED
  47. [INFO] hadoop-mapreduce-client-jobclient ................. SKIPPED
  48. [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
  49. [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
  50. [INFO] hadoop-mapreduce .................................. SKIPPED
  51. [INFO] Apache Hadoop MapReduce Streaming ................. SKIPPED
  52. [INFO] Apache Hadoop Distributed Copy .................... SKIPPED
  53. [INFO] Apache Hadoop Archives ............................ SKIPPED
  54. [INFO] Apache Hadoop Rumen ............................... SKIPPED
  55. [INFO] Apache Hadoop Gridmix ............................. SKIPPED
  56. [INFO] Apache Hadoop Data Join ........................... SKIPPED
  57. [INFO] Apache Hadoop Ant Tasks ........................... SKIPPED
  58. [INFO] Apache Hadoop Extras .............................. SKIPPED
  59. [INFO] Apache Hadoop Pipes ............................... SKIPPED
  60. [INFO] Apache Hadoop OpenStack support ................... SKIPPED
  61. [INFO] Apache Hadoop Amazon Web Services support ......... SKIPPED
  62. [INFO] Apache Hadoop Azure support ....................... SKIPPED
  63. [INFO] Apache Hadoop Client .............................. SKIPPED
  64. [INFO] Apache Hadoop Mini-Cluster ........................ SKIPPED
  65. [INFO] Apache Hadoop Scheduler Load Simulator ............ SKIPPED
  66. [INFO] Apache Hadoop Tools Dist .......................... SKIPPED
  67. [INFO] Apache Hadoop Tools ............................... SKIPPED
  68. [INFO] Apache Hadoop Distribution ........................ SKIPPED
  69. [INFO] ------------------------------------------------------------------------
  70. [INFO] BUILD FAILURE
  71. [INFO] ------------------------------------------------------------------------
  72. [INFO] Total time: 2:25.054s
  73. [INFO] Finished at: Wed Apr 22 03:50:50 PDT 2015
  74. [INFO] Final Memory: 30M/71M
  75. [INFO] ------------------------------------------------------------------------
  76. [ERROR] Failed to execute goal org.apache.maven.plugins:maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) on project hadoop-project: Execution hadoop-java-sources of goal org.apache.maven.plugins:maven-source-plugin:2.3:jar-no-fork failed: Plugin org.apache.maven.plugins:maven-source-plugin:2.3 or one of its dependencies could not be resolved: Could not transfer artifact org.apache.maven.reporting:maven-reporting-api:jar:2.0.6 from/to central (http://repo.maven.apache.org/maven2): GET request of: org/apache/maven/reporting/maven-reporting-api/2.0.6/maven-reporting-api-2.0.6.jar from central failed: Read timed out -> [Help 1]
  77. [ERROR]
  78. [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
  79. [ERROR] Re-run Maven using the -X switch to enable full debug logging.
  80. [ERROR]
  81. [ERROR] For more information about the errors and possible solutions, please read the following articles:
  82. [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
  83. [ERROR]
  84. [ERROR] After correcting the problems, you can resume the build with the command
  85. [ERROR]   mvn <goals> -rf :hadoop-project
复制代码

执行了下

  1. mvn install
复制代码

第二个过了,后面又遇到了问题,文件夹里,还有文件里,而且还是root,最后直接切换root,免得在遇到权限问题

切换root遇到问题

  1. [ERROR] The build could not read 1 project -> [Help 1]
  2. [ERROR]   
  3. [ERROR]   The project org.apache.hadoop:hadoop-main:2.7.0 (/root/hadoop-2.7.0-src/pom.xml) has 1 error
  4. [ERROR]     Unresolveable build extension: Plugin org.apache.felix:maven-bundle-plugin:2.5.0 or one of its dependencies could not be resolved: Failed to collect dependencies for org.apache.felix:maven-bundle-plugin:jar:2.5.0 (): Failed to read artifact descriptor for org.apache.maven:maven-plugin-parameter-documenter:jar:2.0.6: Could not transfer artifact org.apache.maven:maven-plugin-parameter-documenter:pom:2.0.6 from/to central (http://repo.maven.apache.org/maven2): Read timed out -> [Help 2]
  5. [ERROR]
  6. [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
  7. [ERROR] Re-run Maven using the -X switch to enable full debug logging.
  8. [ERROR]
  9. [ERROR] For more information about the errors and possible solutions, please read the following articles:
  10. [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
  11. [ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
复制代码

又遇到

  1. [INFO] ------------------------------------------------------------------------
  2. [ERROR] Failed to execute goal on project hadoop-minikdc: Could not resolve dependencies for project org.apache.hadoop:hadoop-minikdc:jar:2.7.0: The following artifacts could not be resolved: org.apache.directory.server:apacheds-core-api:jar:2.0.0-M15, org.apache.directory.api:api-ldap-codec-core:jar:1.0.0-M20, org.apache.directory.api:api-ldap-extras-aci:jar:1.0.0-M20, org.apache.directory.api:api-ldap-extras-util:jar:1.0.0-M20, org.apache.directory.api:api-ldap-model:jar:1.0.0-M20, org.apache.directory.api:api-util:jar:1.0.0-M20, org.apache.mina:mina-core:jar:2.0.0-M5, net.sf.ehcache:ehcache-core:jar:2.4.4, org.apache.directory.server:apacheds-interceptors-admin:jar:2.0.0-M15: Could not transfer artifact org.apache.directory.server:apacheds-core-api:jar:2.0.0-M15 from/to central (http://repo.maven.apache.org/maven2): GET request of: org/apache/directory/server/apacheds-core-api/2.0.0-M15/apacheds-core-api-2.0.0-M15.jar from central failed: Read timed out -> [Help 1]
复制代码

总结:

遇到这种情况,一般都是插件没有下载

  1. Could not resolve dependencies for project org.apache.hadoop:hadoop-minikdc:jar:2.7.0:
复制代码

可以多次执行,下载完毕。

参考:

从零教你在Linux环境下(ubuntu 12.04)如何编译hadoop2.4
我来评几句
登录后评论

已发表评论数()

相关站点

+订阅
热门文章