hadoop2.2.0单节点安装配置

配置hadoop-env.sh、yarn-env.sh中的环境变量JAVA_HOME

修改core-site.xml

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
<description>The name of the default file system.</description>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/scott/hadoop-${user.name}</value>
</property>
<property>
<name>hadoop.proxyuser.hue.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hue.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.scott.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.scott.groups</name>
<value>*</value>
</property>
</configuration>

修改hdfs-site.xml

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.webhdfs.enable</name>
<value>true</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master:50090</value>
</property>
<property>
<name>dfs.namenode.http-address</name>
<value>master:50070</value>
</property>
</configuration>

修改mapred-site.xml

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<final>true</final>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapred.jobtracker.plugins</name>
<value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
<description>Comma-separated list of jobtracker plug-ins to be activated.</description>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>

修改yarn-site.xml

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address</name>
<value>master:8090</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
</configuration>

修改httpfs-site.xml

<configuration>
<property>
<name>httpfs.proxyuser.hue.hosts</name>
<value>*</value>
</property>
<property>
<name>httpfs.proxyuser.hue.groups</name>
<value>*</value>
</property>
<property>
<name>httpfs.proxyuser.scott.hosts</name>
<value>*</value>
</property>
<property>
<name>httpfs.proxyuser.scott.groups</name>
<value>*</value>
</property>
</configuration>

注:
此文件可以暂时不修改,这里是添加hdfs文件系统用户,其中hue和scott均为linux系统用户

bin/hdfs namenode -format #格式化namenode

bin/hdfs namenode -format

sbin/start-dfs.sh #启动namenode、datanode

sbin/start-dfs.sh

sbin/start-yarn.sh #启动resourcemanager、datamanager

sbin/start-yarn.sh

sbin/mr-jobhistory-daemon.sh start historyserver #启动historyserver

sbin/mr-jobhistory-daemon.sh start historyserver

执行例子

bin/hdfs dfs -put word.txt /word.txt (将当前目录下的word.txt上传到hdfs文件系统上)

bin/hdfs dfs -put word.txt /word.txt
  • 查看hdfs文件系统

  • 执行官方提供的例子
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /word.txt /output

  • 查看运行结果
bin/hdfs dfs -cat /output/*

注:

上文中出现的master均为namenode所在主机名
上文中出现的scott和hue均为本地linux用户名