At the end of this exercise you will have Hadoop, Hive & HBase installed on your Mac and you will be ready to start implementing some BigData projects
Installing - Apache Hadoop 2.7.2 on macOS Sierra
This is my step by step guide to installing Hadoop on macOS Sierra.
Prerequisites
Hardware
Model Name: MacBook Pro
Model Identifier: MacBookPro12,1
Processor Name: Intel Core i7
Processor Speed: 3.1 GHz
Number of Processors: 1
Total Number of Cores: 2
L2 Cache (per Core): 256 KB
L3 Cache: 4 MB
Memory: 16 GB
Software
OS: macOS Sierra - OS X 10.12. - pre-release version
Package Manager: Home Brew - 1.18
Big Data: Hadoop
Installation
SET JAVA_HOME ON MAC OS X
- Open Terminal
- Confirm you have JDK by typing “which java”. It should show something like /usr/bin/java.
$ which java
/usr/local/bin/java
- Check you have the needed version of Java, by typing “java -version”.
$ java -version
openjdk version "1.8.0-adoptopenjdk"
OpenJDK Runtime Environment (build 1.8.0-adoptopenjdk-jenkins_2018_05_19_02_01-b00)
OpenJDK 64-Bit Server VM (build 25.71-b00, mixed mode)
- JAVA_HOME is essentially the full path of the directory that contains a sub-directory named bin which in turn contains the java.
- For Mac OSX – it is /Library/Java/Home
- Set JAVA_HOME using this command in Terminal: export JAVA_HOME=/Library/Java/Home
- echo $JAVA_HOME on Terminal to confirm the path
- You should now be able to run your application
Note that this sets JAVA_HOME only for this session. If you want it to persist, you will have to add the command to your ~/.profile file. Below are instructions on how to accomplish this instead:
- Open up Terminal.app (Applications >> Utilities >> Terminal)
- Type: emacs .profile
- add this to the end of the .profile file:
JAVA_HOME=/Library/Java/Home
export JAVA_HOME;
- Save and exit emacs (ctrl-x, ctrl-s; ctrl-x, ctrl-c)
- Open a new Terminal window (cmd-n) and type: $JAVA_HOME/bin/java -version
If you see something like:
java version “1.5.0_16″
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b06-284)
Java HotSpot(TM) Client VM (build 1.5.0_16-133, mixed mode, sharing)
Then you’ve successfully set your JAVA_HOME environment variable to the binary stored in /Library/Java/Home/bin/java
Install Home Brew
Open a terminal in MAC.
CMD + Space & type Terminal or open terminal from launcher
bash-3.2$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
==> This script will install:
/usr/local/bin/brew
/usr/local/Library/...
/usr/local/share/doc/homebrew
/usr/local/share/man/man1/brew.1
/usr/local/share/zsh/site-functions/_brew
/usr/local/etc/bash_completion.d/brew
==> The following directories will be made group writable:
/usr/local/.
/usr/local/bin
==> The following directories will have their owner set to *************:
/usr/local/.
/usr/local/bin
==> The following directories will have their group set to admin:
/usr/local/.
/usr/local/bin
Press RETURN to continue or any other key to abort
==> /usr/bin/sudo /bin/chmod g+rwx /usr/local/. /usr/local/bin
Password:
==> /usr/bin/sudo /usr/sbin/chown ************* /usr/local/. /usr/local/bin
==> /usr/bin/sudo /usr/bin/chgrp admin /usr/local/. /usr/local/bin
==> /usr/bin/sudo /bin/mkdir -p /Users/*************/Library/Caches/Homebrew
==> /usr/bin/sudo /bin/chmod g+rwx /Users/*************/Library/Caches/Homebrew
==> /usr/bin/sudo /usr/sbin/chown ************* /Users/*************/Library/Caches/Homebrew
==> Searching online for the Command Line Tools
==> /usr/bin/sudo /usr/bin/touch /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
==> Installing Command Line Tools (OS X 10.11) for Xcode-7.3
==> /usr/bin/sudo /usr/sbin/softwareupdate -i Command\ Line\ Tools\ (OS\ X\ 10.11)\ for\ Xcode-7.3
Software Update Tool
Copyright 2002-2015 Apple Inc.
Downloading Command Line Tools (OS X 10.11) for Xcode
Downloaded Command Line Tools (OS X 10.11) for Xcode
Installing Command Line Tools (OS X 10.11) for Xcode
Done with Command Line Tools (OS X 10.11) for Xcode
Done.
==> /usr/bin/sudo /bin/rm -f /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
==> /usr/bin/sudo /usr/bin/xcode-select --switch /Library/Developer/CommandLineTools
==> Downloading and installing Homebrew...
remote: Counting objects: 1033, done.
remote: Compressing objects: 100% (925/925), done.
remote: Total 1033 (delta 92), reused 645 (delta 67), pack-reused 0
Receiving objects: 100% (1033/1033), 1.03 MiB | 1.41 MiB/s, done.
Resolving deltas: 100% (92/92), done.
From https://github.com/Homebrew/brew
* [new branch] master -> origin/master
HEAD is now at 5d5b9a7 Merge pull request #842 from penman/docs_plist_options
==> Tapping homebrew/core
Cloning into '/usr/local/Library/Taps/homebrew/homebrew-core'...
remote: Counting objects: 3717, done.
remote: Compressing objects: 100% (3604/3604), done.
remote: Total 3717 (delta 16), reused 1690 (delta 4), pack-reused 0
Receiving objects: 100% (3717/3717), 2.87 MiB | 4.91 MiB/s, done.
Resolving deltas: 100% (16/16), done.
Checking connectivity... done.
Tapped 3595 formulae (3,744 files, 8.9M)
==> Installation successful!
==> Next steps
Run `brew help` to get started
Further documentation: https://git.io/brew-docs
==> Homebrew has enabled anonymous aggregate user behaviour analytics
Read the analytics documentation (and how to opt-out) here:
https://git.io/brew-analytics
Install Hadoop
command: brew install hadoop
------
Hadoop will be installed in the following directory/usr/local/Cellar/hadoop
------
bash-3.2$ brew install hadoop
==> Auto-updated Homebrew!
Updated Homebrew from 5d5b9a7 to 069d5f0.
Updated 1 tap (homebrew/core).
==> New Formulae
folly gron libngspice openssl@1.1 sslsplit xcenv
==> Updated Formulae
acmetool csfml gor libflowmanager nvm remarshal vapoursynth
activemq deheader gputils libiscsi oniguruma reposurgeon vnstat
ammonite-repl depqbf gst-plugins-bad libpointing open-mpi sfml wakatime-cli
ansible-cmdb dovecot gst-plugins-good libprotoident pgbadger sile wartremover
aws-elasticbeanstalk dtach gst-plugins-ugly libvirt plowshare sonar-runner xonsh
aws-sdk-cpp etcd gtk-gnutella libvpx pre-commit sqlmap youtube-dl
aws-shell fdk-aac-encoder heroku mailhog prometheus subliminal zimg
awscli flow inspircd mawk pushpin swiftgen zurl
bfg git jenkins mbelib pypy tcpkali
cereal git-credential-manager jsoncpp media-info qemu terraform
clang-format gitlab-ci-multi-runner jsonschema2pojo mediaconch rainbarf tgui
codequery gitversion khard mtools re2 transcrypt
crystal-lang gjstest lftp mysqltuner rebar urweb
==> Deleted Formulae
box2d cfitsio git-stree
Warning: You are using OS X 10.12.
We do not provide support for this pre-release version.
You may encounter build failures or other breakages.
Please create pull-requests instead of filing issues.
==> Using the sandbox
==> Downloading https://www.apache.org/dyn/closer.cgi?path=hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz
==> Best Mirror http://mirrors.ocf.berkeley.edu/apache/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz
######################################################################## 100.0%
==> Caveats
In Hadoop's config file:
/usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop/hadoop-env.sh,
/usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop/mapred-env.sh and
/usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop/yarn-env.sh
$JAVA_HOME has been set to be the output of:
/usr/libexec/java_home
==> Summary
🍺 /usr/local/Cellar/hadoop/2.7.2: 6,304 files, 310M, built in 1 minute 5 seconds
Configuration
Configuring Hadoop
Edit hadoop-env.sh
The file can be located at /usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop/hadoop-env.sh where 2.7.0 is the hadoop version.
One can open and edit the file in the terminal alternatively can also open the file from finder.
In order to do so. You need to activate the finder to show all hidden folders.
Show/Hide Hidden Files
- Open Terminal found in Finder > Applications > Utilities
- In Terminal, paste the following:
defaults write com.apple.finder AppleShowAllFiles YES
- Press return
- Hold the ‘Option/alt’ key, then right click on the Finder icon in the dock and click Relaunch.
- Relaunch Finder by right clicking the Finder Icon whilst holding the ‘Option/alt’ key
- This will show all hidden files.
- To hide them again, follow the same steps but replace the Terminal command with: defaults write com.apple.finder AppleShowAllFiles NO
I used text wrangler to edit the file. Alternatively you can edit it on terminal:
edit the file vi hadoop-env.sh
Find the line with
export
HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"
as highlighted in the file:
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Set Hadoop-specific environment variables here.
# The only required environment variable is JAVA_HOME. All others are
# optional. When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.
# The java implementation to use.
export JAVA_HOME="$(/usr/libexec/java_home)"
# The jsvc implementation to use. Jsvc is required to run secure datanodes
# that bind to privileged ports to provide authentication of data transfer
# protocol. Jsvc is not required if SASL is configured for authentication of
# data transfer protocol using non-privileged ports.
#export JSVC_HOME=${JSVC_HOME}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"
# Extra Java CLASSPATH elements. Automatically insert capacity-scheduler.
for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
if [ "$HADOOP_CLASSPATH" ]; then
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
else
export HADOOP_CLASSPATH=$f
fi
done
# The maximum amount of heap to use, in MB. Default is 1000.
#export HADOOP_HEAPSIZE=
#export HADOOP_NAMENODE_INIT_HEAPSIZE=""
# Extra Java runtime options. Empty by default.
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"
# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_NFS3_OPTS="$HADOOP_NFS3_OPTS"
export HADOOP_PORTMAP_OPTS="-Xmx512m $HADOOP_PORTMAP_OPTS"
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS"
#HADOOP_JAVA_PLATFORM_OPTS="-XX:-UsePerfData $HADOOP_JAVA_PLATFORM_OPTS"
# On secure datanodes, user to run the datanode as after dropping privileges.
# This **MUST** be uncommented to enable secure HDFS if using privileged ports
# to provide authentication of data transfer protocol. This **MUST NOT** be
# defined if SASL is configured for authentication of data transfer protocol
# using non-privileged ports.
export HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER}
# Where log files are stored. $HADOOP_HOME/logs by default.
#export HADOOP_LOG_DIR=${HADOOP_LOG_DIR}/$USER
# Where log files are stored in the secure data environment.
export HADOOP_SECURE_DN_LOG_DIR=${HADOOP_LOG_DIR}/${HADOOP_HDFS_USER}
###
# HDFS Mover specific parameters
###
# Specify the JVM options to be used when starting the HDFS Mover.
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
# export HADOOP_MOVER_OPTS=""
###
# Advanced Users Only!
###
# The directory where pid files are stored. /tmp by default.
# NOTE: this should be set to a directory that can only be written to by
# the user that will run the hadoop daemons. Otherwise there is the
# potential for a symlink attack.
export HADOOP_PID_DIR=${HADOOP_PID_DIR}
export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}
# A string representing this instance of hadoop. $USER by default
export HADOOP_IDENT_STRING=$USER
and change it to
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc="
Edit Core-site.xml
The file can be located at/usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop/core-site.xml
from:
16/09/06 13:20:22 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = *************-mbp.home/192.168.1.10
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.7.2
STARTUP_MSG: classpath = /usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/activation-1.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/asm-3.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/curator-client-2.7.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/curator-framework-2.7.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/hadoop-annotations-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/hadoop-auth-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jsr305-3.0.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/junit-4.11.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/xz-1.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/hadoop-common-2.7.2-tests.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/hadoop-common-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/hadoop-nfs-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/hadoop-hdfs-2.7.2-tests.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/hadoop-hdfs-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/activation-1.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/asm-3.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/guice-3.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-api-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-client-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-common-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-registry-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-server-common-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2-tests.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar:/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r b165c4fe8a74265c792ce23f546c64604acf0e41; compiled by 'jenkins' on 2016-01-26T00:08Z
STARTUP_MSG: java = 1.8.0_92
************************************************************/
16/09/06 13:20:22 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
16/09/06 13:20:22 INFO namenode.NameNode: createNameNode [-format]
16/09/06 13:20:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Formatting using clusterid: CID-ed403bb1-56a9-4a3b-8751-15b17ba93ca3
16/09/06 13:20:23 INFO namenode.FSNamesystem: No KeyProvider found.
16/09/06 13:20:23 INFO namenode.FSNamesystem: fsLock is fair:true
16/09/06 13:20:23 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
16/09/06 13:20:23 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
16/09/06 13:20:23 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
16/09/06 13:20:23 INFO blockmanagement.BlockManager: The block deletion will start around 2016 Sep 06 13:20:23
16/09/06 13:20:23 INFO util.GSet: Computing capacity for map BlocksMap
16/09/06 13:20:23 INFO util.GSet: VM type = 64-bit
16/09/06 13:20:23 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB
16/09/06 13:20:23 INFO util.GSet: capacity = 2^21 = 2097152 entries
16/09/06 13:20:23 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
16/09/06 13:20:23 INFO blockmanagement.BlockManager: defaultReplication = 1
16/09/06 13:20:23 INFO blockmanagement.BlockManager: maxReplication = 512
16/09/06 13:20:23 INFO blockmanagement.BlockManager: minReplication = 1
16/09/06 13:20:23 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
16/09/06 13:20:23 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
16/09/06 13:20:23 INFO blockmanagement.BlockManager: encryptDataTransfer = false
16/09/06 13:20:23 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
16/09/06 13:20:23 INFO namenode.FSNamesystem: fsOwner = ************* (auth:SIMPLE)
16/09/06 13:20:23 INFO namenode.FSNamesystem: supergroup = supergroup
16/09/06 13:20:23 INFO namenode.FSNamesystem: isPermissionEnabled = true
16/09/06 13:20:23 INFO namenode.FSNamesystem: HA Enabled: false
16/09/06 13:20:23 INFO namenode.FSNamesystem: Append Enabled: true
16/09/06 13:20:23 INFO util.GSet: Computing capacity for map INodeMap
16/09/06 13:20:23 INFO util.GSet: VM type = 64-bit
16/09/06 13:20:23 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB
16/09/06 13:20:23 INFO util.GSet: capacity = 2^20 = 1048576 entries
16/09/06 13:20:23 INFO namenode.FSDirectory: ACLs enabled? false
16/09/06 13:20:23 INFO namenode.FSDirectory: XAttrs enabled? true
16/09/06 13:20:23 INFO namenode.FSDirectory: Maximum size of an xattr: 16384
16/09/06 13:20:23 INFO namenode.NameNode: Caching file names occuring more than 10 times
16/09/06 13:20:23 INFO util.GSet: Computing capacity for map cachedBlocks
16/09/06 13:20:23 INFO util.GSet: VM type = 64-bit
16/09/06 13:20:23 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB
16/09/06 13:20:23 INFO util.GSet: capacity = 2^18 = 262144 entries
16/09/06 13:20:23 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
16/09/06 13:20:23 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
16/09/06 13:20:23 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
16/09/06 13:20:23 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
16/09/06 13:20:23 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
16/09/06 13:20:23 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
16/09/06 13:20:23 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
16/09/06 13:20:23 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
16/09/06 13:20:23 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/09/06 13:20:23 INFO util.GSet: VM type = 64-bit
16/09/06 13:20:23 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
16/09/06 13:20:23 INFO util.GSet: capacity = 2^15 = 32768 entries
16/09/06 13:20:23 INFO namenode.FSImage: Allocated new BlockPoolId: BP-154566474-192.168.1.10-1473182423595
16/09/06 13:20:23 INFO common.Storage: Storage directory /usr/local/Cellar/hadoop/hdfs/tmp/dfs/name has been successfully formatted.
16/09/06 13:20:23 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
16/09/06 13:20:23 INFO util.ExitUtil: Exiting with status 0
16/09/06 13:20:23 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at *************-mbp.home/192.168.1.10
************************************************************/
SSH Localhost
Generate ~/.ssh/id_rsa and the ~/.ssh/id_rsa.pub files using
$ ssh-keygen -t rsa
Enable Remote Login
“System Preferences” -> “Sharing”. Check “Remote Login”
Authorize SSH Keys
To allow your system to accept login, we have to make it aware of the keys that will be used
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Login.
$ ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
ECDSA key fingerprint is SHA256:WmIZBk7JuaSCNEKV81Lj1skRPCI4G+J/vvixV5MyXiY.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Last login: Tue Sep 6 11:36:38 2016
$ exit
logout
Connection to localhost closed.
Running Hadoop
Start Hadoop
$ hstart
$ hstart
16/09/06 13:52:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/Cellar/hadoop/2.7.2/libexec/logs/hadoop-*************-namenode-*************-MBP.home.out
localhost: starting datanode, logging to /usr/local/Cellar/hadoop/2.7.2/libexec/logs/hadoop-*************-datanode-*************-MBP.home.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/Cellar/hadoop/2.7.2/libexec/logs/hadoop-*************-secondarynamenode-*************-MBP.home.out
16/09/06 13:52:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/local/Cellar/hadoop/2.7.2/libexec/logs/yarn-*************-resourcemanager-*************-MBP.home.out
localhost: starting nodemanager, logging to /usr/local/Cellar/hadoop/2.7.2/libexec/logs/yarn-*************-nodemanager-*************-MBP.home.out
Stop Hadoop
$ hstop
$ hstop
stopping yarn daemons
stopping resourcemanager
localhost: stopping nodemanager
no proxyserver to stop
16/09/06 13:51:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Stopping namenodes on [localhost]
localhost: stopping namenode
localhost: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
16/09/06 13:51:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Web Interface
Resource Manager: http://localhost:50070
JobTracker: http://localhost:8088
Specific Node Information: http://localhost:8042
Command
$ jps
7379 DataNode
7459 SecondaryNameNode
7316 NameNode
7636 NodeManager
7562 ResourceManager
7676 Jps
$ yarn // For resource management more information than the web interface.
$ mapred // Detailed information about jobs