Wednesday, September 7, 2016

Installing - Apache Spark on macOS Sierra

Installing - Apache Spark on macOS Sierra






This is my step by step guide to installing Apache Spark 2.0.0 on macOS Sierra.

Pre-Requisirtes

Hardware

  Model Name: MacBook Pro
  Model Identifier: MacBookPro12,1
  Processor Name: Intel Core i7
  Processor Speed: 3.1 GHz
  Number of Processors: 1
  Total Number of Cores: 2
  L2 Cache (per Core): 256 KB
  L3 Cache: 4 MB

  Memory: 16 GB

Software

OS: macOS Sierra - OS X 10.12. - pre-release version
Package Manager: Home Brew - 1.18
Big Data: Hadoop
Cluster Computing Framework: Apache Spark 2.0.0

Installation

Install Xcode 8.0.

Xcode can be updated from
  https://developer.apple.com/xcode/downloads/
If 8.0 is installed, you may need to:
  sudo xcode-select --switch /path/to/Xcode-beta.app
bash-3.2$ sudo xcode-select --switch /Applications/Xcode-beta.app 

Install Home Brew

Open a terminal in MAC.
CMD + Space & type Terminal or open terminal from launcher

Install Spark


bash-3.2$ brew install spark
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
graphicsmagick

Warning: You are using OS X 10.12.
We do not provide support for this pre-release version.
You may encounter build failures or other breakages.
Please create pull-requests instead of filing issues.
==> Using the sandbox
==> Downloading https://github.com/holman/spark/archive/v1.0.1.tar.gz
==> Downloading from https://codeload.github.com/holman/spark/tar.gz/v1.0.1
######################################################################## 100.0%
🍺  /usr/local/Cellar/spark/1.0.1: 5 files, 6.3K, built in 3 seconds

Captain Nemo New York
- No comments:
Labels: , , ,

Tuesday, September 6, 2016

Installing Hadoop, Hive and HBase



Intention:
At the end of this exercise you will have Hadoop, Hive & HBase installed on your Mac and you will be ready to start implementing some BigData projects

Installing - Apache Hadoop 2.7.2 on macOS Sierra

This is my step by step guide to installing Hadoop on macOS Sierra.

Prerequisites

Hardware

  Model Name: MacBook Pro
  Model Identifier: MacBookPro12,1
  Processor Name: Intel Core i7
  Processor Speed: 3.1 GHz
  Number of Processors: 1
  Total Number of Cores: 2
  L2 Cache (per Core): 256 KB
  L3 Cache: 4 MB

  Memory: 16 GB

Software

OS: macOS Sierra - OS X 10.12. - pre-release version
Package Manager: Home Brew - 1.18
Big Data: Hadoop

Installation

SET JAVA_HOME ON MAC OS X

  • Open Terminal
  • Confirm you have JDK by typing “which java”. It should show something like /usr/bin/java.
  • $ which java
    /usr/local/bin/java

  • Check you have the needed version of Java, by typing “java -version”.
  • $ java -version
    openjdk version "1.8.0-adoptopenjdk"
    OpenJDK Runtime Environment (build 1.8.0-adoptopenjdk-jenkins_2018_05_19_02_01-b00)
    OpenJDK 64-Bit Server VM (build 25.71-b00, mixed mode)

    • JAVA_HOME is essentially the full path of the directory that contains a sub-directory named bin which in turn contains the java.
    • For Mac OSX – it is /Library/Java/Home
  • Set JAVA_HOME using this command in Terminal: export JAVA_HOME=/Library/Java/Home
  • echo $JAVA_HOME on Terminal to confirm the path
  • You should now be able to run your application
Note that this sets JAVA_HOME only for this session. If you want it to persist, you will have to add the command to your ~/.profile file.  Below are instructions on how to accomplish this instead:
  • Open up Terminal.app (Applications >> Utilities >> Terminal)
  • Type: emacs .profile
  • add this to the end of the .profile file:
JAVA_HOME=/Library/Java/Home
export JAVA_HOME;
  • Save and exit emacs (ctrl-x, ctrl-s; ctrl-x, ctrl-c)
  • Open a new Terminal window (cmd-n) and type: $JAVA_HOME/bin/java -version
If you see something like:
java version “1.5.0_16″
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b06-284)
Java HotSpot(TM) Client VM (build 1.5.0_16-133, mixed mode, sharing)
Then you’ve successfully set your JAVA_HOME environment variable to the binary stored in /Library/Java/Home/bin/java


Install Home Brew

Open a terminal in MAC.
CMD + Space & type Terminal or open terminal from launcher
















bash-3.2$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
==> This script will install:
/usr/local/bin/brew
/usr/local/Library/...
/usr/local/share/doc/homebrew
/usr/local/share/man/man1/brew.1
/usr/local/share/zsh/site-functions/_brew
/usr/local/etc/bash_completion.d/brew
==> The following directories will be made group writable:
/usr/local/.
/usr/local/bin
==> The following directories will have their owner set to *************:
/usr/local/.
/usr/local/bin
==> The following directories will have their group set to admin:
/usr/local/.
/usr/local/bin

Press RETURN to continue or any other key to abort
==> /usr/bin/sudo /bin/chmod g+rwx /usr/local/. /usr/local/bin
Password:
==> /usr/bin/sudo /usr/sbin/chown ************* /usr/local/. /usr/local/bin
==> /usr/bin/sudo /usr/bin/chgrp admin /usr/local/. /usr/local/bin
==> /usr/bin/sudo /bin/mkdir -p /Users/*************/Library/Caches/Homebrew
==> /usr/bin/sudo /bin/chmod g+rwx /Users/*************/Library/Caches/Homebrew
==> /usr/bin/sudo /usr/sbin/chown ************* /Users/*************/Library/Caches/Homebrew
==> Searching online for the Command Line Tools
==> /usr/bin/sudo /usr/bin/touch /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
==> Installing Command Line Tools (OS X 10.11) for Xcode-7.3
==> /usr/bin/sudo /usr/sbin/softwareupdate -i Command\ Line\ Tools\ (OS\ X\ 10.11)\ for\ Xcode-7.3
Software Update Tool
Copyright 2002-2015 Apple Inc.


Downloading Command Line Tools (OS X 10.11) for Xcode
Downloaded Command Line Tools (OS X 10.11) for Xcode
Installing Command Line Tools (OS X 10.11) for Xcode
Done with Command Line Tools (OS X 10.11) for Xcode
Done.
==> /usr/bin/sudo /bin/rm -f /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
==> /usr/bin/sudo /usr/bin/xcode-select --switch /Library/Developer/CommandLineTools
==> Downloading and installing Homebrew...
remote: Counting objects: 1033, done.
remote: Compressing objects: 100% (925/925), done.
remote: Total 1033 (delta 92), reused 645 (delta 67), pack-reused 0
Receiving objects: 100% (1033/1033), 1.03 MiB | 1.41 MiB/s, done.
Resolving deltas: 100% (92/92), done.
From https://github.com/Homebrew/brew
 * [new branch]      master     -> origin/master
HEAD is now at 5d5b9a7 Merge pull request #842 from penman/docs_plist_options
==> Tapping homebrew/core
Cloning into '/usr/local/Library/Taps/homebrew/homebrew-core'...
remote: Counting objects: 3717, done.
remote: Compressing objects: 100% (3604/3604), done.
remote: Total 3717 (delta 16), reused 1690 (delta 4), pack-reused 0
Receiving objects: 100% (3717/3717), 2.87 MiB | 4.91 MiB/s, done.
Resolving deltas: 100% (16/16), done.
Checking connectivity... done.
Tapped 3595 formulae (3,744 files, 8.9M)
==> Installation successful!
==> Next steps
Run `brew help` to get started
Further documentation: https://git.io/brew-docs
==> Homebrew has enabled anonymous aggregate user behaviour analytics
Read the analytics documentation (and how to opt-out) here:

  https://git.io/brew-analytics

Install Hadoop

command: brew install hadoop
------
Hadoop will be installed in the following directory/usr/local/Cellar/hadoop
------

bash-3.2$ brew install hadoop
==> Auto-updated Homebrew!
Updated Homebrew from 5d5b9a7 to 069d5f0.
Updated 1 tap (homebrew/core).
==> New Formulae
folly                       gron                        libngspice                  openssl@1.1                 sslsplit                    xcenv                     
==> Updated Formulae
acmetool                csfml                   gor                     libflowmanager          nvm                     remarshal               vapoursynth           
activemq                deheader                gputils                 libiscsi                oniguruma               reposurgeon             vnstat                
ammonite-repl           depqbf                  gst-plugins-bad         libpointing             open-mpi                sfml                    wakatime-cli          
ansible-cmdb            dovecot                 gst-plugins-good        libprotoident           pgbadger                sile                    wartremover           
aws-elasticbeanstalk    dtach                   gst-plugins-ugly        libvirt                 plowshare               sonar-runner            xonsh                 
aws-sdk-cpp             etcd                    gtk-gnutella            libvpx                  pre-commit              sqlmap                  youtube-dl            
aws-shell               fdk-aac-encoder         heroku                  mailhog                 prometheus              subliminal              zimg                  
awscli                  flow                    inspircd                mawk                    pushpin                 swiftgen                zurl                  
bfg                     git                     jenkins                 mbelib                  pypy                    tcpkali               
cereal                  git-credential-manager  jsoncpp                 media-info              qemu                    terraform             
clang-format            gitlab-ci-multi-runner  jsonschema2pojo         mediaconch              rainbarf                tgui                  
codequery               gitversion              khard                   mtools                  re2                     transcrypt            
crystal-lang            gjstest                 lftp                    mysqltuner              rebar                   urweb                 
==> Deleted Formulae
box2d                                                   cfitsio                                                 git-stree                                             

Warning: You are using OS X 10.12.
We do not provide support for this pre-release version.
You may encounter build failures or other breakages.
Please create pull-requests instead of filing issues.
==> Using the sandbox
==> Downloading https://www.apache.org/dyn/closer.cgi?path=hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz
==> Best Mirror http://mirrors.ocf.berkeley.edu/apache/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz
######################################################################## 100.0%
==> Caveats
In Hadoop's config file:
  /usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop/hadoop-env.sh,
  /usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop/mapred-env.sh and
  /usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop/yarn-env.sh
$JAVA_HOME has been set to be the output of:
  /usr/libexec/java_home
==> Summary
🍺  /usr/local/Cellar/hadoop/2.7.2: 6,304 files, 310M, built in 1 minute 5 seconds

Configuration

Configuring Hadoop

Edit hadoop-env.sh

The file can be located at /usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop/hadoop-env.sh where 2.7.0 is the hadoop version.

One can open and edit the file in the terminal alternatively can also open the file from finder.
In order to do so. You need to activate the finder to show all hidden folders.

Show/Hide Hidden Files

  1. Open Terminal found in Finder > Applications > Utilities
  2. In Terminal, paste the following: defaults write com.apple.finder AppleShowAllFiles YES
  3. Press return
  4. Hold the ‘Option/alt’ key, then right click on the Finder icon in the dock and click Relaunch.
  5. Relaunch Finder by right clicking the Finder Icon whilst holding the ‘Option/alt’ key

  1. This will show all hidden files. 
  2. To hide them again, follow the same steps but replace the Terminal command with: defaults write com.apple.finder AppleShowAllFiles NO

I used text wrangler to edit the file. Alternatively you can edit it on terminal:
edit the file vi hadoop-env.sh 

Find the line with
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"
as highlighted in the file:


# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME. All others are
# optional. When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.

# The java implementation to use.
export JAVA_HOME="$(/usr/libexec/java_home)"

# The jsvc implementation to use. Jsvc is required to run secure datanodes
# that bind to privileged ports to provide authentication of data transfer
# protocol. Jsvc is not required if SASL is configured for authentication of
# data transfer protocol using non-privileged ports.
#export JSVC_HOME=${JSVC_HOME}

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"

# Extra Java CLASSPATH elements. Automatically insert capacity-scheduler.
for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
if [ "$HADOOP_CLASSPATH" ]; then
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
else
export HADOOP_CLASSPATH=$f
fi
done

# The maximum amount of heap to use, in MB. Default is 1000.
#export HADOOP_HEAPSIZE=
#export HADOOP_NAMENODE_INIT_HEAPSIZE=""

# Extra Java runtime options. Empty by default.
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"

# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"

export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS"

export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_NFS3_OPTS="$HADOOP_NFS3_OPTS"
export HADOOP_PORTMAP_OPTS="-Xmx512m $HADOOP_PORTMAP_OPTS"

# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS"
#HADOOP_JAVA_PLATFORM_OPTS="-XX:-UsePerfData $HADOOP_JAVA_PLATFORM_OPTS"
# On secure datanodes, user to run the datanode as after dropping privileges.
# This **MUST** be uncommented to enable secure HDFS if using privileged ports
# to provide authentication of data transfer protocol. This **MUST NOT** be
# defined if SASL is configured for authentication of data transfer protocol
# using non-privileged ports.
export HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER}
# Where log files are stored. $HADOOP_HOME/logs by default.
#export HADOOP_LOG_DIR=${HADOOP_LOG_DIR}/$USER
# Where log files are stored in the secure data environment.
export HADOOP_SECURE_DN_LOG_DIR=${HADOOP_LOG_DIR}/${HADOOP_HDFS_USER}

###
# HDFS Mover specific parameters
###
# Specify the JVM options to be used when starting the HDFS Mover.
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
# export HADOOP_MOVER_OPTS=""

###
# Advanced Users Only!
###

# The directory where pid files are stored. /tmp by default.
# NOTE: this should be set to a directory that can only be written to by
# the user that will run the hadoop daemons. Otherwise there is the
# potential for a symlink attack.
export HADOOP_PID_DIR=${HADOOP_PID_DIR}
export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}

# A string representing this instance of hadoop. $USER by default
export HADOOP_IDENT_STRING=$USER

and change it to
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc="

Edit Core-site.xml

The file can be located at/usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop/core-site.xml
from:



to:

  

     hadoop.tmp.dir
     /usr/local/Cellar/hadoop/hdfs/tmp
     A base for other temporary directories.
  
  
     fs.default.name                                     
     hdfs://localhost:9000

Edit mapred-site.xml

The file can be located at /usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop/mapred-site.xml The file doesn't exist by default. Need to create it from mapred-site.xml.template 


mapred.job.tracker
localhost:9010

Edit hdfs-site.xml

The file can be located at /usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop/hdfs-site.xml .


dfs.replication
1
Create and edit your ~/.profile 
The .profile file in MacOSX works exactly how you would expect. Simply create the .profile file in your user directory if it doesn't exist.
There is no ~/.profile by default. 
When you start your Terminal.app, it loads a shell – in your case Bash. Shells can have configuration files, and .profile is one of these. They are read when the shell starts up, in a specific order. You can read man bash, section "Invocation", for more details.*
touch ~/.profile
open -e !$
Save it, and the next time you start a new instance of Bash (e.g. by opening a new Terminal window), your ~/.profile will be read and all commands therein will be executed.
* Note that once you create a file called ~/.bash_profile, your ~/.profile will not be read anymore.
.profile works with sh or bash. However .bash_profile is bash-specific and will override .profile if present.
In macOS Sierra the .bash_profile exists in the home directory (its empty by default)

Add the following two commands. 

alias hstart="/usr/local/Cellar/hadoop/2.6.0/sbin/start-dfs.sh;/usr/local/Cellar/hadoop/2.7.2/sbin/start-yarn.sh"
alias hstop="/usr/local/Cellar/hadoop/2.6.0/sbin/stop-yarn.sh;/usr/local/Cellar/hadoop/2.7.2/sbin/stop-dfs.sh"

. ~/.bash_profile to reload the bash profile.

Format the hdfs file before executing Hadoop
$ hdfs namenode -format

16/09/06 13:20:22 INFO namenode.NameNode: STARTUP_MSG: 

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = *************-mbp.home/192.168.1.10

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 2.7.2

STARTUP_MSG:   classpath = /usr/local/Cellar/hadoop/2.7.2/libexec/etc/hadoop:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/activation-1.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/asm-3.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/curator-client-2.7.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/curator-framework-2.7.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/hadoop-annotations-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/hadoop-auth-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/jsr305-3.0.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/junit-4.11.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/xz-1.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/hadoop-common-2.7.2-tests.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/hadoop-common-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/hadoop-nfs-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/hadoop-hdfs-2.7.2-tests.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/hadoop-hdfs-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/activation-1.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/asm-3.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/guice-3.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-api-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-client-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-common-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-registry-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-server-common-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2-tests.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.2.jar:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar:/contrib/capacity-scheduler/*.jar

STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r b165c4fe8a74265c792ce23f546c64604acf0e41; compiled by 'jenkins' on 2016-01-26T00:08Z

STARTUP_MSG:   java = 1.8.0_92
************************************************************/
16/09/06 13:20:22 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
16/09/06 13:20:22 INFO namenode.NameNode: createNameNode [-format]
16/09/06 13:20:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Formatting using clusterid: CID-ed403bb1-56a9-4a3b-8751-15b17ba93ca3
16/09/06 13:20:23 INFO namenode.FSNamesystem: No KeyProvider found.
16/09/06 13:20:23 INFO namenode.FSNamesystem: fsLock is fair:true
16/09/06 13:20:23 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
16/09/06 13:20:23 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
16/09/06 13:20:23 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
16/09/06 13:20:23 INFO blockmanagement.BlockManager: The block deletion will start around 2016 Sep 06 13:20:23
16/09/06 13:20:23 INFO util.GSet: Computing capacity for map BlocksMap
16/09/06 13:20:23 INFO util.GSet: VM type       = 64-bit
16/09/06 13:20:23 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB
16/09/06 13:20:23 INFO util.GSet: capacity      = 2^21 = 2097152 entries
16/09/06 13:20:23 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
16/09/06 13:20:23 INFO blockmanagement.BlockManager: defaultReplication         = 1
16/09/06 13:20:23 INFO blockmanagement.BlockManager: maxReplication             = 512
16/09/06 13:20:23 INFO blockmanagement.BlockManager: minReplication             = 1
16/09/06 13:20:23 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
16/09/06 13:20:23 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
16/09/06 13:20:23 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
16/09/06 13:20:23 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
16/09/06 13:20:23 INFO namenode.FSNamesystem: fsOwner             = ************* (auth:SIMPLE)
16/09/06 13:20:23 INFO namenode.FSNamesystem: supergroup          = supergroup
16/09/06 13:20:23 INFO namenode.FSNamesystem: isPermissionEnabled = true
16/09/06 13:20:23 INFO namenode.FSNamesystem: HA Enabled: false
16/09/06 13:20:23 INFO namenode.FSNamesystem: Append Enabled: true
16/09/06 13:20:23 INFO util.GSet: Computing capacity for map INodeMap
16/09/06 13:20:23 INFO util.GSet: VM type       = 64-bit
16/09/06 13:20:23 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB
16/09/06 13:20:23 INFO util.GSet: capacity      = 2^20 = 1048576 entries
16/09/06 13:20:23 INFO namenode.FSDirectory: ACLs enabled? false
16/09/06 13:20:23 INFO namenode.FSDirectory: XAttrs enabled? true
16/09/06 13:20:23 INFO namenode.FSDirectory: Maximum size of an xattr: 16384
16/09/06 13:20:23 INFO namenode.NameNode: Caching file names occuring more than 10 times
16/09/06 13:20:23 INFO util.GSet: Computing capacity for map cachedBlocks
16/09/06 13:20:23 INFO util.GSet: VM type       = 64-bit
16/09/06 13:20:23 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB
16/09/06 13:20:23 INFO util.GSet: capacity      = 2^18 = 262144 entries
16/09/06 13:20:23 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
16/09/06 13:20:23 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
16/09/06 13:20:23 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
16/09/06 13:20:23 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
16/09/06 13:20:23 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
16/09/06 13:20:23 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
16/09/06 13:20:23 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
16/09/06 13:20:23 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
16/09/06 13:20:23 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/09/06 13:20:23 INFO util.GSet: VM type       = 64-bit
16/09/06 13:20:23 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
16/09/06 13:20:23 INFO util.GSet: capacity      = 2^15 = 32768 entries
16/09/06 13:20:23 INFO namenode.FSImage: Allocated new BlockPoolId: BP-154566474-192.168.1.10-1473182423595
16/09/06 13:20:23 INFO common.Storage: Storage directory /usr/local/Cellar/hadoop/hdfs/tmp/dfs/name has been successfully formatted.
16/09/06 13:20:23 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
16/09/06 13:20:23 INFO util.ExitUtil: Exiting with status 0
16/09/06 13:20:23 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at *************-mbp.home/192.168.1.10
************************************************************/



SSH Localhost


Generate ~/.ssh/id_rsa and the ~/.ssh/id_rsa.pub files using

$ ssh-keygen -t rsa

Enable Remote Login
“System Preferences” -> “Sharing”. Check “Remote Login”






Authorize SSH Keys
To allow your system to accept login, we have to make it aware of the keys that will be used

$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Login.

$ ssh localhost




The authenticity of host 'localhost (::1)' can't be established.



ECDSA key fingerprint is SHA256:WmIZBk7JuaSCNEKV81Lj1skRPCI4G+J/vvixV5MyXiY.



Are you sure you want to continue connecting (yes/no)? yes



Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.



Last login: Tue Sep  6 11:36:38 2016








$ exit








logout








Connection to localhost closed.

Running Hadoop

Start Hadoop

$ hstart



$ hstart




16/09/06 13:52:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable




Starting namenodes on [localhost]




localhost: starting namenode, logging to /usr/local/Cellar/hadoop/2.7.2/libexec/logs/hadoop-*************-namenode-*************-MBP.home.out




localhost: starting datanode, logging to /usr/local/Cellar/hadoop/2.7.2/libexec/logs/hadoop-*************-datanode-*************-MBP.home.out




Starting secondary namenodes [0.0.0.0]




0.0.0.0: starting secondarynamenode, logging to /usr/local/Cellar/hadoop/2.7.2/libexec/logs/hadoop-*************-secondarynamenode-*************-MBP.home.out




16/09/06 13:52:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable




starting yarn daemons




starting resourcemanager, logging to /usr/local/Cellar/hadoop/2.7.2/libexec/logs/yarn-*************-resourcemanager-*************-MBP.home.out




localhost: starting nodemanager, logging to /usr/local/Cellar/hadoop/2.7.2/libexec/logs/yarn-*************-nodemanager-*************-MBP.home.out

Stop Hadoop

$ hstop



$ hstop



stopping yarn daemons



stopping resourcemanager



localhost: stopping nodemanager



no proxyserver to stop



16/09/06 13:51:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable



Stopping namenodes on [localhost]



localhost: stopping namenode



localhost: stopping datanode



Stopping secondary namenodes [0.0.0.0]



0.0.0.0: stopping secondarynamenode



16/09/06 13:51:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable



Web Interface

Resource Manager: http://localhost:50070





JobTracker: http://localhost:8088 



Specific Node Information: http://localhost:8042 

Command
$ jps 
7379 DataNode
7459 SecondaryNameNode
7316 NameNode
7636 NodeManager
7562 ResourceManager
7676 Jps

$ yarn  // For resource management more information than the web interface. 
$ mapred  // Detailed information about jobs



Captain Nemo New York

 Reference: Special thanks to MAREK for Installing Hadoop on Mac part 1

Installing Hive on MacOS



Prerequisite

MySQL 5.6.22 is already installed.
Versions (2018/03/14)
Hadoop 3.0.0 (Since my earlier installed version of Hadoop was 2.7 , I upgraded it to 3.0.0 using brew)

Hive 2.3.1

Upgrade Hadoop & Install Hive


$ brew update
$ brew upgrade hadoop
$ brew install hive
Configure environment variables



# ~/.bashrc
export HADOOP_HOME=/usr/local/Cellar/hadoop/3.3.0/libexec
export HIVE_HOME=/usr/local/Cellar/hive/2.7.1/libexec

Install MySQL & Download JDBC



$ brew install mysql
To start the MySql Server:

We've installed your MySQL database without a root password. To secure it run:

    mysql_secure_installation


MySQL is configured to only allow connections from localhost by default


To connect run:

    mysql -uroot


To have launchd start mysql now and restart at login:

  brew services start mysql

Or, if you don't want/need a background service you can just run:

  mysql.server start

Go to mysql page and download the latest jdbc (sign up is required)
http://dev.mysql.com/downloads/connector/j/


$ tar zxvf mysql-connector-java-5.1.35.tar.gz
$ sudo cp mysql-connector-java-5.1.35/mysql-connector-java-5.1.35-bin.jar /usr/local/Cellar/hive/2.7.1/libexec/lib/


Setup MySQL database





$ mysql
mysql> CREATE DATABASE metastore;
mysql> USE metastore;
mysql> CREATE USER 'hiveuser'@'localhost' IDENTIFIED BY 'password';
mysql> GRANT SELECT,INSERT,UPDATE,DELETE,ALTER,CREATE ON metastore.* TO 'hiveuser'@'localhost';


Copy hive-default-xml to hive-site.xml


$ cd /usr/local/Cellar/hive/2.7.1/libexec/conf
$ cp hive-default.xml.template hive-site.xml


Edit following lines in hive-site.xml



































<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://localhost/metastore</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.jdbc.Driver</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>hiveuser</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>password</value>
</property>
<property>
  <name>datanucleus.fixedDatastore</name>
  <value>false</value>
</property>
<property>
    <name>hive.exec.local.scratchdir</name>
    <value>/tmp/hive</value>
    <description>Local scratch space for Hive jobs</description>
</property>
<property>
    <name>hive.downloaded.resources.dir</name>
    <value>/tmp/hive</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
</property>
<property>
    <name>hive.querylog.location</name>
    <value>/tmp/hive</value>
    <description>Location of Hive run time structured log file</description>
</property>


Run hive


$ hive
hive > show tables;


Install Apache HBase on Mac

 

 Prerequisite

Zookeeper: brew install zookeeper

To have launchd start zookeeper now and restart at login:

  brew services start zookeeper

Or, if you don't want/need a background service you can just run:

  zkServer start

Install Hbase

brew install hbase

Updating Homebrew...

==> Downloading https://homebrew.bintray.com/bottles/lzo-2.10.catalina.bottle.tar.gz

######################################################################## 100.0%

==> Downloading https://homebrew.bintray.com/bottles/openjdk%4011-11.0.9.catalina.bottle.tar.gz

==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/c640eade77c3ad69fef4d66872bbccc2e8782fcd5beee84ecb6c5b7dbb28081b?response-content-disposition=attachment%3Bfilename%3D%22openjdk%4011-11.

######################################################################## 100.0%

==> Downloading https://homebrew.bintray.com/bottles/hbase-2.3.2.catalina.bottle.tar.gz

==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/b18c993c5f4bffe90725bec08d91869878ade5c407d51d93567fbbbd5a1754da?response-content-disposition=attachment%3Bfilename%3D%22hbase-2.3.2.cata

######################################################################## 100.0%

==> Installing dependencies for hbase: lzo and openjdk@11

==> Installing hbase dependency: lzo

==> Pouring lzo-2.10.catalina.bottle.tar.gz

🍺  /usr/local/Cellar/lzo/2.10: 31 files, 546.7KB

==> Installing hbase dependency: openjdk@11

==> Pouring openjdk@11-11.0.9.catalina.bottle.tar.gz

==> Caveats

For the system Java wrappers to find this JDK, symlink it with

  sudo ln -sfn /usr/local/opt/openjdk@11/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk-11.jdk


openjdk@11 is keg-only, which means it was not symlinked into /usr/local,

because this is an alternate version of another formula.


If you need to have openjdk@11 first in your PATH run:

  echo 'export PATH="/usr/local/opt/openjdk@11/bin:$PATH"' >> ~/.zshrc


For compilers to find openjdk@11 you may need to set:

  export CPPFLAGS="-I/usr/local/opt/openjdk@11/include"


==> Summary

🍺  /usr/local/Cellar/openjdk@11/11.0.9: 653 files, 297.2MB

==> Installing hbase

==> Pouring hbase-2.3.2.catalina.bottle.tar.gz

==> Caveats

To have launchd start hbase now and restart at login:

  brew services start hbase

Or, if you don't want/need a background service you can just run:

  /usr/local/opt/hbase/bin/start-hbase.sh

==> Summary

🍺  /usr/local/Cellar/hbase/2.3.2: 499 files, 280.5MB

==> Caveats

==> openjdk@11

For the system Java wrappers to find this JDK, symlink it with

  sudo ln -sfn /usr/local/opt/openjdk@11/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk-11.jdk


openjdk@11 is keg-only, which means it was not symlinked into /usr/local,

because this is an alternate version of another formula.


If you need to have openjdk@11 first in your PATH run:

  echo 'export PATH="/usr/local/opt/openjdk@11/bin:$PATH"' >> ~/.zshrc


For compilers to find openjdk@11 you may need to set:

  export CPPFLAGS="-I/usr/local/opt/openjdk@11/include"


==> hbase

To have launchd start hbase now and restart at login:

  brew services start hbase

Or, if you don't want/need a background service you can just run:

  /usr/local/opt/hbase/bin/start-hbase.sh

Starting Hbase

 brew services start hbase

==> Tapping homebrew/services

Cloning into '/usr/local/Homebrew/Library/Taps/homebrew/homebrew-services'...

remote: Enumerating objects: 46, done.

remote: Counting objects: 100% (46/46), done.

remote: Compressing objects: 100% (46/46), done.

remote: Total 958 (delta 22), reused 9 (delta 0), pack-reused 912

Receiving objects: 100% (958/958), 270.77 KiB | 997.00 KiB/s, done.

Resolving deltas: 100% (396/396), done.

Tapped 1 command (40 files, 352.8KB).

==> Successfully started `hbase` (label: homebrew.mxcl.hbase)

Validating Install

hbase shell

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/3.3.0/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/Cellar/hbase/2.3.2/libexec/lib/client-facing-thirdparty/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.apache.hadoop.hbase.util.UnsafeAvailChecker (file:/usr/local/Cellar/hbase/2.3.2/libexec/lib/hbase-common-2.3.2.jar) to method java.nio.Bits.unaligned()

WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.hbase.util.UnsafeAvailChecker

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

2020-11-04 15:48:11,821 WARN  [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

HBase Shell

Use "help" to get list of supported commands.

Use "exit" to quit this interactive shell.

For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell

Version 2.3.2, r8dc7de1c75ab615b8992b928150e8b476cb053e3, Sun Sep 20 13:28:26 UTC 2020

Took 0.0006 seconds                                                                                                                                                                                 

hbase(main):001:0> 

Stopping Hbase

brew services stop hbase 

Stopping `hbase`... (might take a while)

==> Successfully stopped `hbase` (label: homebrew.mxcl.hbase)

- No comments:
Labels: , , , , ,
Subscribe to: Posts (Atom)

Scala & Spark for Managing & Analyzing Big Data (Using Machine Learning)

Managing & Analyzing Big Data using Apache Scala & Apache Spark In this blog we will see how to use Scala and Spark to analyze Big D...