Big Data

HADOOP

Startup scripts

The $HADOOP_INSTALL/hadoop/bin directory contains some scripts used to launch Hadoop DFS and Hadoop Map/Reduce daemons. These are:

 start-dfs.sh - Starts the Hadoop DFS daemons, the namenode and datanodes. Use this before start-mapred.sh
 stop-dfs.sh - Stops the Hadoop DFS daemons.
 start-mapred.sh - Starts the Hadoop Map/Reduce daemons, the jobtracker and tasktrackers.
 stop-mapred.sh - Stops the Hadoop Map/Reduce daemons.
 start-all.sh - Starts all Hadoop daemons, the namenode, datanodes, the jobtracker and tasktrackers. Deprecated; use start-dfs.sh then start-mapred.sh
 stop-all.sh - Stops all Hadoop daemons. Deprecated; use stop-mapred.sh then stop-dfs.sh

File System Shell

mkdir

Usage:

hdfs dfs -mkdir [-p] <paths>

Takes path uri's as argument and creates directories.

Options: The -p option behavior is much like Unix mkdir -p, creating parent directories along the path.

Example:(click to open)

 hdfs dfs -mkdir /user/hadoop/dir1 /user/hadoop/dir2
 hdfs dfs -mkdir hdfs://nn1.example.com/user/hadoop/dir hdfs://nn2.example.com/user/hadoop/dir

Exit Code: Returns 0 on success and -1 on error.

ls

Usage:

 hdfs dfs -ls <args> 

For a file returns stat on the file with the following format:

 permissions number_of_replicas userid groupid filesize modification_date modification_time filename

For a directory it returns list of its direct children as in Unix. A directory is listed as:

 permissions userid groupid modification_date modification_time dirname

Example:(click to open)

 hdfs dfs -ls /user/hadoop/file1

Exit Code: Returns 0 on success and -1 on error.

rm

Usage:

 hdfs dfs -rm [-skipTrash] URI [URI ...] 

Delete files specified as args. Only deletes non empty directory and files. If the -skipTrash option is specified, the trash, if enabled, will be bypassed and the specified file(s) deleted immediately. This can be useful when it is necessary to delete files from an over-quota directory. Refer to rmr for recursive deletes.

Example:(click to open)

 hdfs dfs -rm hdfs://nn.example.com/file /user/hadoop/emptydir

Exit Code: Returns 0 on success and -1 on error.