edu.isi.hadoop.icmptrain

Class ICMPTrainRecordReader

java.lang.Object

edu.isi.hadoop.icmptrain.ICMPTrainRecordReader

public class ICMPTrainRecordReader
extends java.lang.Object

This class can serve as input reader (option -inputreader ICMPTrainRecordReader to hadoop streaming), for reading LANDER icmptrain data files. Due to deprecation of -inputreader the newer -inputformat edu.isi.hadoop.icmptrain.ICMPTrainInputFormat option should be used.

This reader parses and prints out contents of binary icmptrain output file, possibly compressed. These are the files that we use to record data from Internet censuses and surveys.

CONFIGURATION

What is printed can be configured using jobconf (option -D) as follows:

• stream.recordreader.icmptrain.keys - specify fields used as keys
• stream.recordreader.icmptrain.vals - specify fields used as values

Fields can be:

Name	Length	Comment
type	1 byte	ICMP type field
code	1 byte	ICMP code field
typeandcode	2 bytes	ICMP type and code concatenated
typecode	1 byte	ICMP type and code fiedls bunched together (legacy)
ttl	1 byte	ICMP response remaining time to live
time	4 bytes	Seconds since the Epoch that the probe was sent
rtt	4 bytes	Round trip time in microseconds (if replied or zero)
flags	1 byte	Format flags
probeaddr	4 bytes	probed IP adddress (or zero if didn't match)
replyaddr	4 bytes	IP address of the responder (if any or zero)

These fields can be grouped by using underscore('_') or comma(',') as a separator.

Examples:

        -D stream.recordreader.icmptrain.keys=probeaddr
        -D stream.recordreader.icmptrain.vals=type_code_typecode
        -D stream.recordreader.icmptrain.vals=type,code,typeandcode
 

As mentioned earlier that data to this reader can be a compressed stream. Few codecs split the input file into chunks while many process one whole file. If a codec can handle split file, it will be done so seamlessly. But this behavior can be over-ridden by setting option mapred.input.codec.nosplitting to true as follows:

        -D mapred.input.codec.nosplitting=true
 

As said earlier that ICMP data is binary. So whenever ICMP file is split, this reader needs to align itself with an ICMP record boundary. This reader uses few heuristics for that. The strength of this heuristic can be controlled by the option stream.recordreader.icmptrain.lookaheadcount as follows:

        -D stream.recordreader.icmptrain.lookaheadcount=100
 

In this case the reader will use 100 ICMP records to verify that its alignment choice is right.

OUTPUT

All keys and values are printed in hex (without leading 0x) in fixed length format (number of hex digits being the number of bytes times two). All fields are separated by TABs.

USAGE

  HADOOP=$HADOOP_HOME/bin/hadoop
  STREAMING_JAR=$HADOOP_HOME/build/hadoop-streaming.jar
  INPUTREADER_JAR=$HADOOP_HOME/build/hadoop-icmptrain.jar
  INPUTREADER_CLASS=edu.isi.hadoop.icmptrain.ICMPTrainRecordReader ... export
  INPUTFORMAT_CLASS=edu.isi.hadoop.icmptrain.ICMPTrainInputFormat
  HADOOP_CLASSPATH=${INPUTREADER_JAR} ... $HADOOP jar $STREAMING_JAR -file
  $INPUTREADER_JAR -inputreader $INPUTREADER_CLASS ... ...
  or
  $INPUTREADER_JAR -inputformat $INPUTFORMAT_CLASS ... ...
 

Constructor Summary

Constructors

Constructor and Description
ICMPTrainRecordReader(Configuration job, FileSplit split)
ICMPTrainRecordReader(FSDataInputStream in, FileSplit split, Reporter reporter, JobConf job, FileSystem fs)

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
Text	createKey()
Text	createValue()
long	getPos()
float	getProgress()
boolean	next(Text key, Text value)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ICMPTrainRecordReader

public ICMPTrainRecordReader(FSDataInputStream in,
                     FileSplit split,
                     Reporter reporter,
                     JobConf job,
                     FileSystem fs)
                      throws java.io.IOException,
                             java.lang.NumberFormatException

Throws:

java.io.IOException

java.lang.NumberFormatException

ICMPTrainRecordReader

public ICMPTrainRecordReader(Configuration job,
                     FileSplit split)
                      throws java.io.IOException,
                             java.lang.NumberFormatException

Throws:

java.io.IOException

java.lang.NumberFormatException

Method Detail

next

public boolean next(Text key,
           Text value)
             throws java.io.IOException

Throws:

java.io.IOException

createKey

public Text createKey()

createValue

public Text createValue()

getProgress

public float getProgress()
                  throws java.io.IOException

Throws:

java.io.IOException

getPos

public long getPos()
            throws java.io.IOException

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Throws:

java.io.IOException