public static class ReduceTask.ReduceCopier<K,V> extends java.lang.Object implements ShuffleConsumerPlugin
Modifier and Type | Class and Description |
---|---|
static class |
ReduceTask.ReduceCopier.MapOutput
Describes the output of a map; could either be on disk or in-memory.
|
class |
ReduceTask.ReduceCopier.MapOutputCopier
Copies map outputs as they become available
|
class |
ReduceTask.ReduceCopier.MapOutputLocation
Abstraction to track a map-output.
|
class |
ReduceTask.ReduceCopier.ShuffleClientMetrics
This class contains the methods that should be used for metrics-reporting
the specific metrics for shuffle.
|
ShuffleConsumerPlugin.Context
Modifier and Type | Field and Description |
---|---|
static long |
COUNTER_UPDATE_INTERVAL |
static long |
DEFAULT_DISK_HEALTH_CHECK_INTERVAL
How often TaskTracker needs to check the health of its disks, if not
configured using mapred.disk.healthChecker.interval
|
static int |
FILE_NOT_FOUND |
static java.lang.String |
FOR_REDUCE_TASK
The reduce task number for which this map output is being transferred
|
static java.lang.String |
FROM_MAP_TASK
The map task from which the map output data is being transferred
|
static int |
HEARTBEAT_INTERVAL_MIN_DEFAULT |
static java.lang.String |
MAP_OUTPUT_LENGTH
The custom http header used for the map output length.
|
static java.lang.String |
RAW_MAP_OUTPUT_LENGTH
The custom http header used for the "raw" map output length.
|
static int |
SUCCESS |
static java.lang.String |
WORKDIR |
Constructor and Description |
---|
ReduceTask.ReduceCopier() |
Modifier and Type | Method and Description |
---|---|
protected void |
checkAndInformJobTracker(int failures,
TaskAttemptID mapId,
boolean readError) |
void |
close()
close and clean any resource associated with this object.
|
protected boolean |
closeMerger() |
RawKeyValueIterator |
createKVIterator()
Create a RawKeyValueIterator from copied map outputs.
|
boolean |
fetchOutputs()
To fetch the map outputs.
|
java.lang.Throwable |
getMergeThrowable()
To get any exception from merge.
|
void |
init(ShuffleConsumerPlugin.Context context)
To initialize the reduce copier plugin.
|
protected void |
initMerger() |
protected ReduceTask.ReduceCopier.MapOutput |
shuffle(ReduceTask.ReduceCopier.MapOutputCopier copier,
ReduceTask.ReduceCopier.MapOutputLocation mapOutputLoc,
java.net.URLConnection connection,
java.io.InputStream input,
ReduceTask.ReduceCopier.ShuffleClientMetrics shuffleClientMetrics,
org.apache.hadoop.fs.Path filename,
long decompressedLength,
long compressedLength) |
public static final int HEARTBEAT_INTERVAL_MIN_DEFAULT
public static final long COUNTER_UPDATE_INTERVAL
public static final long DEFAULT_DISK_HEALTH_CHECK_INTERVAL
public static final int SUCCESS
public static final int FILE_NOT_FOUND
public static final java.lang.String MAP_OUTPUT_LENGTH
public static final java.lang.String RAW_MAP_OUTPUT_LENGTH
public static final java.lang.String FROM_MAP_TASK
public static final java.lang.String FOR_REDUCE_TASK
public static final java.lang.String WORKDIR
public void init(ShuffleConsumerPlugin.Context context) throws java.lang.ClassNotFoundException, java.io.IOException
ShuffleConsumerPlugin
init
in interface ShuffleConsumerPlugin
context
- reduce copier context.java.lang.ClassNotFoundException
java.io.IOException
protected void initMerger() throws java.io.IOException
java.io.IOException
public boolean fetchOutputs() throws java.io.IOException
ShuffleConsumerPlugin
fetchOutputs
in interface ShuffleConsumerPlugin
java.io.IOException
protected boolean closeMerger()
protected ReduceTask.ReduceCopier.MapOutput shuffle(ReduceTask.ReduceCopier.MapOutputCopier copier, ReduceTask.ReduceCopier.MapOutputLocation mapOutputLoc, java.net.URLConnection connection, java.io.InputStream input, ReduceTask.ReduceCopier.ShuffleClientMetrics shuffleClientMetrics, org.apache.hadoop.fs.Path filename, long decompressedLength, long compressedLength) throws java.io.IOException, java.lang.InterruptedException
java.io.IOException
java.lang.InterruptedException
protected void checkAndInformJobTracker(int failures, TaskAttemptID mapId, boolean readError)
public RawKeyValueIterator createKVIterator() throws java.io.IOException
createKVIterator
in interface ShuffleConsumerPlugin
java.io.IOException
public java.lang.Throwable getMergeThrowable()
ShuffleConsumerPlugin
getMergeThrowable
in interface ShuffleConsumerPlugin
public void close()
ShuffleConsumerPlugin
close
in interface ShuffleConsumerPlugin
Copyright © 2009 The Apache Software Foundation