Package | Description |
---|---|
org.apache.hadoop.contrib.index.example | |
org.apache.hadoop.contrib.index.mapred | |
org.apache.hadoop.contrib.utils.join | |
org.apache.hadoop.examples |
Hadoop example code.
|
org.apache.hadoop.examples.dancing |
This package is a distributed implementation of Knuth's dancing links
algorithm that can run under Hadoop.
|
org.apache.hadoop.examples.terasort |
This package consists of 3 map/reduce applications for Hadoop to
compete in the annual terabyte sort
competition.
|
org.apache.hadoop.mapred |
A software framework for easily writing applications which process vast
amounts of data (multi-terabyte data-sets) parallelly on large clusters
(thousands of nodes) built of commodity hardware in a reliable, fault-tolerant
manner.
|
org.apache.hadoop.mapred.jobcontrol |
Utilities for managing dependent jobs.
|
org.apache.hadoop.mapred.join |
Given a set of sorted datasets keyed with the same class and yielding equal
partitions, it is possible to effect a join of those datasets prior to the map.
|
org.apache.hadoop.mapred.lib |
Library of generally useful mappers, reducers, and partitioners.
|
org.apache.hadoop.mapred.lib.aggregate |
Classes for performing various counting and aggregations.
|
org.apache.hadoop.mapred.lib.db |
org.apache.hadoop.mapred.lib.db Package
|
org.apache.hadoop.mapred.pipes |
Hadoop Pipes allows C++ code to use Hadoop DFS and map/reduce.
|
org.apache.hadoop.mapreduce.task | |
org.apache.hadoop.streaming |
Hadoop Streaming is a utility which allows users to create and run
Map-Reduce jobs with any executables (e.g.
|
org.apache.hadoop.util |
Modifier and Type | Method and Description |
---|---|
void |
IdentityLocalAnalysis.configure(JobConf job) |
void |
LineDocLocalAnalysis.configure(JobConf job) |
RecordReader<DocumentID,LineDocTextAndOp> |
LineDocInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter) |
Modifier and Type | Method and Description |
---|---|
void |
IndexUpdateCombiner.configure(JobConf job) |
void |
IndexUpdateMapper.configure(JobConf job) |
void |
IndexUpdatePartitioner.configure(JobConf job) |
void |
IndexUpdateReducer.configure(JobConf job) |
RecordWriter<Shard,org.apache.hadoop.io.Text> |
IndexUpdateOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem fs,
JobConf job,
java.lang.String name,
org.apache.hadoop.util.Progressable progress) |
Modifier and Type | Field and Description |
---|---|
protected JobConf |
DataJoinMapperBase.job |
protected JobConf |
DataJoinReducerBase.job |
Modifier and Type | Method and Description |
---|---|
static JobConf |
DataJoinJob.createDataJoinJob(java.lang.String[] args) |
Modifier and Type | Method and Description |
---|---|
TaggedMapOutput |
TaggedMapOutput.clone(JobConf job) |
void |
DataJoinMapperBase.configure(JobConf job) |
void |
DataJoinReducerBase.configure(JobConf job) |
void |
JobBase.configure(JobConf job)
Initializes a new instance from a
JobConf . |
static boolean |
DataJoinJob.runJob(JobConf job)
Submit/run a map/reduce job.
|
Modifier and Type | Method and Description |
---|---|
JobConf |
SleepJob.setupJobConf(int numMapper,
int numReducer,
long mapSleepTime,
int mapSleepCount,
long reduceSleepTime,
int reduceSleepCount) |
Modifier and Type | Method and Description |
---|---|
void |
PiEstimator.PiReducer.configure(JobConf job)
Store job configuration.
|
void |
SleepJob.configure(JobConf job) |
static java.math.BigDecimal |
PiEstimator.estimate(int numMaps,
long numPoints,
JobConf jobConf)
Run a map/reduce job for estimating Pi.
|
RecordReader<org.apache.hadoop.io.IntWritable,org.apache.hadoop.io.IntWritable> |
SleepJob.SleepInputFormat.getRecordReader(InputSplit ignored,
JobConf conf,
Reporter reporter) |
InputSplit[] |
SleepJob.SleepInputFormat.getSplits(JobConf conf,
int numSplits) |
Modifier and Type | Method and Description |
---|---|
void |
DistributedPentomino.PentMap.configure(JobConf conf) |
Modifier and Type | Method and Description |
---|---|
static boolean |
TeraOutputFormat.getFinalSync(JobConf conf)
Does the user want a final sync at close?
|
RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text> |
TeraInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter) |
RecordWriter<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text> |
TeraOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
JobConf job,
java.lang.String name,
org.apache.hadoop.util.Progressable progress) |
InputSplit[] |
TeraInputFormat.getSplits(JobConf conf,
int splits) |
static void |
TeraOutputFormat.setFinalSync(JobConf conf,
boolean newValue)
Set the requirement for a final sync before the stream is closed.
|
static void |
TeraInputFormat.writePartitionFile(JobConf conf,
org.apache.hadoop.fs.Path partFile)
Use the input splits to take samples of the input and generate sample
keys.
|
Modifier and Type | Field and Description |
---|---|
protected JobConf |
Task.conf |
protected JobConf |
Task.CombinerRunner.job |
protected JobConf |
JobLocalizer.ttConf |
Modifier and Type | Method and Description |
---|---|
JobConf |
JobTracker.getConf()
Returns a handle to the JobTracker's Configuration
|
JobConf |
TaskTracker.getJobConf()
Get the default job conf for this tracker.
|
JobConf |
JobContext.getJobConf()
Get the job Configuration
|
JobConf |
TaskAttemptContext.getJobConf()
Deprecated.
|
JobConf |
MapOutputCollector.Context.getJobConf() |
JobConf |
JobContextImpl.getJobConf()
Deprecated.
Get the job Configuration
|
JobConf |
ShuffleConsumerPlugin.Context.getJobConf() |
JobConf |
TaskAttemptContextImpl.getJobConf()
Deprecated.
|
JobConf |
TaskTracker.getJobConf(JobID jobId)
Get the specific job conf for a running job.
|
Modifier and Type | Method and Description |
---|---|
static void |
FileInputFormat.addInputPath(JobConf conf,
org.apache.hadoop.fs.Path path)
Add a
Path to the list of inputs for the map-reduce job. |
static void |
FileInputFormat.addInputPaths(JobConf conf,
java.lang.String commaSeparatedPaths)
Add the given comma separated paths to the list of inputs for
the map-reduce job.
|
void |
OutputFormat.checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored,
JobConf job)
Check for validity of the output-specification for the job.
|
void |
FileOutputFormat.checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored,
JobConf job) |
void |
SequenceFileAsBinaryOutputFormat.checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored,
JobConf job) |
void |
JobConfigurable.configure(JobConf job)
Initializes a new instance from a
JobConf . |
void |
KeyValueTextInputFormat.configure(JobConf conf) |
void |
MapReduceBase.configure(JobConf job)
Default implementation that does nothing.
|
void |
MapRunner.configure(JobConf job) |
void |
TextInputFormat.configure(JobConf conf) |
static <K,V> Task.CombinerRunner<K,V> |
Task.CombinerRunner.create(JobConf job,
TaskAttemptID taskId,
Counters.Counter inputCounter,
Task.TaskReporter reporter,
OutputCommitter committer) |
void |
JobLocalizer.createWorkDir(JobConf jConf) |
static boolean |
FileOutputFormat.getCompressOutput(JobConf conf)
Is the job output compressed?
|
static org.apache.hadoop.fs.PathFilter |
FileInputFormat.getInputPathFilter(JobConf conf)
Get a PathFilter instance of the filter set for the input paths.
|
static org.apache.hadoop.fs.Path[] |
FileInputFormat.getInputPaths(JobConf conf)
Get the list of input
Path s for the map-reduce job. |
static java.lang.String |
JobHistory.JobInfo.getJobHistoryFileName(JobConf jobConf,
JobID id)
Recover the job history filename from the history folder.
|
static org.apache.hadoop.fs.Path |
JobHistory.JobInfo.getJobHistoryLogLocationForUser(java.lang.String logFileName,
JobConf jobConf)
Get the user job history file path
|
static org.apache.hadoop.io.SequenceFile.CompressionType |
SequenceFileOutputFormat.getOutputCompressionType(JobConf conf)
Get the
SequenceFile.CompressionType for the output SequenceFile . |
static java.lang.Class<? extends org.apache.hadoop.io.compress.CompressionCodec> |
FileOutputFormat.getOutputCompressorClass(JobConf conf,
java.lang.Class<? extends org.apache.hadoop.io.compress.CompressionCodec> defaultValue)
Get the
CompressionCodec for compressing the job outputs. |
static org.apache.hadoop.fs.Path |
FileOutputFormat.getOutputPath(JobConf conf)
Get the
Path to the output directory for the map-reduce job. |
static org.apache.hadoop.fs.Path |
FileOutputFormat.getPathForCustomFile(JobConf conf,
java.lang.String name)
Helper function to generate a
Path for a file that is unique for
the task within the job output directory. |
RecordReader<K,V> |
InputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter)
Get the
RecordReader for the given InputSplit . |
abstract RecordReader<K,V> |
FileInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter) |
RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text> |
KeyValueTextInputFormat.getRecordReader(InputSplit genericSplit,
JobConf job,
Reporter reporter) |
abstract RecordReader<K,V> |
MultiFileInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter)
Deprecated.
|
RecordReader<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable> |
SequenceFileAsBinaryInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter) |
RecordReader<K,V> |
SequenceFileInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter) |
RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text> |
SequenceFileAsTextInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter) |
RecordReader<K,V> |
SequenceFileInputFilter.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter)
Create a record reader for the given split
|
RecordReader<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text> |
TextInputFormat.getRecordReader(InputSplit genericSplit,
JobConf job,
Reporter reporter) |
RecordWriter<K,V> |
OutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
JobConf job,
java.lang.String name,
org.apache.hadoop.util.Progressable progress)
Get the
RecordWriter for the given job. |
abstract RecordWriter<K,V> |
FileOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
JobConf job,
java.lang.String name,
org.apache.hadoop.util.Progressable progress) |
RecordWriter<org.apache.hadoop.io.WritableComparable,org.apache.hadoop.io.Writable> |
MapFileOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
JobConf job,
java.lang.String name,
org.apache.hadoop.util.Progressable progress) |
RecordWriter<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable> |
SequenceFileAsBinaryOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
JobConf job,
java.lang.String name,
org.apache.hadoop.util.Progressable progress) |
RecordWriter<K,V> |
SequenceFileOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
JobConf job,
java.lang.String name,
org.apache.hadoop.util.Progressable progress) |
RecordWriter<K,V> |
TextOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
JobConf job,
java.lang.String name,
org.apache.hadoop.util.Progressable progress) |
java.lang.String |
TaskController.getRunAsUser(JobConf conf)
Returns the local unix user that a given job will run as.
|
static java.lang.Class<? extends org.apache.hadoop.io.WritableComparable> |
SequenceFileAsBinaryOutputFormat.getSequenceFileOutputKeyClass(JobConf conf)
Get the key class for the
SequenceFile |
static java.lang.Class<? extends org.apache.hadoop.io.Writable> |
SequenceFileAsBinaryOutputFormat.getSequenceFileOutputValueClass(JobConf conf)
Get the value class for the
SequenceFile |
InputSplit[] |
InputFormat.getSplits(JobConf job,
int numSplits)
Logically split the set of input files for the job.
|
InputSplit[] |
FileInputFormat.getSplits(JobConf job,
int numSplits)
Splits files returned by
FileInputFormat.listStatus(JobConf) when
they're too big. |
InputSplit[] |
MultiFileInputFormat.getSplits(JobConf job,
int numSplits)
Deprecated.
|
static long |
TaskLog.getTaskLogLength(JobConf conf)
Get the desired maximum length of task's logs.
|
static JobClient.TaskStatusFilter |
JobClient.getTaskOutputFilter(JobConf job)
Get the task output filter out of the JobConf.
|
static org.apache.hadoop.fs.Path |
FileOutputFormat.getTaskOutputPath(JobConf conf,
java.lang.String name)
Helper function to create the task's temporary output directory and
return the path to the task's output file.
|
static java.lang.String |
FileOutputFormat.getUniqueName(JobConf conf,
java.lang.String name)
Helper function to generate a name that is unique for the task.
|
static java.lang.String |
JobHistory.JobInfo.getUserName(JobConf jobConf)
Get the user name from the job conf
|
static org.apache.hadoop.fs.Path |
FileOutputFormat.getWorkOutputPath(JobConf conf)
Get the
Path to the task's temporary output directory
for the map-reduce job
Tasks' Side-Effect Files |
void |
JobClient.init(JobConf conf)
Connect to the default
JobTracker . |
static void |
JobHistory.init(JobTracker jobTracker,
JobConf conf,
java.lang.String hostname,
long jobTrackerStartTime)
Initialize JobHistory files.
|
void |
Task.initialize(JobConf job,
JobID id,
Reporter reporter,
boolean useNewApi) |
protected org.apache.hadoop.fs.FileStatus[] |
FileInputFormat.listStatus(JobConf job)
List input directories.
|
protected org.apache.hadoop.fs.FileStatus[] |
SequenceFileInputFormat.listStatus(JobConf job) |
void |
Task.localizeConfiguration(JobConf conf)
Localize the given JobConf to be specific for this task.
|
void |
ReduceTask.localizeConfiguration(JobConf conf)
Localize the given JobConf to be specific for this task.
|
void |
JobLocalizer.localizeJobFiles(JobID jobid,
JobConf jConf,
org.apache.hadoop.fs.Path localJobFile,
org.apache.hadoop.fs.Path localJobTokenFile,
TaskUmbilicalProtocol taskTracker) |
void |
JobLocalizer.localizeJobFiles(JobID jobid,
JobConf jConf,
org.apache.hadoop.fs.Path localJobTokenFile,
TaskUmbilicalProtocol taskTracker) |
static void |
JobEndNotifier.localRunnerNotification(JobConf conf,
JobStatus status) |
static void |
JobHistory.JobInfo.logSubmitted(JobID jobId,
JobConf jobConf,
java.lang.String jobConfPath,
long submitTime)
Deprecated.
|
static void |
JobHistory.JobInfo.logSubmitted(JobID jobId,
JobConf jobConf,
java.lang.String jobConfPath,
long submitTime,
boolean restarted) |
boolean |
JobClient.monitorAndPrintJob(JobConf conf,
RunningJob job)
Monitor a job and print status in real-time as progress is made and tasks
fail.
|
static org.apache.hadoop.fs.Path |
JobHistory.JobInfo.recoverJobHistoryFile(JobConf conf,
org.apache.hadoop.fs.Path logFilePath)
Selects one of the two files generated as a part of recovery.
|
static void |
JobEndNotifier.registerNotification(JobConf jobConf,
JobStatus status) |
abstract void |
Task.run(JobConf job,
TaskUmbilicalProtocol umbilical)
Run this task as a part of the named job.
|
void |
MapTask.run(JobConf job,
TaskUmbilicalProtocol umbilical) |
void |
ReduceTask.run(JobConf job,
TaskUmbilicalProtocol umbilical) |
static RunningJob |
JobClient.runJob(JobConf job)
Utility that submits a job, then polls for progress until the job is
complete.
|
static void |
FileOutputFormat.setCompressOutput(JobConf conf,
boolean compress)
Set whether the output of the job is compressed.
|
static void |
FileInputFormat.setInputPathFilter(JobConf conf,
java.lang.Class<? extends org.apache.hadoop.fs.PathFilter> filter)
Set a PathFilter to be applied to the input paths for the map-reduce job.
|
static void |
FileInputFormat.setInputPaths(JobConf conf,
org.apache.hadoop.fs.Path... inputPaths)
Set the array of
Path s as the list of inputs
for the map-reduce job. |
static void |
FileInputFormat.setInputPaths(JobConf conf,
java.lang.String commaSeparatedPaths)
Sets the given comma separated paths as the list of inputs
for the map-reduce job.
|
static void |
SequenceFileOutputFormat.setOutputCompressionType(JobConf conf,
org.apache.hadoop.io.SequenceFile.CompressionType style)
Set the
SequenceFile.CompressionType for the output SequenceFile . |
static void |
FileOutputFormat.setOutputCompressorClass(JobConf conf,
java.lang.Class<? extends org.apache.hadoop.io.compress.CompressionCodec> codecClass)
Set the
CompressionCodec to be used to compress job outputs. |
static void |
FileOutputFormat.setOutputPath(JobConf conf,
org.apache.hadoop.fs.Path outputDir)
Set the
Path of the output directory for the map-reduce job. |
static void |
SequenceFileAsBinaryOutputFormat.setSequenceFileOutputKeyClass(JobConf conf,
java.lang.Class<?> theClass)
Set the key class for the
SequenceFile |
static void |
SequenceFileAsBinaryOutputFormat.setSequenceFileOutputValueClass(JobConf conf,
java.lang.Class<?> theClass)
Set the value class for the
SequenceFile |
static void |
SkipBadRecords.setSkipOutputPath(JobConf conf,
org.apache.hadoop.fs.Path path)
Set the directory to which skipped records are written.
|
static void |
JobClient.setTaskOutputFilter(JobConf job,
JobClient.TaskStatusFilter newValue)
Modify the JobConf to set the task output filter.
|
void |
JobTrackerHADaemon.JobTrackerRunner.startJobTracker(JobConf conf) |
static JobTracker |
JobTracker.startTracker(JobConf conf)
Start the JobTracker with given configuration.
|
static JobTracker |
JobTracker.startTracker(JobConf conf,
java.lang.String identifier) |
RunningJob |
JobClient.submitJob(JobConf job)
Submit a job to the MR system.
|
RunningJob |
JobClient.submitJobInternal(JobConf job)
Internal method for submitting jobs to the system.
|
protected boolean |
Task.supportIsolationRunner(JobConf conf) |
void |
Task.writeFilesRequiredForRerun(JobConf conf)
Write files that the IsolationRunner will need to rerun the task.
|
void |
MapTask.writeFilesRequiredForRerun(JobConf conf) |
static void |
JobLocalizer.writeLocalJobFile(org.apache.hadoop.fs.Path jobFile,
JobConf conf)
Write the task specific job-configuration file.
|
void |
SpillRecord.writeToFile(org.apache.hadoop.fs.Path loc,
JobConf job)
Write this spill record to the location provided.
|
void |
SpillRecord.writeToFile(org.apache.hadoop.fs.Path loc,
JobConf job,
java.util.zip.Checksum crc) |
Constructor and Description |
---|
FileSplit(org.apache.hadoop.fs.Path file,
long start,
long length,
JobConf conf)
Deprecated.
|
JobClient(JobConf conf)
Build a job client with the given
JobConf , and connect to the
default JobTracker . |
JobInProgress(JobID jobid,
JobConf conf,
JobTracker tracker)
Create an almost empty JobInProgress, which can be used only for tests
|
JobLocalizer(JobConf ttConf,
java.lang.String user,
java.lang.String jobid) |
JobLocalizer(JobConf ttConf,
java.lang.String user,
java.lang.String jobid,
java.lang.String... localDirs) |
LocalJobRunner(JobConf conf) |
MapOutputCollector.Context(MapTask mapTask,
JobConf jobConf,
Task.TaskReporter reporter) |
MultiFileSplit(JobConf job,
org.apache.hadoop.fs.Path[] files,
long[] lengths)
Deprecated.
|
ReduceTask.ReduceCopier.MapOutputCopier(JobConf job,
Reporter reporter,
javax.crypto.SecretKey jobTokenSecret) |
ShuffleConsumerPlugin.Context(TaskUmbilicalProtocol umbilical,
JobConf conf,
Task.TaskReporter reporter,
ReduceTask reduceTask) |
SpillRecord(org.apache.hadoop.fs.Path indexFileName,
JobConf job,
java.util.zip.Checksum crc,
java.lang.String expectedIndexOwner) |
SpillRecord(org.apache.hadoop.fs.Path indexFileName,
JobConf job,
java.lang.String expectedIndexOwner) |
Task.OldCombinerRunner(java.lang.Class<? extends Reducer<K,V,K,V>> cls,
JobConf conf,
Counters.Counter inputCounter,
Task.TaskReporter reporter) |
TaskInProgress(JobID jobid,
java.lang.String jobFile,
int numMaps,
int partition,
JobTracker jobtracker,
JobConf conf,
JobInProgress job,
int numSlotsRequired)
Constructor for ReduceTask
|
TaskInProgress(JobID jobid,
java.lang.String jobFile,
JobSplit.TaskSplitMetaInfo split,
JobTracker jobtracker,
JobConf conf,
JobInProgress job,
int partition,
int numSlotsRequired)
Constructor for MapTask
|
TaskTracker(JobConf conf)
Start with the local machine name, and the default JobTracker
|
Modifier and Type | Method and Description |
---|---|
JobConf |
Job.getJobConf()
Deprecated.
|
Modifier and Type | Method and Description |
---|---|
void |
Job.setJobConf(JobConf jobConf)
Deprecated.
Set the mapred job conf for this job.
|
Constructor and Description |
---|
Job(JobConf conf)
Deprecated.
|
Job(JobConf jobConf,
java.util.ArrayList<?> dependingJobs)
Deprecated.
Construct a job.
|
Modifier and Type | Method and Description |
---|---|
ComposableRecordReader<K,V> |
ComposableInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter) |
ComposableRecordReader<K,TupleWritable> |
CompositeInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter)
Construct a CompositeRecordReader for the children of this InputFormat
as defined in the init expression.
|
InputSplit[] |
CompositeInputFormat.getSplits(JobConf job,
int numSplits)
Build a CompositeInputSplit from the child InputFormats by assigning the
ith split from each child to the ith composite split.
|
void |
CompositeInputFormat.setFormat(JobConf job)
Interpret a given string as a composite expression.
|
Constructor and Description |
---|
JoinRecordReader(int id,
JobConf conf,
int capacity,
java.lang.Class<? extends org.apache.hadoop.io.WritableComparator> cmpcl) |
MultiFilterRecordReader(int id,
JobConf conf,
int capacity,
java.lang.Class<? extends org.apache.hadoop.io.WritableComparator> cmpcl) |
Modifier and Type | Field and Description |
---|---|
protected JobConf |
CombineFileRecordReader.jc |
Modifier and Type | Method and Description |
---|---|
JobConf |
CombineFileSplit.getJob() |
Modifier and Type | Method and Description |
---|---|
static void |
MultipleInputs.addInputPath(JobConf conf,
org.apache.hadoop.fs.Path path,
java.lang.Class<? extends InputFormat> inputFormatClass)
Add a
Path with a custom InputFormat to the list of
inputs for the map-reduce job. |
static void |
MultipleInputs.addInputPath(JobConf conf,
org.apache.hadoop.fs.Path path,
java.lang.Class<? extends InputFormat> inputFormatClass,
java.lang.Class<? extends Mapper> mapperClass)
|
static <K1,V1,K2,V2> |
ChainMapper.addMapper(JobConf job,
java.lang.Class<? extends Mapper<K1,V1,K2,V2>> klass,
java.lang.Class<? extends K1> inputKeyClass,
java.lang.Class<? extends V1> inputValueClass,
java.lang.Class<? extends K2> outputKeyClass,
java.lang.Class<? extends V2> outputValueClass,
boolean byValue,
JobConf mapperConf)
Adds a Mapper class to the chain job's JobConf.
|
static <K1,V1,K2,V2> |
ChainReducer.addMapper(JobConf job,
java.lang.Class<? extends Mapper<K1,V1,K2,V2>> klass,
java.lang.Class<? extends K1> inputKeyClass,
java.lang.Class<? extends V1> inputValueClass,
java.lang.Class<? extends K2> outputKeyClass,
java.lang.Class<? extends V2> outputValueClass,
boolean byValue,
JobConf mapperConf)
Adds a Mapper class to the chain job's JobConf.
|
static void |
MultipleOutputs.addMultiNamedOutput(JobConf conf,
java.lang.String namedOutput,
java.lang.Class<? extends OutputFormat> outputFormatClass,
java.lang.Class<?> keyClass,
java.lang.Class<?> valueClass)
Adds a multi named output for the job.
|
static void |
MultipleOutputs.addNamedOutput(JobConf conf,
java.lang.String namedOutput,
java.lang.Class<? extends OutputFormat> outputFormatClass,
java.lang.Class<?> keyClass,
java.lang.Class<?> valueClass)
Adds a named output for the job.
|
void |
NullOutputFormat.checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored,
JobConf job) |
void |
BinaryPartitioner.configure(JobConf job) |
void |
ChainMapper.configure(JobConf job)
Configures the ChainMapper and all the Mappers in the chain.
|
void |
ChainReducer.configure(JobConf job)
Configures the ChainReducer, the Reducer and all the Mappers in the chain.
|
void |
DelegatingMapper.configure(JobConf conf) |
void |
FieldSelectionMapReduce.configure(JobConf job) |
void |
HashPartitioner.configure(JobConf job) |
void |
KeyFieldBasedComparator.configure(JobConf job) |
void |
KeyFieldBasedPartitioner.configure(JobConf job) |
void |
MultithreadedMapRunner.configure(JobConf jobConf) |
void |
NLineInputFormat.configure(JobConf conf) |
void |
RegexMapper.configure(JobConf job) |
void |
TotalOrderPartitioner.configure(JobConf job)
Read in the partition file and build indexing data structures.
|
protected void |
CombineFileInputFormat.createPool(JobConf conf,
java.util.List<org.apache.hadoop.fs.PathFilter> filters)
Create a new pool and add the filters to it.
|
protected void |
CombineFileInputFormat.createPool(JobConf conf,
org.apache.hadoop.fs.PathFilter... filters)
Create a new pool and add the filters to it.
|
protected abstract RecordWriter<K,V> |
MultipleOutputFormat.getBaseRecordWriter(org.apache.hadoop.fs.FileSystem fs,
JobConf job,
java.lang.String name,
org.apache.hadoop.util.Progressable arg3) |
protected RecordWriter<K,V> |
MultipleSequenceFileOutputFormat.getBaseRecordWriter(org.apache.hadoop.fs.FileSystem fs,
JobConf job,
java.lang.String name,
org.apache.hadoop.util.Progressable arg3) |
protected RecordWriter<K,V> |
MultipleTextOutputFormat.getBaseRecordWriter(org.apache.hadoop.fs.FileSystem fs,
JobConf job,
java.lang.String name,
org.apache.hadoop.util.Progressable arg3) |
static boolean |
MultipleOutputs.getCountersEnabled(JobConf conf)
Returns if the counters for the named outputs are enabled or not.
|
protected java.lang.String |
MultipleOutputFormat.getInputFileBasedOutputFileName(JobConf job,
java.lang.String name)
Generate the outfile name based on a given anme and the input file name.
|
static java.lang.Class<? extends OutputFormat> |
MultipleOutputs.getNamedOutputFormatClass(JobConf conf,
java.lang.String namedOutput)
Returns the named output OutputFormat.
|
static java.lang.Class<? extends org.apache.hadoop.io.WritableComparable> |
MultipleOutputs.getNamedOutputKeyClass(JobConf conf,
java.lang.String namedOutput)
Returns the key class for a named output.
|
static java.util.List<java.lang.String> |
MultipleOutputs.getNamedOutputsList(JobConf conf)
Returns list of channel names.
|
static java.lang.Class<? extends org.apache.hadoop.io.Writable> |
MultipleOutputs.getNamedOutputValueClass(JobConf conf,
java.lang.String namedOutput)
Returns the value class for a named output.
|
static java.lang.String |
TotalOrderPartitioner.getPartitionFile(JobConf job)
Get the path to the SequenceFile storing the sorted partition keyset.
|
abstract RecordReader<K,V> |
CombineFileInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter)
This is not implemented yet.
|
RecordReader<K,V> |
DelegatingInputFormat.getRecordReader(InputSplit split,
JobConf conf,
Reporter reporter) |
RecordReader<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text> |
NLineInputFormat.getRecordReader(InputSplit genericSplit,
JobConf job,
Reporter reporter) |
RecordWriter<K,V> |
MultipleOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem fs,
JobConf job,
java.lang.String name,
org.apache.hadoop.util.Progressable arg3)
Create a composite record writer that can write key/value data to different
output files
|
RecordWriter<K,V> |
NullOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
JobConf job,
java.lang.String name,
org.apache.hadoop.util.Progressable progress) |
K[] |
InputSampler.Sampler.getSample(InputFormat<K,V> inf,
JobConf job)
For a given job, collect and return a subset of the keys from the
input data.
|
K[] |
InputSampler.SplitSampler.getSample(InputFormat<K,V> inf,
JobConf job)
From each split sampled, take the first numSamples / numSplits records.
|
K[] |
InputSampler.RandomSampler.getSample(InputFormat<K,V> inf,
JobConf job)
Randomize the split order, then take the specified number of keys from
each split sampled, where each key is selected with the specified
probability and possibly replaced by a subsequently selected key when
the quota of keys from that split is satisfied.
|
K[] |
InputSampler.IntervalSampler.getSample(InputFormat<K,V> inf,
JobConf job)
For each split sampled, emit when the ratio of the number of records
retained to the total record count is less than the specified
frequency.
|
InputSplit[] |
CombineFileInputFormat.getSplits(JobConf job,
int numSplits) |
InputSplit[] |
DelegatingInputFormat.getSplits(JobConf conf,
int numSplits) |
InputSplit[] |
NLineInputFormat.getSplits(JobConf job,
int numSplits)
Logically splits the set of input files for the job, splits N lines
of the input as one split.
|
static boolean |
MultipleOutputs.isMultiNamedOutput(JobConf conf,
java.lang.String namedOutput)
Returns if a named output is multiple.
|
static void |
MultipleOutputs.setCountersEnabled(JobConf conf,
boolean enabled)
Enables or disables counters for the named outputs.
|
static void |
TotalOrderPartitioner.setPartitionFile(JobConf job,
org.apache.hadoop.fs.Path p)
Set the path to the SequenceFile storing the sorted partition keyset.
|
static <K1,V1,K2,V2> |
ChainReducer.setReducer(JobConf job,
java.lang.Class<? extends Reducer<K1,V1,K2,V2>> klass,
java.lang.Class<? extends K1> inputKeyClass,
java.lang.Class<? extends V1> inputValueClass,
java.lang.Class<? extends K2> outputKeyClass,
java.lang.Class<? extends V2> outputValueClass,
boolean byValue,
JobConf reducerConf)
Sets the Reducer class to the chain job's JobConf.
|
static <K,V> void |
InputSampler.writePartitionFile(JobConf job,
InputSampler.Sampler<K,V> sampler)
Write a partition file for the given job, using the Sampler provided.
|
Constructor and Description |
---|
CombineFileRecordReader(JobConf job,
CombineFileSplit split,
Reporter reporter,
java.lang.Class<RecordReader<K,V>> rrClass)
A generic RecordReader that can hand out different recordReaders
for each chunk in the CombineFileSplit.
|
CombineFileSplit(JobConf job,
org.apache.hadoop.fs.Path[] files,
long[] lengths) |
CombineFileSplit(JobConf job,
org.apache.hadoop.fs.Path[] files,
long[] start,
long[] lengths,
java.lang.String[] locations) |
InputSampler(JobConf conf) |
MultipleOutputs(JobConf job)
Creates and initializes multiple named outputs support, it should be
instantiated in the Mapper/Reducer configure method.
|
Modifier and Type | Method and Description |
---|---|
static JobConf |
ValueAggregatorJob.createValueAggregatorJob(java.lang.String[] args)
Create an Aggregate based map/reduce job.
|
static JobConf |
ValueAggregatorJob.createValueAggregatorJob(java.lang.String[] args,
java.lang.Class<?> caller)
Create an Aggregate based map/reduce job.
|
static JobConf |
ValueAggregatorJob.createValueAggregatorJob(java.lang.String[] args,
java.lang.Class<? extends ValueAggregatorDescriptor>[] descriptors) |
static JobConf |
ValueAggregatorJob.createValueAggregatorJob(java.lang.String[] args,
java.lang.Class<? extends ValueAggregatorDescriptor>[] descriptors,
java.lang.Class<?> caller) |
Modifier and Type | Method and Description |
---|---|
void |
UserDefinedValueAggregatorDescriptor.configure(JobConf job)
Do nothing.
|
void |
ValueAggregatorDescriptor.configure(JobConf job)
Configure the object
|
void |
ValueAggregatorBaseDescriptor.configure(JobConf job)
get the input file name.
|
void |
ValueAggregatorCombiner.configure(JobConf job)
Combiner does not need to configure.
|
void |
ValueAggregatorJobBase.configure(JobConf job) |
static void |
ValueAggregatorJob.setAggregatorDescriptors(JobConf job,
java.lang.Class<? extends ValueAggregatorDescriptor>[] descriptors) |
Constructor and Description |
---|
UserDefinedValueAggregatorDescriptor(java.lang.String className,
JobConf job) |
Modifier and Type | Method and Description |
---|---|
void |
DBOutputFormat.checkOutputSpecs(org.apache.hadoop.fs.FileSystem filesystem,
JobConf job)
Check for validity of the output-specification for the job.
|
void |
DBInputFormat.configure(JobConf job)
Initializes a new instance from a
JobConf . |
static void |
DBConfiguration.configureDB(JobConf job,
java.lang.String driverClass,
java.lang.String dbUrl)
Sets the DB access related fields in the JobConf.
|
static void |
DBConfiguration.configureDB(JobConf job,
java.lang.String driverClass,
java.lang.String dbUrl,
java.lang.String userName,
java.lang.String passwd)
Sets the DB access related fields in the JobConf.
|
RecordReader<org.apache.hadoop.io.LongWritable,T> |
DBInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter)
Get the
RecordReader for the given InputSplit . |
RecordWriter<K,V> |
DBOutputFormat.getRecordWriter(org.apache.hadoop.fs.FileSystem filesystem,
JobConf job,
java.lang.String name,
org.apache.hadoop.util.Progressable progress)
Get the
RecordWriter for the given job. |
InputSplit[] |
DBInputFormat.getSplits(JobConf job,
int chunks)
Logically split the set of input files for the job.
|
static void |
DBInputFormat.setInput(JobConf job,
java.lang.Class<? extends DBWritable> inputClass,
java.lang.String inputQuery,
java.lang.String inputCountQuery)
Initializes the map-part of the job with the appropriate input settings.
|
static void |
DBInputFormat.setInput(JobConf job,
java.lang.Class<? extends DBWritable> inputClass,
java.lang.String tableName,
java.lang.String conditions,
java.lang.String orderBy,
java.lang.String... fieldNames)
Initializes the map-part of the job with the appropriate input settings.
|
static void |
DBOutputFormat.setOutput(JobConf job,
java.lang.String tableName,
int fieldCount)
Initializes the reduce-part of the job with the appropriate output settings
|
static void |
DBOutputFormat.setOutput(JobConf job,
java.lang.String tableName,
java.lang.String... fieldNames)
Initializes the reduce-part of the job with the appropriate output settings
|
Constructor and Description |
---|
DBInputFormat.DBRecordReader(DBInputFormat.DBInputSplit split,
java.lang.Class<T> inputClass,
JobConf job,
java.sql.Connection conn,
DBConfiguration dbConfig,
java.lang.String cond,
java.lang.String[] fields,
java.lang.String table) |
Modifier and Type | Method and Description |
---|---|
static java.lang.String |
Submitter.getExecutable(JobConf conf)
Get the URI of the application's executable.
|
static boolean |
Submitter.getIsJavaMapper(JobConf conf)
Check whether the job is using a Java Mapper.
|
static boolean |
Submitter.getIsJavaRecordReader(JobConf conf)
Check whether the job is using a Java RecordReader
|
static boolean |
Submitter.getIsJavaRecordWriter(JobConf conf)
Will the reduce use a Java RecordWriter?
|
static boolean |
Submitter.getIsJavaReducer(JobConf conf)
Check whether the job is using a Java Reducer.
|
static boolean |
Submitter.getKeepCommandFile(JobConf conf)
Does the user want to keep the command file for debugging? If this is
true, pipes will write a copy of the command data to a file in the
task directory named "downlink.data", which may be used to run the C++
program under the debugger.
|
static RunningJob |
Submitter.jobSubmit(JobConf conf)
Submit a job to the Map-Reduce framework.
|
static RunningJob |
Submitter.runJob(JobConf conf)
Submit a job to the map/reduce cluster.
|
static void |
Submitter.setExecutable(JobConf conf,
java.lang.String executable)
Set the URI for the application's executable.
|
static void |
Submitter.setIsJavaMapper(JobConf conf,
boolean value)
Set whether the Mapper is written in Java.
|
static void |
Submitter.setIsJavaRecordReader(JobConf conf,
boolean value)
Set whether the job is using a Java RecordReader.
|
static void |
Submitter.setIsJavaRecordWriter(JobConf conf,
boolean value)
Set whether the job will use a Java RecordWriter.
|
static void |
Submitter.setIsJavaReducer(JobConf conf,
boolean value)
Set whether the Reducer is written in Java.
|
static void |
Submitter.setKeepCommandFile(JobConf conf,
boolean keep)
Set whether to keep the command file for debugging
|
static RunningJob |
Submitter.submitJob(JobConf conf)
Deprecated.
|
Modifier and Type | Field and Description |
---|---|
protected JobConf |
JobContextImpl.conf |
Modifier and Type | Field and Description |
---|---|
protected JobConf |
StreamJob.jobConf_ |
Modifier and Type | Method and Description |
---|---|
static JobConf |
StreamJob.createJob(java.lang.String[] argv)
This method creates a streaming job from the given argument list.
|
Modifier and Type | Method and Description |
---|---|
void |
AutoInputFormat.configure(JobConf job) |
void |
PipeMapRed.configure(JobConf job) |
void |
PipeMapper.configure(JobConf job) |
void |
PipeReducer.configure(JobConf job) |
static FileSplit |
StreamUtil.getCurrentSplit(JobConf job) |
RecordReader |
AutoInputFormat.getRecordReader(InputSplit split,
JobConf job,
Reporter reporter) |
RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text> |
StreamInputFormat.getRecordReader(InputSplit genericSplit,
JobConf job,
Reporter reporter) |
static org.apache.hadoop.streaming.StreamUtil.TaskId |
StreamUtil.getTaskInfo(JobConf job) |
static boolean |
StreamUtil.isLocalJobTracker(JobConf job) |
Constructor and Description |
---|
StreamBaseRecordReader(org.apache.hadoop.fs.FSDataInputStream in,
FileSplit split,
Reporter reporter,
JobConf job,
org.apache.hadoop.fs.FileSystem fs) |
StreamXmlRecordReader(org.apache.hadoop.fs.FSDataInputStream in,
FileSplit split,
Reporter reporter,
JobConf job,
org.apache.hadoop.fs.FileSystem fs) |
Constructor and Description |
---|
MRAsyncDiskService(JobConf conf)
Initialize MRAsyncDiskService based on conf.
|
Copyright © 2009 The Apache Software Foundation