public class DistributedCache
extends java.lang.Object
DistributedCache
is a facility provided by the Map-Reduce
framework to cache files (text, archives, jars etc.) needed by applications.
Applications specify the files, via urls (hdfs:// or http://) to be cached
via the JobConf
.
The DistributedCache
assumes that the
files specified via hdfs:// urls are already present on the
FileSystem
at the path specified by the url.
The framework will copy the necessary files on to the slave node before any tasks for the job are executed on that node. Its efficiency stems from the fact that the files are only copied once per job and the ability to cache archives which are un-archived on the slaves.
DistributedCache
can be used to distribute simple, read-only
data/text files and/or more complex types such as archives, jars etc.
Archives (zip, tar and tgz/tar.gz files) are un-archived at the slave nodes.
Jars may be optionally added to the classpath of the tasks, a rudimentary
software distribution mechanism. Files have execution permissions.
Optionally users can also direct it to symlink the distributed cache file(s)
into the working directory of the task.
DistributedCache
tracks modification timestamps of the cache
files. Clearly the cache files should not be modified by the application
or externally while the job is executing.
Here is an illustrative example on how to use the
DistributedCache
:
It is also very common to use the DistributedCache by using// Setting up the cache for the application 1. Copy the requisite files to theFileSystem
: $ bin/hadoop fs -copyFromLocal lookup.dat /myapp/lookup.dat $ bin/hadoop fs -copyFromLocal map.zip /myapp/map.zip $ bin/hadoop fs -copyFromLocal mylib.jar /myapp/mylib.jar $ bin/hadoop fs -copyFromLocal mytar.tar /myapp/mytar.tar $ bin/hadoop fs -copyFromLocal mytgz.tgz /myapp/mytgz.tgz $ bin/hadoop fs -copyFromLocal mytargz.tar.gz /myapp/mytargz.tar.gz 2. Setup the application'sJobConf
: JobConf job = new JobConf(); DistributedCache.addCacheFile(new URI("/myapp/lookup.dat#lookup.dat"), job); DistributedCache.addCacheArchive(new URI("/myapp/map.zip", job); DistributedCache.addFileToClassPath(new Path("/myapp/mylib.jar"), job); DistributedCache.addCacheArchive(new URI("/myapp/mytar.tar", job); DistributedCache.addCacheArchive(new URI("/myapp/mytgz.tgz", job); DistributedCache.addCacheArchive(new URI("/myapp/mytargz.tar.gz", job); 3. Use the cached files in theMapper
orReducer
: public static class MapClass extends MapReduceBase implements Mapper<K, V, K, V> { private Path[] localArchives; private Path[] localFiles; public void configure(JobConf job) { // Get the cached archives/files localArchives = DistributedCache.getLocalCacheArchives(job); localFiles = DistributedCache.getLocalCacheFiles(job); } public void map(K key, V value, OutputCollector<K, V> output, Reporter reporter) throws IOException { // Use data from the cached archives/files here // ... // ... output.collect(k, v); } }
GenericOptionsParser
.
This class includes methods that should be used by users
(specifically those mentioned in the example above, as well
as addArchiveToClassPath(Path, Configuration)
),
as well as methods intended for use by the MapReduce framework
(e.g., JobClient
). For implementation
details, see TrackerDistributedCacheManager
and
TaskDistributedCacheManager
.Modifier and Type | Field and Description |
---|---|
static java.lang.String |
CACHE_ARCHIVES
Warning:
CACHE_ARCHIVES is not a *public* constant. |
static java.lang.String |
CACHE_ARCHIVES_SIZES
Warning:
CACHE_ARCHIVES_SIZES is not a *public* constant. |
static java.lang.String |
CACHE_ARCHIVES_TIMESTAMPS
Warning:
CACHE_ARCHIVES_TIMESTAMPS is not a *public* constant. |
static java.lang.String |
CACHE_FILES
Warning:
CACHE_FILES is not a *public* constant. |
static java.lang.String |
CACHE_FILES_SIZES
Warning:
CACHE_FILES_SIZES is not a *public* constant. |
static java.lang.String |
CACHE_FILES_TIMESTAMPS
Warning:
CACHE_FILES_TIMESTAMPS is not a *public* constant. |
static java.lang.String |
CACHE_LOCALARCHIVES
Warning:
CACHE_LOCALARCHIVES is not a *public* constant. |
static java.lang.String |
CACHE_LOCALFILES
Warning:
CACHE_LOCALFILES is not a *public* constant. |
static java.lang.String |
CACHE_SYMLINK
Warning:
CACHE_SYMLINK is not a *public* constant. |
Constructor and Description |
---|
DistributedCache() |
Modifier and Type | Method and Description |
---|---|
static void |
addArchiveToClassPath(org.apache.hadoop.fs.Path archive,
org.apache.hadoop.conf.Configuration conf)
Add an archive path to the current set of classpath entries.
|
static void |
addArchiveToClassPath(org.apache.hadoop.fs.Path archive,
org.apache.hadoop.conf.Configuration conf,
org.apache.hadoop.fs.FileSystem fs)
Add an archive path to the current set of classpath entries.
|
static void |
addCacheArchive(java.net.URI uri,
org.apache.hadoop.conf.Configuration conf)
Add a archives to be localized to the conf.
|
static void |
addCacheFile(java.net.URI uri,
org.apache.hadoop.conf.Configuration conf)
Add a file to be localized to the conf.
|
static void |
addFileToClassPath(org.apache.hadoop.fs.Path file,
org.apache.hadoop.conf.Configuration conf)
Add a file path to the current set of classpath entries.
|
static void |
addFileToClassPath(org.apache.hadoop.fs.Path file,
org.apache.hadoop.conf.Configuration conf,
org.apache.hadoop.fs.FileSystem fs)
Add a file path to the current set of classpath entries.
|
static void |
addLocalArchives(org.apache.hadoop.conf.Configuration conf,
java.lang.String str)
Add a archive that has been localized to the conf.
|
static void |
addLocalFiles(org.apache.hadoop.conf.Configuration conf,
java.lang.String str)
Add a file that has been localized to the conf..
|
static boolean |
checkURIs(java.net.URI[] uriFiles,
java.net.URI[] uriArchives)
This method checks if there is a conflict in the fragment names
of the uris.
|
static void |
createAllSymlink(org.apache.hadoop.conf.Configuration conf,
java.io.File jobCacheDir,
java.io.File workDir)
Deprecated.
Internal to MapReduce framework. Use DistributedCacheManager
instead.
|
static void |
createSymlink(org.apache.hadoop.conf.Configuration conf)
This method allows you to create symlinks in the current working directory
of the task to all the cache files/archives.
|
static org.apache.hadoop.fs.Path[] |
getArchiveClassPaths(org.apache.hadoop.conf.Configuration conf)
Get the archive entries in classpath as an array of Path.
|
static long[] |
getArchiveTimestamps(org.apache.hadoop.conf.Configuration conf)
Get the timestamps of the archives.
|
static java.net.URI[] |
getCacheArchives(org.apache.hadoop.conf.Configuration conf)
Get cache archives set in the Configuration.
|
static java.net.URI[] |
getCacheFiles(org.apache.hadoop.conf.Configuration conf)
Get cache files set in the Configuration.
|
static org.apache.hadoop.fs.Path[] |
getFileClassPaths(org.apache.hadoop.conf.Configuration conf)
Get the file entries in classpath as an array of Path.
|
static org.apache.hadoop.fs.FileStatus |
getFileStatus(org.apache.hadoop.conf.Configuration conf,
java.net.URI cache)
Returns
FileStatus of a given cache file on hdfs. |
static long[] |
getFileTimestamps(org.apache.hadoop.conf.Configuration conf)
Get the timestamps of the files.
|
static org.apache.hadoop.fs.Path[] |
getLocalCacheArchives(org.apache.hadoop.conf.Configuration conf)
Return the path array of the localized caches.
|
static org.apache.hadoop.fs.Path[] |
getLocalCacheFiles(org.apache.hadoop.conf.Configuration conf)
Return the path array of the localized files.
|
static boolean |
getSymlink(org.apache.hadoop.conf.Configuration conf)
This method checks to see if symlinks are to be create for the
localized cache files in the current working directory
Used by internal DistributedCache code.
|
static long |
getTimestamp(org.apache.hadoop.conf.Configuration conf,
java.net.URI cache)
Returns mtime of a given cache file on hdfs.
|
static void |
setArchiveTimestamps(org.apache.hadoop.conf.Configuration conf,
java.lang.String timestamps)
This is to check the timestamp of the archives to be localized.
|
static void |
setCacheArchives(java.net.URI[] archives,
org.apache.hadoop.conf.Configuration conf)
Set the configuration with the given set of archives.
|
static void |
setCacheFiles(java.net.URI[] files,
org.apache.hadoop.conf.Configuration conf)
Set the configuration with the given set of files.
|
static void |
setFileTimestamps(org.apache.hadoop.conf.Configuration conf,
java.lang.String timestamps)
This is to check the timestamp of the files to be localized.
|
static void |
setLocalArchives(org.apache.hadoop.conf.Configuration conf,
java.lang.String str)
Set the conf to contain the location for localized archives.
|
static void |
setLocalFiles(org.apache.hadoop.conf.Configuration conf,
java.lang.String str)
Set the conf to contain the location for localized files.
|
public static final java.lang.String CACHE_FILES_SIZES
CACHE_FILES_SIZES
is not a *public* constant.public static final java.lang.String CACHE_ARCHIVES_SIZES
CACHE_ARCHIVES_SIZES
is not a *public* constant.public static final java.lang.String CACHE_ARCHIVES_TIMESTAMPS
CACHE_ARCHIVES_TIMESTAMPS
is not a *public* constant.public static final java.lang.String CACHE_FILES_TIMESTAMPS
CACHE_FILES_TIMESTAMPS
is not a *public* constant.public static final java.lang.String CACHE_ARCHIVES
CACHE_ARCHIVES
is not a *public* constant.public static final java.lang.String CACHE_FILES
CACHE_FILES
is not a *public* constant.public static final java.lang.String CACHE_LOCALARCHIVES
CACHE_LOCALARCHIVES
is not a *public* constant.public static final java.lang.String CACHE_LOCALFILES
CACHE_LOCALFILES
is not a *public* constant.public static final java.lang.String CACHE_SYMLINK
CACHE_SYMLINK
is not a *public* constant.public static org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.conf.Configuration conf, java.net.URI cache) throws java.io.IOException
FileStatus
of a given cache file on hdfs.conf
- configurationcache
- cache fileFileStatus
of a given cache file on hdfsjava.io.IOException
public static long getTimestamp(org.apache.hadoop.conf.Configuration conf, java.net.URI cache) throws java.io.IOException
conf
- configurationcache
- cache filejava.io.IOException
public static void createAllSymlink(org.apache.hadoop.conf.Configuration conf, java.io.File jobCacheDir, java.io.File workDir) throws java.io.IOException
conf
- the configurationjobCacheDir
- the target directory for creating symlinksworkDir
- the directory in which the symlinks are createdjava.io.IOException
public static void setCacheArchives(java.net.URI[] archives, org.apache.hadoop.conf.Configuration conf)
archives
- The list of archives that need to be localizedconf
- Configuration which will be changedpublic static void setCacheFiles(java.net.URI[] files, org.apache.hadoop.conf.Configuration conf)
files
- The list of files that need to be localizedconf
- Configuration which will be changedpublic static java.net.URI[] getCacheArchives(org.apache.hadoop.conf.Configuration conf) throws java.io.IOException
conf
- The configuration which contains the archivesjava.io.IOException
public static java.net.URI[] getCacheFiles(org.apache.hadoop.conf.Configuration conf) throws java.io.IOException
conf
- The configuration which contains the filesjava.io.IOException
public static org.apache.hadoop.fs.Path[] getLocalCacheArchives(org.apache.hadoop.conf.Configuration conf) throws java.io.IOException
conf
- Configuration that contains the localized archivesjava.io.IOException
public static org.apache.hadoop.fs.Path[] getLocalCacheFiles(org.apache.hadoop.conf.Configuration conf) throws java.io.IOException
conf
- Configuration that contains the localized filesjava.io.IOException
public static long[] getArchiveTimestamps(org.apache.hadoop.conf.Configuration conf)
conf
- The configuration which stored the timestampsjava.io.IOException
public static long[] getFileTimestamps(org.apache.hadoop.conf.Configuration conf)
conf
- The configuration which stored the timestampsjava.io.IOException
public static void setArchiveTimestamps(org.apache.hadoop.conf.Configuration conf, java.lang.String timestamps)
conf
- Configuration which stores the timestamp'stimestamps
- comma separated list of timestamps of archives.
The order should be the same as the order in which the archives are added.public static void setFileTimestamps(org.apache.hadoop.conf.Configuration conf, java.lang.String timestamps)
conf
- Configuration which stores the timestamp'stimestamps
- comma separated list of timestamps of files.
The order should be the same as the order in which the files are added.public static void setLocalArchives(org.apache.hadoop.conf.Configuration conf, java.lang.String str)
conf
- The conf to modify to contain the localized cachesstr
- a comma separated list of local archivespublic static void setLocalFiles(org.apache.hadoop.conf.Configuration conf, java.lang.String str)
conf
- The conf to modify to contain the localized cachesstr
- a comma separated list of local filespublic static void addLocalArchives(org.apache.hadoop.conf.Configuration conf, java.lang.String str)
conf
- The conf to modify to contain the localized cachesstr
- a comma separated list of local archivespublic static void addLocalFiles(org.apache.hadoop.conf.Configuration conf, java.lang.String str)
conf
- The conf to modify to contain the localized cachesstr
- a comma separated list of local filespublic static void addCacheArchive(java.net.URI uri, org.apache.hadoop.conf.Configuration conf)
uri
- The uri of the cache to be localizedconf
- Configuration to add the cache topublic static void addCacheFile(java.net.URI uri, org.apache.hadoop.conf.Configuration conf)
uri
- The uri of the cache to be localizedconf
- Configuration to add the cache topublic static void addFileToClassPath(org.apache.hadoop.fs.Path file, org.apache.hadoop.conf.Configuration conf) throws java.io.IOException
file
- Path of the file to be addedconf
- Configuration that contains the classpath settingjava.io.IOException
public static void addFileToClassPath(org.apache.hadoop.fs.Path file, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs) throws java.io.IOException
file
- Path of the file to be addedconf
- Configuration that contains the classpath settingfs
- FileSystem with respect to which archivefile
should
be interpreted.java.io.IOException
public static org.apache.hadoop.fs.Path[] getFileClassPaths(org.apache.hadoop.conf.Configuration conf)
conf
- Configuration that contains the classpath settingpublic static void addArchiveToClassPath(org.apache.hadoop.fs.Path archive, org.apache.hadoop.conf.Configuration conf) throws java.io.IOException
archive
- Path of the archive to be addedconf
- Configuration that contains the classpath settingjava.io.IOException
public static void addArchiveToClassPath(org.apache.hadoop.fs.Path archive, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs) throws java.io.IOException
archive
- Path of the archive to be addedconf
- Configuration that contains the classpath settingfs
- FileSystem with respect to which archive
should be interpreted.java.io.IOException
public static org.apache.hadoop.fs.Path[] getArchiveClassPaths(org.apache.hadoop.conf.Configuration conf)
conf
- Configuration that contains the classpath settingpublic static void createSymlink(org.apache.hadoop.conf.Configuration conf)
conf
- the jobconfpublic static boolean getSymlink(org.apache.hadoop.conf.Configuration conf)
conf
- the jobconfpublic static boolean checkURIs(java.net.URI[] uriFiles, java.net.URI[] uriArchives)
uriFiles
- The uri array of urifilesuriArchives
- the uri array of uri archivesCopyright © 2009 The Apache Software Foundation