public class NonJavaCallout extends AbstractPerJob
pegasus.selector.site
must be set to value
NonJavaCallout
.
This site selector implements a popen()
like call to an
external application that conforms to the API laid out here. The name
of the application to run is specified by the property
pegasus.selector.site.path
. Its value points to a locally
available application.
If the external executable requires certain environment variables to
be set for execution, these can be specified in the property files,
using the prefix pegasus.selector.site.env
. The name of the
environment variable is obtained by stripping the prefix. For
example, to set the variable PATH to a certain value, use the
following entry in your user property file:
pegasus.selector.site.env.PATH = /usr/bin:/bin:/usr/X11R6/binThe site selector populates the environment of the external application with the following default properties, which can be overwritten by user-specified properties:
key | value |
---|---|
PEGASUS_HOME | As set by the system |
CLASSPATH | From java.class.path |
JAVA_HOME | From java.home |
USER | From user.name, if present |
LOGNAME | From user.name, if present |
HOME | From user.home, if present |
TMP | From java.io.tmpdir, if present |
TZ | From user.timezone, if present |
The user can specify the environment variables, by specifying the properties with the prefix pegasus.selector.site.env. prefix. for e.g user can override the default user.name property by setting the property pegasus.selector.site.env.user.home .
The external application is invoked with one commandline argument. This argument is the name of a temporary file. The temporary file is created for each invocation anew by the site selecting caller. Being temporary, the file is deleted after the site selector returns with success. The deletion of the file is governed by the property pegasus.selector.site.keep.tmp. It can have a tristate value with the valid values being
ALWAYS NEVER ONERROR
The external application is expected to write one line to stdout.
The line starts with the string SOLUTION:
, followed
by the chosen site handle. Optionally, separated by a colon, the
name of a jobmanager for the site can be provided by the site
selector. Two examples for successful site selections are:
SOLUTION:mysite:my.job.mgr/jobmanager-batch SOLUTION:siteYNote, these are two examples. The site selector only returns one line with the appropriate solution. If no site is found to be eligble, the poolhandle should be set to NONE by the site selector.
The temporary file is the corner stone of the communication between the site selecting caller and the external site selector. It is a collection of key-value pairs. Each pair is separated by an equals (=) sign, and stands on a line of its own. There are no multi-line values permitted.
The following pairs are generated for the siteselector temporary file:
# | key | value |
---|---|---|
1 | version | The version of the site selector API, currently 2.0 |
1 | transformation | The fully-qualified definition identifier for the TR, ns::id:vs. |
1 | derivation | The fully-qualified definition identifier for the DV, ns::id:vs. |
1 | job.level | The job's depth in the DFS tree of the workflow DAG |
1 | job.id | The job's ID, as used in the DAX file. |
N | resource.id | A pool handle, followed by a whitespace, followed by a gridftp server. Typically, each gridftp server is enumerated once, so you may have multiple occurances of the same site. |
M | input.lfn | An input LFN, optionally followed by a whitespace and filesize. |
1 | wf.name | The label of the DAX, as found in the DAX's root element. |
1 | wf.index | The DAX index, which is incremented for each partition. |
1 | wf.time | The mtime of the workflow. |
1 | wf.manager | The name of the workflow manager to be used, e.g. dagman. |
1 | vo.name | unused at present, name of the virtual organization who runs this WF. |
1 | vo.group | unused at present, usage not clear . |
In order to detect malfunctioning site selectors, a timeout is
attached with each site selector, see property
pegasus.selector.site.timeout
. By default, a site selector
is given up upon after 60 s.
Runtime
,
Process
Modifier and Type | Field and Description |
---|---|
static int |
KEEP_ALWAYS
The state denoting always to keep the temporary files.
|
static int |
KEEP_NEVER
The state denoting never to keep the temporary files.
|
static int |
KEEP_ONERROR
The state denoting to keep the temporary files only in case of error.
|
private edu.isi.pegasus.planner.classes.ADag |
mAbstractDag
The abstract DAG.
|
private static java.lang.String |
mDescription
The description of the site selector.
|
private java.util.Map |
mEnvVar
The map that contains the environment variables including the
default ones that are set while calling out to the site selector
unless they are overridden by the values set in the properties
file.
|
private int |
mKeepTMP
The tristate value for whether keeping the temporary files generated or
not.
|
private java.lang.String |
mSiteSelectorPath
The path to the site selector.
|
private int |
mTimeout
The timeout value in seconds after which to timeout, in the case
where the external site selector does nothing (nothing on stdout
nor stderr).
|
static java.lang.String |
PREFIX_PROPERTIES
The prefix of the property names that specify the environment
variables that need to be set before calling out to the site
selector.
|
static java.lang.String |
PREFIX_TEMPORARY_FILE
The prefix to be used while creating a temporary file to pass to
the external siteselector.
|
static java.lang.String |
SOLUTION_PREFIX
The prefix that the site selector writes out on its stdout to
designate that it is sending a solution.
|
static java.lang.String |
SUFFIX_TEMPORARY_FILE
The suffix to be used while creating a temporary file to pass to
the external siteselector.
|
static java.lang.String |
VERSION
The version number associated with this API of non java callout
site selection.
|
mBag, mLogger, mProps, mSiteStore, mTCMapper
SITE_NOT_FOUND
Constructor and Description |
---|
NonJavaCallout()
The default constructor.
|
Modifier and Type | Method and Description |
---|---|
java.lang.String |
description()
Returns a brief description of the site selection technique
implemented by this class.
|
private java.lang.String[] |
getEnvArrFromMap()
Generates an array of environment variables.
|
private int |
getKeepTMPValue(java.lang.String value)
Returns the int value corresponding to the string value passed.
|
private java.lang.String |
getTempFilename()
Creates a temporary file and obtains its name.
|
void |
initialize(edu.isi.pegasus.planner.classes.PegasusBag bag)
Initializes the site selector.
|
private void |
loadEnvironmentVariables()
Initializes the internal hash that collects environment variables.
|
static void |
main(java.lang.String[] args)
The main program that allows you to test.
|
void |
mapJob(edu.isi.pegasus.planner.classes.Job job,
java.util.List sites)
Calls out to the external site selector.
|
void |
mapWorkflow(edu.isi.pegasus.planner.classes.ADag workflow,
java.util.List sites)
Maps the jobs in the workflow to the various grid sites.
|
private boolean |
parseStdOut(edu.isi.pegasus.planner.classes.Job job,
java.lang.String s)
Extracts the chosen site from the site selector's answer.
|
private java.io.File |
prepareInputFile(edu.isi.pegasus.planner.classes.Job job,
java.util.List pools)
Writes job knowledge into the temporary file passed to the external
site selector.
|
public static final java.lang.String PREFIX_TEMPORARY_FILE
public static final java.lang.String SUFFIX_TEMPORARY_FILE
public static final java.lang.String PREFIX_PROPERTIES
public static final java.lang.String SOLUTION_PREFIX
public static final java.lang.String VERSION
public static final int KEEP_NEVER
public static final int KEEP_ONERROR
public static final int KEEP_ALWAYS
private static final java.lang.String mDescription
private java.util.Map mEnvVar
private int mTimeout
private int mKeepTMP
private java.lang.String mSiteSelectorPath
private edu.isi.pegasus.planner.classes.ADag mAbstractDag
public void initialize(edu.isi.pegasus.planner.classes.PegasusBag bag)
initialize
in interface SiteSelector
initialize
in class Abstract
bag
- the bag of objects that is useful for initialization.public void mapWorkflow(edu.isi.pegasus.planner.classes.ADag workflow, java.util.List sites)
mapWorkflow
in interface SiteSelector
mapWorkflow
in class AbstractPerJob
workflow
- the workflow.sites
- the list of String
objects representing the
execution sites that can be used.public java.lang.String description()
public void mapJob(edu.isi.pegasus.planner.classes.Job job, java.util.List sites)
Job
object into an API-compliant temporary file.
The file's name is provided as single commandline argument to the
site selector executable when it is invoked. The executable,
representing the external site selector, provides its answer
on stdout. The answer is captures, and returned.mapJob
in class AbstractPerJob
job
- is a representation of the DAX compute job whose site of
execution need to be determined.sites
- the list of String
objects representing the
execution sites that can be used.
FIXME: Some site selector return an empty string on failures. Also:
NONE could be a valid site name.org.griphyn.cPlanner.classes.Job
private java.io.File prepareInputFile(edu.isi.pegasus.planner.classes.Job job, java.util.List pools)
Job
record, and the a list of site
candidates. The format of the file is laid out in the class's
introductory documentation.job
- is a representation of the DAX compute job whose site of
execution need to be determined.pools
- is a list of site candidates. The items of the list are
String
objects.null
implies that an error occured while writing
the file.getTempFilename()
private boolean parseStdOut(edu.isi.pegasus.planner.classes.Job job, java.lang.String s)
job
- the job that has to be mapped.s
- is the stdout received from the site selector.private java.lang.String getTempFilename()
private void loadEnvironmentVariables()
private java.lang.String[] getEnvArrFromMap()
null
if
the map is empty.loadEnvironmentVariables()
private int getKeepTMPValue(java.lang.String value)
value
- the string value for keeping the temporary files.KEEP_ALWAYS
,
KEEP_NEVER
,
KEEP_ONERROR
public static void main(java.lang.String[] args)
args
- the arguments