org.apache.oozie.action.hadoop

Class JavaActionExecutor

java.lang.Object

org.apache.oozie.action.ActionExecutor

org.apache.oozie.action.hadoop.JavaActionExecutor

Direct Known Subclasses:

DistcpActionExecutor, MapReduceActionExecutor, ScriptLanguageActionExecutor, ShellActionExecutor, SparkActionExecutor, SqoopActionExecutor

public class JavaActionExecutor
extends ActionExecutor

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.oozie.action.ActionExecutor

ActionExecutor.Context, ActionExecutor.RETRYPOLICY

Field Summary

Fields

Modifier and Type	Field and Description
static String	ACL_MODIFY_JOB
static String	ACL_VIEW_JOB
static String	CONF_HADOOP_YARN_UBER_MODE
static String	HADOOP_CHILD_JAVA_ENV
static String	HADOOP_CHILD_JAVA_OPTS
static String	HADOOP_JOB_CLASSLOADER
static String	HADOOP_JOB_TRACKER
static String	HADOOP_JOB_TRACKER_2
static String	HADOOP_MAP_JAVA_ENV
static String	HADOOP_MAP_JAVA_OPTS
static String	HADOOP_MAP_MEMORY_MB
static String	HADOOP_NAME_NODE
static String	HADOOP_REDUCE_JAVA_OPTS
protected static String	HADOOP_USER
static String	HADOOP_USER_CLASSPATH_FIRST
static String	HADOOP_YARN_KILL_CHILD_JOBS_ON_AMRESTART
static String	HADOOP_YARN_RM
static String	HADOOP_YARN_TIMELINE_SERVICE_ENABLED
static String	HADOOP_YARN_UBER_MODE
protected XLog	LOG
static String	MAX_EXTERNAL_STATS_SIZE
static String	OOZIE_COMMON_LIBDIR
static String	OOZIE_CREDENTIALS_SKIP
XConfiguration	workflowConf
static String	YARN_AM_COMMAND_OPTS
static String	YARN_AM_ENV
static String	YARN_AM_RESOURCE_MB
static int	YARN_MEMORY_MB_MIN

Fields inherited from class org.apache.oozie.action.ActionExecutor

ACTION_RETRY_INTERVAL, ACTION_RETRY_POLICY, CONF_PREFIX, ERROR_OTHER, MAX_RETRIES, OOZIE_ACTION_YARN_TAG, RETRY_INTERVAL

Constructor Summary

Constructors

Modifier	Constructor and Description
	JavaActionExecutor()
protected	JavaActionExecutor(String type)

Method Summary

Methods

Modifier and Type	Method and Description
protected void	addActionLibs(org.apache.hadoop.fs.Path appPath, org.apache.hadoop.conf.Configuration conf)
protected void	addShareLib(org.apache.hadoop.conf.Configuration conf, String[] actionShareLibNames)
protected void	addSystemShareLibForAction(org.apache.hadoop.conf.Configuration conf)
void	check(ActionExecutor.Context context, WorkflowAction action) Check if an action has completed.
org.apache.hadoop.mapred.JobConf	createBaseHadoopConf(ActionExecutor.Context context, org.jdom.Element actionXml)
protected org.apache.hadoop.mapred.JobConf	createBaseHadoopConf(ActionExecutor.Context context, org.jdom.Element actionXml, boolean loadResources)
protected org.apache.hadoop.mapred.JobClient	createJobClient(ActionExecutor.Context context, org.apache.hadoop.mapred.JobConf jobConf) Create job client object
void	end(ActionExecutor.Context context, WorkflowAction action) End an action after it has executed.
int	extractHeapSizeMB(String input)
protected HashMap<String,CredentialsProperties>	getActionCredentialsProperties(ActionExecutor.Context context, WorkflowAction action)
protected void	getActionData(org.apache.hadoop.fs.FileSystem actionFs, org.apache.hadoop.mapred.RunningJob runningJob, WorkflowAction action, ActionExecutor.Context context) Get the output data of an action.
protected String	getActualExternalId(WorkflowAction action) Useful for overriding in actions that do subsequent job runs such as the MapReduce Action, where the launcher job is not the actual job that then gets monitored.
protected boolean	getCaptureOutput(WorkflowAction action)
static List<Class>	getCommonLauncherClasses()
protected CredentialsProperties	getCredProperties(ActionExecutor.Context context, String credName)
protected String	getDefaultShareLibName(org.jdom.Element actionXml) Returns the default sharelib name for the action if any.
List<Class>	getLauncherClasses()
protected String	getLauncherMain(org.apache.hadoop.conf.Configuration launcherConf, org.jdom.Element actionXml)
static int	getMaxExternalStatsSize() Get the maximum allowed size of stats
protected org.apache.hadoop.mapred.RunningJob	getRunningJob(ActionExecutor.Context context, WorkflowAction action, org.apache.hadoop.mapred.JobClient jobClient)
String[]	getShareLibFilesForActionConf()
protected String[]	getShareLibNames(ActionExecutor.Context context, org.jdom.Element actionXml, org.apache.hadoop.conf.Configuration conf) Return the sharelib names for the action.
void	initActionType() Invoked once at system initialization time.
protected void	injectCallback(ActionExecutor.Context context, org.apache.hadoop.conf.Configuration conf)
boolean	isCompleted(String externalStatus) Return if the external status indicates that the action has completed.
void	kill(ActionExecutor.Context context, WorkflowAction action) Kill an action.
protected org.apache.hadoop.mapred.JobConf	loadHadoopDefaultResources(ActionExecutor.Context context, org.jdom.Element actionXml)
static void	parseJobXmlAndConfiguration(ActionExecutor.Context context, org.jdom.Element element, org.apache.hadoop.fs.Path appPath, org.apache.hadoop.conf.Configuration conf)
static void	parseJobXmlAndConfiguration(ActionExecutor.Context context, org.jdom.Element element, org.apache.hadoop.fs.Path appPath, org.apache.hadoop.conf.Configuration conf, boolean isLauncher)
void	prepareActionDir(org.apache.hadoop.fs.FileSystem actionFs, ActionExecutor.Context context)
boolean	requiresNameNodeJobTracker() Returns true if this action type requires a NameNode and JobTracker.
protected void	setActionCompletionData(ActionExecutor.Context context, org.apache.hadoop.fs.FileSystem actionFs) Sets some data for the action on completion
protected HashMap<String,CredentialsProperties>	setCredentialPropertyToActionConf(ActionExecutor.Context context, WorkflowAction action, org.apache.hadoop.conf.Configuration actionConf)
protected void	setCredentialTokens(org.apache.hadoop.mapred.JobConf jobconf, ActionExecutor.Context context, WorkflowAction action, HashMap<String,CredentialsProperties> credPropertiesMap)
void	setLibFilesArchives(ActionExecutor.Context context, org.jdom.Element actionXml, org.apache.hadoop.fs.Path appPath, org.apache.hadoop.conf.Configuration conf)
void	start(ActionExecutor.Context context, WorkflowAction action) Start an action.
void	submitLauncher(org.apache.hadoop.fs.FileSystem actionFs, ActionExecutor.Context context, WorkflowAction action)
boolean	supportsConfigurationJobXML() Returns true if this action type supports a Configuration and JobXML.

Methods inherited from class org.apache.oozie.action.ActionExecutor

convertException, disableInit, enableInit, getActionDir, getActionDirPath, getActionSignal, getActionYarnTag, getMaxRetries, getOozieConf, getOozieRuntimeDir, getOozieSystemId, getRetryInterval, getRetryPolicy, getType, registerError, resetInitInfo, setMaxRetries, setRetryInterval, setRetryPolicy

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HADOOP_USER

protected static final String HADOOP_USER

See Also:

Constant Field Values

HADOOP_JOB_TRACKER

public static final String HADOOP_JOB_TRACKER

See Also:

Constant Field Values

HADOOP_JOB_TRACKER_2

public static final String HADOOP_JOB_TRACKER_2

See Also:

Constant Field Values

HADOOP_YARN_RM

public static final String HADOOP_YARN_RM

See Also:

Constant Field Values

HADOOP_NAME_NODE

public static final String HADOOP_NAME_NODE

See Also:

Constant Field Values

OOZIE_COMMON_LIBDIR

public static final String OOZIE_COMMON_LIBDIR

See Also:

Constant Field Values

MAX_EXTERNAL_STATS_SIZE

public static final String MAX_EXTERNAL_STATS_SIZE

See Also:

Constant Field Values

ACL_VIEW_JOB

public static final String ACL_VIEW_JOB

See Also:

Constant Field Values

ACL_MODIFY_JOB

public static final String ACL_MODIFY_JOB

See Also:

Constant Field Values

HADOOP_YARN_TIMELINE_SERVICE_ENABLED

public static final String HADOOP_YARN_TIMELINE_SERVICE_ENABLED

See Also:

Constant Field Values

HADOOP_YARN_UBER_MODE

public static final String HADOOP_YARN_UBER_MODE

See Also:

Constant Field Values

HADOOP_YARN_KILL_CHILD_JOBS_ON_AMRESTART

public static final String HADOOP_YARN_KILL_CHILD_JOBS_ON_AMRESTART

See Also:

Constant Field Values

HADOOP_MAP_MEMORY_MB

public static final String HADOOP_MAP_MEMORY_MB

See Also:

Constant Field Values

HADOOP_CHILD_JAVA_OPTS

public static final String HADOOP_CHILD_JAVA_OPTS

See Also:

Constant Field Values

HADOOP_MAP_JAVA_OPTS

public static final String HADOOP_MAP_JAVA_OPTS

See Also:

Constant Field Values

HADOOP_REDUCE_JAVA_OPTS

public static final String HADOOP_REDUCE_JAVA_OPTS

See Also:

Constant Field Values

HADOOP_CHILD_JAVA_ENV

public static final String HADOOP_CHILD_JAVA_ENV

See Also:

Constant Field Values

HADOOP_MAP_JAVA_ENV

public static final String HADOOP_MAP_JAVA_ENV

See Also:

Constant Field Values

YARN_AM_RESOURCE_MB

public static final String YARN_AM_RESOURCE_MB

See Also:

Constant Field Values

YARN_AM_COMMAND_OPTS

public static final String YARN_AM_COMMAND_OPTS

See Also:

Constant Field Values

YARN_AM_ENV

public static final String YARN_AM_ENV

See Also:

Constant Field Values

YARN_MEMORY_MB_MIN

public static final int YARN_MEMORY_MB_MIN

See Also:

Constant Field Values

LOG

protected XLog LOG

CONF_HADOOP_YARN_UBER_MODE

public static final String CONF_HADOOP_YARN_UBER_MODE

See Also:

Constant Field Values

HADOOP_JOB_CLASSLOADER

public static final String HADOOP_JOB_CLASSLOADER

See Also:

Constant Field Values

HADOOP_USER_CLASSPATH_FIRST

public static final String HADOOP_USER_CLASSPATH_FIRST

See Also:

Constant Field Values

OOZIE_CREDENTIALS_SKIP

public static final String OOZIE_CREDENTIALS_SKIP

See Also:

Constant Field Values

workflowConf

public XConfiguration workflowConf

Constructor Detail

JavaActionExecutor

public JavaActionExecutor()

JavaActionExecutor

protected JavaActionExecutor(String type)

Method Detail

getCommonLauncherClasses

public static List<Class> getCommonLauncherClasses()

getLauncherClasses

public List<Class> getLauncherClasses()

initActionType

public void initActionType()

Description copied from class: ActionExecutor

Invoked once at system initialization time.

It can be used to register error information for the expected exceptions. Exceptions should be register from subclasses to superclasses to ensure proper detection, same thing that it is done in a normal catch.

This method should invoke the ActionExecutor.registerError(java.lang.String, org.apache.oozie.action.ActionExecutorException.ErrorType, java.lang.String) method to register all its possible errors.

Subclasses overriding must invoke super.

Overrides:

initActionType in class ActionExecutor

getMaxExternalStatsSize

public static int getMaxExternalStatsSize()

Get the maximum allowed size of stats

Returns:

maximum size of stats

createBaseHadoopConf

public org.apache.hadoop.mapred.JobConf createBaseHadoopConf(ActionExecutor.Context context,
                                                    org.jdom.Element actionXml)

createBaseHadoopConf

protected org.apache.hadoop.mapred.JobConf createBaseHadoopConf(ActionExecutor.Context context,
                                                    org.jdom.Element actionXml,
                                                    boolean loadResources)

loadHadoopDefaultResources

protected org.apache.hadoop.mapred.JobConf loadHadoopDefaultResources(ActionExecutor.Context context,
                                                          org.jdom.Element actionXml)

extractHeapSizeMB

public int extractHeapSizeMB(String input)

parseJobXmlAndConfiguration

public static void parseJobXmlAndConfiguration(ActionExecutor.Context context,
                               org.jdom.Element element,
                               org.apache.hadoop.fs.Path appPath,
                               org.apache.hadoop.conf.Configuration conf)
                                        throws IOException,
                                               ActionExecutorException,
                                               HadoopAccessorException,
                                               URISyntaxException

Throws:

IOException

ActionExecutorException

HadoopAccessorException

URISyntaxException

parseJobXmlAndConfiguration

public static void parseJobXmlAndConfiguration(ActionExecutor.Context context,
                               org.jdom.Element element,
                               org.apache.hadoop.fs.Path appPath,
                               org.apache.hadoop.conf.Configuration conf,
                               boolean isLauncher)
                                        throws IOException,
                                               ActionExecutorException,
                                               HadoopAccessorException,
                                               URISyntaxException

Throws:

IOException

ActionExecutorException

HadoopAccessorException

URISyntaxException

prepareActionDir

public void prepareActionDir(org.apache.hadoop.fs.FileSystem actionFs,
                    ActionExecutor.Context context)
                      throws ActionExecutorException

Throws:

ActionExecutorException

addShareLib

protected void addShareLib(org.apache.hadoop.conf.Configuration conf,
               String[] actionShareLibNames)
                    throws ActionExecutorException

Throws:

ActionExecutorException

addSystemShareLibForAction

protected void addSystemShareLibForAction(org.apache.hadoop.conf.Configuration conf)
                                   throws ActionExecutorException

Throws:

ActionExecutorException

addActionLibs

protected void addActionLibs(org.apache.hadoop.fs.Path appPath,
                 org.apache.hadoop.conf.Configuration conf)
                      throws ActionExecutorException

Throws:

ActionExecutorException

setLibFilesArchives

public void setLibFilesArchives(ActionExecutor.Context context,
                       org.jdom.Element actionXml,
                       org.apache.hadoop.fs.Path appPath,
                       org.apache.hadoop.conf.Configuration conf)
                         throws ActionExecutorException

Throws:

ActionExecutorException

getLauncherMain

protected String getLauncherMain(org.apache.hadoop.conf.Configuration launcherConf,
                     org.jdom.Element actionXml)

injectCallback

protected void injectCallback(ActionExecutor.Context context,
                  org.apache.hadoop.conf.Configuration conf)

submitLauncher

public void submitLauncher(org.apache.hadoop.fs.FileSystem actionFs,
                  ActionExecutor.Context context,
                  WorkflowAction action)
                    throws ActionExecutorException

Throws:

ActionExecutorException

setCredentialPropertyToActionConf

protected HashMap<String,CredentialsProperties> setCredentialPropertyToActionConf(ActionExecutor.Context context,
                                                                      WorkflowAction action,
                                                                      org.apache.hadoop.conf.Configuration actionConf)
                                                                           throws Exception

Throws:

Exception

setCredentialTokens

protected void setCredentialTokens(org.apache.hadoop.mapred.JobConf jobconf,
                       ActionExecutor.Context context,
                       WorkflowAction action,
                       HashMap<String,CredentialsProperties> credPropertiesMap)
                            throws Exception

Throws:

Exception

getActionCredentialsProperties

protected HashMap<String,CredentialsProperties> getActionCredentialsProperties(ActionExecutor.Context context,
                                                                   WorkflowAction action)
                                                                        throws Exception

Throws:

Exception

getCredProperties

protected CredentialsProperties getCredProperties(ActionExecutor.Context context,
                                      String credName)
                                           throws Exception

Throws:

Exception

start

public void start(ActionExecutor.Context context,
         WorkflowAction action)
           throws ActionExecutorException

Description copied from class: ActionExecutor

Start an action.

The ActionExecutor.Context.setStartData(java.lang.String, java.lang.String, java.lang.String) method must be called within this method.

If the action has completed, the ActionExecutor.Context.setExecutionData(java.lang.String, java.util.Properties) method must be called within this method.

Specified by:

start in class ActionExecutor

Parameters:

context - executor context.

action - the action to start.

Throws:

ActionExecutorException - thrown if the action could not start.

end

public void end(ActionExecutor.Context context,
       WorkflowAction action)
         throws ActionExecutorException

Description copied from class: ActionExecutor

End an action after it has executed.

The ActionExecutor.Context.setEndData(org.apache.oozie.client.WorkflowAction.Status, java.lang.String) method must be called within this method.

Specified by:

end in class ActionExecutor

Parameters:

context - executor context.

action - the action to end.

Throws:

ActionExecutorException - thrown if the action could not end.

createJobClient

protected org.apache.hadoop.mapred.JobClient createJobClient(ActionExecutor.Context context,
                                                 org.apache.hadoop.mapred.JobConf jobConf)
                                                      throws HadoopAccessorException

Create job client object

Parameters:

context -

jobConf -

Returns:

JobClient

Throws:

HadoopAccessorException

getRunningJob

protected org.apache.hadoop.mapred.RunningJob getRunningJob(ActionExecutor.Context context,
                                                WorkflowAction action,
                                                org.apache.hadoop.mapred.JobClient jobClient)
                                                     throws Exception

Throws:

Exception

getActualExternalId

protected String getActualExternalId(WorkflowAction action)

Useful for overriding in actions that do subsequent job runs such as the MapReduce Action, where the launcher job is not the actual job that then gets monitored.

check

public void check(ActionExecutor.Context context,
         WorkflowAction action)
           throws ActionExecutorException

Description copied from class: ActionExecutor

Check if an action has completed. This method must be implemented by Async Action Executors.

If the action has completed, the ActionExecutor.Context.setExecutionData(java.lang.String, java.util.Properties) method must be called within this method.

If the action has not completed, the ActionExecutor.Context.setExternalStatus(java.lang.String) method must be called within this method.

Specified by:

check in class ActionExecutor

Parameters:

context - executor context.

action - the action to end.

Throws:

ActionExecutorException - thrown if the action could not be checked.

getActionData

protected void getActionData(org.apache.hadoop.fs.FileSystem actionFs,
                 org.apache.hadoop.mapred.RunningJob runningJob,
                 WorkflowAction action,
                 ActionExecutor.Context context)
                      throws HadoopAccessorException,
                             org.jdom.JDOMException,
                             IOException,
                             URISyntaxException

Get the output data of an action. Subclasses should override this method to get action specific output data.

Parameters:

actionFs - the FileSystem object

runningJob - the runningJob

action - the Workflow action

context - executor context

Throws:

HadoopAccessorException

org.jdom.JDOMException

IOException

URISyntaxException

getCaptureOutput

protected boolean getCaptureOutput(WorkflowAction action)
                            throws org.jdom.JDOMException

Throws:

org.jdom.JDOMException

kill

public void kill(ActionExecutor.Context context,
        WorkflowAction action)
          throws ActionExecutorException

Description copied from class: ActionExecutor

Kill an action.

The ActionExecutor.Context.setEndData(org.apache.oozie.client.WorkflowAction.Status, java.lang.String) method must be called within this method.

Specified by:

kill in class ActionExecutor

Parameters:

context - executor context.

action - the action to kill.

Throws:

ActionExecutorException - thrown if the action could not be killed.

isCompleted

public boolean isCompleted(String externalStatus)

Description copied from class: ActionExecutor

Return if the external status indicates that the action has completed.

Specified by:

isCompleted in class ActionExecutor

Parameters:

externalStatus - external status to check.

Returns:

if the external status indicates that the action has completed.

getShareLibNames

protected String[] getShareLibNames(ActionExecutor.Context context,
                        org.jdom.Element actionXml,
                        org.apache.hadoop.conf.Configuration conf)

Return the sharelib names for the action.

If NULL or empty, it means that the action does not use the action sharelib.

If a non-empty string, i.e. foo, it means the action uses the action sharelib sub-directory foo and all JARs in the sharelib foo directory will be in the action classpath. Multiple sharelib sub-directories can be specified as a comma separated list.

The resolution is done using the following precedence order:

• action.sharelib.for.#ACTIONTYPE# in the action configuration
• action.sharelib.for.#ACTIONTYPE# in the job configuration
• action.sharelib.for.#ACTIONTYPE# in the oozie configuration
• Action Executor getDefaultShareLibName() method

Parameters:

context - executor context.

actionXml -

conf - action configuration.

Returns:

the action sharelib names.

getDefaultShareLibName

protected String getDefaultShareLibName(org.jdom.Element actionXml)

Returns the default sharelib name for the action if any.

Parameters:

actionXml - the action XML fragment.

Returns:

the sharelib name for the action, NULL if none.

getShareLibFilesForActionConf

public String[] getShareLibFilesForActionConf()

setActionCompletionData

protected void setActionCompletionData(ActionExecutor.Context context,
                           org.apache.hadoop.fs.FileSystem actionFs)
                                throws IOException,
                                       HadoopAccessorException,
                                       URISyntaxException

Sets some data for the action on completion

Parameters:

context - executor context

actionFs - the FileSystem object

Throws:

IOException

HadoopAccessorException

URISyntaxException

requiresNameNodeJobTracker

public boolean requiresNameNodeJobTracker()

Description copied from class: ActionExecutor

Returns true if this action type requires a NameNode and JobTracker. These can either be specified directly in the action via <name-node> and <job-tracker>, from the fields in the global section, or from their default values. If false, Oozie won't ensure (i.e. won't throw an Exception if non-existant) that this action type has these values.

Overrides:

requiresNameNodeJobTracker in class ActionExecutor

Returns:

true if a NameNode and JobTracker are required; false if not

supportsConfigurationJobXML

public boolean supportsConfigurationJobXML()

Description copied from class: ActionExecutor

Returns true if this action type supports a Configuration and JobXML. In this case, Oozie will include the <configuration> and <job-xml> elements from the global section (if provided) with the action. If false, Oozie won't add these.

Overrides:

supportsConfigurationJobXML in class ActionExecutor

Returns:

true if the global section's Configuration and JobXML should be given; false if not