org.apache.oozie.action.hadoop
Class JavaActionExecutor

java.lang.Object
  org.apache.oozie.action.ActionExecutor
      org.apache.oozie.action.hadoop.JavaActionExecutor

Direct Known Subclasses:

DistcpActionExecutor, HiveActionExecutor, MapReduceActionExecutor, PigActionExecutor, ShellActionExecutor, SqoopActionExecutor

public class JavaActionExecutor
extends ActionExecutor

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.oozie.action.ActionExecutor

ActionExecutor.Context

Field Summary

static String	ACL_MODIFY_JOB
static String	ACL_VIEW_JOB
static String	HADOOP_JOB_TRACKER
static String	HADOOP_JOB_TRACKER_2
static String	HADOOP_NAME_NODE
static String	HADOOP_YARN_RM
protected XLog	log
static String	MAX_EXTERNAL_STATS_SIZE
static int	MAX_EXTERNAL_STATS_SIZE_DEFAULT
static String	OOZIE_COMMON_LIBDIR

Fields inherited from class org.apache.oozie.action.ActionExecutor

CONF_PREFIX, ERROR_OTHER, MAX_RETRIES, requiresNNJT, RETRY_INTERVAL

Constructor Summary

	JavaActionExecutor()
protected	JavaActionExecutor(String type)

Method Summary

protected void	addActionLibs(org.apache.hadoop.fs.Path appPath, org.apache.hadoop.conf.Configuration conf)
protected void	addShareLib(org.apache.hadoop.fs.Path appPath, org.apache.hadoop.conf.Configuration conf, String actionShareLibName)
void	check(ActionExecutor.Context context, org.apache.oozie.client.WorkflowAction action) Check if an action has completed.
org.apache.hadoop.mapred.JobConf	createBaseHadoopConf(ActionExecutor.Context context, org.jdom.Element actionXml)
protected org.apache.hadoop.mapred.JobClient	createJobClient(ActionExecutor.Context context, org.apache.hadoop.mapred.JobConf jobConf) Create job client object
void	end(ActionExecutor.Context context, org.apache.oozie.client.WorkflowAction action) End an action after it has executed.
protected HashMap<String,CredentialsProperties>	getActionCredentialsProperties(ActionExecutor.Context context, org.apache.oozie.client.WorkflowAction action)
protected void	getActionData(org.apache.hadoop.fs.FileSystem actionFs, org.apache.hadoop.mapred.RunningJob runningJob, org.apache.oozie.client.WorkflowAction action, ActionExecutor.Context context) Get the output data of an action.
protected boolean	getCaptureOutput(org.apache.oozie.client.WorkflowAction action)
protected CredentialsProperties	getCredProperties(ActionExecutor.Context context, String credName)
protected String	getDefaultShareLibName(org.jdom.Element actionXml) Returns the default sharelib name for the action if any.
protected List<Class>	getLauncherClasses()
protected String	getLauncherJarName()
protected String	getLauncherMain(org.apache.hadoop.conf.Configuration launcherConf, org.jdom.Element actionXml)
static int	getMaxExternalStatsSize() Get the maximum allowed size of stats
protected String	getShareLibName(ActionExecutor.Context context, org.jdom.Element actionXml, org.apache.hadoop.conf.Configuration conf) Return the sharelib name for the action.
void	initActionType() Invoked once at system initialization time.
boolean	isCompleted(String externalStatus) Return if the external status indicates that the action has completed.
void	kill(ActionExecutor.Context context, org.apache.oozie.client.WorkflowAction action) Kill an action.
static void	parseJobXmlAndConfiguration(ActionExecutor.Context context, org.jdom.Element element, org.apache.hadoop.fs.Path appPath, org.apache.hadoop.conf.Configuration conf)
void	prepareActionDir(org.apache.hadoop.fs.FileSystem actionFs, ActionExecutor.Context context)
protected void	setActionCompletionData(ActionExecutor.Context context, org.apache.hadoop.fs.FileSystem actionFs) Sets some data for the action on completion
protected HashMap<String,CredentialsProperties>	setCredentialPropertyToActionConf(ActionExecutor.Context context, org.apache.oozie.client.WorkflowAction action, org.apache.hadoop.conf.Configuration actionConf)
protected void	setCredentialTokens(org.apache.hadoop.mapred.JobConf jobconf, ActionExecutor.Context context, org.apache.oozie.client.WorkflowAction action, HashMap<String,CredentialsProperties> credPropertiesMap)
void	start(ActionExecutor.Context context, org.apache.oozie.client.WorkflowAction action) Start an action.
void	submitLauncher(org.apache.hadoop.fs.FileSystem actionFs, ActionExecutor.Context context, org.apache.oozie.client.WorkflowAction action)

Methods inherited from class org.apache.oozie.action.ActionExecutor

convertException, disableInit, enableInit, getActionDir, getActionDirPath, getActionSignal, getMaxRetries, getOozieConf, getOozieRuntimeDir, getOozieSystemId, getRetryInterval, getType, registerError, resetInitInfo, setMaxRetries, setRetryInterval

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HADOOP_JOB_TRACKER

public static final String HADOOP_JOB_TRACKER

See Also:

Constant Field Values

HADOOP_JOB_TRACKER_2

public static final String HADOOP_JOB_TRACKER_2

See Also:

Constant Field Values

HADOOP_YARN_RM

public static final String HADOOP_YARN_RM

See Also:

Constant Field Values

HADOOP_NAME_NODE

public static final String HADOOP_NAME_NODE

See Also:

Constant Field Values

OOZIE_COMMON_LIBDIR

public static final String OOZIE_COMMON_LIBDIR

See Also:

Constant Field Values

MAX_EXTERNAL_STATS_SIZE_DEFAULT

public static final int MAX_EXTERNAL_STATS_SIZE_DEFAULT

See Also:

Constant Field Values

MAX_EXTERNAL_STATS_SIZE

public static final String MAX_EXTERNAL_STATS_SIZE

See Also:

Constant Field Values

ACL_VIEW_JOB

public static final String ACL_VIEW_JOB

See Also:

Constant Field Values

ACL_MODIFY_JOB

public static final String ACL_MODIFY_JOB

See Also:

Constant Field Values

log

protected XLog log

Constructor Detail

JavaActionExecutor

public JavaActionExecutor()

JavaActionExecutor

protected JavaActionExecutor(String type)

Method Detail

getLauncherJarName

protected String getLauncherJarName()

getLauncherClasses

protected List<Class> getLauncherClasses()

initActionType

public void initActionType()

Description copied from class: ActionExecutor

Invoked once at system initialization time.

It can be used to register error information for the expected exceptions. Exceptions should be register from subclasses to superclasses to ensure proper detection, same thing that it is done in a normal catch.

This method should invoke the ActionExecutor.registerError(java.lang.String, org.apache.oozie.action.ActionExecutorException.ErrorType, java.lang.String) method to register all its possible errors.

Subclasses overriding must invoke super.

Overrides:

initActionType in class ActionExecutor

getMaxExternalStatsSize

public static int getMaxExternalStatsSize()

Get the maximum allowed size of stats

Returns:

maximum size of stats

createBaseHadoopConf

public org.apache.hadoop.mapred.JobConf createBaseHadoopConf(ActionExecutor.Context context,
                                                             org.jdom.Element actionXml)

parseJobXmlAndConfiguration

public static void parseJobXmlAndConfiguration(ActionExecutor.Context context,
                                               org.jdom.Element element,
                                               org.apache.hadoop.fs.Path appPath,
                                               org.apache.hadoop.conf.Configuration conf)
                                        throws IOException,
                                               ActionExecutorException,
                                               HadoopAccessorException,
                                               URISyntaxException

Throws:

IOException

ActionExecutorException

HadoopAccessorException

URISyntaxException

prepareActionDir

public void prepareActionDir(org.apache.hadoop.fs.FileSystem actionFs,
                             ActionExecutor.Context context)
                      throws ActionExecutorException

Throws:

ActionExecutorException

addShareLib

protected void addShareLib(org.apache.hadoop.fs.Path appPath,
                           org.apache.hadoop.conf.Configuration conf,
                           String actionShareLibName)
                    throws ActionExecutorException

Throws:

ActionExecutorException

addActionLibs

protected void addActionLibs(org.apache.hadoop.fs.Path appPath,
                             org.apache.hadoop.conf.Configuration conf)
                      throws ActionExecutorException

Throws:

ActionExecutorException

getLauncherMain

protected String getLauncherMain(org.apache.hadoop.conf.Configuration launcherConf,
                                 org.jdom.Element actionXml)

submitLauncher

public void submitLauncher(org.apache.hadoop.fs.FileSystem actionFs,
                           ActionExecutor.Context context,
                           org.apache.oozie.client.WorkflowAction action)
                    throws ActionExecutorException

Throws:

ActionExecutorException

setCredentialPropertyToActionConf

protected HashMap<String,CredentialsProperties> setCredentialPropertyToActionConf(ActionExecutor.Context context,
                                                                                  org.apache.oozie.client.WorkflowAction action,
                                                                                  org.apache.hadoop.conf.Configuration actionConf)
                                                                           throws Exception

Throws:

Exception

setCredentialTokens

protected void setCredentialTokens(org.apache.hadoop.mapred.JobConf jobconf,
                                   ActionExecutor.Context context,
                                   org.apache.oozie.client.WorkflowAction action,
                                   HashMap<String,CredentialsProperties> credPropertiesMap)
                            throws Exception

Throws:

Exception

getActionCredentialsProperties

protected HashMap<String,CredentialsProperties> getActionCredentialsProperties(ActionExecutor.Context context,
                                                                               org.apache.oozie.client.WorkflowAction action)
                                                                        throws Exception

Throws:

Exception

getCredProperties

protected CredentialsProperties getCredProperties(ActionExecutor.Context context,
                                                  String credName)
                                           throws Exception

Throws:

Exception

start

public void start(ActionExecutor.Context context,
                  org.apache.oozie.client.WorkflowAction action)
           throws ActionExecutorException

Description copied from class: ActionExecutor

Start an action.

The ActionExecutor.Context.setStartData(java.lang.String, java.lang.String, java.lang.String) method must be called within this method.

If the action has completed, the ActionExecutor.Context.setExecutionData(java.lang.String, java.util.Properties) method must be called within this method.

Specified by:

start in class ActionExecutor

Parameters:

context - executor context.

action - the action to start.

Throws:

ActionExecutorException - thrown if the action could not start.

end

public void end(ActionExecutor.Context context,
                org.apache.oozie.client.WorkflowAction action)
         throws ActionExecutorException

Description copied from class: ActionExecutor

End an action after it has executed.

The ActionExecutor.Context.setEndData(org.apache.oozie.client.WorkflowAction.Status, java.lang.String) method must be called within this method.

Specified by:

end in class ActionExecutor

Parameters:

context - executor context.

action - the action to end.

Throws:

ActionExecutorException - thrown if the action could not end.

createJobClient

protected org.apache.hadoop.mapred.JobClient createJobClient(ActionExecutor.Context context,
                                                             org.apache.hadoop.mapred.JobConf jobConf)
                                                      throws HadoopAccessorException

Create job client object

Parameters:

context -

jobConf -

Returns:

Throws:

HadoopAccessorException

check

public void check(ActionExecutor.Context context,
                  org.apache.oozie.client.WorkflowAction action)
           throws ActionExecutorException

Description copied from class: ActionExecutor

Check if an action has completed. This method must be implemented by Async Action Executors.

If the action has completed, the ActionExecutor.Context.setExecutionData(java.lang.String, java.util.Properties) method must be called within this method.

If the action has not completed, the ActionExecutor.Context.setExternalStatus(java.lang.String) method must be called within this method.

Specified by:

check in class ActionExecutor

Parameters:

context - executor context.

action - the action to end.

Throws:

ActionExecutorException - thrown if the action could not be checked.

getActionData

protected void getActionData(org.apache.hadoop.fs.FileSystem actionFs,
                             org.apache.hadoop.mapred.RunningJob runningJob,
                             org.apache.oozie.client.WorkflowAction action,
                             ActionExecutor.Context context)
                      throws HadoopAccessorException,
                             org.jdom.JDOMException,
                             IOException,
                             URISyntaxException

Get the output data of an action. Subclasses should override this method to get action specific output data.

Parameters:

actionFs - the FileSystem object

runningJob - the runningJob

action - the Workflow action

context - executor context

Throws:

HadoopAccessorException

org.jdom.JDOMException

IOException

URISyntaxException

getCaptureOutput

protected boolean getCaptureOutput(org.apache.oozie.client.WorkflowAction action)
                            throws org.jdom.JDOMException

Throws:

org.jdom.JDOMException

kill

public void kill(ActionExecutor.Context context,
                 org.apache.oozie.client.WorkflowAction action)
          throws ActionExecutorException

Description copied from class: ActionExecutor

Kill an action.

The ActionExecutor.Context.setEndData(org.apache.oozie.client.WorkflowAction.Status, java.lang.String) method must be called within this method.

Specified by:

kill in class ActionExecutor

Parameters:

context - executor context.

action - the action to kill.

Throws:

ActionExecutorException - thrown if the action could not be killed.

isCompleted

public boolean isCompleted(String externalStatus)

Description copied from class: ActionExecutor

Return if the external status indicates that the action has completed.

Specified by:

isCompleted in class ActionExecutor

Parameters:

externalStatus - external status to check.

Returns:

if the external status indicates that the action has completed.

getShareLibName

protected String getShareLibName(ActionExecutor.Context context,
                                 org.jdom.Element actionXml,
                                 org.apache.hadoop.conf.Configuration conf)

Return the sharelib name for the action.

If NULL or emtpy, it means that the action does not use the action sharelib.

If a non-empty string, i.e. foo, it means the action uses the action sharelib subdirectory foo and all JARs in the sharelib foo directory will be in the action classpath.

The resolution is done using the following precedence order:

• action.sharelib.for.#ACTIONTYPE# in the action configuration
• action.sharelib.for.#ACTIONTYPE# in the job configuration
• action.sharelib.for.#ACTIONTYPE# in the oozie configuration
• Action Executor getDefaultShareLibName() method

Parameters:

context - executor context.

actionXml -

conf - action configuration. @return the action sharelib name.

getDefaultShareLibName

protected String getDefaultShareLibName(org.jdom.Element actionXml)

Returns the default sharelib name for the action if any.

Parameters:

actionXml - the action XML fragment.

Returns:

the sharelib name for the action, NULL if none.

setActionCompletionData

protected void setActionCompletionData(ActionExecutor.Context context,
                                       org.apache.hadoop.fs.FileSystem actionFs)
                                throws IOException,
                                       HadoopAccessorException,
                                       URISyntaxException

Sets some data for the action on completion

Parameters:

context - executor context

actionFs - the FileSystem object

Throws:

IOException

HadoopAccessorException

URISyntaxException