org.apache.oozie.command.coord

Class CoordSubmitXCommand

java.lang.Object

org.apache.oozie.command.XCommand<T>

org.apache.oozie.command.TransitionXCommand<String>

org.apache.oozie.command.SubmitTransitionXCommand

org.apache.oozie.command.coord.CoordSubmitXCommand

All Implemented Interfaces:

Callable<String>, XCallable<String>

Direct Known Subclasses:

BundleCoordSubmitXCommand, CoordUpdateXCommand

public class CoordSubmitXCommand
extends SubmitTransitionXCommand

This class provides the functionalities to resolve a coordinator job XML and write the job information into a DB table.

Specifically it performs the following functions: 1. Resolve all the variables or properties using job configurations. 2. Insert all datasets definition as part of the <data-in> and <data-out> tags. 3. Validate the XML at runtime.

Field Summary

Fields

Modifier and Type	Field and Description
protected String	bundleId
protected org.apache.hadoop.conf.Configuration	conf
static String	CONF_CHECK_MAX_FREQUENCY
static String	CONF_DEFAULT_CONCURRENCY
static String	CONF_DEFAULT_MAX_TIMEOUT Default MAX timeout in minutes, after which coordinator input check will timeout
static String	CONF_DEFAULT_THROTTLE
static String	CONF_DEFAULT_TIMEOUT_NORMAL Default timeout for normal jobs, in minutes, after which coordinator input check will timeout
static String	CONF_MAT_THROTTLING_FACTOR
static String	CONF_QUEUE_SIZE
static String	CONFIG_DEFAULT
String	COORD_INPUT_EVENTS
String	COORD_INPUT_EVENTS_DATA_IN
String	COORD_OUTPUT_EVENTS
String	COORD_OUTPUT_EVENTS_DATA_OUT
static String	COORDINATOR_XML_FILE
protected CoordinatorJobBean	coordJob
protected String	coordName
protected boolean	dryrun
protected JPAService	jpaService

Fields inherited from class org.apache.oozie.command.TransitionXCommand

insertList, job, updateList

Fields inherited from class org.apache.oozie.command.XCommand

DEFAULT_LOCK_TIMEOUT, DEFAULT_REQUEUE_DELAY, eventService, instrumentation, INSTRUMENTATION_GROUP, LOG

Constructor Summary

Constructors

Modifier	Constructor and Description
	CoordSubmitXCommand(boolean dryrun, org.apache.hadoop.conf.Configuration conf) Constructor to create the Coordinator Submit Command.
	CoordSubmitXCommand(org.apache.hadoop.conf.Configuration conf) Constructor to create the Coordinator Submit Command.
protected	CoordSubmitXCommand(org.apache.hadoop.conf.Configuration conf, String bundleId, String coordName) Constructor to create the Coordinator Submit Command by bundle job.

Method Summary

Methods

Modifier and Type	Method and Description
org.jdom.Element	basicResolveAndIncludeDS(String appXml, org.apache.hadoop.conf.Configuration conf, CoordinatorJobBean coordJob) The method resolve all the variables that are defined in configuration.
protected String	getDryRun(CoordinatorJobBean coordJob) Gets the dryrun output.
String	getEntityKey() Return the entity key for the command.
Job	getJob() Get the Job for the command.
protected void	includeDataSets(org.jdom.Element resolvedXml, org.apache.hadoop.conf.Configuration conf) Include referred datasets into XML.
protected void	initEvaluators() Initialize all the required EL Evaluators.
protected boolean	isLockRequired() Indicate if the the command requires locking.
protected void	loadState() Load the necessary state to perform the precondition check and to execute the command.
protected void	mergeDefaultConfig() Merge default configuration with user-defined configuration.
void	notifyParent() This will be used to notify the parent about the status of that perticular job.
void	performWrites() This will be used to perform atomically all the writes within this command.
protected void	queueMaterializeTransitionXCommand(String jobId) Queue MaterializeTransitionXCommand
protected String	readAndValidateXml() Read the application XML and validate against coordinator Schema
protected String	readDefinition(String appPath) Read coordinator definition.
protected org.jdom.Element	resolveInitial(org.apache.hadoop.conf.Configuration conf, String appXml, CoordinatorJobBean coordJob) Resolve basic entities using job Configuration.
protected String	storeToDB(String appXML, org.jdom.Element eJob, CoordinatorJobBean coordJob) Write a coordinator job into database
protected String	submit() Submit the job
protected String	submitJob()
void	updateJob() Update the parent job.
protected void	verifyPrecondition() Verify the precondition for the command after a lock has been obtain, just before executing the command.

Methods inherited from class org.apache.oozie.command.SubmitTransitionXCommand

execute, transitToNext

Methods inherited from class org.apache.oozie.command.TransitionXCommand

generateEvents, setJob

Methods inherited from class org.apache.oozie.command.XCommand

call, eagerLoadState, eagerVerifyPrecondition, executeInterrupts, getCreatedTime, getInstrumentation, getKey, getLockTimeOut, getLog, getName, getPriority, getRequeueDelay, getType, inInterruptMode, isReQueueRequired, queue, queue, resetUsed, setInterruptMode, setLogInfo, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

conf

protected org.apache.hadoop.conf.Configuration conf

bundleId

protected final String bundleId

coordName

protected final String coordName

dryrun

protected boolean dryrun

jpaService

protected JPAService jpaService

CONFIG_DEFAULT

public static final String CONFIG_DEFAULT

See Also:

Constant Field Values

COORDINATOR_XML_FILE

public static final String COORDINATOR_XML_FILE

See Also:

Constant Field Values

COORD_INPUT_EVENTS

public final String COORD_INPUT_EVENTS

See Also:

Constant Field Values

COORD_OUTPUT_EVENTS

public final String COORD_OUTPUT_EVENTS

See Also:

Constant Field Values

COORD_INPUT_EVENTS_DATA_IN

public final String COORD_INPUT_EVENTS_DATA_IN

See Also:

Constant Field Values

COORD_OUTPUT_EVENTS_DATA_OUT

public final String COORD_OUTPUT_EVENTS_DATA_OUT

See Also:

Constant Field Values

coordJob

protected CoordinatorJobBean coordJob

CONF_DEFAULT_TIMEOUT_NORMAL

public static final String CONF_DEFAULT_TIMEOUT_NORMAL

Default timeout for normal jobs, in minutes, after which coordinator input check will timeout

See Also:

Constant Field Values

CONF_DEFAULT_CONCURRENCY

public static final String CONF_DEFAULT_CONCURRENCY

See Also:

Constant Field Values

CONF_DEFAULT_THROTTLE

public static final String CONF_DEFAULT_THROTTLE

See Also:

Constant Field Values

CONF_MAT_THROTTLING_FACTOR

public static final String CONF_MAT_THROTTLING_FACTOR

See Also:

Constant Field Values

CONF_DEFAULT_MAX_TIMEOUT

public static final String CONF_DEFAULT_MAX_TIMEOUT

Default MAX timeout in minutes, after which coordinator input check will timeout

See Also:

Constant Field Values

CONF_QUEUE_SIZE

public static final String CONF_QUEUE_SIZE

See Also:

Constant Field Values

CONF_CHECK_MAX_FREQUENCY

public static final String CONF_CHECK_MAX_FREQUENCY

See Also:

Constant Field Values

Constructor Detail

CoordSubmitXCommand

public CoordSubmitXCommand(org.apache.hadoop.conf.Configuration conf)

Constructor to create the Coordinator Submit Command.

Parameters:

conf - : Configuration for Coordinator job

CoordSubmitXCommand

protected CoordSubmitXCommand(org.apache.hadoop.conf.Configuration conf,
                   String bundleId,
                   String coordName)

Constructor to create the Coordinator Submit Command by bundle job.

Parameters:

conf - : Configuration for Coordinator job

bundleId - : bundle id

coordName - : coord name

CoordSubmitXCommand

public CoordSubmitXCommand(boolean dryrun,
                   org.apache.hadoop.conf.Configuration conf)

Constructor to create the Coordinator Submit Command.

Parameters:

dryrun - : if dryrun

conf - : Configuration for Coordinator job

Method Detail

submit

protected String submit()
                 throws CommandException

Description copied from class: SubmitTransitionXCommand

Submit the job

Specified by:

submit in class SubmitTransitionXCommand

Returns:

the id

Throws:

CommandException - thrown if unable to submit

submitJob

protected String submitJob()
                    throws CommandException

Throws:

CommandException

getDryRun

protected String getDryRun(CoordinatorJobBean coordJob)
                    throws Exception

Gets the dryrun output.

Parameters:

coordJob - the coordinatorJobBean

Returns:

the dry run

Throws:

Exception - the exception

queueMaterializeTransitionXCommand

protected void queueMaterializeTransitionXCommand(String jobId)

Queue MaterializeTransitionXCommand

readAndValidateXml

protected String readAndValidateXml()
                             throws CoordinatorJobException

Read the application XML and validate against coordinator Schema

Returns:

validated coordinator XML

Throws:

CoordinatorJobException - thrown if unable to read or validate coordinator xml

mergeDefaultConfig

protected void mergeDefaultConfig()
                           throws CommandException

Merge default configuration with user-defined configuration.

Throws:

CommandException - thrown if failed to read or merge configurations

basicResolveAndIncludeDS

public org.jdom.Element basicResolveAndIncludeDS(String appXml,
                                        org.apache.hadoop.conf.Configuration conf,
                                        CoordinatorJobBean coordJob)
                                          throws CoordinatorJobException,
                                                 Exception

The method resolve all the variables that are defined in configuration. It also include the data set definition from dataset file into XML.

Parameters:

appXml - : Original job XML

conf - : Configuration of the job

coordJob - : Coordinator job bean to be populated.

Returns:

Resolved and modified job XML element.

Throws:

CoordinatorJobException - thrown if failed to resolve basic entities or include referred datasets

Exception - thrown if failed to resolve basic entities or include referred datasets

initEvaluators

protected void initEvaluators()

Initialize all the required EL Evaluators.

resolveInitial

protected org.jdom.Element resolveInitial(org.apache.hadoop.conf.Configuration conf,
                              String appXml,
                              CoordinatorJobBean coordJob)
                                   throws CoordinatorJobException,
                                          Exception

Resolve basic entities using job Configuration.

Parameters:

conf - :Job configuration

appXml - : Original job XML

coordJob - : Coordinator job bean to be populated.

Returns:

Resolved job XML element.

Throws:

CoordinatorJobException - thrown if failed to resolve basic entities

Exception - thrown if failed to resolve basic entities

includeDataSets

protected void includeDataSets(org.jdom.Element resolvedXml,
                   org.apache.hadoop.conf.Configuration conf)
                        throws CoordinatorJobException

Include referred datasets into XML.

Parameters:

resolvedXml - : Job XML element.

conf - : Job configuration

Throws:

CoordinatorJobException - thrown if failed to include referred datasets into XML

readDefinition

protected String readDefinition(String appPath)
                         throws CoordinatorJobException

Read coordinator definition.

Parameters:

appPath - application path.

Returns:

coordinator definition.

Throws:

CoordinatorJobException - thrown if the definition could not be read.

storeToDB

protected String storeToDB(String appXML,
               org.jdom.Element eJob,
               CoordinatorJobBean coordJob)
                    throws CommandException

Write a coordinator job into database

Parameters:

appXML - : Coordinator definition xml

eJob - : XML element of job

coordJob - : Coordinator job bean

Returns:

Job id

Throws:

CommandException - thrown if unable to save coordinator job to db

getEntityKey

public String getEntityKey()

Description copied from class: XCommand

Return the entity key for the command.

Specified by:

getEntityKey in interface XCallable<String>

Specified by:

getEntityKey in class XCommand<String>

Returns:

the entity key for the command.

isLockRequired

protected boolean isLockRequired()

Description copied from class: XCommand

Indicate if the the command requires locking.

Subclasses should override this method if they require locking.

Specified by:

isLockRequired in class XCommand<String>

Returns:

true/false

loadState

protected void loadState()
                  throws CommandException

Description copied from class: XCommand

Load the necessary state to perform the precondition check and to execute the command.

Subclasses must implement this method and load the state needed to do the precondition check and execute the command.

Specified by:

loadState in class XCommand<String>

Throws:

CommandException

verifyPrecondition

protected void verifyPrecondition()
                           throws CommandException

Description copied from class: XCommand

Verify the precondition for the command after a lock has been obtain, just before executing the command.

Specified by:

verifyPrecondition in class XCommand<String>

Throws:

CommandException - thrown if the precondition is not met.

notifyParent

public void notifyParent()
                  throws CommandException

Description copied from class: TransitionXCommand

This will be used to notify the parent about the status of that perticular job.

Specified by:

notifyParent in class TransitionXCommand<String>

Throws:

CommandException

updateJob

public void updateJob()
               throws CommandException

Description copied from class: TransitionXCommand

Update the parent job.

Specified by:

updateJob in class TransitionXCommand<String>

Throws:

CommandException

getJob

public Job getJob()

Description copied from class: TransitionXCommand

Get the Job for the command.

Overrides:

getJob in class TransitionXCommand<String>

Returns:

the job

performWrites

public void performWrites()
                   throws CommandException

Description copied from class: TransitionXCommand

This will be used to perform atomically all the writes within this command.

Specified by:

performWrites in class TransitionXCommand<String>

Throws:

CommandException