org.apache.oozie

Class DagEngine

java.lang.Object

org.apache.oozie.BaseEngine

org.apache.oozie.DagEngine

public class DagEngine
extends BaseEngine

The DagEngine provides all the DAG engine functionality for WS calls.

Field Summary

Fields inherited from class org.apache.oozie.BaseEngine

USE_XCOMMAND, user

Constructor Summary

Constructors

Constructor and Description
DagEngine() Create a system Dag engine, with no user and no group.
DagEngine(String user) Create a Dag engine to perform operations on behave of a user.

Method Summary

Modifier and Type	Method and Description
void	change(String jobId, String changeValue) Change a coordinator job.
void	changeSLA(String id, String actions, String dates, String childIds, String newParams) Change SLA properties for job
void	disableSLAAlert(String id, String actions, String dates, String childIds) Disable SLA alert for job
String	dryRunSubmit(org.apache.hadoop.conf.Configuration conf) Dry run a job; like BaseEngine.submitJob(org.apache.hadoop.conf.Configuration, boolean) but doesn't actually execute the job.
void	enableSLAAlert(String id, String actions, String dates, String childIds) Enable SLA alert for job
CoordinatorJob	getCoordJob(String jobId) Return the info about a coord job.
CoordinatorJob	getCoordJob(String jobId, String filter, int start, int length, boolean desc) Return the info about a coord job with actions subset.
String	getDefinition(String jobId) Return the a job definition.
WorkflowJob	getJob(String jobId) Return the info about a job.
WorkflowJob	getJob(String jobId, int start, int length) Return the info about a job with actions subset.
String	getJobIdForExternalId(String externalId) Return the workflow Job ID for an external ID.
WorkflowsInfo	getJobs(String filter, int start, int len) Return the info about a set of jobs.
String	getJobStatus(String jobId) Return the status for a Job ID
WorkflowActionBean	getWorkflowAction(String actionId)
List<Map<String,String>>	getWorkflowActionRetries(String actionId) Gets the workflow action retries.
void	kill(String jobId) Kill a job.
WorkflowsInfo	killJobs(String filter, int start, int len) return the jobs that've been killed
protected Map<String,List<String>>	parseFilter(String filter) Validate a jobs filter.
void	processCallback(String actionId, String externalStatus, Properties actionData) Process an action callback.
void	reRun(String jobId, org.apache.hadoop.conf.Configuration conf) Rerun a job.
void	resume(String jobId) Resume a job.
WorkflowsInfo	resumeJobs(String filter, int start, int len) return the jobs that've been resumed
void	start(String jobId) Start a job.
protected void	streamJobLog(XLogStreamer logStreamer, String jobId, Writer writer) Stream job log.
String	submitHttpJob(org.apache.hadoop.conf.Configuration conf, String jobType) Submit a pig/hive/mapreduce job through HTTP.
String	submitJob(org.apache.hadoop.conf.Configuration conf, boolean startJob) Submit a workflow job.
String	submitJobFromCoordinator(org.apache.hadoop.conf.Configuration conf, String parentId) Submit a workflow through a coordinator.
void	suspend(String jobId) Suspend a job.
WorkflowsInfo	suspendJobs(String filter, int start, int len) return the jobs that've been suspended

Methods inherited from class org.apache.oozie.BaseEngine

getJMSTopicName, getUser, streamAuditLog, streamErrorLog, streamLog

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DagEngine

public DagEngine()

Create a system Dag engine, with no user and no group.

DagEngine

public DagEngine(String user)

Create a Dag engine to perform operations on behave of a user.

Parameters:

user - user name.

Method Detail

submitJob

public String submitJob(org.apache.hadoop.conf.Configuration conf,
                        boolean startJob)
                 throws DagEngineException

Submit a workflow job.

It validates configuration properties.

Specified by:

submitJob in class BaseEngine

Parameters:

conf - job configuration.

startJob - indicates if the job should be started or not.

Returns:

the job Id.

Throws:

DagEngineException - thrown if the job could not be created.

submitJobFromCoordinator

public String submitJobFromCoordinator(org.apache.hadoop.conf.Configuration conf,
                                       String parentId)
                                throws DagEngineException

Submit a workflow through a coordinator. It validates configuration properties.

Parameters:

conf - job conf

parentId - parent of workflow

Returns:

jobId returns jobId of the launched workflow

Throws:

DagEngineException - if the job can't be submitted

submitHttpJob

public String submitHttpJob(org.apache.hadoop.conf.Configuration conf,
                            String jobType)
                     throws DagEngineException

Submit a pig/hive/mapreduce job through HTTP.

It validates configuration properties.

Parameters:

conf - job configuration.

jobType - job type - can be "pig", "hive", "sqoop" or "mapreduce".

Returns:

the job Id.

Throws:

DagEngineException - thrown if the job could not be created.

start

public void start(String jobId)
           throws DagEngineException

Start a job.

Specified by:

start in class BaseEngine

Parameters:

jobId - job Id.

Throws:

DagEngineException - thrown if the job could not be started.

resume

public void resume(String jobId)
            throws DagEngineException

Resume a job.

Specified by:

resume in class BaseEngine

Parameters:

jobId - job Id.

Throws:

DagEngineException - thrown if the job could not be resumed.

suspend

public void suspend(String jobId)
             throws DagEngineException

Suspend a job.

Specified by:

suspend in class BaseEngine

Parameters:

jobId - job Id.

Throws:

DagEngineException - thrown if the job could not be suspended.

kill

public void kill(String jobId)
          throws DagEngineException

Kill a job.

Specified by:

kill in class BaseEngine

Parameters:

jobId - job Id.

Throws:

DagEngineException - thrown if the job could not be killed.

change

public void change(String jobId,
                   String changeValue)
            throws DagEngineException

Description copied from class: BaseEngine

Change a coordinator job.

Specified by:

change in class BaseEngine

Parameters:

jobId - job Id.

changeValue - change value.

Throws:

DagEngineException

reRun

public void reRun(String jobId,
                  org.apache.hadoop.conf.Configuration conf)
           throws DagEngineException

Rerun a job.

Specified by:

reRun in class BaseEngine

Parameters:

jobId - job Id to rerun.

conf - configuration information for the rerun.

Throws:

DagEngineException - thrown if the job could not be rerun.

processCallback

public void processCallback(String actionId,
                            String externalStatus,
                            Properties actionData)
                     throws DagEngineException

Process an action callback.

Parameters:

actionId - the action Id.

externalStatus - the action external status.

actionData - the action output data, null if none.

Throws:

DagEngineException - thrown if the callback could not be processed.

getJob

public WorkflowJob getJob(String jobId)
                   throws DagEngineException

Return the info about a job.

Specified by:

getJob in class BaseEngine

Parameters:

jobId - job Id.

Returns:

the workflow job info.

Throws:

DagEngineException - thrown if the job info could not be obtained.

getJob

public WorkflowJob getJob(String jobId,
                          int start,
                          int length)
                   throws DagEngineException

Return the info about a job with actions subset.

Specified by:

getJob in class BaseEngine

Parameters:

jobId - job Id

start - starting from this index in the list of actions belonging to the job

length - number of actions to be returned

Returns:

the workflow job info.

Throws:

DagEngineException - thrown if the job info could not be obtained.

getDefinition

public String getDefinition(String jobId)
                     throws DagEngineException

Return the a job definition.

Specified by:

getDefinition in class BaseEngine

Parameters:

jobId - job Id.

Returns:

the job definition.

Throws:

DagEngineException - thrown if the job definition could no be obtained.

streamJobLog

protected void streamJobLog(XLogStreamer logStreamer,
                            String jobId,
                            Writer writer)
                     throws IOException,
                            DagEngineException

Description copied from class: BaseEngine

Stream job log.

Specified by:

streamJobLog in class BaseEngine

Parameters:

logStreamer - the log streamer

jobId - the job id

writer - the writer

Throws:

IOException - Signals that an I/O exception has occurred.

DagEngineException

parseFilter

protected Map<String,List<String>> parseFilter(String filter)
                                        throws DagEngineException

Validate a jobs filter.

Parameters:

filter - filter to validate.

Returns:

the parsed filter.

Throws:

DagEngineException - thrown if the filter is invalid.

getJobs

public WorkflowsInfo getJobs(String filter,
                             int start,
                             int len)
                      throws DagEngineException

Return the info about a set of jobs.

Parameters:

filter - job filter. Refer to the OozieClient for the filter syntax.

start - offset, base 1.

len - number of jobs to return.

Returns:

job info for all matching jobs, the jobs don't contain node action information.

Throws:

DagEngineException - thrown if the jobs info could not be obtained.

getJobIdForExternalId

public String getJobIdForExternalId(String externalId)
                             throws DagEngineException

Return the workflow Job ID for an external ID.

This is reverse lookup for recovery purposes.

Specified by:

getJobIdForExternalId in class BaseEngine

Parameters:

externalId - external ID provided at job submission time.

Returns:

the associated workflow job ID if any, null if none.

Throws:

DagEngineException - thrown if the lookup could not be done.

getCoordJob

public CoordinatorJob getCoordJob(String jobId)
                           throws BaseEngineException

Description copied from class: BaseEngine

Return the info about a coord job.

Specified by:

getCoordJob in class BaseEngine

Parameters:

jobId - job Id.

Returns:

the coord job info.

Throws:

BaseEngineException - thrown if the job info could not be obtained.

getCoordJob

public CoordinatorJob getCoordJob(String jobId,
                                  String filter,
                                  int start,
                                  int length,
                                  boolean desc)
                           throws BaseEngineException

Description copied from class: BaseEngine

Return the info about a coord job with actions subset.

Specified by:

getCoordJob in class BaseEngine

Parameters:

jobId - job Id.

filter - the status filter

start - starting from this index in the list of actions belonging to the job

length - number of actions to be returned

desc - true if actions are sorted in a descending order of nominal time, false if asc order

Returns:

the coord job info.

Throws:

BaseEngineException - thrown if the job info could not be obtained.

getWorkflowAction

public WorkflowActionBean getWorkflowAction(String actionId)
                                     throws BaseEngineException

Throws:

BaseEngineException

getWorkflowActionRetries

public List<Map<String,String>> getWorkflowActionRetries(String actionId)
                                                  throws BaseEngineException

Gets the workflow action retries.

Parameters:

actionId - the action id

Returns:

the workflow action retries

Throws:

BaseEngineException - the base engine exception

dryRunSubmit

public String dryRunSubmit(org.apache.hadoop.conf.Configuration conf)
                    throws BaseEngineException

Description copied from class: BaseEngine

Dry run a job; like BaseEngine.submitJob(org.apache.hadoop.conf.Configuration, boolean) but doesn't actually execute the job.

It validates configuration properties.

Specified by:

dryRunSubmit in class BaseEngine

Parameters:

conf - job configuration.

Returns:

the result of the dryrun

Throws:

BaseEngineException - thrown if there was a problem doing the dryrun

getJobStatus

public String getJobStatus(String jobId)
                    throws DagEngineException

Return the status for a Job ID

Specified by:

getJobStatus in class BaseEngine

Parameters:

jobId - job Id.

Returns:

the job's status

Throws:

DagEngineException - thrown if the job's status could not be obtained

enableSLAAlert

public void enableSLAAlert(String id,
                           String actions,
                           String dates,
                           String childIds)
                    throws BaseEngineException

Description copied from class: BaseEngine

Enable SLA alert for job

Specified by:

enableSLAAlert in class BaseEngine

Parameters:

id - job ID

actions - list of actions

dates - dates

childIds - child IDs

Throws:

BaseEngineException - thrown if SLA alert could not be enabled

disableSLAAlert

public void disableSLAAlert(String id,
                            String actions,
                            String dates,
                            String childIds)
                     throws BaseEngineException

Description copied from class: BaseEngine

Disable SLA alert for job

Specified by:

disableSLAAlert in class BaseEngine

Parameters:

id - job ID

actions - list of actions

dates - dates

childIds - child IDs

Throws:

BaseEngineException - thrown if SLA alert could not be disabled

changeSLA

public void changeSLA(String id,
                      String actions,
                      String dates,
                      String childIds,
                      String newParams)
               throws BaseEngineException

Description copied from class: BaseEngine

Change SLA properties for job

Specified by:

changeSLA in class BaseEngine

Parameters:

id - job ID

actions - list of actions

dates - dates

childIds - child IDs

newParams - parameters to

Throws:

BaseEngineException - thrown if SLA alert could not be enabled

killJobs

public WorkflowsInfo killJobs(String filter,
                              int start,
                              int len)
                       throws DagEngineException

return the jobs that've been killed

Parameters:

filter - Jobs that satisfy the filter will be killed

start - start index in the database of jobs

len - maximum number of jobs that will be killed

Returns:

workflowsInfo return the jobs that've been killed

Throws:

DagEngineException - if the jobs could not be killed

suspendJobs

public WorkflowsInfo suspendJobs(String filter,
                                 int start,
                                 int len)
                          throws DagEngineException

return the jobs that've been suspended

Parameters:

filter - Filter for jobs that will be suspended, can be name, user, group, status, id or combination of any

start - Offset for the jobs that will be suspended

len - maximum number of jobs that will be suspended

Returns:

workflowsInfo return the jobs that've been suspended

Throws:

DagEngineException - if the jobs cloud not be suspended

resumeJobs

public WorkflowsInfo resumeJobs(String filter,
                                int start,
                                int len)
                         throws DagEngineException

return the jobs that've been resumed

Parameters:

filter - Filter for jobs that will be resumed, can be name, user, group, status, id or combination of any

start - Offset for the jobs that will be resumed

len - maximum number of jobs that will be resumed

Returns:

workflowsInfo returns the jobs that've been resumed

Throws:

DagEngineException - if the jobs cloud not be resumed