org.apache.oozie

Class DagEngine

java.lang.Object

org.apache.oozie.BaseEngine

org.apache.oozie.DagEngine

public class DagEngine
extends BaseEngine

The DagEngine provides all the DAG engine functionality for WS calls.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.oozie.BaseEngine

BaseEngine.LOG_TYPE

Field Summary

Fields inherited from class org.apache.oozie.BaseEngine

USE_XCOMMAND, user

Constructor Summary

Constructors

Constructor and Description
DagEngine() Create a system Dag engine, with no user and no group.
DagEngine(String user) Create a Dag engine to perform operations on behave of a user.

Method Summary

Methods

Modifier and Type	Method and Description
void	change(String jobId, String changeValue) Change a coordinator job.
void	changeSLA(String id, String actions, String dates, String childIds, String newParams) Change SLA properties for job
void	disableSLAAlert(String id, String actions, String dates, String childIds) Disable SLA alert for job
String	dryRunSubmit(org.apache.hadoop.conf.Configuration conf) Dry run a job; like {@link BaseEngine#submitJob(org.apache.hadoop.conf.Configuration, boolean) but doesn't actually execute the job.
void	enableSLAAlert(String id, String actions, String dates, String childIds) Enable SLA alert for job
CoordinatorJob	getCoordJob(String jobId) Return the info about a coord job.
CoordinatorJob	getCoordJob(String jobId, String filter, int start, int length, boolean desc) Return the info about a coord job with actions subset.
String	getDefinition(String jobId) Return the a job definition.
WorkflowJob	getJob(String jobId) Return the info about a job.
WorkflowJob	getJob(String jobId, int start, int length) Return the info about a job with actions subset.
String	getJobIdForExternalId(String externalId) Return the workflow Job ID for an external ID.
WorkflowsInfo	getJobs(String filter, int start, int len) Return the info about a set of jobs.
String	getJobStatus(String jobId) Return the status for a Job ID
WorkflowActionBean	getWorkflowAction(String actionId)
void	kill(String jobId) Kill a job.
WorkflowsInfo	killJobs(String filter, int start, int len) return the jobs that've been killed
protected Map<String,List<String>>	parseFilter(String filter) Validate a jobs filter.
void	processCallback(String actionId, String externalStatus, Properties actionData) Process an action callback.
void	reRun(String jobId, org.apache.hadoop.conf.Configuration conf) Rerun a job.
void	resume(String jobId) Resume a job.
WorkflowsInfo	resumeJobs(String filter, int start, int len) return the jobs that've been resumed
void	start(String jobId) Start a job.
void	streamAuditLog(String jobId, Writer writer, Map<String,String[]> params) Stream the audit log of a job.
void	streamErrorLog(String jobId, Writer writer, Map<String,String[]> params) Stream the error log of a job.
void	streamLog(String jobId, Writer writer, Map<String,String[]> params) Stream the log of a job.
String	submitHttpJob(org.apache.hadoop.conf.Configuration conf, String jobType) Submit a pig/hive/mapreduce job through HTTP.
String	submitJob(org.apache.hadoop.conf.Configuration conf, boolean startJob) Submit a workflow job.
String	submitJobFromCoordinator(org.apache.hadoop.conf.Configuration conf, String parentId) Submit a workflow through a coordinator.
void	suspend(String jobId) Suspend a job.
WorkflowsInfo	suspendJobs(String filter, int start, int len) return the jobs that've been suspended

Methods inherited from class org.apache.oozie.BaseEngine

fetchLog, getJMSTopicName, getUser

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DagEngine

public DagEngine()

Create a system Dag engine, with no user and no group.

DagEngine

public DagEngine(String user)

Create a Dag engine to perform operations on behave of a user.

Parameters:

user - user name.

Method Detail

submitJob

public String submitJob(org.apache.hadoop.conf.Configuration conf,
               boolean startJob)
                 throws DagEngineException

Submit a workflow job.

It validates configuration properties.

Specified by:

submitJob in class BaseEngine

Parameters:

conf - job configuration.

startJob - indicates if the job should be started or not.

Returns:

the job Id.

Throws:

DagEngineException - thrown if the job could not be created.

submitJobFromCoordinator

public String submitJobFromCoordinator(org.apache.hadoop.conf.Configuration conf,
                              String parentId)
                                throws DagEngineException

Submit a workflow through a coordinator. It validates configuration properties.

Parameters:

conf - job conf

parentId - parent of workflow

Returns:

Throws:

DagEngineException

submitHttpJob

public String submitHttpJob(org.apache.hadoop.conf.Configuration conf,
                   String jobType)
                     throws DagEngineException

Submit a pig/hive/mapreduce job through HTTP.

It validates configuration properties.

Parameters:

conf - job configuration.

jobType - job type - can be "pig", "hive", "sqoop" or "mapreduce".

Returns:

the job Id.

Throws:

DagEngineException - thrown if the job could not be created.

start

public void start(String jobId)
           throws DagEngineException

Start a job.

Specified by:

start in class BaseEngine

Parameters:

jobId - job Id.

Throws:

DagEngineException - thrown if the job could not be started.

resume

public void resume(String jobId)
            throws DagEngineException

Resume a job.

Specified by:

resume in class BaseEngine

Parameters:

jobId - job Id.

Throws:

DagEngineException - thrown if the job could not be resumed.

suspend

public void suspend(String jobId)
             throws DagEngineException

Suspend a job.

Specified by:

suspend in class BaseEngine

Parameters:

jobId - job Id.

Throws:

DagEngineException - thrown if the job could not be suspended.

kill

public void kill(String jobId)
          throws DagEngineException

Kill a job.

Specified by:

kill in class BaseEngine

Parameters:

jobId - job Id.

Throws:

DagEngineException - thrown if the job could not be killed.

change

public void change(String jobId,
          String changeValue)
            throws DagEngineException

Description copied from class: BaseEngine

Change a coordinator job.

Specified by:

change in class BaseEngine

Parameters:

jobId - job Id.

changeValue - change value.

Throws:

DagEngineException

reRun

public void reRun(String jobId,
         org.apache.hadoop.conf.Configuration conf)
           throws DagEngineException

Rerun a job.

Specified by:

reRun in class BaseEngine

Parameters:

jobId - job Id to rerun.

conf - configuration information for the rerun.

Throws:

DagEngineException - thrown if the job could not be rerun.

processCallback

public void processCallback(String actionId,
                   String externalStatus,
                   Properties actionData)
                     throws DagEngineException

Process an action callback.

Parameters:

actionId - the action Id.

externalStatus - the action external status.

actionData - the action output data, null if none.

Throws:

DagEngineException - thrown if the callback could not be processed.

getJob

public WorkflowJob getJob(String jobId)
                   throws DagEngineException

Return the info about a job.

Specified by:

getJob in class BaseEngine

Parameters:

jobId - job Id.

Returns:

the workflow job info.

Throws:

DagEngineException - thrown if the job info could not be obtained.

getJob

public WorkflowJob getJob(String jobId,
                 int start,
                 int length)
                   throws DagEngineException

Return the info about a job with actions subset.

Specified by:

getJob in class BaseEngine

Parameters:

jobId - job Id

start - starting from this index in the list of actions belonging to the job

length - number of actions to be returned

Returns:

the workflow job info.

Throws:

DagEngineException - thrown if the job info could not be obtained.

getDefinition

public String getDefinition(String jobId)
                     throws DagEngineException

Return the a job definition.

Specified by:

getDefinition in class BaseEngine

Parameters:

jobId - job Id.

Returns:

the job definition.

Throws:

DagEngineException - thrown if the job definition could no be obtained.

streamLog

public void streamLog(String jobId,
             Writer writer,
             Map<String,String[]> params)
               throws IOException,
                      DagEngineException

Stream the log of a job.

Specified by:

streamLog in class BaseEngine

Parameters:

jobId - job Id.

writer - writer to stream the log to.

params - additional parameters from the request

Throws:

IOException - thrown if the log cannot be streamed.

DagEngineException - thrown if there is error in getting the Workflow Information for jobId.

streamErrorLog

public void streamErrorLog(String jobId,
                  Writer writer,
                  Map<String,String[]> params)
                    throws IOException,
                           DagEngineException

Stream the error log of a job.

Specified by:

streamErrorLog in class BaseEngine

Parameters:

jobId - job Id.

writer - writer to stream the log to.

params - additional parameters from the request

Throws:

IOException - thrown if the log cannot be streamed.

DagEngineException - thrown if there is error in getting the Workflow Information for jobId.

streamAuditLog

public void streamAuditLog(String jobId,
                  Writer writer,
                  Map<String,String[]> params)
                    throws IOException,
                           DagEngineException

Stream the audit log of a job.

Specified by:

streamAuditLog in class BaseEngine

Parameters:

jobId - job Id.

writer - writer to stream the log to.

params - additional parameters from the request

Throws:

IOException - thrown if the log cannot be streamed.

DagEngineException - thrown if there is error in getting the Workflow Information for jobId.

parseFilter

protected Map<String,List<String>> parseFilter(String filter)
                                        throws DagEngineException

Validate a jobs filter.

Parameters:

filter - filter to validate.

Returns:

the parsed filter.

Throws:

DagEngineException - thrown if the filter is invalid.

getJobs

public WorkflowsInfo getJobs(String filter,
                    int start,
                    int len)
                      throws DagEngineException

Return the info about a set of jobs.

Parameters:

filter - job filter. Refer to the OozieClient for the filter syntax.

start - offset, base 1.

len - number of jobs to return.

Returns:

job info for all matching jobs, the jobs don't contain node action information.

Throws:

DagEngineException - thrown if the jobs info could not be obtained.

getJobIdForExternalId

public String getJobIdForExternalId(String externalId)
                             throws DagEngineException

Return the workflow Job ID for an external ID.

This is reverse lookup for recovery purposes.

Specified by:

getJobIdForExternalId in class BaseEngine

Parameters:

externalId - external ID provided at job submission time.

Returns:

the associated workflow job ID if any, null if none.

Throws:

DagEngineException - thrown if the lookup could not be done.

getCoordJob

public CoordinatorJob getCoordJob(String jobId)
                           throws BaseEngineException

Description copied from class: BaseEngine

Return the info about a coord job.

Specified by:

getCoordJob in class BaseEngine

Parameters:

jobId - job Id.

Returns:

the coord job info.

Throws:

BaseEngineException - thrown if the job info could not be obtained.

getCoordJob

public CoordinatorJob getCoordJob(String jobId,
                         String filter,
                         int start,
                         int length,
                         boolean desc)
                           throws BaseEngineException

Description copied from class: BaseEngine

Return the info about a coord job with actions subset.

Specified by:

getCoordJob in class BaseEngine

Parameters:

jobId - job Id.

filter - the status filter

start - starting from this index in the list of actions belonging to the job

length - number of actions to be returned

Returns:

the coord job info.

Throws:

BaseEngineException - thrown if the job info could not be obtained.

getWorkflowAction

public WorkflowActionBean getWorkflowAction(String actionId)
                                     throws BaseEngineException

Throws:

BaseEngineException

dryRunSubmit

public String dryRunSubmit(org.apache.hadoop.conf.Configuration conf)
                    throws BaseEngineException

Description copied from class: BaseEngine

Dry run a job; like {@link BaseEngine#submitJob(org.apache.hadoop.conf.Configuration, boolean) but doesn't actually execute the job.

It validates configuration properties.

Specified by:

dryRunSubmit in class BaseEngine

Parameters:

conf - job configuration.

Returns:

the result of the dryrun

Throws:

BaseEngineException - thrown if there was a problem doing the dryrun

getJobStatus

public String getJobStatus(String jobId)
                    throws DagEngineException

Return the status for a Job ID

Specified by:

getJobStatus in class BaseEngine

Parameters:

jobId - job Id.

Returns:

the job's status

Throws:

DagEngineException - thrown if the job's status could not be obtained

enableSLAAlert

public void enableSLAAlert(String id,
                  String actions,
                  String dates,
                  String childIds)
                    throws BaseEngineException

Description copied from class: BaseEngine

Enable SLA alert for job

Specified by:

enableSLAAlert in class BaseEngine

Throws:

BaseEngineException

disableSLAAlert

public void disableSLAAlert(String id,
                   String actions,
                   String dates,
                   String childIds)
                     throws BaseEngineException

Description copied from class: BaseEngine

Disable SLA alert for job

Specified by:

disableSLAAlert in class BaseEngine

Throws:

BaseEngineException

changeSLA

public void changeSLA(String id,
             String actions,
             String dates,
             String childIds,
             String newParams)
               throws BaseEngineException

Description copied from class: BaseEngine

Change SLA properties for job

Specified by:

changeSLA in class BaseEngine

Throws:

BaseEngineException

killJobs

public WorkflowsInfo killJobs(String filter,
                     int start,
                     int len)
                       throws DagEngineException

return the jobs that've been killed

Parameters:

filter - Jobs that satisfy the filter will be killed

start - start index in the database of jobs

len - maximum number of jobs that will be killed

Returns:

Throws:

DagEngineException

suspendJobs

public WorkflowsInfo suspendJobs(String filter,
                        int start,
                        int len)
                          throws DagEngineException

return the jobs that've been suspended

Parameters:

filter - Filter for jobs that will be suspended, can be name, user, group, status, id or combination of any

start - Offset for the jobs that will be suspended

len - maximum number of jobs that will be suspended

Returns:

Throws:

DagEngineException

resumeJobs

public WorkflowsInfo resumeJobs(String filter,
                       int start,
                       int len)
                         throws DagEngineException

return the jobs that've been resumed

Parameters:

filter - Filter for jobs that will be resumed, can be name, user, group, status, id or combination of any

start - Offset for the jobs that will be resumed

len - maximum number of jobs that will be resumed

Returns:

Throws:

DagEngineException