::Go back to Oozie Documentation Index::
Oozie provides a command line utility, oozie , to perform job and admin tasks. All operations are done via sub-commands of the oozie CLI.
The oozie CLI interacts with Oozie via its WS API.
usage:
the env variable 'OOZIE_URL' is used as default value for the '-oozie' option
custom headers for Oozie web services can be specified using '-Dheader:NAME=VALUE' oozie help : display usage
.
oozie version : show client version
.
oozie job <OPTIONS> : job operations
-action <arg> coordinator rerun on action ids (requires -rerun); coordinator log retrieval on action ids (requires -log)
-auth <arg> select authentication type [SIMPLE|KERBEROS]
-change <arg> change a coordinator/bundle job
-config <arg> job configuration file '.xml' or '.properties'
-D <property=value> set/override value for given property
-date <arg> coordinator/bundle rerun on action dates (requires -rerun)
-definition <arg> job definition
-doas <arg> doAs user, impersonates as the specified user
-dryrun Supported in Oozie-2.0 or later versions ONLY - dryrun or test
run a coordinator job, job is not queued
-info <arg> info of a job
-kill <arg> kill a job
-len <arg> number of actions (default TOTAL ACTIONS, requires -info)
-localtime use local time (default GMT)
-log <arg> job log
-nocleanup do not clean up output-events of the coordinator rerun actions
(requires -rerun)
-offset <arg> job info offset of actions (default '1', requires -info)
-oozie <arg> Oozie URL
-refresh re-materialize the coordinator rerun actions (requires -rerun)
-rerun <arg> rerun a job (coordinator requires -action or -date; bundle requires -coordinator or -date)
-resume <arg> resume a job
-run run a job
-start <arg> start a job
-submit submit a job
-suspend <arg> suspend a job
-value <arg> new endtime/concurrency/pausetime value for changing a
coordinator job; new pausetime value for changing a bundle job
-verbose verbose mode
.
oozie jobs <OPTIONS> : jobs status
-auth <arg> select authentication type [SIMPLE|KERBEROS]
-doas <arg> doAs user, impersonates as the specified user.
-filter <arg> user=<U>;name=<N>;group=<G>;status=<S>;...
-jobtype <arg> job type ('Supported in Oozie-2.0 or later versions ONLY -
coordinator' or 'wf' (default))
-len <arg> number of jobs (default '100')
-localtime use local time (default GMT)
-offset <arg> jobs offset (default '1')
-oozie <arg> Oozie URL
-verbose verbose mode
.
oozie admin <OPTIONS> : admin operations
-auth <arg> select authentication type [SIMPLE|KERBEROS]
-doas <arg> doAs user, impersonates as the specified user.
-oozie <arg> Oozie URL
-queuedump show Oozie server queue elements
-status show the current system status
-systemmode <arg> Supported in Oozie-2.0 or later versions ONLY. Change oozie
system mode [NORMAL|NOWEBSERVICE|SAFEMODE]
-version show Oozie server build version
.
oozie validate <ARGS> : validate a workflow XML file
.
oozie sla <OPTIONS> : sla operations (Supported in Oozie-2.0 or later)
-auth <arg> select authentication type [SIMPLE|KERBEROS]
-len <arg> number of results (default '100')
-offset <arg> start offset (default '0')
-oozie <arg> Oozie URL
.
oozie pig <OPTIONS> -X <ARGS> : submit a pig job, everything after '-X' are pass-through parameters to pig
-auth <arg> select authentication type [SIMPLE|KERBEROS]
-doas <arg> doAs user, impersonates as the specified user.
-config <arg> job configuration file '.properties'
-D <property=value> set/override value for given property
-file <arg> Pig script
-oozie <arg> Oozie URL
The oozie CLI automatically perform authentication if the Oozie server requests it. By default it supports both pseudo/simple authentication and Kerberos HTTP SPNEGO authentication.
To perform a specific authentication, the auth option with authentication type requests Oozie client to run the specified authentication mechanism only. Oozie client provides two types simple and kerberos which supports pseudo/simple and Kerberos .
For pseudo/simple authentication the oozie CLI uses the user name of the current OS user.
For Kerberos HTTP SPNEGO authentication the oozie CLI uses the default principal for the OS Kerberos cache (normally the principal that did kinit ).
Oozie uses Apache Hadoop-Auth (Java HTTP SPENGO) library for authentication. This library can be extended to support other authentication mechanisms.
Once authentication is performed successfully the received authentication token is cached in the user home directory in the .oozie-auth-token file with owner-only permissions. Subsequent requests reuse the cached token while valid.
The use of the cache file can be disabled by invoking the oozie CLI with the -Doozie.auth.token.cache false= option.
To use an custom authentication mechanism, a Hadoop-Auth Authenticator implementation must be specified with the -Dauthenticator.class =CLASS option.
The -doas option allows the current user to impersonate other users when interacting with the Oozie system. The current user must be configured as a proxyuser in the Oozie system. The proxyuser configuration may restrict from which hosts a user may impersonate users, as well as users of which groups can be impersonated.
All oozie CLI sub-commands expect the -oozie OOZIE_URL option indicating the URL of the Oozie system to run the command against.
If the -oozie option is not specified, the oozie CLI will look for the OOZIE_URL environment variable and uses it if set.
If the option is not provided and the environment variable is not set, the oozie CLI will fail.
* Submitting bundle feature is only supported in Oozie 3.0 or later. Similarly, all bundle operation features below are only supported in Oozie 3.0 or later.
Example:
$ oozie job -oozie http://localhost:8080/oozie -config job.properties -submit . job: 14-20090525161321-oozie-joe
The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML Configuration file (.xml). This file must be specified with the -config option.
The workflow application path must be specified in the file with the oozie.wf.application.path property. The coordinator application path must be specified in the file with the oozie.coord.application.path property.The bundle application path must be specified in the file with the oozie.bundle.application.path property. Specified path must be an HDFS path.
The job will be created, but it will not be started, it will be in PREP status.
Example:
$ oozie job -oozie http://localhost:8080/oozie -start 14-20090525161321-oozie-joe
The start option starts a previously submitted workflow job, coordinator job or bundle job that is in PREP status.
After the command is executed the workflow job will be in RUNNING status, coordinator job will be in RUNNING status and bundle job will be in RUNNING status.
Example:
$ oozie job -oozie http://localhost:8080/oozie -config job.properties -run . job: 15-20090525161321-oozie-joe
The run option creates and starts a workflow job, coordinator job or bundle job.
The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML Configuration file (.xml). This file must be specified with the -config option.
The workflow application path must be specified in the file with the oozie.wf.application.path property. The coordinator application path must be specified in the file with the oozie.coord.application.path property. The bundle application path must be specified in the file with the oozie.bundle.application.path property.The specified path must be an HDFS path.
The job will be created and it will started, the job will be in RUNNING status.
Example:
$ oozie job -oozie http://localhost:8080/oozie -suspend 14-20090525161321-oozie-joe
The suspend option suspends a workflow job in RUNNING status. After the command is executed the workflow job will be in SUSPENDED status.
The suspend option suspends a coordinator/bundle job in RUNNING , RUNNIINGWITHERROR or PREP status. When the coordinator job is suspended, running coordinator actions will stay in running and the workflows will be suspended. If the coordinator job is in RUNNING status, it will transit to SUSPENDED status; if it is in RUNNINGWITHERROR status, it will transit to SUSPENDEDWITHERROR ; if it is in PREP status, it will transit to PREPSUSPENDED status.
When the bundle job is suspended, running coordinators will be suspended. If the bundle job is in RUNNING status, it will transit to SUSPENDED status; if it is in RUNNINGWITHERROR status, it will transit to SUSPENDEDWITHERROR ; if it is in PREP status, it will transit to PREPSUSPENDED status.
Example:
$ oozie job -oozie http://localhost:8080/oozie -resume 14-20090525161321-oozie-joe
The resume option resumes a workflow job in SUSPENDED status.
After the command is executed the workflow job will be in RUNNING status.
The suspend option suspends a coordinator/bundle job in SUSPENDED , SUSPENDEDWITHERROR or PREPSUSPENDED status. If the coordinator job is in SUSPENDED status, it will transit to RUNNING status; if it is in SUSPENDEDWITHERROR status, it will transit to RUNNINGWITHERROR ; if it is in PREPSUSPENDED status, it will transit to PREP status.
When the coordinator job is resumed it will create all the coordinator actions that should have been created during the time it was suspended, actions will not be lost, they will delayed.
When the bundle job is resumed, suspended coordinators will resume running. If the bundle job is in SUSPENDED status, it will transit to RUNNING status; if it is in SUSPENDEDWITHERROR status, it will transit to RUNNINGWITHERROR ; if it is in PREPSUSPENDED status, it will transit to PREP status.
Example:
$ oozie job -oozie http://localhost:8080/oozie -kill 14-20090525161321-oozie-joe
The kill option kills a workflow job in PREP , SUSPENDED or RUNNING status and a coordinator/bundle job in =PREP=, RUNNING , PREPSUSPENDED , SUSPENDED , PREPPAUSED , or PAUSED status.
After the command is executed the job will be in KILLED status.
Example:
$ oozie job -oozie http://localhost:8080/oozie -change 14-20090525161321-oozie-joe -value endtime=2011-12-01T05:00Z\;concurrency=100\;2011-10-01T05:00Z
The change option changes a coordinator job that is not in KILLED status.
Valid value names are:
Repeated value names are not allowed. New end time must not be before job's start time and last action time. New concurrency value has to be a valid integer. All lookahead actions will be revoked according to the new pause time. Also empty string "" can be used to reset pause time to none.
After the command is executed the job's end time, concurrency or pause time should be changed. If an already-succeeded job changes its end time, its status will become running.
Example:
$ oozie job -oozie http://localhost:8080/oozie -change 14-20090525161321-oozie-joe -value pausetime=2011-12-01T05:00Z
The change option changes a bundle job that is not in KILLED status.
Valid value names are:
Repeated value names are not allowed. An empty string "" can be used to reset pause time to none.
After the command is executed the job's pause time should be changed.
Example:
$ oozie job -oozie http://localhost:8080/oozie -config job.properties -rerun 14-20090525161321-oozie-joe
The rerun option reruns a completed ( SUCCCEDED , FAILED or KILLED ) job skipping the specified nodes.
The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML Configuration file (.xml). This file must be specified with the -config option.
The workflow application path must be specified in the file with the oozie.wf.application.path property. The specified path must be an HDFS path.
The list of nodes to skipped must be provided in the oozie.wf.rerun.skip.nodes property separated by commas.
After the command is executed the job will be in RUNNING status.
Refer to the Rerunning Workflow Jobs for details on rerun.
Example:
$oozie job -rerun <coord_Job_id> [-nocleanup] [-refresh] [-action 1, 3-4, 7-40] (-action or -date is required to rerun.) [-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z] (if neither -action nor -date is given, the exception will be thrown.)
The rerun option reruns a terminated (=TIMEDOUT=, SUCCEEDED , KILLED , FAILED ) coordiantor action when coordiator job is not in FAILED or KILLED state.
After the command is executed the rerun coordiator action will be in WAITING status.
Refer to the Rerunning Coordinator Actions for details on rerun.
Example:
$oozie job -rerun <bundle_Job_id> [-nocleanup] [-refresh] [-coordinator c1, c3, c4] (-coordinator or -date is required to rerun.) [-date 2009-01-01T01:00Z::2009-05-31T23:59Z, 2009-11-10T01:00Z, 2009-12-31T22:00Z] (if neither -coordinator nor -date is given, the exception will be thrown.)
The rerun option reruns coordinator actions belonging to specified coordinators within the specified date range.
After the command is executed the rerun coordiator action will be in WAITING status.
Example:
$ oozie job -oozie http://localhost:8080/oozie -info 14-20090525161321-oozie-joe . .---------------------------------------------------------------------------------------------------------------------------------------------------------------- Workflow Name : map-reduce-wf App Path : hdfs://localhost:9000/user/joe/workflows/map-reduce Status : SUCCEEDED Run : 0 User : joe Group : users Created : 2009-05-26 05:01 +0000 Started : 2009-05-26 05:01 +0000 Ended : 2009-05-26 05:01 +0000 Actions .---------------------------------------------------------------------------------------------------------------------------------------------------------------- Action Name Type Status Transition External Id External Status Error Code Start End .---------------------------------------------------------------------------------------------------------------------------------------------------------------- hadoop1 map-reduce OK end job_200904281535_0254 SUCCEEDED - 2009-05-26 05:01 +0000 2009-05-26 05:01 +0000 .----------------------------------------------------------------------------------------------------------------------------------------------------------------
The info option can display information about a workflow job or coordinator job or coordinator action. The info option for a Coordinator job will retrieve the Coordinator actions ordered by nominal time. However, the info command may timeout if the number of Coordinator actions are very high. In that case, info should be used with offset and len option.
The offset and len option specifies the offset and number of actions to display, if checking a workflow job or coordinator job.
The localtime option displays times in local time, if not specified times are displayed in GMT.
The verbose option gives more detailed information for all the actions, if checking for workflow job or coordinator job.
The filter option can be used to filter coordinator actions based on their status. The filter option syntax is: [status=VALUE][;status=VALUE]* . Multiple values must be specified as different name value pairs. When multiple filters are specified, all Coordinator actions that satisfy any one of the filters will be retrieved. (The query will do an OR among all the filter values for the status) Currently, the filter option can be used only with an info option on Coordinator job.
An example below shows how the verbose option can be used to gather action statistics information for a job:
$ oozie job -oozie http://localhost:8080/oozie -info 0000001-111219170928042-oozie-para-W@mr-node -verbose
ID : 0000001-111219170928042-oozie-para-W@mr-node------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------
Console URL : http://localhost:50030/jobdetails.jsp?jobid=job_201112191708_0006
Error Code : -
Error Message : -
External ID : job_201112191708_0006
External Status : SUCCEEDED
Name : mr-node
Retries : 0
Tracker URI : localhost:9001
Type : map-reduce
Started : 2011-12-20 01:12
Status : OK
Ended : 2011-12-20 01:12
External Stats : {"org.apache.hadoop.mapred.JobInProgress$Counter":{"TOTAL_LAUNCHED_REDUCES":1,"TOTAL_LAUNCHED_MAPS":1,"DATA_LOCAL_MAPS":1},"ACTION_TYPE":"MAP_REDUCE","FileSystemCounters":{"FILE_BYTES_READ":1746,"HDFS_BYTES_READ":1409,"FILE_BYTES_WRITTEN":3524,"HDFS_BYTES_WRITTEN":1547},"org.apache.hadoop.mapred.Task$Counter":{"REDUCE_INPUT_GROUPS":33,"COMBINE_OUTPUT_RECORDS":0,"MAP_INPUT_RECORDS":33,"REDUCE_SHUFFLE_BYTES":0,"REDUCE_OUTPUT_RECORDS":33,"SPILLED_RECORDS":66,"MAP_OUTPUT_BYTES":1674,"MAP_INPUT_BYTES":1409,"MAP_OUTPUT_RECORDS":33,"COMBINE_INPUT_RECORDS":0,"REDUCE_INPUT_RECORDS":33}}
External ChildIDs : null
------------------------------------------------------------------------------------------------------------------------------------
The two fields External Stats and External ChildIDs display the action statistics information (that includes counter information in case of MR action and PigStats information in case of a pig action) and child ids of the given job.
Note that the user can turn on/off External Stats by specifying the property oozie.action.external.stats.write as true or false in workflow.xml. By default, it is set to false (not to collect External Stats). External ChildIDs will always be stored.
Example:
$ oozie job -oozie http://localhost:8080/oozie -definition 14-20090525161321-oozie-joe<workflow-app xmlns="uri:oozie:workflow:0.2" name="sm3-segment-2413">
<start to="p0"/>
<action name="p0">
</action>
<end name="end"/>
</workflow-app>
Example:
$ oozie job -oozie http://localhost:8080/oozie -log 14-20090525161321-oozie-joe
Example:
$ oozie job -log <coord_job_id> [-action 1, 3-4, 7-40] (-action is optional.)
* This feature is only supported in Oozie 2.0 or later.
Example:
$ oozie job -oozie http://localhost:8080/oozie job -dryrun -config job.properties
***coordJob after parsing: ***
<coordinator-app xmlns="uri:oozie:coordinator:0.1" name="sla_coord" frequency="20"
start="2009-03-06T010:00Z" end="2009-03-20T11:00Z" timezone="America/Los_Angeles">
<output-events>
<data-out name="Output" dataset="DayLogs">
<dataset name="DayLogs" frequency="1440" initial-instance="2009-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE">
<uri-template>hdfs://localhost:9000/user/angeloh/coord_examples/${YEAR}/${MONTH}/${DAY}</uri-template>
</dataset>
<instance>${coord:current(0)}</instance>
</data-out>
</output-events>
<action>
</action>
</coordinator-app>
***actions for instance***
***total coord actions is 1 ***
------------------------------------------------------------------------------------------------------------------------------------
coordAction instance: 1:
<coordinator-app xmlns="uri:oozie:coordinator:0.1" name="sla_coord" frequency="20"
start="2009-03-06T010:00Z" end="2009-03-20T11:00Z" timezone="America/Los_Angeles">
<output-events>
<data-out name="Output" dataset="DayLogs">
<uris>hdfs://localhost:9000/user/angeloh/coord_examples/2009/03/06</uris>
<dataset name="DayLogs" frequency="1440" initial-instance="2009-01-01T00:00Z" timezone="UTC" freq_timeunit="MINUTE" end_of_duration="NONE">
<uri-template>hdfs://localhost:9000/user/angeloh/coord_examples/${YEAR}/${MONTH}/${DAY}</uri-template>
</dataset>
</data-out>
</output-events>
<action>
</action>
</coordinator-app>
------------------------------------------------------------------------------------------------------------------------------------
The dryrun option tests running a coordinator job with given job properties and does not create the job.
The parameters for the job must be provided in a file, either a Java Properties file (.properties) or a Hadoop XML Configuration file (.xml). This file must be specified with the -config option.
The coordinator application path must be specified in the file with the oozie.coord.application.path property. The specified path must be an HDFS path.
Example:
$ oozie jobs -oozie http://localhost:8080/oozie -localtime -len 2 -fliter status=RUNNING . Job Id Workflow Name Status Run User Group Created Started Ended .---------------------------------------------------------------------------------------------------------------------------------------------------------------- 4-20090527151008-oozie-joe hadoopel-wf RUNNING 0 joe other 2009-05-27 15:34 +0530 2009-05-27 15:34 +0530 - 0-20090527151008-oozie-joe hadoopel-wf RUNNING 0 joe other 2009-05-27 15:15 +0530 2009-05-27 15:15 +0530 - .----------------------------------------------------------------------------------------------------------------------------------------------------------------
The jobs sub-command will display information about multiple jobs.
The offset and len option specified the offset and number of jobs to display, default values are 1 and 100 respectively.
The localtime option displays times in local time, if not specified times are displayed in GMT.
The verbose option gives more detailed information for each job.
A filter can be specified after all options.
The filter option syntax is: [NAME=VALUE][;NAME=VALUE]* .
Valid filter names are:
The query will do an AND among all the filter names. The query will do an OR among all the filter values for the same name. Multiple values must be specified as different name value pairs.
* This feature is only supported in Oozie 2.0 or later.
Example:
$ oozie jobs -oozie http://localhost:8080/oozie -jobtype coordinator . Job ID App Name Status Freq Unit Started Next Materialized .---------------------------------------------------------------------------------------------------------------------------------------------------------------- 0004165-100531045722929-oozie-wrkf-C smaggs-xaggsptechno-coordinator SUCCEEDED 1440 MINUTE 2010-05-27 00:00 2010-05-29 00:00 .---------------------------------------------------------------------------------------------------------------------------------------------------------------- 0003823-100531045722929-oozie-wrkf-C coordcal2880minutescurrent SUCCEEDED 2880 MINUTE 2010-02-01 16:30 2010-02-05 16:30 .----------------------------------------------------------------------------------------------------------------------------------------------------------------
The jobtype option specified the job type to display, default value is 'wf'. To see the coordinator jobs, value is 'coordinator'.
* This feature is only supported in Oozie 3.0 or later.
Example:
$ oozie jobs -oozie http://localhost:8080/oozie -jobtype bundle Job ID Bundle Name Status Kickoff Created User Group------------------------------------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------------------------------------ 0000027-110322105610515-oozie-chao-B BUNDLE-TEST RUNNING 2012-01-15 00:24 2011-03-22 18:07 joe users ------------------------------------------------------------------------------------------------------------------------------------ 0000001-110322105610515-oozie-chao-B BUNDLE-TEST RUNNING 2012-01-15 00:24 2011-03-22 18:06 joe users ------------------------------------------------------------------------------------------------------------------------------------ 0000000-110322105610515-oozie-chao-B BUNDLE-TEST DONEWITHERROR2012-01-15 00:24 2011-03-22 17:58 joe users ------------------------------------------------------------------------------------------------------------------------------------
The jobtype option specified the job type to display, default value is 'wf'. To see the bundle jobs, value is 'bundle'.
Example:
$ oozie admin -oozie http://localhost:8080/oozie -status . Safemode: OFF
It returns the current status of the Oozie system.
* This feature is only supported in Oozie 2.0 or later.
Example:
$ oozie admin -oozie http://localhost:8080/oozie -systemmode [NORMAL|NOWEBSERVICE|SAFEMODE] . Safemode: ON
It returns the current status of the Oozie system.
Example:
$ oozie admin -oozie http://localhost:8080/oozie -version . Oozie server build version: 2.0.2.1-0.20.1.3092118008--
It returns the Oozie server build version.
* This feature is for administrator debugging purpose
Example:
$ oozie admin -oozie http://localhost:8080/oozie -queuedump[Server Queue Dump]: (coord_action_start,1),(coord_action_start,1),(coord_action_start,1) (coord_action_ready,1) (action.check,2)
It returns the Oozie server current queued commands.
Example:
$ oozie validate myApp/workflow.xml
.
Error: Invalid workflow-app, org.xml.sax.SAXParseException: cvc-complex-type.2.4.a:
Invalid content was found starting with element 'xend'. One of '{"uri:oozie:workflow:0.1":decision,
"uri:oozie:workflow:0.1":fork, "uri:oozie:workflow:0.1":join, "uri:oozie:workflow:0.1":kill,
"uri:oozie:workflow:0.1":action, "uri:oozie:workflow:0.1":end}' is expected.
It performs an XML Schema validation on the specified workflow XML file.
* This feature is only supported in Oozie 2.0 or later.
Example:
$ oozie sla -oozie http://localhost:8080/oozie -len 1
.
<sla-message>
<event>
<sequence-id>1091</sequence-id>
<status>
<sla-id>0000004-100521225917912-oozie-wrkf-C@1</sla-id>
<status-timestamp>2010-05-22T00:41Z</status-timestamp>
<job-status>STARTED</job-status>
<job-data />
</status>
</event>
<last-sequence-id>1091</last-sequence-id>
</sla-message>
The offset and len option specified the offset and number of sla events to display, default values are 1 and 100 respectively.
The return message is XML format that can be easily consumed by SLA users.
Example:
$ oozie pig -oozie http://localhost:8080/oozie -file pigScriptFile -config job.properties -X -param_file params . job: 14-20090525161321-oozie-joe-W . $cat job.properties fs.default.name=hdfs://localhost:9000 mapreduce.jobtracker.kerberos.principal=ccc dfs.namenode.kerberos.principal=ddd oozie.libpath=hdfs://localhost:9000/user/oozie/pig/lib/
The parameters for the job must be provided in a Java Properties file (.properties). jobtracker, namenode, libpath must be specified in this file. pigScriptFile is a local file. All jar files (including pig jar file) and all other files needed by the pig job (e.g., parameter file in above example) need to be uploaded onto HDFS under libpath beforehand. The workflow.xml will be created in Oozie server internally. Users can get the workflow.xml from console or command line(-definition).
The job will be created and run right away.