Setting Up Oozie with an Alternate Tomcat

Use the addtowar.sh script to prepare the Oozie server only if Oozie will run with a different servlet container than the embedded Tomcat provided with the distribution.

The addtowar.sh script adds Hadoop JARs, JDBC JARs and the ExtJS library to the Oozie WAR file.

The addtowar.sh script options are:

 Usage  : addtowar <OPTIONS>
 Options: -inputwar INPUT_OOZIE_WAR
          -outputwar OUTPUT_OOZIE_WAR
          [-hadoop HADOOP_VERSION HADOOP_PATH]
          [-extjs EXTJS_PATH]
          [-jars JARS_PATH] (multiple JAR path separated by ':')
          [-secureWeb WEB_XML_PATH] (path to secure web.xml)

The original oozie.war file is in the Oozie server installation directory.

After the Hadoop JARs and the ExtJS library has been added to the oozie.war file Oozie is ready to run.

Delete any previous deployment of the oozie.war from the servlet container (if using Tomcat, delete =oozie.war= and oozie directory from Tomcat's webapps/ directory)

Deploy the prepared oozie.war file (the one that contains the Hadoop JARs and the ExtJS library) in the servlet container (if using Tomcat, copy the prepared oozie.war file to Tomcat's webapps/ directory).

IMPORTANT: Only one Oozie instance can be deployed per Tomcat instance.

Oozie Configuration Properties

All Oozie configuration properties and their default values are defined in the oozie-default.xml file.

Oozie resolves configuration property values in the following order:

• If a Java System property is defined, it uses its value
• Else, if the Oozie configuration file (=oozie-site.xml=) contains the property, it uses its value
• Else, it uses the default value documented in the oozie-default.xml file

NOTE: The oozie-default.xml file found in Oozie's conf/ directory is not used by Oozie, it is there for reference purposes only.

Logging Configuration

By default, Oozie log configuration is defined in the oozie-log4j.properties configuration file.

If the Oozie log configuration file changes, Oozie reloads the new settings automatically.

By default, Oozie logs to Oozie's logs/ directory.

Oozie logs in 4 different files:

• oozie.log: web services log streaming works from this log
• oozie-ops.log: messages for Admin/Operations to monitor
• oozie-instrumentation.log: instrumentation data, every 60 seconds (configurable)
• oozie-audit.log: audit messages, workflow jobs changes

The embedded Tomcat and embedded Derby log files are also written to Oozie's logs/ directory.

Oozie User Authentication Configuration

Oozie supports Kerberos HTTP SPNEGO authentication, pseudo/simple authentication and anonymous access for client connections.

Anonymous access (*default*) does not require the user to authenticate and the user ID is obtained from the job properties on job submission operations, other operations are anonymous.

Pseudo/simple authentication requires the user to specify the user name on the request, this is done by the PseudoAuthenticator class by injecting the user.name parameter in the query string of all requests. The user.name parameter value is taken from the client process Java System property user.name .

Kerberos HTTP SPNEGO authentication requires the user to perform a Kerberos HTTP SPNEGO authentication sequence.

If Pseudo/simple or Kerberos HTTP SPNEGO authentication mechanisms are used, Oozie will return the user an authentication token HTTP Cookie that can be used in later requests as identity proof.

Oozie uses Apache Hadoop-Auth (Java HTTP SPNEGO) library for authentication. This library can be extended to support other authentication mechanisms.

Oozie user authentication is configured using the following configuration properties (default values shown):

  oozie.authentication.type=simple
  oozie.authentication.token.validity=36000
  oozie.authentication.signature.secret=
  oozie.authentication.cookie.domain=
  oozie.authentication.simple.anonymous.allowed=true
  oozie.authentication.kerberos.principal=HTTP/localhost@${local.realm}
  oozie.authentication.kerberos.keytab=${oozie.service.HadoopAccessorService.keytab.file}

The type defines authentication used for Oozie HTTP endpoint, the supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME#.

The token.validity indicates how long (in seconds) an authentication token is valid before it has to be renewed.

The signature.secret is the signature secret for signing the authentication tokens. It is recommended to not set this, in which case Oozie will randomly generate one on startup.

The oozie.authentication.cookie.domain The domain to use for the HTTP cookie that stores the authentication token. In order to authentication to work correctly across all Hadoop nodes web-consoles the domain must be correctly set.

The simple.anonymous.allowed indicates if anonymous requests are allowed. This setting is meaningful only when using 'simple' authentication.

The kerberos.principal indicates the Kerberos principal to be used for HTTP endpoint. The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO specification.

The kerberos.keytab indicates the location of the keytab file with the credentials for the principal. It should be the same keytab file Oozie uses for its Kerberos credentials for Hadoop.

Oozie Hadoop Authentication Configuration

Oozie works with Hadoop versions which support Kerberos authentication.

Oozie Hadoop authentication is configured using the following configuration properties (default values shown):

  oozie.service.HadoopAccessorService.kerberos.enabled=false
  local.realm=LOCALHOST
  oozie.service.HadoopAccessorService.keytab.file=${user.home}/oozie.keytab
  oozie.service.HadoopAccessorService.kerberos.principal=${user.name}/localhost@{local.realm}

The above default values are for a Hadoop 0.20 secure distribution (with support for Kerberos authentication).

To enable Kerberos authentication, the following property must be set:

  oozie.service.HadoopAccessorService.kerberos.enabled=true

When using Kerberos authentication, the following properties must be set to the correct values (default values shown):

  local.realm=LOCALHOST
  oozie.service.HadoopAccessorService.keytab.file=${user.home}/oozie.keytab
  oozie.service.HadoopAccessorService.kerberos.principal=${user.name}/localhost@{local.realm}

IMPORTANT: When using Oozie with a Hadoop 20 with Security distribution, the Oozie user in Hadoop must be configured as a proxy user.

User ProxyUser Configuration

Oozie supports impersonation or proxyuser functionality (identical to Hadoop proxyuser capabilities and conceptually similar to Unix 'sudo').

Proxyuser enables other systems that are Oozie clients to submit jobs on behalf of other users.

Because proxyuser is a powerful capability, Oozie provides the following restriction capabilities (similar to Hadoop):

• Proxyuser is an explicit configuration on per proxyuser user basis.
• A proxyuser user can be restricted to impersonate other users from a set of hosts.
• A proxyuser user can be restricted to impersonate users belonging to a set of groups.

There are 2 configuration properties needed to set up a proxyuser:

• oozie.service.ProxyUserService.proxyuser.#USER#.hosts: hosts from where the user #USER# can impersonate other users.
• oozie.service.ProxyUserService.proxyuser.#USER#.groups: groups the users being impersonated by user #USER# must belong to.

Both properties support the '*' wildcard as value. Although this is recommended only for testing/development.

User Authorization Configuration

Oozie has a basic authorization model:

• Users have read access to all jobs
• Users have write access to their own jobs
• Users have write access to jobs based on an Access Control List (list of users and groups)
• Users have read access to admin operations
• Admin users have write access to all jobs
• Admin users have write access to admin operations

If security is disabled all users are admin users.

Oozie security is set via the following configuration property (default value shown):

  oozie.service.AuthorizationService.security.enabled=false

NOTE: the old ACL model where a group was provided is still supported if the following property is set in oozie-site.xml :

  oozie.service.AuthorizationService.default.group.as.acl=true

Admin users are determined from the list of admin groups, specified in oozie.service.AuthorizationService.admin.groups property. Use commas to separate multiple groups, spaces, tabs and ENTER characters are trimmed.

If the above property for admin groups is not set, then the admin users are the users specified in the conf/adminusers.txt file. The syntax of this file is:

• One user name per line
• Empty lines and lines starting with '#' are ignored

Oozie System ID Configuration

Oozie has a system ID that is is used to generate the Oozie temporary runtime directory, the workflow job IDs, and the workflow action IDs.

Two Oozie systems running with the same ID will not have any conflict but in case of troubleshooting it will be easier to identify resources created/used by the different Oozie systems if they have different system IDs (default value shown):

  oozie.system.id=oozie-${user.name}

Filesystem Configuration

Oozie lets you to configure the allowed Filesystems by using the following configuration property in oozie-site.xml:

  <property>
    <name>oozie.service.HadoopAccessorService.supported.filesystems</name>
    <value>hdfs</value>
  </property>

The above value, hdfs , which is the default, means that Oozie will only allow HDFS filesystems to be used. Examples of other filesystems that Oozie is compatible with are: hdfs, hftp, webhdfs, and viewfs. Multiple filesystems can be specified as comma-separated values. Putting a * will allow any filesystem type, effectively disabling this check.

HCatalog Configuration

Refer to the Oozie HCatalog Integration document for a overview of HCatalog and integration of Oozie with HCatalog. This section explains the various settings to be configured in oozie-site.xml on the Oozie server to enable Oozie to work with HCatalog.

Adding HCatalog jars to Oozie war:

For Oozie server to talk to HCatalog server, HCatalog and hive jars need to be in the server classpath. hive-site.xml which has the configuration to talk to the HCatalog server also needs to be in the classpath or specified by the following configuration property in oozie-site.xml:

  <property>
    <name>oozie.service.HCatAccessorService.hcat.configuration</name>
    <value>/local/filesystem/path/to/hive-site.xml</value>
  </property>

The oozie-[version]-hcataloglibs.tar.gz in the oozie distribution bundles the required hcatalog and hive jars that needs to be placed in the Oozie server classpath. If using a version of HCatalog bundled in Oozie hcataloglibs/, copy the corresponding HCatalog jars from hcataloglibs/ to the libext/ directory. If using a different version of HCatalog, copy the required HCatalog jars from such version in the libext/ directory. This needs to be done before running the oozie-setup.sh script so that these jars get added to the Oozie WAR file.

Configure HCatalog URI Handling:

  <property>
    <name>oozie.service.URIHandlerService.uri.handlers</name>
    <value>org.apache.oozie.dependency.FSURIHandler,org.apache.oozie.dependency.HCatURIHandler</value>
    <description>
        Enlist the different uri handlers supported for data availability checks.
    </description>
  </property>

The above configuration defines the different uri handlers which check for existence of data dependencies defined in a Coordinator. The default value is org.apache.oozie.dependency.FSURIHandler . FSURIHandler supports uris with schemes defined in the configuration oozie.service.HadoopAccessorService.supported.filesystems which are hdfs, hftp and webhcat by default. HCatURIHandler supports uris with the scheme as hcat.

Configure HCatalog services:

  <property>
    <name>oozie.services.ext</name>
    <value>
        org.apache.oozie.service.JMSAccessorService,
        org.apache.oozie.service.PartitionDependencyManagerService,
        org.apache.oozie.service.HCatAccessorService
      </value>
    <description>
          To add/replace services defined in 'oozie.services' with custom implementations.
          Class names must be separated by commas.
    </description>
  </property>

PartitionDependencyManagerService and HCatAccessorService are required to work with HCatalog and support Coordinators having HCatalog uris as data dependency. If the HCatalog server is configured to publish partition availability notifications to a JMS compliant messaging provider like ActiveMQ, then JMSAccessorService needs to be added to oozie.services.ext to handle those notifications.

Configure JMS Provider JNDI connection mapping for HCatalog:

  <property>
    <name>oozie.service.HCatAccessorService.jmsconnections</name>
    <value>
      hcat://hcatserver.colo1.com:8020=java.naming.factory.initial#Dummy.Factory;java.naming.provider.url#tcp://broker.colo1.com:61616,
      default=java.naming.factory.initial#org.apache.activemq.jndi.ActiveMQInitialContextFactory;java.naming.provider.url#tcp://broker.colo.com:61616;connectionFactoryNames#ConnectionFactory
    </value>
    <description>
        Specify the map  of endpoints to JMS configuration properties. In general, endpoint
        identifies the HCatalog server URL. "default" is used if no endpoint is mentioned
        in the query. If some JMS property is not defined, the system will use the property
        defined jndi.properties. jndi.properties files is retrieved from the application classpath.
        Mapping rules can also be provided for mapping Hcatalog servers to corresponding JMS providers.
        hcat://${1}.${2}.com:8020=java.naming.factory.initial#Dummy.Factory;java.naming.provider.url#tcp://broker.${2}.com:61616
    </description>
  </property>

Currently HCatalog does not provide APIs to get the connection details to connect to the JMS Provider it publishes notifications to. It only has APIs which provide the topic name in the JMS Provider to which the notifications are published for a given database table. So the JMS Provider's connection properties needs to be manually configured in Oozie using the above setting. You can either provide a default JNDI configuration which will be used as the JMS Provider for all HCatalog servers, or can specify a configuration per HCatalog server URL or provide a configuration based on a rule matching multiple HCatalog server URLs. For example: With the configuration of hcat://${1}.${2}.com:8020=java.naming.factory.initial#Dummy.Factory;java.naming.provider.url#tcp://broker.${2}.com:61616, request URL of hcat://server1.colo1.com:8020 will map to tcp://broker.colo1.com:61616, hcat://server2.colo2.com:8020 will map to tcp://broker.colo2.com:61616 and so on.

Configure HCatalog Polling Frequency:

  <property>
    <name>oozie.service.coord.push.check.requeue.interval
        </name>
    <value>600000</value>
    <description>Command re-queue interval for push dependencies (in millisecond).
    </description>
  </property>

If there is no JMS Provider configured for a HCatalog Server, then oozie polls HCatalog based on the frequency defined in oozie.service.coord.input.check.requeue.interval . This config also applies to HDFS polling. If there is a JMS provider configured for a HCatalog Server, then oozie polls HCatalog based on the frequency defined in oozie.service.coord.push.check.requeue.interval as a fallback. The defaults for oozie.service.coord.input.check.requeue.interval and oozie.service.coord.push.check.requeue.interval are 1 minute and 10 minutes respectively.

Notifications Configuration

Oozie supports publishing notifications to a JMS Provider for job status changes and SLA met and miss events. For more information on the feature, refer JMS Notifications documentation. Oozie can also send email notifications on SLA misses.

• Message Broker Installation :
  

• Services :
  

     <property>
        <name>oozie.services.ext</name>
        <value>
            org.apache.oozie.service.JMSAccessorService,
            org.apache.oozie.service.JMSTopicService,
            org.apache.oozie.service.EventHandlerService,
            org.apache.oozie.sla.service.SLAService
        </value>
     </property>
     

• Event Handlers :
  

     <property>
        <name>oozie.service.EventHandlerService.event.listeners</name>
        <value>
            org.apache.oozie.jms.JMSJobEventListener,
            org.apache.oozie.sla.listener.SLAJobEventListener,
            org.apache.oozie.jms.JMSSLAEventListener,
            org.apache.oozie.sla.listener.SLAEmailEventListener
        </value>
     </property>
     

• JMS properties :
  

As an example, if using ActiveMQ in local env, the property can be set to

     <property>
        <name>oozie.jms.producer.connection.properties</name>
        <value>
            java.naming.factory.initial#org.apache.activemq.jndi.ActiveMQInitialContextFactory;java.naming.provider.url#tcp://localhost:61616;connectionFactoryNames#ConnectionFactory
        </value>
     </property>
     

• JMS Topic name :
  

     <property>
        <name>oozie.service.JMSTopicService.topic.name</name>
        <value>
            default=${username}
        </value>
        <description>
            Topic options are ${username}, ${jobId}, or a fixed string which can be specified as default or for a
            particular job type.
            For e.g To have a fixed string topic for workflows, coordinators and bundles,
            specify in the following comma-separated format: {jobtype1}={some_string1}, {jobtype2}={some_string2}
            where job type can be WORKFLOW, COORDINATOR or BUNDLE.
            Following example defines topic for workflow job, workflow action, coordinator job, coordinator action,
            bundle job and bundle action
            WORKFLOW=workflow,
            COORDINATOR=coordinator,
            BUNDLE=bundle
            For jobs with no defined topic, default topic will be ${username}
        </description>
     </property>
     

Another related property is the topic prefix.

     <property>
        <name>oozie.service.JMSTopicService.topic.prefix</name>
        <value></value>
        <description>
            This can be used to append a prefix to the topic in oozie.service.JMSTopicService.topic.name. For eg: oozie.
        </description>
     </property>
     

Setting Up Oozie with HTTPS (SSL)

IMPORTANT : The default HTTPS configuration will cause all Oozie URLs to use HTTPS except for the JobTracker callback URLs. This is to simplify configuration (no changes needed outside of Oozie), but this is okay because Oozie doesn't inherently trust the callbacks anyway; they are used as hints.

The related environment variables are explained at Environment Setup .

You can use either a certificate from a Certificate Authority or a Self-Signed Certificate. Using a self-signed certificate requires some additional configuration on each Oozie client machine. If possible, a certificate from a Certificate Authority is recommended because it's simpler to configure.

There's also some additional considerations when using Oozie HA with HTTPS.

keytool -genkeypair -alias tomcat -keyalg RSA -dname "CN=hostname" -storepass password -keypass password

keytool -genkeypair -alias tomcat -keyalg RSA -dname "CN=oozie.int.example.com" -storepass password -keypass password

keytool -genkeypair -alias tomcat -keyalg RSA -dname "CN=*.int.example.com" -storepass password -keypass password

keytool -exportcert -alias tomcat -file path/to/anywhere/certificate.cert -storepass password

keytool -import -alias tomcat -file path/to/certificate.cert -keystore /path/to/anywhere/oozie.truststore -storepass password2

keytool -import -alias tomcat -file path/to/certificate.cert

oozie-setup.sh prepare-war -secure

export OOZIE_CLIENT_OPTS='-Djavax.net.ssl.trustStore=/path/to/oozie.truststore'

Fine Tuning an Oozie Server

Refer to the oozie-default.xml for details.

Using Metrics instead of Instrumentation

As of version 4.1.0, Oozie includes a replacement for the Instrumentation based on Codahale's Metrics library. It includes a number of improvements over the original Instrumentation included in Oozie. They both report most of the same information, though the formatting is slightly different and there's some additional information in the Metrics version; the format of the output to the oozie-instrumentation log is also different. The Metrics version can be enabled by adding the MetricsInstrumentationService to the list of services:

     <property>
        <name>oozie.services.ext</name>
        <value>
            org.apache.oozie.service.MetricsInstrumentationService
        </value>
     </property>
     

Once enabled, the admin/instrumentation REST endpoint will no longer be available and instead the admin/metrics endpoint should be used (see the Web Services API documentation for more details); the Oozie Web UI will also replace the "Instrumentation" tab with a "Metrics" tab.

We can also publish the instrumentation metrics to the external server graphite or ganglia. For this the following properties should be specified in oozie-site.xml :

    <property>
        <name>oozie.external_monitoring.enable</name>
        <value>false</value>
        <description>
            If the oozie functional metrics needs to be exposed to the metrics-server backend, set it to true
            If set to true, the following properties has to be specified : oozie.metrics.server.name,
            oozie.metrics.host, oozie.metrics.prefix, oozie.metrics.report.interval.sec, oozie.metrics.port
        </description>
    </property>    <property>
        <name>oozie.external_monitoring.type</name>
        <value>graphite</value>
        <description>
            The name of the server to which we want to send the metrics, would be graphite or ganglia.
        </description>
    </property>
    <property>
        <name>oozie.external_monitoring.address</name>
        <value>http://localhost:2020</value>
    </property>
    <property>
        <name>oozie.external_monitoring.metricPrefix</name>
        <value>oozie</value>
    </property>
    <property>
        <name>oozie.external_monitoring.reporterIntervalSecs</name>
        <value>60</value>
    </property>
    

We can also publish the instrumentation metrics via JMX interface. For this the following property should be specified in oozie-site.xml :

    <property>
         <name>oozie.jmx_monitoring.enable</name>
         <value>false</value>
         <description>
             If the oozie functional metrics needs to be exposed via JMX interface, set it to true.
         </description>
     </property>>
    

High Availability (HA)

Multiple Oozie Servers can be configured against the same database to provide High Availability (HA) of the Oozie service.

<property>
    <name>oozie.services.ext</name>
    <value>
        org.apache.oozie.service.ZKLocksService,
        org.apache.oozie.service.ZKXLogStreamingService,
        org.apache.oozie.service.ZKJobsConcurrencyService,
        org.apache.oozie.service.ZKUUIDService
    </value>
</property>

<property>
   <name>oozie.zookeeper.connection.string</name>
   <value>localhost:2181</value>
</property>

<property>
    <name>oozie.zookeeper.namespace</name>
    <value>oozie</value>
</property>

<property>
    <name>oozie.base.url</name>
    <value>http://my.loadbalancer.hostname:11000/oozie</value>
</property>

<property>
    <name>oozie.instance.id</name>
    <value>hostname</value>
</property>

<property>
    <name>oozie.server.authentication.type</name>
    <value>kerberos</value>
</property>

<property>
    <name>oozie.zookeeper.secure</name>
    <value>true</value>
</property>

authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
kerberos.removeHostFromPrincipal=true
kerberos.removeRealmFromPrincipal=true

<property>
    <name>oozie.service.ZKUUIDService.jobid.sequence.max</name>
    <value>99999999990</value>
</property>