Oozie Installation and Configuration

Overriding Configuration Values

Overriding happens for following configuration entries with oozie.launcher prefix, by switching oozie.launcher.override (on by default).

For those, following is the general approach:

• check whether a YARN compatible entry is present. If yes, use it to override default value
• check whether a MapReduce v2 compatible entry is present. If yes, use it to override default value
• check whether a MapReduce v1 compatible entry is present. If yes, use it to override default value
• use default value

Such properties are (legend: YARN / MapReduce v2 / MapReduce v1):

• max attempts of the MapReduce Application Master:
  • N / A
  • mapreduce.map.maxattempts
  • mapred.map.max.attempts
• memory amount in MB of the MapReduce Application Master:
  • yarn.app.mapreduce.am.resource.mb
  • mapreduce.map.memory.mb
  • mapred.job.map.memory.mb
• CPU vcore count of the MapReduce Application Master:
  • yarn.app.mapreduce.am.resource.cpu-vcores
  • mapreduce.map.cpu.vcores
  • N / A
• logging level of the MapReduce Application Master:
  • N / A
  • mapreduce.map.log.level
  • mapred.map.child.log.level
• MapReduce Application Master JVM options:
  • yarn.app.mapreduce.am.command-opts
  • mapreduce.map.java.opts
  • mapred.child.java.opts
• MapReduce Application Master environment variable settings:
  • yarn.app.mapreduce.am.env
  • mapreduce.map.env
  • mapred.child.env
• MapReduce Application Master job priority:
  • N / A
  • mapreduce.job.priority
  • mapred.job.priority
• MapReduce Application Master job queue name:
  • N / A
  • mapreduce.job.queuename
  • mapred.job.queue.name
• MapReduce View ACL settings:
  • N / A
  • mapreduce.job.acl-view-job
  • N / A
• MapReduce Modify ACL settings:
  • N / A
  • mapreduce.job.acl-modify-job
  • N / A

This list can be extended or modified by adding new configuration entries or updating existing values beginning with oozie.launcher.override. within oozie-site.xml. Examples can be found in oozie-default.xml.

Prepending Configuration Values

Prepending happens for following configuration entries with oozie.launcher prefix, by switching oozie.launcher.prepend (on by default).

For those, following is the general approach:

• check whether a YARN compatible entry is present. If yes, use it to prepend to default value
• use default value

Such properties are (legend: YARN only):

• MapReduce Application Master JVM options: yarn.app.mapreduce.am.admin-command-opts
• MapReduce Application Master environment settings: yarn.app.mapreduce.am.admin.user.env

This list can be extended or modified by adding new configuration entries or updating existing values beginning with oozie.launcher.prepend. within oozie-site.xml. Examples can be found in oozie-default.xml.

Defining Admin Users

Admin users are determined from the list of admin groups, specified in oozie.service.AuthorizationService.admin.groups property. Use commas to separate multiple groups, spaces, tabs and ENTER characters are trimmed.

If the above property for admin groups is not set, then defining the admin users can happen in the following manners. The list of admin users can be in the conf/adminusers.txt file. The syntax of this file is:

• One user name per line
• Empty lines and lines starting with ‘#’ are ignored

Admin users can also be defined in oozie.serviceAuthorizationService.admin.users property. Use commas to separate multiple admin users, spaces, tabs and ENTER characters are trimmed.

In case there are admin users defined using both methods, the effective list of admin users will be the union of the admin users found in the adminusers.txt and those specified with oozie.serviceAuthorizationService.admin.users.

Defining Access Control Lists

Access Control Lists are defined in the following ways:

• workflow job submission over CLI: configuration property group.name of job.properties
• workflow job submission over HTTP: configuration property group.name of the XML submitted over HTTP
• workflow job re-run: configuration property oozie.job.acl (preferred) or configuration property group.name of job.properties
• coordinator job submission over CLI: configuration property oozie.job.acl (preferred) or configuration property group.name of job.properties
• bundle job submission over CLI: configuration property oozie.job.acl (preferred) or configuration property group.name of job.properties

For all other workflow, coordinator, or bundle actions the ACL set in beforehand will be used as basis.

Once the ACL for the job is defined, Oozie will check over HDFS whether the user trying to perform a specific action is part of the necessary group(s). For implementation details please check out org.apache.hadoop.security.Groups#getGroups(String user).

Note that it’s enough that the submitting user be part of at least one group of the ACL. Note also that the ACL can contain user names as well. If there is an ACL defined and the submitting user isn’t part of any group or user name present in the ACL, an AuthorizationException is thrown.

Example: A typical ACL setup

Detail of job.properties on workflow job submission:

user.name=joe
group.name=marketing,admin,qa,root

HDFS group membership of HDFS user joe is qa. That is, the check to org.apache.hadoop.security.Groups#getGroups("joe") returns qa. Hence, ACL check will pass inside AuthorizationService, because the user.name provided belongs to at least of the ACL list elements provided as group.name.

To use a Self-Signed Certificate

There are many ways to create a Self-Signed Certificate, this is just one way. We will be using the keytool program, which is included with your JRE. If it’s not on your path, you should be able to find it in $JAVA_HOME/bin.

1. Run the following command (as the Oozie user) to create the keystore file, which will be named .keystore and located in the Oozie user’s home directory.

   keytool -genkeypair -alias jetty -keyalg RSA -dname "CN=hostname" -storepass password -keypass password
   

   The hostname should be the host name of the Oozie Server or a wildcard on the subdomain it belongs to. Make sure to include the “CN=” part. You can change storepass and keypass values, but they should be the same. If you do want to use something other than password, you’ll also need to change the value of the oozie.https.keystore.pass property in oozie-site.xml to match; password is the default.

   For example, if your Oozie server was at oozie.int.example.com, then you would do this:

   keytool -genkeypair -alias jetty -keyalg RSA -dname "CN=oozie.int.example.com" -storepass password -keypass password
   

   If you’re going to be using Oozie HA, it’s simplest if you have a single certificate that all Oozie servers in the HA group can use. To do that, you’ll need to use a wildcard on the subdomain it belongs to:

   keytool -genkeypair -alias jetty -keyalg RSA -dname "CN=*.int.example.com" -storepass password -keypass password
   

   The above would work on any server in the int.example.com domain.

2. Run the following command (as the Oozie user) to export a certificate file from the keystore file:

   keytool -exportcert -alias jetty -file path/to/anywhere/certificate.cert -storepass password
   

3. Run the following command (as any user) to create a truststore containing the certificate we just exported:

   keytool -import -alias jetty -file path/to/certificate.cert -keystore /path/to/anywhere/oozie.truststore -storepass password2
   

   You’ll need the oozie.truststore later if you’re using the Oozie client (or other Java-based client); otherwise, you can skip this step. The storepass value here is only used to verify or change the truststore and isn’t typically required when only reading from it; so it does not have to be given to users only using the client.

To use a Certificate from a Certificate Authority

1. You will need to make a request to a Certificate Authority in order to obtain a proper Certificate; please consult a Certificate Authority on this procedure. If you’re going to be using Oozie HA, it’s simplest if you have a single certificate that all Oozie servers in the HA group can use. To do that, you’ll need to use a wild on the subdomain it belongs to (e.g. “*.int.example.com”).

2. Once you have your .cert file, run the following command (as the Oozie user) to create a keystore file from your certificate:

   keytool -import -alias jetty -file path/to/certificate.cert
   

   The keystore file will be named .keystore and located in the Oozie user’s home directory.

Configure the Oozie Server to use SSL (HTTPS)

1. Make sure the Oozie server isn’t running

2. Configure settings necessary for enabling SSL/TLS support in oozie-site.xml.

   2a. Set oozie.https.enabled to true. To revert back to HTTP, set oozie.https.enabled to false.

   2b. Set location and password for the keystore and location for truststore by setting oozie.https.keystore.file, oozie.https.keystore.pass, oozie.https.truststore.file.

   Note: oozie.https.truststore.file can be overridden by setting javax.net.ssl.trustStore system property.

   The default HTTPS port Oozie listens on for secure connections is 11443; it can be changed via oozie.https.port.

   It is possible to specify other HTTPS settings via oozie-site.xml: - To include / exclude cipher suites, set oozie.https.include.cipher.suites / oozie.https.exclude.cipher.suites. - To include / exclude TLS protocols, set oozie.https.include.protocols / oozie.https.exclude.protocols. Note: Exclude is always preferred over include (i.e. if you both include and exclude an entity, it will be excluded).

   Note: When SSL is enabled, HTTP Strict-Transport_security (HSTS) is also enabled. The default value for max-age is 31536000 (one year). This can be changed by setting oozie.hsts.max.age.seconds property. Setting it to 0 or negative value, will disable HSTS.

3. Start the Oozie server

   Note: If using Oozie HA, make sure that each Oozie server has a copy of the .keystore file.

Configure the Oozie Client to connect using SSL (HTTPS)

The first two steps are only necessary if you are using a Self-Signed Certificate; the third is required either way. Also, these steps must be done on every machine where you intend to use the Oozie Client.

1. Copy or download the oozie.truststore file onto the client machine

2. When using any Java-based program, you’ll need to pass -Djavax.net.ssl.trustStore to the JVM. To do this for the Oozie client:

   export OOZIE_CLIENT_OPTS='-Djavax.net.ssl.trustStore=/path/to/oozie.truststore'
   

3. When using the Oozie Client, you will need to use https://oozie.server.hostname:11443/oozie instead of http://oozie.server.hostname:11000/oozie – Java will not automatically redirect from the http address to the https address.

Connect to the Oozie Web UI using SSL (HTTPS)

1. Use https://oozie.server.hostname:11443/oozie though most browsers should automatically redirect you if you use http://oozie.server.hostname:11000/oozie

   IMPORTANT: If using a Self-Signed Certificate, your browser will warn you that it can’t verify the certificate or something similar. You will probably have to add your certificate as an exception.

Additional considerations for Oozie HA with SSL

You’ll need to configure the load balancer to do SSL pass-through. This will allow the clients talking to Oozie to use the SSL certificate provided by the Oozie servers (so the load balancer does not need one). Please consult your load balancer’s documentation on how to configure this. Make sure to point the load balancer at the https://HOST:HTTPS_PORT addresses for your Oozie servers. Clients can then connect to the load balancer at https://LOAD_BALANCER_HOST:PORT.

Important: Callbacks from the ApplicationMaster are done via http or https depending on what you enter for the OOZIE_BASE_URL property. If you are using a Certificate from a Certificate Authority, you can simply put the https address here. If you are using a self-signed certificate, you have to do one of the following options (Option 1 is recommended):

Option 1) You’ll need to follow the steps in the Configure the Oozie Client to connect using SSL (HTTPS) section, but on the host of the ApplicationMaster. You can then set OOZIE_BASE_URL to the load balancer https address. This will allow the ApplicationMaster to contact the Oozie server with https (like the Oozie client, they are also Java programs).

Option 2) You’ll need setup another load balancer, or another “pool” on the existing load balancer, with the http addresses of the Oozie servers. You can then set OOZIE_BASE_URL to the load balancer http address. Clients should use the https load balancer address. This will allow clients to use https while the ApplicationMaster uses http for callbacks.

Pre-requisites

1. A database that supports multiple concurrent connections. In order to have full HA, the database should also have HA support, or it becomes a single point of failure.

   NOTE: The default derby database does not support this

2. A ZooKeeper ensemble.

   Apache ZooKeeper is a distributed, open-source coordination service for distributed applications; the Oozie servers use it for coordinating access to the database and communicating with each other. In order to have full HA, there should be at least 3 ZooKeeper servers. More information on ZooKeeper can be found here.

3. Multiple Oozie servers.

   IMPORTANT: While not strictly required for all configuration properties, all of the servers should ideally have exactly the same configuration for consistency’s sake.

4. A Loadbalancer, Virtual IP, or Round-Robin DNS.

   This is used to provide a single entry-point for users and for callbacks from the JobTracker/ResourceManager. The load balancer should be configured for round-robin between the Oozie servers to distribute the requests. Users (using either the Oozie client, a web browser, or the REST API) should connect through the load balancer. In order to have full HA, the load balancer should also have HA support, or it becomes a single point of failure.

Installation/Configuration Steps

1. Install identically configured Oozie servers normally. Make sure they are all configured against the same database and make sure that you DO NOT start them yet.

2. Add the following services to the extension services configuration property in oozie-site.xml in all Oozie servers. This will make Oozie use the ZooKeeper versions of these services instead of the default implementations.

   <property>
       <name>oozie.services.ext</name>
       <value>
           org.apache.oozie.service.ZKLocksService,
           org.apache.oozie.service.ZKXLogStreamingService,
           org.apache.oozie.service.ZKJobsConcurrencyService,
           org.apache.oozie.service.ZKUUIDService
       </value>
   </property>
   

3. Add the following property to oozie-site.xml in all Oozie servers. It should be a comma-separated list of host:port pairs of the ZooKeeper servers. The default value is shown below.

   <property>
      <name>oozie.zookeeper.connection.string</name>
      <value>localhost:2181</value>
   </property>
   

4. (Optional) Add the following property to oozie-site.xml in all Oozie servers to specify the namespace to use. All of the Oozie Servers that are planning on talking to each other should have the same namespace. If there are multiple Oozie setups each doing their own HA, they should have their own namespace. The default value is shown below.

   <property>
       <name>oozie.zookeeper.namespace</name>
       <value>oozie</value>
   </property>
   

5. Change the value of OOZIE_BASE_URL in oozie-site.xml to point to the loadbalancer or virtual IP, for example:

   <property>
       <name>oozie.base.url</name>
       <value>http://my.loadbalancer.hostname:11000/oozie</value>
   </property>
   

6. (Optional) If using a secure cluster, see Security below on configuring Kerberos with Oozie HA.

7. Start the ZooKeeper servers.

8. Start the Oozie servers.

   Note: If one of the Oozie servers becomes unavailable, querying Oozie for the logs from a job in the Web UI, REST API, or client may be missing information until that server comes back up.

Security

Oozie HA works with the existing Oozie security framework and settings. For HA features (log streaming, share lib, etc) to work properly in a secure setup, following property can be set on each server. If oozie.server.authentication.type is not set, then server-server authentication will fall back on oozie.authentication.type.

<property>
    <name>oozie.server.authentication.type</name>
    <value>kerberos</value>
</property>

Below are some additional steps and information specific to Oozie HA:

1. (Optional) To prevent unauthorized users or programs from interacting with or reading the znodes used by Oozie in ZooKeeper, you can tell Oozie to use Kerberos-backed ACLs. To enforce this for all of the Oozie-related znodes, simply add the following property to oozie-site.xml in all Oozie servers and set it to true. The default is false.

   <property>
       <name>oozie.zookeeper.secure</name>
       <value>true</value>
   </property>
   

   Note: The Kerberos principals of each of the Oozie servers should have the same primary name (i.e. in primary/instance@REALM, each server should have the same value for primary).

   Important: Once this property is set to true, it will set the ACLs on all existing Oozie-related znodes to only allow Kerberos authenticated users with a principal that has the same primary as described above (also for any subsequently created new znodes). This means that if you ever want to turn this feature off, you will have to manually connect to ZooKeeper using a Kerberos principal with the same primary and either delete all znodes under and including the namespace (i.e. if oozie.zookeeper.namespace oozie= then that would be /oozie); alternatively, instead of deleting them all, you can manually set all of their ACLs to world:anyone. In either case, make sure that no Oozie servers are running while this is being done.

   Also, in your zoo.cfg for ZooKeeper, make sure to set the following properties:

   authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
   kerberos.removeHostFromPrincipal=true
   kerberos.removeRealmFromPrincipal=true
   

2. Until Hadoop 2.5.0 and later, there is a known limitation where each Oozie server can only use one HTTP principal. However, for Oozie HA, we need to use two HTTP principals: HTTP/oozie-server-host@realm and HTTP/load-balancer-host@realm. This allows access to each Oozie server directly and through the load balancer. While users should always go through the load balancer, certain features (e.g. log streaming) require the Oozie servers to talk to each other directly; it can also be helpful for an administrator to talk directly to an Oozie server. So, if using a Hadoop version prior to 2.5.0, you will have to choose which HTTP principal to use as you cannot use both; it is recommended to choose HTTP/load-balancer-host@realm so users can connect through the load balancer. This will prevent Oozie servers from talking to each other directly, which will effectively disable log streaming.

   For Hadoop 2.5.0 and later:

   2a. When creating the keytab used by Oozie, make sure to include Oozie’s principal and the two HTTP principals mentioned above.

   2b. Set oozie.authentication.kerberos.principal to * (that is, an asterisks) so it will use both HTTP principals.

   For earlier versions of Hadoop:

   2a. When creating the keytab used by Oozie, make sure to include Oozie’s principal and the load balancer HTTP principal

   2b. Set oozie.authentication.kerberos.principal to HTTP/load-balancer-host@realm.

3. With Hadoop 2.6.0 and later, a rolling random secret that is synchronized across all Oozie servers will be used for signing the Oozie auth tokens. This is done automatically when HA is enabled; no additional configuration is needed.

   For earlier versions of Hadoop, each server will have a different random secret. This will still work but will likely result in additional calls to the KDC to authenticate users to the Oozie server (because the auth tokens will not be accepted by other servers, which will cause a fallback to Kerberos).

4. If you’d like to use HTTPS (SSL) with Oozie HA, there’s some additional considerations that need to be made. See the Setting Up Oozie with HTTPS (SSL) section for more information.

JobId sequence

Oozie in HA mode, uses ZK to generate job id sequence. Job Ids are of following format. <Id sequence>-<yyMMddHHmmss(server start time)>-<system_id>-<W/C/B>

Where, <systemId> is configured as oozie.system.id (default is “oozie-” + “user.name”) W/C/B is suffix to job id indicating that generated job is a type of workflow or coordinator or bundle.

Maximum allowed character for job id sequence is 40. “Id sequence” is stored in ZK and reset to 0 once maximum job id sequence is reached. Maximum job id sequence is configured as oozie.service.ZKUUIDService.jobid.sequence.max, default value is 99999999990.

<property>
    <name>oozie.service.ZKUUIDService.jobid.sequence.max</name>
    <value>99999999990</value>
</property>