org.apache.oozie.service

Class HadoopAccessorService

java.lang.Object

org.apache.oozie.service.HadoopAccessorService

All Implemented Interfaces:

Service

public class HadoopAccessorService
extends Object
implements Service

The HadoopAccessorService returns HadoopAccessor instances configured to work on behalf of a user-group.

The default accessor used is the base accessor which just injects the UGI into the configuration instance used to create/obtain JobClient and FileSystem instances.

Field Summary

Fields

Modifier and Type	Field and Description
static String	ACTION_CONFS
static String	ACTION_CONFS_LOAD_DEFAULT_RESOURCES
static String	CONF_PREFIX
static String	HADOOP_CONFS
protected static String	HADOOP_JOB_TRACKER
protected static String	HADOOP_JOB_TRACKER_2
protected static String	HADOOP_YARN_RM
static String	JOB_TRACKER_WHITELIST
protected static String	JT_PRINCIPAL The Kerberos principal for the job tracker.
static String	KERBEROS_AUTH_ENABLED
static String	KERBEROS_KEYTAB
static String	KERBEROS_PRINCIPAL
static org.apache.hadoop.io.Text	MR_TOKEN_ALIAS
static String	NAME_NODE_WHITELIST
protected static String	OOZIE_HADOOP_ACCESSOR_SERVICE_CREATED
protected static String	RM_PRINCIPAL The Kerberos principal for the resource manager.
static String	SUPPORTED_FILESYSTEMS Supported filesystem schemes for namespace federation

Fields inherited from interface org.apache.oozie.service.Service

DEFAULT_LOCK_TIMEOUT, lockTimeout

Constructor Summary

Constructors

Constructor and Description
HadoopAccessorService()

Method Summary

Methods

Modifier and Type	Method and Description
void	addFileToClassPath(String user, org.apache.hadoop.fs.Path file, org.apache.hadoop.conf.Configuration conf)
void	checkSupportedFilesystem(URI uri) checks configuration parameter if filesystem scheme is among the list of supported ones this makes system robust to filesystems other than HDFS also
XConfiguration	createActionDefaultConf(String hostPort, String action) Returns a Configuration containing any defaults for an action for a particular cluster.
org.apache.hadoop.fs.FileSystem	createFileSystem(String user, URI uri, org.apache.hadoop.conf.Configuration conf) Return a FileSystem created with the provided user for the specified URI.
org.apache.hadoop.mapred.JobClient	createJobClient(String user, org.apache.hadoop.mapred.JobConf conf) Return a JobClient created with the provided user/group.
org.apache.hadoop.mapred.JobConf	createJobConf(String hostPort) Creates a JobConf using the site configuration for the specified hostname:port.
void	destroy() Destroy the service.
org.apache.hadoop.conf.Configuration	getCachedConf()
Class<? extends Service>	getInterface() Return the public interface of the service.
org.apache.hadoop.io.Text	getMRDelegationTokenRenewer(org.apache.hadoop.mapred.JobConf jobConf)
Set<String>	getSupportedSchemes()
void	init(org.apache.hadoop.conf.Configuration conf)
void	init(Services services) Initialize the service.
protected void	validateJobTracker(String jobTrackerUri) Validate Job tracker
protected void	validateNameNode(String nameNodeUri) Validate Namenode list

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CONF_PREFIX

public static final String CONF_PREFIX

See Also:

Constant Field Values

JOB_TRACKER_WHITELIST

public static final String JOB_TRACKER_WHITELIST

See Also:

Constant Field Values

NAME_NODE_WHITELIST

public static final String NAME_NODE_WHITELIST

See Also:

Constant Field Values

HADOOP_CONFS

public static final String HADOOP_CONFS

See Also:

Constant Field Values

ACTION_CONFS

public static final String ACTION_CONFS

See Also:

Constant Field Values

ACTION_CONFS_LOAD_DEFAULT_RESOURCES

public static final String ACTION_CONFS_LOAD_DEFAULT_RESOURCES

See Also:

Constant Field Values

KERBEROS_AUTH_ENABLED

public static final String KERBEROS_AUTH_ENABLED

See Also:

Constant Field Values

KERBEROS_KEYTAB

public static final String KERBEROS_KEYTAB

See Also:

Constant Field Values

KERBEROS_PRINCIPAL

public static final String KERBEROS_PRINCIPAL

See Also:

Constant Field Values

MR_TOKEN_ALIAS

public static final org.apache.hadoop.io.Text MR_TOKEN_ALIAS

OOZIE_HADOOP_ACCESSOR_SERVICE_CREATED

protected static final String OOZIE_HADOOP_ACCESSOR_SERVICE_CREATED

See Also:

Constant Field Values

JT_PRINCIPAL

protected static final String JT_PRINCIPAL

The Kerberos principal for the job tracker.

See Also:

Constant Field Values

RM_PRINCIPAL

protected static final String RM_PRINCIPAL

The Kerberos principal for the resource manager.

See Also:

Constant Field Values

HADOOP_JOB_TRACKER

protected static final String HADOOP_JOB_TRACKER

See Also:

Constant Field Values

HADOOP_JOB_TRACKER_2

protected static final String HADOOP_JOB_TRACKER_2

See Also:

Constant Field Values

HADOOP_YARN_RM

protected static final String HADOOP_YARN_RM

See Also:

Constant Field Values

SUPPORTED_FILESYSTEMS

public static final String SUPPORTED_FILESYSTEMS

Supported filesystem schemes for namespace federation

See Also:

Constant Field Values

Constructor Detail

HadoopAccessorService

public HadoopAccessorService()

Method Detail

init

public void init(Services services)
          throws ServiceException

Description copied from interface: Service

Initialize the service.

Invoked by the Service singleton at start up time.

Specified by:

init in interface Service

Parameters:

services - services singleton initializing the service.

Throws:

ServiceException - thrown if the service could not initialize.

init

public void init(org.apache.hadoop.conf.Configuration conf)
          throws ServiceException

Throws:

ServiceException

destroy

public void destroy()

Description copied from interface: Service

Destroy the service.

Invoked by the Service singleton at shutdown time.

Specified by:

destroy in interface Service

getInterface

public Class<? extends Service> getInterface()

Description copied from interface: Service

Return the public interface of the service.

Services are retrieved by its public interface. Specializations of services must return the public interface.

Specified by:

getInterface in interface Service

Returns:

the interface of the service.

createJobConf

public org.apache.hadoop.mapred.JobConf createJobConf(String hostPort)

Creates a JobConf using the site configuration for the specified hostname:port.

If the specified hostname:port is not defined it falls back to the '*' site configuration if available. If the '*' site configuration is not available, the JobConf has all Hadoop defaults.

Parameters:

hostPort - hostname:port to lookup Hadoop site configuration.

Returns:

a JobConf with the corresponding site configuration for hostPort.

getCachedConf

public org.apache.hadoop.conf.Configuration getCachedConf()

createActionDefaultConf

public XConfiguration createActionDefaultConf(String hostPort,
                                     String action)

Returns a Configuration containing any defaults for an action for a particular cluster.

This configuration is used as default for the action configuration and enables cluster level default values per action.

Parameters:

hostPort - hostname"port to lookup the action default confiugration.

action - action name.

Returns:

the default configuration for the action for the specified cluster.

createJobClient

public org.apache.hadoop.mapred.JobClient createJobClient(String user,
                                                 org.apache.hadoop.mapred.JobConf conf)
                                                   throws HadoopAccessorException

Return a JobClient created with the provided user/group.

Parameters:

conf - JobConf with all necessary information to create the JobClient.

Returns:

JobClient created with the provided user/group.

Throws:

HadoopAccessorException - if the client could not be created.

createFileSystem

public org.apache.hadoop.fs.FileSystem createFileSystem(String user,
                                               URI uri,
                                               org.apache.hadoop.conf.Configuration conf)
                                                 throws HadoopAccessorException

Return a FileSystem created with the provided user for the specified URI.

Parameters:

uri - file system URI.

conf - Configuration with all necessary information to create the FileSystem.

Returns:

FileSystem created with the provided user/group.

Throws:

HadoopAccessorException - if the filesystem could not be created.

validateJobTracker

protected void validateJobTracker(String jobTrackerUri)
                           throws HadoopAccessorException

Validate Job tracker

Parameters:

jobTrackerUri -

Throws:

HadoopAccessorException

validateNameNode

protected void validateNameNode(String nameNodeUri)
                         throws HadoopAccessorException

Validate Namenode list

Parameters:

nameNodeUri -

Throws:

HadoopAccessorException

getMRDelegationTokenRenewer

public org.apache.hadoop.io.Text getMRDelegationTokenRenewer(org.apache.hadoop.mapred.JobConf jobConf)
                                                      throws IOException

Throws:

IOException

addFileToClassPath

public void addFileToClassPath(String user,
                      org.apache.hadoop.fs.Path file,
                      org.apache.hadoop.conf.Configuration conf)
                        throws IOException

Throws:

IOException

checkSupportedFilesystem

public void checkSupportedFilesystem(URI uri)
                              throws HadoopAccessorException

checks configuration parameter if filesystem scheme is among the list of supported ones this makes system robust to filesystems other than HDFS also

Throws:

HadoopAccessorException

getSupportedSchemes

public Set<String> getSupportedSchemes()