org.apache.oozie.service

Class HadoopAccessorService

java.lang.Object

org.apache.oozie.service.HadoopAccessorService

All Implemented Interfaces:

Service

public class HadoopAccessorService
extends Object
implements Service

The HadoopAccessorService returns HadoopAccessor instances configured to work on behalf of a user-group.

The default accessor used is the base accessor which just injects the UGI into the configuration instance used to create/obtain JobClient and FileSystem instances.

Field Summary

Fields

Modifier and Type	Field and Description
static String	ACTION_CONFS
static String	ACTION_CONFS_LOAD_DEFAULT_RESOURCES
static String	CONF_PREFIX
static String	HADOOP_CONFS
static String	JOB_TRACKER_WHITELIST
static String	KERBEROS_AUTH_ENABLED
static String	KERBEROS_KEYTAB
static String	KERBEROS_PRINCIPAL
static String	NAME_NODE_WHITELIST
static String	SUPPORTED_FILESYSTEMS Supported filesystem schemes for namespace federation

Fields inherited from interface org.apache.oozie.service.Service

DEFAULT_LOCK_TIMEOUT, lockTimeout

Constructor Summary

Constructors

Constructor and Description
HadoopAccessorService()

Method Summary

Modifier and Type	Method and Description
void	addFileToClassPath(String user, org.apache.hadoop.fs.Path file, org.apache.hadoop.conf.Configuration conf)
void	checkSupportedFilesystem(URI uri) checks configuration parameter if filesystem scheme is among the list of supported ones this makes system robust to filesystems other than HDFS also
XConfiguration	createActionDefaultConf(String hostPort, String action) Returns a Configuration containing any defaults for an action for a particular cluster.
org.apache.hadoop.conf.Configuration	createConfiguration(String hostPort) Creates a Configuration using the site configuration for the specified hostname:port.
org.apache.hadoop.fs.FileSystem	createFileSystem(String user, URI uri, org.apache.hadoop.conf.Configuration conf) Return a FileSystem created with the provided user for the specified URI.
org.apache.hadoop.mapred.JobClient	createJobClient(String user, org.apache.hadoop.conf.Configuration conf) Return a JobClient created with the provided user/group.
org.apache.hadoop.mapred.JobClient	createJobClient(String user, org.apache.hadoop.mapred.JobConf conf) Return a JobClient created with the provided user/group.
org.apache.hadoop.yarn.api.records.LocalResource	createLocalResourceForConfigurationFile(String filename, String user, org.apache.hadoop.conf.Configuration conf, URI uri, org.apache.hadoop.fs.Path dir) Creates a LocalResource for the Configuration to localize it for a Yarn Container.
org.apache.hadoop.yarn.client.api.YarnClient	createYarnClient(String user, org.apache.hadoop.conf.Configuration conf) Return a YarnClient created with the provided user and configuration.
void	destroy() Destroy the service.
org.apache.hadoop.conf.Configuration	getCachedConf()
Class<? extends Service>	getInterface() Return the public interface of the service.
Set<String>	getSupportedSchemes()
void	init(org.apache.hadoop.conf.Configuration conf)
void	init(Services services) Initialize the service.
protected void	validateJobTracker(String jobTrackerUri) Validate Job tracker
protected void	validateNameNode(String nameNodeUri) Validate Namenode list

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CONF_PREFIX

public static final String CONF_PREFIX

See Also:

Constant Field Values

JOB_TRACKER_WHITELIST

public static final String JOB_TRACKER_WHITELIST

See Also:

Constant Field Values

NAME_NODE_WHITELIST

public static final String NAME_NODE_WHITELIST

See Also:

Constant Field Values

HADOOP_CONFS

public static final String HADOOP_CONFS

See Also:

Constant Field Values

ACTION_CONFS

public static final String ACTION_CONFS

See Also:

Constant Field Values

ACTION_CONFS_LOAD_DEFAULT_RESOURCES

public static final String ACTION_CONFS_LOAD_DEFAULT_RESOURCES

See Also:

Constant Field Values

KERBEROS_AUTH_ENABLED

public static final String KERBEROS_AUTH_ENABLED

See Also:

Constant Field Values

KERBEROS_KEYTAB

public static final String KERBEROS_KEYTAB

See Also:

Constant Field Values

KERBEROS_PRINCIPAL

public static final String KERBEROS_PRINCIPAL

See Also:

Constant Field Values

SUPPORTED_FILESYSTEMS

public static final String SUPPORTED_FILESYSTEMS

Supported filesystem schemes for namespace federation

See Also:

Constant Field Values

Constructor Detail

HadoopAccessorService

public HadoopAccessorService()

Method Detail

init

public void init(Services services)
          throws ServiceException

Description copied from interface: Service

Initialize the service.

Invoked by the Service singleton at start up time.

Specified by:

init in interface Service

Parameters:

services - services singleton initializing the service.

Throws:

ServiceException - thrown if the service could not initialize.

init

public void init(org.apache.hadoop.conf.Configuration conf)
          throws ServiceException

Throws:

ServiceException

destroy

public void destroy()

Description copied from interface: Service

Destroy the service.

Invoked by the Service singleton at shutdown time.

Specified by:

destroy in interface Service

getInterface

public Class<? extends Service> getInterface()

Description copied from interface: Service

Return the public interface of the service.

Services are retrieved by its public interface. Specializations of services must return the public interface.

Specified by:

getInterface in interface Service

Returns:

the interface of the service.

createConfiguration

public org.apache.hadoop.conf.Configuration createConfiguration(String hostPort)

Creates a Configuration using the site configuration for the specified hostname:port.

If the specified hostname:port is not defined it falls back to the '*' site configuration if available. If the '*' site configuration is not available, the JobConf has all Hadoop defaults.

Parameters:

hostPort - hostname:port to lookup Hadoop site configuration.

Returns:

a Configuration with the corresponding site configuration for hostPort.

getCachedConf

public org.apache.hadoop.conf.Configuration getCachedConf()

createActionDefaultConf

public XConfiguration createActionDefaultConf(String hostPort,
                                              String action)

Returns a Configuration containing any defaults for an action for a particular cluster.

This configuration is used as default for the action configuration and enables cluster level default values per action.

Parameters:

hostPort - hostname"port to lookup the action default confiugration.

action - action name.

Returns:

the default configuration for the action for the specified cluster.

createJobClient

public org.apache.hadoop.mapred.JobClient createJobClient(String user,
                                                          org.apache.hadoop.mapred.JobConf conf)
                                                   throws HadoopAccessorException

Return a JobClient created with the provided user/group.

Parameters:

conf - JobConf with all necessary information to create the JobClient.

Returns:

JobClient created with the provided user/group.

Throws:

HadoopAccessorException - if the client could not be created.

createJobClient

public org.apache.hadoop.mapred.JobClient createJobClient(String user,
                                                          org.apache.hadoop.conf.Configuration conf)
                                                   throws HadoopAccessorException

Return a JobClient created with the provided user/group.

Parameters:

conf - Configuration with all necessary information to create the JobClient.

Returns:

JobClient created with the provided user/group.

Throws:

HadoopAccessorException - if the client could not be created.

createYarnClient

public org.apache.hadoop.yarn.client.api.YarnClient createYarnClient(String user,
                                                                     org.apache.hadoop.conf.Configuration conf)
                                                              throws HadoopAccessorException

Return a YarnClient created with the provided user and configuration. The caller is responsible for closing it when done.

Parameters:

user - The username to impersonate

conf - The conf

Returns:

a YarnClient with the provided user and configuration

Throws:

HadoopAccessorException - if the client could not be created.

createFileSystem

public org.apache.hadoop.fs.FileSystem createFileSystem(String user,
                                                        URI uri,
                                                        org.apache.hadoop.conf.Configuration conf)
                                                 throws HadoopAccessorException

Return a FileSystem created with the provided user for the specified URI.

Parameters:

user - The username to impersonate

uri - file system URI.

conf - Configuration with all necessary information to create the FileSystem.

Returns:

FileSystem created with the provided user/group.

Throws:

HadoopAccessorException - if the filesystem could not be created.

validateJobTracker

protected void validateJobTracker(String jobTrackerUri)
                           throws HadoopAccessorException

Validate Job tracker

Parameters:

jobTrackerUri -

Throws:

HadoopAccessorException

validateNameNode

protected void validateNameNode(String nameNodeUri)
                         throws HadoopAccessorException

Validate Namenode list

Parameters:

nameNodeUri -

Throws:

HadoopAccessorException

addFileToClassPath

public void addFileToClassPath(String user,
                               org.apache.hadoop.fs.Path file,
                               org.apache.hadoop.conf.Configuration conf)
                        throws IOException

Throws:

IOException

checkSupportedFilesystem

public void checkSupportedFilesystem(URI uri)
                              throws HadoopAccessorException

checks configuration parameter if filesystem scheme is among the list of supported ones this makes system robust to filesystems other than HDFS also

Throws:

HadoopAccessorException

getSupportedSchemes

public Set<String> getSupportedSchemes()

createLocalResourceForConfigurationFile

public org.apache.hadoop.yarn.api.records.LocalResource createLocalResourceForConfigurationFile(String filename,
                                                                                                String user,
                                                                                                org.apache.hadoop.conf.Configuration conf,
                                                                                                URI uri,
                                                                                                org.apache.hadoop.fs.Path dir)
                                                                                         throws IOException,
                                                                                                HadoopAccessorException,
                                                                                                URISyntaxException

Creates a LocalResource for the Configuration to localize it for a Yarn Container. This involves also writing it to HDFS. Example usage: *

 
 LocalResource res1 = createLocalResourceForConfigurationFile(filename1, user, conf, uri, dir);
 LocalResource res2 = createLocalResourceForConfigurationFile(filename2, user, conf, uri, dir);
 ...
 Map<String, LocalResource> localResources = new HashMap<String, LocalResource>();
 localResources.put(filename1, res1);
 localResources.put(filename2, res2);
 ...
 containerLaunchContext.setLocalResources(localResources);
 
 

Parameters:

filename - The filename to use on the remote filesystem and once it has been localized.

user - The user

conf - The configuration to process

uri - The URI of the remote filesystem (e.g. HDFS)

dir - The directory on the remote filesystem to write the file to

Returns:

localResource

Throws:

IOException - A problem occurred writing the file

HadoopAccessorException - A problem occured with Hadoop

URISyntaxException - A problem occurred parsing the URI