Troubleshooting using Kerberos with Talend Big Data

author
Talend Documentation Team
EnrichVersion
6.4
6.3
6.2
6.1
6.0
EnrichProdName
Talend Data Fabric
Talend Real-Time Big Data Platform
Talend Big Data
Talend Big Data Platform
task
Design and Development > Designing Jobs > Hadoop distributions
Data Governance > Third-party systems > Authentication components > Kerberos components
Data Quality and Preparation > Third-party systems > Authentication components > Kerberos components
Design and Development > Third-party systems > Authentication components > Kerberos components
EnrichPlatform
Talend Studio

Troubleshooting using Kerberos with Talend Big Data

You may encounter error messages or unexpected issues when you use a Kerberos authentication to connect to a cluster. This article provides troubleshooting information when you get such errors.

For information on how to use Kerberos with Talend Big Data, see:

  • How to use Kerberos in Talend Studio with Big Data v5.x (part 1)
  • How to use Kerberos in Talend Studio with Big Data v6.x
Enabling the debugger

When the debugger is enabled, additional information is captured which can help diagnose problems.To enable the debugger:

  1. Navigate to the Run view of the Talend Studio.
  2. Click to open the Advanced settings tab. Select the Use specific JVM arguments check box.
  3. Click the New... button and add this argument:

-Dsun.security.krb5.debug=true

To collect debugging information, after enabling the debugger:

  1. Navigate to the Basic Run tab.
  2. Run the job.

Troubleshooting (Errors, Possible Causes and Resolution)

Once the detailed error is identified, search for it in the errors below.

Caused by: java.lang.IllegalArgumentException: Illegal principal name user@BIGDATA.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to user@BIGDATA.COM

Possible Cause

Possible realm conflict.

Resolution

Check your /etc/krb5.conf . If there are multiple realms, try setting your realm as the default one.

GSSException: No valid credentials provided (Mechanism level: Connection refused: connect)

Possible Cause

In krb5.ini, the KDC hostname is incorrect or the KDC daemon is not started on this server.

Resolution

Check for the daemon status or ask to verify the right KDC.

GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt

Possible Cause

User who executes the job has no valid ticket in his/her cache.

Resolution

Run kinit where the driver will run (on localhost if from studio, on the server where the JobServer lies if on remote).

java.io.IOException: java.lang.IllegalArgumentException: Server has invalid Kerberos principal: nn/sandbox.hadoop.com@EXAMPLE.COM; Host Details : local host is: "ServerName1/127.0.1.1";

Possible Cause

The server name is not defined in /etc/hosts .

Resolution

Check etc/hosts on the client machine and make sure you have the same hostname.

Or

Replace _HOST by the server's complete hostname.

java.io.IOException: Login failure for user1 from keytab C:/Users/user1.DOMAIN1/Documents/sko/user1.keytab Caused by: javax.security.auth.login.LoginException: Checksum failed

Possible Cause

The principal hash does not correspond to the hash the keytab knows. It can happen when the principal was once deleted and then recreated in the KDC database.

Resolution

Renew the obsolete keytab.

java.io.IOException: Login failure for user1 from keytab C:/Users/user1.DOMAIN/Documents/sko/user1.keytab Caused by: KrbException: Client not found in Kerberos database (devil) - CLIENT_NOT_FOUND

Possible Cause

The keytab correctly contains the user principal but KDC does not know this principal anymore.

Resolution

Renew the obsolete keytab.

java.io.IOException: Login failure for user1 from keytab C:/Users/user1.DOMAIN/Documents/sko/user1.keytab Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

Possible Cause

The keytab used does not contain any credentials corresponding to the specified principal or the keytab is not readable for the current user.

Resolution

Check the username specified in the Talend components, if ok, check the keytab privileges (read access) and validity.

kinit: Client not found in Kerberos database while getting initial credentials

Possible Cause

The user has no principal in the KDC database.

Resolution

Create the principal or use the right one (via kadmin or kadmin.local).

krb5_get_init_creds: unable to reach any KDC in realm EXAMPLE.COM, tried 1 KDC

Possible Cause

The KDC server is configured to use only UDP or TCP and not both, as supposed by your krb5.conf .

Resolution

Try to force the protocol in the krb5.conf by adding a line:

kdc = tcp/<kdc_server_hostname>:88

org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered coprocessor service found for name AuthenticationService in region hbase:meta,,1

Possible Cause

The HBase server side configurations for the coprocessor security are missing.

Resolution

Add the following to hbase-site.xml :

<property> 
	<name>hbase.coprocessor.region.classes</name> 
	<value>org.apache.hadoop.hbase.security.token.TokenProvider,
		   org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,
		   org.apache.hadoop.hbase.security.access.AccessController </value> 
</property>
<property> 
	<name>hbase.rpc.engine</name> 
	<value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> 
</property> 
<property> 
	<name>hbase.coprocessor.master.classes</name> 
	<value>org.apache.hadoop.hbase.security.access.AccessController</value> 
</property>

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): KERBEROS authentication is not enabled. Available:[SIMPLE]

Possible Cause

The Studio is configured to run a Job targeting a kerberized cluster whereas the server is not configured to use Kerberos.

Resolution

Remove the Kerberos credentials and configure the Job to access the cluster using simple authentication (user-based).

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]

Possible Cause

Studio tries to connect to a non-kerberized cluster whereas this is a kerberized environment.

Resolution

Activate the Kerberos credentials in the Job / Talend components.

org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

Possible Cause

Due to a bug of the studio for MapReduce Jobs, this error is happening when you activated Kerberos on a M/R Job but a username was previously set and the Studio still use it to access the cluster without taking Kerberos credentials into consideration.

Resolution

Deactivate the Kerberos authentication, set the username to blank and reactivate the Kerberos authentication.

org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to hdfs@TALEND.EXAMPLE.COM

Possible Cause

You face a cross-realm Kerberos environment and the mapping rules to translate principal from one realm to the Hadoop realm does not match the expected result.

Resolution

Check that the mapping rules are specified in the core-site.xml embedded in hadoop-conf-kerberos.jar . If yes, you have to correct these rules to obtain the right mapping.

Server has invalid Kerberos principal: hdfs/talend-cdh5-nn1@TALEND.COM

Possible Cause

The realm configured in krb5.conf or the realm configured in the Job does not match the server realm.

Resolution

Check the Job configuration and the krb5.ini to make sure this is aligned with the target server realm.

[WARN ]: org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException as:user@EXAMPLE.COM (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for user@EXAMPLE.COM to hbase/talend-cdh5@EXAMPLE.COM

Possible Cause

The server name is not defined in /etc/hosts .

Resolution

Add FQDN to /etc/hosts .