Troubleshooting using Kerberos with Talend Big Data - 7.2

author
Talend Documentation Team
EnrichVersion
7.2
EnrichProdName
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Real-Time Big Data Platform
task
Data Governance > Third-party systems > Authentication components > Kerberos components
Data Quality and Preparation > Third-party systems > Authentication components > Kerberos components
Design and Development > Designing Jobs > Hadoop distributions
Design and Development > Third-party systems > Authentication components > Kerberos components
EnrichPlatform
Talend Studio

Troubleshooting using Kerberos with Talend Big Data

You may encounter error messages or unexpected issues when you use a Kerberos authentication to connect to a cluster. This article provides troubleshooting information when you get such errors.

For information on how to use Kerberos with Talend Big Data, see:

  • How to use Kerberos in Talend Studio with Big Data v5.x (part 1)
  • How to use Kerberos in Talend Studio with Big Data v6.x
Enabling the debugger

When the debugger is enabled, additional information is captured which can help diagnose problems.To enable the debugger:

  1. Navigate to the Run view of the Talend Studio.
  2. Click to open the Advanced settings tab. Select the Use specific JVM arguments check box.
  3. Click the New... button and add this argument:

-Dsun.security.krb5.debug=true

To collect debugging information, after enabling the debugger:

  1. Navigate to the Basic Run tab.
  2. Run the job.
  3. Examine the collected log messages. In doing this, pay attention to the used principal and how the clien, that is to say, the Studio or your Hadoop cluster, is dealing with Kerberos from the cache.
Troubleshooting (Common errors, Possible Causes and Resolution)

Once the detailed error is identified, search for it in the errors below.

  • "Exception in thread "main" java.lang.ExceptionInInitializerError
    
    at java.lang.Class.forName0(Native Method)
    
        at java.lang.Class.forName(Class.java:274)
        at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2147)

    Possible Cause

    You java driver does not support the kerbberos security.

    Resolution

    Get http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html and unzip it on your $JAVAHOME/jre/lib/security.

  • Caused by: java.lang.IllegalArgumentException: Illegal principal name user@BIGDATA.COM: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to user@BIGDATA.COM
    Possible Cause

    Possible realm conflict.

    Resolution

    Check your /etc/krb5.conf . If there are multiple realms, try setting your realm as the default one.

  • GSSException: No valid credentials provided (Mechanism level: Connection refused: connect)
    Possible Cause

    In krb5.ini, the KDC hostname is incorrect or the KDC daemon is not started on this server.

    Resolution

    Check for the daemon status or ask to verify the right KDC.

  • GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt
    Possible Cause

    User who executes the job has no valid ticket in his/her cache.

    Resolution

    Run kinit where the driver will run (on localhost if from studio, on the server where the JobServer lies if on remote).

  • java.io.IOException: java.lang.IllegalArgumentException: Server has invalid Kerberos principal: nn/sandbox.hadoop.com@EXAMPLE.COM; Host Details : local host is: "ServerName1/127.0.1.1";
    Possible Cause

    The server name is not defined in /etc/hosts .

    Resolution

    Check etc/hosts on the client machine and make sure you have the same hostname.

    Or

    Replace _HOST by the server's complete hostname.

  • java.io.IOException: Login failure for user1 from keytab C:/Users/user1.DOMAIN1/Documents/sko/user1.keytab Caused by: javax.security.auth.login.LoginException: Checksum failed
    Possible Cause

    The principal hash does not correspond to the hash the keytab knows. It can happen when the principal was once deleted and then recreated in the KDC database.

    Resolution

    Renew the obsolete keytab.

  • java.io.IOException: Login failure for user1 from keytab C:/Users/user1.DOMAIN/Documents/sko/user1.keytab Caused by: KrbException: Client not found in Kerberos database (devil) - CLIENT_NOT_FOUND
    Possible Cause

    The keytab correctly contains the user principal but KDC does not know this principal anymore.

    Resolution

    Renew the obsolete keytab.

  • java.io.IOException: Login failure for user1 from keytab C:/Users/user1.DOMAIN/Documents/sko/user1.keytab Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
    Possible Cause

    The keytab used does not contain any credentials corresponding to the specified principal or the keytab is not readable for the current user.

    Resolution

    Check the username specified in the Talend components, if ok, check the keytab privileges (read access) and validity.

  • kinit: Client not found in Kerberos database while getting initial credentials
    Possible Cause

    The user has no principal in the KDC database.

    Resolution

    Create the principal or use the right one (via kadmin or kadmin.local).

  • krb5_get_init_creds: unable to reach any KDC in realm EXAMPLE.COM, tried 1 KDC
    Possible Cause

    The KDC server is configured to use only UDP or TCP and not both, as supposed by your krb5.conf .

    Resolution

    Try to force the protocol in the krb5.conf by adding a line:

    kdc = tcp/<kdc_server_hostname>:88
  • org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered coprocessor service found for name AuthenticationService in region hbase:meta,,1
    Possible Cause

    The HBase server side configurations for the coprocessor security are missing.

    Resolution

    Add the following to hbase-site.xml :

    <property> 
    	<name>hbase.coprocessor.region.classes</name> 
    	<value>org.apache.hadoop.hbase.security.token.TokenProvider,
    		   org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,
    		   org.apache.hadoop.hbase.security.access.AccessController </value> 
    </property>
    <property> 
    	<name>hbase.rpc.engine</name> 
    	<value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value> 
    </property> 
    <property> 
    	<name>hbase.coprocessor.master.classes</name> 
    	<value>org.apache.hadoop.hbase.security.access.AccessController</value> 
    </property>
  • org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): KERBEROS authentication is not enabled. Available:[SIMPLE]
    Possible Cause

    The Studio is configured to run a Job targeting a kerberized cluster whereas the server is not configured to use Kerberos.

    Resolution

    Remove the Kerberos credentials and configure the Job to access the cluster using simple authentication (user-based).

  • org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]
    Possible Cause

    Studio tries to connect to a non-kerberized cluster whereas this is a kerberized environment.

    Resolution

    Activate the Kerberos credentials in the Job / Talend components.

  • org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
    Possible Cause

    Due to a bug of the studio for MapReduce Jobs, this error is happening when you activated Kerberos on a M/R Job but a username was previously set and the Studio still use it to access the cluster without taking Kerberos credentials into consideration.

    Resolution

    Deactivate the Kerberos authentication, set the username to blank and reactivate the Kerberos authentication.

  • org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to hdfs@TALEND.EXAMPLE.COM
    Possible Cause

    You face a cross-realm Kerberos environment and the mapping rules to translate principal from one realm to the Hadoop realm does not match the expected result.

    Resolution

    Check that the mapping rules are specified in the core-site.xml embedded in hadoop-conf-kerberos.jar . If yes, you have to correct these rules to obtain the right mapping.

  • Server has invalid Kerberos principal: hdfs/talend-cdh5-nn1@TALEND.COM
    Possible Cause

    The realm configured in krb5.conf or the realm configured in the Job does not match the server realm.

    Resolution

    Check the Job configuration and the krb5.ini to make sure this is aligned with the target server realm.

  • [WARN ]: org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException as:user@EXAMPLE.COM (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for user@EXAMPLE.COM to hbase/talend-cdh5@EXAMPLE.COM
    Possible Cause

    The server name is not defined in /etc/hosts .

    Resolution

    Add FQDN to /etc/hosts .