Using this component in OSGi - 6.3

Talend ESB Mediation Developer Guide

EnrichVersion
6.3
EnrichProdName
Talend Data Fabric
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for ESB
Talend Real-Time Big Data Platform
task
Design and Development
EnrichPlatform
Talend ESB

There are some quirks when running this component in an OSGi environment related to the mechanism Hadoop 2.x uses to discover different org.apache.hadoop.fs.FileSystem implementations. Hadoop 2.x uses java.util.ServiceLoader which looks for /META-INF/services/org.apache.hadoop.fs.FileSystem files defining available filesystem types and implementations. These resources are not available when running inside OSGi.

As with camel-hdfs component, the default configuration files need to be visible from the bundle class loader. A typical way to deal with it is to keep a copy of core-default.xml (and e.g., hdfs-default.xml) in your bundle root.

Using this component with manually defined routes

There are two options:

  1. Package /META-INF/services/org.apache.hadoop.fs.FileSystem resource with bundle that defines the routes. This resource should list all the required Hadoop 2.x filesystem implementations.

  2. Provide boilerplate initialization code which populates internal, static cache inside org.apache.hadoop.fs.FileSystem class:

    org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
    conf.setClass("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class, FileSystem.class);
    conf.setClass("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class, FileSystem.class);
    ...
    FileSystem.get("file:///", conf);
    FileSystem.get("hdfs://localhost:9000/", conf);
    ...

Using this component with Blueprint container

Two options:

  1. Package /META-INF/services/org.apache.hadoop.fs.FileSystem resource with bundle that contains blueprint definition.

  2. Add the following to the blueprint definition file:

    <bean id="hdfsOsgiHelper" class="org.apache.camel.component.hdfs2.HdfsOsgiHelper">
       <argument>
          <map>
             <entry key="file:///" value="org.apache.hadoop.fs.LocalFileSystem"  />
             <entry key="hdfs://localhost:9000/" value="org.apache.hadoop.hdfs.DistributedFileSystem" />
             ...
          </map>
       </argument>
    </bean>
     
    <bean id="hdfs2" class="org.apache.camel.component.hdfs2.HdfsComponent" depends-on="hdfsOsgiHelper" />

This way Hadoop 2.x will have correct mapping of URI schemes to filesystem implementations.