Kafka Consumer Plugin Integration with PDI 6.1

Configuration of Kafka consumer Plugin for PDI (Pentaho data integration)

Issue Background:

Kafka consumer plugin does not work out of the box when they get integrated with PDI Spoon plugins however the Kafka Producer works out of the box.So many would be wondered how to fix this so here we go with some workaround 

Tested Environment:

• ·         PDI  6.1 EE or CE
• ·         pentaho-kafka-producer-v1.6.zip
• ·         pentaho-kafka-consumer-v1.4.zip

Producer and Consumer Installation

·         Download the latest Kafka Consumer & Producer from GitHub - https://github.com/RuckusWirelessIL

·          where ever you have your  PDI installed  extract the Plugins to the appropriate directory

Example:

C:\pentaho\pdi6.1\data-integration\plugins\steps or to your matching Linux path

Note: if you don’t find the steps folder under  data-integration\plugins  please  create a new folder  named – “steps”

Open up the C:\pentaho\pdi6.1\data-integration\plugins\steps\pentaho-kafka-consumer\lib and check if there are 17 jar files listed out as below

Procedure to Copy & Update  Library Files

• ·         compare *.jar files from    C:\pentaho \pdi6.1\data-integration\plugins\steps\pentaho-kafka-consumer\lib  against   pdi6.1\data-integration\lib.

·         

Perform the file copy & replace operation as per the below table instruction.

·         Edit plugin .xml under \pdi6.1\data-integration\plugins\steps\pentaho-kafka-consumer and update the version number values  matching to jar  files under  \data-integration\lib .

Sample – Untouched Original File-   Plugin.xml

                <libraries>

                                <library name="pentaho-kafka-consumer.jar"/>

                                <library name="lib/jline-0.9.94.jar"/>

                                <library name="lib/jopt-simple-3.2.jar"/>

                                <library name="lib/junit-3.8.1.jar"/>

                                <library name="lib/kafka_2.10-0.8.2.1.jar"/>

                                <library name="lib/kafka-clients-0.8.2.1.jar"/>

                                <library name="lib/log4j-1.2.16.jar"/>

                                <library name="lib/lz4-1.2.0.jar"/>

                                <library name="lib/metrics-core-2.2.0.jar"/>

                                <library name="lib/netty-3.7.0.Final.jar"/>

                                <library name="lib/scala-library-2.10.4.jar"/>

                                <library name="lib/slf4j-api-1.7.2.jar"/>

                                <library name="lib/slf4j-log4j12-1.6.1.jar"/>

                                <library name="lib/snappy-java-1.1.1.6.jar"/>

                                <library name="lib/xercesImpl-2.9.1.jar"/>

                                <library name="lib/xml-apis-1.3.04.jar"/>

                                <library name="lib/zkclient-0.3.jar"/>

                                <library name="lib/zookeeper-3.4.6.jar"/>

                </libraries>

Sample   - modified Plugin.XML

<libraries>

                                <library name="pentaho-kafka-consumer.jar"/>

                                <library name="lib/jline-0.9.94.jar"/>

                                <library name="lib/jopt-simple-3.2.jar"/>

                                <library name="lib/junit-3.8.1.jar"/>

                                <library name="lib/kafka_2.10-0.8.2.1.jar"/>

                                <library name="lib/kafka-clients-0.8.2.1.jar"/>

                                <library name="lib/log4j-1.2.17.jar"/>  

                                <library name="lib/lz4-1.2.0.jar"/>

                                <library name="lib/metrics-core-2.2.0.jar"/>

                                <library name="lib/netty-3.7.0.Final.jar"/>

                                <library name="lib/scala-library-2.10.4.jar"/>

                                <library name="lib/slf4j-api-1.7.7.jar"/>

                                <library name="lib/slf4j-log4j12-1.7.7.jar"/>

                                <library name="lib/snappy-java-1.1.0.jar"/>

                                <library name="lib/xercesImpl-2.9.1.jar"/>

                                <library name="lib/xml-apis-2.0.2.jar"/> 

                                <library name="lib/zkclient-0.3.jar"/>

                                <library name="lib/zookeeper-3.4.6.jar"/>

                </libraries>

·         Restart Spoon and test the producer and consumer.