KafkaSpout 引起的 log4j 的问题

今天在测试 KafkaSpout 的时候突然冒出了 log4j 的问题,先是两行醒目的红色警告:

SLF4J: Detected both log4j-over-slf4j.jar AND slf4j-log4j12.jar on the class path, preempting StackOverflowError. 
SLF4J: See also http://www.slf4j.org/codes.html#log4jDelegationLoop for more details.

跟着 topology 就挂掉了,只留下了一堆无奈的日志:

5370 [Thread-14-newKafka] ERROR backtype.storm.util - Async loop died!
java.lang.NoClassDefFoundError: Could not initialize class org.apache.log4j.Log4jLoggerFactory
	at org.apache.log4j.Logger.getLogger(Logger.java:39) ~[log4j-over-slf4j-1.6.6.jar:1.6.6]
	at kafka.utils.Logging$class.logger(Logging.scala:24) ~[kafka_2.10-0.8.2.1.jar:na]
	at kafka.consumer.SimpleConsumer.logger$lzycompute(SimpleConsumer.scala:30) ~[kafka_2.10-0.8.2.1.jar:na]
	at kafka.consumer.SimpleConsumer.logger(SimpleConsumer.scala:30) ~[kafka_2.10-0.8.2.1.jar:na]
	at kafka.utils.Logging$class.info(Logging.scala:67) ~[kafka_2.10-0.8.2.1.jar:na]
	at kafka.consumer.SimpleConsumer.info(SimpleConsumer.scala:30) ~[kafka_2.10-0.8.2.1.jar:na]
	at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:74) ~[kafka_2.10-0.8.2.1.jar:na]
	at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:68) ~[kafka_2.10-0.8.2.1.jar:na]
	at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:127) ~[kafka_2.10-0.8.2.1.jar:na]
	at kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79) ~[kafka_2.10-0.8.2.1.jar:na]
	at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:77) ~[storm-kafka-0.9.3.jar:0.9.3]
	at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:67) ~[storm-kafka-0.9.3.jar:0.9.3]
	at storm.kafka.PartitionManager.<init>(PartitionManager.java:83) ~[storm-kafka-0.9.3.jar:0.9.3]
	at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98) ~[storm-kafka-0.9.3.jar:0.9.3]
	at storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) ~[storm-kafka-0.9.3.jar:0.9.3]
	at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:135) ~[storm-kafka-0.9.3.jar:0.9.3]
	at backtype.storm.daemon.executor$fn__3373$fn__3388$fn__3417.invoke(executor.clj:565) ~[storm-core-0.9.3.jar:0.9.3]
	at backtype.storm.util$async_loop$fn__464.invoke(util.clj:463) ~[storm-core-0.9.3.jar:0.9.3]
	at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
	at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]

追根溯源,发现 KafkaSpout 代码里( storm.kafka.KafkaSpout )使用了 slf4j 的包,而 Kafka 系统本身( kafka.consumer.SimpleConsumer )却使用了 apache 的包,这个结果着实有些尴尬。

折腾了一会儿,最后还是根据 http://stackoverflow.com/questions/20117720/detected-both-log4j-over-slf4j-jar-and-slf4j-log4j12-jar-on-the-class-path-pree 这个问题的提示,在依赖定义中排除问题依赖包(也就是 Kafka 本身的依赖包)中对应的冲突的包

<dependency>
	<groupId>org.apache.kafka</groupId>
	<artifactId>kafka_2.10</artifactId>
	<version>0.8.2.1</version>
	<scope>provided</scope>
	<exclusions>
		<exclusion>
			<groupId>org.slf4j</groupId>
			<artifactId>slf4j-log4j12</artifactId>
		</exclusion>
		<exclusion>
			<groupId>log4j</groupId>
			<artifactId>log4j</artifactId>
		</exclusion>
	</exclusions>
</dependency>

重新编译运行就 OK 了。

TIPS

结合这次经历和以前遇到的各种 log4j 问题(没办法,使用最广泛的包就是这么任性),总结了一下使用 log4j 的几点不算什么经验的心得。

  • 统一采用 slf4j 的包
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MyBolt {
    private static final Logger LOG = LoggerFactory
            .getLogger(MyBolt.class);
}

在代码中不再使用 apache 的包,转而统一使用 slf4j 的包,可以避免大多数包冲突问题(只适用于 Storm 相关服务,这是因为 Storm 所依赖的 logback-classic-1.0.13.jar 是使用 slf4j 的)。

  • Maven 的编译发布插件中最好都要排除冲突的包
<plugins>
	<plugin>
		<groupId>org.apache.maven.plugins</groupId>
		<artifactId>maven-compiler-plugin</artifactId>
		<version>3.1</version>
		<configuration>
			<source>1.7</source>
			<target>1.7</target>
			<excludes>
				<exclude>log4j:log4j:jar:</exclude>
			</excludes>
		</configuration>
	</plugin>
	<plugin>
		<groupId>org.apache.maven.plugins</groupId>
		<artifactId>maven-shade-plugin</artifactId>
		<executions>
			<execution>
				<phase>package</phase>
				<goals>
					<goal>shade</goal>
				</goals>
			</execution>
		</executions>
		<configuration>
			<finalName>${project.artifactId}-${project.version}-shade</finalName>
			<artifactSet>
				<excludes>
					<exclude>log4j:log4j:jar:</exclude>
				</excludes>
			</artifactSet>
			<transformers>
				<transformer
					implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
				<transformer
					implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
					<mainClass>storm.test.topology.SimulationTopology</mainClass>
				</transformer>
			</transformers>
		</configuration>
	</plugin>
</plugins>

<exclude>log4j:log4j:jar:</exclude> 这样的操作可以避免拓扑发布到 Storm 集群之后可能出现的包冲突问题。

我来评几句
登录后评论

已发表评论数()

相关站点

热门文章