Zeppelin HBase2.x兼容问题和解决方案

问题

HBase Shell使用了Ruby脚本编写。HBase Interpreter使用了HBase的Ruby脚本，自带默认的jruby解析器对HBase 2.x中的Ruby脚本语法不兼容。

环境

  • Zeppelin 0.11.2
  • HBase 2.0.0

错误信息

table.rb问题：

org.apache.zeppelin.interpreter.InterpreterException: org.jruby.embed.EvalFailedException: (SyntaxError) /usr/hdp/3.0.1.0-187/hbase/lib/ruby/hbase/table.rb:808: syntax error, unexpected tDOT .select { |s| RegionReplicaUtil.isDefaultReplica(s.getRegion) } ^ at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:861) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:769) at org.apache.zeppelin.scheduler.Job.run(Job.java:186) at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:135) at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.jruby.embed.EvalFailedException: (SyntaxError) /usr/hdp/3.0.1.0-187/hbase/lib/ruby/hbase/table.rb:808: syntax error, unexpected tDOT .select { |s| RegionReplicaUtil.isDefaultReplica(s.getRegion) } ^ at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:136) at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1263) at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1294) at org.apache.zeppelin.hbase.HbaseInterpreter.open(HbaseInterpreter.java:97) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) ... 8 more Caused by: org.jruby.exceptions.RaiseException: (SyntaxError) /usr/hdp/3.0.1.0-187/hbase/lib/ruby/hbase/table.rb:808: syntax error, unexpected tDOT .select { |s| RegionReplicaUtil.isDefaultReplica(s.getRegion) } ^

replication_admin.rb问题

org.apache.zeppelin.interpreter.InterpreterException: org.jruby.embed.EvalFailedException: (SyntaxError) /usr/hdp/3.0.1.0-187/hbase/lib/ruby/hbase/replication_admin.rb:70: syntax error, unexpected tDOT .newBuilder() ^ at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:861) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:769) at org.apache.zeppelin.scheduler.Job.run(Job.java:186) at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:135) at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.jruby.embed.EvalFailedException: (SyntaxError) /usr/hdp/3.0.1.0-187/hbase/lib/ruby/hbase/replication_admin.rb:70: syntax error, unexpected tDOT .newBuilder() ^ at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:136) at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1263) at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1294) at org.apache.zeppelin.hbase.HbaseInterpreter.open(HbaseInterpreter.java:97) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) ... 8 more Caused by: org.jruby.exceptions.RaiseException: (SyntaxError) /usr/hdp/3.0.1.0-187/hbase/lib/ruby/hbase/replication_admin.rb:70: syntax error, unexpected tDOT .newBuilder() ^

  
org.apache.zeppelin.interpreter.InterpreterException: org.jruby.embed.EvalFailedException: (SyntaxError) /usr/hdp/3.0.1.0-187/hbase/lib/ruby/hbase/replication_admin.rb:231: syntax error, unexpected tDOT .newBuilder(rpc) ^ at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:861) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:769) at org.apache.zeppelin.scheduler.Job.run(Job.java:186) at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:135) at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.jruby.embed.EvalFailedException: (SyntaxError) /usr/hdp/3.0.1.0-187/hbase/lib/ruby/hbase/replication_admin.rb:231: syntax error, unexpected tDOT .newBuilder(rpc) ^ at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:136) at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1263) at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1294) at org.apache.zeppelin.hbase.HbaseInterpreter.open(HbaseInterpreter.java:97) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) ... 8 more Caused by: org.jruby.exceptions.RaiseException: (SyntaxError) /usr/hdp/3.0.1.0-187/hbase/lib/ruby/hbase/replication_admin.rb:231: syntax error, unexpected tDOT .newBuilder(rpc) ^

  
org.apache.zeppelin.interpreter.InterpreterException: org.jruby.embed.EvalFailedException: (SyntaxError) /usr/hdp/3.0.1.0-187/hbase/lib/ruby/hbase/replication_admin.rb:231: syntax error, unexpected tDOT .newBuilder(rpc) ^ at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:861) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:769) at org.apache.zeppelin.scheduler.Job.run(Job.java:186) at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:135) at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.jruby.embed.EvalFailedException: (SyntaxError) /usr/hdp/3.0.1.0-187/hbase/lib/ruby/hbase/replication_admin.rb:231: syntax error, unexpected tDOT .newBuilder(rpc) ^ at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:136) at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1263) at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1294) at org.apache.zeppelin.hbase.HbaseInterpreter.open(HbaseInterpreter.java:97) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) ... 8 more Caused by: org.jruby.exceptions.RaiseException: (SyntaxError) /usr/hdp/3.0.1.0-187/hbase/lib/ruby/hbase/replication_admin.rb:231: syntax error, unexpected tDOT .newBuilder(rpc) ^

org.apache.zeppelin.interpreter.InterpreterException: org.jruby.embed.EvalFailedException: (SyntaxError) /usr/hdp/3.0.1.0-187/hbase/lib/ruby/hbase/replication_admin.rb:362: syntax error, unexpected tDOT .newBuilder(replication_peer_config) ^ at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:861) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:769) at org.apache.zeppelin.scheduler.Job.run(Job.java:186) at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:135) at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.jruby.embed.EvalFailedException: (SyntaxError) /usr/hdp/3.0.1.0-187/hbase/lib/ruby/hbase/replication_admin.rb:362: syntax error, unexpected tDOT .newBuilder(replication_peer_config) ^ at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:136) at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1263) at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1294) at org.apache.zeppelin.hbase.HbaseInterpreter.open(HbaseInterpreter.java:97) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) ... 8 more Caused by: org.jruby.exceptions.RaiseException: (SyntaxError) /usr/hdp/3.0.1.0-187/hbase/lib/ruby/hbase/replication_admin.rb:362: syntax error, unexpected tDOT .newBuilder(replication_peer_config) ^

shell.rb问题：

org.apache.zeppelin.interpreter.InterpreterException: org.jruby.embed.EvalFailedException: (SyntaxError) /usr/hdp/3.0.1.0-187/hbase/lib/ruby/shell.rb:56: syntax error, unexpected ':' commands: [], ^ at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:861) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:769) at org.apache.zeppelin.scheduler.Job.run(Job.java:186) at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:135) at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.jruby.embed.EvalFailedException: (SyntaxError) /usr/hdp/3.0.1.0-187/hbase/lib/ruby/shell.rb:56: syntax error, unexpected ':' commands: [], ^ at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:136) at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1263) at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1294) at org.apache.zeppelin.hbase.HbaseInterpreter.open(HbaseInterpreter.java:97) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) ... 8 more Caused by: org.jruby.exceptions.RaiseException: (SyntaxError) /usr/hdp/3.0.1.0-187/hbase/lib/ruby/shell.rb:56: syntax error, unexpected ':' commands: [], ^

解决方式

重新编译Interperter

升级Zeppelin HBase interpreter的jruby版本。修改hbase/pom.xml文件，将jruby版本从1.6.8升级为9.1.17.0。类似如下：

<properties>
    <!--library versions-->
    <interpreter.name>hbase</interpreter.name>
    <hbase.hbase.version>2.4.12</hbase.hbase.version>
    <hbase.hadoop.version>${hadoop2.7.version}</hbase.hadoop.version>
    <jruby.version>9.1.17.0</jruby.version>
    <protobuf.version>2.5.0</protobuf.version>
    <jline.version>2.12.1</jline.version>
  </properties>

注意：尝试最高可升级到如下版本（同时升级了jruby和jline）：

  <properties>
    <!--library versions-->
    <interpreter.name>hbase</interpreter.name>
    <hbase.hbase.version>2.4.12</hbase.hbase.version>
    <hbase.hadoop.version>${hadoop2.7.version}</hbase.hadoop.version>
    <jruby.version>9.2.13.0</jruby.version>
    <protobuf.version>2.5.0</protobuf.version>
    <jline.version>2.14.6</jline.version>
  </properties>

如果使用更加新的jruby 9.3.13.0出现如下问题：

org.apache.zeppelin.interpreter.InterpreterException: org.jruby.embed.EvalFailedException: (NoMethodError) undefined method `runtime' for JRuby:Module
    at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76)
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:861)
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:769)
    at org.apache.zeppelin.scheduler.Job.run(Job.java:186)
    at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:135)
    at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
Caused by: org.jruby.embed.EvalFailedException: (NoMethodError) undefined method `runtime' for JRuby:Module
    at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:134)
    at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1296)
    at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1327)
    at org.apache.zeppelin.hbase.HbaseInterpreter.open(HbaseInterpreter.java:97)
    at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
    ... 8 more
Caused by: org.jruby.exceptions.NoMethodError: (NoMethodError) undefined method `runtime' for JRuby:Module
    at RUBY.`(uri:classloader:/jruby/kernel/jruby/process_manager.rb:10)
    at RUBY.`(uri:classloader:/jruby/kernel/jruby/process_manager.rb:46)
    at RUBY.initialize(/usr/hdp/3.0.1.0-187/hbase/lib/ruby/irb/hirb.rb:46)
    at org.jruby.RubyClass.new(org/jruby/RubyClass.java:883)
    at RUBY.start(hirb.rb:181)
    at RUBY.<main>(hirb.rb:193)

经过查询社区资料：
[Q] about require 'jruby'. · Issue #7221 · jruby/jruby

NoMethodError: undefined method 'runtime' for JRuby:Module with JRuby 9.3.13.0 but not 9.3.9.0 · Issue #8038 · jruby/jruby

原因是新版本的jruby要求require 'jruby'：

JRuby 9.3.0.0 need to require "jruby" to avoid undefined method 'runtime' for JRuby:Module.

可以通过如下方式验证：修改HBase 2.0.0安装目录的bin/hirb.rb，在头部增加require 'jruby'，重启HBase interpreter之后上述错误消失，可以成功运行。

jruby和jline升级完毕后，执行下方命令重新编译HBase Interpreter。

mvn clean package -amd -pl org.apache.zeppelin:zeppelin-hbase -e

编译之后生成的fat jar位于interpreter/hbase/zeppelin-hbase-0.11.2.jar。将其覆盖到Zeppelin安装目录的interpreter/hbase/zeppelin-hbase-0.11.2.jar。

配置Zeppelin env

不要忘记在zeppelin-env.sh配置HBASE_HOME。否则Zeppelin无法找到hbase-site.xml，无法读取HBase的配置。

配置HBase Interperter

登录后点击右侧菜单，选择Interperter。然后搜索到hbase。点击右侧的edit按钮。

配置hbase.home为HBase的安装目录。

在下方Dependencies的Artifact中增加必须的依赖。如果zeppelin服务节点同时安装了HBase和Hadoop，可以使用本地依赖。

  • /usr/hdp/3.0.1.0-187/hbase/lib/hbase-common-2.0.0.3.0.1.0-187.jar
  • /usr/hdp/3.0.1.0-187/hbase/lib/hbase-client-2.0.0.3.0.1.0-187.jar
  • /usr/hdp/3.0.1.0-187/hbase/lib/hbase-protocol-2.0.0.3.0.1.0-187.jar
  • /usr/hdp/3.0.1.0-187/hadoop/hadoop-common-3.1.1.3.0.1.0-187.jar
  • /usr/hdp/3.0.1.0-187/hadoop/lib/woodstox-core-5.0.3.jar
  • /usr/hdp/3.0.1.0-187/hadoop/lib/stax2-api-3.1.4.jar
  • /usr/hdp/3.0.1.0-187/hadoop/lib/guava-11.0.2.jar
  • /usr/hdp/3.0.1.0-187/hadoop/lib/commons-collections-3.2.2.jar
  • /usr/hdp/3.0.1.0-187/hadoop/lib/jackson-core-2.9.5.jar
  • /usr/hdp/3.0.1.0-187/hbase/lib/hbase-server-2.0.0.3.0.1.0-187.jar
  • /usr/hdp/3.0.1.0-187/hadoop/lib/commons-configuration2-2.1.1.jar
  • /usr/hdp/3.0.1.0-187/hadoop/lib/commons-lang-2.6.jar

完美解决方式

上面的方式仍然使用Interpreter自带的Jruby来调用HBase，和jruby解析器的版本强关联，无法保证以后HBase版本升级之后仍可继续兼容，且升级后已知无法兼容老的HBase 1.x。一句话，无法做到HBase全版本兼容。

博主为了解决这个问题，决定不再使用HBase Interpreter自带的Jruby解析器，将HBase Interpreter设计为使用HBase shell命令来执行用户脚本的方案。大致思路为将用户脚本作为临时文件保存到主机上，然后使用hbase shell 临时文件.txt的方式执行脚本，最后将脚本的执行输出展示在Zeppelin界面上。

对应的PR为：[ZEPPELIN-6163] HBase interpreter supports hbase-2.x by paul8263 · Pull Request #4908 · apache/zeppelin

附录

Zeppelin完整重新编译打包命令为：

./mvnw clean package -Pbuild-distr -Pspark-3.3 -Pspark-scala-2.12 -Pflink-117 -DskipTests

Ruby查看所有已加载特性的方式：

$LOADED_FEATURES.each { |str| puts str }
最后编辑于 ：
©著作权归作者所有,转载或内容合作请联系作者
平台声明：文章内容（如有图片或视频亦包括在内）由作者上传并发布，文章内容仅代表作者本人观点，简书系信息发布平台，仅提供信息存储服务。