监控工具集锦

work

监控工具：ZooKeeper，Kafka，Cassandra…

ZooKeeper

node-zk-browser

依赖条件：nodejs, npm

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ git clone https://github.com/killme2008/node-zk-browser.git
$ cd node-zk-browser
$ cat package.json
{
  "name": "node-zk-browser",
  "version": "0.0.2",
  "dependencies": {
    "ejs": "0.7.2",
    "express": "3.2.6",
    "zookeeper":"*",
    "express-namespace":"0.1.1"
  }
}
$ npm install -d
$ node app.js
Express server listening on port 3000
zk session established, id=154a418d5dd000b

http://localhost:3000 admin:admin

node-zk

zkui

1
2
3
4
$ git clone https://github.com/DeemOpen/zkui.git
$ mvn clean install
$ cp target/zkui-2.0-SNAPSHOT-jar-with-dependencies.jar .
$ java -jar zkui-2.0-SNAPSHOT-jar-with-dependencies.jar

http://localhost:9090/ admin:manager

zkui

zk-web

依赖条件：lein

1
2
3
4
$ git clone https://github.com/qiuxiafei/zk-web
$ cd zk-web
$ lein deps
$ lein run

http://localhost:9090/ admin:hello

zkweb

Kafka

trifecta

下载https://github.com/ldaniels528/trifecta/releases中的zip包（web ui）， jar是命令行工具（trifecta_cli）。

依赖条件：scalascript

编译并发布scalascript

需要先将scalascript发布到本地仓库，默认sbt会发布到ivy下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
➜  git clone https://github.com/ldaniels528/scalascript.git
➜  cd scalascript 

修改sbt版本为当前安装的sbt版本，否则如果没有修改，会去下载，尽量使用已有的
➜  cat project/build.properties  
sbt.version=0.13.9
➜  sbt publish-local  
Getting org.scala-sbt sbt 0.13.9 ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt/0.13.9/jars/sbt.jar ...
    [SUCCESSFUL ] org.scala-sbt#sbt;0.13.9!sbt.jar (5084ms)

使用0.13.11就不会下载了
➜  sbt publish-local
[info] Loading global plugins from /Users/zhengqh/.sbt/0.13/plugins
[info] Loading project definition from /Users/zhengqh/Soft/scalascript/project
[info] Packaging /Users/zhengqh/Soft/scalascript/target/scala-2.11/scalascript_sjs0.6_2.11-0.2.20.jar ...
[info] Done packaging.
[info]  published scalascript_sjs0.6_2.11 to /Users/zhengqh/.ivy2/local/com.github.ldaniels528/scalascript_sjs0.6_2.11/0.2.20/poms/scalascript_sjs0.6_2.11.pom
[info]  published scalascript_sjs0.6_2.11 to /Users/zhengqh/.ivy2/local/com.github.ldaniels528/scalascript_sjs0.6_2.11/0.2.20/jars/scalascript_sjs0.6_2.11.jar
[info]  published scalascript_sjs0.6_2.11 to /Users/zhengqh/.ivy2/local/com.github.ldaniels528/scalascript_sjs0.6_2.11/0.2.20/srcs/scalascript_sjs0.6_2.11-sources.jar
[info]  published scalascript_sjs0.6_2.11 to /Users/zhengqh/.ivy2/local/com.github.ldaniels528/scalascript_sjs0.6_2.11/0.2.20/docs/scalascript_sjs0.6_2.11-javadoc.jar
[info]  published ivy to /Users/zhengqh/.ivy2/local/com.github.ldaniels528/scalascript_sjs0.6_2.11/0.2.20/ivys/ivy.xml
[success] Total time: 71 s, completed 2016-5-16 10:11:20

编译并发布trifecta

1
2
3
4
5
6
7
8
9
➜  git clone https://github.com/ldaniels528/trifecta.git
➜  cd trifecta
➜  sbt publish-local
[info]  published trifecta_cli_2.11 to /Users/zhengqh/.ivy2/local/com.github.ldaniels528/trifecta_cli_2.11/0.20.0/poms/trifecta_cli_2.11.pom
[info]  published trifecta_cli_2.11 to /Users/zhengqh/.ivy2/local/com.github.ldaniels528/trifecta_cli_2.11/0.20.0/jars/trifecta_cli_2.11.jar
[info]  published trifecta_cli_2.11 to /Users/zhengqh/.ivy2/local/com.github.ldaniels528/trifecta_cli_2.11/0.20.0/srcs/trifecta_cli_2.11-sources.jar
[info]  published trifecta_cli_2.11 to /Users/zhengqh/.ivy2/local/com.github.ldaniels528/trifecta_cli_2.11/0.20.0/docs/trifecta_cli_2.11-javadoc.jar
[info]  published ivy to /Users/zhengqh/.ivy2/local/com.github.ldaniels528/trifecta_cli_2.11/0.20.0/ivys/ivy.xml
[success] Total time: 75 s, completed 2016-5-16 9:51:09

现在已经编译好了trifecta_cli_2.11，这和release的cli.jar是一样的。 不过我们想要编译ui的话：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
➜  sbt "project trifecta_ui" dist
[info] Loading global plugins from /Users/zhengqh/.sbt/0.13/plugins
[info] Loading project definition from /Users/zhengqh/Soft/trifecta/project
[info] Set current project to trifecta_core (in build file:/Users/zhengqh/Soft/trifecta/)
[info] Set current project to trifecta_cli (in build file:/Users/zhengqh/Soft/trifecta/)
[info] Set current project to trifecta_ui (in build file:/Users/zhengqh/Soft/trifecta/)
...
[info] Done packaging.
[info]
[info] Your package is ready in /Users/zhengqh/Soft/trifecta/app-play/target/universal/trifecta_ui-0.20.0.zip
[info]
[success] Total time: 109 s, completed 2016-5-16 10:15:01

➜  unzip /Users/zhengqh/Soft/trifecta/app-play/target/universal/trifecta_ui-0.20.0.zip
Archive:  /Users/zhengqh/Soft/trifecta/app-play/target/universal/trifecta_ui-0.20.0.zip
➜  cd trifecta_ui-0.20.0/bin
➜  ./trifecta_ui
[info] p.a.l.c.ActorSystemProvider - Starting application default Akka system: application
[info] application - Application has started
[info] play.api.Play$ - Application started (Prod)
[info] p.c.s.NettyServer$ - Listening for HTTP on /0:0:0:0:0:0:0:0:9000
[info] application - Loading Trifecta configuration...
[info] application - Starting Zookeeper client...
[info] application - created actor com.github.ldaniels528.trifecta.actors.SSEClientHandlingActor
[info] application - Registering new SSE session '8d82b54dc125406f875c98e9f05e4a4f'...
[info] application - GET /api/sse/connect ~> 200 [927.0 ms]

http://localhost:9000

trifecta

Kafka Offset Monitor(https://github.com/quantifind/KafkaOffsetMonitor)

Kafka Monitor()

Cassandra

Riemman

准备工作

1
2
3
4
rvm install 1.9.3 --with-gcc=clang
rvm 1.9.3 --default
rvm list
ruby -v
1
2
3
4
5
6
7
8
9
10
11
wget https://aphyr.com/riemann/riemann-0.2.11.tar.bz2
tar xvfj riemann-0.2.11.tar.bz2
cd riemann-0.2.11

➜  riemann-0.2.11 bin/riemann etc/riemann.config
INFO [2016-05-19 11:50:49,069] main - riemann.bin - PID 70661
INFO [2016-05-19 11:50:49,078] main - riemann.bin - Loading /Users/zhengqh/Soft/riemann-0.2.11/etc/riemann.config
INFO [2016-05-19 11:50:49,334] clojure-agent-send-off-pool-1 - riemann.transport.websockets - Websockets server 127.0.0.1 5556 online
INFO [2016-05-19 11:50:49,629] clojure-agent-send-off-pool-2 - riemann.transport.udp - UDP server 127.0.0.1 5555 16384 -1 online
INFO [2016-05-19 11:50:49,656] clojure-agent-send-off-pool-3 - riemann.transport.tcp - TCP server 127.0.0.1 5555 online
INFO [2016-05-19 11:50:49,659] main - riemann.core - Hyperspace core online
1
2
3
4
5
6
7
8
9
10
11
12
gem install riemann-client riemann-tools riemann-dash

➜  ~ riemann-dash
DEPRECATION WARNING:
Sass 3.5 will no longer support Ruby 1.9.3.
Please upgrade to Ruby 2.0.0 or greater as soon as possible.

No configuration loaded; using defaults.
[2016-05-19 12:12:35] INFO  WEBrick 1.3.1
[2016-05-19 12:12:35] INFO  ruby 1.9.3 (2014-11-13) [x86_64-darwin14.1.1]
== Sinatra (v1.4.7) has taken the stage on 4567 for development with backup from WEBrick
[2016-05-19 12:12:35] INFO  WEBrick::HTTPServer#start: pid=20843 port=4567

Mac下command并单击某个区域，这部分会变成深灰色

rieman1

按快捷键e，出现编辑页面

rieman2

点击Apply后，按Esc键，会退出编辑状态，深灰色变为正常颜色

rieman3

1
➜  ~ riemann-health

打包：

1
2
3
➜  riemann-cassandra git:(master) ✗ mvn package
[INFO] Replacing /Users/zhengqh/Github/riemann-cassandra/target/riemann-cassandra-0.0.3.jar with 
/Users/zhengqh/Github/riemann-cassandra/target/riemann-cassandra-0.0.3-shaded.jar

启动客户端：

1
2
3
4
5
6
7
8
9
10
11
java -jar target/riemann-cassandra-0.0.3.jar \
    -riemann_host localhost \
    -riemann_port 5555 \
    -cassandra_host localhost \
    -jmx_port 7199 \
    -jmx_username null \
    -jmx_password null \
    -interval_seconds 5

success start riemann-cassandra-client............@Thu May 19 14:50:04 CST 2016
^Cclosed riemann-cassandra-client@Thu May 19 16:14:28 CST 2016

Event的格式：

1
2
3
4
5
6
7
8
{ 
    host: "www1", 
    service: "http req", 
    metric: 2.53, 
    state: "critical", 
    description: "Request took 2.53 seconds.", 
    tags: ["http"]
}

对应的riemann后台日志：

1
2
INFO [2016-05-19 14:01:46,512] Thread-7 - riemann.config - expired 
{:host www1, :service http req, :state expired, :time 365909426627/250, :ttl 60}

riemann-cassandra中的事件格式：

1
2
3
4
5
6
{ 
    host: "localhost", 
    service: "cassandra.db.forseti.velocity_global.max_row_size_ob", 
    metric: 148,
    tags: ["cassandra"]
}

riemann服务端后台日志：

1
2
INFO [2016-05-19 16:47:54,195] Thread-7 - riemann.config - expired 
{:host 10.57.2.36, :service cassandra.db.forseti.velocity_partner.mean_row_size_kb, :state expired, :time 292729534839/200, :ttl 60}

查询所有表的read_latency, query中支持不同的查询

1
tagged "cassandra"  and service =~ "cassandra.db.forseti.%.read_%"

Tracing

zipkin

准备环境

1
2
3
4
5
6
alias npm="npm --registry=https://registry.npm.taobao.org --disturl=https://npm.taobao.org/mirrors/node"
yum install npm -y
git clone https://github.com/creationix/nvm.git
source nvm/install.sh
node -v && npm -v && nvm -v
nvm install v5.5.0
1
2
3
$ tar zxf zipkin-1.40.1.tar.gz
$ cd zipkin-1.40.1
$ bin/query

vi zipkin-ui/webpack.config.js

1
2
3
4
5
6
7
8
9
10
11
12
13
:zipkin-ui:npmBuild

> zipkin-ui@0.0.0 build /home/qihuang.zheng/zipkin-1.40.1/zipkin-ui
> node node_modules/webpack/bin/webpack.js --bail

ModuleBuildError: Module build failed: Error: /home/qihuang.zheng/zipkin-1.40.1/zipkin-ui/node_modules/node-sass/vendor/linux-x64-47/binding.node: ELF load command past end of file
npm ERR! Linux 3.10.96-1.el6.elrepo.x86_64
npm ERR! argv "/home/qihuang.zheng/zipkin-1.40.1/zipkin-ui/build/nodejs/node-v5.5.0-linux-x64/bin/node" "/home/qihuang.zheng/zipkin-1.40.1/zipkin-ui/build/nodejs/node-v5.5.0-linux-x64/lib/node_modules/npm/bin/npm-cli.js" "run-script" "build"
npm ERR! node v5.5.0
npm ERR! npm  v3.3.12
npm ERR! code ELIFECYCLE
npm ERR! zipkin-ui@0.0.0 build: `node node_modules/webpack/bin/webpack.js --bail`
npm ERR! Exit status 1

cat

HTrace

Operation

FrameGraph

git clone https://github.com/brendangregg/FlameGraph
cd FlameGraph
sudo perf record –call-graph dwarf -p $(pgrep scylla)
sudo perf script | pwd/stackcollapse-perf.pl | pwd/flamegraph.pl > flame.svg

http://blog.csdn.net/justlinux2010/article/details/11520829

本文标题:监控工具集锦

文章作者:任何忧伤,都抵不过世界的美丽

发布时间:2016年12月31日 - 00时00分

最后更新:2019年02月14日 - 21时42分

原始链接:http://github.com/zqhxuyuan/2016/12/31/Tools-BigData-Monitor/

许可协议: "署名-非商用-相同方式共享 3.0" 转载请保留原文链接及作者。


文章目录
  1. 1. ZooKeeper
    1. 1.1. node-zk-browser
    2. 1.2. zkui
    3. 1.3. zk-web
  2. 2. Kafka
    1. 2.1. trifecta
    2. 2.2. Kafka Offset Monitor(https://github.com/quantifind/KafkaOffsetMonitor)
    3. 2.3. Kafka Monitor()
  3. 3. Cassandra
    1. 3.1. Riemman
      1. 3.1.1. 准备工作
  4. 4. Tracing
    1. 4.1. zipkin
    2. 4.2. cat
    3. 4.3. HTrace
  5. 5. Operation
    1. 5.1. FrameGraph