zookeeper

概览

单机安装

###################################################### 安装jdk ######################################################
core-services >> jdk.md

ZooKeeper runs in Java, release 1.8 or greater (JDK 8 LTS, JDK 11 LTS, JDK 12 - Java 9 and 10 are not supported). It runs as an ensemble of ZooKeeper servers. Three ZooKeeper servers is the minimum recommended size for an ensemble, and we also recommend that they run on separate machines. 

###################################################### 下载并解压二进制文件 ###########################################
mkdir -p /xdata && cd /xdata
wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
OR http://192.168.1.8/chfs/shared/linux-pkg/zookeeper-3.4.14.tar.gz
tar xf zookeeper-3.4.14.tar.gz
cd /xdata/zookeeper-3.4.14

######################################################### 配置 ######################################################
egrep -v "^$|^#" conf/zoo_sample.cfg &> conf/zoo.cfg
cat conf/zoo.cfg 
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/lib/zookeeper
clientPort=2181

mkdir -p /var/lib/zookeeper

###################################################### 启动与检查 ######################################################
bin/zkServer.sh start
bin/zkServer.sh
bin/zkServer.sh status

################################################ 连接上zookeeper验证 ######################################################
bin/zkCli.sh -server 127.0.0.1:2181
上面的命令即可连上zookeeper交互终端,可以输入一些zookeeper提供的命令,常用的命令如下
help
quit
ls /
创建与查看节点数据,命令参考如下
create /zk_test my_data
ls /
get /zk_test
修改节点数据
set /zk_test junk
get /zk_test
删除节点数据
delete /zk_test
递归删除 zookeeper-shell zk1:2181 rmr  /brokers/topics/v2_dptask_62.97_moka_production.interviewer_feedbacks.143536

######################################################### 启动,停止,清空环境 ######################################################
bin/zkServer.sh start
bin/zkServer.sh stop
bin/zkServer.sh status
rm -rf /var/lib/zookeeper

######################################################### 冗余信息相关 #############################################################
[root@node4 zookeeper-3.4.14]# bin/zkServer.sh
ZooKeeper JMX enabled by default
Using config: /xdata/zookeeper-3.4.14/bin/../conf/zoo.cfg
Usage: bin/zkServer.sh {start|start-foreground|stop|restart|status|upgrade|print-cmd}


[root@node4 zookeeper-3.4.14]# bin/zkCli.sh -server 127.0.0.1:2181
Connecting to 127.0.0.1:2181
2020-05-25 14:43:57,302 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
2020-05-25 14:43:57,304 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=bogon
2020-05-25 14:43:57,304 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_231
2020-05-25 14:43:57,306 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2020-05-25 14:43:57,306 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/as4k/jdk1.8.0_231/jre
2020-05-25 14:43:57,306 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/xdata/zookeeper-3.4.14/bin/../zookeeper-server/target/classes:/xdata/zookeeper-3.4.14/bin/../build/classes:/xdata/zookeeper-3.4.14/bin/../zookeeper-server/target/lib/*.jar:/xdata/zookeeper-3.4.14/bin/../build/lib/*.jar:/xdata/zookeeper-3.4.14/bin/../lib/slf4j-log4j12-1.7.25.jar:/xdata/zookeeper-3.4.14/bin/../lib/slf4j-api-1.7.25.jar:/xdata/zookeeper-3.4.14/bin/../lib/netty-3.10.6.Final.jar:/xdata/zookeeper-3.4.14/bin/../lib/log4j-1.2.17.jar:/xdata/zookeeper-3.4.14/bin/../lib/jline-0.9.94.jar:/xdata/zookeeper-3.4.14/bin/../lib/audience-annotations-0.5.0.jar:/xdata/zookeeper-3.4.14/bin/../zookeeper-3.4.14.jar:/xdata/zookeeper-3.4.14/bin/../zookeeper-server/src/main/resources/lib/*.jar:/xdata/zookeeper-3.4.14/bin/../conf:/as4k/jdk1.8.0_231/lib:/as4k/jdk1.8.0_231/jre/lib:
2020-05-25 14:43:57,306 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2020-05-25 14:43:57,314 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2020-05-25 14:43:57,314 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2020-05-25 14:43:57,314 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2020-05-25 14:43:57,314 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2020-05-25 14:43:57,314 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-957.el7.x86_64
2020-05-25 14:43:57,314 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
2020-05-25 14:43:57,314 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
2020-05-25 14:43:57,314 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/xdata/zookeeper-3.4.14
2020-05-25 14:43:57,315 [myid:] - INFO  [main:ZooKeeper@442] - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5ce65a89
2020-05-25 14:43:57,338 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1025] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
Welcome to ZooKeeper!
JLine support is enabled
2020-05-25 14:43:57,449 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@879] - Socket connection established to localhost/127.0.0.1:2181, initiating session
2020-05-25 14:43:57,473 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x10000144d2b0000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: 127.0.0.1:2181(CONNECTED) 0] quit

[root@node4 zookeeper-3.4.14]# netstat -lntup | grep 2181
tcp6       0      0 :::2181                 :::*                    LISTEN      3783/java           

ZooKeeper -server host:port cmd args
	stat path [watch]
	set path data [version]
	ls path [watch]
	delquota [-n|-b] path
	ls2 path [watch]
	setAcl path acl
	setquota -n|-b val path
	history 
	redo cmdno
	printwatches on|off
	delete path [version]
	sync path
	listquota path
	rmr path
	get path [watch]
	create [-s] [-e] path data acl
	addauth scheme auth
	quit 
	getAcl path
	close 
	connect host:port

集群安装

集群环境可以提供高可用,奇数机器,多数原则,3台机器可以允许1台挂掉,下面以3台机器为例

###################################################### 机器IP信息 ######################################################
192.168.1.113 zk1
192.168.1.114 zk2
192.168.1.118 zk3

如果后面配置集群信息写IP或者别名的时候,最好用主机名,并且这个主机名可以解析成对应的IP

[root@zk3 zookeeper-3.4.14]# cat  /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.113 zk1
192.168.1.114 zk2
192.168.1.118 zk3

###################################################### 安装jdk(3台) ######################################################
core-services >> jdk.md

###################################################### 下载并解压二进制文件(3台) ###########################################
mkdir -p /xdata && cd /xdata
wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
OR http://192.168.1.8/chfs/shared/linux-pkg/zookeeper-3.4.14.tar.gz
tar xf zookeeper-3.4.14.tar.gz
cd /xdata/zookeeper-3.4.14

######################################################### 配置 ######################################################

机器1
mkdir -p /xdata/zookeeper-data; echo 1 > /xdata/zookeeper-data/myid
机器2
mkdir -p /xdata/zookeeper-data; echo 2 > /xdata/zookeeper-data/myid
机器3
mkdir -p /xdata/zookeeper-data; echo 3 > /xdata/zookeeper-data/myid

3台机器分别执行
cd /xdata/zookeeper-3.4.14
egrep -v "^$|^#" conf/zoo_sample.cfg &> conf/zoo.cfg
cat << 'EOF' > conf/zoo.cfg
tickTime=2000
dataDir=/xdata/zookeeper-data
clientPort=2181
initLimit=10
syncLimit=5
server.1=zk1:2888:3888
server.2=zk2:2888:3888
server.3=zk3:2888:3888
EOF
echo; echo; cat conf/zoo.cfg

集群参考配置 https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_zkMulitServerSetup

###################################################### 启动与检查 ######################################################
bin/zkServer.sh start
bin/zkServer.sh
bin/zkServer.sh status
jps
echo mntr | nc 127.0.0.1 2181

bin/zkCli.sh -server 127.0.0.1:2181 create /zk_test my_data
bin/zkCli.sh -server 127.0.0.1:2181 ls /

其它一些基本操作,可以参考上文的单机教程

彻底删除 kafka topic

https://www.cnblogs.com/musen/p/11475695.html

https://github.com/darrenfu/bigdata/issues/6

rmr /config/topics/<topic_name>
rmr /brokers/topics/<topic_name>
rmr /admin/delete_topics/<topic_name>

docker exec -it zk1 zookeeper-shell zk1:2181 rmr /brokers/topics/__debezium-heartbeat.dptask_1

docker exec -it zk1 zookeeper-shell zk1:2181 ls /
docker exec zk1 kafka-topics --zookeeper zk1:2181 --list  &> all-kafka-topics.txt
fgrep "marked for deletion" all-kafka-topics.txt &> marked-for-deletion-topics.txt
sed 's# - mared for deletion##g' marked-for-deletion-topics.txt  &> marked-for-deletion-topics-name.txt

cat tmp.sh
for i in `cat marked-for-deletion-topics-name.txt`; do
	echo "docker exec   zk1 zookeeper-shell zk1:2181 rmr /config/topics/$i"  >> final-del-zk-topics.sh
	echo "docker exec   zk1 zookeeper-shell zk1:2181 rmr /brokers/topics/$i"  >> final-del-zk-topics.sh
	echo "docker exec   zk1 zookeeper-shell zk1:2181 rmr /admin/delete_topics/$i"  >> final-del-zk-topics.sh
	echo >> final-del-zk-topics.sh
done

nohup bash -x final-del-zk-topics.sh &

使用Java API访问

https://mvnrepository.com/artifact/org.apache.zookeeper/zookeeper/3.4.6
https://www.cnblogs.com/swordfall/p/8672341.html

参考资料

ZooKeeper Releases
https://zookeeper.apache.org/

http://mirror.bit.edu.cn/apache/zookeeper/
http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
https://archive.apache.org/dist/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

The Four Letter Words
https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_4lw