kafka集群扩容记录

发表于 | 分类于 | 阅读量 次
字数统计 1,236 | 阅读时长 7

kafka集群扩容记录

因为我们需要扩容 3 个broker，即broker从 0,1,2,3,4,5 扩容到 0,1,2,3,4,5,6,7,8 ，所以先进行第一部分工作

集群扩容

  • 新机器的 6,7,8 的准备：JAVA、文件句柄数、防火墙、主机名以及hosts映射

  • 从 0,1,2,3,4,5 选一台机器，拷贝其上的 kafka 安装目录到 6,7,8 三台机器，删除其中的logs目录，修改 server.properties 文件中的 broker.id 项。

  • 新机器启动kafka：

    sh kafka-server-start.sh -daemon ../config/server.properties

  • 检查新broker是否加入成功，检查zookeeper上的/broker/ids下的node是否包含新增broker.id

准备工作

查看现有Topic的partition、副本以及leader和follower分布。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@gs-kafka1 kafka_2.11-0.10.2.1]# bin/kafka-topics.sh --describe --zookeeper gs-kafka1:2181 --topic WATCH-LOCATION
Topic:WATCH-LOCATION	PartitionCount:18	ReplicationFactor:3	Configs:
	Topic: WATCH-LOCATION	Partition: 0	Leader: 4	Replicas: 4,2,3	Isr: 3,4,2
	Topic: WATCH-LOCATION	Partition: 1	Leader: 5	Replicas: 5,3,4	Isr: 5,3,4
	Topic: WATCH-LOCATION	Partition: 2	Leader: 0	Replicas: 0,4,5	Isr: 5,0,4
	Topic: WATCH-LOCATION	Partition: 3	Leader: 1	Replicas: 1,5,0	Isr: 5,0,1
	Topic: WATCH-LOCATION	Partition: 4	Leader: 2	Replicas: 2,0,1	Isr: 2,0,1
	Topic: WATCH-LOCATION	Partition: 5	Leader: 3	Replicas: 3,1,2	Isr: 3,2,1
	Topic: WATCH-LOCATION	Partition: 6	Leader: 4	Replicas: 4,3,5	Isr: 5,3,4
	Topic: WATCH-LOCATION	Partition: 7	Leader: 5	Replicas: 5,4,0	Isr: 5,0,4
	Topic: WATCH-LOCATION	Partition: 8	Leader: 0	Replicas: 0,5,1	Isr: 5,0,1
	Topic: WATCH-LOCATION	Partition: 9	Leader: 1	Replicas: 1,0,2	Isr: 0,1,2
	Topic: WATCH-LOCATION	Partition: 10	Leader: 2	Replicas: 2,1,3	Isr: 2,3,1
	Topic: WATCH-LOCATION	Partition: 11	Leader: 3	Replicas: 3,2,4	Isr: 3,2,4
	Topic: WATCH-LOCATION	Partition: 12	Leader: 4	Replicas: 4,5,0	Isr: 5,0,4
	Topic: WATCH-LOCATION	Partition: 13	Leader: 5	Replicas: 5,0,1	Isr: 5,0,1
	Topic: WATCH-LOCATION	Partition: 14	Leader: 0	Replicas: 0,1,2	Isr: 0,2,1
	Topic: WATCH-LOCATION	Partition: 15	Leader: 1	Replicas: 1,2,3	Isr: 1,3,2
	Topic: WATCH-LOCATION	Partition: 16	Leader: 2	Replicas: 2,3,4	Isr: 2,4,3
	Topic: WATCH-LOCATION	Partition: 17	Leader: 3	Replicas: 3,4,5	Isr: 5,3,4

因为现有topic的副本及其leader都在 0,1,2,3,4,5 broker，所以我们需要迁移三分之一的副本到新broker，并且将三分之一的leader转到新broker，才能做到网络和磁盘IO的扩容。

迁移

参考扩容方案中的方法，我们直接编辑 topic-result.json，格式如下
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
{"version":1,"partitions":[{"topic":"WATCH-LOCATION","partition":0, "replicas":[4,2,6]},
                           {"topic":"WATCH-LOCATION","partition":1, "replicas":[5,3,7]},
                           {"topic":"WATCH-LOCATION","partition":2, "replicas":[0,4,8]},
                           {"topic":"WATCH-LOCATION","partition":3, "replicas":[1,5,6]},
                           {"topic":"WATCH-LOCATION","partition":4, "replicas":[2,0,7]},
                           {"topic":"WATCH-LOCATION","partition":5, "replicas":[3,1,8]},
                           {"topic":"WATCH-LOCATION","partition":6, "replicas":[4,3,6]},
                           {"topic":"WATCH-LOCATION","partition":7, "replicas":[5,4,7]},
                           {"topic":"WATCH-LOCATION","partition":8, "replicas":[0,5,8]},
                           {"topic":"WATCH-LOCATION","partition":9, "replicas":[1,0,6]},
                           {"topic":"WATCH-LOCATION","partition":10,"replicas":[2,1,7]},
                           {"topic":"WATCH-LOCATION","partition":11,"replicas":[3,2,8]},
                           {"topic":"WATCH-LOCATION","partition":12,"replicas":[4,5,6]},
                           {"topic":"WATCH-LOCATION","partition":13,"replicas":[5,0,7]},
                           {"topic":"WATCH-LOCATION","partition":14,"replicas":[0,1,8]},
                           {"topic":"WATCH-LOCATION","partition":15,"replicas":[1,2,6]},
                           {"topic":"WATCH-LOCATION","partition":16,"replicas":[2,3,7]},
                           {"topic":"WATCH-LOCATION","partition":17,"replicas":[3,4,8]}
                           ]}

但是上述只考虑到了副本迁移，未考虑到leader迁移，参考扩容方案，我们迁移的时候顺便 修改replicas顺序，修改如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
{"version":1,"partitions":[{"topic":"WATCH-LOCATION","partition":0, "replicas":[4,2,6]},
                           {"topic":"WATCH-LOCATION","partition":1, "replicas":[5,3,7]},
                           {"topic":"WATCH-LOCATION","partition":2, "replicas":[0,4,8]},
                           {"topic":"WATCH-LOCATION","partition":3, "replicas":[1,5,6]},
                           {"topic":"WATCH-LOCATION","partition":4, "replicas":[2,0,7]},
                           {"topic":"WATCH-LOCATION","partition":5, "replicas":[3,1,8]},
                           {"topic":"WATCH-LOCATION","partition":6, "replicas":[4,3,6]},
                           {"topic":"WATCH-LOCATION","partition":7, "replicas":[5,4,7]},
                           {"topic":"WATCH-LOCATION","partition":8, "replicas":[0,5,8]},
                           {"topic":"WATCH-LOCATION","partition":9, "replicas":[6,0,1]},
                           {"topic":"WATCH-LOCATION","partition":10,"replicas":[7,1,2]},
                           {"topic":"WATCH-LOCATION","partition":11,"replicas":[8,2,3]},
                           {"topic":"WATCH-LOCATION","partition":12,"replicas":[6,5,4]},
                           {"topic":"WATCH-LOCATION","partition":13,"replicas":[7,0,5]},
                           {"topic":"WATCH-LOCATION","partition":14,"replicas":[8,1,0]},
                           {"topic":"WATCH-LOCATION","partition":15,"replicas":[1,2,6]},
                           {"topic":"WATCH-LOCATION","partition":16,"replicas":[2,3,7]},
                           {"topic":"WATCH-LOCATION","partition":17,"replicas":[3,4,8]}
                           ]}

其他 topic 也可参考此方法一起修改副本位置和副本顺序。然后开始迁移

1
bin/kafka-reassign-partitions.sh --zookeeper gs-kafka1:2181 --reassignment-json-file topic-result.json  --execute

等所有topic迁移完成后，开始leader迁移，两个方法

  • 一次性平衡所有topic
    bin/kafka-preferred-replica-election.sh --zookeeper gs-kafka1:2181
  • 每次平衡部分topic，编写 topicPartition.json ，格式如下。
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    {"version":1,"partitions":[{"topic":"WATCH-LOCATION","partition":0},
                               {"topic":"WATCH-LOCATION","partition":1},
                               {"topic":"WATCH-LOCATION","partition":2},
                               {"topic":"WATCH-LOCATION","partition":3},
                               {"topic":"WATCH-LOCATION","partition":4},
                               {"topic":"WATCH-LOCATION","partition":5},
                               {"topic":"WATCH-LOCATION","partition":6},
                               {"topic":"WATCH-LOCATION","partition":7},
                               {"topic":"WATCH-LOCATION","partition":8},
                               {"topic":"WATCH-LOCATION","partition":9},
                               {"topic":"WATCH-LOCATION","partition":10},
                               {"topic":"WATCH-LOCATION","partition":11},
                               {"topic":"WATCH-LOCATION","partition":12},
                               {"topic":"WATCH-LOCATION","partition":13},
                               {"topic":"WATCH-LOCATION","partition":14},
                               {"topic":"WATCH-LOCATION","partition":15},
                               {"topic":"WATCH-LOCATION","partition":16},
                               {"topic":"WATCH-LOCATION","partition":17}
                               ]}
    然后平衡 bin/kafka-preferred-replica-election.sh --zookeeper gs-kafka1:2181 --path-to-json-file topicPartition.json