压测实操--kafka-consumer压测方案

avatar
作者
猴君
阅读量:0

作者:九月

环境信息:

操作系统centos7.9,kafka版本为hdp集群中的2.0版本。

Consumer相关参数

使用Kafka自带的kafka-consumer-perf-test.sh脚本进行压测,该脚本参数为:

请添加图片描述

请添加图片描述

请添加图片描述

thread:测试时的单机线程数;

fetch-size:抓取数据量;

partition:分区数,主要是和线程复合来测试;

replication:副本数;

througout:我们所需要的吞吐量,单位时间内处理消息的数量,可能对我们处理消息的延迟有影响;

consumer测试

fetch-size

fetch-size=524288
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size  524288  --threads 4 
fetch-size=1048576
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size  1048576  --threads 4 
fetch-size=2097152
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size  2097152  --threads 4 
fetch-size=3145728
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size  3145728  --threads 4 
fetch-size=4194304
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size  4194304  --threads 4 
fetch-size=5242880
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 1000000 --fetch-size  5242880  --threads 4 

partitions

创建不同partitions的topic
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-topics.sh --create  --zookeeper mutong01:2181,mutong02:2181,mutong03:2181  --topic test_kafka_consumer1  --partitions 1 --replication-factor 1  /usr/hdp/3.1.5.0-152/kafka/bin/kafka-topics.sh --create  --zookeeper mutong01:2181,mutong02:2181,mutong03:2181  --topic test_kafka_consumer2  --partitions 2 --replication-factor 1  /usr/hdp/3.1.5.0-152/kafka/bin/kafka-topics.sh --create  --zookeeper mutong01:2181,mutong02:2181,mutong03:2181  --topic test_kafka_consumer3  --partitions 3 --replication-factor 1  /usr/hdp/3.1.5.0-152/kafka/bin/kafka-topics.sh --create  --zookeeper mutong01:2181,mutong02:2181,mutong03:2181  --topic test_kafka_consumer4  --partitions 4 --replication-factor 1 
topic生产数据
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-producer-perf-test.sh  --topic test_kafka_consumer1 --num-records  100000000 --record-size 687  --producer-props bootstrap.servers=172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667  batch.size=5000 --throughput 30000  /usr/hdp/3.1.5.0-152/kafka/bin/kafka-producer-perf-test.sh  --topic test_kafka_consumer2 --num-records  100000000 --record-size 687  --producer-props bootstrap.servers=172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667  batch.size=5000 --throughput 30000  /usr/hdp/3.1.5.0-152/kafka/bin/kafka-producer-perf-test.sh  --topic test_kafka_consumer3 --num-records  100000000 --record-size 687  --producer-props bootstrap.servers=172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667  batch.size=5000 --throughput 30000  /usr/hdp/3.1.5.0-152/kafka/bin/kafka-producer-perf-test.sh  --topic test_kafka_consumer4 --num-records  100000000 --record-size 687  --producer-props bootstrap.servers=172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667  batch.size=5000 --throughput 30000 
消费不同partitions的topic

消费不同partitions的topic,获取不同partition下的吞吐量

/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer1 --messages 10000000 --fetch-size  1048576  --threads 4   /usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer2 --messages 10000000 --fetch-size  1048576  --threads 4   /usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer3 --messages 10000000 --fetch-size  1048576  --threads 4   /usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer4 --messages 10000000 --fetch-size  1048576  --threads 4 

fetch-threads

fetch-thread=1
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --fetch-size 1048576  --messages 10000000  --threads 1  --num-fetch-threads 1 
fetch-thread=4
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --fetch-size 1048576  --messages 10000000  --threads 1  --num-fetch-threads 4 
fetch-thread=7
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --fetch-size 1048576  --messages 10000000  --threads 1  --num-fetch-threads 7 
fetch-thread=10
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --fetch-size 1048576  --messages 10000000  --threads 1  --num-fetch-threads 10 

总结

使用kafka自带的kafka-consumer-perf-test.sh脚本对集群消费情况进行压测,根据业务情况设置要测试的messager条数。不同的变量因素,通过修改其中一个变量条件,固定其余变量值,获取对应的吞吐速率。整理不同条件下的指标结果,形成曲线就可以观察具体的消费趋势。

请添加图片描述

广告一刻

为您即时展示最新活动产品广告消息,让您随时掌握产品活动新动态!