Skip to content

README.md

Latest commit

 

History

History
50 lines (31 loc) · 1.12 KB

README.md

File metadata and controls

50 lines (31 loc) · 1.12 KB

Kafka with orjson vs msgspec

This project is to help profiling memory usage of the Kafka with two different serialization libraries:

  • orjson
  • msgspec

Usage

python test_<serializer>.py [-h] [--kafka-bootstrap-server KAFKA_BOOTSTRAP_SERVER] --count COUNT --kafka-topic KAFKA_TOPIC

Setup

Python Virutal Environment

virtualenv ./venv # or python3 -m venv venv

. venv/bin/activate

pip install -r requirements.txt

Kafka

docker run -d --name zookeeper --network host zookeeper:latest

docker run -d --name=kafka --network host -e KAFKA_ZOOKEEPER_CONNECT=localhost:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 confluentinc/cp-kafka:latest

Run

PYTHONMALLOC=malloc memray run --follow-fork test_orjson.py -s localhost:9092 -t test -c 99999

PYTHONMALLOC=malloc memray run --follow-fork test_msgspec.py -s localhost:9092 -t test -c 99999

Results

Find the result files starting with memray- in the current directory.

memray table --leaks <result file>

Then open the generated HTML report in your browser.