java seems to be the theme of the week.

still have the same java as the minecraft experiment.

pulled the “community” version of kafka from confluent: https://www.confluent.io/installation/ (may have to register)

have extracted the tar file onto the nvme drive:
/nvme/community/confluent-7.3.2

edit the properties files to point to the nvme also:

etc/kafka/server.properties:log.dirs=/nvme/kafka/kafka-logs
etc/kafka/zookeeper.properties:dataDir=/nvme/kafka/zookeeper

now using “tmux” to start multiple components
(https://www.redhat.com/sysadmin/introduction-tmux-linux)

and manually start the components each in their own “tmux tab”
zookeeper
./bin/zookeeper-server-start etc/kafka/zookeeper.properties

kafka-server
./bin/kafka-server-start etc/kafka/server.properties

and create a sample topic: (this is all on one line)
./bin/kafka-topics --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic riscv-topic

now also use python, so installed kafka-python

and used it to create some random data

import csv
import json
from kafka import KafkaProducer
import random

MIN_LETTER = 65
MAX_LETTER = 90

producer = KafkaProducer(
    bootstrap_servers="visionfive2:9092",
    client_id="my_client",
    acks=1,
    value_serializer=lambda v: json.dumps(v).encode("utf-8"),
)

# simple random data test
for r in range(0, 100):
    random_string = ""
    for s in range(6):
        random_integer = random.randint(MIN_LETTER, MAX_LETTER)
        random_string += chr(random_integer)

    row = {"count": r, "code": random_string, "number": random_integer}
    producer.send(topic="riscv-topic", value=row)
    producer.flush()

then to see them on cli: (in its own tab kafka-consumer)

./bin/kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic riscv-topic

this is putty, from my windows machine, you can see the various “tabs” running the different servers.