Kafka Broker

The technical documentation of the kafka-broker microservice, which handles the communication between the microservices and the Kafka broker.

The Kafka broker in the United Manufacturing Hub is a Apache Kafka , that with the help of Zookeeper manages the topics and the partitions. It’s a core component of the stack and is used to communicate between the different microservices.

How it works

Kafka is an event streaming platform that is used to store and process messages. It’s a distributed system that is made up of a cluster of brokers. Each broker is responsible for a subset of the data. The brokers communicate with each other to keep the data in sync. The brokers are also responsible for the replication of the data. The replication factor is the number of copies of each partition that are kept on different brokers.

Kubernetes resources

  • StatefulSet: united-manufacturing-hub-kafka
  • Service:
    • Internal ClusterIP: united-manufacturing-hub-kafka at port 9092
    • Internal ClusterIP (headless): united-manufacturing-hub-kafka-headless at port 9092
    • External LoadBalancer: united-manufacturing-hub-kafka-0-external at port 9094
  • ConfigMap: united-manufacturing-hub-kafka-scripts
  • Secret: united-manufacturing-hub-kafka-broker-secrets
  • PersistentVolumeClaim: data-united-manufacturing-hub-kafka-0

Configuration

You shouldn’t need to configure the Kafka broker manually, as it’s configured automatically when the cluster is deployed. However, if you need to change the configuration, you can do it by editing the kafka section of the Helm chart values file.

Environment variables

Environment variables
Variable nameDescriptionTypeAllowed valuesDefault
ALLOW_PLAINTEXT_LISTENERAllow to use the PLAINTEXT listenerstringAnyyes
BITNAMI_DEBUGSpecify if debug values should be setbooltrue, falsefalse
KAFKA_CFG_ALLOW_EVERYONE_IF_NO_ACL_FOUNDAllow resources with no associated ACLbooltrue, falsetrue
KAFKA_CFG_AUTHORIZER_CLASS_NAMEClass name of the AuthorizerstringAny""
KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLEWhether topics get generated automaticallybooltrue, falsetrue
KAFKA_CFG_DEFAULT_REPLICATION_FACTORDefault replication factors for automatically created topicsintAny1
KAFKA_CFG_DELETE_TOPIC_ENABLEWhether topic deletion is enabledbooltrue, falsefalse
KAFKA_CFG_LISTENERSThe address(es) the socket server listens onstringAnyINTERNAL://:9093,CLIENT://:9092,EXTERNAL://:9094
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAPThe protocol to listener mappingstringAnyINTERNAL:PLAINTEXT,CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_CFG_LOG_DIRSA comma separated list of directories in which kafka’s log data is keptstringAny/bitnami/kafka/data
KAFKA_CFG_LOG_FLUSH_INTERVAL_MESSAGESThe number of messages to accept before forcing a flush of data to diskintAny10000
KAFKA_CFG_LOG_FLUSH_INTERVAL_MSThe maximum amount of time a message can sit in a log before a forced flushintAny1000
KAFKA_CFG_LOG_RETENTION_BYTESA size-based retention policy for logsintAny26214400
KAFKA_CFG_LOG_RETENTION_CHECK_INTERVAL_MSThe interval at which log segments are checked to see if they can be deletedintAny300000
KAFKA_CFG_LOG_RETENTION_HOURSThe minimum age of a log file to be eligible for deletion due to ageintAny168
KAFKA_CFG_LOG_SEGMENT_BYTESThe maximum size of a log segment file. When this size is reached a new log segment will be createdintAny10485760
KAFKA_CFG_MESSAGE_MAX_BYTESThe largest record batch size allowed by KafkaintAny1000012
KAFKA_CFG_NUM_IO_THREADSThe number of threads doing disk I/OintAny8
KAFKA_CFG_NUM_NETWORK_THREADSThe number of threads handling network requestsintAny3
KAFKA_CFG_NUM_PARTITIONSThe default number of log partitions per topicintAny6
KAFKA_CFG_NUM_RECOVERY_THREADS_PER_DATA_DIRThe number of threads per data directory to be used for log recovery at startup and flushing at shutdownintAny1
KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTORThe replication factor for the offsets topicintAny1
KAFKA_CFG_SOCKET_RECEIVE_BUFFER_BYTESThe receive buffer (SO_RCVBUF) used by the socket serverintAny102400
KAFKA_CFG_SOCKET_REQUEST_MAX_BYTESThe maximum size of a request that the socket server will accept (protection against OOM)intAny104857600
KAFKA_CFG_SOCKET_SEND_BUFFER_BYTESThe send buffer (SO_SNDBUF) used by the socket serverintAny102400
KAFKA_CFG_SUPER_USERSList of super usersstringAnyUser:admin
KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISROverridden min.insync.replicas config for the transaction topicintAny1
KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTORThe replication factor for the transaction topicintAny1
KAFKA_CFG_ZOOKEEPER_CONNECTURL of ZooKeeperstringAnyunited-manufacturing-hub-zookeeper
KAFKA_CFG_ZOOKEEPER_CONNECTION_TIMEOUT_MSTimeout in ms for connecting to ZooKeeperintAny6000
KAFKA_HEAP_OPTSKafka Java Heap sizestringAny-Xmx2048m -Xms2048m
KAFKA_INTER_BROKER_LISTENER_NAMEThe listener that the brokers should communicate onstringAnyINTERNAL
KAFKA_LOG_DIRMount path of the Kafka logs volumestringAny/opt/bitnami/kafka/logs
KAFKA_VOLUME_DIRMount path of the Kafka data volumestringAny/bitnami/kafka
KAFKA_ZOOKEEPER_PROTOCOLThe protocol to use with ZooKeeperstringAnyPLAINTEXT
Last modified May 5, 2023: feat: finishing touches (1147002)