kafka

The kafka source receives inputs from Kafka.

The kafka source consumes kafka topics and turns messages into inputs for fns. It will make a message as consumed only if the fn returns an output successfully.

fnrunner Key

fnrun.source/kafka

Configuration

The following table describes the keys on the configuration object for the kafka source. Configuration data is required because the source will not know how to connect to Kafka without at least the brokers and topics.

KeyTypeDefaultDescription
brokersString""A comma-delimited list of broker addresses
groupString""The consumer group name
versionString2.1.1The kafka version
topicsString""A comma-delimited list of topics to consume
assignorStringrangeThe rebalance strategy; one of sticky, roundrobin, and range
oldestBoolfalseIndicates whether the source should start with the oldest available message for a partition
ignoreErrorsBoolfalseIndicates whether messages are marked as consumed when the fn returns an error

Inputs

Inputs generated by the http source are objects with the following keys.

KeyTypeDescription
keyStringThe message key
valueStringThe message value
offsetIntegerThe message offset
partitionIntegerThe partition from which the message was received
topicStringThe topic from which the message was received
timestampTimestampThe inner message timestamp

Examples

The following example shows a system that consumes messages from a topic called test-topic and processes the event value with a bash script. Note that the message will be marked as received only if the script returns successfully.

source:
  fnrun.source/kafka:
    brokers: 127.0.0.1:9092
    group: fnrun-example
    topics: test-topic
middleware:
  - fnrun.middleware/key: value
fn:
  fnrun.fn/cli:
    command: ./fn.sh
    script: true

Edit this page on GitHub

← http
lambda →