Bloblang

Bloblang is a declarative mapping language for transforming structured data. It creates new documents by assigning values from input documents, literals, and function calls.

Bloblang Queries

Some configuration fields support Bloblang function interpolations, which allow you to query message contents and perform arithmetic. The syntax is ${!<bloblang expression>}, where the contents are a Bloblang query (the right-hand-side of a Bloblang mapping).

For example, with the following config:

output:
  kafka:
    addresses: ['localhost:9092']
    topic: 'meow-${! json("topic") }'

A message with contents {"topic":"foo","message":"hello world"} would be routed to the Kafka topic meow-foo.

Escaping

If a literal string matching this pattern is required (e.g., ${!foo}), escape it with double brackets: ${{!foo}} is read as the literal ${!foo}.

Bloblang Core Concepts

Keyword	Description
`root`	The output document being constructed. Assignments to `root` or its fields build the new message.
`this`	The input document context (read-only). Access input fields via `this.fieldname`.
`content()`	Returns the raw bytes of the input message.

Basic assignment:

root = this                           # Copy entire input to output
root.name = this.user.name            # Copy nested field
root.count = this.items.length()      # Use methods on values

Functions

Function	Description
`uuid_v4()`	Generates a random UUID v4 string.
`now()`	Returns current timestamp as RFC 3339 string.
`timestamp_unix()`	Returns current Unix timestamp in seconds.
`counter()`	Returns an incrementing integer (starts at 1).
`random_int()`	Returns a random 64-bit integer.
`content()`	Returns raw message bytes.
`json()`	Parses message content as JSON.
`metadata()`	Returns all metadata as an object.
`env()`	Returns the value of an environment variable.
`hostname()`	Returns the hostname of the machine.
`range()`	Generates an array of integers from start to stop-1.
`throw()`	Throws an error with the given message.
`deleted()`	Marks a field or message for deletion.

Methods

String Methods

Method	Description
`.string()`	Converts value to string.
`.bytes()`	Converts value to byte array.
`.uppercase()`	Converts string to uppercase.
`.lowercase()`	Converts string to lowercase.
`.capitalize()`	Capitalizes first letter of each word.
`.trim()`	Removes leading/trailing whitespace.
`.replace_all(old, new)`	Replaces all occurrences of old with new.
`.split(delimiter)`	Splits string into array by delimiter.
`.contains(substring)`	Returns true if string contains substring.
`.has_prefix(prefix)`	Returns true if string starts with prefix.
`.has_suffix(suffix)`	Returns true if string ends with suffix.
`.slice(start, end)`	Extracts substring from start to end index.
`.length()`	Returns length of string.
`.re_match(pattern)`	Returns true if string matches regex pattern.
`.re_find_all(pattern)`	Returns array of all regex matches.
`.re_replace_all(pattern, repl)`	Replaces all regex matches.

Number Methods

Method	Description
`.number()`	Converts value to number.
`.int32()`	Converts value to integer.
`.float32()`	Converts value to float.
`.round()`	Rounds to nearest integer.
`.floor()`	Rounds down to integer.
`.ceil()`	Rounds up to integer.
`.abs()`	Returns absolute value.
`.max(other)`	Returns larger of two values.
`.min(other)`	Returns smaller of two values.

Array Methods

Method	Description
`.length()`	Returns number of elements.
`.append(value)`	Adds value to end of array.
`.concat(array)`	Concatenates two arrays.
`.contains(value)`	Returns true if array contains value.
`.flatten()`	Flattens nested arrays into single array.
`.index(i)`	Returns element at index i.
`.slice(start, end)`	Extracts subarray from start to end.
`.sort()`	Sorts array elements.
`.sort_by(path)`	Sorts objects by field path.
`.reverse()`	Reverses array order.
`.unique()`	Removes duplicate values.
`.join(delimiter)`	Joins array elements into string.
`.map_each(mapping)`	Applies mapping to each element (use `this` for current element).
`.filter(condition)`	Keeps elements where condition is true.
`.any(condition)`	Returns true if any element matches condition.
`.all(condition)`	Returns true if all elements match condition.
`.sum()`	Sums numeric array elements.
`.fold(init, mapping)`	Reduces array to single value (use `tally` and `value`).

Object Methods

Method	Description
`.keys()`	Returns array of object keys.
`.values()`	Returns array of object values.
`.merge(object)`	Merges two objects (right takes precedence).
`.without(keys...)`	Returns object without specified keys.
`.exists(path)`	Returns true if path exists.
`.get(path)`	Returns value at path or null if missing.
`.assign(key, value)`	Sets key to value.
`.map_each(mapping)`	Applies mapping to each key-value pair.
`.map_each_key(mapping)`	Applies mapping to each key.

Type and Error Handling

Method	Description
`.type()`	Returns type as string: "string", "number", "bool", "array", "object", "null".
`.bool()`	Converts value to boolean.
`.not_null()`	Returns value if not null, otherwise fails.
`.or(default)`	Returns value if not null, otherwise returns default.
`.catch()`	Returns fallback value if expression fails.

Encoding and Parsing

Method	Description
`.parse_json()`	Parses string as JSON.
`.format_json()`	Formats value as JSON string.
`.format_json(indent)`	Formats as indented JSON (true for pretty print).
`.parse_yaml()`	Parses string as YAML.
`.format_yaml()`	Formats value as YAML string.
`.parse_csv()`	Parses CSV string into array of objects.
`.parse_url()`	Parses URL into components object.
`.parse_form_url_encoded()`	Parses URL-encoded form data.
`.encode(encoding)`	Encodes to: base64, base64url, hex, ascii85.
`.decode(encoding)`	Decodes from: base64, base64url, hex, ascii85.
`.compress(algorithm)`	Compresses using: gzip, zlib, deflate, lz4, snappy, zstd.
`.decompress(algorithm)`	Decompresses using above algorithms.
`.hash(algorithm)`	Hashes value using: md5, sha1, sha256, sha512, xxhash64, crc32.

Variables

Use let to create variables for reuse:

let user_id = this.user.id
let timestamp = now()

root.events = this.events.map_each(event -> {
  "user": $user_id,
  "time": $timestamp,
  "type": event.type,
  "data": event.payload
})

Conditional Logic

Match Expressions

root.doc.type = match {
  this.exists("header.id") => "foo"
  this.exists("body.data") => "bar"
  _ => throw("unknown type")
}
root.doc.contents = (this.body.content | this.thing.body)

# In:  {"header":{"id":"first"},"thing":{"body":"hello world"}}
# Out: {"doc":{"contents":"hello world","type":"foo"}}

# In:  {"nothing":"matches"}
# Out: Error("failed assignment (line 1): unknown type")

If Expressions

root.priority = if this.urgent == true {
  "high"
} else if this.important == true {
  "medium"
} else {
  "low"
}

Practical Examples

Restructure a message:

root.id = uuid_v4()
root.created_at = now()
root.user = {
  "name": this.firstName + " " + this.lastName,
  "email": this.email.lowercase()
}
root.tags = this.categories.map_each(c -> c.lowercase())

Handle missing or null values:

root.name = this.name.or("Anonymous")
root.score = this.score.catch(0)
root.metadata = this.meta.or({})

Delete fields or messages:

root = this
root.password = deleted()
root.internal_id = deleted()

# Delete entire message if condition met
root = if this.test == true { deleted() } else { this }