Synadia Connect
HDFS
| NAME | RUNTIME | TYPE | VERSION |
|---|---|---|---|
| hdfs | wombat | sink | latest |
Sends message parts as files to a HDFS directory.
Each file is written with the path specified with the 'path' field, in order to have a different path for each object you should use function interpolations described here.
Configuration
| Name | Type | Optional | Secret | Description |
|---|---|---|---|---|
| hosts | []string | no | no | A list of target host addresses to connect to. |
| user | string | yes | no | A user ID to connect as. The value defaults to "". |
| directory | string | no | no | A directory to store message files within. If the directory does not exist it will be created. |
| path | string | yes | no | The path to upload messages as, interpolation functions should be used in order to generate unique file paths. The value defaults to "${!counter()}-${!timestamp_unix_nano()}.txt". |
| max_in_flight | int | yes | no | The maximum number of messages to have in flight at a given time. Increase this to improve throughput. The value defaults to 64. |
| batching | batching | no | no | Allows you to configure a batching policy. |
Batching
| Name | Type | Optional | Secret | Description |
|---|---|---|---|---|
| count | int | yes | no | A number of messages at which the batch should be flushed. If 0 disables count based batching. The value defaults to 0. |
| byte_size | int | yes | no | An amount of bytes at which the batch should be flushed. If 0 disables size based batching. The value defaults to 0. |
| period | string | yes | no | A period in which an incomplete batch should be flushed regardless of its size. The value defaults to "". |
| check | string | yes | no | A Bloblang query that should return a boolean value indicating whether a message should end a batch. The value defaults to "". |