NATS.io Newsletter Contact Us

Synadia Connect

AWS S3

NAME	RUNTIME	TYPE	VERSION	STATUS
aws_s3	wombat	source	latest	stable

Downloads objects within an Amazon S3 bucket, optionally filtered by a prefix, either by walking the items in the bucket or by streaming upload notifications in realtime.

Configuration

name	type	optional	secret	description
bucket	string	yes	no	The bucket to consume from. If the field `sqs.url` is specified this field is optional.
prefix	string	yes	no	An optional path prefix, if set only objects with the prefix are consumed when walking a bucket.
region	string	no	no	The AWS region to target.
endpoint	string	yes	no	Allows you to specify a custom endpoint for the AWS API. The value defaults to ``.
credentials	credentials	no	no	Optional manual configuration of AWS credentials to use. More information can be found in xref:guides:cloud/aws.adoc[].
force_path_style_urls	bool	yes	no	Forces the client API to use path style URLs for downloading keys, which is often required when connecting to custom endpoints. The value defaults to `false`.
delete_objects	bool	yes	no	Whether to delete downloaded objects from the bucket once they are processed. The value defaults to `false`.
scanner	scanner	no	no	The scanner by which the stream of bytes consumed will be broken out into individual messages. Scanners are useful for processing large sources of data without holding the entirety of it within memory. For example, the `csv` scanner allows you to process individual CSV rows without loading the entire CSV file in memory at once.
sqs	sqs	no	no	Consume SQS messages in order to trigger key downloads.

Credentials

name	type	optional	secret	description
id	string	yes	no	The ID of credentials to use. The value defaults to ``.
secret	string	yes	no	The secret for the credentials being used. The value defaults to ``.
token	string	yes	no	The token for the credentials being used, required when using short term credentials. The value defaults to ``.

SQS

name	type	optional	secret	description
url	string	yes	no	An optional SQS URL to connect to. When specified this queue will control which objects are downloaded. The value defaults to ``.
endpoint	string	yes	no	A custom endpoint to use when connecting to SQS. The value defaults to ``.
key_path	string	yes	no	A dot path whereby object keys are found in SQS messages. The value defaults to `Records.*.s3.object.key`.
bucket_path	string	yes	no	A dot path whereby the bucket name can be found in SQS messages. The value defaults to `Records.*.s3.bucket.name`.
envelope_path	string	yes	no	A dot path of a field to extract an enveloped JSON payload for further extracting the key and bucket from SQS messages. This is specifically useful when subscribing an SQS queue to an SNS topic that receives bucket events. The value defaults to ``.
delay_period	string	yes	no	An optional period of time to wait from when a notification was originally sent to when the target key download is attempted. The value defaults to ``.
max_messages	int	yes	no	The maximum number of SQS messages to consume from each request. The value defaults to `10`.
wait_time_seconds	int	yes	no	Whether to set the wait time. Enabling this activates long-polling. Valid values: 0 to 20. The value defaults to `0`.

Previous: AWS Kinesis
Next: AWS SQS