Synadia Cloud
JetStream Management
JetStream manages and stores messages. More specifically, JetStream promotes persistence such that data remains available (for a time period that you configure) for access, message playback, and use. The basic NATS messaging API (which allows message exchange) is referred to as NATS Core; JetStream resides atop NATS Core.
The primary JetStream window (depicted below) allows you to view account usage specific to Streams, Consumers, File Storage, and Memory Storage. Streams represent data. You can manage each stream individually along with its settings and consumers (message recipients). You can also monitor streams and enable Live Mode to view real-time messages.
Create
Stream, Mirror, KV Bucket, and Object Bucket are JetStream assets. You can create and move JetStream assets in real time across regions, while connections and data remain available and intact. You can also relocate a stream globally to another cluster.
Stream
A stream is a location or channel that stores sent and received messages. A stream is also commonly considered a log or list of messages organized in the order or sequence received along with its related subject; clients can filter logs to access messages. Replicas are copies of streams. This redundancy helps safeguard your data. Replicas can exist on multiple NATS servers as data backups. If a server goes down, the actual stream still functions without issue. Replicas promote durability.
Replicas include only odd numbers. Currently, only Replica 1 (R1) and Replica 3 (R3) exist. R1 stores data on one server; R3 stores data on three servers. Just as some customers might configure accounts to use business functions (Order, Ship, Purchase), for example, others might configure accounts to reflect DEV/TEST/PROD environments. For these customers, you must not use R1 streams in PROD because we cannot guarantee data durability.
Our billing plans include Standard Streams, Standard Consumers, HA Storage, HA Streams, and HA Consumers. Most customers prefer (and use) the term R1 as opposed to Standard and R3 as opposed to HA. Regarding terminology, we’ll be changing Standard to R1 and HA to R3, going forward, to accommodate you and enhance your user experience. HA includes failover safeguards that prevent server downtime, making HA (or R3) more durable than Standard (or R1).
WARNING: DO NOT USE R1 streams in PROD environments because we cannot guarantee data durability. This applies to organizations that have established accounts that reflect DEV/TEST/PROD environments or similar. R3 is HA and, therefore, more durable than R1 (Standard). The Personal/Free tier does not include R3, but all paid billing plans provide this offering.
E-commerce App
This example app receives orders and has a West Coast data center. The app ships these orders to its East Coast warehouse. The warehouse receives the orders and sends the shipment information to the app – ensuring all information will be available in this cloud app. We won’t complete this process in its entirety. Instead, we’ll explore cross-account sharing. We will move data, a stream for example, from the cloud app to the warehouse. This example originates from a tutorial video our team recorded, which aligns with the image below.
The instructions below apply, if you need to create assets. You are neither required to follow the E-commerce scenario nor include its parameters. Simply follow the instructions. However, to guide you through our cloud/warehouse scenario, we have provided the parameters for the E-commerce app such that you fully understand how this process works.
To create a stream:
Stream parameters for E-commerce app:
| Field | Parameter |
|---|---|
| Name | orders |
| Subject | orders |
| Replicas | R3 is preferred; however, R1 is the only option available with the Personal/Free tier. |
| Cluster Enabled | ngsprod-aws-uswest2 |
Click Create Stream.
Designate your stream parameters.
Save your updates.
The table displays the stream you created. You can click this new stream – orders, for example – to view Stream Activity. You can also filter streams and inspect messages.
Overview, Configuration, Consumers, and Settings appear in the Stream Activity window.
Overview
You can view additional data for this stream. You can view data by Start Date, for example, and even slide the Live Mode toggle (to the right or ON position) to view real-time data entering the stream. You can also inspect the message, view the value, and decode in JSON or XML format.
Configuration
This window includes the Basic Info, Limits, and Subjects cards.
Consumers
Consumers are message recipients. Since JetStream provides persistence, no messages are missed. Consumers process streams and have message playback capabilities. You can manage consumers associated with this stream.
Settings
This window allows you to view all specifications for this stream. You can also Delete Stream (remove stream and all data) and Purge Stream (remove all data only).
E-commerce App
Next, the messages specific to the orders stream must reach the East Coast warehouse, which is a separate account. We must share this stream. The Cross-Account Sharing describes this process. After you export the orders stream and create an order_mirror (all via the export/import process which is a subset of account sharing), we would have completed our goal of sharing data across accounts and also would have reached our Export limit using the Personal/Free tier.
Mirror
A mirror is a copy of a stream with a different name. Similar to replicas, mirrors can be R1 or R3 (Standard or HA, respectively). You might have a main or source stream in California, for example, but want to process messages in Europe. Without mirrors, Europe might have to communicate with a cluster in California. However, a mirror in Europe makes streaming faster and more efficient – as the data is in closer proximity. As the source stream updates, the mirror updates. You can create a mirror of any JetStream asset: Stream, KV Bucket, and Object Bucket.
The difference between mirrors and replicas is specific to the level at which they operate. Replicas operate at the server level and are indicative of stream resiliency. Mirrors operate at the JetStream level and automatically copy messages from one stream to another.
If you need to secure any JetStream asset – Stream, Mirror, KV Bucket, Object Bucket – such that you won’t lose data if a server goes down, you want to use R3 (HA). With R1, we cannot guarantee data durability.
To create a mirror:
Click Create > Mirror.
Designate your mirror parameters.
Save your updates.
The table displays the mirror you created. You can click this new table entry to view Mirror Activity. Similar to the details described above for streams, you can access and modify content in the Overview, Configuration, Consumers, and Settings windows.
KV Bucket
This asset represents the storage location for key-value pairs, such as Shipping IDs and Shipping Product Names.
To create a KV Bucket:
Click Create > KV Bucket.
Designate your KV Bucket parameters.
Save your updates.
The table displays the KV Bucket you created. You can click this new table entry to view KV Bucket Activity. KV Buckets have fewer configurations than streams. You can access and modify content in the Overview, Consumers, and Settings windows.
Object Bucket
This asset represents the storage location for large files, such as documents and graphics.
To create an Object Bucket:
Click Create > Object Bucket.
Designate your Object Bucket parameters.
Save your updates.
The table displays the Object Bucket you created. You can click this new table entry to view Object Bucket Activity. Object Buckets have fewer configurations than streams. You can access and modify content in the Overview, Consumers, and Settings windows.
Object store is not a distributed storage system. All files in a bucket will need to fit on the target file system.