NGS
NGS Day 0
Congratulations on the start of your next-generation connective fabric journey! This guide offers a brief sampling of technologies and solutions now available to you with NGS account in-hand.
In this guide we use the flexible NATS.io CLI tool for demonstrating service communication, but your real applications will use the rich NATS.io client libraries available for Go, Rust, Java, .NET, Python, JavaScript, C, and many other languages. The NATS.io CLI has no "special powers" -- it uses the Go client library!
This guide takes approximately 5 minutes.
After you complete this "Day 0" guide, you may be interested in The Battery Spa end-to-end scenario on "Day 1".
Guide Pre-requisites
- NGS account
- NGS profile setup on your host
- NATS.io CLI tool Version 0.0.32 or greater
- A terminal window and network connectivity to NGS
Multi-Geo, Multi-Cloud, and Edge Pub-Sub
Send a message to any NGS-connected subscriber who has expressed interest in "foo" subject:
nats pub foo 'Hello, World!'
17:52:59 Published 13 bytes to "foo"
Why is this cool?
An app publishes messages to purely functional addresses (subject) without physical concern, pre-knowledge, or configuration headache of delivery URLs, DNS resolvers, geographic locations, or traffic routing.
Any app that has current interest in a subject (subscribed) will be sent the message asyncronously. NGS transparently takes care of routing, access control, and more; all at extremely high speed and low latency.
Only subscribers that are permitted within your account and any external accounts that have accepted your share offer (export defined by subject) is eligible to receive your published messages.
Multi-Geo, Multi-Cloud, and Edge Services
Have a "conversation" with a service
nats req ngs.echo 'You out there NGS Echo service?'
17:55:30 Sending request on "ngs.echo"
17:55:30 Received with rtt 42.685134ms
[Oregon, US]: "You out there NGS Echo service?"
Load-balanced service ngs.echo lives in every geo region of NGS and can be invoked by any NGS account. NGS will route your "echo" request to the instance nearest your NGS connection.
Why is this cool?
Asynchronous API! An app sends requests to purely functional endpoints (subject) and receives replies over a dynamic reply subject. Unlike HTTP-based APIs, the service conversation is non-blocking and sessionless which enables great speed and scale; replies can be singular or a stream of reply messages!
Have a scatter-gather service conversation
nats req ngs.echo.status --replies=5 'All NGS Echo Status services report!'
Service ngs.echo.status lives in every geo region of NGS. NGS will route your "echo status" request to every instance.
Make a service available
Providing a service on NGS is as easy as subscribing your service app to a subject -- an endpoint -- and responding to request messages that are sent.
nats reply my.echo --echo --queue echopool
17:57:32 Listening on "my.echo" in group "echopool"
Run as many instances of my.echo as you like from anywhere in the world. NGS will pick one instance of "echopool" for each client request.
Field a time service my.weather such that a caller will get a localized weather report from every instance on NGS.
nats reply my.weather --command 'curl -s https://wttr.in/?format=3'
17:59:19 Listening on "my.weather" in group "NATS-RPLY-22"
Here's an example request for my.weather (in another terminal):
nats req my.weather --replies=3 'Give me local weather please!'
18:02:51 Sending request on "my.weather"
18:02:52 Received with rtt 1.208008744s
University Place, Washington, United States: ⛅️ +58°F
Why is this cool?
We can easily field global services deployed as a group member of a load-balanced pool or working independently. The endpoint of our service is simply the subscribed subject. NGS will immediately route requests on that subject to your service -- no additional configuration required!
Reliable messaging: Instant Streams, Queues, and Topics
NGS provides real-time communication with At-Most-Once quality of service as in the Pub-Sub and Service examples.
For At-Least-Once and even Once-and-Only-Once messaging, simply employ JetStream between publishers and subscribers on a specific subject (or set of subjects). JetStream ingests published messages and stores them for later read by subscribers -- an extremely flexible, durable communication buffer!
JetStreams can be placed in any Geo location or Cloud Provider of NGS (or near you by default), and has sophisticated message, growth, lifecycle, and availability policies.
Let's create a JetStream in the western region of the US that captures any message published that matches the some.event.> subject hierarchy, keeping a rolling max of 10 MiB or 1,000 messages with lifetime of up to 7 days. Messages will be limited to 128 KiB in size and any duplicates within 2 minute window ignored.
nats stream add SOME-EVENTS \
--description 'Capture some events in an NGS stream' \
--tags 'geo:us-west' \
--subjects 'some.event.>' \
--retention limits --discard old --dupe-window 2m \
--max-age 7d --max-bytes 10485760 --max-msgs 1000 --max-msgs-per-subject=-1 --max-msg-size 131072 \
--replicas 1 --storage file \
--no-deny-delete --no-deny-purge --no-allow-rollup
By specifying different retention policies, JetStream provides Stream, Queue, or Topic styles of consumption:
| Retention Policy | Pattern | Description |
|---|---|---|
| Limits | Stream | N consumers can read (and re-read!) the stream independently |
| WorkQueue | Queue | Queue - Message consumption deletes messages |
| Interest | Topic (Multi-Queue) | When all N consumers have read, messages are deleted |
Here's the output of nats stream add:
Stream SOME-EVENTS was created
Information for Stream SOME-EVENTS created 2022-04-17T18:07:06-07:00
Configuration:
Description: Capture some events in an NGS stream
Subjects: some.event.>
Acknowledgements: true
Retention: File - Limits
Replicas: 1
Discard Policy: Old
Duplicate Window: 2m0s
Allows Msg Delete: true
Allows Purge: true
Allows Rollups: false
Maximum Messages: 1,000
Maximum Bytes: 10 MiB
Maximum Age: 7d0h0m0s
Maximum Message Size: 128 KiB
Maximum Consumers: unlimited
Placement Tags: geo:us-west
Cluster Information:
Name: ngstest-az-uswest2
Leader: az-uswest2-natsjs-1
State:
Messages: 0
Bytes: 0 B
FirstSeq: 0
LastSeq: 0
Active Consumers: 0
Why is this cool?
With JetStream we temporally decouple publishers from subscribers enabling durable subscription and rich consumption patterns. With JetStream, a published message's journey to one or many subscribing apps can now occur in milliseconds or months.
You choose how a JetStream operates -- as a continuous and replayable stream, as a traditional queue, or as a multi- consumer topic!
Materialized Views: JetStream-enabled Key-Value and Object Stores!
Streams, Queues, and Topics are all message-passing patterns, but JetStream also "materializes" other access patterns including Key-Value store and Object store solutions!
Key-Value Store
Each KV store in NGS is a JetStream with each unique subject materializing a key.
KV stores access is by familiar key-value idioms such as put, get, and history. Watches can be placed -- at store level or key level -- so apps can act on change events!
Create a KV store to hold the last 10 values of key1:
nats kv add KV-STORE \
--description 'KV store for key values' \
--replicas 1 --storage file --history 10 --max-bucket-size 10485760 --max-value-size 131072
Information for Key-Value Store Bucket KV-STORE created 2022-04-17T18:19:05-07:00
Configuration:
Bucket Name: KV-STORE
History Kept: 10
Values Stored: 0
Backing Store Kind: JetStream
Description: KV store for key values
Maximum Bucket Size: 10485760
Maximum Value Size: 131072
JetStream Stream: KV_KV-STORE
Storage: File
Cluster Information:
Name: ngstest-az-uswest2
Leader: az-uswest2-natsjs-4
Put three values for key1 into KV-STORE:
nats kv put KV-STORE 'key1' \
'{ "alpha": "23456", "beta": "1648583305" }'
nats kv put KV-STORE 'key1' \
'{ "alpha": "13890", "beta": "1648647305" }'
nats kv put KV-STORE 'key1' \
'{ "alpha": "23890", "beta": "1648594301" }'
The latest value of key1:
nats kv get KV-STORE 'key1'
KV-STORE > key1 created @ 18 Apr 22 01:20 UTC
{ "alpha": "23890", "beta": "1648594301" }
A history of key1:
nats kv history KV-STORE 'key1'
╭──────────────────────────────────────────────────────────────────────────────────────────╮
│ History for KV-STORE > key1 │
├──────┬──────────┬─────┬─────────────────────┬────────┬───────────────────────────────────┤
│ Key │ Revision │ Op │ Created │ Length │ Value │
├──────┼──────────┼─────┼─────────────────────┼────────┼───────────────────────────────────┤
│ key1 │ 1 │ PUT │ 17 Apr 22 18:20 PDT │ 42 │ { "alpha": "234... "1648583305" } │
│ key1 │ 2 │ PUT │ 17 Apr 22 18:20 PDT │ 42 │ { "alpha": "138... "1648647305" } │
│ key1 │ 3 │ PUT │ 17 Apr 22 18:20 PDT │ 42 │ { "alpha": "238... "1648594301" } │
╰──────┴──────────┴─────┴─────────────────────┴────────┴───────────────────────────────────╯
Watch the KV-STORE for any key updates:
nats kv watch KV-STORE
Watch the KV-STORE for any updates to key1:
nats kv watch KV-STORE 'key1'
Why is this cool?
Instant Key-Value stores that you can place anywhere in the world, accessible from any application you connect to NGS. JetStream provides easy to use Key-Value access methods as an abstraction over message stream.
Object Store
Each Object store is a JetStream with ordered messages of a subject (taken together) materializing an object (or blob). Object stores directly leverage blob-storage idioms like add, get, and ls. Watches can be placed on an Object store so clients can be evented any object adds or deletes!
Grab a file from the Synadia web site (or provide a file of your own):
curl -s https://synadia.com/static/OG.jpg > /tmp/ad1234.jpg
Create an Object store OBJ-STORE to hold your files:
nats object add OBJ-STORE \
--description 'Object store for my files' \
--max-bucket-size 10485760
Information for Object Store Bucket OBJ-STORE created 2022-04-17T18:28:47-07:00
Configuration:
Bucket Name: OBJ-STORE
Description: Object store for my files
Replicas: 1
TTL: unlimited
Sealed: false
Size: 0 B
Maximum Bucket Size: 10 MiB
Backing Store Kind: JetStream
JetStream Stream: OBJ_OBJ-STORE
Cluster Information:
Name: ngstest-az-uswest2
Leader: az-uswest2-natsjs-3
Upload your file to OBJ-STORE:
nats object put OBJ-STORE '/tmp/ad1234.jpg' --name 'ad1234.jpg'
104 KiB / 104 KiB [========================================================================================================]
Object information for OBJ-STORE > ad1234.jpg
Size: 104 KiB
Modification Time: 18 Apr 22 01:29 +0000
Chunks: 1
Digest: sha-256 a9e3e78fe3c796c3b9fd469230ccb33f124c8a9335d0025d9f941fc5349a
List files in OBJ-STORE:
nats object ls OBJ-STORE
╭──────────────────────────────────────────────────╮
│ Bucket Contents │
├────────────┬─────────┬───────────────────────────┤
│ Name │ Size │ Time │
├────────────┼─────────┼───────────────────────────┤
│ ad1234.jpg │ 104 KiB │ 2022-04-17T18:29:59-07:00 │
╰────────────┴─────────┴───────────────────────────╯
Download your file from OBJ-STORE:
nats object get OBJ-STORE 'ad1234.jpg' --output '/tmp/edgecache-ad1234.jpg'
104 KiB / 104 KiB [========================================================================================================]
Wrote: 104 KiB to /tmp/edgecache-ad1234.jpg in 0.00s
Monitor OBJ-STORE for new uploads or deletes:
nats object watch OBJ-STORE
Why is this cool?
Instant blob storage that you can place anywhere in the world, accessible from any application you connect to NGS! JetStream provides easy-to-use access methods as an Object store abstraction over message stream.