What types of Application communication are available?
Synchronous and Asynchronous
What synchronous models are available for application communication that are scalable?
- SQS - queue model
- SNS - pub/sub model
- Kinesis - real-time streaming model
What is SQS?
A synchronous application communication model that can have many producers (push) and consumers (poll and can consume up to 10 message at a time) from a single queue.
What are the pros and cons of and SQS Standard Queue?
Pros
- Fully managed by AWS
- scales from 1 message/sec to 10,000s/sec
- no message limit in queue
- low latency
- no consumer limit
- can delay individual messages
Cons
- Can contain duplicate messages
- Can process messages out of order
- Has limitation of 256kb/message
- Data is deleted after it is consumed
- no ordering gaurantee (except in FIFO queues)
What is an SQS delay queue?
This is a delay setting that allows a message to be hidden for up to 15 mins from consumers. It will override the delay delivery parameter that can be set on the individual message.
What is SQS visibility?
This is the time the message is visible by the consumer to pick up. Timeouts range from 0 sec and 12 hours (default 30 sec).
if set too low another consumer could receive the same message.
if set too high and consumer fails to process the message then you have to wait until after timeout to try again.
What is an SQS dead letter queue?
This is when the redrive policy (i.e. number of consumption tries) has been met and the message has still not been processed, the message is sent to the dead letter queue (DLQ). Note: You want to make sure you process the messages in the DLQ before they expire.
What is SQS Long Polling?
This is the process of a consumer waiting for a message to arrive in the queue. This can range from 1 - 20 secs and can be enabled at the queue or API level using the WaitTImeSeconds setting.
What is SQS FIFO Queue?
A newer offering that sets the queue to allow polling of First In-First Out messages.
- name of the queue must end in .fifo
- messages are processed in order by the consumer
- message are sent exactly once
- ability to do content based de-duplication at a 5min interval using Duplication ID.
- ability to group messages using Message Group ID (i.e. just an extra tag on the message)
Can SQS be used with ASG?
Yes, by setting the SQS Queue to poll for messages from a cluster of EC2 instances and setting up a CloudWatch custom metric based on Queue length/Number of instances, you can trigger a CloudWatch Alarm to autoscale.
What is AWS SNS?
This allows message to be sent to multiple consumers using a pub/sub model. The publisher sends the message to one SNS topic and the subscriber polls the SNS topic for notifications. Each subscriber to the topic will receive all the messages unless a filter is used.
What are the characteristics of SNS?
- up to 10,000,000 subscriptions/topic
- limit of 100,000 topics
- can push data to many subscribers
- Subscribers can use (SQS, HTTP/HTTPS, Lambda, Emails, SMS Messages, or Mobile Notifications)
- data is lost if not delivered
- does not require provisioned throughput
- can be used in conjunction with SQS where the SQS Queue is a subscriber to the SNS Topic. (called SQS Fan-out pattern)
What AWS services can SNS integrate with?
- CloudWatch
- Auto Scaling Groups
- S3 (on bucket events)
- Cloud Formation
How do we publish and SNS Model?
- Individual Topic publication
1. Create a topic
2. create one or more subscriptions
3. publish to the topic
or
*Direct publish (works with Google GCM, Apple APNS, Amazon ADM, etc.)
1. Create a platform application
2. Create a platform endpoint
3. Publish to the platform endpoint
What is AWS Kinesis?
A managed alternative to Apache Kafka. Great for real-time big data, logs, metrics, IoT, and clickstreams and is automatically replicated to 3 AZs.
- Consumers “pull data”
- no consumer limit
- Ability to replay data
- Can order data at the shard level
- Data expires after X days
- must provision throughput
What is Kinesis streams?
Low latency streaming that can ingest at scale
What is Kinesis Analytics?
- A real-time analytics tool using SQL
- No servers to provision as it is managed for you
- Ability to autoscale
- Pay for actual consuption rate
- Can create streams out of the real-time queries
What is Kinesis Firehose?
- Fully managed service (no admin, automaitc scaling & serverless)
- Loads data into S3, Redshift, ElasticSearch, Splunk.
- Near real-time (60sec latency for non full batches or min 32MB of data at a time)
- Supports many data formats, conversions, transformations and compression
- Pay for the amount of data going through Firehose only.
- No data storage
What are the characteristics of Kinesis streams?
- Streams are divided into Shards/Partitions
- Data retention is 1 day by default (up to 7 days)
- Ability to reprocess/replay data
- Multiple applications can consume the same stream
- Real-time processing with scale of throughput
- Data is immutable (cannot be deleted)
What are the characteristics of a Kinesis Stream Shard?
- One stream is mad up of many shards
- 1mb/sec or 100 message at write per shard
- 2mb/s at read per shard
- Billing is per provisioned shard
- Batching available or per message calls
- The number of shards can evolve over time (increase or decrease)
- Records are ordered per shard
What is an Kinesis API Put record?
- A record that contains the data plus a message key that is produced by hashing the record + partition key to determine a shard id.
- The same key goes to the same shard/partition which helps with ordering for a specific key by using a sequence number.
- Best to use a unique partition key such as user id to prevent a “hot partition”
- Can be used with batching to reduce costs
- Can use CLI, AWS SDK, or producer libraries from various frameworks
What is the AWS Kinesis exception?
- Provisioned Throughput exception happens when we send more data than the shard can handle.
Solutions:
- Avoid a bad partition key which could send too much data to the same shard.
- Implement retries with backoff
- Increase shards (using scaling)
What is Kinesis Security?
A means to control access/authorization using IAM policies
Characteristics include:
- Encryption in flight using HTTPS endpoints
- Encryption at rest using KMS
- Ability to encrypt/decrypt on client side (harder)
- Ability to use VPC endpoints to access within VPC
What is an Amazon MQ?
A managed Apache ActiveMQ that doesn’t scale as much as SNS or SQS. It runs on a dedicated machine (with HA failover if needed) and has both queue feature and topic features included.
