Kinesis — Data Streams, Firehose, Analytics & Video Streams

Tổng quan

Kinesis là platform streaming data của AWS. 4 service: Data Streams (ingest + process), Firehose (delivery), Data Analytics / Managed Flink (SQL/Flink trên stream), Video Streams (video real-time).

Kinesis Data Streams

• Shard-based: mỗi shard 1 MB/s in, 2 MB/s out, 1000 records/s in.
• On-demand mode: tự scale shard, không cần capacity planning.
• Retention: 24h mặc định, tối đa 365 ngày.
• Enhanced Fan-Out: mỗi consumer có 2 MB/s riêng (push via HTTP/2).
• KPL (Kinesis Producer Library): batching, aggregation, retry.
• KCL (Kinesis Client Library): checkpoint, shard balancing, exactly-once processing.

Kinesis Data Firehose

• Fully managed delivery: buffer (1 MB hoặc 60s) → target.
• Targets: S3, Redshift (qua S3 COPY), OpenSearch, Splunk, HTTP endpoint, 3rd party.
• Transform: Lambda function inline, convert JSON → Parquet/ORC.
• Không lưu data → không replay. Near real-time (không phải real-time).
• Compression: GZIP, Snappy, Zip cho S3 delivery.

Analytics & Video Streams

• Managed Service for Apache Flink (trước là Kinesis Data Analytics): chạy Flink job trên stream.
• Input: Kinesis Data Streams, MSK (Kafka). Output: stream, Firehose, Lambda.
• Video Streams: ingest video từ camera, xử lý với Rekognition, SageMaker.
• Use case video: smart home, industrial monitoring, traffic analysis.

Kinesis vs SQS vs MSK

• Kinesis: ordered per shard, multiple consumers, replay, real-time.
• SQS: queue, 1 consumer group, delete after consume, no replay.
• MSK (Managed Kafka): open-source compatible, topic-based, consumer groups, replay.