Partitioning in Amazon Kinesis for Scalable and Reliable Streams

Amazon Kinesis is a powerful service for streaming real-time data, and partitioning plays a big role in ensuring your streams are scalable and perform well. Let’s take a closer look at partitioning and how you can use it effectively.

What’s a Shard in Amazon Kinesis?

In Kinesis, a shard is a unit of capacity within a stream. Each shard has:

• Ingestion Capacity: It can handle up to 1 MB per second or 1,000 records per second.
• Egress Capacity: It can output data at 2 MB per second.

This means that data is spread across multiple shards to balance the load and keep things running smoothly.

How Partitioning Works in Kinesis

Partitioning in Kinesis is done by dividing your data across shards using a partition key. Here’s the breakdown:

1. Every record you send to Kinesis has a partition key.
2. This partition key is hashed using an MD5 hash function.
3. The hash value determines which shard the record will be sent to.
4. If records share the same partition key, they will always go to the same shard.
5. If different partition keys are used, the records cannot guarantee always go to the same shard(depending on the capacity of shards and hash values).

Why Partition Keys Matter

Partition keys are important because they determine how your data is distributed across shards. Here’s why it’s crucial:

• Shards Have Limits: Each shard can handle a maximum of 1,000 records per second or 1 MB per second. If you exceed these limits, you’ll get a ProvisionedThroughputExceededException.
• For example, if you have 4 shards, you can scale this limit to 4,000 records per second or 4 MB per second. But, you need to ensure you’re using different partition keys to spread the load.
• Same Partition Key = Same Shard: If all your records use the same partition key, they will all end up on the same shard and could cause bottlenecks. This is where partition key rotation comes in to help distribute data more evenly.

Hot Shards: What Are They and How to Avoid Them?

A hot shard happens when one shard gets overloaded with too much data. This can cause performance issues, and you might get throttling or delays.

To avoid hot shards, you should use different partition keys to balance the data load. For instance, if you have a single process generating data, you can use unique values like a random string or some component from the record itself (e.g., a unique record ID).

Tips for Using Partition Keys Effectively

• Use Random Partition Keys:
• If you don’t need related data to be grouped together, you can generate random partition keys. This helps distribute the data evenly across all shards, preventing overload.
• Use Calculated Partition Keys:
• If you need related data (e.g., from the same customer or transaction) to be in the same shard, you can calculate partition keys based on attributes like customer ID or transaction type.

How Partition Keys Impact Throughput and Ordering

• Throughput Limits: Each partition key contributes to the overall write size of your stream. It counts against the total size, so be mindful of the length of the partition key. Using too much text could waste space. If you want to keep it random but efficient, you could calculate a hash from the key and use that instead.
• Ordering: One important thing to remember is that Kinesis guarantees ordering only within a single shard. However, that doesn’t always match the order your application might expect.
• Single-threaded clients: When using the PutRecord API, ordering is consistent between the client and partition.
• Multi-threaded clients: If you’re using multiple threads, the order can be affected by factors like thread scheduling or network routing.
• Batch writes with PutRecords API: Records in a batch may not be written in the exact order. If a shard is full, some records may be rejected and need to be retried.
• Resharding and Consistency: If you add or remove shards (resharding), the order of records could change when reading. This can cause potential inconsistencies, so it’s important to track the changes in shard structure carefully.

Best Practices for Scaling and Handling Data

1. Distribute Data Across Shards: Use unique partition keys to avoid writing all data to the same shard.
2. Track Shard Changes: When resharding occurs, ensure your system can handle the changes to maintain consistency and avoid data loss.
3. Use PutRecords for High Volume: When dealing with high volumes of data, use the PutRecords API, but be aware that records might be rejected if a shard is full.

4. Dynamic Resharding: Adjust the number of shards in your stream based on real-time data throughput to accommodate changing workloads.

5. Use Appropriate Capacity Mode: Kinesis Data Streams offers two capacity modes:

• On-Demand Mode: Automatically manages shard scaling based on data throughput, ideal for unpredictable workloads.
• Provisioned Mode: Allows manual specification of shard count, suitable for steady and predictable traffic patterns.

PutRecord vs. PutRecords

There are two main ways to write data to a Kinesis stream: PutRecord and PutRecords. Here's how they differ:

1. PutRecord

• Single Record: This API is used to write a single record at a time to a Kinesis stream.
• Use Case: Ideal for writing small amounts of data or for individual record writes.
• Throughput: Limited to 1,000 records per second or 1 MB per second per shard. Exceeding this may result in throttling.

Example:

response = kinesis_client.put_record(
StreamName='my-stream',
Data='my-record-data',
PartitionKey='my-partition-key'

2. PutRecords

• Multiple Records: This API allows you to write up to 500 records in a single batch request.
• Use Case: Perfect for high-volume data streams where you need to batch multiple records together.
• Throughput: Still Limited to 1,000 records per second or 1 MB per second per shard. Exceeding this may result in throttling. because this limit is related to shard.

Example:python

records = [
{'Data': 'data1', 'PartitionKey': 'key1'},
{'Data': 'data2', 'PartitionKey': 'key2'},
# More records here...
]
response = kinesis_client.put_records(
StreamName='my-stream',
Records=records
)

Key Differences:

• PutRecord is for single-record writes, whereas PutRecords allows batching multiple records in a single request.
• PutRecords is generally used for high-throughput applications, while PutRecord is suitable for smaller, individual writes.
• Both APIs ensure the data is hashed and assigned to the appropriate shard based on the partition key.

Summary

• Shards are containers in Kinesis that hold your records.
• The partition key decides which shard gets a record by hashing it.
• Different partition keys ensure data is distributed across multiple shards.
• Hot shards can be avoided by rotating partition keys and spreading the load.
• Ordering within a shard is guaranteed, but not across shards — be careful when relying on it.
• Use strategies like random keys or calculated keys depending on your need for data grouping.

With these tips, you’ll be able to scale your Kinesis streams efficiently and avoid common pitfalls like hot shards and throughput issues! Let me know if you need more details or have questions. 😊