AWS S3 Performance

S3 是AWS的最常用的服务之一,但是对于S3的并发连接数,Amazon S3 宣布提高请求速率性能](https://aws.amazon.com/cn/about-aws/whats-new/2018/07/amazon-s3-announces-increased-request-rate-performance/)

那在大并发的场景下,如何监控S3的性能呢?下面的文章给出了解答。

S3 Best Performance

Your applications can easily achieve thousands of transactions per second in request performance when uploading and retrieving storage from Amazon S3. Amazon S3 automatically scales to high request rates. For example, your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in a bucket. There are no limits to the number of prefixes in a bucket. You can increase your read or write performance by parallelizing reads. For example, if you create 10 prefixes in an Amazon S3 bucket to parallelize reads, you could scale your read performance to 55,000 read requests per second.

image-20201028230839926

S3 控制台和都能找到对应的监控指标。Awesome!

额外的小议题:

我一度设想将S3 bucket 挂在到EKS的pods上,作为日志的输出目录,参照文献:利用 S3FS 将 S3 作为共享存储挂载到 Kubernetes Pod。但是此处存在一个问题,是一档pods中的容器重启后,日志文件将会被重新覆盖。

采用S3FS的一写注意事项:

  • 随机写或追加写需要重写整个文件;
  • 由于网络延迟,所以元数据的操作如列出目录等的性能较差;(备注: 在网络存储方案上,如EFS上,也会有同样的问题)
  • 最终一致性会暂时产生中间数据(AMAZON S3数据一致性模型);
  • 没有文件和目录的原子重命名;
  • 安装相同存储桶的多个客户端之间没有协调;
  • 没有硬链接;
Author: Chandler Kwok
Link: http://yoursite.com/2020/10/26/AWS-S3-Performance/
Copyright Notice: All articles in this blog are licensed under CC BY-NC-SA 4.0 unless stating additionally.