Avoiding Common Pitfalls with s3:TestEvent in AWS S3 Notifications

When configuring event notifications for S3 buckets, s3:TestEvent messages are automatically sent by S3. If this test message is not handled properly, it may cause unexpected issues.

Takahiro Iwasa

2020-12-29

2 min read

When configuring event notifications for S3 buckets, s3:TestEvent messages are automatically sent by S3. If this test message is not handled properly, it may cause unexpected issues.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html

When you configure an event notification on a bucket, Amazon S3 sends a test message with the s3:TestEvent.

Building

Create a CloudFormation stack template:

1
AWSTemplateFormatVersion: "2010-09-09"
2
Description: Example of CloudWatch events not queueing to SSE SQS
3
Resources:
4
  Bucket:
5
    Type: AWS::S3::Bucket
6
    Properties:
7
      BucketEncryption:
8
        ServerSideEncryptionConfiguration:
9
          - ServerSideEncryptionByDefault:
10
              SSEAlgorithm: AES256
11
      NotificationConfiguration:
12
        QueueConfigurations:
13
          - Event: 's3:ObjectCreated:Put'
14
            Queue: !GetAtt Queue.Arn
15
      PublicAccessBlockConfiguration:
16
        BlockPublicAcls: true
17
        BlockPublicPolicy: true
18
        IgnorePublicAcls: true
19
        RestrictPublicBuckets: true
20

21
  Queue:
22
    Type: AWS::SQS::Queue
23
    Properties:
24
      QueueName: s3-event-notification-test-queue
25
      ReceiveMessageWaitTimeSeconds: 20
26

27
  QueuePolicy:
28
    Type: AWS::SQS::QueuePolicy
29
    Properties:
30
      PolicyDocument:
31
        Version: '2008-10-17'
32
        Statement:
33
          - Effect: Allow
34
            Principal:
35
              Service: s3.amazonaws.com
36
            Action:
37
              - SQS:SendMessage
38
              - SQS:ReceiveMessage
39
            Resource: !GetAtt Queue.Arn
40
            Condition:
41
              StringEquals:
42
                aws:SourceAccount: !Ref AWS::AccountId
43
      Queues:
44
        - !Ref Queue

Deploy the stack:

aws cloudformation deploy \
  --template-file template.yaml \
  --stack-name s3-event-notification-test

Testing

The SQS messages confirm the setup is working:

aws sqs receive-message \
  --queue-url https://sqs.ap-northeast-1.amazonaws.com/{AccountId}/s3-event-notification-test-queue

You should observe the s3:TestEvent message in the output, even if no objects have been added to the bucket.

1
{
2
  "Messages": [
3
    {
4
      "MessageId": "...",
5
      "ReceiptHandle": "...",
6
      "MD5OfBody": "...",
7
      "Body": "{\"Service\":\"Amazon S3\",\"Event\":\"s3:TestEvent\",\"Time\":\"2020-12-29T18:53:47.874Z\",\"Bucket\":\"s3-event-notification-test-bucket-xxxxxxxx\",\"RequestId\":\"...\",\"HostId\":\"...\"}"
8
    }
9
  ]
10
}

Cleaning Up

Delete the stack afterward:

aws cloudformation delete-stack --stack-name s3-event-notification-test

Conclusion

Deploying an S3 bucket with an SQS event notification and polling the queue showed an s3:TestEvent message arriving before any object was ever uploaded. The NotificationConfiguration in this template only subscribes to s3:ObjectCreated:Put, yet that very first message has no matching Event field at all — S3 sends it automatically the moment the notification configuration is attached, regardless of which event types were actually requested. Any Lambda or application code consuming this queue needs to check the Event field and either skip or explicitly handle s3:TestEvent, since treating it like a real object-created payload will produce a parsing error the first time the stack is deployed rather than when actual objects start arriving.

Back to posts