Event-Driven AWS Glue Crawlers: Setting Up SQS-Based Triggers

Use S3 event notifications and SQS to run incremental AWS Glue crawler scans instead of repeatedly scanning an entire bucket.

Takahiro Iwasa

2023-01-03

4 min read

AWS Glue crawlers can use S3 event notifications to process only newly added or changed objects instead of scanning an entire bucket on every run, reducing crawl time and cost.

Configuration

The CloudFormation template below sets up the queue, bucket, and crawler:

1
AWSTemplateFormatVersion: '2010-09-09'
2
Description: Glue crawler test
3

4
Resources:
5
  SqsQueue:
6
    Type: AWS::SQS::Queue
7
    Properties:
8
      SqsManagedSseEnabled: true
9
      QueueName: glue-crawler-test-queue
10

11
  SqsQueuePolicy:
12
    Type: AWS::SQS::QueuePolicy
13
    Properties:
14
      Queues:
15
        - !Ref SqsQueue
16
      PolicyDocument:
17
        Version: '2008-10-17'
18
        Id: __default_policy_ID
19
        Statement:
20
          - Effect: Allow
21
            Principal:
22
              AWS:
23
                - !Sub arn:aws:iam::${AWS::AccountId}:root
24
                - !GetAtt IAMRoleGlueCrawler.Arn
25
            Action: sqs:*
26
            Resource: !GetAtt SqsQueue.Arn
27
          - Effect: Allow
28
            Principal:
29
              Service: s3.amazonaws.com
30
            Action: sqs:*
31
            Resource: !GetAtt SqsQueue.Arn
32

33
  S3Bucket:
34
    Type: AWS::S3::Bucket
35
    DependsOn: SqsQueuePolicy
36
    Properties:
37
      BucketName: !Sub glue-crawler-test-${AWS::AccountId}-${AWS::Region}
38
      BucketEncryption:
39
        ServerSideEncryptionConfiguration:
40
          - ServerSideEncryptionByDefault:
41
              SSEAlgorithm: AES256
42
      NotificationConfiguration:
43
        QueueConfigurations:
44
          - Event: 's3:ObjectCreated:*'
45
            Queue: !GetAtt SqsQueue.Arn
46
      PublicAccessBlockConfiguration:
47
        BlockPublicAcls: TRUE
48
        BlockPublicPolicy: TRUE
49
        IgnorePublicAcls: TRUE
50
        RestrictPublicBuckets: TRUE
51

52
  IAMRoleGlueCrawler:
53
    Type: AWS::IAM::Role
54
    Properties:
55
      Path: /service-role/
56
      RoleName: !Sub glue-crawler-test-service-role
57
      AssumeRolePolicyDocument:
58
        Version: '2012-10-17'
59
        Statement:
60
          - Effect: Allow
61
            Principal:
62
              Service: glue.amazonaws.com
63
            Action: sts:AssumeRole
64
      Policies:
65
        - PolicyName: cw-logs
66
          PolicyDocument:
67
            Version: '2012-10-17'
68
            Statement:
69
              - Effect: Allow
70
                Action:
71
                  - logs:CreateLogGroup
72
                  - logs:CreateLogStream
73
                  - logs:PutLogEvents
74
                Resource: "*"
75
        - PolicyName: glue
76
          PolicyDocument:
77
            Version: '2012-10-17'
78
            Statement:
79
              - Effect: Allow
80
                Action:
81
                  - glue:CreateTable
82
                  - glue:GetDatabase
83
                  - glue:GetTable
84
                  - glue:UpdateTable
85
                Resource: "*"
86
        - PolicyName: s3
87
          PolicyDocument:
88
            Version: '2012-10-17'
89
            Statement:
90
              - Effect: Allow
91
                Action:
92
                  - s3:GetObject
93
                  - s3:ListBucket
94
                  - s3:PutObject
95
                Resource: '*'
96

97
  GlueDatabase:
98
    Type: AWS::Glue::Database
99
    Properties:
100
      CatalogId: !Ref AWS::AccountId
101
      DatabaseInput:
102
        Name: glue-crawler-test-db
103

104
  GlueTable:
105
    Type: AWS::Glue::Table
106
    Properties:
107
      CatalogId: !Ref AWS::AccountId
108
      DatabaseName: !Ref GlueDatabase
109
      TableInput:
110
        Name: glue-crawler-test-table
111
        TableType: EXTERNAL_TABLE
112
        Parameters:
113
          classification: json
114
        StorageDescriptor:
115
          Location: !Sub 's3://${S3Bucket}/'
116
          Compressed: false
117
          InputFormat: org.apache.hadoop.mapred.TextInputFormat
118
          OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
119
          SerdeInfo:
120
            SerializationLibrary: org.openx.data.jsonserde.JsonSerDe
121

122
  GlueCrawler:
123
    Type: AWS::Glue::Crawler
124
    Properties:
125
      Name: glue-crawler-test
126
      Role: !Sub service-role/${IAMRoleGlueCrawler}
127
      Targets:
128
        CatalogTargets:
129
          - DatabaseName: !Ref GlueDatabase
130
            Tables:
131
              - !Ref GlueTable
132
      SchemaChangePolicy:
133
        UpdateBehavior: UPDATE_IN_DATABASE
134
        DeleteBehavior: LOG

Deploy the stack:

STACK_NAME=glue-crawler-test

aws cloudformation package \
  --template-file template.yaml \
  --s3-bucket <YOUR_CFN_BUCKET> \
  --s3-prefix "$STACK_NAME/$(date +%Y)/$(date +%m)/$(date +%d)/$(date +%H)/$(date +%M)" \
  --output-template-file package.template

aws cloudformation deploy \
  --stack-name $STACK_NAME \
  --template-file package.template \
  --capabilities CAPABILITY_NAMED_IAM

Updating the Glue Crawler Target Setting

As of January 2023, CloudFormation does not support configuring S3 event notifications for Glue crawlers. Update the crawler target manually:

https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/947

CloudFormation Support For S3 Event isn’t currently available. S3 Event Crawler’s integration with CloudFormation is in scope and in the works. We plan on releasing this coverage to Cloudformation some later this year. Thank you for patience.

Select the S3 target and click Edit.

Choose Crawl based on events in the Subsequent crawler runs section.

Testing

Uploading Version 1

Upload a sample JSON file to trigger an S3 event notification:

echo '{"message": "Hello World"}' > sample1.json
aws s3 cp sample1.json s3://glue-crawler-test-<ACCOUNT_ID>-<REGION>/

Start the Glue crawler:

aws glue start-crawler --name glue-crawler-test

Monitor its status until it shows STOPPING:

$ aws glue get-crawler --name glue-crawler-test | jq -r '.Crawler.State'
STOPPING

Verify the updated Glue table schema:

aws glue get-table \
  --database-name glue-crawler-test-db \
  --name glue-crawler-test-table \
| jq '.Table.StorageDescriptor.Columns'

Expected output:

1
[
2
  {
3
    "Name": "message",
4
    "Type": "string"
5
  }
6
]

Uploading Version 2

Repeat the process with a new JSON file version:

echo '{"message": "Hello World", "statusCode": 200}' > sample2.json
aws s3 cp sample2.json s3://glue-crawler-test-<ACCOUNT_ID>-<REGION>/

Start the Glue crawler:

aws glue start-crawler --name glue-crawler-test

Monitor its status until it shows STOPPING:

$ aws glue get-crawler --name glue-crawler-test | jq -r '.Crawler.State'
STOPPING

Verify the updated Glue table schema:

aws glue get-table \
  --database-name glue-crawler-test-db \
  --name glue-crawler-test-table \
| jq '.Table.StorageDescriptor.Columns'

Expected output:

1
[
2
  {
3
    "Name": "message",
4
    "Type": "string"
5
  },
6
  {
7
    "Name": "statuscode",
8
    "Type": "int"
9
  }
10
]

Checking SQS Message Count

Verify that no messages are left in the SQS queue:

queue_url=$(aws sqs get-queue-url --queue-name glue-crawler-test-queue | jq -r '.QueueUrl')
aws sqs get-queue-attributes \
  --queue-url $queue_url \
  --attribute-names ApproximateNumberOfMessages

1
{
2
    "Attributes": {
3
        "ApproximateNumberOfMessages": "0"
4
    }
5
}

Cleaning Up

Remove the resources provisioned by this example with:

aws s3 rm s3://glue-crawler-test-<ACCOUNT_ID>-<REGION>/ --recursive
aws cloudformation delete-stack --stack-name $STACK_NAME

Conclusion

Sending S3 event notifications through SQS to a Glue crawler allows successive file uploads to update the table schema incrementally. The empty SQS queue confirms that the crawler consumed the notifications.

With event-based crawling enabled, the crawler processes the objects referenced by the notifications rather than rescanning the entire bucket. This keeps runtime and cost lower as the dataset grows.

In January 2023, switching Subsequent crawler runs to Crawl based on events still requires the console because CloudFormation does not expose the setting. Track the linked GitHub issue if the configuration must be managed entirely as code.

Back to posts