Limitation of Access to Internet-facing Elasticsearch Only from Lambda with IAM Role

Takahiro Iwasa

Apr 13, 2020

3 min read

Elasticsearch IAM Lambda

aws.amazon.com

Announcing Amazon OpenSearch Service which Supports OpenSearch 1.0 | Amazon Web Services

Developers embrace open-source software for many reasons. One of the most important reasons is the freedom to use that software where and how they want. On January 21, 2021, Elastic NV announced that they would change their software licensing strategy. After Elasticsearch version 7.10.2 and Kibana 7.10.2, they will not release new versions of Elasticsearch […]

Amazon Elasticsearch users should basically place Elasticsearch clusters within VPCs. However, this requires NAT Gateway or NAT instances, by which you would incur additional costs. When placing your clusters in public, you can use a Lambda function as a proxy to your Elasticsearch domain.

Install the following on you computer.

• AWS SAM CLI
• Python 3.x

/
|-- es-proxy-lambda/
|   |-- __init__.py
|   |-- lambda_function.py
|   `-- requirements.txt
|-- samconfig.toml
`-- template.yaml

Please note the following:

• An Elasticsearch domain limits access using AccessPolicies. (lines 8-16)
• A Lambda function needs appropriate policies to access the Elasticsearch domain. (lines 64-70)

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31

Resources:
  Elasticsearch:
    Type: AWS::Elasticsearch::Domain
    Properties:
      AccessPolicies:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              AWS:
                - !GetAtt IamRole.Arn
            Action: es:*
            Resource: !Sub arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/es-for-lambda/*
      DomainName: es-for-lambda
      EBSOptions:
        EBSEnabled: true
        VolumeSize: 10
        VolumeType: standard
      ElasticsearchClusterConfig:
        DedicatedMasterEnabled: false
        InstanceCount: 1
        InstanceType: t2.small.elasticsearch
      ElasticsearchVersion: 7.4

  Lambda:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: es-proxy-lambda/
      Environment:
        Variables:
          ES_DOMAIN: !GetAtt Elasticsearch.DomainEndpoint
      FunctionName: es_proxy_lambda
      Handler: lambda_function.lambda_handler
      Role: !GetAtt IamRole.Arn
      Runtime: python3.8

  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub
        - /aws/lambda/${name}
        - {name: !Ref Lambda}
      RetentionInDays: 1

  IamRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - es:ESHttpHead
                  - es:DescribeElasticsearchDomain
                  - es:ESHttpGet
                  - es:DescribeElasticsearchDomainConfig
                Resource: !Sub arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/es-for-lambda
          PolicyName: policy
      RoleName: es-proxy-lambda-role

The AWS Lambda runtime environment has boto3 installed by default, so there is no need to include it in your requirements.txt.

certifi==2019.11.28
chardet==3.0.4
elasticsearch==7.5.1
idna==2.9
requests==2.23.0
requests-aws4auth==0.9
urllib3==1.25.8

The following script uses requests_aws4auth to generate an Auth header including AWS credentials. (lines 10-11 and 14)

import os

import boto3
from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth


es_domain = os.environ.get('ES_DOMAIN')
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(
    credentials.access_key, credentials.secret_key, 'ap-northeast-1', 'es', session_token=credentials.token)
es = Elasticsearch(
    hosts=[{'host': es_domain, 'port': 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)


def lambda_handler(event, context):
    response = es.info()
    print(response)

Replace <YOUR_S3_BUCKET> with the actual value.

version = 0.1
[default]
[default.deploy]
[default.deploy.parameters]
stack_name = "es-proxy-lambda"
s3_bucket = "<YOUR_S3_BUCKET>"
s3_prefix = "es-proxy-lambda"
region = "ap-northeast-1"
capabilities = "CAPABILITY_IAM CAPABILITY_NAMED_IAM"

Build and deploy with the following command. Creating the Elasticsearch domain may take 10 to 20 minutes.

sam build
sam deploy

Execute the Lambda function. It should succeed to access the Elasticsearch domain.

Try to access the Elasticsearch domain using the following command. It should be rejected by the Elasticsearch domain.

$ curl https://search-es-for-lambda-rase3snu6yozl6xhjcuq34cu5m.ap-northeast-1.es.amazonaws.com/
{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet"}

Clean up the provisioned AWS resources with the following command.

sam delete --stack-name es-proxy-lambda

Using an IAM role attached to a Lambda function, we can start an Elasticsearch cluster in public and allow only the Lambda function to access.

Please note that when prioritizing security, Elasticsearch clusters should be placed essentially within a VPC.

I hope you will find this post useful.