Deploying Containers on EKS Fargate in Private Subnets Behind an ALB

Run containers on EKS Fargate in private subnets and expose the application only through an internal ALB.

Takahiro Iwasa

2022-12-24

15 min read

Running containers on EKS Fargate within private subnets, managed behind an Application Load Balancer (ALB), keeps the cluster’s pods off the public internet while still exposing a controlled entry point.

Setting Up the VPC

Creating the VPC

Create a dedicated VPC:

aws ec2 create-vpc \
  --cidr-block 192.168.0.0/16 \
  --tag-specifications "ResourceType=vpc,Tags=[{Key=Name,Value=eks-fargate-vpc}]"

aws ec2 modify-vpc-attribute \
  --vpc-id vpc-xxxxxxxxxxxxxxxxx \
  --enable-dns-hostnames

❗ Important

Please make sure to enable DNS hostnames for VPC endpoints. For more details, please refer to an official documentation.

If you use custom DNS domain names defined in a private hosted zone in Amazon Route 53, or use private DNS with interface VPC endpoints (AWS PrivateLink), you must set both the enableDnsHostnames and enableDnsSupport attributes to true.

Adding Subnets

Create private subnets for Fargate pods and a public subnet for the bastion EC2 instance.

aws ec2 create-subnet \
  --vpc-id vpc-xxxxxxxxxxxxxxxxx \
  --availability-zone ap-northeast-1a \
  --cidr-block 192.168.0.0/20 \
  --tag-specifications "ResourceType=subnet,Tags=[{Key=Name,Value=eks-fargate-private-subnet-1a}]"

aws ec2 create-subnet \
  --vpc-id vpc-xxxxxxxxxxxxxxxxx \
  --availability-zone ap-northeast-1c \
  --cidr-block 192.168.16.0/20 \
  --tag-specifications "ResourceType=subnet,Tags=[{Key=Name,Value=eks-fargate-private-subnet-1c}]"

aws ec2 create-subnet \
  --vpc-id vpc-xxxxxxxxxxxxxxxxx \
  --availability-zone ap-northeast-1a \
  --cidr-block 192.168.32.0/20 \
  --tag-specifications "ResourceType=subnet,Tags=[{Key=Name,Value=eks-fargate-public-subnet-1a}]"

Adding Internet Gateway

To enable internet access for resources in the public subnet, create an Internet Gateway and attach it to your VPC:

aws ec2 create-internet-gateway \
  --tag-specifications "ResourceType=internet-gateway,Tags=[{Key=Name,Value=igw-eks-fargate}]"

aws ec2 attach-internet-gateway \
  --internet-gateway-id igw-xxxxxxxxxxxxxxxxx \
  --vpc-id vpc-xxxxxxxxxxxxxxxxx

Next, create a route table, add a default route through the Internet Gateway, and associate the route table with the public subnet:

aws ec2 create-route-table \
  --vpc-id vpc-xxxxxxxxxxxxxxxxx \
  --tag-specifications "ResourceType=route-table,Tags=[{Key=Name,Value=rtb-eks-fargate-public}]"

aws ec2 create-route \
  --route-table-id rtb-xxxxxxxx \
  --destination-cidr-block 0.0.0.0/0 \
  --gateway-id igw-xxxxxxxxxxxxxxxxx

aws ec2 associate-route-table \
  --route-table-id rtb-xxxxxxxx \
  --subnet-id subnet-xxxxxxxxxxxxxxxxx

Adding VPC Endpoints

To enable secure communication for an EKS private cluster, create the necessary VPC endpoints. Refer to the official documentation for detailed information.

Type	Endpoint
Interface	com.amazonaws.region-code.ecr.api
Interface	com.amazonaws.region-code.ecr.dkr
Interface	com.amazonaws.region-code.ec2
Interface	com.amazonaws.region-code.elasticloadbalancing
Interface	com.amazonaws.region-code.sts
Gateway	com.amazonaws.region-code.s3

Create a security group for the VPC endpoints:

aws ec2 create-security-group \
  --description "VPC endpoints" \
  --group-name eks-fargate-vpc-endpoints-sg \
  --vpc-id vpc-xxxxxxxxxxxxxxxxx \
  --tag-specifications "ResourceType=security-group,Tags=[{Key=Name,Value=eks-fargate-vpc-endpoints-sg}]"

aws ec2 authorize-security-group-ingress \
  --group-id sg-xxxxxxxxxxxxxxxxx \
  --protocol tcp \
  --port 443 \
  --cidr 192.168.0.0/16

Create the interface VPC endpoints:

for name in com.amazonaws.<REGION>.ecr.api com.amazonaws.<REGION>.ecr.dkr com.amazonaws.<REGION>.ec2 com.amazonaws.<REGION>.elasticloadbalancing com.amazonaws.<REGION>.sts; do \
aws ec2 create-vpc-endpoint \
  --vpc-id vpc-xxxxxxxxxxxxxxxxx \
  --vpc-endpoint-type Interface \
  --service-name $name \
  --security-group-ids sg-xxxxxxxxxxxxxxxxx \
  --subnet-ids subnet-xxxxxxxxxxxxxxxxx subnet-xxxxxxxxxxxxxxxxx;
done;

Create the gateway VPC endpoint for S3:

aws ec2 create-vpc-endpoint \
  --vpc-id vpc-xxxxxxxxxxxxxxxxx \
  --service-name com.amazonaws.<REGION>.s3 \
  --route-table-ids rtb-xxxxxxxxxxxxxxxxx

By adding these endpoints, your private cluster can securely access AWS services such as ECR, S3, and Elastic Load Balancing.

Bastion EC2 Instance

Use a bastion EC2 instance to access the EKS private cluster. The instance can reach the Kubernetes API server from within the VPC after public endpoint access is disabled.

https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html#private-access

If you have disabled public access for your cluster’s Kubernetes API server endpoint, you can only access the API server from within your VPC or a connected network.

Creating an Instance IAM Role

Create an IAM role for the bastion instance and attach the AmazonSSMManagedInstanceCore managed policy for Session Manager access.

Create an IAM role:

echo '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}' > policy.json

aws iam create-role \
  --role-name eks-fargate-bastion-ec2-role \
  --assume-role-policy-document file://./policy.json

Create an instance profile:

aws iam create-instance-profile \
  --instance-profile-name eks-fargate-bastion-ec2-instance-profile

aws iam add-role-to-instance-profile \
  --instance-profile-name eks-fargate-bastion-ec2-instance-profile \
  --role-name eks-fargate-bastion-ec2-role

Attach the AmazonSSMManagedInstanceCore policy to allow Session Manager access:

aws iam attach-role-policy \
  --role-name eks-fargate-bastion-ec2-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

Attach an additional policy for the EKS, EC2, IAM, and CloudFormation operations used in this example. For a production environment, narrow these permissions according to the service authorization reference.

echo '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudformation:CreateStack",
                "cloudformation:DeleteStack",
                "cloudformation:DescribeStacks",
                "cloudformation:DescribeStackEvents",
                "cloudformation:ListStacks",
                "ec2:*",
                "eks:*",
                "iam:AttachRolePolicy",
                "iam:CreateOpenIDConnectProvider",
                "iam:CreateRole",
                "iam:DetachRolePolicy",
                "iam:DeleteOpenIDConnectProvider",
                "iam:GetOpenIDConnectProvider",
                "iam:GetRole",
                "iam:ListPolicies",
                "iam:PassRole",
                "iam:PutRolePolicy",
                "iam:TagOpenIDConnectProvider"
            ],
            "Resource": "*"
        }
    ]
}' > policy.json

aws iam put-role-policy \
  --role-name eks-fargate-bastion-ec2-role \
  --policy-name eks-cluster \
  --policy-document file://./policy.json

Starting the Bastion EC2 Instance

After configuring the IAM role, launch the EC2 instance. Replace the example AMI ID with an Amazon Linux AMI available in your region; see the official documentation for details.

instanceProfileRole=$( \
aws iam list-instance-profiles-for-role \
  --role-name eks-fargate-bastion-ec2-role \
| jq -r '.InstanceProfiles[0].Arn')

aws ec2 run-instances \
  --image-id ami-0bba69335379e17f8 \
  --instance-type t2.micro \
  --iam-instance-profile "Arn=$instanceProfileRole" \
  --subnet-id subnet-xxxxxxxxxxxxxxxxx \
  --associate-public-ip-address \
  --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=eks-fargate-bastion-ec2}]"

Connecting to the Instance with Session Manager

Connect to the bastion EC2 instance with AWS Systems Manager Session Manager. This avoids the need for an SSH key pair and provides auditable session access.

After connecting, switch to the ec2-user account:

sh-4.2$ sudo su - ec2-user

❗ Important

Ensure that the instance IAM role has the AmazonSSMManagedInstanceCore policy attached for Session Manager connectivity.

Updating the AWS CLI

Update the AWS CLI on the bastion instance:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install --bin-dir /usr/local/bin --install-dir /usr/local/aws-cli --update

Verify the installation:

aws --version

Installing `kubectl`

To manage your EKS cluster, install kubectl on the bastion instance.

Download the kubectl binary for your EKS cluster version:

curl -o kubectl https://s3.us-west-2.amazonaws.com/amazon-eks/1.24.7/2022-10-31/bin/linux/amd64/kubectl

Make the binary executable:

chmod +x ./kubectl

Add kubectl to your PATH:

mkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && export PATH=$PATH:$HOME/bin
echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc

Verify the installation:

kubectl version --short --client

Installing `eksctl`

Install eksctl to create and manage the EKS cluster.

Download and extract eksctl:

curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp

Move the binary to a location in your PATH:

sudo mv /tmp/eksctl /usr/local/bin

Verify the installation:

eksctl version

The bastion instance can now manage the EKS cluster with kubectl and eksctl.

EKS

Creating the EKS Cluster

Create an EKS cluster with the eksctl --fargate option. Fargate runs the pods without self-managed or managed node groups.

Refer to the official documentation for detailed instructions.

ℹ️ Note

Creating the cluster may take approximately 20 minutes or more.

eksctl create cluster \
  --name eks-fargate-cluster \
  --region ap-northeast-1 \
  --version 1.24 \
  --vpc-private-subnets subnet-xxxxxxxxxxxxxxxxx,subnet-xxxxxxxxxxxxxxxxx \
  --without-nodegroup \
  --fargate

After creation, verify the cluster:

$ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   20m

Appendix: Troubleshooting Cluster Access

Issue 1: Credential Error

If kubectl get svc returns the following error:

1
Unable to connect to the server: getting credentials: decoding stdout: no kind "ExecCredential" is registered for version "client.authentication.k8s.io/v1alpha1" in scheme "pkg/client/auth/exec/exec.go:62"

Update the AWS CLI to the latest version:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install --bin-dir /usr/local/bin --install-dir /usr/local/aws-cli --update

Retry the command:

$ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   20m

Issue 2: Connection Refused

If the command returns the following connection error:

1
The connection to the server localhost:8080 was refused - did you specify the right host or port?

Update the Kubernetes configuration file (~/.kube/config):

aws eks update-kubeconfig \
  --region ap-northeast-1 \
  --name eks-fargate-cluster

Retry the command:

$ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   20m

Adding IAM Users and Roles

Grant access to additional IAM users or roles so cluster administration does not depend on the creator’s IAM identity. By default, only the IAM entity that created the cluster has administrative access.

Refer to the official documentation for best practices.

The IAM user or role that created the cluster is the only IAM entity that has access to the cluster. Grant permissions to other IAM users or roles so they can access your cluster.

To add an IAM user to the system:masters group, use the following command:

eksctl create iamidentitymapping \
  --cluster eks-fargate-cluster \
  --region ap-northeast-1 \
  --arn arn:aws:iam::000000000000:user/xxxxxx \
  --group system:masters \
  --no-duplicate-arns

The mapped IAM identity now has administrative access to the EKS cluster.

Enabling the Private Cluster Endpoint

Enable private endpoint access and disable public endpoint access to restrict the Kubernetes API to the VPC and connected networks.

ℹ️ Note

Enabling the private cluster endpoint may take about 10 minutes.

aws eks update-cluster-config \
  --region ap-northeast-1 \
  --name eks-fargate-cluster \
  --resources-vpc-config endpointPublicAccess=false,endpointPrivateAccess=true

Ensure that your EKS control plane security group allows ingress traffic on port 443 from your bastion EC2 instance.

https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html#private-access

You must ensure that your Amazon EKS control plane security group contains rules to allow ingress traffic on port 443 from your bastion host.

sgId=$(aws eks describe-cluster --name eks-fargate-cluster | jq -r .cluster.resourcesVpcConfig.clusterSecurityGroupId)
aws ec2 authorize-security-group-ingress \
  --group-id $sgId \
  --protocol tcp \
  --port 443 \
  --cidr 192.168.0.0/16

Test the connectivity between the bastion EC2 instance and the EKS cluster:

$ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.100.0.1   <none>        443/TCP   153m

Creating a Fargate Profile

Create a Fargate profile for the application’s namespace:

eksctl create fargateprofile \
  --region ap-northeast-1 \
  --cluster eks-fargate-cluster  \
  --name fargate-app-profile \
  --namespace fargate-app

Installing AWS Load Balancer Controller

Install the AWS Load Balancer Controller to run application containers behind an Application Load Balancer (ALB).

Create an IAM OIDC provider for the cluster if it does not already exist:

oidc_id=$(aws eks describe-cluster --name eks-fargate-cluster --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)
aws iam list-open-id-connect-providers | grep $oidc_id

# If no response is returned, run the following:
eksctl utils associate-iam-oidc-provider \
  --region ap-northeast-1 \
  --cluster eks-fargate-cluster \
  --approve

Download the policy file for the AWS Load Balancer Controller:

curl -o iam_policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.4.4/docs/install/iam_policy.json

Create the IAM policy:

aws iam create-policy \
  --policy-name AWSLoadBalancerControllerIAMPolicy \
  --policy-document file://iam_policy.json

Create the IAM service account:

eksctl create iamserviceaccount \
  --region ap-northeast-1 \
  --cluster=eks-fargate-cluster \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --role-name "AmazonEKSLoadBalancerControllerRole" \
  --attach-policy-arn=arn:aws:iam::111122223333:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve

Installing Helm and the Load Balancer Controller

Install Helm v3:

$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 > get_helm.sh
$ chmod 700 get_helm.sh
$ ./get_helm.sh
$ helm version --short | cut -d + -f 1
v3.10.3

Install the Load Balancer Controller add-on:

helm repo add eks https://aws.github.io/eks-charts
helm repo update
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set region=ap-northeast-1 \
  --set vpcId=vpc-xxxxxxxxxxxxxxxxx \
  --set image.repository=602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/amazon/aws-load-balancer-controller \
  --set clusterName=eks-fargate-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set enableShield=false \
  --set enableWaf=false \
  --set enableWafv2=false

ℹ️ Note

You need to add enableShield=false, enableWaf=false, and enableWafv2=false to the command because VPC endpoints are not currently provided. For more information, please refer to the official documentation.

When deploying it, you should use command line flags to set enable-shield, enable-waf, and enable-wafv2 to false. Certificate discovery with hostnames from Ingress objects isn’t supported. This is because the controller needs to reach AWS Certificate Manager, which doesn’t have a VPC interface endpoint.

Verify the deployment:

$ kubectl get deployment -n kube-system aws-load-balancer-controller
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
aws-load-balancer-controller   2/2     2            2           105s

The cluster can now provision an Application Load Balancer for the application.

Tagging Subnets

Tag the private subnets to indicate their use for internal load balancers. This is required for Kubernetes and the AWS Load Balancer Controller to identify the subnets correctly.

aws ec2 create-tags \
  --resources subnet-xxxxxxxxxxxxxxxxx subnet-xxxxxxxxxxxxxxxxx \
  --tags Key=kubernetes.io/role/internal-elb,Value=1

Refer to the official documentation for additional details.

Must be tagged in the following format. This is so that Kubernetes and the AWS load balancer controller know that the subnets can be used for internal load balancers.

Deploying Application

Building Application

This example uses FastAPI to create a small demonstration API.

Define the necessary dependencies for the application:

1
anyio==3.6.2
2
click==8.1.3
3
fastapi==0.88.0
4
h11==0.14.0
5
httptools==0.5.0
6
idna==3.4
7
pydantic==1.10.2
8
python-dotenv==0.21.0
9
PyYAML==6.0
10
sniffio==1.3.0
11
starlette==0.22.0
12
typing_extensions==4.4.0
13
uvicorn==0.20.0
14
uvloop==0.17.0
15
watchfiles==0.18.1
16
websockets==10.4

Create a basic API endpoint:

1
from fastapi import FastAPI
2

3
app = FastAPI()
4

5

6
@app.get('/')
7
def read_root():
8
    return {'message': 'Hello world!'}

Create a Dockerfile to build the application container:

1
FROM python:3.10-alpine@sha256:d8a484baabf7d2337d34cdef6730413ea1feef4ba251784f9b7a8d7b642041b3
2
COPY ./src ./
3
RUN pip install --no-cache-dir -r requirements.txt
4
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "80"]

Pushing the Image to ECR

Build and push the application image to ECR:

Create an ECR repository:

aws ecr create-repository --repository-name api

Retrieve the repository URI:

uri=$(aws ecr describe-repositories | jq -r '.repositories[] | select(.repositoryName == "api") | .repositoryUri')

Authenticate Docker to ECR:

aws ecr get-login-password --region ap-northeast-1 | docker login --username AWS --password-stdin 000000000000.dkr.ecr.ap-northeast-1.amazonaws.com

Build, tag, and push the image:

docker build .
docker tag xxxxxxxxxxxx $uri:latest
docker push $uri:latest

Deploying to Fargate

Create a Kubernetes manifest file fargate-app.yaml.

Replace 000000000000.dkr.ecr.ap-northeast-1.amazonaws.com/api:latest with the actual image URI.

For more information about the AWS Load Balancer Controller v2.4 specification, refer to the official documentation.

1
---
2
apiVersion: v1
3
kind: Namespace
4
metadata:
5
  name: fargate-app
6
---
7
apiVersion: apps/v1
8
kind: Deployment
9
metadata:
10
  name: fargate-app-deployment
11
  namespace: fargate-app
12
  labels:
13
    app: api
14
spec:
15
  replicas: 1
16
  selector:
17
    matchLabels:
18
      app: api
19
  template:
20
    metadata:
21
      labels:
22
        app: api
23
    spec:
24
      affinity:
25
        nodeAffinity:
26
          requiredDuringSchedulingIgnoredDuringExecution:
27
            nodeSelectorTerms:
28
              - matchExpressions:
29
                - key: kubernetes.io/arch
30
                  operator: In
31
                  values:
32
                    - amd64
33
      containers:
34
        - name: api
35
          image: 000000000000.dkr.ecr.ap-northeast-1.amazonaws.com/api:latest
36
          imagePullPolicy: IfNotPresent
37
          ports:
38
            - name: http
39
              containerPort: 80
40
      nodeSelector:
41
        kubernetes.io/os: linux
42
---
43
apiVersion: v1
44
kind: Service
45
metadata:
46
  name: fargate-app-service
47
  namespace: fargate-app
48
  labels:
49
    app: api
50
spec:
51
  selector:
52
    app: api
53
  ports:
54
    - protocol: TCP
55
      port: 80
56
      targetPort: 80
57
  type: NodePort
58
---
59
apiVersion: networking.k8s.io/v1
60
kind: Ingress
61
metadata:
62
  name: fargate-app-ingress
63
  namespace: fargate-app
64
  annotations:
65
    alb.ingress.kubernetes.io/scheme: internal
66
    alb.ingress.kubernetes.io/target-type: ip
67
spec:
68
  ingressClassName: alb
69
  rules:
70
    - http:
71
        paths:
72
          - path: /
73
            pathType: Prefix
74
            backend:
75
              service:
76
                name: fargate-app-service
77
                port:
78
                  number: 80

Apply the manifest file:

kubectl apply -f fargate-app.yaml

Verify the deployed resources:

$ kubectl get all -n fargate-app
NAME                                          READY   STATUS    RESTARTS   AGE
pod/fargate-app-deployment-6db55f9b7b-4hp8z   1/1     Running   0          55s

NAME                          TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/fargate-app-service   NodePort   10.100.190.97   <none>        80:31985/TCP   6m

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/fargate-app-deployment   1/1     1            1           6m

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/fargate-app-deployment-6db55f9b7b   1         1         1       6m

ℹ️ Note

Provisioning ALB may take about ten minutes or longer.

Testing the API

Retrieve the DNS name of the ALB:

kubectl describe ingress -n fargate-app fargate-app-ingress

Example output:

1
Name:             fargate-app-ingress
2
Labels:           <none>
3
Namespace:        fargate-app
4
Address:          internal-k8s-fargatea-fargatea-0579eb4ce2-1731550123.ap-northeast-1.elb.amazonaws.com
5
Ingress Class:    alb
6
Default backend:  <default>
7
Rules:
8
  Host        Path  Backends
9
  ----        ----  --------
10
  *
11
              /   fargate-app-service:80 (192.168.4.97:80)
12
Annotations:  alb.ingress.kubernetes.io/scheme: internal
13
              alb.ingress.kubernetes.io/target-type: ip
14
Events:
15
  Type    Reason                  Age    From     Message
16
  ----    ------                  ----   ----     -------
17
  Normal  SuccessfullyReconciled  4m17s  ingress  Successfully reconciled

Test the API endpoint:

curl internal-k8s-fargatea-fargatea-xxxxxxxxxx-xxxxxxxxxx.ap-northeast-1.elb.amazonaws.com

Expected output:

1
{"message":"Hello world!"}

Deleting EKS Cluster

When the EKS cluster and its associated resources are no longer needed, delete them as follows.

Remove the deployed application and uninstall the AWS Load Balancer Controller:

kubectl delete -f fargate-app.yaml
helm uninstall aws-load-balancer-controller -n kube-system

Retrieve the ARN of the AWSLoadBalancerControllerIAMPolicy and detach it:

arn=$(aws iam list-policies --scope Local \
| jq -r '.Policies[] | select(.PolicyName == "AWSLoadBalancerControllerIAMPolicy").Arn')

aws iam detach-role-policy \
  --role-name AmazonEKSLoadBalancerControllerRole \
  --policy-arn $arn

Delete the service account associated with the AWS Load Balancer Controller:

eksctl delete iamserviceaccount \
  --region ap-northeast-1 \
  --cluster eks-fargate-cluster \
  --namespace kube-system \
  --name aws-load-balancer-controller

Remove Fargate profiles created during the setup:

aws eks delete-fargate-profile \
  --cluster-name eks-fargate-cluster \
  --fargate-profile-name fargate-app-profile

aws eks delete-fargate-profile \
  --cluster-name eks-fargate-cluster \
  --fargate-profile-name fp-default

Retrieve and detach the AmazonEKSFargatePodExecutionRolePolicy:

arn=$(aws iam list-policies --scope AWS \
| jq -r '.Policies[] | select(.PolicyName == "AmazonEKSFargatePodExecutionRolePolicy").Arn')

aws iam detach-role-policy \
  --role-name eksctl-eks-fargate-cluster-FargatePodExecutionRole-xxxxxxxxxxxxx \
  --policy-arn $arn

Use eksctl to delete the cluster:

eksctl delete cluster \
  --region ap-northeast-1 \
  --name eks-fargate-cluster

Appendix: Troubleshooting Deletion Issues

If deleting the AWS Load Balancer Controller ingress fails, you may need to remove its finalizers manually, as described here:

kubectl patch ingress fargate-app-ingress -n fargate-app -p '{"metadata":{"finalizers":[]}}' --type=merge

This command ensures that Kubernetes can finalize the ingress resource for deletion.

Conclusion

A private-subnet VPC, an EKS Fargate cluster, and the AWS Load Balancer Controller run the FastAPI container behind an internal ALB. The --fargate option on eksctl create cluster removes the need to manage worker nodes, while the alb.ingress.kubernetes.io/scheme: internal Ingress annotation keeps the application entry point off the public internet.

The private subnets must have the kubernetes.io/role/internal-elb=1 tag so the Load Balancer Controller can discover subnets for the internal ALB.

After the VPC endpoints, bastion host, and controller are in place, additional services can reuse the same networking and IAM foundation and be deployed with a manifest and kubectl apply.

Back to posts

About the author

Takahiro Iwasa

Software Developer

This blog shares technical notes from hands-on projects—architecture, implementation, and AWS service integrations.

Setting Up the VPC

Creating the VPC

Adding Subnets

Adding Internet Gateway

Adding VPC Endpoints

Bastion EC2 Instance

Creating an Instance IAM Role

Starting the Bastion EC2 Instance

Connecting to the Instance with Session Manager

Updating the AWS CLI

Installing kubectl

Installing eksctl

EKS

Creating the EKS Cluster

Appendix: Troubleshooting Cluster Access

Issue 1: Credential Error

Issue 2: Connection Refused

Adding IAM Users and Roles

Enabling the Private Cluster Endpoint

Creating a Fargate Profile

Installing AWS Load Balancer Controller

Installing Helm and the Load Balancer Controller

Tagging Subnets

Deploying Application

Building Application

Pushing the Image to ECR

Deploying to Fargate

Testing the API

Deleting EKS Cluster

Appendix: Troubleshooting Deletion Issues

Conclusion

Related posts

About the author

Installing `kubectl`

Installing `eksctl`