Full-Text Search for Movies with AWS CloudSearch

Indexing a public movies dataset in AWS CloudSearch and running free-text, numeric, and range queries against it.

Takahiro Iwasa

2022-11-27

5 min read

AWS CloudSearch can index and query a movies dataset for full-text search without standing up a separate search cluster.

Setting Up a CloudSearch Domain

Create a CloudSearch domain:

aws cloudsearch create-domain \
  --domain-name searching-movies-data

According to the official documentation, creating a CloudSearch domain typically takes approximately 10 minutes to complete.

The domain creation status can be checked with:

aws cloudsearch describe-domains \
  --domain-name searching-movies-data

When the command output shows Processing: false, the domain and its endpoints are ready to use.

1
{
2
  "DomainStatusList": [
3
    {
4
      "DomainId": "123456789012/searching-movies-data",
5
      "DomainName": "searching-movies-data",
6
      "ARN": "arn:aws:cloudsearch:ap-northeast-1:123456789012:domain/searching-movies-data",
7
      "Created": true,
8
      "Deleted": false,
9
      "DocService": {
10
        "Endpoint": "doc-searching-movies-data-xxxxxxxxxx.ap-northeast-1.cloudsearch.amazonaws.com"
11
      },
12
      "SearchService": {
13
        "Endpoint": "search-searching-movies-data-xxxxxxxxxx.ap-northeast-1.cloudsearch.amazonaws.com"
14
      },
15
      "RequiresIndexDocuments": false,
16
      "Processing": false,
17
      "SearchInstanceType": "search.small",
18
      "SearchPartitionCount": 1,
19
      "SearchInstanceCount": 1,
20
      "Limits": {
21
        "MaximumReplicationCount": 5,
22
        "MaximumPartitionCount": 10
23
      }
24
    }
25
  ]
26
}

To enhance security, update the domain’s access policy to allow access only from your IP address:

aws cloudsearch update-service-access-policies \
  --domain-name searching-movies-data \
  --access-policies '
  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": "*",
        "Action": ["cloudsearch:*"],
        "Condition": {"IpAddress": {"aws:SourceIp": "xxx.xxx.xxx.xxx/32"}}
      }
    ]
  }'

Configuring Index Fields

Define the index fields based on the dataset’s structure. This example uses The Movies Dataset by Kaggle (CC0: Public Domain). Below is an example command for defining an index field:

aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name adult --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name belongs_to_collection --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name budget --type double
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name genres --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name homepage --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name id --type int
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name imdb_id --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name original_language --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name original_title --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name overview --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name popularity --type double
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name poster_path --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name production_companies --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name production_countries --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name release_date --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name revenue --type int
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name runtime --type double
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name spoken_languages --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name status --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name tagline --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name title --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name video --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name vote_average --type double
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name vote_count --type int

After defining the fields, tell the search domain to start indexing its documents:

aws cloudsearch index-documents \
  --domain-name searching-movies-data

Indexing Dataset

Download the dataset from Kaggle and prepare a sample file with the first 1,000 rows:

head -1000 movies_metadata.csv > sample.csv

CloudSearch can index data directly from CSV files through the console interface, though the AWS CLI’s aws cloudsearchdomain upload-documents command only accepts JSON or XML formats.

Navigate to Actions > Upload documents.

Choose your CSV file and click Next.

Review the detected fields and click Upload documents.

When the process completes, confirm that 998 records, excluding the header, were indexed.

Running Queries

Searching for Text

To search for movies with the keyword house in the title and overview fields:

curl --location \
  -g \
  --request GET \
  'https://search-searching-movies-data-xxxxxxxxxx.ap-northeast-1.cloudsearch.amazonaws.com/2013-01-01/search?q=house&q.options={fields:["title","overview"]}&return=title,overview' | jq .

The response will include matching results, such as:

1
{
2
  "status": {
3
    "rid": "8fDJv8swsgEK1DyD",
4
    "time-ms": 1
5
  },
6
  "hits": {
7
    "found": 26,
8
    "start": 0,
9
    "hit": [
10
      {
11
        "id": "local_file_466",
12
        "fields": {
13
          "overview": "Hip Hop duo Kid & Play return...",
14
          "title": "House Party 3"
15
        }
16
      },
17
      ...
18
    ]
19
  }
20
}

For more information, refer to the official documentation.

Searching for Numbers

For numeric searches, such as movies with a vote_average of 5.0:

curl --location --request GET 'https://search-searching-movies-data-xxxxxxxxxx.ap-northeast-1.cloudsearch.amazonaws.com/2013-01-01/search?q.parser=structured&q=vote_average:5.0&return=title,overview' | jq .

The response will include matching results, such as:

1
{
2
  "status": {
3
    "rid": "w+Xgv8swwQEK1DyD",
4
    "time-ms": 0
5
  },
6
  "hits": {
7
    "found": 35,
8
    "start": 0,
9
    "hit": [
10
      {
11
        "id": "local_file_144",
12
        "fields": {
13
          "overview": "Far from home...",
14
          "title": "The Amazing Panda Adventure"
15
        }
16
      },
17
      ...
18
    ]
19
  }
20
}

For more information, refer to the official documentation.

Searching for Ranges

To find movies with a vote_average of 7.0 or greater:

curl --location \
  -g \
  --request GET \
  'https://search-searching-movies-data-xxxxxxxxxx.ap-northeast-1.cloudsearch.amazonaws.com/2013-01-01/search?q.parser=structured&q=vote_average:[7.0,}&return=title,overview' | jq .

The response will include matching results, such as:

1
{
2
  "status": {
3
    "rid": "vJ/vv8swyAEK1DyD",
4
    "time-ms": 2
5
  },
6
  "hits": {
7
    "found": 254,
8
    "start": 0,
9
    "hit": [
10
      {
11
        "id": "local_file_1",
12
        "fields": {
13
          "overview": "Led by Woody, Andy's toys...",
14
          "title": "Toy Story"
15
        }
16
      },
17
      ...
18
    ]
19
  }
20
}

For more information, refer to the official documentation.

Cleaning Up

Delete the domain once finished:

aws cloudsearch delete-domain \
  --domain-name searching-movies-data

Conclusion

AWS CloudSearch can run free-text and numeric searches against the same domain without a separate search cluster. Switching between q=house and the structured query q.parser=structured&q=vote_average:[7.0,} covers both free-text and numeric range searches without additional index configuration.

Because CloudSearch endpoints are reachable over the public internet by default, restrict the domain’s access policy to a specific aws:SourceIp with update-service-access-policies immediately after creating the domain.

Allow for the roughly 10-minute domain creation time when automating this configuration as part of a larger setup process.