Full-Text Search for Movies with AWS CloudSearch

Indexing a public movies dataset in AWS CloudSearch and running free-text, numeric, and range queries against it.

Takahiro Iwasa

2022-11-27

5 min read

AWS CloudSearch can index and query a movies dataset for full-text search without standing up a separate search cluster.

Setting Up CloudSearch Domain

Create a CloudSearch domain:

aws cloudsearch create-domain \
  --domain-name searching-movies-data

According to the official documentation, creating a CloudSearch domain typically takes approximately 10 minutes to complete.

The domain creation status can be checked with:

aws cloudsearch describe-domains \
  --domain-name searching-movies-data

When the command output shows Processing: false, this indicates that the domain and its endpoints are fully created and ready to use.

1
{
2
  "DomainStatusList": [
3
    {
4
      "DomainId": "123456789012/searching-movies-data",
5
      "DomainName": "searching-movies-data",
6
      "ARN": "arn:aws:cloudsearch:ap-northeast-1:123456789012:domain/searching-movies-data",
7
      "Created": true,
8
      "Deleted": false,
9
      "DocService": {
10
        "Endpoint": "doc-searching-movies-data-xxxxxxxxxx.ap-northeast-1.cloudsearch.amazonaws.com"
11
      },
12
      "SearchService": {
13
        "Endpoint": "search-searching-movies-data-xxxxxxxxxx.ap-northeast-1.cloudsearch.amazonaws.com"
14
      },
15
      "RequiresIndexDocuments": false,
16
      "Processing": false,
17
      "SearchInstanceType": "search.small",
18
      "SearchPartitionCount": 1,
19
      "SearchInstanceCount": 1,
20
      "Limits": {
21
        "MaximumReplicationCount": 5,
22
        "MaximumPartitionCount": 10
23
      }
24
    }
25
  ]
26
}

To enhance security, update the domain’s access policy to allow access only from your IP address:

aws cloudsearch update-service-access-policies \
  --domain-name searching-movies-data \
  --access-policies '
  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": "*",
        "Action": ["cloudsearch:*"],
        "Condition": {"IpAddress": {"aws:SourceIp": "xxx.xxx.xxx.xxx/32"}}
      }
    ]
  }'

Configuring Index Fields

Define the index fields based on the dataset’s structure. This example uses The Movies Dataset by Kaggle (CC0: Public Domain). Below is an example command for defining an index field:

aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name adult --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name belongs_to_collection --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name budget --type double
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name genres --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name homepage --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name id --type int
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name imdb_id --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name original_language --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name original_title --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name overview --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name popularity --type double
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name poster_path --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name production_companies --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name production_countries --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name release_date --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name revenue --type int
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name runtime --type double
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name spoken_languages --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name status --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name tagline --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name title --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name video --type text
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name vote_average --type double
aws cloudsearch define-index-field \
  --domain-name searching-movies-data --name vote_count --type int

Once done, tells the search domain to start indexing its documents:

aws cloudsearch index-documents \
  --domain-name searching-movies-data

Indexing Dataset

Download the dataset from Kaggle and prepare a sample file with the first 1,000 rows:

head -1000 movies_metadata.csv > sample.csv

CloudSearch can index data directly from CSV files through the console interface, though the AWS CLI’s aws cloudsearchdomain upload-documents command only accepts JSON or XML formats.

Navigate to Actions > Upload documents.

Choose your CSV file and click Next.

Review the detected fields and click Upload documents.

Once the process completes, you can confirm that 998 records (excluding headers) are successfully indexed.

Running Queries

Searching for Text

To search for movies with the keyword house in the title and overview fields:

curl --location \
  -g \
  --request GET \
  'https://search-searching-movies-data-xxxxxxxxxx.ap-northeast-1.cloudsearch.amazonaws.com/2013-01-01/search?q=house&q.options={fields:["title","overview"]}&return=title,overview' | jq .

The response will include matching results, such as:

1
{
2
  "status": {
3
    "rid": "8fDJv8swsgEK1DyD",
4
    "time-ms": 1
5
  },
6
  "hits": {
7
    "found": 26,
8
    "start": 0,
9
    "hit": [
10
      {
11
        "id": "local_file_466",
12
        "fields": {
13
          "overview": "Hip Hop duo Kid & Play return...",
14
          "title": "House Party 3"
15
        }
16
      },
17
      ...
18
    ]
19
  }
20
}

For more information, refer to the official documentation.

Searching for Numbers

For numeric searches, such as movies with a vote_average of 5.0:

curl --location --request GET 'https://search-searching-movies-data-xxxxxxxxxx.ap-northeast-1.cloudsearch.amazonaws.com/2013-01-01/search?q.parser=structured&q=vote_average:5.0&return=title,overview' | jq .

The response will include matching results, such as:

1
{
2
  "status": {
3
    "rid": "w+Xgv8swwQEK1DyD",
4
    "time-ms": 0
5
  },
6
  "hits": {
7
    "found": 35,
8
    "start": 0,
9
    "hit": [
10
      {
11
        "id": "local_file_144",
12
        "fields": {
13
          "overview": "Far from home...",
14
          "title": "The Amazing Panda Adventure"
15
        }
16
      },
17
      ...
18
    ]
19
  }
20
}

For more information, refer to the official documentation.

Searching for Ranges

To find movies with a vote_average greater than 7.0:

curl --location \
  -g \
  --request GET \
  'https://search-searching-movies-data-xxxxxxxxxx.ap-northeast-1.cloudsearch.amazonaws.com/2013-01-01/search?q.parser=structured&q=vote_average:[7.0,}&return=title,overview' | jq .

The response will include matching results, such as:

1
{
2
  "status": {
3
    "rid": "vJ/vv8swyAEK1DyD",
4
    "time-ms": 2
5
  },
6
  "hits": {
7
    "found": 254,
8
    "start": 0,
9
    "hit": [
10
      {
11
        "id": "local_file_1",
12
        "fields": {
13
          "overview": "Led by Woody, Andy's toys...",
14
          "title": "Toy Story"
15
        }
16
      },
17
      ...
18
    ]
19
  }
20
}

For more information, refer to the official documentation.

Cleaning Up

Delete the domain once finished:

aws cloudsearch delete-domain \
  --domain-name searching-movies-data

Conclusion

Indexing 998 rows of a public movies dataset in AWS CloudSearch and querying them made both free-text and numeric searches work against the same domain with no separate search cluster to manage. Switching between q=house and the structured parser with q.parser=structured&q=vote_average:[7.0,} covers free-text and numeric range queries with no separate configuration, which is what makes CloudSearch convenient here. Restricting the domain’s access policy to a specific aws:SourceIp with update-service-access-policies is worth doing immediately after creating the domain, since CloudSearch endpoints are reachable over the public internet by default. The roughly 10-minute domain creation time is also worth planning around if this ends up scripted into a larger setup process.