Searching Across Multiple Elasticsearch Indices

Querying multiple Elasticsearch indices at once with a wildcard pattern versus an explicit comma-separated list.

Takahiro Iwasa

2020-12-31

2 min read

Elasticsearch supports querying across several indices at once, either with a wildcard pattern or an explicit comma-separated list.

Launching Elasticsearch

Below is an example docker-compose.yml file to configure and start the cluster.

1
version: '3'
2

3
services:
4
  elasticsearch:
5
    image: elasticsearch:7.10.1
6
    container_name: elasticsearch
7
    environment:
8
      - discovery.type=single-node
9
      - bootstrap.memory_lock=true
10
    ports:
11
      - 9200:9200

Start the cluster:

docker-compose up -d

Preparing Data

To experiment with multiple indices, index some data into users-2020-11 and users-2020-12 using the curl command:

curl -X POST -H 'Content-Type: application/json' -d '{"name": "hoge"}' localhost:9200/users-2020-11/_doc/
curl -X POST -H 'Content-Type: application/json' -d '{"name": "fuga"}' localhost:9200/users-2020-12/_doc/

Searching Data

Searching with Wildcards

A wildcard expression matches data across multiple indices in a single query:

curl localhost:9200/users-2020-*/_search | jq .hits.hits

The response will contain data from both users-2020-11 and users-2020-12:

1
[
2
  {
3
    "_index": "users-2020-11",
4
    "_type": "_doc",
5
    "_id": "PNQ3tXYBKT-fwQ71grcz",
6
    "_score": 1,
7
    "_source": {
8
      "name": "hoge"
9
    }
10
  },
11
  {
12
    "_index": "users-2020-12",
13
    "_type": "_doc",
14
    "_id": "PdQ3tXYBKT-fwQ71p7cy",
15
    "_score": 1,
16
    "_source": {
17
      "name": "fuga"
18
    }
19
  }
20
]

Searching with CSV Format

Another way to search is by specifying multiple indices as a comma-separated list:

🔥 Caution

Ensure that your URL length does not exceed the maximum limit when using this approach.

curl localhost:9200/users-2020-11,users-2020-12/_search | jq .hits.hits

The response will contain data from both users-2020-11 and users-2020-12:

1
[
2
  {
3
    "_index": "users-2020-11",
4
    "_type": "_doc",
5
    "_id": "PNQ3tXYBKT-fwQ71grcz",
6
    "_score": 1,
7
    "_source": {
8
      "name": "hoge"
9
    }
10
  },
11
  {
12
    "_index": "users-2020-12",
13
    "_type": "_doc",
14
    "_id": "PdQ3tXYBKT-fwQ71p7cy",
15
    "_score": 1,
16
    "_source": {
17
      "name": "fuga"
18
    }
19
  }
20
]

Conclusion

Querying users-2020-11 and users-2020-12 with both a wildcard pattern and an explicit comma-separated list returned identical hits, so the real difference between the two approaches shows up only in how the query scales. A wildcard pattern like users-2020-* stays a fixed length no matter how many monthly indices accumulate, which makes it the more durable choice for time-based index naming schemes, while the comma-separated form is easier to reason about when the exact set of indices is small and known in advance — though it’s the one to watch for the URL length limit as that list grows.

Back to posts