Example of how to connect and use Elasticsearch on IBM Cloud Databases using bash automation and CURL commands

Example how to connect to Elasticsearch on IBM Cloud with Databases for Elasticsearch using bash scripting with cURL.

Content:

• Setup
• Example usage of search indexes using bash automation
• Elasticsearch example CURL search command

Useful resources:

• Elasticsearch API documentation here you can easily copy the commands
• IBM Cloud Elasticsearch CURL documentation

1. Setup IBM Cloud

Step 1: Clone the repo to the local machine

git clone https://github.com/thomassuedbroecker/example-connect-to-elastic-search
cd example-connect-to-elastic-search

Step 2: Create a Standard Eleasticsearch instance on IBM Cloud

open https://cloud.ibm.com/databases/databases-for-elasticsearch/create

Step 3: Create new Service Credentials

Step 4: Download the self-signed certificate to your local computer

Download the certificate file into the folder ./code/cert.

Step 5: Create a .env file for environment variables of the bash automation

cat code/.env_template > .env

Step 6: Insert the needed values for the environment variables

# IBM Cloud
export IBM_CLOUD_API_KEY=XXXX
export IBM_CLOUD_REGION=us-south
export IBM_CLOUD_RESOURCE_GROUP=default

# Elasticsearch service
export E_SEARCH_SERVICE=YOUR_DatabasesForElasticsearch
export E_CERT_FILE_NAME=d5290bfc-XXXXX-XXX-9337-XXXX40bd
export E_HOST=XXXXX-XXXXX-XXXXX-XXXXX.XXXX.databases.appdomain.cloud
export E_PORT=0815
export E_ADMIN_USER=admin
export E_ADMIN_PASSWORD=YOUR_AWESOME_PASSWORD

Step 7: Change the admin password for the database

cd /code
sh change_admin_password.sh

• Example output:

Key                   Value
ID                    crn:XXX
Deployment ID         crn:vXXX
Description           Updating user
Created At            2023-XX
Status                running
Progress Percentage   0
                      
Status                completed
Progress Percentage   100
Location              https://api.us-east.databases.cloud.ibm.com/XXXX
OK

2. Example usage of search indexes using bash automation

Step 1: Test the connection

cd /code
sh test_connection.sh

• Example output:

{
  "cluster_name" : "XXX",
  "status" : "green",
 ...
}

Step 2: Create indexes

cd /code
sh create_indexes.sh

• Example output:

Create: documents
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 85

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "documents"
}
Create: passages
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 84

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "passages"
}

Step 3: Upload data to index

cd /code
sh upload-data_to_indexes.sh

• Example output:

[
  {
    "chunckid": "1.0815",
    "text": [
      "Hello world",
      "This is the first hello world example!"    ],
      "url": "http://example.com/docs/0001",
    "title": "Hello world"
  },
  {
    "chunckid": "1.0815",
    "text": [
      "This is the second hello world example!"    ],
      "url": "http://example.com/docs/0001",
    "title": "Hello world"
  }
]HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 451

{"took":119,"errors":false,"items":[{"index":{"_index":"passages","_type":"_doc","_id":"http://example.com/docs/0001","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":0,"_primary_term":1,"status":201}},{"index":{"_index":"passages","_type":"_doc","_id":"http://example.com/docs/0001","_version":2,"result":"updated","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":1,"_primary_term":1,"status":200}}]}Upload next file: /Users/thomassuedbroecker/Downloads/dev/example-connect-to-elastic-search/code/full/full1.json
[
    {
      "title": "Hello world",
      "url": "http://example.com/docs/0001",
      "text": [
        "Hello world",
        "This is the first hello world example!", 
        "This is the second hello world example!"      
       ]
    }
]HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 245

{"took":117,"errors":false,"items":[{"index":{"_index":"documents","_type":"_doc","_id":"http://example.com/docs/0001","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":0,"_primary_term":1,"status":201}}]}

Step 4: Search in index

cd /code
sh search_in_indexes.sh

Step 5: Delete indexes

cd /code
sh delete_indexes.sh

3. Elasticsearch example CURL search command

Step 1: Open a terminal

Step 2: Create a folder called cert

mkdir cert

Step 3: Navigate to the folder

cd cert

Step 4: Set the environment variable path

export E_CERT_PATH=$(pwd)

Step 5: Download the certificate file for your Databases for Elasticsearch instance

Download the certificate file to your local computer into the newly created folder.

Step 5: Set the following environment variables to your values

• Example:

export E_CERT_FILE_NAME=d5290bfc-XXXXX-XXX-9337-XXXX40bd
export E_HOST=XXXXX-XXXXX-XXXXX-XXXXX.XXXX.databases.appdomain.cloud
export E_PORT=0815
export E_ADMIN_USER=admin
export E_ADMIN_PASSWORD=YOUR_AWESOME_PASSWORD

Step 6: Execute an index search

CURL_CA_BUNDLE=$E_CERT_PATH/$E_CERT_FILE_NAME curl -u ${E_ADMIN_USER}:${E_ADMIN_PASSWORD} -XGET -H "Content-Type: application/json" "https://${E_HOST}:${E_PORT}/documents/_search" -d '{ "query": {"multi_match" : {"query" : "Second Hello World?","fields": ["title", "text"]}}}' | jq '.'

• Example output:

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:100   453  100   423  100    30   1193     84 --:--:-- --:--:-- --:--:--  1279
{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "documents",
        "_type": "_doc",
        "_id": "http://example.com/docs/0001",
        "_score": 1,
        "_source": {
          "title": "Hello world",
          "url": "http://example.com/docs/0001",
          "text": [
            "Hello world",
            "This is the first hello world example!",
            "This is the second hello world example!"
          ]
        }
      }
    ]
  }
}
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:  0     0    0     0    0     0      0      0 --:--:-- --:--:100   526  100   435  100    91   1264    264 --:--:-- --:--:-- --:--:--  1529
{
  "took": 7,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1.1918257,
    "hits": [
      {
        "_index": "documents",
        "_type": "_doc",
        "_id": "http://example.com/docs/0001",
        "_score": 1.1918257,
        "_source": {
          "title": "Hello world",
          "url": "http://example.com/docs/0001",
          "text": [
            "Hello world",
            "This is the first hello world example!",
            "This is the second hello world example!"
          ]
        }
      }
    ]
  }
}

4. Setup Local

Start Elasticsearch on your local computer using the Quarkus documentation

Step 1: Start container

docker run --name elasticsearch  -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Xms512m -Xmx512m"\
       --rm -p 9200:9200 docker.io/elastic/elasticsearch:7.16.3

Example how to start with basic authentication.

docker run --name elasticsearch  \ 
       -e "discovery.type=single-node" \ 
       -e "ES_JAVA_OPTS=-Xms512m -Xmx512m" \
       -e "xpack.security.enabled=true" \
       -e "ELASTIC_USERNAME=elastic" \
       -e "ELASTIC_PASSWORD=123456789" \
       --rm -p 9200:9200 docker.io/elastic/elasticsearch:7.16.3

Step 2: Create a .env file for environment variables of the bash automation

cat code/.env_template > .env

Step 3: Define the following variables

export E_PORT=9200
export E_HOST=localhost

Step 4: Execute the bash scripts

sh create_indexes_local.sh
sh upload-data_to_indexes_local.sh
sh search_in_index_local.sh
sh delete_indexes_local.sh