How To: Access Your AWS VPC-based Elasticsearch Cluster Locally

Posted in How To, Programming & Serverless on November 1, 2017
Last Updated: February 15, 2018

AWS recently announced that their Elasticsearch Service now supports VPC, which is awesome, for a number of reasons:

1. No more signing every request

Remember this?

Every request had to be signed with AWS’s SigV4 so that the Elasticsearch endpoint could be properly authorized. That meant additional code to sign all your requests, and additional time for the endpoint to decode it. It might only be a few milliseconds of extra processing time, but those can add up. Now we can call our VPC Elasticsearch endpoint with a simple HTTP request.

2. No need to set up NATs or Internet Gateways

If your apps don’t require outgoing access to the Internet, there is no longer a need to set up NATs and IGs to access your Elasticsearch cluster. This saves you both complexity and money by not needing to maintain the extra configurations. Especially in a multi-availability zone deployment.

3. More secure, no more publicly available URLs protected by weak IP restrictions

Elasticsearch has no built-in security, so we used to simply restrict access to our EC2 instances that were running ES using security groups. AWS’s Elasticsearch Service, however, only allowed for a publicly accessible URL, requiring additional levels of security to authorize access, like signing the request. This meant managing your cluster locally from the command line, or accessing Kibana, required you to compromise security by authorizing specific IP addresses to have access to the cluster. This was a terrible idea and opened up huge security risks. VPC-based ES clusters are no longer publicly accessible, which closes that security hole.

Accessing Your Elasticsearch Cluster Locally

All this new VPC stuff is great, but if you read the ES documentation you probably noticed this:

To access the default installation of Kibana for a domain that resides within a VPC, users must first connect to the VPC. This process varies by network configuration, but likely involves connecting to a VPN or corporate network.

This is also true if you want to access your ES cluster from the command line. The URL is no longer publicly accessible, and in fact, routes to an internal VPC IP address. If you already have a VPN solution that allows you to connect to your VPC, then configuring your security groups correctly should work. However, if you don’t have a VPN configured, you can solve your problem using a simple SSH tunnel with port forwarding.

Step 1:
You need to have an EC2 instance running in the same VPC as your Elasticsearch cluster. If you don’t, fire up a micro Linux instance with a secure key pair.

NOTE: Make sure your instance’s security group has access to the Elasticsearch cluster and that your Elasticsearch cluster’s access policy uses the “Do not require signing request with IAM credential” template.

Step 2:
Create an entry in your SSH config file (~/.ssh/config on a Mac):

NOTE: The “HostName” should be your instance’s PUBLIC IP address or DNS. “User” should be your Linux distro’s default user (ec2-user if using Amazon Linux).

Step 3:
Run ssh estunnel -N from the command line

Step 4:
localhost:9200 should now be forwarded to your secure Elasticsearch cluster.

Access via a web browser, ignore the invalid SSL certificate:
Search: https://localhost:9200
Kibana: https://localhost:9200/_plugin/kibana

Access via cURL, be sure to use the -k option to ignore the security certificate:
curl -k https://localhost:9200/

Access programmatically (example in Node.js). Be sure to use the corresponding option to ignore SSL certificates (as in strictSSL below):

And that’s it! Now you can take advantage of the benefits of VPC-based Elasticsearch clusters and still maintain your local development workflows.

Tags: , , , , , , ,

Did you like this post? 👍  Do you want more? 🙌  Follow me on Twitter or check out some of the projects I’m working on.

🗓 This post was originally published on November 1, 2017 by Jeremy Daly. It was last updated on February 15, 2018.

59 thoughts on “How To: Access Your AWS VPC-based Elasticsearch Cluster Locally”

  1. Good Post Sir.
    I am trying to access the ElasticSearch Server hosted in private subnet in a VPC from the EC2 instance which is hosted in public subnet.
    But when i try to do ssh tunneling from locally to public EC2 instance, then the connection fails and I am not able to access the underlying ElasticSearch server hosted in private subnet.

    Reply

    1. Nitin,
      There should be no problem accessing a private VPC subnet from a public VPC subnet so long as the security group allows it. Make sure that the machine trying to access the Elasticsearch cluster has a security group that is authorized in the Elasticsearch cluster configuration.
      Hope that helps,
      Jeremy

    2. Hi Nitin/Jeremy – Did it work? I have the same situation ( ES is on a private subnet. I have created 2 EC2s – 1) with private subnet , same subnet as ES , 2) public subnet ) . It is not working.

  2. Hi Jeremy,
    Thank you for a good reference. The approach worked with one additional step to provide access policy to the ES domain : “dont require signing request with IAM credentials”.

    Please verify my understanding.

    Reply

    1. Pratik,
      Yes, you are correct. You have to have your Elasticsearch Access Policy set up using the “Do not require signing request with IAM credential” template. I’ll update the post.

  3. I followed all the steps correctly but i am getting ssh: connect to host my-IP-Address port 22: Operation timed out.
    so i have some doubts
    1) HostName 12.34.56.78 – what IP address is this? I pasted my EC2 instance’s private IP address. is that correct? If not what should be the IP here?
    2) User ec2-user – what user is this? is this a IAM user , a AWS user?

    Any help would be appriciated 🙂

    Reply

    1. Hi Mahesh,

      The HostName should be your server’s PUBLIC IP address or public DNS, not your private IP.

      The User is your Linux username, so it depends on the distro you’re using. If you are using Amazon Linux, then it will be “ec2-user”. If you were using Ubuntu, then it would be “ubuntu”. Debian would be “admin”. Etc.

      Give that a try!
      Jeremy

  4. Thanks for you prompt reply.

    I dont see Public IP /DNS assigned to my ec2 instance. But after further reading i got to know that we can create a Elastic IP and associate with our ec2 instance. So i did that, but no luck got the same message ‘Operation timed out.’

    Reply

  6. Hi Jeremy,

    I found that the subnet you attach the ec2 instance should have a route table with 1 rule for
    intenet gateway. So once i had it i was able to SSH the instance but port forwarding did not work.
    So the other solution which worked for me was i created an Lambda function under the same VPC as Elastic search and wrote a logic to get data depending on query we invoke.

    Thanks for all the help and sharing this post. this was really helpful.

    Reply

    1. Hi William,

      The point of the VPC is to create a private cloud that isn’t directly accessible from the Internet. You could do port forwarding to your VPC Elasticsearch cluster from one of your EC2 instances and then open up that port to the web. However, this is HIGHLY insecure and I do not suggest you do that. Your only real security would be to assign IP restrictions, which can easily be spoofed.

      If you have a VPN that can tunnel into your VPC, then there is no reason to use the method I describe in the post. The DNS for the VPC Elasticsearch cluster resolves to an internal VPC IP address, so that FQDN can be used in local applications as well.

      Hope that helps!

  8. Hello Jeremy,

    I followed the whole tuto and read each comment but it does not work for me :/
    I’m trying to connect to ES from my EC2-instance as you describe but like Nitin Ware (first comment), connexion fails.

    I’ve checked the security groups. My EC2-instance has the same group as my ES cluster.
    When i’m on my EC2-instance (threw ssh) , it can access the aws ES so I suppose its ok for the security group.

    Have you any idea ?

    Reply

    1. Hi Florian,

      If you can access your Elasticsearch cluster when you’re connected via ssh, then there should be no problem using the port forwarding. Are you SSHing into your EC2 instance via your config file, or are you using a full connection string? The first step is to make sure you can connect to your EC2 instance by adding an entry to your config file. Try adding everything above except for the LocalForward 9200 vpc-YOUR-ES-CLUSTER.us-east-1.es.amazonaws.com:443 line. If this doesn’t allow you to connect to your EC2 instance, then your configuration (hostname, user, etc.) is incorrect somehow. Once you can connect, adding the LocalForward line should work.

      Jeremy

  9. Hi Jeremy,
    Thanks for your answer.

    The above configuration did not work so I’ve searched and made the following changes on AWS
    – Port 443 was added to the SG for the ES domain.
    – The SG of the ES domain was added to the SG for the instance.

    Upon making the above changes, I again attempted SSH estunnel on my mac.
    ssh estunnel -N

    Now, when I use the browser on my local machine I am able to access both of the following URLs.

    Everything is good now !

    Reply

    1. Hi Vishal,

      The IdentityFile is the private key that you generated as part of the “ec2-key-pairs.html” page you mentioned. If you are getting a permission denied error, then it is possible that your permissions are set incorrectly on that file. Make sure that you are referencing the location of your private key correctly, and then be sure to chmod 400 MY-KEY.pem so that the system has read permissions.

      Hope that helps!

  12. Hello, thanks for great article.
    I’m wondering – would it be possible to access Kibana/ES deployed in VPC by proxying the traffic via internet-facing load balancer ?

    Reply

    1. I would think so. An earlier version of the AWS Database Migration Service required a proxy server on EC2-Classic instances in order to transfer data into a VPC. Same concept applies here. I would think that you could easily do this with NGINX as described in Appendix A of this old doc: https://d1.awsstatic.com/whitepapers/RDS/Moving_RDS_MySQL_DB_to_VPC.pdf. You’ll have to configure your security groups correctly as well. This would be a bit of a security risk, however, depending on your use case.

    2. Yes, it works using load balancer as reverse proxy “without authentication”.
      I haven’t got it to work using Cognito.

  16. Hi Jeremy,

    Hope things are going well. I just setup elasticsearch in a VPC and also started an EC2 instance. I am able to SSH into the ec2 and curl elasticsearch though the vpc endpoint. I have also loaded some data into elasticsearch through logstash on my EC2. But I am not able to open kibana. I am new to SSH, networking and not sure where I am making a mistake. Below are my settings:

    L9200 https://my-vcp.us-east-1.amazonaws.com:443

    Google chrome says connection refused to localhost 9200. Can you please help me with this issue.

    Thanks,
    Nanda

    Reply

    1. Hi Nanda,

      Are you starting the tunnel by running ssh estunnel -N in your terminal? What happens when you run that command? You can also run ssh estunnel -Nv to see some additional information about the status of the connection. Let me know!

      – Jeremy

  17. Hi,

    I’m using Putty on Windows.
    I set up the ssh configs in Putty. I’m able to connect to my EC2 instance.
    But on my Windows browser, I cannot connect to ES or Kibana.
    Please help.
    Thanks.

    Reply

  19. This is great example of how to access and interact with the ES Service. If you wanted to access this securely (i.e. without using the -k option in curl, how would yo do that?

    Reply

    1. Hi Harry,
      The -k option just skips verification of the certificate, the data itself is still encrypted. If you wanted to verify certificates, you’d need to be connected via a VPN.
      – Jeremy

  21. Hi Jeremy,

    This is great example how to access securely ES/Kibana in VPC but what about authorization. Have you tried to use AWS Cognito to authorize Kibana when hosted in VPC?
    I am asking because I have a use case where not only one person (the admin) will access the Kibana so I need more granular control who use it.

    Reply

  22. When using aws-elasticsearch-client or the lower-level http-aws-es, I couldn’t find a way to disable strict SSL. This doesn’t appear to be a supported option.

    You can side-step this issue by adding an entry mapping the Elasticsearch cluster domain name to 127.0.0.1 in your /etc/hosts file. That way the SSL check passes.

    Reply

  23. Hi Jeremy,
    This is a great example of how to access the ES service within a VPC.
    I’ve got it all working fine however since configuring my ES cluster with Cognito, whenever I now login it then gives me a 400 error code.
    I’ve checked all the configurations including App client settings within the cognito pool settings and even the access policy for ES to allow the identity pool associated with my cognito user pool.

    Can you think of anything that could be causing this?

    Reply

    1. I don’t think you can use Cognito to access ES, just Kibana. What exactly are you trying to accomplish? Cognito permissions to restrict access to ES API calls?

  25. thanks for the post. As an experiment, I am trying to load sql server table in AWS elastic service using local logstash on my mac. I followed these instructions as the ES is in vpc

    I get the following error

    [2019-12-02T21:29:55,621][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://localhost:9200/]}}
    [2019-12-02T21:29:56,136][ERROR][logstash.javapipeline ][main] Pipeline aborted due to error {:pipeline_id=>”main”, :exception=>#

    Any help is much appreciated.

    Reply

  27. Hi Jeremy! Great article. I am close and I am trying to connect through my EC2 instance to Kibana, but I am getting a 400 Bad Request error after I authenticate. I am using CentOS 7. Any ideas? Thank you.

    Reply

  30. I am getting {“Message”:”User: anonymous is not authorized to perform: es:ESHttpGet”}. Anyone else getting this error and how to go about it

    Reply

  31. I am able to be forwarded to aws elasticsearch through ssh.
    But I am not able to access the elastic search link https://localhost:9200 in web browser
    It shows PR_CONNECT_RESET_ERROR in firefox
    ERR_CONNECTION_RESET in chrome
    (curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to localhost:9200) in curl

    Can you please tell me how to fix this issue ?

    Reply

  32. Hey Jeremy great article. I was able to access es just fine following this. However I am running into an issue with accessing Kibana when using Cognito. The problem I believe is when you sign in with cognito it takes you to your custom page test.auth.us-west-2.amazoncognito.com, and passes a redirect url of the ES Endpoint, not localhost. So when the login is successful it tries to request something that is not accessible.

    Reply

    1. i do get “Message”:”User: anonymous is not authorized to perform: es:ESHttpGet”} Can some please help on what to do.

  34. Hi Jeremy,
    Thanks for this great article. What is the approach to migrate data with public endpoint hosted on EC2-Classic to EC2-VPC of ES domain hosted inside VPC private subnet.
    Looking forward to a work around solution

    Reply

  35. I am getting bellow error how to solve this

    [ec2-user@ip-10-0-0-243 .ssh]$ cat config/ec.sh
    # Elasticsearch Tunnel
    Host estunnel
    HostName 18.223.106.105 # your server’s public IP address
    User ec2-user
    IdentitiesOnly yes
    IdentityFile ~/.ssh/new.pem
    LocalForward 9200 vpc-newdomain-q3o4gempvoyb6esprenquk5zcq.us-east-2.es.amazonaws.com:443
    [ec2-user@ip-10-0-0-243 .ssh]$ ssh estunnel -N
    ssh: Could not resolve hostname estunnel: Name or service not known
    [ec2-user@ip-10-0-0-243 .ssh]$

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.