
AWS recently announced that their Elasticsearch Service now supports VPC, which is awesome, for a number of reasons:
1. No more signing every request
Remember this?
1 2 3 |
let creds = new AWS.SharedIniFileCredentials({profile: esProfile }); let signer = new AWS.Signers.V4(req, 'es'); signer.addAuthorization(creds, new Date()); |
Every request had to be signed with AWS’s SigV4 so that the Elasticsearch endpoint could be properly authorized. That meant additional code to sign all your requests, and additional time for the endpoint to decode it. It might only be a few milliseconds of extra processing time, but those can add up. Now we can call our VPC Elasticsearch endpoint with a simple HTTP request.
2. No need to set up NATs or Internet Gateways
If your apps don’t require outgoing access to the Internet, there is no longer a need to set up NATs and IGs to access your Elasticsearch cluster. This saves you both complexity and money by not needing to maintain the extra configurations. Especially in a multi-availability zone deployment.
3. More secure, no more publicly available URLs protected by weak IP restrictions
Elasticsearch has no built-in security, so we used to simply restrict access to our EC2 instances that were running ES using security groups. AWS’s Elasticsearch Service, however, only allowed for a publicly accessible URL, requiring additional levels of security to authorize access, like signing the request. This meant managing your cluster locally from the command line, or accessing Kibana, required you to compromise security by authorizing specific IP addresses to have access to the cluster. This was a terrible idea and opened up huge security risks. VPC-based ES clusters are no longer publicly accessible, which closes that security hole.
Accessing Your Elasticsearch Cluster Locally
All this new VPC stuff is great, but if you read the ES documentation you probably noticed this:
To access the default installation of Kibana for a domain that resides within a VPC, users must first connect to the VPC. This process varies by network configuration, but likely involves connecting to a VPN or corporate network.
This is also true if you want to access your ES cluster from the command line. The URL is no longer publicly accessible, and in fact, routes to an internal VPC IP address. If you already have a VPN solution that allows you to connect to your VPC, then configuring your security groups correctly should work. However, if you don’t have a VPN configured, you can solve your problem using a simple SSH tunnel with port forwarding.
Step 1:
You need to have an EC2 instance running in the same VPC as your Elasticsearch cluster. If you don’t, fire up a micro Linux instance with a secure key pair.
NOTE: Make sure your instance’s security group has access to the Elasticsearch cluster and that your Elasticsearch cluster’s access policy uses the “Do not require signing request with IAM credential” template.
Step 2:
Create an entry in your SSH config file (~/.ssh/config
on a Mac):
1 2 3 4 5 6 7 |
# Elasticsearch Tunnel Host estunnel HostName 12.34.56.78 # your server's public IP address User ec2-user IdentitiesOnly yes IdentityFile ~/.ssh/MY-KEY.pem LocalForward 9200 vpc-YOUR-ES-CLUSTER.us-east-1.es.amazonaws.com:443 |
NOTE: The “HostName” should be your instance’s PUBLIC IP address or DNS. “User” should be your Linux distro’s default user (ec2-user if using Amazon Linux).
Step 3:
Run ssh estunnel -N
from the command line
Step 4:
localhost:9200
should now be forwarded to your secure Elasticsearch cluster.
Access via a web browser, ignore the invalid SSL certificate:
Search: https://localhost:9200
Kibana: https://localhost:9200/_plugin/kibana
Access via cURL, be sure to use the -k
option to ignore the security certificate:
curl -k https://localhost:9200/
Access programmatically (example in Node.js). Be sure to use the corresponding option to ignore SSL certificates (as in strictSSL
below):
1 2 3 4 5 6 7 8 9 10 |
const REQUEST = require('request-promise') const options = { method: 'GET', uri: 'https://localhost:9200/', strictSSL: false, json: true }; REQUEST(options).then(res => { console.log(res) }) |
And that’s it! Now you can take advantage of the benefits of VPC-based Elasticsearch clusters and still maintain your local development workflows.
Tags: amazon web services, aws vpc, elasticsearch, nodejs, security, server management, serverless, ssh
Did you like this post? 👍 Do you want more? 🙌 Follow me on Twitter or check out some of the projects I’m working on.
Good Post Sir.
I am trying to access the ElasticSearch Server hosted in private subnet in a VPC from the EC2 instance which is hosted in public subnet.
But when i try to do ssh tunneling from locally to public EC2 instance, then the connection fails and I am not able to access the underlying ElasticSearch server hosted in private subnet.
Nitin,
There should be no problem accessing a private VPC subnet from a public VPC subnet so long as the security group allows it. Make sure that the machine trying to access the Elasticsearch cluster has a security group that is authorized in the Elasticsearch cluster configuration.
Hope that helps,
Jeremy
Hi Nitin/Jeremy – Did it work? I have the same situation ( ES is on a private subnet. I have created 2 EC2s – 1) with private subnet , same subnet as ES , 2) public subnet ) . It is not working.
Hi Jeremy,
Thank you for a good reference. The approach worked with one additional step to provide access policy to the ES domain : “dont require signing request with IAM credentials”.
Please verify my understanding.
Pratik,
Yes, you are correct. You have to have your Elasticsearch Access Policy set up using the “Do not require signing request with IAM credential” template. I’ll update the post.
I followed all the steps correctly but i am getting ssh: connect to host my-IP-Address port 22: Operation timed out.
so i have some doubts
1) HostName 12.34.56.78 – what IP address is this? I pasted my EC2 instance’s private IP address. is that correct? If not what should be the IP here?
2) User ec2-user – what user is this? is this a IAM user , a AWS user?
Any help would be appriciated 🙂
Hi Mahesh,
The HostName should be your server’s PUBLIC IP address or public DNS, not your private IP.
The User is your Linux username, so it depends on the distro you’re using. If you are using Amazon Linux, then it will be “ec2-user”. If you were using Ubuntu, then it would be “ubuntu”. Debian would be “admin”. Etc.
Give that a try!
Jeremy
Thanks for you prompt reply.
I dont see Public IP /DNS assigned to my ec2 instance. But after further reading i got to know that we can create a Elastic IP and associate with our ec2 instance. So i did that, but no luck got the same message ‘Operation timed out.’
Mahesh,
Are you able to SSH into that instance? You may need to open up port 22 on a security group associated with that instance.
thanks for sharing, sir. it’s so good
Hi Jeremy,
I found that the subnet you attach the ec2 instance should have a route table with 1 rule for
intenet gateway. So once i had it i was able to SSH the instance but port forwarding did not work.
So the other solution which worked for me was i created an Lambda function under the same VPC as Elastic search and wrote a logic to get data depending on query we invoke.
Thanks for all the help and sharing this post. this was really helpful.
Is there any other way? I was under the impression that just settings the firewall would do. How about the VPN way?
Hi William,
The point of the VPC is to create a private cloud that isn’t directly accessible from the Internet. You could do port forwarding to your VPC Elasticsearch cluster from one of your EC2 instances and then open up that port to the web. However, this is HIGHLY insecure and I do not suggest you do that. Your only real security would be to assign IP restrictions, which can easily be spoofed.
If you have a VPN that can tunnel into your VPC, then there is no reason to use the method I describe in the post. The DNS for the VPC Elasticsearch cluster resolves to an internal VPC IP address, so that FQDN can be used in local applications as well.
Hope that helps!
Hello Jeremy,
I followed the whole tuto and read each comment but it does not work for me :/
I’m trying to connect to ES from my EC2-instance as you describe but like Nitin Ware (first comment), connexion fails.
I’ve checked the security groups. My EC2-instance has the same group as my ES cluster.
When i’m on my EC2-instance (threw ssh) , it can access the aws ES so I suppose its ok for the security group.
Have you any idea ?
Hi Florian,
If you can access your Elasticsearch cluster when you’re connected via ssh, then there should be no problem using the port forwarding. Are you SSHing into your EC2 instance via your
config
file, or are you using a full connection string? The first step is to make sure you can connect to your EC2 instance by adding an entry to yourconfig
file. Try adding everything above except for theLocalForward 9200 vpc-YOUR-ES-CLUSTER.us-east-1.es.amazonaws.com:443
line. If this doesn’t allow you to connect to your EC2 instance, then your configuration (hostname, user, etc.) is incorrect somehow. Once you can connect, adding theLocalForward
line should work.Jeremy
Hi Jeremy,
Thanks for your answer.
The above configuration did not work so I’ve searched and made the following changes on AWS
– Port 443 was added to the SG for the ES domain.
– The SG of the ES domain was added to the SG for the instance.
Upon making the above changes, I again attempted SSH estunnel on my mac.
ssh estunnel -N
Now, when I use the browser on my local machine I am able to access both of the following URLs.
Everything is good now !
What is sg?, can you paste your configuration here?
“SG” is in reference to the AWS Security Group.
Thanks Florian. The tip to add port 443 to the SG was the solution for me.
Hi Jeremy,
I’m not sure what to enter here :
IdentityFile ~/.ssh/MY-KEY.pem
I followed https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html to generate the file, however, my ssh tunnel command fails with “Permission denied (publickey).”
Can you please help?
Hi Vishal,
The
IdentityFile
is the private key that you generated as part of the “ec2-key-pairs.html” page you mentioned. If you are getting apermission denied
error, then it is possible that your permissions are set incorrectly on that file. Make sure that you are referencing the location of your private key correctly, and then be sure tochmod 400 MY-KEY.pem
so that the system has read permissions.Hope that helps!
You, sir, are a genius! Thank you so much for this.
Hello, thanks for great article.
I’m wondering – would it be possible to access Kibana/ES deployed in VPC by proxying the traffic via internet-facing load balancer ?
I would think so. An earlier version of the AWS Database Migration Service required a proxy server on EC2-Classic instances in order to transfer data into a VPC. Same concept applies here. I would think that you could easily do this with NGINX as described in Appendix A of this old doc: https://d1.awsstatic.com/whitepapers/RDS/Moving_RDS_MySQL_DB_to_VPC.pdf. You’ll have to configure your security groups correctly as well. This would be a bit of a security risk, however, depending on your use case.
Yes, it works using load balancer as reverse proxy “without authentication”.
I haven’t got it to work using Cognito.
You rock man! Amazing!.Works like a charm. I spent 2-3 hours troubleshooting incognito configuration.
It is giving me following error,
ssh: Could not resolve hostname estunnel: Name or service not known
Any help?
Looks like you figured it out! Let me know if you need more help.
Thanks,
Jeremy
Awesome. It worked.
Hi Jeremy,
Hope things are going well. I just setup elasticsearch in a VPC and also started an EC2 instance. I am able to SSH into the ec2 and curl elasticsearch though the vpc endpoint. I have also loaded some data into elasticsearch through logstash on my EC2. But I am not able to open kibana. I am new to SSH, networking and not sure where I am making a mistake. Below are my settings:
L9200 https://my-vcp.us-east-1.amazonaws.com:443
Google chrome says connection refused to localhost 9200. Can you please help me with this issue.
Thanks,
Nanda
Hi Nanda,
Are you starting the tunnel by running
ssh estunnel -N
in your terminal? What happens when you run that command? You can also runssh estunnel -Nv
to see some additional information about the status of the connection. Let me know!– Jeremy
Hi,
I’m using Putty on Windows.
I set up the ssh configs in Putty. I’m able to connect to my EC2 instance.
But on my Windows browser, I cannot connect to ES or Kibana.
Please help.
Thanks.
Hi Kent,
Once you are logged into your EC2 instance, can you connect to your Elasticsearch cluster?
– Jeremy
Did you resolve the problem? I have same issue.
Thanks for that. Saved my day.
This is great example of how to access and interact with the ES Service. If you wanted to access this securely (i.e. without using the -k option in curl, how would yo do that?
Hi Harry,
The
-k
option just skips verification of the certificate, the data itself is still encrypted. If you wanted to verify certificates, you’d need to be connected via a VPN.– Jeremy
Simply genius !
A great thanks.
Hi Jeremy,
This is great example how to access securely ES/Kibana in VPC but what about authorization. Have you tried to use AWS Cognito to authorize Kibana when hosted in VPC?
I am asking because I have a use case where not only one person (the admin) will access the Kibana so I need more granular control who use it.
Hi Petar,
I think it would be possible to use API Gateway to access your VPC (https://aws.amazon.com/about-aws/whats-new/2017/11/amazon-api-gateway-supports-endpoint-integrations-with-private-vpcs/) and apply Lambda Authorizers to control access.
If you make that work, I’d love to know about.
– Jeremy
When using aws-elasticsearch-client or the lower-level http-aws-es, I couldn’t find a way to disable strict SSL. This doesn’t appear to be a supported option.
You can side-step this issue by adding an entry mapping the Elasticsearch cluster domain name to 127.0.0.1 in your /etc/hosts file. That way the SSL check passes.
Hi Jeremy,
This is a great example of how to access the ES service within a VPC.
I’ve got it all working fine however since configuring my ES cluster with Cognito, whenever I now login it then gives me a 400 error code.
I’ve checked all the configurations including App client settings within the cognito pool settings and even the access policy for ES to allow the identity pool associated with my cognito user pool.
Can you think of anything that could be causing this?
I don’t think you can use Cognito to access ES, just Kibana. What exactly are you trying to accomplish? Cognito permissions to restrict access to ES API calls?
Thank you for the clear instructions. Spent all morning looking for a solution before finding this.
M@
thanks for the post. As an experiment, I am trying to load sql server table in AWS elastic service using local logstash on my mac. I followed these instructions as the ES is in vpc
I get the following error
[2019-12-02T21:29:55,621][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://localhost:9200/]}}
[2019-12-02T21:29:56,136][ERROR][logstash.javapipeline ][main] Pipeline aborted due to error {:pipeline_id=>”main”, :exception=>#
Any help is much appreciated.
Hi Suchi,
I don’t know enough about the logstash implementation to offer any meaningful help. Is there more context to the error message?
– Jeremy
Hey Jeremy,
I am following this article https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-vpc.html#kibana-test
I am able to load Kibana thru https://localhost:9200/_plugin/kibana/. However, when the user tries to sign in, he gets message on chrome: This site can’t be reached
Can you point what I am missing? I am pretty naive in thinking about the whole proxy stuff.
Do you know where Kibana is posting the login to? I’ve been able to use it without a problem.
could you resolve that? i have the same problem. thanks
Hi Jeremy! Great article. I am close and I am trying to connect through my EC2 instance to Kibana, but I am getting a 400 Bad Request error after I authenticate. I am using CentOS 7. Any ideas? Thank you.
Very very useful post. Thank you !
ssh: Could not resolve hostname estunnel: Name or service not known
I’m stuck on this, any help?
I am getting {“Message”:”User: anonymous is not authorized to perform: es:ESHttpGet”}. Anyone else getting this error and how to go about it
I am able to be forwarded to aws elasticsearch through ssh.
But I am not able to access the elastic search link https://localhost:9200 in web browser
It shows PR_CONNECT_RESET_ERROR in firefox
ERR_CONNECTION_RESET in chrome
(curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to localhost:9200) in curl
Can you please tell me how to fix this issue ?
Hey Jeremy great article. I was able to access es just fine following this. However I am running into an issue with accessing Kibana when using Cognito. The problem I believe is when you sign in with cognito it takes you to your custom page test.auth.us-west-2.amazoncognito.com, and passes a redirect url of the ES Endpoint, not localhost. So when the login is successful it tries to request something that is not accessible.
Thank you for this great article. I created a docker config for it:
https://gist.github.com/Schwankenson/f02297f24df82eb613466499b0a0400e
i do get “Message”:”User: anonymous is not authorized to perform: es:ESHttpGet”} Can some please help on what to do.
Hi Jeremy,
Thanks for this great article. What is the approach to migrate data with public endpoint hosted on EC2-Classic to EC2-VPC of ES domain hosted inside VPC private subnet.
Looking forward to a work around solution
I am getting bellow error how to solve this
[ec2-user@ip-10-0-0-243 .ssh]$ cat config/ec.sh
# Elasticsearch Tunnel
Host estunnel
HostName 18.223.106.105 # your server’s public IP address
User ec2-user
IdentitiesOnly yes
IdentityFile ~/.ssh/new.pem
LocalForward 9200 vpc-newdomain-q3o4gempvoyb6esprenquk5zcq.us-east-2.es.amazonaws.com:443
[ec2-user@ip-10-0-0-243 .ssh]$ ssh estunnel -N
ssh: Could not resolve hostname estunnel: Name or service not known
[ec2-user@ip-10-0-0-243 .ssh]$