Organizations with sensitive data spread across multiple databases, data warehouses or lakes can strengthen their data security by reviewing audit logs. However, the large volume of audit data makes it difficult to analyze and extract value in real-time. We look at how integrating Satori and AWS OpenSearch sends the Satori audit logs to an OpenSearch instance so that users can analyze these logs at scale.
The integration looks like:
Now let’s drill down!
What is AWS OpenSearch?
AWS OpenSearch is an open-source tool that allows developers to add search functionality to their applications and websites. Using AWS OpeanSearch developers can use dashboards to monitor and analyze large volumes of data, conduct log analytics and website searches. A significant benefit of AWS OpenSearch is that it scales easily and provides quick access and responses to a large amount of data.
Satori and AWS OpenSearch
Satori’s data security platform secures and centralizes all your data access and audit logs from all data stores, into one place. The data access logs, enriched with metadata, and detailed identity and security policies are presented in a single view. Users can see who accessed production data where and when, and the application of any security or identify policies.
Integrating Satori’s rich audit and access logs with OpenSearch enables users to develop a dashboard that consolidates all of this information in real-time. Users can then run analytics from an OpenSearch dashboard on the large volume of audit data gaining insights to strengthen security and compliance requirements.
We use the following tools to integrate Satori with OpenSearch in this example. Please note there are numerous ways to solve this type of problem using a variety of languages, platforms, and components. This solution is just one of many possible solutions.
- Set up Satori account
- Set up OpenSearch domain, yourdomain-opensearch
- AWS Secret to store usernames and passwords, your-lambda-secrets
- AWS Lambda to actually join Satori with OpenSearch, your-opensearch-python
- Add the following to your Lambda layer
- AWS Layer – “DataWrangler”, AWSDataWrangler-Python38
- AWS CloudWatch trigger
- Permission to access AWS Secret
- Add the following to your Lambda layer
- Configure AWS OpenSearch
1. Satori Account Setup Prerequisites
- Administrator access to a Satori platform account.
- A Satori service account and key.
- Make sure that you make a note of these three pieces of information to continue this integration:
- Account ID
- Service ID
- Service Key
2. AWS OpenSearch Deployment
Create a simple, lightweight AWS OpenSearch domain instance with the following characteristics:
- Fine-grained access using a username/password
- Public network access
- No custom endpoint
During the creation of your AWS domain instance: your-opensearch, enter a master username and password.
Take note of this information for the next step below.
After the domain is deployed, you can log in with your username and password. The endpoint is displayed in the configuration details.
3. AWS Secrets
In this case, we used AWS Secrets Manager for storing secrets, usernames and passwords:
In the actual secret text, we use multiple name value pairs. Later on, our python sample code will parse the values as needed:
If you use the sample python code, then you must fill in all of the keys/names shown above, exactly as shown, as well as the relevant values.
4. AWS Lambda
AWS Lambda ties the Satori Rest API to the AWS OpenSearch Domain. We have provided a sample github gist to get you started.
Create a new lambda function with the following characteristics:
- Using Python 3.8 engine
- Add the AWS layer called “DataWrangler”
- Add a trigger for AWS CloudWatch, set to every 15 minutes.
For the layer, we selected AWS “DataWrangler” as it contains all of the useful python modules we need for our sample code:
Now we can change the python code, using this python gist provided by Satori. You can paste it into the Lambda code editor, but please note, you need to make changes as listed below:
You need to change the following lines at the top:
Then in the AWS code editor, save the file, and also click the Deploy button.
To add the AWS CloudWatch trigger to your Lambda function, navigate to Configuration>Triggers:
You also need to add permissions for this Lambda function to access AWS Secret Manager. Note: there are numerous ways to attach AWS security to resources, this is just one possible example.
Don’t forget to deploy your code changes in AWS Lambda!
5. AWS OpenSearch Configuration
We now need to tell OpenSearch that our timestamp field is actually a timestamp. To do this, we modify the mapping for the index when creating it for the first time. From the homepage of OpenSearch, navigate to the API page:
And run the following command:
You may also need to create and/or refresh the Index patterns for your new Satori audit data.
From the home page navigation of OpenSearch, choose Stack Management > Index Patterns:
If you click on the name of your Satori index, make sure that it looks like the following:
6. The Integration
You can now start creating OpenSearch views and dashboards for your Satori audit data!
Integrating Satori and AWS OpenSearch allows you to use the OpenSearch dashboards and analyze large volumes of enriched audit logs. You can gain insights from the analyzed audit logs and strengthen your ability to meet security and compliance requirements.
To learn more about Satori: