Collecting CloudWatch Logs
This page contains instructions required to be fulfilled, to enable BluSapphire Log-Agent to collect log data from Amazon CloudWatch Log Groups.
Overview
Amazon CloudWatch can be used to store log files from Amazon Elastic Compute Cloud (EC2), AWS CloudTrail, Route53, and other sources. CloudWatch Logs enables you to centralize the logs from all your systems, applications, and AWS services that you use, in a single, highly scalable service.
A log group is a group of log streams that share the same retention, monitoring, and access control settings. You can define log groups and specify which streams to put into each group. There is no limit on the number of log streams that can belong to one log group.
A log stream is a sequence of log events that share the same source. Each separate source of logs in CloudWatch Logs makes up a separate log stream.
BluSapphire Log-Agent makes AWS API calls to retrieve and process logs from all log streams within a specific log group. This page showcases the steps required to be fulfilled and configured for setting up BluSapphire Log-Agent.
Perquisites
AWS account with sufficient access to Amazon IAM, CloudWatch Services.
Already have created and configured log groups within your CloudWatch Management Console for the required AWS Service and the logs are being written to their respective CloudWatch log groups.
If not, please follow the documentation to configure Amazon CloudWatch as a logging destination:
For instructions on configuring CloudTrail to publish log data to the Amazon CloudWatch log group.
For instructions to configure VPC Flow to publish flow log data directly to Amazon CloudWatch.
Step 01: AWS IAM Policies and Permissions
An AWS IAM policy is an entity that defines permissions to an object within your AWS environment. Creating a customized IAM policy for BluSapphire Log Shipper with the required set of permissions is needed for API calls to access and process log data from AWS CloudWatch Log-Groups.
| AWS Permissions Required | Description |
|---|---|
logs:DescribeLogGroups logs:FilterLogEvents | BluSapphire Agent requires these permissions to access and retrieve data from the CloudWatch log group. |
Create IAM Policy
1) To create a new IAM policy, from AWS Management Console - go to the IAM management console via the services menu -> navigate to “Policies” under “Access Management” from the left side menu and click on “create policy”.
2) One way to configure the policy is using the visual editor within Create Policy Screen. Choose “CloudWatch Logs” for the service and select “DescribeLogGroups, FilterLogEvents” for actions.
3) To specify a resource, choose “Add ARN” and provide the ARN of the respective CloudWatch log group you want access to, and click on Add as shown. Note: Log-Group ARN can be found in the log group details section within the CloudWatch management console.
4) Another way to configure the policy is to switch to the JSON tab within Create Policy Screen. Copy-Paste below JSON policy and update the “LOG_GROUP-ARN” with the respective CloudWatch log group.
5) Continue to review the policy - provide name, appropriate description, and save.
Note: In case, if you want to send logs to BluSapphire from multiple CloudWatch Log Groups, then you must update the above-created policy with ARN’s of all the Log Groups and make them accessible for the BluSapphire agent.
Step 02: Configure IAM User
It's recommended to create a new user with the required permission and policies for collecting logs from AWS.
Create a New IAM User
1) To create a new IAM user, from AWS Management Console - go to the IAM management console via the services menu -> navigate to “Users” under “Access Management” from the left side menu, and click on “Add users".
2) Provide the appropriate username, and select “Access key” for AWS credential type.
3) Choose to attach existing policies directly, search and select the IAM policy that was created earlier, and continue.
4) Review the user details and permissions for the new user and click create a user.
5) Download the CSV file and make note of “Access key ID” and “Secret access key” as these are needed while configuring BluSapphire log-shipper agent.
Configure Existing IAM User
1) Updating existing IAM User’s permission with the newly created Policy for CloudWatch, from AWS Management Console - go to IAM management console via the services menu -> navigate to “Users” under “Access Management” from the left side menu. Select an existing user created for the purpose of shipping logs, and click on “Add Permissions”.
2) Choose to attach existing policies directly and select the previously created CloudWatch policy, review and save the changes.
3) Generate, and make note of “Access key ID” and “Secret access key” for the log shipping user, as these are needed while configuring BluSapphire log-shipper agent.
Step 03: Configure BluSapphire Log-Shipper Agent
The following information is required to configure the BluSapphire agent for collecting logs from CloudWatch Log Groups:
| Required AWS Information | Description |
|---|---|
Access key ID | AWS access key of the IAM user created/updated in “Step 02” |
Secret access key | AWS access secret of the IAM user created/updated in “Step 02” |
Log Group ARN (log_group_arn) | ARN of the log group you want the BluSapphire agent to collect data from |
Log Group Name (log_group_name) | Name of the log groups you want the BluSapphire agent to collect data from |
Region Name (region_name) | Region that the specified log group or log group prefix belongs to |
(optional) Prefix for Log Group Name (log_group_name_prefix) | Prefix for a group of log group names |
Note: Having all the above configuration steps performed, and the BluSapphire agent is configured, it may take some time to get the logs populated in the BluSapphire portal.
Last updated
