Collecting CloudWatch Logs
This page contains instructions required to be fulfilled, to enable BluSapphire Log-Agent to collect log data from Amazon CloudWatch Log Groups.
Last updated
This page contains instructions required to be fulfilled, to enable BluSapphire Log-Agent to collect log data from Amazon CloudWatch Log Groups.
Last updated
Amazon CloudWatch can be used to store log files from Amazon Elastic Compute Cloud (EC2), AWS CloudTrail, Route53, and other sources. CloudWatch Logs enables you to centralize the logs from all your systems, applications, and AWS services that you use, in a single, highly scalable service.
A log group is a group of log streams that share the same retention, monitoring, and access control settings. You can define log groups and specify which streams to put into each group. There is no limit on the number of log streams that can belong to one log group.
A log stream is a sequence of log events that share the same source. Each separate source of logs in CloudWatch Logs makes up a separate log stream.
BluSapphire Log-Agent makes AWS API calls to retrieve and process logs from all log streams within a specific log group. This page showcases the steps required to be fulfilled and configured for setting up BluSapphire Log-Agent.
AWS account with sufficient access to Amazon IAM, CloudWatch Services.
Already have created and configured log groups within your CloudWatch Management Console for the required AWS Service and the logs are being written to their respective CloudWatch log groups.
If not, please follow the documentation to configure Amazon CloudWatch as a logging destination:
For instructions on configuring CloudTrail to publish log data to the Amazon CloudWatch log group.
For instructions to configure VPC Flow to publish flow log data directly to Amazon CloudWatch.
An AWS IAM policy is an entity that defines permissions to an object within your AWS environment. Creating a customized IAM policy for BluSapphire Log Shipper with the required set of permissions is needed for API calls to access and process log data from AWS CloudWatch Log-Groups.
logs:DescribeLogGroups
logs:FilterLogEvents
BluSapphire Agent requires these permissions to access and retrieve data from the CloudWatch log group.
1) To create a new IAM policy, from AWS Management Console - go to the IAM management console via the services menu -> navigate to “Policies” under “Access Management” from the left side menu and click on “create policy”.
2) One way to configure the policy is using the visual editor within Create Policy Screen. Choose “CloudWatch Logs” for the service and select “DescribeLogGroups, FilterLogEvents” for actions.
3) To specify a resource, choose “Add ARN” and provide the ARN of the respective CloudWatch log group you want access to, and click on Add as shown. Note: Log-Group ARN can be found in the log group details section within the CloudWatch management console.
4) Another way to configure the policy is to switch to the JSON tab within Create Policy Screen. Copy-Paste below JSON policy and update the “LOG_GROUP-ARN” with the respective CloudWatch log group.
5) Continue to review the policy - provide name, appropriate description, and save.
Note: In case, if you want to send logs to BluSapphire from multiple CloudWatch Log Groups, then you must update the above-created policy with ARN’s of all the Log Groups and make them accessible for the BluSapphire agent.
It's recommended to create a new user with the required permission and policies for collecting logs from AWS.
1) To create a new IAM user, from AWS Management Console - go to the IAM management console via the services menu -> navigate to “Users” under “Access Management” from the left side menu, and click on “Add users".
2) Provide the appropriate username, and select “Access key” for AWS credential type.
3) Choose to attach existing policies directly, search and select the IAM policy that was created earlier, and continue.
4) Review the user details and permissions for the new user and click create a user.
5) Download the CSV file and make note of “Access key ID” and “Secret access key” as these are needed while configuring BluSapphire log-shipper agent.
1) Updating existing IAM User’s permission with the newly created Policy for CloudWatch, from AWS Management Console - go to IAM management console via the services menu -> navigate to “Users” under “Access Management” from the left side menu. Select an existing user created for the purpose of shipping logs, and click on “Add Permissions”.
2) Choose to attach existing policies directly and select the previously created CloudWatch policy, review and save the changes.
3) Generate, and make note of “Access key ID” and “Secret access key” for the log shipping user, as these are needed while configuring BluSapphire log-shipper agent.
The following information is required to configure the BluSapphire agent for collecting logs from CloudWatch Log Groups:
Access key ID
AWS access key of the IAM user created/updated in “Step 02”
Secret access key
AWS access secret of the IAM user created/updated in “Step 02”
Log Group ARN (log_group_arn)
ARN of the log group you want the BluSapphire agent to collect data from
Log Group Name (log_group_name)
Name of the log groups you want the BluSapphire agent to collect data from
Region Name
(region_name)
Region that the specified log group or log group prefix belongs to
(optional) Prefix for Log Group Name (log_group_name_prefix)
Prefix for a group of log group names
Note: Having all the above configuration steps performed, and the BluSapphire agent is configured, it may take some time to get the logs populated in the BluSapphire portal.
