GCP LoadBalancer (bucket)

GCP Load Balancer log Ingestion using Filebeat

Phase 1: Configure Your GCP Project

1. Enable GCP Load Balancer Logging

1. In the Google Cloud console, go to the Load Balancing page.

2. Click the name of your load balancer.

3. Click edit.

4. Click Backend Configuration.

5. Click edit next to your backend service.

6. In the Logging section, select the Enable logging checkbox.

7. To finish editing the backend service, click Update.

8. To finish editing the load balancer, click Update.

2. Create a GCS bucket (destination)

Create a bucket to store LB logs.

3. Create a Log Sink

1. The sink will filter your firewall logs and send them to the GCP Bucket.

2. Navigate to Logging > Log Router.

3. Click Create Sink.

4. Name the sink (e.g., lb-log-sink) and click Next.

5. For the Sink destination, select "Cloud Storage bucket" as the sink service and choose the GCP Bucket you just created. Click Next.

6. Please modify the Build Inclusion Filter and apply the appropriate filter to include GCP load balancer logs.

7. Click Create Sink.

4. Set Sink Permissions

1. Cloud Logging needs permission to write to your bucket.

2. After creating the sink, look at the Writer Identity column in the Log Router list. It will look like [e-mail].

3. Copy this service account address.

4. Navigate to Cloud Storage > Buckets and select your log bucket.

5. Click the Permissions tab > Grant Access.

6. Paste the service account address in the "New principals" field.

7. Assign the role: Storage Object Creator.

8. Click Save.

Phase 2: Create a Service Account for Filebeat

Filebeat needs its own identity to read the logs from GCP.

1. Navigate to IAM & Admin > Service Accounts.

2. Click Create Service Account, give it a name (e.g., filebeat-agent), and click Create and Continue.

3. In the "Grant access" step, assign the role of Storage Object Viewer. Click Done.

4. Find the service account you just created, click on the three dots at the right and select Manage keys option.

5. Go to its KEYS tab, click ADD KEY, and select Create new key.

6. Choose JSON and click CREATE. A JSON key file will be downloaded to your computer.

Caution: This integration method has a limitation, as logs are exported to the Cloud Storage bucket at periodic intervals (every few minutes). Consequently, log ingestion into the platform is not real-time, and there may be a delay of few mins before events become available for monitoring and analysis.