AQUILA GCP Integration

This Google Cloud integration collects and analyzes a wide range of logs and metrics to provide comprehensive visibility into your cloud environment. It ingests Firewall Logs, VPC Flow Logs, DNS Logs, and Load Balancing Logs exported from Cloud Logging via a Pub/Sub topic sink. Additionally, it gathers detailed metrics and metadata from Google Cloud Monitoring across core services, including Compute Engine, Cloud SQL, Cloud Run, GKE, Firestore, Dataproc, Pub/Sub, Redis, Storage, Load Balancing, and Billing. This enables in-depth monitoring of infrastructure, application performance, network activity, and cost trends.

Logs

Metrics
Authentication

To use this Google Cloud Platform (GCP) integration, you need to set up a Service Account with a Role and a Service Account Key to access data on your GCP project.

1. Service Account

First, you need to create a Service Account. A Service Account (SA) is a particular type of Google account intended to represent a non-human user who needs to access the GCP resources.

The AQUILA Agent uses the SA to access data on Google Cloud Platform using the Google APIs.

2. Required IAM Service Account Roles:
3. Logs Collection Configuration 

With a properly configured Service Account and the integration setting in place, it’s time to start collecting some logs. 

Requirements 

You need to create a few dedicated Google Cloud resources before starting, in detail: 

  • Pub/Sub Topic: A messaging endpoint where publishers send messages that can then be delivered to one or more subscribers.

  • Subscription: A configuration attached to a Pub/Sub topic that delivers messages to subscribers, either by push or pull.

  • Log Sink: A configuration that routes logs from Cloud Logging to a specified destination such as Pub/Sub, Cloud Storage, or BigQuery.

 It’s recommended to have separate Pub/Sub topics for each of the log types so that they can be parsed and stored in a specific data stream. 

Here’s an example of collecting Audit Logs using a Pub/Sub topic, a subscription, and a Log Router. We will create the resources in the Google Cloud Console and then configure the Google Cloud Platform integration. 

Example Setup Using Google Cloud Console
  1. Navigate to "Logging" > "Log Router" > "Create Sink".

  2. Provide a Sink name and description.

  3. For Sink destination, select "Cloud Pub/Sub topic". Choose an existing topic or create a new one.

  4. If a new topic is created, you must also create a subscription for it.

  5. Under "Choose logs to include in sink", use a filter like: logName:"cloudaudit.googleapis.com"

4. Enable API Services
5. Service Account Key  

Next, with the Service Account (SA) with access to Google Cloud Platform (GCP) resources setup, you need some credentials to associate with it: a Service Account Key.  

From the list of SA (Service Accounts):  

  1. Go to IAM & Admin > Service Accounts in the GCP Console.
  2. Click the service account you created.
  3. Under the "Keys" section, click "Add Key" > "Create new key".
  4. Choose JSON as the key type.
  5. Download and securely store the generated private key (it cannot be retrieved again from GCP if lost).

Please provide the following information to CyTech: 

  • Project ID  - The Project ID is the Google Cloud project ID where your resources exist. 

  • Credentials File - Save the JSON file with the private key in a secure location of the file system, and make sure that the Log Collector Agent has at least read-only privileges to this file. Specify the file path in the Log Collector Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json.

  • Pub/Sub Topic Name of the topic where the logs are written to.

  • Subscription - Use the short subscription name here, not the full-blown path with the project ID. You can find it as "Subscription ID" on the Google Cloud Console. 

 

If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.


Revision #1
Created 26 June 2025 09:30:31 by Richmond Abella
Updated 26 June 2025 09:34:09 by Richmond Abella