System Integrations
- Cyber Incident Monitoring Integration Procedure
- Microsoft 365
- GitHub
- Add Windows Integrations
- Sysmon for Linux
- 1 Password Integrations
- Atlassian Bitbucket Integrations
- AWS Cloudtrails Integrations
- AWS GuardDuty Integrations
- AWS Security Hub Integrations
- AWS Integrations
- CISCO Meraki Integrations
- CISCO Secure Endpoint Integrations
- CISCO Umbrella Integrations
- Cloudflare Integration
- Crowdstrike Integrations
- Dropbox Integrations
- F5 Integrations
- Fortinet-Fortigate Integrations
- GCP Integrations
- GitLab Integrations
- Google Workspace Integrations
- Jumpcloud Integrations
- Mimecast Integrations
- MongoDB Integrations
- OKTA Integrations
- Pulse Connect Secure Integrations
- Slack Integrations
- System Integrations
- Team Viewer Integrations
- Z Scaler Integrations
- gcp
- VMware vSphere Integration
Cyber Incident Monitoring Integration Procedure
Go to > Cyber Incident Monitoring
Microsoft 365
Microsoft Office 365 integration currently supports user, admin, system, and policy actions and events from Office 365 and Azure AD activity logs exposed by the Office 365 Management Activity API.
Procedures
To perform the setup, please confirm that you have the following access:
-
A Microsoft Office 365 account with Administrative Privileges
-
A Microsoft Azure account with Administrative Privileges
Register a new Office 365 web application To get started collecting Office 365 logs, register an Office 365 web application:
-
Log into the Office 365 portal as an Active Directory tenant administrator.
-
Click Show all to expand the left navigation area, and then click Azure Active Directory.
-
Select App Registrations, and then click + New application registration.
-
Provide the following information in the fields:
-
-
-
Name – for example, o365cytech.
-
Select Single tenant for supported account types.
-
Leave the Redirect URI blank.
-
The Audit Log Search needs to be enabled.
-
Click Register and note the Application (client) ID.
-
-
Setup Active Directory security permissions
The Active Directory security permissions allow the application you created to read threat intelligence data and activity reports for your organization.
To set up Active Directory permissions:
- On the main panel under the new application, click API Permissions, and then click + Add a permission.
- Locate and click on Office 365 Management APIs.
- In Application permissions, expand and select ActivityFeed.Read, ActivityFeed.ReadDlp, ActivityReports.Read, and ServiceHealth.Read
- Ensure all necessary permissions are selected, and then click Add permissions.
- Click Grant admin consent, and then click Accept to confirm.
- On the left navigation area, select Certificates & secrets, and then click + New client secret.
- Make Sure to Copy the Value (Client Secret (Api Key) will disappear
- Type a key Description and set the duration to Never or Maximum Grant time.
- Click Add.
- Click Overview to return to the application summary, and then click the link under Managed application in local directory.
- Click Properties, and then note the Object ID associated with the application.
GitHub
Introduction
The GitHub integration collects events from the GitHub API.
Logs
Audit
The GitHub audit log records all events related to the GitHub organization.
To use this integration, you must be an organization owner, and you must use an Personal Access Token with the admin:org scope.
This integration is not compatible with GitHub Enterprise server.
Code Scanning
The Code Scanning lets you retrieve all security vulnerabilities and coding errors from a repository setup using Github Advanced Security Code Scanning feature.
To use this integration, GitHub Apps must have the security_events read permission. Or use a personal access token with the security_events scope for private repos or public_repo scope for public repos.
Secret Scanning
The Github Secret Scanning lets you retrieve secret scanning for advanced security alerts from a repository setup using Github Advanced Security Secret Scanning feature.
To use this integration, GitHub Apps must have the secret_scanning_alerts read permission. Or you must be an administrator for the repository or for the organization that owns the repository, and you must use a personal access token with the repo scope or security_events scope. For public repositories, you may instead use the public_repo scope.
Dependabot
The Github Dependabot lets you retrieve known vulnerabilities in dependencies from a repository setup using Github Advanced Security Dependabot feature.
To use this integration, you must be an administrator for the repository or for the organization that owns the repository, and you must use a personal access token with the repo scope or security_events scope. For public repositories, you may instead use the public_repo scope.
Issues
The Github Issues datastream lets you retrieve github issues, including pull requests, issue assignees, comments, labels, and milestones. See About Issues for more details. You can retrieve issues for specific repository or for entire organization. Since Github API considers pull requests as issues, users can use github.issues.is_pr field to filter for only pull requests.
All issues including closed are retrieved by default. If users want to retrieve only open requests, you need to change State parameter to open.
To use this integration, users must use Github Apps or Personal Access Token with read permission to repositories or organization. Please refer to Github Apps Permissions Required and Personal Access Token Permissions Required for more details.
GitHub Integration Procedures
Please provide the following information to CyTech:
1.Select Settings
2. Select Developer Settings
3. Select token (classic)
4. Select scope admin:scope
Collect GitHub logs via API
-
Personal Access Token - the GitHub Personal Access Token. Requires the 'admin:org' scope
-
Organization Name - The GitHub organization name/ID
GHAS Code Scanning
-
Personal Access Token - the GitHub Personal Access Token. Requires the 'public_repo' scope for public repositories and 'security_events' scope for private repositories. \nSee List code scanning alerts for a repository
-
Repository owner - The owner of GitHub Repository. If repository belongs to an organization, owner is name of the organization
GHAS Dependabot
-
Personal Access Token - The GitHub Personal Access Token. \nSee Authenticating with GraphQL
-
Repository owner - The owner of GitHub Repository
Github Issues
1. Personal Access Token - the GitHub Personal Access Token.
2. Repository owner - The owner of GitHub Repository. If repository belongs to an organization, owner is name of the organization.
GHAS Secret Scanning
1. Personal Access Token - the GitHub Personal Access Token. Requires admin access to the repository or organization owning the repository along with a personal access token with 'public_repo' scope for public repositories and repo or security_events scope for private repositories. \nSee List secret scanning alerts for a repository
2. Repository owner - The owner of GitHub Repository
Add Windows Integrations
Introduction
The Windows integration allows you to monitor the Windows OS, services, applications, and more.
Use the Windows integration to collect metrics and logs from your machine. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue.
For example, if you wanted to know if a Windows service unexpectedly stops running, you could install the Windows integration to send service metrics to Elastic. Then, you could view real-time changes to service status in Kibana's [Metrics Windows] Services dashboard.
Data streams
The Windows integration collects two types of data: logs and metrics.
Logs help you keep a record of events that happen on your machine. Log data streams collected by the Windows integration include forwarded events, PowerShell events, and Sysmon events. Log collection for the Security, Application, and System event logs is handled by the System integration. See more details in the Logs reference.
-
https://aquila-elk.kb.us-east-1.aws.found.io:9243/app/integrations/detail/windows-1.15.2/overview#logs-reference
Metrics give you insight into the state of the machine. Metric data streams collected by the Windows integration include service details and performance counter values. See more details in the Metrics reference.
Note: For 7.11, security, application and system logs have been moved to the system package.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
Each data stream collects different kinds of metric data, which may require dedicated permissions to be fetched and which may vary across operating systems.
Setup
For step-by-step instructions on how to set up an integration, see the Getting started guide.
Note: Because the Windows integration always applies to the local server, the hosts config option is not needed.
Ingesting Windows Events via Splunk
This integration allows you to seamlessly ingest data from a Splunk Enterprise instance. The integration uses the httpjson input in Elastic Agent to run a Splunk search via the Splunk REST API and then extract the raw event from the results. The raw event is then processed via the Elastic Agent. You can customize both the Splunk search query and the interval between searches. For more information see Ingest data from Splunk.
Note: This integration requires Windows Events from Splunk to be in XML format. To achieve this, renderXml needs to be set to 1 in your inputs.conf file.
Logs reference
Forwarded
The Windows forwarded data stream provides events from the Windows ForwardedEvents event log. The fields will be the same as the channel specific data streams.
Powershell
The Windows powershell data stream provides events from the Windows Windows PowerShell event log.
System Integration Procedures
Collect events from the following Windows event log channels: (Enable Yes/No)
-
Preserve original event (Enable Yes/No)
-
Preserves a raw copy of the original XML event, added to the field event.original
-
Event ID
-
A list of included and excluded (blocked) event IDs. The value is a comma-separated list. The accepted values are single event IDs to include (e.g. 4624), a range of event IDs to include (e.g. 4700-4800), and single event IDs to exclude (e.g. -4735). Limit 22 IDs.
-
Ignore events older than
-
If this option is specified, events that are older than the specified amount of time are ignored. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
Language ID
-
The language ID the events will be rendered in. The language will be forced regardless of the system language. A complete list of language IDs can be found https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c[here]. It defaults to 0, which indicates to use the system language. E.g.: 0x0409 for en-US
-
Tags
-
Processors
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
Synthetic source (Enable Yes/No)
Powershell (Enable Yes/No)
-
Preserve original event (Enable Yes/No)
-
Preserves a raw copy of the original XML event, added to the field event.original
-
Event ID
-
A list of included and excluded (blocked) event IDs. The value is a comma-separated list. The accepted values are single event IDs to include (e.g. 4624), a range of event IDs to include (e.g. 4700-4800), and single event IDs to exclude (e.g. -4735). Limit 22 IDs.
-
Ignore events older than
-
If this option is specified, events that are older than the specified amount of time are ignored. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
Language ID
-
The language ID the events will be rendered in. The language will be forced regardless of the system language. A complete list of language IDs can be found https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c[here]. It defaults to 0, which indicates to use the system language. E.g.: 0x0409 for en-US
-
Tags
-
Processors
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
Synthetic source (Enable Yes/No)
Powershell Operational (Enable Yes/No)
-
Preserve original event (Enable Yes/No)
-
Preserves a raw copy of the original XML event, added to the field event.original
-
Event ID
-
A list of included and excluded (blocked) event IDs. The value is a comma-separated list. The accepted values are single event IDs to include (e.g. 4624), a range of event IDs to include (e.g. 4700-4800), and single event IDs to exclude (e.g. -4735). Limit 22 IDs.
-
Ignore events older than
-
If this option is specified, events that are older than the specified amount of time are ignored. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
Language ID
-
The language ID the events will be rendered in. The language will be forced regardless of the system language. A complete list of language IDs can be found https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c[here]. It defaults to 0, which indicates to use the system language. E.g.: 0x0409 for en-US
-
Tags
-
Processors
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
Synthetic source (Enable Yes/No)
Sysmon Operational (Enable Yes/No)
-
Preserve original event (Enable Yes/No)
-
Preserves a raw copy of the original XML event, added to the field event.original
-
Event ID
-
A list of included and excluded (blocked) event IDs. The value is a comma-separated list. The accepted values are single event IDs to include (e.g. 4624), a range of event IDs to include (e.g. 4700-4800), and single event IDs to exclude (e.g. -4735). Limit 22 IDs.
-
Ignore events older than
-
If this option is specified, events that are older than the specified amount of time are ignored. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
Language ID
-
The language ID the events will be rendered in. The language will be forced regardless of the system language. A complete list of language IDs can be found https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c[here]. It defaults to 0, which indicates to use the system language. E.g.: 0x0409 for en-US
-
Tags
-
Processors
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
Synthetic source (Enable Yes/No)
Collect Windows perfmon and service metrics (Enable Yes/No)
-
Perfmon Group Measurements By Instance (Enable Yes/No)
-
Enabling this option will send all measurements with a matching perfmon instance as part of a single event
-
Perfmon Ignore Non Existent Counters (Enable Yes/No)
-
Enabling this option will make sure to ignore any errors caused by counters that do not exist
-
Perfmon Queries
-
Will list the perfmon queries to execute, each query will have an object option, an optional instance contiguration and the actual counters
-
Period
-
Synthetic source (Enable Yes/No)
Windows service metrics (Enable Yes/No)
-
Period
-
Processors
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
Synthetic source (Enable Yes/No)
Collect logs from third-party REST API (experimental) (Enable Yes/No)
-
URL of Splunk Enterprise Server
-
i.e. scheme://host:port, path is automatic
-
Splunk REST API Username
-
Splunk Authorization Token
-
Bearer Token or Session Key, e.g. "Bearer eyJFd3e46..." or "Splunk 192fd3e...". Cannot be used with username and password.
-
SSL Configuration
-
i.e. certificate_authorities, supported_protocols, verification_mode etc.
Windows ForwardedEvents via Splunk Enterprise REST API (Enable Yes/No)
-
Interval to query Splunk Enterprise REST API
-
Go Duration syntax (eg. 10s)
-
Preserve original event (Enable Yes/No)
-
Preserves a raw copy of the original event, added to the field event.original
-
Splunk search string
-
Tags
-
Processors
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
Synthetic source (Enable Yes/No)
Windows Powershell Events via Splunk Enterprise REST API (Enable Yes/No)
-
Interval to query Splunk Enterprise REST API
-
Go Duration syntax (eg. 10s)
-
Preserve original event (Enable Yes/No)
-
Preserves a raw copy of the original event, added to the field event.original
-
Splunk search string
-
Tags
-
Processors
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
Synthetic source (Enable Yes/No)
Windows Powershell Operational Events via Splunk Enterprise REST API (Enable Yes/No)
-
Interval to query Splunk Enterprise REST API
-
Go Duration syntax (eg. 10s)
-
Preserve original event (Enable Yes/No)
-
Preserves a raw copy of the original event, added to the field event.original
-
Splunk search string
-
Tags
-
Processors
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
Synthetic source (Enable Yes/No)
Windows Sysmon Operational Events via Splunk Enterprise REST API (Enable Yes/No)
-
Interval to query Splunk Enterprise REST API
-
Go Duration syntax (eg. 10s)
-
Preserve original event (Enable Yes/No)
-
Preserves a raw copy of the original event, added to the field event.original
-
Splunk search string
-
Tags
-
Processors
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
Synthetic source (Enable Yes/No)
Sysmon for Linux
Introduction
The Sysmon for Linux integration allows you to monitor the Sysmon for Linux, which is an open-source system monitor tool developed to collect security events from Linux environments.
Use the Sysmon for Linux integration to collect logs from linux machine which has sysmon tool running. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue.
NOTE: To collect Sysmon events from Windows event log, use Windows sysmon_operational data stream instead.
-
Sysmon for Linux - https://github.com/Sysinternals/SysmonForLinux
-
Windows sysmon_operational data stream - https://docs.elastic.co/en/integrations/windows#sysmonoperational
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
Setup
For step-by-step instructions on how to set up an integration, see the Getting started guide.
Data streams
The Sysmon for Linux log data stream provides events from logs produced by Sysmon tool running on Linux machine.
Sysmon for Linux Integration
Please provide the following information to CyTech:
Collect Sysmon for Linux logs (Enable Yes/No)
-
Paths - /var/log/sysmon*
1 Password Integrations
Introduction
With 1Password Business, you can send your account activity to your security information and event management (SIEM) system, using the 1Password Events API.
Get reports about 1Password activity, such as sign-in attempts and item usage, while you manage all your company’s applications and services from a central location.
With 1Password Events Reporting and Elastic SIEM, you can:
-
Control your 1Password data retention
-
Build custom graphs and dashboards
-
Set up custom alerts that trigger specific actions
-
Cross-reference 1Password events with the data from other services
Events
Sign-in Attempts
Use the 1Password Events API to retrieve information about sign-in attempts. Events include the name and IP address of the user who attempted to sign in to the account, when the attempt was made, and – for failed attempts – the cause of the failure.
Item Usages
This uses the 1Password Events API to retrieve information about items in shared vaults that have been modified, accessed, or used. Events include the name and IP address of the user who accessed the item, when it was accessed, and the vault where the item is stored.
Requirements
You can set up Events Reporting if you’re an owner or administrator.
Ready to get started?
1Password Integration Procedures
Please provide the following information to CyTech:
The 1Password Events API Beat returns information from 1Password through requests to the Events REST API and sends that data securely to Elasticsearch. Requests are authenticated with a bearer token. Issue a token for each application or service you use.
To connect your 1Password account to Elastic:
-
Download and install the 1Password Events API Elastic Beat from the 1Password GitHub repository.
-
Download an example eventsapibeat.yml file .
-
Configure the YAML file for the Beat to include:
-
The bearer token you saved previously in the auth_token fields for each 1Password event type you plan to monitor.
-
The output for events (sent directly to Elasticsearch, or through Logstash).
-
Any other configurations you want to customize.
-
Run the following command: ./eventsapibeat -c eventsapibeat.yml -e
You can now use Elasticsearch with the 1Password Events API Beat to monitor events from your 1Password account. The returned data will follow the Elastic Common Schema (ECS) specifications.
Collect events from 1Password Events API
-
URL of 1Password Events API Server - options: https://events.1password.com, https://events.1password.ca, https://events.1password.eu, https://events.ent.1password.com. path is automatic
-
1Password Authorization Token - Bearer Token, e.g. "eyJhbGciO..."
Atlassian Bitbucket Integrations
Introduction
The Bitbucket integration collects audit logs from the audit log files or the audit API.
Reference: https://developer.atlassian.com/server/bitbucket/reference/rest-api/
Assumptions
The procedures described in Section 3 assume that a Log Collector has already been set up.
Requirements
For more information on auditing in Bitbucket and how it can be configured, see View and configure the audit log on Atlassian's website.
Reference: https://confluence.atlassian.com/bitbucketserver/view-and-configure-the-audit-log-776640417.html
Logs
Audit
The Confluence integration collects audit logs from the audit log files or the audit API from self-hosted Confluence Data Center. It has been tested with Confluence 7.14.2 but is expected to work with newer versions. As of version 1.2.0, this integration added experimental support for Atlassian Confluence Cloud. JIRA Cloud only supports Basic Auth using username and a Personal Access Token.
Atlassian Bitbucket Integration Procedures
Please provide the following information to CyTech:
Collect Bitbucket audit logs via log files
-
Path
-
Preserve Original Event? (Enable Yes/No)
-
Preserves a raw copy of the original event, added to the field event.original
-
Tags
-
Processors (Optional)
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed.
-
Indexing settings (experimental) (Enable Yes/No)
-
Select data streams to configure indexing options. This is an experimental feature and may have effects on other properties.
Collect Bitbucket audit logs via API (Enable Yes/No)
-
API URL - The API URL without the path.
-
Bitbucket Username - JIRA Username. Needs to be used with a Password. Do not fill if you are using a personal access token.
-
Bitbucket Password - JIRA Password. Needs to be used with a Username. Do not fill if you are using a personal access token.
-
Personal Access Token - The Personal Access Token. If set, Username and Password will be ignored.
-
Initial Interval - Initial interval for the first API call. Defaults to 24 hours.
AWS Cloudtrails Integrations
Introduction
The AWS CloudTrail integration allows you to monitor AWS CloudTrail
Reference: https://aws.amazon.com/cloudtrail/
Use the AWS CloudTrail integration to collect and parse logs related to account activity across your AWS infrastructure. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference logs when troubleshooting an issue.
For example, you could use the data from this integration to spot unusual activity in your AWS accounts—like excessive failed AWS console sign in attempts.
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Data streams
The AWS CloudTrail integration collects one type of data: logs.
Logs help you keep a record of every event that CloudTrail receives. These logs are useful for many scenarios, including security and access audits. See more details in the Logs reference.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
Before using any AWS integration you will need:
-
AWS Credentials to connect with your AWS account.
-
AWS Permissions to make sure the user you're using to connect has permission to share the relevant data.
For more details about these requirements, see the AWS integration documentation.
Setup
Use this integration if you only need to collect data from the AWS CloudTrail service.
If you want to collect data from two or more AWS services, consider using the AWS integration. When you configure the AWS integration, you can collect data from as many AWS services as you'd like.
For step-by-step instructions on how to set up an integration, see the Getting started guide.
Logs reference
The cloudtrail data stream collects AWS CloudTrail logs. CloudTrail monitors events like user activity and API usage in AWS services. If a user creates a trail, it delivers those events as log files to a specific Amazon S3 bucket.
AWS CloudTrail integration Procedures
The following will need to be provided in the Configure integration when adding the AWS Audit CloudTrail integration.
Procedures:
Please provide the following information to CyTech:
Configure Integration
-
Collect CloudTrail logs from S3 (Enable)
-
Queue URL is required (URL of the AWS SQS queue that messages will be received from.)
-
Collect CloudTrail logs from CloudWatch(Optional)
-
Log Group ARN (ARN of the log group to collect logs from.)
-
Collect CloudTrail logs from third-party REST API (Optional)
-
URL of Splunk Enterprise Server (i.e. scheme://host:port, path is automatic)
-
Splunk REST API Username
-
Splunk REST API Password
-
Splunk Authorization Token (Bearer Token or Session Key, e.g. "Bearer eyJFd3e46..." or "Splunk 192fd3e...". Cannot be used with username and password.)
AWS GuardDuty Integrations
Introduction
The Amazon GuardDuty integration collects and parses data from Amazon GuardDuty Findings REST APIs.
The Amazon GuardDuty integration can be used in three different modes to collect data:
-
HTTP REST API - Amazon GuardDuty pushes logs directly to an HTTP REST API.
-
AWS S3 polling - Amazon GuardDuty writes data to S3 and Elastic Agent polls the S3 bucket by listing its contents and reading new files.
-
AWS S3 SQS - Amazon GuardDuty writes data to S3, S3 pushes a new object notification to SQS, Elastic Agent receives the notification from SQS, and then reads the S3 object. Multiple Agents can be used in this mode.
Assumptions
The procedures described in Section 3 assume that a Log Collector has already been setup.
Requirements
You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
Note: It is recommended to use AWS SQS for Amazon GuardDuty.
Aws GuardDuty integration Procedures
To collect data from AWS S3 Bucket, follow the steps below:
-
Configure the Data Forwarder to ingest data into an AWS S3 bucket. However, the user can set the parameter "Bucket List Prefix" according to the requirement.
To collect data from AWS SQS, follow the steps below:
-
If data forwarding to an AWS S3 bucket hasn't been configured, then first setup an AWS S3 bucket as mentioned in the documentation above.
-
To setup an SQS queue, follow "Step 1: Create an Amazon SQS queue" mentioned in the Documentation.
-
While creating an SQS queue, please provide the same bucket ARN that has been generated after creating the AWS S3 bucket.
-
Setup event notification for an S3 bucket. Follow this guide.
-
The user has to perform Step 3 for the guardduty data-stream, and the prefix parameter should be set the same as the S3 Bucket List Prefix as created earlier. For example, logs/ for guardduty data stream.
-
For all the event notifications that have been created, select the event type as s3:ObjectCreated:*, select the destination type SQS Queue, and select the queue that has been created in Step 2.
Note:
-
Credentials for the above AWS S3 and SQS input types should be configured according to the input configuration guide.
-
Data collection via AWS S3 Bucket and AWS SQS are mutually exclusive in this case.
To collect data from Amazon GuardDuty API, users must have an Access Key and a Secret Key. To create an API token follow the steps below:
-
Login to https://console.aws.amazon.com/.
-
Go to https://console.aws.amazon.com/iam/ to access the IAM console.
-
On the navigation menu, choose Users.
-
Choose your IAM user name.
-
Select Create access key from the Security Credentials tab.
-
To see the new access key, choose Show.
Note
-
The Secret Access Key and Access Key ID are required for the current integration package.
AWS Security Hub Integrations
Introduction
The AWS Security Hub integration collects and parses data from AWS Security Hub REST APIs.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Compatibility
-
This module is tested against AWS Security Hub API version 1.0.
Requirements
To collect data from AWS Security Hub APIs, users must have an Access Key and a Secret Key. To create API token follow below steps:
-
Login to https://console.aws.amazon.com/.
-
Go to https://console.aws.amazon.com/iam/ to access the IAM console.
-
On the navigation menu, choose Users.
-
Choose your IAM user name.
-
Select Create access key from the Security Credentials tab.
-
To see the new access key, choose Show.
Note:
-
For the current integration package, it is recommended to have interval in hours.
-
For the current integration package, it is compulsory to add Secret Access Key and Access Key ID.
Logs:
-
Findings - This is the securityhub_findings data stream.
-
Insights - This is the securityhub_insights data stream.
AWS Security Hub Integration Procedures
Please provide the following information to CyTech:
Collect AWS Security Hub logs via API
-
AWS Region - AWS Region.
Collect AWS Security Hub Insights from AWS
-
AWS Region - AWS Region.
AWS Integrations
Introduction
This document shows information related to AWS Integration.
The AWS integration is used to fetch logs and metrics from Amazon Web Services.
The usage of the AWS integration is to collect metrics and logs across many AWS services managed by your AWS account.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
Before using the AWS integration you will need:
-
AWS Credentials to connect with your AWS account.
-
AWS Permissions to make sure the user you're using to connect has permission to share the relevant data.
AWS Credentials
Use access keys directly (Option 1 Recommended)
Access keys are long-term credentials for an IAM user or the AWS account root user. To use access keys as credentials, you need to provide:
-
access_key_id: The first part of the access key.
-
secret_access_key: The second part of the access key.
Use an IAM role Amazon Resource Name (ARN)
To use an IAM role ARN, you need to provide either a credential profile or access keys along with the role_arn advanced option. role_arn is used to specify which AWS IAM role to assume for generating temporary credentials.
Instead of providing the access_key_id and secret_access_key directly to the integration, you will provide two advanced options to look up the access keys in the shared credentials file:
-
credential_profile_name: The profile name in shared credentials file.
Access keys are long-term credentials for an IAM user or the AWS account root user. To use access keys as credentials, you need to provide:
-
access_key_id: The first part of the access key.
-
secret_access_key: The second part of the access key.
AWS Permissions
Specific AWS permissions are required for the IAM user to make specific AWS API calls. To enable the AWS integration to collect metrics and logs from all supported services, make sure to give necessary permissions which CyTech to monitor.
Reference permissions:
-
ec2:DescribeInstances
-
ec2:DescribeRegions
-
cloudwatch:GetMetricData
-
cloudwatch:ListMetrics
-
iam:ListAccountAliases
-
rds:DescribeDBInstances
-
rds:ListTagsForResource
-
s3:GetObject
-
sns:ListTopics
-
sqs:ChangeMessageVisibility
-
sqs:DeleteMessage
-
sqs:ListQueues
-
sqs:ReceiveMessage
-
sts:AssumeRole
-
sts:GetCallerIdentity
-
tag:GetResources
AWS Integrations Procedures
Access Key ID and Secret Access Key:
These are associated with AWS Identity and Access Management (IAM) users and are used for programmatic access to AWS services. To find them:
-
Access the AWS Management Console.
-
Go to the "IAM" (Identity and Access Management) service.
-
Select the IAM user for which you want to retrieve the access keys.
-
Under the "Security credentials" tab, you can find the Access Key ID and you can create a new Secret Access Key if needed.
S3 Bucket ARN:
You can find the Amazon Resource Name (ARN) for an S3 bucket in the S3 Management Console or by using the AWS CLI. It typically looks like this:
-
Access the AWS Management Console.
-
Go to S3 Bucket Service
-
Navigate properties.
-
Find the ARN under Bucket overview
Sample: arn:aws:s3:::your-bucket-name
Log Group ARN:
Log Groups are associated with AWS CloudWatch Logs. You can find the ARN for a log group as follows:
-
Access the AWS Management Console.
-
Go to the "CloudWatch" service.
-
In the CloudWatch Logs section, select the log group you're interested in.
-
Select the log group that you want to open.
-
In the details of the log group, you will find the ARN.
SQS Queue URL: (Ignore if you're not using SQS Queue URL)
To find the URL of an Amazon Simple Queue Service (SQS) queue:
-
Access the AWS Management Console.
-
Go to the "SQS" service.
-
Select the specific queue you're interested in.
-
In the queue details, you can find the Queue URL
Please provide the following information to CyTech:
-
Access Key ID
-
Secret Access Key
- S3 Bucket ARN
- Log Group ARN
- SQS Queue URL : (Ignore if you're not using SQS Queue URL)
CISCO Meraki Integrations
Introduction
Cisco Meraki offers a centralized cloud management platform for all Meraki devices such as MX Security Appliances, MR Access Points and so on. Its out-of-band cloud architecture creates secure, scalable, and easy-to-deploy networks that can be managed from anywhere. This can be done from almost any device using web-based Meraki Dashboard and Meraki Mobile App. Each Meraki network generates its own events.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Compatibility
A syslog server can be configured to store messages for reporting purposes from MX Security Appliances, MR Access Points, and MS switches. This package collects events from the configured syslog server. The integration supports collection of events from "MX Security Appliances" and "MR Access Points". The "MS Switch" events are not recognized.
Requirements
Cisco Meraki Dashboard Configuration
SYSLOG
Cisco Meraki dashboard can be used to configure one or more syslog servers and Meraki message types to be sent to the syslog servers. Refer to Syslog Server Overview and Configuration page for more information on how to configure syslog server on Cisco Meraki.
API ENDPOINT (WEBHOOKS)
Cisco Meraki dashboard can be used to configure Meraki webhooks. Refer to the Webhooks Dashboard Setup section.
Configure the Cisco Meraki integration
SYSLOG
Depending on the syslog server setup in your environment check one/more of the following options "Collect syslog from Cisco Meraki via UDP", "Collect syslog from Cisco Meraki via TCP", "Collect syslog from Cisco Meraki via file".
Enter the values for syslog host and port OR file path based on the chosen configuration options.
API Endpoint (Webhooks)
Check the option "Collect events from Cisco Meraki via Webhooks" option.
-
Enter values for "Listen Address", "Listen Port" and "Webhook path" to form the endpoint URL. Make note of the Endpoint URL https://{AGENT_ADDRESS}:8686/meraki/events.
-
Enter value for "Secret value". This must match the "Shared Secret" value entered when configuring the webhook from Meraki cloud.
-
Enter values for "TLS". Cisco Meraki requires that the webhook accept requests over HTTPS. So you must either configure the integration with a valid TLS certificate or use a reverse proxy in front of the integration.
Log Events
Enable to collect Cisco Meraki log events for all the applications configured for the chosen log stream.
Logs
Syslog
The cisco_meraki.log dataset provides events from the configured syslog server. All Cisco Meraki syslog specific fields are available in the cisco_meraki.log field group.
API Endpoint (Webhooks)
Cisco Meraki Integration Procedures
Please provide the following information to CyTech:
- Collect syslog from Cisco Meraki via UDP
- Listen Address - The bind address to listen for UDP connections. Set to 0.0.0.0 to bind to all available interfaces.
- Listen Port - The UDP port number to listen on.
-
Collect syslog from Cisco Meraki via TCP
- Listen Address - The bind address to listen for TCP connections. Set to 0.0.0.0 to bind to all available interfaces.
- Listen Port - The UDP port number to listen on.
-
Collect syslog from Cisco Meraki via file
- Paths
- Collect syslog from Cisco Meraki via Webhooks
- Listen Address - Bind address for the listener. Use 0.0.0.0 to listen on all interfaces.
- Listen Port
CISCO Secure Endpoint Integrations
Introduction
Secure Endpoint offers cloud-delivered, advanced endpoint detection and response across multidomain control points to rapidly detect, contain, and remediate advanced threats.
Assumptions
The procedures described in Section 3 assume that a Log Collector has already been setup.
Requirements
This integration is for Cisco Secure Endpoint logs. It includes the following datasets for receiving logs over syslog or read from a file:
-
event dataset: supports Cisco Secure Endpoint Event logs.
Logs
Secure Endpoint
The event dataset collects Cisco Secure Endpoint logs.
What can the Secure Endpoint API be used for?
-
Generate a list of organizations a user has access to
-
Generate a list of policies for a specified organization
-
Generate specific information about a specified policy such as:
-
-
-
-
General policy data
-
Associated network control lists
-
Associated computers
-
Associated groups
-
Proxy settings
-
Policy XML
-
-
-
-
Generate all policy types and operating systems available for a specified organization
Top Use Cases
-
Generating reports on policy settings across an organization
-
Inspecting a particular policy's settings
-
Querying to find policies matching certain criteria in order to detect which policies should be edited
Response Format
-
Data
-
Meta
-
Errors
Cisco Secure Endpoint Integration Procedures
Please provide the following information to CyTech:
Collect logs from the Cisco Secure Endpoint API.
-
Client ID - Cisco Secure Endpoint Client ID
-
API Key - Cisco Secure Endpoint API Key
CISCO Umbrella Integrations
Introduction
Cisco Umbrella is a cloud security platform that provides an additional line of defense against malicious software and threats on the internet by using threat intelligence. That intelligence helps prevent adware, malware, botnets, phishing attacks, and other known bad Websites from being accessed.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Prerequisites
-
-
You must have Full Admin access to Umbrella to create and manage Umbrella API keys or Umbrella KeyAdmin API keys.
-
Requirements
This integration is for Cisco Umbrella. It includes the following datasets for receiving logs from an AWS S3 bucket using an SQS notification queue and Cisco Managed S3 bucket without SQS:
-
log dataset: supports Cisco Umbrella logs.
Logs
Umbrella
When using Cisco Managed S3 buckets that does not use SQS there is no load balancing possibilities for multiple agents, a single agent should be configured to poll the S3 bucket for new and updated files, and the number of workers can be configured to scale vertically.
The log dataset collects Cisco Umbrella logs.
Advantages of Integrating with the Umbrella API
The Umbrella API features a number of improvements over the Umbrella v1 APIs and the Umbrella Reporting v2 API.
-
Intuitive base URI
-
API paths defined by top-level scopes
-
Intent-based, granular API key scopes
-
API key expiration
-
Updated API key administration dashboard views
-
Programmatic API key administration
-
API authentication and authorization supported by OAuth 2.0 client credentials flow
-
Portable, programmable API interface for client integrations
Before you send a request to the Umbrella API, you must create new Umbrella API credentials and generate an API access token. For more information, see Umbrella API Authentication.
Authentication
The Umbrella API provides a standard REST interface and supports the OAuth 2.0 client credentials flow. To get started, log in to Umbrella and create an Umbrella API key. Then, use your API credentials to generate an API access token.
Note: API keys, passwords, secrets, and tokens allow access to your private customer data. You should never share your credentials with another user or organization.
Log in to Umbrella
-
Log in to Umbrella with the following URL: https://dashboard.umbrella.com
-
You can find your username after Admin in the navigation tree. Confirm that your organization appears under your username.
Cisco Secure Endpoint Integration Procedures
Please provide the following information to CyTech:
Collect logs from the Cisco Umbrella
-
Queue URL - URL of the AWS SQS queue that messages will be received from. For Cisco Managed S3 buckets or S3 without SQS, use Bucket ARN.
-
Bucket ARN - Required for Cisco Managed S3. If the S3 bucket does not use SQS, this is the address for the S3 bucket, one example is arn:aws:s3:::cisco-managed-eu-central-1 For a list of Cisco Managed buckets, please see https://docs.umbrella.com/mssp-deployment/docs/enable-logging-to-a-cisco-managed-s3-bucket.
-
Bucket Region - Required for Cisco Managed S3. The region the bucket is located in.
-
Bucket List Prefix - Required for Cisco Managed S3. This sets the root folder of the S3 bucket that should be monitored, found in the S3 Web UI. Example value: 1235_654vcasd23431e5dd6f7fsad457sdf1fd5.
-
Number of Workers - Required for Cisco Managed S3. Number of workers that will process the S3 objects listed. Minimum is 1.
-
Bucket List Interval - Time interval for polling listing of the S3 bucket. Defaults to 120s.
-
Access Key ID
-
Secret Access Key
-
Preserve original event
Cloudflare Integration
Introduction
Cloudflare integration uses Cloudflare's API to retrieve audit logs and traffic logs from Cloudflare, for a particular zone, and ingest them into Elasticsearch. This allows you to search, observe and visualize the Cloudflare log events through Elasticsearch.
Users of Cloudflare use Cloudflare services to increase the security and performance of their web sites and services.
To enable the Cloudflare Logpush, please refer to Section 5. Currently, the procedures described is for the setup of Amazon S3.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
Configure Cloudflare audit logs data stream
Enter values "Auth Email", "Auth Key" and "Account ID".
-
-
-
Auth Email is the email address associated with your account.
-
Auth Key is the API key generated on the "My Account" page.
-
Account ID can be found on the Cloudflare dashboard. Follow the navigation documentation from here.
-
-
Configure Cloudflare logs
These logs contain data related to the connecting client, the request path through the Cloudflare network, and the response from the origin web server. For more information see here.
The integration can retrieve Cloudflare logs using -
-
-
-
Auth Email and Auth Key
-
API Token More information is available here.
-
-
CONFIGURE USING AUTH EMAIL AND AUTH KEY
Enter values "Auth Email", "Auth Key" and "Zone ID".
-
-
- Auth Email is the email address associated with your account.
- Auth Key is the API key generated on the "My Account" page.
- Zone ID can be found here.
-
CONFIGURE USING API TOKEN
Enter values "API Token" and "Zone ID".
For the Cloudflare integration to be able to successfully get logs the following permissions must be granted to the API token -
-
Account.Access: Audit Logs: Read
-
-
- API Tokens allow for more granular permission settings.
- Zone ID can be found here.
-
Logs
Audit
Audit logs summarize the history of changes made within your Cloudflare account. Audit logs include account-level actions like login and logout, as well as setting changes to DNS, Crypto, Firewall, Speed, Caching, Page Rules, Network, and Traffic features, etc.
Logpull
These logs contain data related to the connecting client, the request path through the Cloudflare network, and the response from the origin web server.
Cloudflare Integration Procedures
Please provide the following information to CyTech:
See the Screenshot Below
Audit Logs
-
Auth Email -
-
Auth Key
-
Account ID
Cloudflare Logs
-
Auth Token
-
Zone ID
Enable Logpush to Amazon S3
To enable the Cloudflare Logpush service:
-
Log in to the Cloudflare dashboard.
-
Select the Enterprise account or domain you want to use with Logpush.
-
Go to Analytics & Logs > Logs.
-
Select Add Logpush job. A modal window opens where you will need to complete several steps.
-
Select the dataset you want to push to a storage service.
-
Select the data fields to include in your logs. Add or remove fields later by modifying your settings in Logs > Logpush.
-
Select Amazon S3.
-
Enter or select the following destination information:
-
-
-
Bucket path
-
Daily subfolders
-
Bucket region
-
Encryption constraint in bucket policy
-
For Grant Cloudflare access to upload files to your bucket, make sure your bucket has a policy (if you did not add it already):
-
Copy the JSON policy, then go to your bucket in the Amazon S3 console and paste the policy in Permissions > Bucket Policy and click Save.
-
-
-
Click Validate access.
-
Enter the Ownership token (included in a file or log Cloudflare sends to your provider) and click Prove ownership. To find the ownership token, click the Open button in the Overview tab of the ownership challenge file.
-
Click Save and Start Pushing to finish enabling Logpush.
Once connected, Cloudflare lists Amazon S3 as a connected service under Logs > Logpush. Edit or remove connected services from here.
Crowdstrike Integrations
Introduction
This integration is for CrowdStrike products. It includes the following datasets for receiving logs:
falcon dataset consists of endpoint data and Falcon platform audit data forwarded from Falcon SIEM Connector.
fdr dataset consists of logs forwarded using the Falcon Data Replicator.
Assumptions
The procedures described in Section 3 assume that a Log Collector has already been setup.
Compatibility
This integration supports CrowdStrike Falcon SIEM-Connector-v2.0.
Requirements
Logs
Falcon
Contains endpoint data and CrowdStrike Falcon platform audit data forwarded from Falcon SIEM Connector.
FDR
The CrowdStrike Falcon Data Replicator (FDR) allows CrowdStrike users to replicate FDR data from CrowdStrike managed S3 buckets. CrowdStrike writes notification events to a CrowdStrike managed SQS queue when new data is available in S3.
This integration can be used in two ways. It can consume SQS notifications directly from the CrowdStrike managed SQS queue or it can be used in conjunction with the FDR tool that replicates the data to a self-managed S3 bucket and the integration can read from there.
In both cases SQS messages are deleted after they are processed. This allows you to operate more than one Elastic Agent with this integration if needed and not have duplicate events, but it means you cannot ingest the data a second time.
CrowdStrike Integration Procedures
Please provide the following information to CyTech:
Collect CrowdStrike Falcon Data Replicator logs (input: aws-s3) Option 1
-
AWS: Access Key ID
-
AWS: Secret Access Key
-
AWS: Queue URL - URL of the AWS SQS queue that messages will be received from.
Collect CrowdStrike logs via API. Option 2 (Recommended)
-
Client ID: Client ID for the CrowdStrike.
-
Client Secret: Client Secret for the CrowdStrike.
-
URL: Token URL of CrowdStrike.
Dropbox Integrations
Introduction
Connecting Dropbox
Use the Workplace Search Dropbox connector to automatically capture, sync and index the following items from your Dropbox service:
Stored Files
Including ID, File Metadata, File Content, Updated by, and timestamps.
Dropbox Paper
Including ID, Metadata, Content, Updated by, and timestamps.
This document helps you configure and connect your Dropbox service to Workplace Search. To do this, you must register your Elastic deployment in the Dropbox Developer platform, by creating an OAuth 2.0 app. This gives your app permission to access Dropbox data. You will need your Workplace Search OAuth redirect URL, so have that handy.
If you need a primer on OAuth, read the official OAuth 2.0 authorization flow RFC. The diagrams provide a good mental model of the protocol flow. Dropbox also has its own OAuth guide.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
Configuring the Dropbox Connector
-
To register your Elastic deployment with Dropbox, create a new OAuth app in your organization’s Dropbox developer platform. Make sure to use a trusted and stable Dropbox account.
-
Provide basic information about the app and define the access scopes, or specific permissions, the app needs to interact with Dropbox. A good rule of thumb is that these should be read-only permissions. Choose only the following permissions:
-
-
-
files.content.read
-
sharing.read
-
account_info.read
-
files.metadata.read
-
-
To fetch document-level permissions, you must create an OAuth App using a team-owned (Dropbox Business) account. Enable document-level permissions by adding the following permissions:
-
-
-
team_info.read
-
team_data.member
-
team_data.team_space
-
members.read
-
-
-
Register a redirect URL for the app to use. This is where the OAuth 2.0 service will return the user after they authorize the application. This is the Workplace Search OAuth redirect URL for your deployment. This must be a https endpoint for production use cases. Only use http for local development.
-
Find and record the client_id and client_secret for the app. Dropbox calls these App Key and App Secret.
-
Switch back to your organization’s Workplace Search administrative dashboard
-
In the Sources —> Add Source tab, add a new Dropbox source
-
Configure the service with Workplace Search using the App Key and App Secret.
Your Dropbox service is now configured, and you can connect it to Workplace Search.
Dropbox Integration Procedures
Connecting Dropbox to Workplace Search
Once the Dropbox connector is configured, you can connect a Dropbox instance to your organization’s Workplace Search deployment.
-
Follow the Dropbox authentication flow as presented in the Workplace Search administrative dashboard.
-
If the authentication flow succeeds, you will be redirected to Workplace Search.
Your Dropbox content becomes searchable as soon as syncing starts. Once configured and connected, Dropbox synchronizes automatically every 2 hours.
Limiting the content to be indexed
If you don’t need to index all available content, you can use the API to apply indexing rules. This reduces indexing load and overall index size. See Customizing indexing.
The path_template and file_extension rules are applicable for Dropbox.
Synchronized fields
The following table lists the fields synchronized from the connected source to Workplace Search. The attributes in the table apply to the default search application, as follows:
-
Display name - The label used when displayed in the UI
-
Field name - The name of the underlying field attribute
-
Faceted filter - whether the field is a faceted filter by default, or can be enabled (see also: Customizing filters)
-
Automatic query refinement preceding phrases - The default list of phrases that must precede a value of this field in a search query in order to automatically trigger query refinement. If "None," a value from this field may trigger refinement regardless of where it is found in the query string. If '', a value from this field must be the first token(s) in the query string. If N.A., automatic query refinement is not available for this field by default. All fields that have a faceted filter (default or configurable) can also be configured for automatic query refinement; see also Update a content source, Get a content source’s automatic query refinement details and Customizing filters.
F5 Integrations
Introduction
This document shows information related to F5 Integration.
The F5 BIG-IP integration allows users to monitor LTM, AFM, APM, ASM, and AVR activity. F5 BIG-IP covers software and hardware designed around application availability, access control, and security solutions.
The F5 BIG-IP integration can be used in three different modes to collect data:
HTTP Endpoint mode - F5 BIG-IP pushes logs directly to an HTTP endpoint hosted by users’ Elastic Agent.
AWS S3 polling mode - F5 BIG-IP writes data to S3 and Elastic Agent polls the S3 bucket by listing its contents and reading new files.
AWS S3 SQS mode - F5 BIG-IP writes data to S3, S3 pushes a new object notification to SQS, Elastic Agent receives the notification from SQS, and then reads the S3 object. Multiple Agents can be used in this mode.
For example, users can use the data from this integration to analyze the traffic that passes through their F5 BIG-IP network.
Data streams
The F5 BIG-IP integration collects one type of data stream: log.
Log help users to keep a record of events happening on the network using telemetry streaming. The log data stream collected by the F5 BIG-IP integration includes events that are related to network traffic. See more details in the Logs.
This integration targets the five types of events as mentioned below:
LTM provides the platform for creating virtual servers, performance, service, protocol, authentication, and security profiles to define and shape users’ application traffic. For more information, refer to the link here.
AFM is designed to reduce the hardware and extra hops required when ADC's are paired with traditional firewalls and helps to protect traffic destined for the user's data center. For more information, refer to the link here.
APM provides federation, SSO, application access policies, and secure web tunneling and allows granular access to users' various applications, virtualized desktop environments, or just go full VPN tunnel. For more information, refer to the link here.
ASM is F5's web application firewall (WAF) solution. It allows users to tailor acceptable and expected application behavior on a per-application basis. For more information, refer to the link here.
AVR provides detailed charts and graphs to give users more insight into the performance of web applications, with detailed views on HTTP and TCP stats, as well as system performance (CPU, memory, etc.). For more information, refer to the link here.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
Elasticsearch is needed to store and search data, and Kibana is needed for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your hardware.
The reference link for requirements of telemetry streaming is here.
-
https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/prereqs.html
The reference link for requirements of Application Services 3(AS3) Extension is here.
-
https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/userguide/prereqs.html
This module has been tested against F5 BIG-IP version 16.1.0, Telemetry Streaming version 1.32.0 and AS3 version 3.40.0.
Setup
To collect LTM, AFM, APM, ASM, and AVR data from F5 BIG-IP, the user has to configure modules in F5 BIG-IP as per the requirements.
To set up the F5 BIG-IP environment, users can use the BIG-IP system browser-based Configuration Utility or the command line tools that are provided. For more information related to the configuration of F5 BIG-IP servers, refer to F5 support website here.
https://support.f5.com/csp/knowledge-center/software
Configuration of Telemetry Streaming in F5
For downloading and installing Telemetry Streaming, refer to the link here.
https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/installation.html
Telemetry Streaming will send logs in the JSON format to the destination. Telemetry Streaming is compatible with BIG-IP versions 13.0 and later. Users have to prepare F5 servers for it and set up the Telemetry Streaming Consumer.
To use telemetry streaming, user have to send POST request on https://<BIG-IP>/mgmt/shared/telemetry/declare for declaration.
F5 BIG-IP modules named LTM, AFM, ASM, and APM are not configured by Telemetry Streaming, they must be configured with AS3 or another method. Reference link for setup AS3 extension in F5 BIG-IP is here.
To configure logging using AS3, refer to the link here.
To collect data from AWS S3 Bucket, follow the below steps:
- Create an Amazon S3 bucket. Refer to the link here.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html
- The default value of the "Bucket List Prefix" is listed below. However, the user can set the parameter "Bucket List Prefix" according to the requirement.
To collect data from AWS SQS, follow the below steps:
- If data forwarding to an AWS S3 Bucket hasn't been configured, then first set up an AWS S3 Bucket as mentioned in the above documentation.
- To set up an SQS queue, follow "Step 1: Create an Amazon SQS queue" mentioned in the Documentation. https://docs.aws.amazon.com/AmazonS3/latest/userguide/ways-to-add-notification-config-to-bucket.html
-
-
While creating an SQS Queue, please provide the same bucket ARN that has been generated after creating an AWS S3 Bucket.
-
- Set up event notifications for an S3 bucket. Follow this link. https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications.html
-
- Users have to set the prefix parameter the same as the S3 Bucket List Prefix as created earlier. (for example, log/ for a log data stream.)
- Select the event type as s3:ObjectCreated:*, select the destination type SQS Queue, and select the queue that has been created in Step 2.
Note:
- Credentials for the above AWS S3 and SQS input types should be configured using the link. https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-s3.html#aws-credentials-config
- Data collection via AWS S3 Bucket and AWS SQS are mutually exclusive in this case.
Enabling the integration in Elastic
-
In Kibana go to Management > Integrations.
-
In the "Search for integrations" search bar, type F5 BIG-IP.
-
Click on F5 BIG-IP integration from the search results.
-
Click on the Add F5 BIG-IP button to add F5 BIG-IP integration.
-
Enable the Integration to collect logs via AWS S3 or HTTP endpoint input.
F5 BIG-IP integration
The "Integration name" and either the " Description" and the following will need to be provided in the Configure integration when adding the F5 BIG-IP integration.
Procedures:
Please provide the following information to CyTech:
Collect F5 BIG-IP logs via HTTP Endpoint:
-
Listen Address - The bind address to listen for http endpoint connections. Set to 0.0.0.0 to bind to all available interfaces.
F5 BIG-IP logs via HTTP Endpoint:
-
Listen Port - The port number the listener binds to.
Fortinet-Fortigate Integrations
Introduction
This integration is for Fortinet FortiGate logs sent in the syslog format.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Compatibility
This integration has been tested against FortiOS version 6.0.x and 6.2.x. Versions above this are expected to work but have not been tested.
Fortinet FortiGate Integration Procedures
Please provide the following information to CyTech:
Collect Fortinet FortiGate logs (input: tcp)
-
Listen Address - The bind address to listen for TCP connections.
-
Listen Port - The TCP port number to listen on.
Collect Fortinet FortiGate logs (input: udp)
-
Listen Address - The bind address to listen for UDP connections.
-
Listen Port - The UDP port number to listen on.
GCP Integrations
Introduction
This document shows information related to GCP Integration.
The Google Cloud integration collects and parses Google Cloud Audit Logs, VPC Flow Logs, Firewall Rules Logs and Cloud DNS Logs that have been exported from Cloud Logging to a Google Pub/Sub topic sink and collects Google Cloud metrics and metadata from Google Cloud Monitoring.
Requirements
To use this Google Cloud Platform (GCP) integration, the following needs to be set up:
-
Service Account with a Role. (Section 3.1.1)
-
Service Account Key to access data on your GCP project. (Section 3.1.2)
Service Accounts
First, please create a Service Account. A Service Account (SA) is a particular type of Google account intended to represent a non-human user who needs to access the GCP resources.
The Log Collector uses the SA (Service Account) to access data on Google Cloud Platform using the Google APIs.
Here is a reference from Google related to Service Accounts:
https://cloud.google.com/iam/docs/best-practices-for-securing-service-accounts
Service Account with a Role
You need to grant your Service Account (SA) access to Google Cloud Platform resources by assigning a role to the account. In order to assign minimal privileges, create a custom role that has only the privileges required by the Log Collector. Those privileges are below:
-
compute.instances.list (required for GCP Compute instance metadata collection) (**2)
-
monitoring.metricDescriptors.list
-
monitoring.timeSeries.list
-
pubsub.subscriptions.consume
-
pubsub.subscriptions.create (*1)
-
pubsub.subscriptions.get
-
pubsub.topics.attachSubscription (*1)
*1 Only required if Agent is expected to create a new subscription. If you create the subscriptions yourself, you may omit these privileges.
**2 Only required if corresponding collection will be enabled.
After you have created the custom role, assign the role to your service account.
Service Account Key
Next, with the Service Account (SA) with access to Google Cloud Platform (GCP) resources setup in Section 3.1.1, you need some credentials to associate with it: a Service Account Key.
From the list of SA (Service Accounts):
-
Click the one you just created to open the detailed view.
-
From the Keys section, click "Add key" > "Create new key" and select JSON as the type.
-
Download and store the generated private key securely (remember, the private key can't be recovered from GCP if lost).
GCP Integrations Procedures
-
GCP Audit Logs
The audit dataset collects audit logs of administrative activities and accesses within your Google Cloud resources.
Procedures
The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration.
Please provide the following information to CyTech:
-
Project ID
The Project ID is the Google Cloud project ID where your resources exist.
-
Credentials File vs JSON
Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field.
OPTION 1: CREDENTIALS FILE
Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file.
Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json.
OPTION 2: CREDENTIALS JSON
Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration.
-
GCP DNS Logs
The dns dataset collects queries that name servers resolve for your Virtual Private Cloud (VPC) networks, as well as queries from an external entity directly to a public zone.
Procedures
The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration.
Please provide the following information to CyTech:
- Project ID
The Project ID is the Google Cloud project ID where your resources exist.
-
Credentials File vs JSON
Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field.
OPTION 1: CREDENTIALS FILE
Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file.
Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json.
OPTION 2: CREDENTIALS JSON
Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration.
-
GCP Firewall Logs
The firewall dataset collects logs from Firewall Rules in your Virtual Private Cloud (VPC) networks.
Procedures
The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration.
Please provide the following information to CyTech:
-
Project ID
The Project ID is the Google Cloud project ID where your resources exist.
-
Credentials File vs JSON
Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field.
OPTION 1: CREDENTIALS FILE
Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file.
Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json.
OPTION 2: CREDENTIALS JSON
Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration.
-
GCP VPC Flow Logs
The vpcflow dataset collects logs sent from and received by VM instances, including instances used as GKE nodes.
Procedures
The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration.
Please provide the following information to CyTech:
-
Project ID
The Project ID is the Google Cloud project ID where your resources exist.
-
Credentials File vs JSON
Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field.
OPTION 1: CREDENTIALS FILE
Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file.
Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json.
OPTION 2: CREDENTIALS JSON
Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration.
GitLab Integrations
Introduction
Introduced in GitLab Starter 8.4. Support for Amazon Elasticsearch was introduced in GitLab Starter 9.0.
This document describes how to set up Elasticsearch with GitLab. Once enabled, you'll have the benefit of fast search response times and the advantage of two special searches:
-
Advance Global Search
-
Advanced Syntax Search
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
Elasticsearch 6.0+ is not supported currently. We will support 6.0+ in the future.
GitLabIntegration Procedures
Procedures:
Please provide the following information to CyTech:
Elasticsearch is not included in the Omnibus packages. You will have to install it yourself whether you are using the Omnibus package or installed GitLab from source. Providing detailed information on installing Elasticsearch is out of the scope of this document.
Once the data is added to the database or repository and Elasticsearch is enabled in the admin area the search index will be updated automatically. Elasticsearch can be installed on the same machine as GitLab, or on a separate server, or you can use the Amazon Elasticsearch service.
You can follow the steps as described in the official web site or use the packages that are available for your OS.
Google Workspace Integrations
Introduction
Google Workspace (formerly G Suite) is a suite of cloud computing, productivity and collaboration tools, software and products developed and marketed by Google. It allows users to create, edit, and share documents, spreadsheets, presentations, and more. It also includes email, calendar, chat, and video conferencing tools.
Google Workspace is designed for businesses of all sizes, from small businesses to large enterprises. It is a popular choice for businesses because it is affordable, easy to use, and secure.
The Google Workspace integration collects and parses data from the different Google Workspace audit reports APIs.
If you want to know more about how you can fully leverage the Google Workspace integration, there is a multipart blog from our Security Labs that will help you:
-
To understand what Google Workspace is in Part One - Surveying the Land
-
To set it up, step by step, in Part Two - Setup Threat Detection with Elastic
-
And to use the collected information to your advantage in Part Three - Detecting Common Threats
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Compatibility
It is compatible with a subset of applications under the Google Reports API v1.
Requirements
The procedures described in Section 3 assumes that a Log Collector has already been setup.
In order to ingest data from the Google Reports API you must:
-
Have an administrator account.
-
Set up a ServiceAccount using the administrator account.
-
Set up access to the Admin SDK API for the ServiceAccount.
-
Enable Domain-Wide Delegation for your ServiceAccount.
Create access credentials
Credentials are used to obtain an access token from Google's authorization servers so your app can call Google Workspace APIs. This guide describes how to choose and set up the credentials your app needs.
Choose the access credential that is right for you
The required credentials depends on the type of data, platform, and access methodology of your app. There are three types of credential types available:
-
API key credentials - An API key is a long string containing upper and lower case letters, numbers, underscores, and hyphens, such as AIzaSyDaGmWKa4JsXZ-HjGw7ISLn_3namBGewQe. This authentication method is used to anonymously access publicly-available data, such as Google Workspace files shared using the "Anyone on the Internet with this link" sharing setting. For more details, see Using API keys. To create an API key:
-
-
In the Google Cloud console, go to Menu > APIs & Services > Credentials.
-
Click Create credentials > API key.
-
Your new API key is displayed.
-
-
-
-
Click Copy to copy your API key for use in your app's code. The API key can also be found in the "API keys" section of your project's credentials.
-
Click Restrict key to update advanced settings and limit use of your API key. For more details, see Applying API key restrictions.
-
-
-
OAuth client ID credentials - To authenticate as an end user and access user data in your app, you need to create one or more OAuth 2.0 Client IDs. A client ID is used to identify a single app to Google's OAuth servers. If your app runs on multiple platforms, you must create a separate client ID for each platform. Choose your application type for specific instructions about how to create an OAuth client ID:
-
-
-
-
-
Web application
-
Android
-
iOS
-
Chrome app
-
Desktop app
-
TVs & limited-input devices
-
Universal Windows Platform (UWP)
-
-
-
-
-
Service account credentials - A service account is a special kind of account used by an application, rather than a person. You can use a service account to access data or perform actions by the robot account, or to access data on behalf of Google Workspace or Cloud Identity users. For more information, see Understanding service accounts. Create a service account:
-
-
-
-
-
In the Google Cloud console, go to Menu > IAM & Admin > Service Accounts.
-
Click Create service account.
-
Fill in the service account details, then click Create and continue.
-
Note: By default, Google creates a unique service account ID. If you would like to change the ID, modify the ID in the service account ID field.
-
Optional: Assign roles to your service account to grant access to your Google Cloud project's resources. For more details, refer to Granting, changing, and revoking access to resources.
-
Click Continue.
-
Optional: Enter users or groups that can manage and perform actions with this service account. For more details, refer to Managing service account impersonation.
-
Click Done. Make a note of the email address for the service account.
-
-
-
-
Assign a role to a service account:
You must assign a prebuilt or custom role to a service account by a super administrator account.
-
In the Google Admin console, go to Menu > Account > Admin roles.
-
Point to the role that you want to assign, and then click Assign admin.
-
Click Assign service accounts.
-
Enter the email address of the service account.
-
Click Add > Assign role.
Create credentials for a service account:
You need to obtain credentials in the form of a public/private key pair. These credentials are used by your code to authorize service account actions within your app.
To obtain credentials for your service account:
-
In the Google Cloud console, go to Menu > IAM & Admin > Service Accounts.
-
Select your service account.
-
Click Keys > Add key > Create new key.
-
Select JSON, then click Create.
Your new public/private key pair is generated and downloaded to your machine as a new file. Save the downloaded JSON file as credentials.json in your working directory. This file is the only copy of this key. For information about how to store your key securely, see Managing service account keys.
-
Click Close.
Optional: Set up domain-wide delegation for a service account
To call APIs on behalf of users in a Google Workspace organization, your service account needs to be granted domain-wide delegation of authority in the Google Workspace Admin console by a super administrator account. For more information, see Delegating domain-wide authority to a service account.
To set up domain-wide delegation of authority for a service account:
-
In the Google Cloud console, go to Menu > IAM & Admin > Service Accounts.
-
Select your service account.
-
Click Show advanced settings.
-
Under "Domain-wide delegation," find your service account's "Client ID." Click Copy to copy the client ID value to your clipboard.
-
If you have super administrator access to the relevant Google Workspace account, click View Google Workspace Admin Console, then sign in using a super administrator user account and continue following these steps.
If you don't have super administrator access to the relevant Google Workspace account, contact a super administrator for that account and send them your service account's Client ID and list of OAuth Scopes so they can complete the following steps in the Admin console.
-
-
-
-
In the Google Admin console, go to Menu > Security > Access and data control > API controls.
-
-
-
-
-
Click Manage Domain Wide Delegation.
-
-
-
Click Add new.
-
-
-
In the "Client ID" field, paste the client ID that you previously copied.
-
-
-
In the "OAuth Scopes" field, enter a comma-delimited list of the scopes required by your application. This is the same set of scopes you defined when configuring the OAuth consent screen.
-
-
-
Click Authorize.
-
This integration will make use of the following oauth2 scope:
-
https://www.googleapis.com/auth/admin.reports.audit.readonly
Once you have downloaded your service account credentials as a JSON file, you are ready to set up your integration.
Click the Advanced option of Google Workspace Audit Reports. The default value of "API Host" is https://www.googleapis.com. The API Host will be used for collecting access_transparency, admin, device, context_aware_access, drive, gcp, groups, group_enterprise, login, rules, saml, token and user accounts logs.
NOTE: The Delegated Account value in the configuration, is expected to be the email of the administrator account, and not the email of the ServiceAccount.
Google Workspace Integration Procedures
Please provide the following information to CyTech:
Collect access_transparency, admin, alert, context_aware_access, device, drive, gcp, groups, group_enterprise, login, rules, saml, token and user accounts logs (input: httpjson).
-
Jwt File - Specifies the path to the JWT credentials file. NOTE: Please use either JWT File or JWT JSON parameter.
-
Jwt JSON - Raw contents of the JWT file. Useful when hosting a file along with the agent is not possible. NOTE: Please use either JWT File or JWT JSON parameter.
-
Delegated Account - Delegated Account is required. Email of the admin user used to access the API.
Jumpcloud Integrations
Introduction
The JumpCloud integration allows you to monitor events related to the JumpCloud Directory as a Service via the Directory Insights API.
You can find out more about JumpCloud and JumpCloud Directory Insights here
Data streams
A single data stream named "jumpcloud.events" is used by this integration.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
An Elastic Stack with an Elastic Agent is a fundamental requirement.
An established JumpCloud tenancy with active users is the the other requirement. Basic Directory Insights API access is available to all subscription levels.
NOTE: The lowest level of subscription currently has retention limits, with access to Directory Insights events for the last 15 days at most. Other subscriptions levels provide 90 days or longer historical event access.
A JumpCloud API key is required, the JumpCloud documentation describing how to create one is here
This JumpCloud Directory Insights API is documented here
JumpCloud Integration Procedures
Procedures:
Please provide the following information to CyTech:
Enabling the integration in Elastic
-
In Kibana go to Management > Integrations
-
In "Search for integrations" search bar type JumpCloud
-
Click on "JumpCloud" integration from the search results.
-
Click on Add JumpCloud button to add the JumpCloud integration.
-
Configure the integration as appropriate
-
Assign the integration to a new Elastic Agent host, or an existing Elastic Agent host
Mimecast Integrations
Introduction
The Mimecast integration collects events from the Mimecast API.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
Configuration
Note: Rate limit quotas may require you to set up different credentials for the different available log types.
Logs
Audit Events
This is the mimecast.audit_events dataset. These logs contain Mimecast audit events with the following details: audit type, event category and detailed information about the event. More information about these logs.
DLP Logs
This is the mimecast.dlp_logs dataset. These logs contain information about messages that triggered a DLP or Content Examination policy. More information about these logs.
SIEM Logs
This is the mimecast.siem_logs dataset. These logs contain information about messages that contains MTA (message transfer agent) log – all inbound, outbound, and internal messages.
Threat Intel Feed Malware: Customer
This is the mimecast.threat_intel_malware_customer dataset. These logs contain information about messages that return identified malware threats at a customer level. Learn more about these logs.
Threat Intel Feed Malware: Grid
This is the mimecast.threat_intel_malware_grid dataset. These logs contain information about messages that return identified malware threats at a regional grid level. More about these logs.
TTP Attachment Logs
This is the mimecast.ttp_ap_logs dataset. These logs contain Mimecast TTP attachment protection logs with the following details: result of attachment analysis (if it is malicious or not etc.), date when file is released, sender and recipient address, filename and type, action triggered for the attachment, the route of the original email containing the attachment and details. Learn more about these logs.
TTP Impersonation Logs
This is the mimecast.ttp_ip_logs dataset. These logs contain information about messages containing information flagged by an Impersonation Protection configuration.
TTP URL Logs
This is the mimecast.ttp_url_logs dataset. These logs contain Mimecast TTP attachment protection logs with the following details: the category of the URL clicked, the email address of the user who clicked the link, the url clicked, the action taken by the user if user awareness was applied, the route of the email that contained the link, the action defined by the administrator for the URL, the date that the URL was clicked, url scan result, the action that was taken for the click, the description of the definition that triggered the URL to be rewritten by Mimecast, the action requested by the user, an array of components of the message where the URL was found. More about these logs.
Mimecast Integration Procedures
Please provide the following information to CyTech:
Mimecast API
-
Application Key - Specifies application key for user.
-
Application ID - Set the Application Id.
-
Access Key - Set Access Key.
-
Secret Key - Set Secret Key.
MongoDB Integrations
Introduction
This integration is used to fetch logs and metrics from MongoDB.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Compatibility
The log dataset is tested with logs from versions v3.2.11 and v4.4.4 in plaintext and json formats. The collstats, dbstats, metrics, replstatus and status datasets are tested with MongoDB 3.4 and 3.0 and are expected to work with all versions >= 2.8.
Requirements
MongoDB Privileges
In order to use the metrics datasets, the MongoDB user specified in the package configuration needs to have certain privileges.
We recommend using the clusterMonitor role to cover all the necessary privileges.
You can use the following command in Mongo shell to create the privileged user (make sure you are using the admin db by using db command in Mongo shell).
db.createUser(
{
user: "beats",
pwd: "pass",
roles: ["clusterMonitor"]
}
)
You can use the following command in Mongo shell to grant the role to an existing user (make sure you are using the admin db by using db command in Mongo shell).
db.grantRolesToUser("user", ["clusterMonitor"])
Logs
log
The log dataset collects the MongoDB logs.
Metrics
collstats
The collstats dataset uses the top administrative command to return usage statistics for each collection. It provides the amount of time, in microseconds, used and a count of operations for the following types: total, readLock, writeLock, queries, getmore, insert, update, remove, and commands.
It requires the following privileges, which is covered by the clusterMonitor role:
-
top action on cluster resource
dbstats
The dbstats dataset collects storage statistics for a given database.
It requires the following privileges, which is covered by the clusterMonitor role:
-
listDatabases
action on cluster resource
-
for each of the databases, also need dbStats
action on the database resource
metrics
It requires the following privileges, which is covered by the clusterMonitor role:
-
serverStatus
action on cluster resource
replstatus
The replstatus dataset collects status of the replica set. It requires the following privileges, which is covered by the clusterMonitor role:
-
find/listCollections action on the local database resource
-
collStats action on the local.oplog.rs collection resource
-
replSetGetStatus action on cluster resource
status
The status returns a document that provides an overview of the database's state.
It requires the following privileges, which is covered by the clusterMonitor role:
-
serverStatus
action on cluster resource
MongoDB Integration Procedures
Please provide the following information to CyTech:
Collect MongoDB application logs
- Paths
Collect MongoDB metrics
- Hosts
Ex: localhost:27017
OKTA Integrations
Introduction
The Okta integration collects events from the Okta API, specifically reading from the Okta System Log API.
Logs
System
The Okta System Log records system events related to your organization in order to provide an audit trail that can be used to understand platform activity and to diagnose problems. This module is implemented using the httpjson input and is configured to paginate through the logs while honoring any rate-limiting headers sent by Okta.
Okta Integration Procedures
Please provide the following information to CyTech:
Collect Okta logs via API
-
API Key - API Key is required.
-
Okta System Log API Url - Okta System Log API Url is required.
Pulse Connect Secure Integrations
Introduction
This integration is for Pulse Connect Secure.
Pulse Connect Secure Integration Procedures
Please provide the following information to CyTech:
Collect Pulse Connect Secure logs (input: udp)
-
Syslog Host
-
Syslog Port
Collect Pulse Connect Secure logs (input: tcp)
-
Syslog Host
-
Syslog Port
Slack Integrations
Introduction
Slack is used by numerous organizations as their primary chat and collaboration tool.
Please note the Audit Logs API is only available to Slack workspaces on an Enterprise Grid plan. These API methods will not work for workspaces on a Free, Standard, or Business+ plan.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
Configuration
Enabling the integration in Elastic
-
In Kibana go to Management > Integrations
-
In the "Search for integrations" search bar type Slack.
-
Click on "Slack" integration from the search results.
-
Click on Add Slack button to add Slack integration.
Configure Slack audit logs data stream
Enter values "OAuth API Token".
CONFIGURE USING API TOKEN
For the Slack integration to be able to successfully get logs the following "User Token Scopes"" must be granted to the Slack App:
-
auditlogs:read
Logs
Audit
Audit logs summarize the history of changes made within the Slack Enterprise.
SLACK Integration Procedures
Please provide the following information to CyTech:
Collect Slack logs via API
-
API URL - The root URL for the API endpoints.
Slack Audit logs
-
OAuth API Token - The OAuth API Token used to authenticate with the Slack API
System Integrations
Introduction
The System integration allows you to monitor servers, personal computers, and more.
Use the System integration to collect metrics and logs from your machines. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue.
For example, if you wanted to be notified when less than 10% of the disk space is still available, you could install the System integration to send file system metrics to Elastic. Then, you could view real-time updates to disk space used on your system in Kibana's [Metrics System] Overview dashboard. You could also set up a new rule in the Elastic Observability Metrics app to alert you when the percent free is less than 10% of the total disk space.
Data streams
The System integration collects two types of data: logs and metrics.
Logs help you keep a record of events that happen on your machine. Log data streams collected by the System integration include application, system, and security events on machines running Windows and auth and syslog events on machines running macOS or Linux. See more details in the Logs reference.
-
https://aquila-elk.kb.us-east-1.aws.found.io:9243/app/integrations/detail/system-1.20.4/overview#logs-reference
Metrics give you insight into the state of the machine. Metric data streams collected by the System integration include CPU usage, load statistics, memory usage, information on network behavior, and more. See more details in the Metrics reference.
-
https://aquila-elk.kb.us-east-1.aws.found.io:9243/app/integrations/detail/system-1.20.4/overview#metrics-reference
You can enable and disable individual data streams. If all data streams are disabled and the System integration is still enabled, Fleet uses the default data streams.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
Each data stream collects different kinds of metric data, which may require dedicated permissions to be fetched and which may vary across operating systems. Details on the permissions needed for each data stream are available in the Metrics reference.
-
https://aquila-elk.kb.us-east-1.aws.found.io:9243/app/integrations/detail/system-1.20.4/overview#metrics-reference
Setup
For step-by-step instructions on how to set up an integration, see the Getting started guide.
-
https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html
Troubleshooting
Note that certain data streams may access /proc to gather process information, and the resulting ptrace_may_access() call by the kernel to check for permissions can be blocked by AppArmor and other LSM software, even though the System module doesn't use ptrace directly.
-
https://gitlab.com/apparmor/apparmor/wikis/TechnicalDoc_Proc_and_ptrace
In addition, when running inside a container the proc filesystem directory of the host should be set using system.hostfs setting to /hostfs.
Logs reference
Application
The Windows application data stream provides events from the Windows Application event log.
SUPPORTED OPERATING SYSTEMS
-
Windows
System Integration Procedures
Please provide the following information to CyTech:
Collect logs from System instances (Enable Yes/No)
System auth logs (log) (Enable Yes/No)
-
-
-
Collect System auth logs using log input.
-
-
-
Paths
-
Preserve original event (Enable Yes/No)
-
-
-
Preserves a raw copy of the original event, added to the field event.original.
-
-
-
Ignore events older than
-
-
-
If this option is specified, events that are older than the specified amount of time are ignored. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
-
-
Tags
-
Processors (Optional)
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System syslog logs (log)
-
Paths
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Ignore events older than
-
-
-
If this option is specified, events that are older than the specified amount of time are ignored. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
-
-
Synthetic source (Enable Yes/No)
Collect events from the Windows event log (Enable Yes/No)
-
Application (Enable Yes/No)
-
-
-
Collect Windows application logs.
-
-
-
Preserve original event (Enable Yes/No)
-
-
-
Preserves a raw copy of the original XML event, added to the field event.original
-
-
-
Event ID
-
-
-
A list of included and excluded (blocked) event IDs. The value is a comma-separated list. The accepted values are single event IDs to include (e.g. 4624), a range of event IDs to include (e.g. 4700-4800), and single event IDs to exclude (e.g. -4735). Limit 22 IDs.
-
-
-
Ignore events older than
-
-
-
If this option is specified, events that are older than the specified amount of time are ignored. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
-
-
Language ID
-
-
-
The language ID the events will be rendered in. The language will be forced regardless of the system language. A complete list of language IDs can be found https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c[here]. It defaults to 0, which indicates to use the system language. E.g.: 0x0409 for en-US
-
-
-
Tags
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
Security (Enable Yes/No)
-
Preserve original event (Enable Yes/No)
-
-
-
Preserves a raw copy of the original XML event, added to the field event.original
-
-
-
Event ID
-
-
-
A list of included and excluded (blocked) event IDs. The value is a comma-separated list. The accepted values are single event IDs to include (e.g. 4624), a range of event IDs to include (e.g. 4700-4800), and single event IDs to exclude (e.g. -4735). Limit 22 IDs.
-
-
-
Ignore events older than
-
-
-
If this option is specified, events that are older than the specified amount of time are ignored. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
-
-
Language ID
-
-
-
The language ID the events will be rendered in. The language will be forced regardless of the system language. A complete list of language IDs can be found https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c[here]. It defaults to 0, which indicates to use the system language. E.g.: 0x0409 for en-US
-
-
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System (Enable Yes/No)
-
Preserve original event (Enable Yes/No)
-
-
-
Preserves a raw copy of the original XML event, added to the field event.original
-
-
-
Event ID
-
-
-
A list of included and excluded (blocked) event IDs. The value is a comma-separated list. The accepted values are single event IDs to include (e.g. 4624), a range of event IDs to include (e.g. 4700-4800), and single event IDs to exclude (e.g. -4735). Limit 22 IDs.
-
-
-
Ignore events older than
-
-
-
If this option is specified, events that are older than the specified amount of time are ignored. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
-
-
Language ID
-
-
-
The language ID the events will be rendered in. The language will be forced regardless of the system language. A complete list of language IDs can be found https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c[here]. It defaults to 0, which indicates to use the system language. E.g.: 0x0409 for en-US
-
-
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
Collect metrics from System instances (Enable Yes/No)
-
Proc Filesystem Directory
-
-
-
The proc filesystem base directory.
-
-
-
Period
-
Core Metrics
-
-
-
How to report core metrics. Can be "percentages" or "ticks"
-
-
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System CPU metrics (Enable Yes/No)
-
Period
-
Cpu Metrics
-
-
-
How to report CPU metrics. Can be "percentages", "normalized_percentages", or "ticks"
-
-
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System diskio metrics (Enable Yes/No)
-
Period
-
Include Devices
-
-
-
Provide a specific list of devices to monitor. By default, all devices are monitored.
-
-
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System filesystem metrics
-
Period
-
List of filesystem types to ignore
-
-
-
The filesystem datastream will ignore any filesystems with a matching type as specified here. By default, this will exclude any filesystems marked as "nodev" in /proc/filesystems on linux.
-
-
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with external metadata.
-
-
-
Indexing settings (experimental)
-
-
-
Select data streams to configure indexing options. This is an experimental feature and may have effects on other properties.
-
-
-
Synthetic source (Enable Yes/No)
System fsstat metrics (Enable Yes/No)
-
Period
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with external metadata.
-
-
-
Tags
-
Synthetic source (Enable Yes/No)
System load metrics (Enable Yes/No)
-
Period
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System memory metrics (Enable Yes/No)
-
Period
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System network metrics (Enable Yes/No)
-
Period
-
Interfaces
-
-
-
List of interfaces to monitor. Will monitor all by default.
-
-
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System process metrics
-
Period
-
Process Include Top N By Cpu
-
-
-
Include the top N processes by CPU usage.
-
-
-
Process Include Top N By Memory
-
-
-
Include the top N processes by memory usage.
-
-
-
Enable cmdline cache (Enable Yes/No)
-
-
-
If false, cmdline of a process is not cached
-
-
-
Enable cgroup reporting (Enable Yes/No)
-
-
-
Enable collection of cgroup metrics from processes on Linux.
-
-
-
Env whitelist
-
-
-
A list of regular expressions used to whitelist environment variables reported with the process metricset's events. Defaults to empty.
-
-
-
Include CPU Ticks (Enable Yes/No)
-
-
-
Include the cumulative CPU tick values with the process metrics.
-
-
-
Processes
-
-
-
A glob to match reported processes. By default all processes are reported.
-
-
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System process_summary metrics (Enable Yes/No)
-
Period
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System socket_summary metrics (Enable Yes/No)
-
Period
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System uptime metrics (Enable Yes/No)
-
Period
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
Collect logs from third-party REST API (experimental) (Enable Yes/No)
-
URL of Splunk Enterprise Server
-
-
-
i.e. scheme://host:port, path is automatic
-
-
-
Splunk REST API Username
-
Splunk REST API Password
-
Splunk Authorization Token
-
-
-
Bearer Token or Session Key, e.g. "Bearer eyJFd3e46..." or "Splunk 192fd3e...". Cannot be used with username and password.
-
-
-
Preserve original event (Enable Yes/No)
-
-
-
Preserves a raw copy of the original event, added to the field event.original
-
-
-
SSL Configuration
-
-
-
i.e. certificate_authorities, supported_protocols, verification_mode etc.
-
-
Windows Application Events via Splunk Enterprise REST API (Enable Yes/No)
-
Interval to query Splunk Enterprise REST API
-
-
-
Go Duration syntax (eg. 10s)
-
-
-
Splunk search string
-
Tags
-
Synthetic source (Enable Yes/No)
Windows Security Events via Splunk Enterprise REST API (Enable Yes/No)
-
Interval to query Splunk Enterprise REST API
-
-
-
Go Duration syntax (eg. 10s)
-
-
-
Splunk search string
-
Tags
-
Synthetic source (Enable Yes/No)
Windows System Events via Splunk Enterprise REST API (Enable Yes/No)
-
Interval to query Splunk Enterprise REST API
-
-
-
Go Duration syntax (eg. 10s)
-
-
-
Splunk search string
-
Tags
-
Synthetic source (Enable Yes/No)
Team Viewer Integrations
Remote File Copy via TeamViewer
Identifies an executable or script file remotely downloaded via a TeamViewer transfer session.
Rule type: eql
Rule indices:
-
winlogbeat-*
-
logs-endpoint.events.*
-
logs-windows.*
Severity: medium
Risk score: 47
Runs every: 5m
Searches indices from: now-9m (Date Math format, see also Additional look-back time)
Maximum alerts per execution: 100
References:
-
https://blog.menasec.net/2019/11/hunting-for-suspicious-use-of.html
-
Elastic
-
Host
-
Windows
-
Threat Detection
-
Command and Control
Version: 5
-
Elastic
Rule license: Elastic License v2
Rule query
file where event.type == "creation" and process.name : "TeamViewer.exe" and
file.extension : ("exe", "dll", "scr", "com", "bat", "ps1", "vbs", "vbe", "js", "wsh", "hta")
Framework: MITRE ATT&CKTM
-
Tactic:
-
-
-
Name: Command and Control
-
ID: TA0011
-
Reference URL: https://attack.mitre.org/tactics/TA0011/
-
-
-
Technique:
-
-
-
Name: Ingress Tool Transfer
-
ID: T1105
-
Reference URL: https://attack.mitre.org/techniques/T1105/
-
-
-
Technique:
-
-
-
Name: Remote Access Software
-
ID: T1219
-
Reference URL: https://attack.mitre.org/techniques/T1219/
-
-
Source: https://www.elastic.co/guide/en/security/master/prebuilt-rule-0-14-2-remote-file-copy-via-teamviewer.html
TeamViewer Integration Procedure
-
Install the Elastic Stack (Elasticsearch, Kibana, and Logstash) on your Ubuntu machine by following the instructions provided on the Elastic website.
-
Once you have installed and configured the Elastic Stack, navigate to the Logstash directory and create a new configuration file for the TeamViewer logs by running the command:
Copy and paste the following Logstash configuration into the file:
-
Save and close the file.
-
Start Logstash by running the command:
-
Ensure that Logstash is properly reading and processing the TeamViewer logs by checking the Logstash logs in the /var/log/logstash/ directory.
-
Navigate to the Kibana web interface by opening a web browser and entering the URL: http://localhost:5601/.
-
In Kibana, click on the "Discover" tab to view your logs.
-
Click on the "Create index pattern" button and enter the name of the TeamViewer index pattern (e.g. teamviewer-*).
-
Select the time range for the logs you want to view, and click on the "Create index pattern" button.
-
You should now see a list of logs from your TeamViewer deployment. You can filter the logs based on various criteria like severity, source, or date.
-
You can also create custom dashboards or visualizations to monitor specific aspects of your TeamViewer deployment, such as usage patterns or connection quality.
-
If you encounter any issues with your TeamViewer deployment, you can use the logs to identify the root cause and take corrective action.
Source: ChatGPT
-
Elastic official documentation: https://www.elastic.co/guide/index.html
-
Logstash input plugin documentation: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html
-
Logstash output plugin documentation: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
-
Kibana official documentation: https://www.elastic.co/guide/en/kibana/current/index.html
Z Scaler Integrations
Introduction
This integration is for Zscaler Internet Access logs. It can be used to receive logs sent by NSS log server on respective TCP ports.
The log message is expected to be in JSON format. The data is mapped to ECS fields where applicable and the remaining fields are written under zscaler_zia.<data-stream-name>.*.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Compatibility
This package has been tested against Zscaler Internet Access version 6.1
Requirements
Steps for setting up NSS Feeds
-
Enable the integration with the TCP input.
-
Configure the Zscaler NSS Server and NSS Feeds to send logs to the Elastic Agent that is running this integration. See Add NSS Server and Add NSS Feeds. Use the IP address hostname of the Elastic Agent as the 'NSS Feed SIEM IP Address/FQDN', and use the listening port of the Elastic Agent as the 'SIEM TCP Port' on the Add NSS Feed configuration screen. To configure Zscaler NSS Server and NSS Feeds follow the following steps.
-
In the ZIA Admin Portal, add an NSS Server.
-
Log in to the ZIA Admin Portal using your admin account. If you're unable to log in, contact Support.
-
Add an NSS server. Refer to Adding NSS Servers to set up an Add NSS Server for Web and/or Firewall.
-
Verify that the state of the NSS Server is healthy.
-
In the ZIA Admin Portal, go to Administration > Nanolog Streaming Service > NSS Servers.
-
In the State column, confirm that the state of the NSS server is healthy.
-
In the ZIA Admin Portal, add an NSS Feed.
-
Refer to Add NSS Feeds and select the type of feed you want to configure. The following fields require specific inputs:
-
SIEM IP Address: Enter the IP address of the Elastic agent you’ll be assigning the Zscaler integration to.
-
SIEM TCP Port: Enter the port number, depending on the logs associated with the NSS Feed. You will need to create an NSS Feed for each log type.
-
Alerts: 9010
-
DNS: 9011
-
Firewall: 9012
-
Tunnel: 9013
-
Web: 9014
-
- Feed Output Type: Select Custom in Feed output type and paste the appropriate response format in Feed output format as follows:
Steps for setting up Cloud NSS Feeds
-
Enable the integration with the HTTP Endpoint input.
-
Configure the Zscaler Cloud NSS Feeds to send logs to the Elastic Agent that is running this integration. Provide API URL to send logs to the Elastic Agent. To configure Zscaler Cloud NSS Feeds follow the following steps.
-
In the ZIA Admin Portal, add a Cloud NSS Feed.
-
Log in to the ZIA Admin Portal using your admin account.
-
Add a Cloud NSS Feed. See to Add Cloud NSS Feed.
-
In the ZIA Admin Portal, go to Administration > Nanolog Streaming Service > Cloud NSS Feeds.
-
Give Feed Name, change status to Enabled.
-
Select NSS Type.
-
Change SIEM Type to other.
-
Add an API URL.
-
Default ports:
-
DNS: 9556
-
Firewall: 9557
-
Tunnel: 9558
-
Web: 9559
-
-
Select JSON as feed output type.
-
Add same custom header along with its value on both the side for additional security.
-
Repeat step 2 for each log type.
Please make sure to use the given response formats for NSS and Cloud NSS Feeds.
Note: Please make sure to use latest version of given response formats.
Zscaler Integration Procedures
Please provide the following information to CyTech:
Collect Zscaler Internet Access logs via TCP input
-
-
Listen Address - The bind address to listen for TCP connections.
-
Types:
-
-
-
-
-
-
-
TCP Listen Port for Zscaler Internet Access Alerts
-
-
-
-
-
-
-
TCP Listen Port for Zscaler Internet Access DNS logs
-
TCP Listen Port for Zscaler Internet Access Firewall Logs
-
TCP Listen Port for Zscaler Internet Access Tunnel Logs
-
TCP Listen Port for Zscaler Internet Access Web Logs
-
Collect Zscaler Internet Access logs via HTTP Endpoint
-
-
Listen Address - The bind address to listen for http endpoint connections.
-
Types:
-
-
-
TCP Listen Port for Zscaler Internet Access DNS logs
-
TCP Listen Port for Zscaler Internet Access Firewall Logs
-
TCP Listen Port for Zscaler Internet Access Tunnel Logs
-
TCP Listen Port for Zscaler Internet Access Web Logs
-
gcp
Google Cloud Platform
Google Cloud Platform Integration
The Google Cloud integration collects and parses Google Cloud Audit Logs, VPC Flow Logs, Firewall Rules Logs and Cloud DNS Logs that have been exported from Cloud Logging to a Google Pub/Sub topic sink and collects Google Cloud metrics and metadata from Google Cloud Monitoring.
Authentication
To use this Google Cloud Platform (GCP) integration, you need to set up a Service Account with a Role and a Service Account Key to access data on your GCP project.
Service Account
First, you need to create a Service Account. A Service Account (SA) is a particular type of Google account intended to represent a non-human user who needs to access the GCP resources.
The Elastic Agent uses the SA to access data on Google Cloud Platform using the Google APIs.
If you haven't already, this might be a good moment to check out the best practices for securing service accounts guide.
Role
You need to grant your Service Account (SA) access to Google Cloud Platform resources by assigning a role to the account. In order to assign minimal privileges, create a custom role that has only the privileges required by Agent. Those privileges are:
compute.instances.list
(required for GCP Compute instance metadata collection) **monitoring.metricDescriptors.list
monitoring.timeSeries.list
pubsub.subscriptions.consume
pubsub.subscriptions.create
*pubsub.subscriptions.get
pubsub.topics.attachSubscription
*
* Only required if Agent is expected to create a new subscription. If you create the subscriptions yourself you may omit these privileges. ** Only required if corresponding collection will be enabled.
After you have created the custom role, assign the role to your service account.
Service Account Keys
Now, with your brand new Service Account (SA) with access to Google Cloud Platform (GCP) resources, you need some credentials to associate with it: a Service Account Key.
From the list of SA:
- Click the one you just created to open the detailed view.
- From the Keys section, click "Add key" > "Create new key" and select JSON as the type.
- Download and store the generated private key securely (remember, the private key can't be recovered from GCP if lost).
Configure the Integration Settings
The next step is to configure the general integration settings used for all logs from the supported services (Audit, DNS, Firewall, and VPC Flow).
The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration.
Project Id
The Project Id is the Google Cloud project ID where your resources exist.
Credentials File vs Json
Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field.
Option 1: Credentials File
Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file.
Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json
.
Option 2: Credentials JSON
Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration.
Recommendations
Elastic recommends using Credentials File, as in this method the credential information doesn’t leave your Google Cloud Platform environment. When using Credentials JSON, the integration stores the info in Elasticsearch, and the access is controlled based on policy permissions or access to underlying Elasticsearch data.
Logs Collection Configuration
With a properly configured Service Account and the integration setting in place, it's time to start collecting some logs.
Requirements
You need to create a few dedicated Google Cloud resources before starting, in detail:
- Log Sink
- Pub/Sub Topic
- Subscription
Elastic recommends separate Pub/Sub topics for each of the log types so that they can be parsed and stored in a specific data stream.
Here's an example of collecting Audit Logs using a Pub/Sub topic, a subscription, and a Log Router. We will create the resources in the Google Cloud Console and then configure the Google Cloud Platform integration.
On the Google Cloud Console
At a high level, the steps required are:
- Visit "Logging" > "Log Router" > "Create Sink" and provide a sink name and description.
- In "Sink destination", select "Cloud Pub/Sub topic" as the sink service. Select an existing topic or "Create a topic". Note the topic name, as it will be provided in the Topic field in the Elastic agent configuration.
- If you created a new topic, you must remember to go to that topic and create a subscription for it. A subscription directs messages on a topic to subscribers. Note the "Subscription ID", as it will need to be entered in the "Subscription name" field in the integration settings.
- Under "Choose logs to include in sink", for example add
logName:"cloudaudit.googleapis.com"
in the "Inclusion filter" to include all audit logs.
This is just an example; you will need to create your filter expression to select the log types you want to export to the Pub/Sub topic.
More example filters for different log types:
#
# VPC Flow: logs for specific subnet
#
resource.type="gce_subnetwork" AND
log_id("compute.googleapis.com/vpc_flows") AND
resource.labels.subnetwork_name"=[SUBNET_NAME]"
#
# Audit: Google Compute Engine firewall rule deletion
#
resource.type="gce_firewall_rule" AND
log_id("cloudaudit.googleapis.com/activity") AND
protoPayload.methodName:"firewalls.delete"
#
# DNS: all DNS queries
#
resource.type="dns_query"
#
# Firewall: logs for a given country
#
resource.type="gce_subnetwork" AND
log_id("compute.googleapis.com/firewall") AND
jsonPayload.remote_location.country=[COUNTRY_ISO_ALPHA_3]
Start working on your query using the Google Cloud Logs Explorer, so you can preview and pinpoint the exact log types you want to forward to your Elastic Stack.
To learn more, please read how to Build queries in the Logs Explorer, and take a look at the Sample queries using the Logs Explorer page in the Google Cloud docs.
On Kibana
Visit "Management" > "Integrations" > "Installed Integrations" > "Google Cloud Platform" and select the "Integration Policies" tab. Select the integration policy you previously created.
From the list of services, select "Google Cloud Platform (GCP) audit logs (gcp-pubsub)" and:
- On the "Topic" field, specify the "topic name" you noted before on the Google Cloud Console.
- On the "Subscription Name", specify the short subscription name you noted before on the Google Cloud Console (note: do NOT use the full-blown subscription name made of project/PROJECT_ID/subscriptions/SUBSCRIPTION_ID). Just pick the Subscription ID from the Google Cloud Console).
- Click on "Save Integration", and make sure the Elastic Agent gets the updated policy.
Troubleshooting
If you don't see Audit logs showing up, check the Agent logs to see if there are errors.
Common error types:
- Missing roles in the Service Account
- Misconfigured settings, like "Project Id", "Topic" or "Subscription Name" fields
Missing Roles in the Service Account
If your Service Account (SA) does not have the required roles, you might find errors like this one in the elastic_agent.filebeat
dataset:
failed to subscribe to pub/sub topic: failed to check if subscription exists: rpc error: code = PermissionDenied desc = User not authorized to perform this action.
Solution: make sure your SA has all the required roles.
Misconfigured Settings
If you specify the wrong "Topic field" or "Subscription Name", you might find errors like this one in the elastic_agent.filebeat
dataset:
[elastic_agent.filebeat][error] failed to subscribe to pub/sub topic: failed to check if subscription exists: rpc error: code = InvalidArgument desc = Invalid resource name given (name=projects/project/subscriptions/projects/project/subscriptions/non-existent-sub). Refer to https://cloud.google.com/pubsub/docs/admin#resource_names for more information.
Solution: double check the integration settings.
Metrics Collection Configuration
With a properly configured Service Account and the integration setting in place, it's time to start collecting some metrics.
Requirements
No additional requirement is needed to collect metrics.
Troubleshooting
If you don't see metrics showing up, check the Agent logs to see if there are errors.
Common error types:
- Period is lower than 60 seconds
- Missing roles in the Service Account
- Misconfigured settings, like "Project Id"
Period is lower than 60 seconds
Usual minimum collection period for GCP metrics is 60 seconds. Any value lower than that cause an error when retrieving the metric metadata. If an error happens, the affected metric is skipped at the metric collection stage, resulting in no data being sent.
Missing Roles in the Service Account
If your Service Account (SA) does not have required roles, you might find errors related to accessing GCP resources.
To check you may add Monitoring Viewer
and Compute Viewer
roles (built-in GCP roles) to your SA. These roles contain the permission added in the previous step and expand them with additional permissions. You can analyze additional missing permissions from the GCP Console > IAM > clicking on the down arrow near the roles on the same line of your SA > View analyzed permissions. From the shown table you can check which permissions from the role the SA is actively using. They should match what you configured in your custom role.
Misconfigured Settings
If you specify a wrong setting you will probably find errors related to missing GCP resources.
Make sure the settings are correct and the SA has proper permissions for the given "Project Id".
Logs
Audit
The audit
dataset collects audit logs of administrative activities and accesses within your Google Cloud resources.
Exported fields
Field | Description | Type |
---|---|---|
@timestamp
|
Event timestamp.
|
date
|
client.user.email
|
User email address.
|
keyword
|
client.user.id
|
Unique identifier of the user.
|
keyword
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
cloud.availability_zone
|
Availability zone in which this host is running.
|
keyword
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
cloud.region
|
Region in which this host is running.
|
keyword
|
container.id
|
Unique container id.
|
keyword
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
container.labels
|
Image labels.
|
object
|
container.name
|
Container name.
|
keyword
|
container.runtime
|
Runtime managing this container.
|
keyword
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
error.code
|
Error code describing the error.
|
keyword
|
error.message
|
Error message.
|
match_only_text
|
event.action
|
The action captured by the event. This describes the information in the event. It is more specific than
event.category . Examples are group-add , process-started , file-created . The value is normally defined by the implementer. |
keyword
|
event.created
|
event.created contains the date/time when the event was first read by an agent, or by your pipeline. This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event. In most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source. In case the two timestamps are identical, @timestamp should be used.
|
date
|
event.dataset
|
Event dataset
|
constant_keyword
|
event.id
|
Unique ID to describe the event.
|
keyword
|
event.ingested
|
Timestamp when an event arrived in the central data store. This is different from
@timestamp , which is when the event originally occurred. It's also different from event.created , which is meant to capture the first time an agent saw the event. In normal conditions, assuming no tampering, the timestamps should chronologically look like this: @timestamp < event.created < event.ingested . |
date
|
event.kind
|
This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy.
event.kind gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. |
keyword
|
event.module
|
Event module
|
constant_keyword
|
event.original
|
Raw text message of entire event. Used to demonstrate log integrity or where the full log message (before splitting it up in multiple parts) may be required, e.g. for reindex. This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from
_source . If users wish to override this and index this field, please see Field data types in the Elasticsearch Reference . |
keyword
|
event.outcome
|
This is one of four ECS Categorization Fields, and indicates the lowest level in the ECS category hierarchy.
event.outcome simply denotes whether the event represents a success or a failure from the perspective of the entity that produced the event. Note that when a single transaction is described in multiple events, each event may populate different values of event.outcome , according to their perspective. Also note that in the case of a compound event (a single event that contains multiple logical events), this field should be populated with the value that best captures the overall success or failure from the perspective of the event producer. Further note that not all events will have an associated outcome. For example, this field is generally not populated for metric events, events with event.type:info , or any events for which an outcome does not make logical sense. |
keyword
|
gcp.audit.authentication_info.authority_selector
|
The authority selector specified by the requestor, if any. It is not guaranteed that the principal was allowed to use this authority.
|
keyword
|
gcp.audit.authentication_info.principal_email
|
The email address of the authenticated user making the request.
|
keyword
|
gcp.audit.authentication_info.principal_subject
|
String representation of identity of requesting party. Populated for both first and third party identities. Only present for APIs that support third-party identities.
|
keyword
|
gcp.audit.authorization_info.granted
|
Whether or not authorization for resource and permission was granted.
|
boolean
|
gcp.audit.authorization_info.permission
|
The required IAM permission.
|
keyword
|
gcp.audit.authorization_info.resource
|
The resource being accessed, as a REST-style string.
|
keyword
|
gcp.audit.authorization_info.resource_attributes.name
|
The name of the resource.
|
keyword
|
gcp.audit.authorization_info.resource_attributes.service
|
The name of the service.
|
keyword
|
gcp.audit.authorization_info.resource_attributes.type
|
The type of the resource.
|
keyword
|
gcp.audit.flattened
|
Contains the full audit document as sent by GCP.
|
flattened
|
gcp.audit.labels
|
A map of key, value pairs that provides additional information about the log entry. The labels can be user-defined or system-defined.
|
flattened
|
gcp.audit.logentry_operation.first
|
Optional. Set this to True if this is the first log entry in the operation.
|
boolean
|
gcp.audit.logentry_operation.id
|
Optional. An arbitrary operation identifier. Log entries with the same identifier are assumed to be part of the same operation.
|
keyword
|
gcp.audit.logentry_operation.last
|
Optional. Set this to True if this is the last log entry in the operation.
|
boolean
|
gcp.audit.logentry_operation.producer
|
Optional. An arbitrary producer identifier. The combination of id and producer must be globally unique.
|
keyword
|
gcp.audit.method_name
|
The name of the service method or operation. For API calls, this should be the name of the API method. For example, 'google.datastore.v1.Datastore.RunQuery'.
|
keyword
|
gcp.audit.num_response_items
|
The number of items returned from a List or Query API method, if applicable.
|
long
|
gcp.audit.request
|
flattened
|
|
gcp.audit.request_metadata.caller_ip
|
The IP address of the caller.
|
ip
|
gcp.audit.request_metadata.caller_supplied_user_agent
|
The user agent of the caller. This information is not authenticated and should be treated accordingly.
|
keyword
|
gcp.audit.request_metadata.raw.caller_ip
|
The raw IP address of the caller.
|
keyword
|
gcp.audit.resource_location.current_locations
|
Current locations of the resource.
|
keyword
|
gcp.audit.resource_name
|
The resource or collection that is the target of the operation. The name is a scheme-less URI, not including the API service name. For example, 'shelves/SHELF_ID/books'.
|
keyword
|
gcp.audit.response
|
flattened
|
|
gcp.audit.service_name
|
The name of the API service performing the operation. For example, datastore.googleapis.com.
|
keyword
|
gcp.audit.status.code
|
The status code, which should be an enum value of google.rpc.Code.
|
integer
|
gcp.audit.status.message
|
A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
|
keyword
|
gcp.audit.type
|
Type property.
|
keyword
|
gcp.destination.instance.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.destination.instance.region
|
Region of the VM.
|
keyword
|
gcp.destination.instance.zone
|
Zone of the VM.
|
keyword
|
gcp.destination.vpc.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.destination.vpc.subnetwork_name
|
Subnetwork on which the VM is operating.
|
keyword
|
gcp.destination.vpc.vpc_name
|
VPC on which the VM is operating.
|
keyword
|
gcp.source.instance.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.source.instance.region
|
Region of the VM.
|
keyword
|
gcp.source.instance.zone
|
Zone of the VM.
|
keyword
|
gcp.source.vpc.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.source.vpc.subnetwork_name
|
Subnetwork on which the VM is operating.
|
keyword
|
gcp.source.vpc.vpc_name
|
VPC on which the VM is operating.
|
keyword
|
host.architecture
|
Operating system architecture.
|
keyword
|
host.containerized
|
If the host is a container.
|
boolean
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
host.ip
|
Host ip addresses.
|
ip
|
host.mac
|
Host mac addresses.
|
keyword
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
host.os.build
|
OS build information.
|
keyword
|
host.os.codename
|
OS codename, if any.
|
keyword
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
input.type
|
Input type
|
keyword
|
log.file.path
|
Full path to the log file this event came from, including the file name. It should include the drive letter, when appropriate. If the event wasn't read from a log file, do not populate this field.
|
keyword
|
log.level
|
Original log level of the log event. If the source of the event provides a log level or textual severity, this is the one that goes in
log.level . If your source doesn't specify one, you may put your event transport's severity here (e.g. Syslog severity). Some examples are warn , err , i , informational . |
keyword
|
log.logger
|
The name of the logger inside an application. This is usually the name of the class which initialized the logger, or can be a custom name.
|
keyword
|
log.offset
|
Log offset
|
long
|
message
|
For log events the message field contains the log message, optimized for viewing in a log viewer. For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event. If multiple messages exist, they can be combined into one message.
|
match_only_text
|
orchestrator.api_version
|
API version being used to carry out the action
|
keyword
|
orchestrator.cluster.name
|
Name of the cluster.
|
keyword
|
orchestrator.cluster.url
|
URL of the API used to manage the cluster.
|
keyword
|
orchestrator.cluster.version
|
The version of the cluster.
|
keyword
|
orchestrator.namespace
|
Namespace in which the action is taking place.
|
keyword
|
orchestrator.organization
|
Organization affected by the event (for multi-tenant orchestrator setups).
|
keyword
|
orchestrator.resource.name
|
Name of the resource being acted upon.
|
keyword
|
orchestrator.resource.type
|
Type of resource being acted upon.
|
keyword
|
orchestrator.type
|
Orchestrator cluster type (e.g. kubernetes, nomad or cloudfoundry).
|
keyword
|
service.name
|
Name of the service data is collected from. The name of the service is normally user given. This allows for distributed services that run on multiple hosts to correlate the related instances based on the name. In the case of Elasticsearch the
service.name could contain the cluster name. For Beats the service.name is by default a copy of the service.type field if no name is specified. |
keyword
|
source.address
|
Some event source addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the
.address field. Then it should be duplicated to .ip or .domain , depending on which one it is. |
keyword
|
source.as.number
|
Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
|
long
|
source.as.organization.name
|
Organization name.
|
keyword
|
source.as.organization.name.text
|
Multi-field of
source.as.organization.name . |
match_only_text
|
source.geo.city_name
|
City name.
|
keyword
|
source.geo.continent_name
|
Name of the continent.
|
keyword
|
source.geo.country_iso_code
|
Country ISO code.
|
keyword
|
source.geo.country_name
|
Country name.
|
keyword
|
source.geo.location
|
Longitude and latitude.
|
geo_point
|
source.geo.region_iso_code
|
Region ISO code.
|
keyword
|
source.geo.region_name
|
Region name.
|
keyword
|
source.ip
|
IP address of the source (IPv4 or IPv6).
|
ip
|
tags
|
List of keywords used to tag each event.
|
keyword
|
user.email
|
User email address.
|
keyword
|
user_agent.device.name
|
Name of the device.
|
keyword
|
user_agent.name
|
Name of the user agent.
|
keyword
|
user_agent.original
|
Unparsed user_agent string.
|
keyword
|
user_agent.original.text
|
Multi-field of
user_agent.original . |
match_only_text
|
user_agent.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
user_agent.os.full
|
Operating system name, including the version or code name.
|
keyword
|
user_agent.os.full.text
|
Multi-field of
user_agent.os.full . |
match_only_text
|
user_agent.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
user_agent.os.name
|
Operating system name, without the version.
|
keyword
|
user_agent.os.name.text
|
Multi-field of
user_agent.os.name . |
match_only_text
|
user_agent.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
user_agent.os.version
|
Operating system version as a raw string.
|
keyword
|
user_agent.version
|
Version of the user agent.
|
keyword
|
An example event for audit
looks as following:
{
"@timestamp": "2019-12-19T00:44:25.051Z",
"agent": {
"ephemeral_id": "a22278bb-5e1f-4ab7-b468-277c8c0b80a9",
"id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.7.1"
},
"client": {
"user": {
"email": "xxx@xxx.xxx"
}
},
"cloud": {
"project": {
"id": "elastic-beats"
},
"provider": "gcp"
},
"data_stream": {
"dataset": "gcp.audit",
"namespace": "ep",
"type": "logs"
},
"ecs": {
"version": "8.8.0"
},
"elastic_agent": {
"id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
"snapshot": false,
"version": "8.7.1"
},
"event": {
"action": "beta.compute.instances.aggregatedList",
"agent_id_status": "verified",
"category": [
"network",
"configuration"
],
"created": "2023-10-25T04:18:46.637Z",
"dataset": "gcp.audit",
"id": "yonau2dg2zi",
"ingested": "2023-10-25T04:18:47Z",
"kind": "event",
"outcome": "success",
"provider": "data_access",
"type": [
"access",
"allowed"
]
},
"gcp": {
"audit": {
"authorization_info": [
{
"granted": true,
"permission": "compute.instances.list",
"resource_attributes": {
"name": "projects/elastic-beats",
"service": "resourcemanager",
"type": "resourcemanager.projects"
}
}
],
"num_response_items": 61,
"request": {
"@type": "type.googleapis.com/compute.instances.aggregatedList"
},
"resource_location": {
"current_locations": [
"global"
]
},
"resource_name": "projects/elastic-beats/global/instances",
"response": {
"@type": "core.k8s.io/v1.Status",
"apiVersion": "v1",
"details": {
"group": "batch",
"kind": "jobs",
"name": "gsuite-exporter-1589294700",
"uid": "2beff34a-945f-11ea-bacf-42010a80007f"
},
"kind": "Status",
"status_value": "Success"
},
"type": "type.googleapis.com/google.cloud.audit.AuditLog"
}
},
"input": {
"type": "gcp-pubsub"
},
"log": {
"level": "INFO",
"logger": "projects/elastic-beats/logs/cloudaudit.googleapis.com%2Fdata_access"
},
"service": {
"name": "compute.googleapis.com"
},
"source": {
"ip": "192.168.1.1"
},
"tags": [
"forwarded",
"gcp-audit"
],
"user_agent": {
"device": {
"name": "Mac"
},
"name": "Firefox",
"original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:71.0) Gecko/20100101 Firefox/71.0,gzip(gfe),gzip(gfe)",
"os": {
"full": "Mac OS X 10.15",
"name": "Mac OS X",
"version": "10.15"
},
"version": "71.0."
}
}
Firewall
The firewall
dataset collects logs from Firewall Rules in your Virtual Private Cloud (VPC) networks.
Exported fields
Field | Description | Type |
---|---|---|
@timestamp
|
Event timestamp.
|
date
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
cloud.availability_zone
|
Availability zone in which this host is running.
|
keyword
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
cloud.region
|
Region in which this host is running.
|
keyword
|
container.id
|
Unique container id.
|
keyword
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
container.labels
|
Image labels.
|
object
|
container.name
|
Container name.
|
keyword
|
container.runtime
|
Runtime managing this container.
|
keyword
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
destination.address
|
Some event destination addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the
.address field. Then it should be duplicated to .ip or .domain , depending on which one it is. |
keyword
|
destination.as.number
|
Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
|
long
|
destination.as.organization.name
|
Organization name.
|
keyword
|
destination.as.organization.name.text
|
Multi-field of
destination.as.organization.name . |
match_only_text
|
destination.domain
|
The domain name of the destination system. This value may be a host name, a fully qualified domain name, or another host naming format. The value may derive from the original event or be added from enrichment.
|
keyword
|
destination.geo.city_name
|
City name.
|
keyword
|
destination.geo.continent_name
|
Name of the continent.
|
keyword
|
destination.geo.country_iso_code
|
Country ISO code.
|
keyword
|
destination.geo.country_name
|
Country name.
|
keyword
|
destination.geo.location
|
Longitude and latitude.
|
geo_point
|
destination.geo.name
|
User-defined description of a location, at the level of granularity they care about. Could be the name of their data centers, the floor number, if this describes a local physical entity, city names. Not typically used in automated geolocation.
|
keyword
|
destination.geo.region_iso_code
|
Region ISO code.
|
keyword
|
destination.geo.region_name
|
Region name.
|
keyword
|
destination.ip
|
IP address of the destination (IPv4 or IPv6).
|
ip
|
destination.port
|
Port of the destination.
|
long
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
event.action
|
The action captured by the event. This describes the information in the event. It is more specific than
event.category . Examples are group-add , process-started , file-created . The value is normally defined by the implementer. |
keyword
|
event.category
|
This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy.
event.category represents the "big buckets" of ECS categories. For example, filtering on event.category:process yields all events relating to process activity. This field is closely related to event.type , which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories. |
keyword
|
event.created
|
event.created contains the date/time when the event was first read by an agent, or by your pipeline. This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event. In most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source. In case the two timestamps are identical, @timestamp should be used.
|
date
|
event.dataset
|
Event dataset
|
constant_keyword
|
event.id
|
Unique ID to describe the event.
|
keyword
|
event.ingested
|
Timestamp when an event arrived in the central data store. This is different from
@timestamp , which is when the event originally occurred. It's also different from event.created , which is meant to capture the first time an agent saw the event. In normal conditions, assuming no tampering, the timestamps should chronologically look like this: @timestamp < event.created < event.ingested . |
date
|
event.kind
|
This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy.
event.kind gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. |
keyword
|
event.module
|
Event module
|
constant_keyword
|
event.original
|
Raw text message of entire event. Used to demonstrate log integrity or where the full log message (before splitting it up in multiple parts) may be required, e.g. for reindex. This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from
_source . If users wish to override this and index this field, please see Field data types in the Elasticsearch Reference . |
keyword
|
event.outcome
|
This is one of four ECS Categorization Fields, and indicates the lowest level in the ECS category hierarchy.
event.outcome simply denotes whether the event represents a success or a failure from the perspective of the entity that produced the event. Note that when a single transaction is described in multiple events, each event may populate different values of event.outcome , according to their perspective. Also note that in the case of a compound event (a single event that contains multiple logical events), this field should be populated with the value that best captures the overall success or failure from the perspective of the event producer. Further note that not all events will have an associated outcome. For example, this field is generally not populated for metric events, events with event.type:info , or any events for which an outcome does not make logical sense. |
keyword
|
event.type
|
This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy.
event.type represents a categorization "sub-bucket" that, when used along with the event.category field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. |
keyword
|
gcp.destination.instance.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.destination.instance.region
|
Region of the VM.
|
keyword
|
gcp.destination.instance.zone
|
Zone of the VM.
|
keyword
|
gcp.destination.vpc.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.destination.vpc.subnetwork_name
|
Subnetwork on which the VM is operating.
|
keyword
|
gcp.destination.vpc.vpc_name
|
VPC on which the VM is operating.
|
keyword
|
gcp.firewall.flattened
|
Contains the full firewall document as sent by GCP.
|
flattened
|
gcp.firewall.rule_details.action
|
Action that the rule performs on match.
|
keyword
|
gcp.firewall.rule_details.destination_range
|
List of destination ranges that the firewall applies to.
|
keyword
|
gcp.firewall.rule_details.direction
|
Direction of traffic that matches this rule.
|
keyword
|
gcp.firewall.rule_details.ip_port_info
|
List of ip protocols and applicable port ranges for rules.
|
nested
|
gcp.firewall.rule_details.priority
|
The priority for the firewall rule.
|
long
|
gcp.firewall.rule_details.reference
|
Reference to the firewall rule.
|
keyword
|
gcp.firewall.rule_details.source_range
|
List of source ranges that the firewall rule applies to.
|
keyword
|
gcp.firewall.rule_details.source_service_account
|
List of all the source service accounts that the firewall rule applies to.
|
keyword
|
gcp.firewall.rule_details.source_tag
|
List of all the source tags that the firewall rule applies to.
|
keyword
|
gcp.firewall.rule_details.target_service_account
|
List of all the target service accounts that the firewall rule applies to.
|
keyword
|
gcp.firewall.rule_details.target_tag
|
List of all the target tags that the firewall rule applies to.
|
keyword
|
gcp.source.instance.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.source.instance.region
|
Region of the VM.
|
keyword
|
gcp.source.instance.zone
|
Zone of the VM.
|
keyword
|
gcp.source.vpc.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.source.vpc.subnetwork_name
|
Subnetwork on which the VM is operating.
|
keyword
|
gcp.source.vpc.vpc_name
|
VPC on which the VM is operating.
|
keyword
|
host.architecture
|
Operating system architecture.
|
keyword
|
host.containerized
|
If the host is a container.
|
boolean
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
host.ip
|
Host ip addresses.
|
ip
|
host.mac
|
Host mac addresses.
|
keyword
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
host.os.build
|
OS build information.
|
keyword
|
host.os.codename
|
OS codename, if any.
|
keyword
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
input.type
|
Input type
|
keyword
|
log.file.path
|
Full path to the log file this event came from, including the file name. It should include the drive letter, when appropriate. If the event wasn't read from a log file, do not populate this field.
|
keyword
|
log.logger
|
The name of the logger inside an application. This is usually the name of the class which initialized the logger, or can be a custom name.
|
keyword
|
log.offset
|
Log offset
|
long
|
message
|
For log events the message field contains the log message, optimized for viewing in a log viewer. For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event. If multiple messages exist, they can be combined into one message.
|
match_only_text
|
network.community_id
|
A hash of source and destination IPs and ports, as well as the protocol used in a communication. This is a tool-agnostic standard to identify flows. Learn more at https://github.com/corelight/community-id-spec.
|
keyword
|
network.direction
|
Direction of the network traffic. When mapping events from a host-based monitoring context, populate this field from the host's point of view, using the values "ingress" or "egress". When mapping events from a network or perimeter-based monitoring context, populate this field from the point of view of the network perimeter, using the values "inbound", "outbound", "internal" or "external". Note that "internal" is not crossing perimeter boundaries, and is meant to describe communication between two hosts within the perimeter. Note also that "external" is meant to describe traffic between two hosts that are external to the perimeter. This could for example be useful for ISPs or VPN service providers.
|
keyword
|
network.iana_number
|
IANA Protocol Number (https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml). Standardized list of protocols. This aligns well with NetFlow and sFlow related logs which use the IANA Protocol Number.
|
keyword
|
network.name
|
Name given by operators to sections of their network.
|
keyword
|
network.transport
|
Same as network.iana_number, but instead using the Keyword name of the transport layer (udp, tcp, ipv6-icmp, etc.) The field value must be normalized to lowercase for querying.
|
keyword
|
network.type
|
In the OSI Model this would be the Network Layer. ipv4, ipv6, ipsec, pim, etc The field value must be normalized to lowercase for querying.
|
keyword
|
related.hash
|
All the hashes seen on your event. Populating this field, then using it to search for hashes can help in situations where you're unsure what the hash algorithm is (and therefore which key name to search).
|
keyword
|
related.hosts
|
All hostnames or other host identifiers seen on your event. Example identifiers include FQDNs, domain names, workstation names, or aliases.
|
keyword
|
related.ip
|
All of the IPs seen on your event.
|
ip
|
related.user
|
All the user names or other user identifiers seen on the event.
|
keyword
|
rule.name
|
The name of the rule or signature generating the event.
|
keyword
|
source.address
|
Some event source addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the
.address field. Then it should be duplicated to .ip or .domain , depending on which one it is. |
keyword
|
source.as.number
|
Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
|
long
|
source.as.organization.name
|
Organization name.
|
keyword
|
source.as.organization.name.text
|
Multi-field of
source.as.organization.name . |
match_only_text
|
source.domain
|
The domain name of the source system. This value may be a host name, a fully qualified domain name, or another host naming format. The value may derive from the original event or be added from enrichment.
|
keyword
|
source.geo.city_name
|
City name.
|
keyword
|
source.geo.continent_name
|
Name of the continent.
|
keyword
|
source.geo.country_iso_code
|
Country ISO code.
|
keyword
|
source.geo.country_name
|
Country name.
|
keyword
|
source.geo.location
|
Longitude and latitude.
|
geo_point
|
source.geo.name
|
User-defined description of a location, at the level of granularity they care about. Could be the name of their data centers, the floor number, if this describes a local physical entity, city names. Not typically used in automated geolocation.
|
keyword
|
source.geo.region_iso_code
|
Region ISO code.
|
keyword
|
source.geo.region_name
|
Region name.
|
keyword
|
source.ip
|
IP address of the source (IPv4 or IPv6).
|
ip
|
source.port
|
Port of the source.
|
long
|
tags
|
List of keywords used to tag each event.
|
keyword
|
An example event for firewall
looks as following:
{
"@timestamp": "2019-10-30T13:52:42.191Z",
"agent": {
"ephemeral_id": "175ae0b3-355c-4ca7-87ea-d5f1ee34102e",
"id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.7.1"
},
"cloud": {
"availability_zone": "us-east1-b",
"project": {
"id": "test-beats"
},
"provider": "gcp",
"region": "us-east1"
},
"data_stream": {
"dataset": "gcp.firewall",
"namespace": "ep",
"type": "logs"
},
"destination": {
"address": "10.42.0.2",
"domain": "test-windows",
"ip": "10.42.0.2",
"port": 3389
},
"ecs": {
"version": "8.8.0"
},
"elastic_agent": {
"id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
"snapshot": false,
"version": "8.7.1"
},
"event": {
"action": "firewall-rule",
"agent_id_status": "verified",
"category": [
"network"
],
"created": "2023-10-25T04:20:37.182Z",
"dataset": "gcp.firewall",
"id": "1f21ciqfpfssuo",
"ingested": "2023-10-25T04:20:41Z",
"kind": "event",
"type": [
"allowed",
"connection"
]
},
"gcp": {
"destination": {
"instance": {
"project_id": "test-beats",
"region": "us-east1",
"zone": "us-east1-b"
},
"vpc": {
"project_id": "test-beats",
"subnetwork_name": "windows-isolated",
"vpc_name": "windows-isolated"
}
},
"firewall": {
"rule_details": {
"action": "ALLOW",
"direction": "INGRESS",
"ip_port_info": [
{
"ip_protocol": "TCP",
"port_range": [
"3389"
]
}
],
"priority": 1000,
"source_range": [
"0.0.0.0/0"
],
"target_tag": [
"allow-rdp"
]
}
}
},
"input": {
"type": "gcp-pubsub"
},
"log": {
"logger": "projects/test-beats/logs/compute.googleapis.com%2Ffirewall"
},
"network": {
"community_id": "1:OdLB9eXsBDLz8m97ao4LepX6q+4=",
"direction": "inbound",
"iana_number": "6",
"name": "windows-isolated",
"transport": "tcp",
"type": "ipv4"
},
"related": {
"ip": [
"192.168.2.126",
"10.42.0.2"
]
},
"rule": {
"name": "network:windows-isolated/firewall:windows-isolated-allow-rdp"
},
"source": {
"address": "192.168.2.126",
"geo": {
"continent_name": "Asia",
"country_name": "omn"
},
"ip": "192.168.2.126",
"port": 64853
},
"tags": [
"forwarded",
"gcp-firewall"
]
}
VPC Flow
The vpcflow
dataset collects logs sent from and received by VM instances, including instances used as GKE nodes.
Exported fields
Field | Description | Type |
---|---|---|
@timestamp
|
Event timestamp.
|
date
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
cloud.availability_zone
|
Availability zone in which this host is running.
|
keyword
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
cloud.region
|
Region in which this host is running.
|
keyword
|
container.id
|
Unique container id.
|
keyword
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
container.labels
|
Image labels.
|
object
|
container.name
|
Container name.
|
keyword
|
container.runtime
|
Runtime managing this container.
|
keyword
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
destination.address
|
Some event destination addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the
.address field. Then it should be duplicated to .ip or .domain , depending on which one it is. |
keyword
|
destination.as.number
|
Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
|
long
|
destination.as.organization.name
|
Organization name.
|
keyword
|
destination.as.organization.name.text
|
Multi-field of
destination.as.organization.name . |
match_only_text
|
destination.domain
|
The domain name of the destination system. This value may be a host name, a fully qualified domain name, or another host naming format. The value may derive from the original event or be added from enrichment.
|
keyword
|
destination.geo.city_name
|
City name.
|
keyword
|
destination.geo.continent_name
|
Name of the continent.
|
keyword
|
destination.geo.country_iso_code
|
Country ISO code.
|
keyword
|
destination.geo.country_name
|
Country name.
|
keyword
|
destination.geo.location
|
Longitude and latitude.
|
geo_point
|
destination.geo.name
|
User-defined description of a location, at the level of granularity they care about. Could be the name of their data centers, the floor number, if this describes a local physical entity, city names. Not typically used in automated geolocation.
|
keyword
|
destination.geo.region_iso_code
|
Region ISO code.
|
keyword
|
destination.geo.region_name
|
Region name.
|
keyword
|
destination.ip
|
IP address of the destination (IPv4 or IPv6).
|
ip
|
destination.port
|
Port of the destination.
|
long
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
event.action
|
The action captured by the event. This describes the information in the event. It is more specific than
event.category . Examples are group-add , process-started , file-created . The value is normally defined by the implementer. |
keyword
|
event.category
|
This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy.
event.category represents the "big buckets" of ECS categories. For example, filtering on event.category:process yields all events relating to process activity. This field is closely related to event.type , which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories. |
keyword
|
event.created
|
event.created contains the date/time when the event was first read by an agent, or by your pipeline. This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event. In most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source. In case the two timestamps are identical, @timestamp should be used.
|
date
|
event.dataset
|
Event dataset
|
constant_keyword
|
event.end
|
event.end contains the date when the event ended or when the activity was last observed.
|
date
|
event.id
|
Unique ID to describe the event.
|
keyword
|
event.ingested
|
Timestamp when an event arrived in the central data store. This is different from
@timestamp , which is when the event originally occurred. It's also different from event.created , which is meant to capture the first time an agent saw the event. In normal conditions, assuming no tampering, the timestamps should chronologically look like this: @timestamp < event.created < event.ingested . |
date
|
event.kind
|
This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy.
event.kind gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. |
keyword
|
event.module
|
Event module
|
constant_keyword
|
event.original
|
Raw text message of entire event. Used to demonstrate log integrity or where the full log message (before splitting it up in multiple parts) may be required, e.g. for reindex. This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from
_source . If users wish to override this and index this field, please see Field data types in the Elasticsearch Reference . |
keyword
|
event.outcome
|
This is one of four ECS Categorization Fields, and indicates the lowest level in the ECS category hierarchy.
event.outcome simply denotes whether the event represents a success or a failure from the perspective of the entity that produced the event. Note that when a single transaction is described in multiple events, each event may populate different values of event.outcome , according to their perspective. Also note that in the case of a compound event (a single event that contains multiple logical events), this field should be populated with the value that best captures the overall success or failure from the perspective of the event producer. Further note that not all events will have an associated outcome. For example, this field is generally not populated for metric events, events with event.type:info , or any events for which an outcome does not make logical sense. |
keyword
|
event.start
|
event.start contains the date when the event started or when the activity was first observed.
|
date
|
event.type
|
This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy.
event.type represents a categorization "sub-bucket" that, when used along with the event.category field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. |
keyword
|
gcp.destination.instance.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.destination.instance.region
|
Region of the VM.
|
keyword
|
gcp.destination.instance.zone
|
Zone of the VM.
|
keyword
|
gcp.destination.vpc.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.destination.vpc.subnetwork_name
|
Subnetwork on which the VM is operating.
|
keyword
|
gcp.destination.vpc.vpc_name
|
VPC on which the VM is operating.
|
keyword
|
gcp.source.instance.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.source.instance.region
|
Region of the VM.
|
keyword
|
gcp.source.instance.zone
|
Zone of the VM.
|
keyword
|
gcp.source.vpc.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.source.vpc.subnetwork_name
|
Subnetwork on which the VM is operating.
|
keyword
|
gcp.source.vpc.vpc_name
|
VPC on which the VM is operating.
|
keyword
|
gcp.vpcflow.flattened
|
Contains the full vpcflow document as sent by GCP.
|
flattened
|
gcp.vpcflow.reporter
|
The side which reported the flow. Can be either 'SRC' or 'DEST'.
|
keyword
|
gcp.vpcflow.rtt.ms
|
Latency as measured (for TCP flows only) during the time interval. This is the time elapsed between sending a SEQ and receiving a corresponding ACK and it contains the network RTT as well as the application related delay.
|
long
|
host.architecture
|
Operating system architecture.
|
keyword
|
host.containerized
|
If the host is a container.
|
boolean
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
host.ip
|
Host ip addresses.
|
ip
|
host.mac
|
Host mac addresses.
|
keyword
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
host.os.build
|
OS build information.
|
keyword
|
host.os.codename
|
OS codename, if any.
|
keyword
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
input.type
|
Input type
|
keyword
|
log.file.path
|
Full path to the log file this event came from, including the file name. It should include the drive letter, when appropriate. If the event wasn't read from a log file, do not populate this field.
|
keyword
|
log.logger
|
The name of the logger inside an application. This is usually the name of the class which initialized the logger, or can be a custom name.
|
keyword
|
log.offset
|
Log offset
|
long
|
message
|
For log events the message field contains the log message, optimized for viewing in a log viewer. For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event. If multiple messages exist, they can be combined into one message.
|
match_only_text
|
network.bytes
|
Total bytes transferred in both directions. If
source.bytes and destination.bytes are known, network.bytes is their sum. |
long
|
network.community_id
|
A hash of source and destination IPs and ports, as well as the protocol used in a communication. This is a tool-agnostic standard to identify flows. Learn more at https://github.com/corelight/community-id-spec.
|
keyword
|
network.direction
|
Direction of the network traffic. When mapping events from a host-based monitoring context, populate this field from the host's point of view, using the values "ingress" or "egress". When mapping events from a network or perimeter-based monitoring context, populate this field from the point of view of the network perimeter, using the values "inbound", "outbound", "internal" or "external". Note that "internal" is not crossing perimeter boundaries, and is meant to describe communication between two hosts within the perimeter. Note also that "external" is meant to describe traffic between two hosts that are external to the perimeter. This could for example be useful for ISPs or VPN service providers.
|
keyword
|
network.iana_number
|
IANA Protocol Number (https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml). Standardized list of protocols. This aligns well with NetFlow and sFlow related logs which use the IANA Protocol Number.
|
keyword
|
network.name
|
Name given by operators to sections of their network.
|
keyword
|
network.packets
|
Total packets transferred in both directions. If
source.packets and destination.packets are known, network.packets is their sum. |
long
|
network.transport
|
Same as network.iana_number, but instead using the Keyword name of the transport layer (udp, tcp, ipv6-icmp, etc.) The field value must be normalized to lowercase for querying.
|
keyword
|
network.type
|
In the OSI Model this would be the Network Layer. ipv4, ipv6, ipsec, pim, etc The field value must be normalized to lowercase for querying.
|
keyword
|
related.hash
|
All the hashes seen on your event. Populating this field, then using it to search for hashes can help in situations where you're unsure what the hash algorithm is (and therefore which key name to search).
|
keyword
|
related.hosts
|
All hostnames or other host identifiers seen on your event. Example identifiers include FQDNs, domain names, workstation names, or aliases.
|
keyword
|
related.ip
|
All of the IPs seen on your event.
|
ip
|
related.user
|
All the user names or other user identifiers seen on the event.
|
keyword
|
rule.name
|
The name of the rule or signature generating the event.
|
keyword
|
source.address
|
Some event source addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the
.address field. Then it should be duplicated to .ip or .domain , depending on which one it is. |
keyword
|
source.as.number
|
Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
|
long
|
source.as.organization.name
|
Organization name.
|
keyword
|
source.as.organization.name.text
|
Multi-field of
source.as.organization.name . |
match_only_text
|
source.bytes
|
Bytes sent from the source to the destination.
|
long
|
source.domain
|
The domain name of the source system. This value may be a host name, a fully qualified domain name, or another host naming format. The value may derive from the original event or be added from enrichment.
|
keyword
|
source.geo.city_name
|
City name.
|
keyword
|
source.geo.continent_name
|
Name of the continent.
|
keyword
|
source.geo.country_iso_code
|
Country ISO code.
|
keyword
|
source.geo.country_name
|
Country name.
|
keyword
|
source.geo.location
|
Longitude and latitude.
|
geo_point
|
source.geo.name
|
User-defined description of a location, at the level of granularity they care about. Could be the name of their data centers, the floor number, if this describes a local physical entity, city names. Not typically used in automated geolocation.
|
keyword
|
source.geo.region_iso_code
|
Region ISO code.
|
keyword
|
source.geo.region_name
|
Region name.
|
keyword
|
source.ip
|
IP address of the source (IPv4 or IPv6).
|
ip
|
source.packets
|
Packets sent from the source to the destination.
|
long
|
source.port
|
Port of the source.
|
long
|
tags
|
List of keywords used to tag each event.
|
keyword
|
An example event for vpcflow
looks as following:
{
"@timestamp": "2019-06-14T03:50:10.845Z",
"agent": {
"ephemeral_id": "0b8165a2-0e25-4e9a-bb68-271697e0993f",
"id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.7.1"
},
"cloud": {
"availability_zone": "us-east1-b",
"instance": {
"name": "kibana"
},
"project": {
"id": "my-sample-project"
},
"provider": "gcp",
"region": "us-east1"
},
"data_stream": {
"dataset": "gcp.vpcflow",
"namespace": "ep",
"type": "logs"
},
"destination": {
"address": "10.139.99.242",
"domain": "elasticsearch",
"ip": "10.139.99.242",
"port": 9200
},
"ecs": {
"version": "8.8.0"
},
"elastic_agent": {
"id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
"snapshot": false,
"version": "8.7.1"
},
"event": {
"agent_id_status": "verified",
"category": [
"network"
],
"created": "2023-10-25T04:21:42.006Z",
"dataset": "gcp.vpcflow",
"end": "2019-06-14T03:49:51.821056075Z",
"id": "ut8lbrffooxz5",
"ingested": "2023-10-25T04:21:43Z",
"kind": "event",
"start": "2019-06-14T03:40:20.510622432Z",
"type": [
"connection"
]
},
"gcp": {
"destination": {
"instance": {
"project_id": "my-sample-project",
"region": "us-east1",
"zone": "us-east1-b"
},
"vpc": {
"project_id": "my-sample-project",
"subnetwork_name": "default",
"vpc_name": "default"
}
},
"source": {
"instance": {
"project_id": "my-sample-project",
"region": "us-east1",
"zone": "us-east1-b"
},
"vpc": {
"project_id": "my-sample-project",
"subnetwork_name": "default",
"vpc_name": "default"
}
},
"vpcflow": {
"reporter": "DEST",
"rtt": {
"ms": 201
}
}
},
"input": {
"type": "gcp-pubsub"
},
"log": {
"logger": "projects/my-sample-project/logs/compute.googleapis.com%2Fvpc_flows"
},
"network": {
"bytes": 11773,
"community_id": "1:FYaJFSEAKLcBCMFoT6sR5TMHf/s=",
"direction": "internal",
"iana_number": "6",
"name": "default",
"packets": 94,
"transport": "tcp",
"type": "ipv4"
},
"related": {
"ip": [
"67.43.156.13",
"10.139.99.242"
]
},
"source": {
"address": "67.43.156.13",
"as": {
"number": 35908
},
"bytes": 11773,
"domain": "kibana",
"geo": {
"continent_name": "Asia",
"country_iso_code": "BT",
"country_name": "Bhutan",
"location": {
"lat": 27.5,
"lon": 90.5
}
},
"ip": "67.43.156.13",
"packets": 94,
"port": 33576
},
"tags": [
"forwarded",
"gcp-vpcflow"
]
}
DNS
The dns
dataset collects queries that name servers resolve for your Virtual Private Cloud (VPC) networks, as well as queries from an external entity directly to a public zone.
Exported fields
Field | Description | Type |
---|---|---|
@timestamp
|
Event timestamp.
|
date
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
cloud.availability_zone
|
Availability zone in which this host is running.
|
keyword
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
cloud.region
|
Region in which this host is running.
|
keyword
|
container.id
|
Unique container id.
|
keyword
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
container.labels
|
Image labels.
|
object
|
container.name
|
Container name.
|
keyword
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
destination.address
|
Some event destination addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the
.address field. Then it should be duplicated to .ip or .domain , depending on which one it is. |
keyword
|
destination.ip
|
IP address of the destination (IPv4 or IPv6).
|
ip
|
dns.answers
|
An array containing an object for each answer section returned by the server. The main keys that should be present in these objects are defined by ECS. Records that have more information may contain more keys than what ECS defines. Not all DNS data sources give all details about DNS answers. At minimum, answer objects must contain the
data key. If more information is available, map as much of it to ECS as possible, and add any additional fields to the answer objects as custom fields. |
group
|
dns.answers.class
|
The class of DNS data contained in this resource record.
|
keyword
|
dns.answers.data
|
The data describing the resource. The meaning of this data depends on the type and class of the resource record.
|
keyword
|
dns.answers.name
|
The domain name to which this resource record pertains. If a chain of CNAME is being resolved, each answer's
name should be the one that corresponds with the answer's data . It should not simply be the original question.name repeated. |
keyword
|
dns.answers.ttl
|
The time interval in seconds that this resource record may be cached before it should be discarded. Zero values mean that the data should not be cached.
|
long
|
dns.answers.type
|
The type of data contained in this resource record.
|
keyword
|
dns.question.name
|
The name being queried. If the name field contains non-printable characters (below 32 or above 126), those characters should be represented as escaped base 10 integers (\DDD). Back slashes and quotes should be escaped. Tabs, carriage returns, and line feeds should be converted to \t, \r, and \n respectively.
|
keyword
|
dns.question.registered_domain
|
The highest registered domain, stripped of the subdomain. For example, the registered domain for "foo.example.com" is "example.com". This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last two labels will not work well for TLDs such as "co.uk".
|
keyword
|
dns.question.subdomain
|
The subdomain is all of the labels under the registered_domain. If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com", the subdomain field should contain "sub2.sub1", with no trailing period.
|
keyword
|
dns.question.top_level_domain
|
The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for example.com is "com". This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last label will not work well for effective TLDs such as "co.uk".
|
keyword
|
dns.question.type
|
The type of record being queried.
|
keyword
|
dns.resolved_ip
|
Array containing all IPs seen in
answers.data . The answers array can be difficult to use, because of the variety of data formats it can contain. Extracting all IP addresses seen in there to dns.resolved_ip makes it possible to index them as IP addresses, and makes them easier to visualize and query for. |
ip
|
dns.response_code
|
The DNS response code.
|
keyword
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
event.created
|
event.created contains the date/time when the event was first read by an agent, or by your pipeline. This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event. In most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source. In case the two timestamps are identical, @timestamp should be used.
|
date
|
event.dataset
|
Event dataset
|
constant_keyword
|
event.id
|
Unique ID to describe the event.
|
keyword
|
event.ingested
|
Timestamp when an event arrived in the central data store. This is different from
@timestamp , which is when the event originally occurred. It's also different from event.created , which is meant to capture the first time an agent saw the event. In normal conditions, assuming no tampering, the timestamps should chronologically look like this: @timestamp < event.created < event.ingested . |
date
|
event.kind
|
This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy.
event.kind gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. |
keyword
|
event.module
|
Event module
|
constant_keyword
|
event.original
|
Raw text message of entire event. Used to demonstrate log integrity or where the full log message (before splitting it up in multiple parts) may be required, e.g. for reindex. This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from
_source . If users wish to override this and index this field, please see Field data types in the Elasticsearch Reference . |
keyword
|
event.outcome
|
This is one of four ECS Categorization Fields, and indicates the lowest level in the ECS category hierarchy.
event.outcome simply denotes whether the event represents a success or a failure from the perspective of the entity that produced the event. Note that when a single transaction is described in multiple events, each event may populate different values of event.outcome , according to their perspective. Also note that in the case of a compound event (a single event that contains multiple logical events), this field should be populated with the value that best captures the overall success or failure from the perspective of the event producer. Further note that not all events will have an associated outcome. For example, this field is generally not populated for metric events, events with event.type:info , or any events for which an outcome does not make logical sense. |
keyword
|
gcp.dns.auth_answer
|
Authoritative answer.
|
boolean
|
gcp.dns.destination_ip
|
Destination IP address, only applicable for forwarding cases.
|
ip
|
gcp.dns.egress_error
|
Egress proxy error.
|
keyword
|
gcp.dns.flattened
|
Contains the full dns document as sent by GCP.
|
flattened
|
gcp.dns.protocol
|
Protocol TCP or UDP.
|
keyword
|
gcp.dns.query_name
|
DNS query name.
|
keyword
|
gcp.dns.query_type
|
DNS query type.
|
keyword
|
gcp.dns.rdata
|
DNS answer in presentation format, truncated to 260 bytes.
|
keyword
|
gcp.dns.response_code
|
Response code.
|
keyword
|
gcp.dns.server_latency
|
Server latency.
|
integer
|
gcp.dns.source_ip
|
Source IP address of the query.
|
ip
|
gcp.dns.source_network
|
Source network of the query.
|
keyword
|
gcp.dns.source_type
|
Type of source generating the DNS query: private-zone, public-zone, forwarding-zone, forwarding-policy, peering-zone, internal, external, internet
|
keyword
|
gcp.dns.target_type
|
Type of target resolving the DNS query: private-zone, public-zone, forwarding-zone, forwarding-policy, peering-zone, internal, external, internet
|
keyword
|
gcp.dns.vm_instance_id
|
Compute Engine VM instance ID, only applicable to queries initiated by Compute Engine VMs.
|
keyword
|
gcp.dns.vm_instance_name
|
Compute Engine VM instance name, only applicable to queries initiated by Compute Engine VMs.
|
keyword
|
gcp.dns.vm_project_id
|
Google Cloud project ID, only applicable to queries initiated by Compute Engine VMs.
|
keyword
|
gcp.dns.vm_zone_name
|
Google Cloud VM zone, only applicable to queries initiated by Compute Engine VMs.
|
keyword
|
host.architecture
|
Operating system architecture.
|
keyword
|
host.containerized
|
If the host is a container.
|
boolean
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
host.ip
|
Host ip addresses.
|
ip
|
host.mac
|
Host mac addresses.
|
keyword
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
host.os.build
|
OS build information.
|
keyword
|
host.os.codename
|
OS codename, if any.
|
keyword
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
input.type
|
Input type
|
keyword
|
log.level
|
Original log level of the log event. If the source of the event provides a log level or textual severity, this is the one that goes in
log.level . If your source doesn't specify one, you may put your event transport's severity here (e.g. Syslog severity). Some examples are warn , err , i , informational . |
keyword
|
log.logger
|
The name of the logger inside an application. This is usually the name of the class which initialized the logger, or can be a custom name.
|
keyword
|
log.offset
|
Log offset
|
long
|
network.iana_number
|
IANA Protocol Number (https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml). Standardized list of protocols. This aligns well with NetFlow and sFlow related logs which use the IANA Protocol Number.
|
keyword
|
network.protocol
|
In the OSI Model this would be the Application Layer protocol. For example,
http , dns , or ssh . The field value must be normalized to lowercase for querying. |
keyword
|
network.transport
|
Same as network.iana_number, but instead using the Keyword name of the transport layer (udp, tcp, ipv6-icmp, etc.) The field value must be normalized to lowercase for querying.
|
keyword
|
related.hosts
|
All hostnames or other host identifiers seen on your event. Example identifiers include FQDNs, domain names, workstation names, or aliases.
|
keyword
|
related.ip
|
All of the IPs seen on your event.
|
ip
|
source.address
|
Some event source addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the
.address field. Then it should be duplicated to .ip or .domain , depending on which one it is. |
keyword
|
source.ip
|
IP address of the source (IPv4 or IPv6).
|
ip
|
tags
|
List of keywords used to tag each event.
|
keyword
|
An example event for dns
looks as following:
{
"@timestamp": "2021-12-12T15:59:40.446Z",
"agent": {
"ephemeral_id": "fd6c4189-cbc6-493a-acfb-c9e7b2b7588c",
"id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.7.1"
},
"cloud": {
"project": {
"id": "key-reference-123456"
},
"provider": "gcp",
"region": "global"
},
"data_stream": {
"dataset": "gcp.dns",
"namespace": "ep",
"type": "logs"
},
"destination": {
"address": "216.239.32.106",
"ip": "216.239.32.106"
},
"dns": {
"answers": [
{
"class": "IN",
"data": "67.43.156.13",
"name": "asdf.gcp.example.com.",
"ttl": 300,
"type": "A"
}
],
"question": {
"name": "asdf.gcp.example.com",
"registered_domain": "example.com",
"subdomain": "asdf.gcp",
"top_level_domain": "com",
"type": "A"
},
"resolved_ip": [
"67.43.156.13"
],
"response_code": "NOERROR"
},
"ecs": {
"version": "8.8.0"
},
"elastic_agent": {
"id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
"snapshot": false,
"version": "8.7.1"
},
"event": {
"action": "dns-query",
"agent_id_status": "verified",
"category": "network",
"created": "2023-10-25T04:19:40.300Z",
"dataset": "gcp.dns",
"id": "zir4wud11tm",
"ingested": "2023-10-25T04:19:41Z",
"kind": "event",
"outcome": "success"
},
"gcp": {
"dns": {
"auth_answer": true,
"destination_ip": "216.239.32.106",
"protocol": "UDP",
"query_name": "asdf.gcp.example.com.",
"query_type": "A",
"response_code": "NOERROR",
"server_latency": 0,
"source_type": "internet",
"target_type": "public-zone"
}
},
"input": {
"type": "gcp-pubsub"
},
"log": {
"level": "INFO",
"logger": "projects/key-reference-123456/logs/dns.googleapis.com%2Fdns_queries"
},
"network": {
"iana_number": "17",
"protocol": "dns",
"transport": "udp"
},
"related": {
"hosts": [
"asdf.gcp.example.com"
],
"ip": [
"67.43.156.13",
"216.239.32.106"
]
},
"tags": [
"forwarded",
"gcp-dns"
]
}
Loadbalancing Logs
The loadbalancing_logs
dataset collects logs of the requests sent to and handled by GCP Load Balancers.
Exported fields
Field | Description | Type |
---|---|---|
@timestamp
|
Event timestamp.
|
date
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
cloud.availability_zone
|
Availability zone in which this host is running.
|
keyword
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
cloud.region
|
Region in which this host is running.
|
keyword
|
container.id
|
Unique container id.
|
keyword
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
container.labels
|
Image labels.
|
object
|
container.name
|
Container name.
|
keyword
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
destination.address
|
Some event destination addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the
.address field. Then it should be duplicated to .ip or .domain , depending on which one it is. |
keyword
|
destination.domain
|
The domain name of the destination system. This value may be a host name, a fully qualified domain name, or another host naming format. The value may derive from the original event or be added from enrichment.
|
keyword
|
destination.ip
|
IP address of the destination (IPv4 or IPv6).
|
ip
|
destination.nat.ip
|
Translated ip of destination based NAT sessions (e.g. internet to private DMZ) Typically used with load balancers, firewalls, or routers.
|
ip
|
destination.nat.port
|
Port the source session is translated to by NAT Device. Typically used with load balancers, firewalls, or routers.
|
long
|
destination.port
|
Port of the destination.
|
long
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
event.created
|
event.created contains the date/time when the event was first read by an agent, or by your pipeline. This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event. In most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source. In case the two timestamps are identical, @timestamp should be used.
|
date
|
event.dataset
|
Event dataset
|
constant_keyword
|
event.module
|
Event module
|
constant_keyword
|
gcp.load_balancer.backend_service_name
|
The backend service to which the load balancer is sending traffic
|
keyword
|
gcp.load_balancer.cache_hit
|
Whether or not an entity was served from cache (with or without validation).
|
boolean
|
gcp.load_balancer.cache_id
|
Indicates the location and cache instance that the cache response was served from. For example, a cache response served from a cache in Amsterdam would have a cacheId value of AMS-85e2bd4b, where AMS is the IATA code, and 85e2bd4b is an opaque identifier of the cache instance (because some Cloud CDN locations have multiple discrete caches).
|
keyword
|
gcp.load_balancer.cache_lookup
|
Whether or not a cache lookup was attempted.
|
boolean
|
gcp.load_balancer.forwarding_rule_name
|
The name of the forwarding rule
|
keyword
|
gcp.load_balancer.status_details
|
Explains why the load balancer returned the HTTP status that it did. See https://cloud.google.com/cdn/docs/cdn-logging-monitoring#statusdetail_http_success_messages for specific messages.
|
keyword
|
gcp.load_balancer.target_proxy_name
|
The target proxy name
|
keyword
|
gcp.load_balancer.url_map_name
|
The URL map name
|
keyword
|
host.architecture
|
Operating system architecture.
|
keyword
|
host.containerized
|
If the host is a container.
|
boolean
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
host.ip
|
Host ip addresses.
|
ip
|
host.mac
|
Host mac addresses.
|
keyword
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
host.os.build
|
OS build information.
|
keyword
|
host.os.codename
|
OS codename, if any.
|
keyword
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
http.request.bytes
|
Total size in bytes of the request (body and headers).
|
long
|
http.request.method
|
HTTP request method. The value should retain its casing from the original event. For example,
GET , get , and GeT are all considered valid values for this field. |
keyword
|
http.request.referrer
|
Referrer for this HTTP request.
|
keyword
|
http.response.bytes
|
Total size in bytes of the response (body and headers).
|
long
|
http.response.status_code
|
HTTP response status code.
|
long
|
http.version
|
HTTP version.
|
keyword
|
input.type
|
Input type
|
keyword
|
log.level
|
Original log level of the log event. If the source of the event provides a log level or textual severity, this is the one that goes in
log.level . If your source doesn't specify one, you may put your event transport's severity here (e.g. Syslog severity). Some examples are warn , err , i , informational . |
keyword
|
log.logger
|
The name of the logger inside an application. This is usually the name of the class which initialized the logger, or can be a custom name.
|
keyword
|
log.offset
|
Log offset
|
long
|
network.protocol
|
In the OSI Model this would be the Application Layer protocol. For example,
http , dns , or ssh . The field value must be normalized to lowercase for querying. |
keyword
|
related.hosts
|
All hostnames or other host identifiers seen on your event. Example identifiers include FQDNs, domain names, workstation names, or aliases.
|
keyword
|
related.ip
|
All of the IPs seen on your event.
|
ip
|
source.address
|
Some event source addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the
.address field. Then it should be duplicated to .ip or .domain , depending on which one it is. |
keyword
|
source.as.number
|
Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
|
long
|
source.as.organization.name
|
Organization name.
|
keyword
|
source.as.organization.name.text
|
Multi-field of
source.as.organization.name . |
match_only_text
|
source.geo.city_name
|
City name.
|
keyword
|
source.geo.continent_name
|
Name of the continent.
|
keyword
|
source.geo.country_iso_code
|
Country ISO code.
|
keyword
|
source.geo.country_name
|
Country name.
|
keyword
|
source.geo.location
|
Longitude and latitude.
|
geo_point
|
source.geo.region_iso_code
|
Region ISO code.
|
keyword
|
source.geo.region_name
|
Region name.
|
keyword
|
source.ip
|
IP address of the source (IPv4 or IPv6).
|
ip
|
source.port
|
Port of the source.
|
long
|
tags
|
List of keywords used to tag each event.
|
keyword
|
url.domain
|
Domain of the url, such as "www.elastic.co". In some cases a URL may refer to an IP and/or port directly, without a domain name. In this case, the IP address would go to the
domain field. If the URL contains a literal IPv6 address enclosed by [ and ] (IETF RFC 2732), the [ and ] characters should also be captured in the domain field. |
keyword
|
url.extension
|
The field contains the file extension from the original request url, excluding the leading dot. The file extension is only set if it exists, as not every url has a file extension. The leading period must not be included. For example, the value must be "png", not ".png". Note that when the file name has multiple extensions (example.tar.gz), only the last one should be captured ("gz", not "tar.gz").
|
keyword
|
url.original
|
Unmodified original url as seen in the event source. Note that in network monitoring, the observed URL may be a full URL, whereas in access logs, the URL is often just represented as a path. This field is meant to represent the URL as it was observed, complete or not.
|
wildcard
|
url.original.text
|
Multi-field of
url.original . |
match_only_text
|
url.path
|
Path of the request, such as "/search".
|
wildcard
|
url.port
|
Port of the request, such as 443.
|
long
|
url.query
|
The query field describes the query string of the request, such as "q=elasticsearch". The
? is excluded from the query string. If a URL contains no ? , there is no query field. If there is a ? but no query, the query field exists with an empty string. The exists query can be used to differentiate between the two cases. |
keyword
|
url.scheme
|
Scheme of the request, such as "https". Note: The
: is not part of the scheme. |
keyword
|
user_agent.device.name
|
Name of the device.
|
keyword
|
user_agent.name
|
Name of the user agent.
|
keyword
|
user_agent.original
|
Unparsed user_agent string.
|
keyword
|
user_agent.original.text
|
Multi-field of
user_agent.original . |
match_only_text
|
user_agent.os.full
|
Operating system name, including the version or code name.
|
keyword
|
user_agent.os.full.text
|
Multi-field of
user_agent.os.full . |
match_only_text
|
user_agent.os.name
|
Operating system name, without the version.
|
keyword
|
user_agent.os.name.text
|
Multi-field of
user_agent.os.name . |
match_only_text
|
user_agent.os.version
|
Operating system version as a raw string.
|
keyword
|
user_agent.version
|
Version of the user agent.
|
keyword
|
An example event for loadbalancing
looks as following:
{
"@timestamp": "2020-06-08T23:41:30.078Z",
"agent": {
"ephemeral_id": "f4dde373-2ff7-464b-afdb-da94763f219b",
"id": "5d3eee86-91a9-4afa-af92-c6b79bd866c0",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.6.0"
},
"cloud": {
"project": {
"id": "PROJECT_ID"
},
"region": "global"
},
"data_stream": {
"dataset": "gcp.loadbalancing_logs",
"namespace": "ep",
"type": "logs"
},
"destination": {
"address": "81.2.69.193",
"ip": "81.2.69.193",
"nat": {
"ip": "10.5.3.1",
"port": 9090
},
"port": 8080
},
"ecs": {
"version": "8.8.0"
},
"elastic_agent": {
"id": "5d3eee86-91a9-4afa-af92-c6b79bd866c0",
"snapshot": true,
"version": "8.6.0"
},
"event": {
"agent_id_status": "verified",
"category": "network",
"created": "2020-06-08T23:41:30.588Z",
"dataset": "gcp.loadbalancing_logs",
"id": "1oek5rg3l3fxj7",
"ingested": "2023-01-13T15:02:22Z",
"kind": "event",
"type": "info"
},
"gcp": {
"load_balancer": {
"backend_service_name": "",
"cache_hit": true,
"cache_id": "SFO-fbae48ad",
"cache_lookup": true,
"forwarding_rule_name": "FORWARDING_RULE_NAME",
"status_details": "response_from_cache",
"target_proxy_name": "TARGET_PROXY_NAME",
"url_map_name": "URL_MAP_NAME"
}
},
"http": {
"request": {
"bytes": 577,
"method": "GET",
"referrer": "https://developer.mozilla.org/en-US/docs/Web/JavaScript"
},
"response": {
"bytes": 157,
"status_code": 304
},
"version": "2.0"
},
"input": {
"type": "gcp-pubsub"
},
"log": {
"level": "INFO",
"logger": "projects/PROJECT_ID/logs/requests"
},
"network": {
"protocol": "http"
},
"related": {
"ip": [
"89.160.20.156",
"81.2.69.193",
"10.5.3.1"
]
},
"source": {
"address": "89.160.20.156",
"as": {
"number": 29518,
"organization": {
"name": "Bredband2 AB"
}
},
"geo": {
"city_name": "Linköping",
"continent_name": "Europe",
"country_iso_code": "SE",
"country_name": "Sweden",
"location": {
"lat": 58.4167,
"lon": 15.6167
},
"region_iso_code": "SE-E",
"region_name": "Östergötland County"
},
"ip": "89.160.20.156",
"port": 9989
},
"tags": [
"forwarded",
"gcp-loadbalancing_logs"
],
"url": {
"domain": "81.2.69.193",
"extension": "jpg",
"original": "http://81.2.69.193:8080/static/us/three-cats.jpg",
"path": "/static/us/three-cats.jpg",
"port": 8080,
"scheme": "http"
},
"user_agent": {
"device": {
"name": "Mac"
},
"name": "Chrome",
"original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36",
"os": {
"full": "Mac OS X 10.14.6",
"name": "Mac OS X",
"version": "10.14.6"
},
"version": "83.0.4103.61"
}
}
Metrics
Billing
The billing
dataset collects GCP Billing information from Google Cloud BigQuery daily cost detail table.
Exported fields
Field | Description | Type |
---|---|---|
@timestamp
|
Event timestamp.
|
date
|
cloud
|
Fields related to the cloud or infrastructure the events are coming from.
|
group
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
cloud.account.name
|
The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name.
|
keyword
|
cloud.availability_zone
|
Availability zone in which this host, resource, or service is located.
|
keyword
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
cloud.region
|
Region in which this host, resource, or service is located.
|
keyword
|
container.id
|
Unique container id.
|
keyword
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
container.labels
|
Image labels.
|
object
|
container.name
|
Container name.
|
keyword
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
error
|
These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error.
|
group
|
error.message
|
Error message.
|
match_only_text
|
event.dataset
|
Event dataset
|
constant_keyword
|
event.module
|
Event module
|
constant_keyword
|
gcp.billing.billing_account_id
|
Project Billing Account ID.
|
keyword
|
gcp.billing.cost_type
|
Cost types include regular, tax, adjustment, and rounding_error.
|
keyword
|
gcp.billing.effective_price
|
The charged price for usage of the Google Cloud SKUs and SKU tiers. Reflects contract pricing if applicable, otherwise, it's the list price.
|
float
|
gcp.billing.invoice_month
|
Billing report month.
|
keyword
|
gcp.billing.project_id
|
Project ID of the billing report belongs to.
|
keyword
|
gcp.billing.project_name
|
Project Name of the billing report belongs to.
|
keyword
|
gcp.billing.service_description
|
The Google Cloud service that reported the Cloud Billing data.
|
keyword
|
gcp.billing.service_id
|
The ID of the service that the usage is associated with.
|
keyword
|
gcp.billing.sku_description
|
A description of the resource type used by the service. For example, a resource type for Cloud Storage is Standard Storage US.
|
keyword
|
gcp.billing.sku_id
|
The ID of the resource used by the service.
|
keyword
|
gcp.billing.tags.key
|
keyword
|
|
gcp.billing.tags.value
|
keyword
|
|
gcp.billing.total
|
Total billing amount.
|
float
|
host.architecture
|
Operating system architecture.
|
keyword
|
host.containerized
|
If the host is a container.
|
boolean
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
host.ip
|
Host ip addresses.
|
ip
|
host.mac
|
Host mac addresses.
|
keyword
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
host.os.build
|
OS build information.
|
keyword
|
host.os.codename
|
OS codename, if any.
|
keyword
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
service.type
|
The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch,
service.type would be elasticsearch . |
keyword
|
An example event for billing
looks as following:
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"cloud": {
"account": {
"id": "01475F-5B1080-1137E7"
},
"project": {
"id": "elastic-bi",
"name": "elastic-containerlib-prod"
},
"provider": "gcp"
},
"event": {
"dataset": "gcp.billing",
"duration": 115000,
"module": "gcp"
},
"gcp": {
"billing": {
"billing_account_id": "01475F-5B1080-1137E7",
"cost_type": "regular",
"invoice_month": "202106",
"project_id": "containerlib-prod-12763",
"project_name": "elastic-containerlib-prod",
"total": 4717.170681,
"sku_id": "0D56-2F80-52A5",
"service_id": "6F81-5844-456A",
"sku_description": "Network Inter Region Ingress from Jakarta to Americas",
"service_description": "Compute Engine",
"effective_price": 0.00292353,
"tags": [
{
"key": "stage",
"value": "prod"
},
{
"key": "size",
"value": "standard"
}
]
}
},
"metricset": {
"name": "billing",
"period": 10000
},
"service": {
"type": "gcp"
}
}
Compute
The compute
dataset is designed to fetch metrics for Compute Engine Virtual Machines in Google Cloud Platform.
Exported fields
Field | Description | Type | Metric Type |
---|---|---|---|
@timestamp
|
Event timestamp.
|
date
|
|
agent.id
|
Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.
|
keyword
|
|
cloud
|
Fields related to the cloud or infrastructure the events are coming from.
|
group
|
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
|
cloud.account.name
|
The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name.
|
keyword
|
|
cloud.availability_zone
|
Availability zone in which this host, resource, or service is located.
|
keyword
|
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
|
cloud.region
|
Region in which this host, resource, or service is located.
|
keyword
|
|
container.id
|
Unique container id.
|
keyword
|
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
|
container.labels
|
Image labels.
|
object
|
|
container.name
|
Container name.
|
keyword
|
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
|
error
|
These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error.
|
group
|
|
error.message
|
Error message.
|
match_only_text
|
|
event.dataset
|
Event dataset
|
constant_keyword
|
|
event.module
|
Event module
|
constant_keyword
|
|
gcp.compute.firewall.dropped.bytes
|
Delta of incoming bytes dropped by the firewall
|
long
|
gauge
|
gcp.compute.firewall.dropped_packets_count.value
|
Delta of incoming packets dropped by the firewall
|
long
|
gauge
|
gcp.compute.instance.cpu.reserved_cores.value
|
Number of cores reserved on the host of the instance
|
double
|
gauge
|
gcp.compute.instance.cpu.usage.pct
|
The fraction of the allocated CPU that is currently in use on the instance
|
double
|
gauge
|
gcp.compute.instance.cpu.usage_time.sec
|
Delta of usage for all cores in seconds
|
double
|
gauge
|
gcp.compute.instance.disk.read.bytes
|
Delta of count of bytes read from disk
|
long
|
gauge
|
gcp.compute.instance.disk.read_ops_count.value
|
Delta of count of disk read IO operations
|
long
|
gauge
|
gcp.compute.instance.disk.write.bytes
|
Delta of count of bytes written to disk
|
long
|
gauge
|
gcp.compute.instance.disk.write_ops_count.value
|
Delta of count of disk write IO operations
|
long
|
gauge
|
gcp.compute.instance.memory.balloon.ram_size.value
|
The total amount of memory in the VM. This metric is only available for VMs that belong to the e2 family.
|
long
|
gauge
|
gcp.compute.instance.memory.balloon.ram_used.value
|
Memory currently used in the VM. This metric is only available for VMs that belong to the e2 family.
|
long
|
gauge
|
gcp.compute.instance.memory.balloon.swap_in.bytes
|
Delta of the amount of memory read into the guest from its own swap space. This metric is only available for VMs that belong to the e2 family.
|
long
|
gauge
|
gcp.compute.instance.memory.balloon.swap_out.bytes
|
Delta of the amount of memory written from the guest to its own swap space. This metric is only available for VMs that belong to the e2 family.
|
long
|
gauge
|
gcp.compute.instance.network.egress.bytes
|
Delta of count of bytes sent over the network
|
long
|
gauge
|
gcp.compute.instance.network.egress.packets.count
|
Delta of count of packets sent over the network
|
long
|
gauge
|
gcp.compute.instance.network.ingress.bytes
|
Delta of count of bytes received from the network
|
long
|
gauge
|
gcp.compute.instance.network.ingress.packets.count
|
Delta of count of packets received from the network
|
long
|
gauge
|
gcp.compute.instance.uptime.sec
|
Delta of number of seconds the VM has been running.
|
long
|
gauge
|
gcp.compute.instance.uptime_total.sec
|
Elapsed time since the VM was started, in seconds. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
|
long
|
gauge
|
gcp.labels.metadata.*
|
object
|
||
gcp.labels.metrics.*
|
object
|
||
gcp.labels.resource.*
|
object
|
||
gcp.labels.system.*
|
object
|
||
gcp.labels.user.*
|
object
|
||
gcp.labels_fingerprint
|
Hashed value of the labels field.
|
keyword
|
|
gcp.metrics.*.*.*.*
|
Metrics that returned from Google Cloud API query.
|
object
|
|
host.architecture
|
Operating system architecture.
|
keyword
|
|
host.containerized
|
If the host is a container.
|
boolean
|
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
|
host.ip
|
Host ip addresses.
|
ip
|
|
host.mac
|
Host mac addresses.
|
keyword
|
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
|
host.os.build
|
OS build information.
|
keyword
|
|
host.os.codename
|
OS codename, if any.
|
keyword
|
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
|
service.type
|
The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch,
service.type would be elasticsearch . |
keyword
|
An example event for compute
looks as following:
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"cloud": {
"account": {
"id": "elastic-obs-integrations-dev",
"name": "elastic-obs-integrations-dev"
},
"instance": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"machine": {
"type": "e2-medium"
},
"provider": "gcp",
"availability_zone": "us-central1-c",
"region": "us-central1"
},
"event": {
"dataset": "gcp.compute",
"duration": 115000,
"module": "gcp"
},
"gcp": {
"compute": {
"firewall": {
"dropped": {
"bytes": 421
},
"dropped_packets_count": {
"value": 4
}
},
"instance": {
"cpu": {
"reserved_cores": {
"value": 1
},
"usage": {
"pct": 0.07259952346383708
},
"usage_time": {
"sec": 4.355971407830225
}
},
"memory": {
"balloon": {
"ram_size": {
"value": 4128378880
},
"ram_used": {
"value": 2190848000
},
"swap_in": {
"bytes": 0
},
"swap_out": {
"bytes": 0
}
}
},
"uptime": {
"sec": 60.00000000000091
}
}
},
"labels": {
"user": {
"goog-gke-node": ""
}
}
},
"host": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"metricset": {
"name": "compute",
"period": 10000
},
"service": {
"type": "gcp"
}
}
Dataproc
The dataproc
dataset is designed to fetch metrics from Dataproc in Google Cloud Platform.
Exported fields
Field | Description | Type | Metric Type |
---|---|---|---|
@timestamp
|
Event timestamp.
|
date
|
|
agent.id
|
Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.
|
keyword
|
|
cloud
|
Fields related to the cloud or infrastructure the events are coming from.
|
group
|
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
|
cloud.account.name
|
The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name.
|
keyword
|
|
cloud.availability_zone
|
Availability zone in which this host, resource, or service is located.
|
keyword
|
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
|
cloud.region
|
Region in which this host, resource, or service is located.
|
keyword
|
|
container.id
|
Unique container id.
|
keyword
|
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
|
container.labels
|
Image labels.
|
object
|
|
container.name
|
Container name.
|
keyword
|
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
|
error
|
These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error.
|
group
|
|
error.message
|
Error message.
|
match_only_text
|
|
event.dataset
|
Event dataset
|
constant_keyword
|
|
event.module
|
Event module
|
constant_keyword
|
|
gcp.dataproc.batch.spark.executors.count
|
Indicates the number of Batch Spark executors.
|
long
|
gauge
|
gcp.dataproc.cluster.hdfs.datanodes.count
|
Indicates the number of HDFS DataNodes that are running inside a cluster.
|
long
|
gauge
|
gcp.dataproc.cluster.hdfs.storage_capacity.value
|
Indicates capacity of HDFS system running on cluster in GB.
|
double
|
gauge
|
gcp.dataproc.cluster.hdfs.storage_utilization.value
|
The percentage of HDFS storage currently used.
|
double
|
gauge
|
gcp.dataproc.cluster.hdfs.unhealthy_blocks.count
|
Indicates the number of unhealthy blocks inside the cluster.
|
long
|
gauge
|
gcp.dataproc.cluster.job.completion_time.value
|
The time jobs took to complete from the time the user submits a job to the time Dataproc reports it is completed.
|
object
|
|
gcp.dataproc.cluster.job.duration.value
|
The time jobs have spent in a given state.
|
object
|
|
gcp.dataproc.cluster.job.failed.count
|
Indicates the delta of the number of jobs that have failed on a cluster.
|
long
|
gauge
|
gcp.dataproc.cluster.job.running.count
|
Indicates the number of jobs that are running on a cluster.
|
long
|
gauge
|
gcp.dataproc.cluster.job.submitted.count
|
Indicates the delta of the number of jobs that have been submitted to a cluster.
|
long
|
gauge
|
gcp.dataproc.cluster.operation.completion_time.value
|
The time operations took to complete from the time the user submits a operation to the time Dataproc reports it is completed.
|
object
|
|
gcp.dataproc.cluster.operation.duration.value
|
The time operations have spent in a given state.
|
object
|
|
gcp.dataproc.cluster.operation.failed.count
|
Indicates the delta of the number of operations that have failed on a cluster.
|
long
|
gauge
|
gcp.dataproc.cluster.operation.running.count
|
Indicates the number of operations that are running on a cluster.
|
long
|
gauge
|
gcp.dataproc.cluster.operation.submitted.count
|
Indicates the delta of the number of operations that have been submitted to a cluster.
|
long
|
gauge
|
gcp.dataproc.cluster.yarn.allocated_memory_percentage.value
|
The percentage of YARN memory is allocated.
|
double
|
gauge
|
gcp.dataproc.cluster.yarn.apps.count
|
Indicates the number of active YARN applications.
|
long
|
gauge
|
gcp.dataproc.cluster.yarn.containers.count
|
Indicates the number of YARN containers.
|
long
|
gauge
|
gcp.dataproc.cluster.yarn.memory_size.value
|
Indicates the YARN memory size in GB.
|
double
|
gauge
|
gcp.dataproc.cluster.yarn.nodemanagers.count
|
Indicates the number of YARN NodeManagers running inside cluster.
|
long
|
gauge
|
gcp.dataproc.cluster.yarn.pending_memory_size.value
|
The current memory request, in GB, that is pending to be fulfilled by the scheduler.
|
double
|
gauge
|
gcp.dataproc.cluster.yarn.virtual_cores.count
|
Indicates the number of virtual cores in YARN.
|
long
|
gauge
|
gcp.labels.metadata.*
|
object
|
||
gcp.labels.metrics.*
|
object
|
||
gcp.labels.resource.*
|
object
|
||
gcp.labels.system.*
|
object
|
||
gcp.labels.user.*
|
object
|
||
gcp.labels_fingerprint
|
Hashed value of the labels field.
|
keyword
|
|
gcp.metrics.*.*.*.*
|
Metrics that returned from Google Cloud API query.
|
object
|
|
host.architecture
|
Operating system architecture.
|
keyword
|
|
host.containerized
|
If the host is a container.
|
boolean
|
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
|
host.ip
|
Host ip addresses.
|
ip
|
|
host.mac
|
Host mac addresses.
|
keyword
|
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
|
host.os.build
|
OS build information.
|
keyword
|
|
host.os.codename
|
OS codename, if any.
|
keyword
|
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
|
service.type
|
The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch,
service.type would be elasticsearch . |
keyword
|
An example event for dataproc
looks as following:
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"cloud": {
"account": {
"id": "elastic-obs-integrations-dev",
"name": "elastic-obs-integrations-dev"
},
"instance": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"machine": {
"type": "e2-medium"
},
"provider": "gcp",
"availability_zone": "us-central1-c",
"region": "us-central1"
},
"event": {
"dataset": "gcp.dataproc",
"duration": 115000,
"module": "gcp"
},
"gcp": {
"dataproc": {
"cluster": {
"hdfs": {
"datanodes": {
"count": 15
}
}
}
},
"labels": {
"user": {
"goog-gke-node": ""
}
}
},
"host": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"metricset": {
"name": "dataproc",
"period": 10000
},
"service": {
"type": "gcp"
}
}
Firestore
The firestore
dataset fetches metrics from Firestore in Google Cloud Platform.
Exported fields
Field | Description | Type | Metric Type |
---|---|---|---|
@timestamp
|
Event timestamp.
|
date
|
|
cloud
|
Fields related to the cloud or infrastructure the events are coming from.
|
group
|
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
|
cloud.account.name
|
The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name.
|
keyword
|
|
cloud.availability_zone
|
Availability zone in which this host, resource, or service is located.
|
keyword
|
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
|
cloud.region
|
Region in which this host, resource, or service is located.
|
keyword
|
|
container.id
|
Unique container id.
|
keyword
|
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
|
container.labels
|
Image labels.
|
object
|
|
container.name
|
Container name.
|
keyword
|
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
|
error
|
These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error.
|
group
|
|
error.message
|
Error message.
|
match_only_text
|
|
event.dataset
|
Event dataset
|
constant_keyword
|
|
event.module
|
Event module
|
constant_keyword
|
|
gcp.firestore.document.delete.count
|
Delta of the number of successful document deletes.
|
long
|
gauge
|
gcp.firestore.document.read.count
|
Delta of the number of successful document reads from queries or lookups.
|
long
|
gauge
|
gcp.firestore.document.write.count
|
Delta of the number of successful document writes.
|
long
|
gauge
|
gcp.labels.metadata.*
|
object
|
||
gcp.labels.metrics.*
|
object
|
||
gcp.labels.resource.*
|
object
|
||
gcp.labels.system.*
|
object
|
||
gcp.labels.user.*
|
object
|
||
gcp.metrics.*.*.*.*
|
Metrics that returned from Google Cloud API query.
|
object
|
|
host.architecture
|
Operating system architecture.
|
keyword
|
|
host.containerized
|
If the host is a container.
|
boolean
|
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
|
host.ip
|
Host ip addresses.
|
ip
|
|
host.mac
|
Host mac addresses.
|
keyword
|
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
|
host.os.build
|
OS build information.
|
keyword
|
|
host.os.codename
|
OS codename, if any.
|
keyword
|
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
|
service.type
|
The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch,
service.type would be elasticsearch . |
keyword
|
An example event for firestore
looks as following:
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"cloud": {
"account": {
"id": "elastic-obs-integrations-dev",
"name": "elastic-obs-integrations-dev"
},
"instance": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"machine": {
"type": "e2-medium"
},
"provider": "gcp",
"availability_zone": "us-central1-c",
"region": "us-central1"
},
"event": {
"dataset": "gcp.firestore",
"duration": 115000,
"module": "gcp"
},
"gcp": {
"firestore": {
"document": {
"delete": {
"count": 3
},
"read": {
"count": 10
},
"write": {
"count": 1
}
}
},
"labels": {
"user": {
"goog-gke-node": ""
}
}
},
"host": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"metricset": {
"name": "firestore",
"period": 10000
},
"service": {
"type": "gcp"
}
}
GKE
The gke
dataset is designed to fetch metrics from GKE in Google Cloud Platform.
Exported fields
Field | Description | Type | Metric Type |
---|---|---|---|
@timestamp
|
Event timestamp.
|
date
|
|
agent.id
|
Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.
|
keyword
|
|
cloud
|
Fields related to the cloud or infrastructure the events are coming from.
|
group
|
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
|
cloud.account.name
|
The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name.
|
keyword
|
|
cloud.availability_zone
|
Availability zone in which this host, resource, or service is located.
|
keyword
|
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
|
cloud.region
|
Region in which this host, resource, or service is located.
|
keyword
|
|
container.id
|
Unique container id.
|
keyword
|
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
|
container.labels
|
Image labels.
|
object
|
|
container.name
|
Container name.
|
keyword
|
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
|
error
|
These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error.
|
group
|
|
error.message
|
Error message.
|
match_only_text
|
|
event.dataset
|
Event dataset
|
constant_keyword
|
|
event.module
|
Event module
|
constant_keyword
|
|
gcp.gke.container.cpu.core_usage_time.sec
|
Cumulative CPU usage on all cores used by the container in seconds. Sampled every 60 seconds.
|
double
|
counter
|
gcp.gke.container.cpu.limit_cores.value
|
CPU cores limit of the container. Sampled every 60 seconds.
|
double
|
gauge
|
gcp.gke.container.cpu.limit_utilization.pct
|
The fraction of the CPU limit that is currently in use on the instance. This value cannot exceed 1 as usage cannot exceed the limit. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds.
|
double
|
gauge
|
gcp.gke.container.cpu.request_cores.value
|
Number of CPU cores requested by the container. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
|
double
|
gauge
|
gcp.gke.container.cpu.request_utilization.pct
|
The fraction of the requested CPU that is currently in use on the instance. This value can be greater than 1 as usage can exceed the request. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds.
|
double
|
gauge
|
gcp.gke.container.ephemeral_storage.limit.bytes
|
Local ephemeral storage limit in bytes. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.container.ephemeral_storage.request.bytes
|
Local ephemeral storage request in bytes. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.container.ephemeral_storage.used.bytes
|
Local ephemeral storage usage in bytes. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.container.memory.limit.bytes
|
Memory limit of the container in bytes. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.container.memory.limit_utilization.pct
|
The fraction of the memory limit that is currently in use on the instance. This value cannot exceed 1 as usage cannot exceed the limit. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
|
double
|
gauge
|
gcp.gke.container.memory.page_fault.count
|
Number of page faults, broken down by type, major and minor.
|
long
|
counter
|
gcp.gke.container.memory.request.bytes
|
Memory request of the container in bytes. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
|
long
|
gauge
|
gcp.gke.container.memory.request_utilization.pct
|
The fraction of the requested memory that is currently in use on the instance. This value can be greater than 1 as usage can exceed the request. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds.
|
double
|
gauge
|
gcp.gke.container.memory.used.bytes
|
Memory usage in bytes. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.container.restart.count
|
Number of times the container has restarted. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
|
long
|
counter
|
gcp.gke.container.uptime.sec
|
Time in seconds that the container has been running. Sampled every 60 seconds.
|
double
|
gauge
|
gcp.gke.node.cpu.allocatable_cores.value
|
Number of allocatable CPU cores on the node. Sampled every 60 seconds.
|
double
|
gauge
|
gcp.gke.node.cpu.allocatable_utilization.pct
|
The fraction of the allocatable CPU that is currently in use on the instance. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds.
|
double
|
gauge
|
gcp.gke.node.cpu.core_usage_time.sec
|
Cumulative CPU usage on all cores used on the node in seconds. Sampled every 60 seconds.
|
double
|
counter
|
gcp.gke.node.cpu.total_cores.value
|
Total number of CPU cores on the node. Sampled every 60 seconds.
|
double
|
gauge
|
gcp.gke.node.ephemeral_storage.allocatable.bytes
|
Local ephemeral storage bytes allocatable on the node. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node.ephemeral_storage.inodes_free.value
|
Free number of inodes on local ephemeral storage. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node.ephemeral_storage.inodes_total.value
|
Total number of inodes on local ephemeral storage. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node.ephemeral_storage.total.bytes
|
Total ephemeral storage bytes on the node. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node.ephemeral_storage.used.bytes
|
Local ephemeral storage bytes used by the node. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node.memory.allocatable.bytes
|
Cumulative memory bytes used by the node. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node.memory.allocatable_utilization.pct
|
The fraction of the allocatable memory that is currently in use on the instance. This value cannot exceed 1 as usage cannot exceed allocatable memory bytes. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
|
double
|
gauge
|
gcp.gke.node.memory.total.bytes
|
Number of bytes of memory allocatable on the node. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node.memory.used.bytes
|
Cumulative memory bytes used by the node. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node.network.received_bytes.count
|
Cumulative number of bytes received by the node over the network. Sampled every 60 seconds.
|
long
|
counter
|
gcp.gke.node.network.sent_bytes.count
|
Cumulative number of bytes transmitted by the node over the network. Sampled every 60 seconds.
|
long
|
counter
|
gcp.gke.node.pid_limit.value
|
The max PID of OS on the node. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node.pid_used.value
|
The number of running process in the OS on the node. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node_daemon.cpu.core_usage_time.sec
|
Cumulative CPU usage on all cores used by the node level system daemon in seconds. Sampled every 60 seconds.
|
double
|
counter
|
gcp.gke.node_daemon.memory.used.bytes
|
Memory usage by the system daemon in bytes. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.pod.network.received.bytes
|
Cumulative number of bytes received by the pod over the network. Sampled every 60 seconds.
|
long
|
counter
|
gcp.gke.pod.network.sent.bytes
|
Cumulative number of bytes transmitted by the pod over the network. Sampled every 60 seconds.
|
long
|
counter
|
gcp.gke.pod.volume.total.bytes
|
Total number of disk bytes available to the pod. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
|
long
|
gauge
|
gcp.gke.pod.volume.used.bytes
|
Number of disk bytes used by the pod. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.pod.volume.utilization.pct
|
The fraction of the volume that is currently being used by the instance. This value cannot be greater than 1 as usage cannot exceed the total available volume space. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
|
double
|
gauge
|
gcp.labels.metadata.*
|
object
|
||
gcp.labels.metrics.*
|
object
|
||
gcp.labels.resource.*
|
object
|
||
gcp.labels.system.*
|
object
|
||
gcp.labels.user.*
|
object
|
||
gcp.labels_fingerprint
|
Hashed value of the labels field.
|
keyword
|
|
gcp.metrics.*.*.*.*
|
Metrics that returned from Google Cloud API query.
|
object
|
|
host.architecture
|
Operating system architecture.
|
keyword
|
|
host.containerized
|
If the host is a container.
|
boolean
|
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
|
host.ip
|
Host ip addresses.
|
ip
|
|
host.mac
|
Host mac addresses.
|
keyword
|
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
|
host.os.build
|
OS build information.
|
keyword
|
|
host.os.codename
|
OS codename, if any.
|
keyword
|
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
|
service.type
|
The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch,
service.type would be elasticsearch . |
keyword
|
An example event for gke
looks as following:
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"cloud": {
"account": {
"id": "elastic-obs-integrations-dev",
"name": "elastic-obs-integrations-dev"
},
"instance": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"machine": {
"type": "e2-medium"
},
"provider": "gcp",
"availability_zone": "us-central1-c",
"region": "us-central1"
},
"event": {
"dataset": "gcp.gke",
"duration": 115000,
"module": "gcp"
},
"gcp": {
"gke": {
"container": {
"cpu": {
"core_usage_time": {
"sec": 15
}
}
}
},
"labels": {
"user": {
"goog-gke-node": ""
}
}
},
"host": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"metricset": {
"name": "gke",
"period": 10000
},
"service": {
"type": "gcp"
}
}
Loadbalancing Metrics
The loadbalancing_metrics
dataset is designed to fetch HTTPS, HTTP, and Layer 3 metrics from Load Balancing in Google Cloud Platform.
Exported fields
Field | Description | Type | Metric Type |
---|---|---|---|
@timestamp
|
Event timestamp.
|
date
|
|
agent.id
|
Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.
|
keyword
|
|
cloud
|
Fields related to the cloud or infrastructure the events are coming from.
|
group
|
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
|
cloud.account.name
|
The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name.
|
keyword
|
|
cloud.availability_zone
|
Availability zone in which this host, resource, or service is located.
|
keyword
|
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
|
cloud.region
|
Region in which this host, resource, or service is located.
|
keyword
|
|
container.id
|
Unique container id.
|
keyword
|
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
|
container.labels
|
Image labels.
|
object
|
|
container.name
|
Container name.
|
keyword
|
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
|
error
|
These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error.
|
group
|
|
error.message
|
Error message.
|
match_only_text
|
|
event.dataset
|
Event dataset
|
constant_keyword
|
|
event.module
|
Event module
|
constant_keyword
|
|
gcp.labels.metadata.*
|
object
|
||
gcp.labels.metrics.*
|
object
|
||
gcp.labels.resource.*
|
object
|
||
gcp.labels.system.*
|
object
|
||
gcp.labels.user.*
|
object
|
||
gcp.labels_fingerprint
|
Hashed value of the labels field.
|
keyword
|
|
gcp.loadbalancing_metrics.https.backend_latencies.value
|
A distribution of the latency calculated from when the request was sent by the proxy to the backend until the proxy received from the backend the last byte of response.
|
object
|
|
gcp.loadbalancing_metrics.https.backend_request.bytes
|
Delta of the number of bytes sent as requests from HTTP/S load balancer to backends.
|
long
|
gauge
|
gcp.loadbalancing_metrics.https.backend_request.count
|
Delta of the number of requests served by backends of HTTP/S load balancer.
|
long
|
gauge
|
gcp.loadbalancing_metrics.https.backend_response.bytes
|
Delta of the number of bytes sent as responses from backends (or cache) to external HTTP(S) load balancer.
|
long
|
gauge
|
gcp.loadbalancing_metrics.https.external.regional.backend_latencies.value
|
A distribution of the latency calculated from when the request was sent by the proxy to the backend until the proxy received from the backend the last byte of response.
|
object
|
|
gcp.loadbalancing_metrics.https.external.regional.total_latencies.value
|
A distribution of the latency calculated from when the request was received by the proxy until the proxy got ACK from client on last response byte.
|
object
|
|
gcp.loadbalancing_metrics.https.frontend_tcp_rtt.value
|
A distribution of the RTT measured for each connection between client and proxy.
|
object
|
|
gcp.loadbalancing_metrics.https.internal.backend_latencies.value
|
A distribution of the latency calculated from when the request was sent by the internal HTTP/S load balancer proxy to the backend until the proxy received from the backend the last byte of response.
|
object
|
|
gcp.loadbalancing_metrics.https.internal.total_latencies.value
|
A distribution of the latency calculated from when the request was received by the internal HTTP/S load balancer proxy until the proxy got ACK from client on last response byte.
|
object
|
|
gcp.loadbalancing_metrics.https.request.bytes
|
Delta of the number of bytes sent as requests from clients to HTTP/S load balancer.
|
long
|
gauge
|
gcp.loadbalancing_metrics.https.request.count
|
Delta of the number of requests served by HTTP/S load balancer.
|
long
|
gauge
|
gcp.loadbalancing_metrics.https.response.bytes
|
Delta of the number of bytes sent as responses from HTTP/S load balancer to clients.
|
long
|
gauge
|
gcp.loadbalancing_metrics.https.total_latencies.value
|
A distribution of the latency calculated from when the request was received by the external HTTP/S load balancer proxy until the proxy got ACK from client on last response byte.
|
object
|
|
gcp.loadbalancing_metrics.l3.external.egress.bytes
|
Delta of the number of bytes sent from external TCP/UDP network load balancer backend to client of the flow. For TCP flows it's counting bytes on application stream only.
|
long
|
gauge
|
gcp.loadbalancing_metrics.l3.external.egress_packets.count
|
Delta of the number of packets sent from external TCP/UDP network load balancer backend to client of the flow.
|
long
|
gauge
|
gcp.loadbalancing_metrics.l3.external.ingress.bytes
|
Delta of the number of bytes sent from client to external TCP/UDP network load balancer backend. For TCP flows it's counting bytes on application stream only.
|
long
|
gauge
|
gcp.loadbalancing_metrics.l3.external.ingress_packets.count
|
Delta of the number of packets sent from client to external TCP/UDP network load balancer backend.
|
long
|
gauge
|
gcp.loadbalancing_metrics.l3.external.rtt_latencies.value
|
A distribution of the round trip time latency, measured over TCP connections for the external network load balancer.
|
object
|
|
gcp.loadbalancing_metrics.l3.internal.egress.bytes
|
Delta of the number of bytes sent from ILB backend to client (for TCP flows it's counting bytes on application stream only).
|
long
|
gauge
|
gcp.loadbalancing_metrics.l3.internal.egress_packets.count
|
Delta of the number of packets sent from ILB backend to client of the flow.
|
long
|
gauge
|
gcp.loadbalancing_metrics.l3.internal.ingress.bytes
|
Delta of the number of bytes sent from client to ILB backend (for TCP flows it's counting bytes on application stream only).
|
long
|
gauge
|
gcp.loadbalancing_metrics.l3.internal.ingress_packets.count
|
Delta of the number of packets sent from client to ILB backend.
|
long
|
gauge
|
gcp.loadbalancing_metrics.l3.internal.rtt_latencies.value
|
A distribution of RTT measured over TCP connections for internal TCP/UDP load balancer flows.
|
object
|
|
gcp.loadbalancing_metrics.tcp_ssl_proxy.closed_connections.value
|
Delta of the number of connections that were terminated over TCP/SSL proxy.
|
long
|
gauge
|
gcp.loadbalancing_metrics.tcp_ssl_proxy.egress.bytes
|
Delta of the number of bytes sent from VM to client using proxy.
|
long
|
gauge
|
gcp.loadbalancing_metrics.tcp_ssl_proxy.frontend_tcp_rtt.value
|
A distribution of the smoothed RTT (in ms) measured by the proxy's TCP stack, each minute application layer bytes pass from proxy to client.
|
object
|
|
gcp.loadbalancing_metrics.tcp_ssl_proxy.ingress.bytes
|
Delta of the number of bytes sent from client to VM using proxy.
|
long
|
gauge
|
gcp.loadbalancing_metrics.tcp_ssl_proxy.new_connections.value
|
Delta of the number of connections that were created over TCP/SSL proxy.
|
long
|
gauge
|
gcp.loadbalancing_metrics.tcp_ssl_proxy.open_connections.value
|
Current number of outstanding connections through the TCP/SSL proxy.
|
long
|
gauge
|
gcp.metrics.*.*.*.*
|
Metrics that returned from Google Cloud API query.
|
object
|
|
host.architecture
|
Operating system architecture.
|
keyword
|
|
host.containerized
|
If the host is a container.
|
boolean
|
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
|
host.ip
|
Host ip addresses.
|
ip
|
|
host.mac
|
Host mac addresses.
|
keyword
|
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
|
host.os.build
|
OS build information.
|
keyword
|
|
host.os.codename
|
OS codename, if any.
|
keyword
|
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
|
service.type
|
The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch,
service.type would be elasticsearch . |
keyword
|
An example event for loadbalancing
looks as following:
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"cloud": {
"account": {
"id": "elastic-observability"
},
"provider": "gcp",
"region": "us-central1",
"availability_zone": "us-central1-a"
},
"event": {
"dataset": "gcp.loadbalancing_metrics",
"duration": 115000,
"module": "gcp"
},
"gcp": {
"labels": {
"metrics": {
"client_network": "ocp-be-c5kjr-network",
"client_subnetwork": "ocp-be-c5kjr-worker-subnet",
"client_zone": "us-central1-a"
},
"resource": {
"backend_name": "ocp-be-c5kjr-master-us-central1-a",
"backend_scope": "us-central1-a",
"backend_scope_type": "ZONE",
"backend_subnetwork_name": "ocp-be-c5kjr-master-subnet",
"backend_target_name": "ocp-be-c5kjr-api-internal",
"backend_target_type": "BACKEND_SERVICE",
"backend_type": "INSTANCE_GROUP",
"forwarding_rule_name": "ocp-be-c5kjr-api-internal",
"load_balancer_name": "ocp-be-c5kjr-api-internal",
"network_name": "ocp-be-c5kjr-network",
"region": "us-central1"
}
},
"loadbalancing_metrics": {
"l3": {
"internal": {
"egress_packets": {
"count": 100
},
"egress": {
"bytes": 1247589
}
}
}
}
},
"metricset": {
"name": "loadbalancing",
"period": 10000
},
"service": {
"type": "gcp"
}
}
Redis
The redis
dataset is designed to fetch metrics from GCP Memorystore for Redis in Google Cloud Platform.
Exported fields
Field | Description | Type | Unit | Metric Type |
---|---|---|---|---|
@timestamp
|
Event timestamp.
|
date
|
||
agent.id
|
Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.
|
keyword
|
||
cloud
|
Fields related to the cloud or infrastructure the events are coming from.
|
group
|
||
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
||
cloud.account.name
|
The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name.
|
keyword
|
||
cloud.availability_zone
|
Availability zone in which this host, resource, or service is located.
|
keyword
|
||
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
||
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
||
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
||
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
||
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
||
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
||
cloud.region
|
Region in which this host, resource, or service is located.
|
keyword
|
||
container.id
|
Unique container id.
|
keyword
|
||
container.image.name
|
Name of the image the container was built on.
|
keyword
|
||
container.labels
|
Image labels.
|
object
|
||
container.name
|
Container name.
|
keyword
|
||
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
||
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
||
data_stream.type
|
Data stream type.
|
constant_keyword
|
||
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
||
error
|
These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error.
|
group
|
||
error.message
|
Error message.
|
match_only_text
|
||
event.dataset
|
Event dataset
|
constant_keyword
|
||
event.module
|
Event module
|
constant_keyword
|
||
gcp.labels.metadata.*
|
object
|
|||
gcp.labels.metrics.*
|
object
|
|||
gcp.labels.resource.*
|
object
|
|||
gcp.labels.system.*
|
object
|
|||
gcp.labels.user.*
|
object
|
|||
gcp.labels_fingerprint
|
Hashed value of the labels field.
|
keyword
|
||
gcp.metrics.*.*.*.*
|
Metrics that returned from Google Cloud API query.
|
object
|
||
gcp.redis.clients.blocked.count
|
Number of blocked clients.
|
long
|
gauge
|
|
gcp.redis.clients.connected.count
|
Number of client connections.
|
long
|
gauge
|
|
gcp.redis.commands.calls.count
|
Delta of the number of calls for this command in one minute.
|
long
|
gauge
|
|
gcp.redis.commands.total_time.us
|
Delta of the amount of time in microseconds that this command took in the last second.
|
long
|
micros
|
gauge
|
gcp.redis.commands.usec_per_call.sec
|
Average time per call over 1 minute by command.
|
double
|
s
|
gauge
|
gcp.redis.keyspace.avg_ttl.sec
|
Average TTL for keys in this database.
|
double
|
s
|
gauge
|
gcp.redis.keyspace.keys.count
|
Number of keys stored in this database.
|
long
|
gauge
|
|
gcp.redis.keyspace.keys_with_expiration.count
|
Number of keys with an expiration in this database.
|
long
|
gauge
|
|
gcp.redis.persistence.rdb.bgsave_in_progress
|
Flag indicating a RDB save is on-going.
|
long
|
gauge
|
|
gcp.redis.replication.master.slaves.lag.sec
|
The number of seconds that replica is lagging behind primary.
|
long
|
s
|
gauge
|
gcp.redis.replication.master.slaves.offset.bytes
|
The number of bytes that have been acknowledged by replicas.
|
long
|
byte
|
gauge
|
gcp.redis.replication.master_repl_offset.bytes
|
The number of bytes that master has produced and sent to replicas.
|
long
|
byte
|
gauge
|
gcp.redis.replication.offset_diff.bytes
|
The largest number of bytes that have not been replicated across all replicas. This is the biggest difference between replication byte offset (master) and replication byte offset (replica) of all replicas.
|
long
|
byte
|
gauge
|
gcp.redis.replication.role
|
Returns a value indicating the node role. 1 indicates primary and 0 indicates replica.
|
long
|
gauge
|
|
gcp.redis.server.uptime.sec
|
Uptime in seconds.
|
long
|
s
|
gauge
|
gcp.redis.stats.cache_hit_ratio
|
Cache Hit ratio as a fraction.
|
double
|
gauge
|
|
gcp.redis.stats.connections.total.count
|
Delta of the total number of connections accepted by the server.
|
long
|
gauge
|
|
gcp.redis.stats.cpu_utilization.sec
|
CPU-seconds consumed by the Redis server, broken down by system/user space and parent/child relationship.
|
double
|
s
|
gauge
|
gcp.redis.stats.evicted_keys.count
|
Delta of the number of evicted keys due to maxmemory limit.
|
long
|
gauge
|
|
gcp.redis.stats.expired_keys.count
|
Delta of the total number of key expiration events.
|
long
|
gauge
|
|
gcp.redis.stats.keyspace_hits.count
|
Delta of the number of successful lookup of keys in the main dictionary.
|
long
|
gauge
|
|
gcp.redis.stats.keyspace_misses.count
|
Delta of the number of failed lookup of keys in the main dictionary.
|
long
|
gauge
|
|
gcp.redis.stats.memory.maxmemory.mb
|
Maximum amount of memory Redis can consume.
|
long
|
m
|
gauge
|
gcp.redis.stats.memory.system_memory_overload_duration.us
|
The amount of time in microseconds the instance is in system memory overload mode.
|
long
|
micros
|
gauge
|
gcp.redis.stats.memory.system_memory_usage_ratio
|
Memory usage as a ratio of maximum system memory.
|
double
|
gauge
|
|
gcp.redis.stats.memory.usage.bytes
|
Total number of bytes allocated by Redis.
|
long
|
byte
|
gauge
|
gcp.redis.stats.memory.usage_ratio
|
Memory usage as a ratio of maximum memory.
|
double
|
gauge
|
|
gcp.redis.stats.network_traffic.bytes
|
Delta of the total number of bytes sent to/from redis (includes bytes from commands themselves, payload data, and delimiters).
|
long
|
byte
|
gauge
|
gcp.redis.stats.pubsub.channels.count
|
Global number of pub/sub channels with client subscriptions.
|
long
|
gauge
|
|
gcp.redis.stats.pubsub.patterns.count
|
Global number of pub/sub pattern with client subscriptions.
|
long
|
gauge
|
|
gcp.redis.stats.reject_connections.count
|
Number of connections rejected because of maxclients limit.
|
long
|
gauge
|
|
host.architecture
|
Operating system architecture.
|
keyword
|
||
host.containerized
|
If the host is a container.
|
boolean
|
||
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
||
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
||
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
||
host.ip
|
Host ip addresses.
|
ip
|
||
host.mac
|
Host mac addresses.
|
keyword
|
||
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
||
host.os.build
|
OS build information.
|
keyword
|
||
host.os.codename
|
OS codename, if any.
|
keyword
|
||
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
||
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
||
host.os.name
|
Operating system name, without the version.
|
keyword
|
||
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
||
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
||
host.os.version
|
Operating system version as a raw string.
|
keyword
|
||
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
||
service.type
|
The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch,
service.type would be elasticsearch . |
keyword
|
An example event for redis
looks as following:
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"cloud": {
"account": {
"id": "elastic-obs-integrations-dev",
"name": "elastic-obs-integrations-dev"
},
"instance": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"machine": {
"type": "e2-medium"
},
"provider": "gcp",
"availability_zone": "us-central1-c",
"region": "us-central1"
},
"event": {
"dataset": "gcp.redis",
"duration": 115000,
"module": "gcp"
},
"gcp": {
"redis": {
"clients": {
"blocked": {
"count": 4
}
}
},
"labels": {
"user": {
"goog-gke-node": ""
}
}
},
"host": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"metricset": {
"name": "metrics",
"period": 10000
},
"service": {
"type": "gcp"
}
}
Storage
The storage
dataset fetches metrics from Storage in Google Cloud Platform.
Exported fields
Field | Description | Type | Metric Type |
---|---|---|---|
@timestamp
|
Event timestamp.
|
date
|
|
agent.id
|
Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.
|
keyword
|
|
cloud
|
Fields related to the cloud or infrastructure the events are coming from.
|
group
|
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
|
cloud.account.name
|
The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name.
|
keyword
|
|
cloud.availability_zone
|
Availability zone in which this host, resource, or service is located.
|
keyword
|
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
|
cloud.region
|
Region in which this host, resource, or service is located.
|
keyword
|
|
container.id
|
Unique container id.
|
keyword
|
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
|
container.labels
|
Image labels.
|
object
|
|
container.name
|
Container name.
|
keyword
|
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
|
error
|
These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error.
|
group
|
|
error.message
|
Error message.
|
match_only_text
|
|
event.dataset
|
Event dataset
|
constant_keyword
|
|
event.module
|
Event module
|
constant_keyword
|
|
gcp.labels.metadata.*
|
object
|
||
gcp.labels.metrics.*
|
object
|
||
gcp.labels.resource.*
|
object
|
||
gcp.labels.system.*
|
object
|
||
gcp.labels.user.*
|
object
|
||
gcp.labels_fingerprint
|
Hashed value of the labels field.
|
keyword
|
|
gcp.metrics.*.*.*.*
|
Metrics that returned from Google Cloud API query.
|
object
|
|
gcp.storage.api.request.count
|
Delta count of API calls, grouped by the API method name and response code.
|
long
|
gauge
|
gcp.storage.authz.acl_based_object_access.count
|
Delta count of requests that result in an object being granted access solely due to object ACLs.
|
long
|
gauge
|
gcp.storage.authz.acl_operations.count
|
Usage of ACL operations broken down by type.
|
long
|
gauge
|
gcp.storage.authz.object_specific_acl_mutation.count
|
Delta count of changes made to object specific ACLs.
|
long
|
gauge
|
gcp.storage.network.received.bytes
|
Delta count of bytes received over the network, grouped by the API method name and response code.
|
long
|
gauge
|
gcp.storage.network.sent.bytes
|
Delta count of bytes sent over the network, grouped by the API method name and response code.
|
long
|
gauge
|
gcp.storage.storage.object.count
|
Total number of objects per bucket, grouped by storage class. This value is measured once per day, and the value is repeated at each sampling interval throughout the day.
|
long
|
gauge
|
gcp.storage.storage.total.bytes
|
Total size of all objects in the bucket, grouped by storage class. This value is measured once per day, and the value is repeated at each sampling interval throughout the day.
|
long
|
gauge
|
gcp.storage.storage.total_byte_seconds.bytes
|
Delta count of bytes received over the network, grouped by the API method name and response code.
|
long
|
gauge
|
host.architecture
|
Operating system architecture.
|
keyword
|
|
host.containerized
|
If the host is a container.
|
boolean
|
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
|
host.ip
|
Host ip addresses.
|
ip
|
|
host.mac
|
Host mac addresses.
|
keyword
|
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
|
host.os.build
|
OS build information.
|
keyword
|
|
host.os.codename
|
OS codename, if any.
|
keyword
|
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
|
service.type
|
The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch,
service.type would be elasticsearch . |
keyword
|
An example event for storage
looks as following:
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"cloud": {
"account": {
"id": "elastic-obs-integrations-dev",
"name": "elastic-obs-integrations-dev"
},
"instance": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"machine": {
"type": "e2-medium"
},
"provider": "gcp",
"availability_zone": "us-central1-c",
"region": "us-central1"
},
"event": {
"dataset": "gcp.storage",
"duration": 115000,
"module": "gcp"
},
"gcp": {
"storage": {
"storage": {
"total": {
"bytes": 4472520191
}
},
"network": {
"received": {
"bytes": 4472520191
}
}
},
"labels": {
"user": {
"goog-gke-node": ""
}
}
},
"host": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"metricset": {
"name": "storage",
"period": 10000
},
"service": {
"type": "gcp"
}
}
VMware vSphere Integration
This integration periodically fetches logs and metrics from vSphere vCenter servers.
Compatibility
The integration uses the Govmomi library to collect metrics and logs from any Vmware SDK URL (ESXi/VCenter). This library is built for and tested against ESXi and vCenter 6.5, 6.7 and 7.0.
Installation Guide:
VMware vSphere 7.0 Installation
Govmomi Library
Integration Process
Go> Cyber Incident Management (XDR and MDR)
Go> Cyber Incident Management (XDR and MDR)> Settings
Go> Cyber Incident Management (XDR and MDR)> Settings> Integration
Go> Cyber Incident Management (XDR and MDR)> Settings> Integration>
In search bar type “Vmware”
Click Add Agent
Choose your Log Collector
Click the vSphere logs and metrics
Keep it as is
Enter the IP address and port
Example: https://127.0.0.1:8989/sdk
127.0.0.1: This is the IP address of the local machine (localhost).
8989: This is the port number on which the SDK service is running. (Keep it as is)
/sdk: This indicates that the SDK is accessible at this path. (Keep it as is)
Notes: To add multiple hosts, enter each IP address following the same format (https://<IP_or_hostname>:port/sdk) and press enter.
Enter the Username and password of vSphere account
Notes: The insecure option bypasses the verification of the server's certificate chain, which can be useful in certain scenarios but comes with significant security risks. It is recommended to use this option only when necessary and in environments where security concerns are minimal.
Logs collection
Collect logs from vSphere via UDP
UDP host to listen on: This is the IP address of the machine where the log collector is running.
UDP port to listen on: This is the port on which the log collector will listen for incoming log data. (Keep it as is)
Notes: Enabling "Preserve original event" ensures raw log data is always available, crucial for troubleshooting, compliance, and verifying log accuracy. It adds raw data to event.original, doubling storage needs and potentially slowing processing if storage isn't scaled, impacting efficiency.
Collect logs from vSphere via TCP
TCP host to listen on: This is the IP address of the machine where the log collector is running.
TCP port to listen on: This is the port on which the log collector will listen for incoming log data. (Keep it as is)
Notes: Enabling "Preserve original event" ensures raw log data is always available, crucial for troubleshooting, compliance, and verifying log accuracy. It adds raw data to event.original, doubling storage needs and potentially slowing processing if storage isn't scaled, impacting efficiency.
Click Next to complete the integration.