System Integrations
- Cyber Incident Monitoring Integration Procedure
- Microsoft 365
- GitHub
- Add Windows Integrations
- Sysmon for Linux
- 1 Password Integrations
- Atlassian Bitbucket Integrations
- AWS Cloudtrails Integrations
- AWS GuardDuty Integrations
- AWS Security Hub Integrations
- AWS Integrations
- CISCO Meraki Integrations
- CISCO Secure Endpoint Integrations
- CISCO Umbrella Integrations
- Cloudflare Integration
- Crowdstrike Integrations
- Dropbox Integrations
- F5 Integrations
- Fortinet-Fortigate Integrations
- GCP Integrations
- GitLab Integrations
- Google Workspace Integrations
- Jumpcloud Integrations
- Mimecast Integrations
- MongoDB Integrations
- OKTA Integrations
- Pulse Connect Secure Integrations
- Slack Integrations
- System Integrations
- Team Viewer Integrations
- Z Scaler Integrations
- gcp
- VMware vSphere Integration
- SentinelOne Integrations
- Custom Windows Event Logs - Integration
- Windows Event Forwarding to Linux server using Nxlog
- Windows Event Forwarding to Linux server using Powershell script
- Sophos Integration
- Atlassian Bitbucket Integrations (New)
- Palo Alto Cortex XDR Integration
- Active Directory Integrations
- Microsoft SQL Server Integration
- Azure Logs Integration
- ESET Protect Integration
- ESET Threat Intelligence Integrations
- CSPM for Azure Integration
- Resource Manager Endpoint Integration
- CISCO Secure Email Gateway Integrations
- CISCO Nexus Integrations
- BitDefender Integrations
- Bitwarden Integrations
- Forwarding logs from rsyslog client to a remote rsyslogs server
- New script for logs forwarding
Cyber Incident Monitoring Integration Procedure
Go to > Cyber Incident Monitoring
Microsoft 365
Microsoft Office 365 integration currently supports user, admin, system, and policy actions and events from Office 365 and Azure AD activity logs exposed by the Office 365 Management Activity API.
Procedures
To perform the setup, please confirm that you have the following access:
-
A Microsoft Office 365 account with Administrative Privileges
-
A Microsoft Azure account with Administrative Privileges
Register a new Office 365 web application To get started collecting Office 365 logs, register an Office 365 web application:
-
Log into the Office 365 portal as an Active Directory tenant administrator.
-
Click Show all to expand the left navigation area, and then click Azure Active Directory.
-
Select App Registrations, and then click + New application registration.
-
Provide the following information in the fields:
-
-
-
Name – for example, o365cytech.
-
Select Single tenant for supported account types.
-
Leave the Redirect URI blank.
-
The Audit Log Search needs to be enabled.
-
Click Register and note the Application (client) ID.
-
-
Setup Active Directory security permissions
The Active Directory security permissions allow the application you created to read threat intelligence data and activity reports for your organization.
To set up Active Directory permissions:
- On the main panel under the new application, click API Permissions, and then click + Add a permission.
- Locate and click on Office 365 Management APIs.
- In Application permissions, expand and select ActivityFeed.Read, ActivityFeed.ReadDlp, ActivityReports.Read, and ServiceHealth.Read
- Ensure all necessary permissions are selected, and then click Add permissions.
- Click Grant admin consent, and then click Accept to confirm.
- On the left navigation area, select Certificates & secrets, and then click + New client secret.
- Make Sure to Copy the Value (Client Secret (Api Key) will disappear
- Type a key Description and set the duration to Never or Maximum Grant time.
- Click Add.
- Click Overview to return to the application summary, and then click the link under Managed application in local directory.
- Click Properties, and then note the Object ID associated with the application.
Steps to Renew the Client Secret (API Key):
-
Log into the Azure Portal:
- Go to the Azure Portal and log in using an account with administrative privileges.
-
Navigate to Azure Active Directory:
- In the left navigation pane, select Azure Active Directory.
- If it's not visible, click Show all to expand the list and find it.
-
Go to App Registrations:
- Under Azure Active Directory, select App Registrations.
- Find your registered application (e.g., "o365cytech") in the list, or use the search bar to locate it.
-
Open Certificates & Secrets:
- Click on the registered app to open its details page.
- In the left-hand menu, select Certificates & Secrets.
-
Generate a New Client Secret:
- Under Client Secrets, you'll see a list of previously created secrets, along with their expiration dates.
- Click + New client secret to create a new one.
-
Configure the New Secret:
- Enter a description for the new key (e.g., "Renewed Key for o365cytech").
- Set the duration for the new client secret:
-
Save and Copy the New Secret:
- Click Add.
- Once the new secret is generated, copy the value immediately. This is your new client secret (API key). The secret value will be hidden after you leave this page, so make sure to store it securely.
-
Update Any Services Using the Key:
- If any services or scripts are using the previous client secret, you'll need to update them with the new one.
-
Remove the Old Secret (Optional):
- If the old client secret is no longer needed, you can delete it to avoid confusion. Simply click the trash icon next to the old key under Client Secrets.
GitHub
Introduction
The GitHub integration collects events from the GitHub API.
Logs
Audit
The GitHub audit log records all events related to the GitHub organization.
To use this integration, you must be an organization owner, and you must use an Personal Access Token with the admin:org scope.
This integration is not compatible with GitHub Enterprise server.
Code Scanning
The Code Scanning lets you retrieve all security vulnerabilities and coding errors from a repository setup using Github Advanced Security Code Scanning feature.
To use this integration, GitHub Apps must have the security_events read permission. Or use a personal access token with the security_events scope for private repos or public_repo scope for public repos.
Secret Scanning
The Github Secret Scanning lets you retrieve secret scanning for advanced security alerts from a repository setup using Github Advanced Security Secret Scanning feature.
To use this integration, GitHub Apps must have the secret_scanning_alerts read permission. Or you must be an administrator for the repository or for the organization that owns the repository, and you must use a personal access token with the repo scope or security_events scope. For public repositories, you may instead use the public_repo scope.
Dependabot
The Github Dependabot lets you retrieve known vulnerabilities in dependencies from a repository setup using Github Advanced Security Dependabot feature.
To use this integration, you must be an administrator for the repository or for the organization that owns the repository, and you must use a personal access token with the repo scope or security_events scope. For public repositories, you may instead use the public_repo scope.
Issues
The Github Issues datastream lets you retrieve github issues, including pull requests, issue assignees, comments, labels, and milestones. See About Issues for more details. You can retrieve issues for specific repository or for entire organization. Since Github API considers pull requests as issues, users can use github.issues.is_pr field to filter for only pull requests.
All issues including closed are retrieved by default. If users want to retrieve only open requests, you need to change State parameter to open.
To use this integration, users must use Github Apps or Personal Access Token with read permission to repositories or organization. Please refer to Github Apps Permissions Required and Personal Access Token Permissions Required for more details.
GitHub Integration Procedures
Please provide the following information to CyTech:
1.Select Settings
2. Select Developer Settings
3. Select token (classic)
4. Select scope admin:scope
Collect GitHub logs via API
-
Personal Access Token - the GitHub Personal Access Token. Requires the 'admin:org' scope
-
Organization Name - The GitHub organization name/ID
GHAS Code Scanning
-
Personal Access Token - the GitHub Personal Access Token. Requires the 'public_repo' scope for public repositories and 'security_events' scope for private repositories. \nSee List code scanning alerts for a repository
-
Repository owner - The owner of GitHub Repository. If repository belongs to an organization, owner is name of the organization
GHAS Dependabot
-
Personal Access Token - The GitHub Personal Access Token. \nSee Authenticating with GraphQL
-
Repository owner - The owner of GitHub Repository
Github Issues
1. Personal Access Token - the GitHub Personal Access Token.
2. Repository owner - The owner of GitHub Repository. If repository belongs to an organization, owner is name of the organization.
GHAS Secret Scanning
1. Personal Access Token - the GitHub Personal Access Token. Requires admin access to the repository or organization owning the repository along with a personal access token with 'public_repo' scope for public repositories and repo or security_events scope for private repositories. \nSee List secret scanning alerts for a repository
2. Repository owner - The owner of GitHub Repository
Add Windows Integrations
Introduction
The Windows integration allows you to monitor the Windows OS, services, applications, and more.
Use the Windows integration to collect metrics and logs from your machine. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue.
For example, if you wanted to know if a Windows service unexpectedly stops running, you could install the Windows integration to send service metrics to Elastic. Then, you could view real-time changes to service status in Kibana's [Metrics Windows] Services dashboard.
Data streams
The Windows integration collects two types of data: logs and metrics.
Logs help you keep a record of events that happen on your machine. Log data streams collected by the Windows integration include forwarded events, PowerShell events, and Sysmon events. Log collection for the Security, Application, and System event logs is handled by the System integration. See more details in the Logs reference.
-
https://aquila-elk.kb.us-east-1.aws.found.io:9243/app/integrations/detail/windows-1.15.2/overview#logs-reference
Metrics give you insight into the state of the machine. Metric data streams collected by the Windows integration include service details and performance counter values. See more details in the Metrics reference.
Note: For 7.11, security, application and system logs have been moved to the system package.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
Each data stream collects different kinds of metric data, which may require dedicated permissions to be fetched and which may vary across operating systems.
Setup
For step-by-step instructions on how to set up an integration, see the Getting started guide.
Note: Because the Windows integration always applies to the local server, the hosts config option is not needed.
Ingesting Windows Events via Splunk
This integration allows you to seamlessly ingest data from a Splunk Enterprise instance. The integration uses the httpjson input in Elastic Agent to run a Splunk search via the Splunk REST API and then extract the raw event from the results. The raw event is then processed via the Elastic Agent. You can customize both the Splunk search query and the interval between searches. For more information see Ingest data from Splunk.
Note: This integration requires Windows Events from Splunk to be in XML format. To achieve this, renderXml needs to be set to 1 in your inputs.conf file.
Logs reference
Forwarded
The Windows forwarded data stream provides events from the Windows ForwardedEvents event log. The fields will be the same as the channel specific data streams.
Powershell
The Windows powershell data stream provides events from the Windows Windows PowerShell event log.
System Integration Procedures
Collect events from the following Windows event log channels: (Enable Yes/No)
-
Preserve original event (Enable Yes/No)
-
Preserves a raw copy of the original XML event, added to the field event.original
-
Event ID
-
A list of included and excluded (blocked) event IDs. The value is a comma-separated list. The accepted values are single event IDs to include (e.g. 4624), a range of event IDs to include (e.g. 4700-4800), and single event IDs to exclude (e.g. -4735). Limit 22 IDs.
-
Ignore events older than
-
If this option is specified, events that are older than the specified amount of time are ignored. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
Language ID
-
The language ID the events will be rendered in. The language will be forced regardless of the system language. A complete list of language IDs can be found https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c[here]. It defaults to 0, which indicates to use the system language. E.g.: 0x0409 for en-US
-
Tags
-
Processors
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
Synthetic source (Enable Yes/No)
Powershell (Enable Yes/No)
-
Preserve original event (Enable Yes/No)
-
Preserves a raw copy of the original XML event, added to the field event.original
-
Event ID
-
A list of included and excluded (blocked) event IDs. The value is a comma-separated list. The accepted values are single event IDs to include (e.g. 4624), a range of event IDs to include (e.g. 4700-4800), and single event IDs to exclude (e.g. -4735). Limit 22 IDs.
-
Ignore events older than
-
If this option is specified, events that are older than the specified amount of time are ignored. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
Language ID
-
The language ID the events will be rendered in. The language will be forced regardless of the system language. A complete list of language IDs can be found https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c[here]. It defaults to 0, which indicates to use the system language. E.g.: 0x0409 for en-US
-
Tags
-
Processors
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
Synthetic source (Enable Yes/No)
Powershell Operational (Enable Yes/No)
-
Preserve original event (Enable Yes/No)
-
Preserves a raw copy of the original XML event, added to the field event.original
-
Event ID
-
A list of included and excluded (blocked) event IDs. The value is a comma-separated list. The accepted values are single event IDs to include (e.g. 4624), a range of event IDs to include (e.g. 4700-4800), and single event IDs to exclude (e.g. -4735). Limit 22 IDs.
-
Ignore events older than
-
If this option is specified, events that are older than the specified amount of time are ignored. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
Language ID
-
The language ID the events will be rendered in. The language will be forced regardless of the system language. A complete list of language IDs can be found https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c[here]. It defaults to 0, which indicates to use the system language. E.g.: 0x0409 for en-US
-
Tags
-
Processors
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
Synthetic source (Enable Yes/No)
Sysmon Operational (Enable Yes/No)
-
Preserve original event (Enable Yes/No)
-
Preserves a raw copy of the original XML event, added to the field event.original
-
Event ID
-
A list of included and excluded (blocked) event IDs. The value is a comma-separated list. The accepted values are single event IDs to include (e.g. 4624), a range of event IDs to include (e.g. 4700-4800), and single event IDs to exclude (e.g. -4735). Limit 22 IDs.
-
Ignore events older than
-
If this option is specified, events that are older than the specified amount of time are ignored. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
Language ID
-
The language ID the events will be rendered in. The language will be forced regardless of the system language. A complete list of language IDs can be found https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c[here]. It defaults to 0, which indicates to use the system language. E.g.: 0x0409 for en-US
-
Tags
-
Processors
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
Synthetic source (Enable Yes/No)
Collect Windows perfmon and service metrics (Enable Yes/No)
-
Perfmon Group Measurements By Instance (Enable Yes/No)
-
Enabling this option will send all measurements with a matching perfmon instance as part of a single event
-
Perfmon Ignore Non Existent Counters (Enable Yes/No)
-
Enabling this option will make sure to ignore any errors caused by counters that do not exist
-
Perfmon Queries
-
Will list the perfmon queries to execute, each query will have an object option, an optional instance contiguration and the actual counters
-
Period
-
Synthetic source (Enable Yes/No)
Windows service metrics (Enable Yes/No)
-
Period
-
Processors
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
Synthetic source (Enable Yes/No)
Collect logs from third-party REST API (experimental) (Enable Yes/No)
-
URL of Splunk Enterprise Server
-
i.e. scheme://host:port, path is automatic
-
Splunk REST API Username
-
Splunk Authorization Token
-
Bearer Token or Session Key, e.g. "Bearer eyJFd3e46..." or "Splunk 192fd3e...". Cannot be used with username and password.
-
SSL Configuration
-
i.e. certificate_authorities, supported_protocols, verification_mode etc.
Windows ForwardedEvents via Splunk Enterprise REST API (Enable Yes/No)
-
Interval to query Splunk Enterprise REST API
-
Go Duration syntax (eg. 10s)
-
Preserve original event (Enable Yes/No)
-
Preserves a raw copy of the original event, added to the field event.original
-
Splunk search string
-
Tags
-
Processors
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
Synthetic source (Enable Yes/No)
Windows Powershell Events via Splunk Enterprise REST API (Enable Yes/No)
-
Interval to query Splunk Enterprise REST API
-
Go Duration syntax (eg. 10s)
-
Preserve original event (Enable Yes/No)
-
Preserves a raw copy of the original event, added to the field event.original
-
Splunk search string
-
Tags
-
Processors
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
Synthetic source (Enable Yes/No)
Windows Powershell Operational Events via Splunk Enterprise REST API (Enable Yes/No)
-
Interval to query Splunk Enterprise REST API
-
Go Duration syntax (eg. 10s)
-
Preserve original event (Enable Yes/No)
-
Preserves a raw copy of the original event, added to the field event.original
-
Splunk search string
-
Tags
-
Processors
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
Synthetic source (Enable Yes/No)
Windows Sysmon Operational Events via Splunk Enterprise REST API (Enable Yes/No)
-
Interval to query Splunk Enterprise REST API
-
Go Duration syntax (eg. 10s)
-
Preserve original event (Enable Yes/No)
-
Preserves a raw copy of the original event, added to the field event.original
-
Splunk search string
-
Tags
-
Processors
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
Synthetic source (Enable Yes/No)
Sysmon for Linux
Introduction
The Sysmon for Linux integration allows you to monitor the Sysmon for Linux, which is an open-source system monitor tool developed to collect security events from Linux environments.
Use the Sysmon for Linux integration to collect logs from linux machine which has sysmon tool running. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue.
NOTE: To collect Sysmon events from Windows event log, use Windows sysmon_operational data stream instead.
-
Sysmon for Linux - https://github.com/Sysinternals/SysmonForLinux
-
Windows sysmon_operational data stream - https://docs.elastic.co/en/integrations/windows#sysmonoperational
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
Setup
For step-by-step instructions on how to set up an integration, see the Getting started guide.
Data streams
The Sysmon for Linux log data stream provides events from logs produced by Sysmon tool running on Linux machine.
Sysmon for Linux Integration
Please provide the following information to CyTech:
Collect Sysmon for Linux logs (Enable Yes/No)
-
Paths - /var/log/sysmon*
1 Password Integrations
Introduction
With 1Password Business, you can send your account activity to your security information and event management (SIEM) system, using the 1Password Events API.
Get reports about 1Password activity, such as sign-in attempts and item usage, while you manage all your company’s applications and services from a central location.
With 1Password Events Reporting and Elastic SIEM, you can:
-
Control your 1Password data retention
-
Build custom graphs and dashboards
-
Set up custom alerts that trigger specific actions
-
Cross-reference 1Password events with the data from other services
Events
Sign-in Attempts
Use the 1Password Events API to retrieve information about sign-in attempts. Events include the name and IP address of the user who attempted to sign in to the account, when the attempt was made, and – for failed attempts – the cause of the failure.
Item Usages
This uses the 1Password Events API to retrieve information about items in shared vaults that have been modified, accessed, or used. Events include the name and IP address of the user who accessed the item, when it was accessed, and the vault where the item is stored.
Requirements
You can set up Events Reporting if you’re an owner or administrator.
Ready to get started?
1Password Integration Procedures
Please provide the following information to CyTech:
The 1Password Events API Beat returns information from 1Password through requests to the Events REST API and sends that data securely to Elasticsearch. Requests are authenticated with a bearer token. Issue a token for each application or service you use.
To connect your 1Password account to Elastic:
-
Download and install the 1Password Events API Elastic Beat from the 1Password GitHub repository.
-
Download an example eventsapibeat.yml file .
-
Configure the YAML file for the Beat to include:
-
The bearer token you saved previously in the auth_token fields for each 1Password event type you plan to monitor.
-
The output for events (sent directly to Elasticsearch, or through Logstash).
-
Any other configurations you want to customize.
-
Run the following command: ./eventsapibeat -c eventsapibeat.yml -e
You can now use Elasticsearch with the 1Password Events API Beat to monitor events from your 1Password account. The returned data will follow the Elastic Common Schema (ECS) specifications.
Collect events from 1Password Events API
-
URL of 1Password Events API Server - options: https://events.1password.com, https://events.1password.ca, https://events.1password.eu, https://events.ent.1password.com. path is automatic
-
1Password Authorization Token - Bearer Token, e.g. "eyJhbGciO..."
Atlassian Bitbucket Integrations
Introduction
The Bitbucket integration collects audit logs from the audit log files or the audit API.
Reference: https://developer.atlassian.com/server/bitbucket/reference/rest-api/
Assumptions
The procedures described in Section 3 assume that a Log Collector has already been set up.
Requirements
For more information on auditing in Bitbucket and how it can be configured, see View and configure the audit log on Atlassian's website.
Reference: https://confluence.atlassian.com/bitbucketserver/view-and-configure-the-audit-log-776640417.html
Logs
Audit
The Confluence integration collects audit logs from the audit log files or the audit API from self-hosted Confluence Data Center. It has been tested with Confluence 7.14.2 but is expected to work with newer versions. As of version 1.2.0, this integration added experimental support for Atlassian Confluence Cloud. JIRA Cloud only supports Basic Auth using username and a Personal Access Token.
Atlassian Bitbucket Integration Procedures
Please provide the following information to CyTech:
Collect Bitbucket audit logs via log files
-
Path
-
Preserve Original Event? (Enable Yes/No)
-
Preserves a raw copy of the original event, added to the field event.original
-
Tags
-
Processors (Optional)
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed.
-
Indexing settings (experimental) (Enable Yes/No)
-
Select data streams to configure indexing options. This is an experimental feature and may have effects on other properties.
Collect Bitbucket audit logs via API (Enable Yes/No)
-
API URL - The API URL without the path.
-
Bitbucket Username - JIRA Username. Needs to be used with a Password. Do not fill if you are using a personal access token.
-
Bitbucket Password - JIRA Password. Needs to be used with a Username. Do not fill if you are using a personal access token.
-
Personal Access Token - The Personal Access Token. If set, Username and Password will be ignored.
-
Initial Interval - Initial interval for the first API call. Defaults to 24 hours.
AWS Cloudtrails Integrations
Introduction
The AWS CloudTrail integration allows you to monitor AWS CloudTrail
Reference: https://aws.amazon.com/cloudtrail/
Use the AWS CloudTrail integration to collect and parse logs related to account activity across your AWS infrastructure. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference logs when troubleshooting an issue.
For example, you could use the data from this integration to spot unusual activity in your AWS accounts—like excessive failed AWS console sign in attempts.
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Data streams
The AWS CloudTrail integration collects one type of data: logs.
Logs help you keep a record of every event that CloudTrail receives. These logs are useful for many scenarios, including security and access audits. See more details in the Logs reference.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
Before using any AWS integration you will need:
-
AWS Credentials to connect with your AWS account.
-
AWS Permissions to make sure the user you're using to connect has permission to share the relevant data.
For more details about these requirements, see the AWS integration documentation.
Setup
Use this integration if you only need to collect data from the AWS CloudTrail service.
If you want to collect data from two or more AWS services, consider using the AWS integration. When you configure the AWS integration, you can collect data from as many AWS services as you'd like.
For step-by-step instructions on how to set up an integration, see the Getting started guide.
Logs reference
The cloudtrail data stream collects AWS CloudTrail logs. CloudTrail monitors events like user activity and API usage in AWS services. If a user creates a trail, it delivers those events as log files to a specific Amazon S3 bucket.
AWS CloudTrail integration Procedures
The following will need to be provided in the Configure integration when adding the AWS Audit CloudTrail integration.
Procedures:
Please provide the following information to CyTech:
Configure Integration
-
Collect CloudTrail logs from S3 (Enable)
-
Queue URL is required (URL of the AWS SQS queue that messages will be received from.)
-
Collect CloudTrail logs from CloudWatch(Optional)
-
Log Group ARN (ARN of the log group to collect logs from.)
-
Collect CloudTrail logs from third-party REST API (Optional)
-
URL of Splunk Enterprise Server (i.e. scheme://host:port, path is automatic)
-
Splunk REST API Username
-
Splunk REST API Password
-
Splunk Authorization Token (Bearer Token or Session Key, e.g. "Bearer eyJFd3e46..." or "Splunk 192fd3e...". Cannot be used with username and password.)
AWS GuardDuty Integrations
Introduction
The Amazon GuardDuty integration collects and parses data from Amazon GuardDuty Findings REST APIs.
The Amazon GuardDuty integration can be used in three different modes to collect data:
-
HTTP REST API - Amazon GuardDuty pushes logs directly to an HTTP REST API.
-
AWS S3 polling - Amazon GuardDuty writes data to S3 and Elastic Agent polls the S3 bucket by listing its contents and reading new files.
-
AWS S3 SQS - Amazon GuardDuty writes data to S3, S3 pushes a new object notification to SQS, Elastic Agent receives the notification from SQS, and then reads the S3 object. Multiple Agents can be used in this mode.
Assumptions
The procedures described in Section 3 assume that a Log Collector has already been setup.
Requirements
You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
Note: It is recommended to use AWS SQS for Amazon GuardDuty.
Aws GuardDuty integration Procedures
To collect data from AWS S3 Bucket, follow the steps below:
-
Configure the Data Forwarder to ingest data into an AWS S3 bucket. However, the user can set the parameter "Bucket List Prefix" according to the requirement.
To collect data from AWS SQS, follow the steps below:
-
If data forwarding to an AWS S3 bucket hasn't been configured, then first setup an AWS S3 bucket as mentioned in the documentation above.
-
To setup an SQS queue, follow "Step 1: Create an Amazon SQS queue" mentioned in the Documentation.
-
While creating an SQS queue, please provide the same bucket ARN that has been generated after creating the AWS S3 bucket.
-
Setup event notification for an S3 bucket. Follow this guide.
-
The user has to perform Step 3 for the guardduty data-stream, and the prefix parameter should be set the same as the S3 Bucket List Prefix as created earlier. For example, logs/ for guardduty data stream.
-
For all the event notifications that have been created, select the event type as s3:ObjectCreated:*, select the destination type SQS Queue, and select the queue that has been created in Step 2.
Note:
-
Credentials for the above AWS S3 and SQS input types should be configured according to the input configuration guide.
-
Data collection via AWS S3 Bucket and AWS SQS are mutually exclusive in this case.
To collect data from Amazon GuardDuty API, users must have an Access Key and a Secret Key. To create an API token follow the steps below:
-
Login to https://console.aws.amazon.com/.
-
Go to https://console.aws.amazon.com/iam/ to access the IAM console.
-
On the navigation menu, choose Users.
-
Choose your IAM user name.
-
Select Create access key from the Security Credentials tab.
-
To see the new access key, choose Show.
Note
-
The Secret Access Key and Access Key ID are required for the current integration package.
AWS Security Hub Integrations
Introduction
The AWS Security Hub integration collects and parses data from AWS Security Hub REST APIs.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Compatibility
-
This module is tested against AWS Security Hub API version 1.0.
Requirements
To collect data from AWS Security Hub APIs, users must have an Access Key and a Secret Key. To create API token follow below steps:
-
Login to https://console.aws.amazon.com/.
-
Go to https://console.aws.amazon.com/iam/ to access the IAM console.
-
On the navigation menu, choose Users.
-
Choose your IAM user name.
-
Select Create access key from the Security Credentials tab.
-
To see the new access key, choose Show.
Note:
-
For the current integration package, it is recommended to have interval in hours.
-
For the current integration package, it is compulsory to add Secret Access Key and Access Key ID.
Logs:
-
Findings - This is the securityhub_findings data stream.
-
Insights - This is the securityhub_insights data stream.
AWS Security Hub Integration Procedures
Please provide the following information to CyTech:
Collect AWS Security Hub logs via API
-
AWS Region - AWS Region.
Collect AWS Security Hub Insights from AWS
-
AWS Region - AWS Region.
AWS Integrations
Introduction
This document shows information related to AWS Integration.
The AWS integration is used to fetch logs and metrics from Amazon Web Services.
The usage of the AWS integration is to collect metrics and logs across many AWS services managed by your AWS account.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
Before using the AWS integration you will need:
-
AWS Credentials to connect with your AWS account.
-
AWS Permissions to make sure the user you're using to connect has permission to share the relevant data.
AWS Credentials
Use access keys directly (Option 1 Recommended)
Access keys are long-term credentials for an IAM user or the AWS account root user. To use access keys as credentials, you need to provide:
-
access_key_id: The first part of the access key.
-
secret_access_key: The second part of the access key.
Use an IAM role Amazon Resource Name (ARN)
To use an IAM role ARN, you need to provide either a credential profile or access keys along with the role_arn advanced option. role_arn is used to specify which AWS IAM role to assume for generating temporary credentials.
Instead of providing the access_key_id and secret_access_key directly to the integration, you will provide two advanced options to look up the access keys in the shared credentials file:
-
credential_profile_name: The profile name in shared credentials file.
Access keys are long-term credentials for an IAM user or the AWS account root user. To use access keys as credentials, you need to provide:
-
access_key_id: The first part of the access key.
-
secret_access_key: The second part of the access key.
AWS Permissions
Specific AWS permissions are required for the IAM user to make specific AWS API calls. To enable the AWS integration to collect metrics and logs from all supported services, make sure to give necessary permissions which CyTech to monitor.
Reference permissions:
-
ec2:DescribeInstances
-
ec2:DescribeRegions
-
cloudwatch:GetMetricData
-
cloudwatch:ListMetrics
-
iam:ListAccountAliases
-
rds:DescribeDBInstances
-
rds:ListTagsForResource
-
s3:GetObject
-
sns:ListTopics
-
sqs:ChangeMessageVisibility
-
sqs:DeleteMessage
-
sqs:ListQueues
-
sqs:ReceiveMessage
-
sts:AssumeRole
-
sts:GetCallerIdentity
-
tag:GetResources
AWS Integrations Procedures
Access Key ID and Secret Access Key:
These are associated with AWS Identity and Access Management (IAM) users and are used for programmatic access to AWS services. To find them:
-
Access the AWS Management Console.
-
Go to the "IAM" (Identity and Access Management) service.
-
Select the IAM user for which you want to retrieve the access keys.
-
Under the "Security credentials" tab, you can find the Access Key ID and you can create a new Secret Access Key if needed.
S3 Bucket ARN:
You can find the Amazon Resource Name (ARN) for an S3 bucket in the S3 Management Console or by using the AWS CLI. It typically looks like this:
-
Access the AWS Management Console.
-
Go to S3 Bucket Service
-
Navigate properties.
-
Find the ARN under Bucket overview
Sample: arn:aws:s3:::your-bucket-name
Log Group ARN:
Log Groups are associated with AWS CloudWatch Logs. You can find the ARN for a log group as follows:
-
Access the AWS Management Console.
-
Go to the "CloudWatch" service.
-
In the CloudWatch Logs section, select the log group you're interested in.
-
Select the log group that you want to open.
-
In the details of the log group, you will find the ARN.
SQS Queue URL: (Ignore if you're not using SQS Queue URL)
To find the URL of an Amazon Simple Queue Service (SQS) queue:
-
Access the AWS Management Console.
-
Go to the "SQS" service.
-
Select the specific queue you're interested in.
-
In the queue details, you can find the Queue URL
Please provide the following information to CyTech:
-
Access Key ID
-
Secret Access Key
- S3 Bucket ARN
- Log Group ARN
- SQS Queue URL : (Ignore if you're not using SQS Queue URL)
CISCO Meraki Integrations
Introduction
Cisco Meraki offers a centralized cloud management platform for all Meraki devices such as MX Security Appliances, MR Access Points and so on. Its out-of-band cloud architecture creates secure, scalable, and easy-to-deploy networks that can be managed from anywhere. This can be done from almost any device using web-based Meraki Dashboard and Meraki Mobile App. Each Meraki network generates its own events.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Compatibility
A syslog server can be configured to store messages for reporting purposes from MX Security Appliances, MR Access Points, and MS switches. This package collects events from the configured syslog server. The integration supports collection of events from "MX Security Appliances" and "MR Access Points". The "MS Switch" events are not recognized.
Requirements
Cisco Meraki Dashboard Configuration
SYSLOG
Cisco Meraki dashboard can be used to configure one or more syslog servers and Meraki message types to be sent to the syslog servers. Refer to Syslog Server Overview and Configuration page for more information on how to configure syslog server on Cisco Meraki.
API ENDPOINT (WEBHOOKS)
Cisco Meraki dashboard can be used to configure Meraki webhooks. Refer to the Webhooks Dashboard Setup section.
Configure the Cisco Meraki integration
SYSLOG
Depending on the syslog server setup in your environment check one/more of the following options "Collect syslog from Cisco Meraki via UDP", "Collect syslog from Cisco Meraki via TCP", "Collect syslog from Cisco Meraki via file".
Enter the values for syslog host and port OR file path based on the chosen configuration options.
API Endpoint (Webhooks)
Check the option "Collect events from Cisco Meraki via Webhooks" option.
-
Enter values for "Listen Address", "Listen Port" and "Webhook path" to form the endpoint URL. Make note of the Endpoint URL https://{AGENT_ADDRESS}:8686/meraki/events.
-
Enter value for "Secret value". This must match the "Shared Secret" value entered when configuring the webhook from Meraki cloud.
-
Enter values for "TLS". Cisco Meraki requires that the webhook accept requests over HTTPS. So you must either configure the integration with a valid TLS certificate or use a reverse proxy in front of the integration.
Log Events
Enable to collect Cisco Meraki log events for all the applications configured for the chosen log stream.
Logs
Syslog
The cisco_meraki.log dataset provides events from the configured syslog server. All Cisco Meraki syslog specific fields are available in the cisco_meraki.log field group.
API Endpoint (Webhooks)
Cisco Meraki Integration Procedures
Please provide the following information to CyTech:
- Collect syslog from Cisco Meraki via UDP
- Listen Address - The bind address to listen for UDP connections. Set to 0.0.0.0 to bind to all available interfaces.
- Listen Port - The UDP port number to listen on.
-
Collect syslog from Cisco Meraki via TCP
- Listen Address - The bind address to listen for TCP connections. Set to 0.0.0.0 to bind to all available interfaces.
- Listen Port - The UDP port number to listen on.
-
Collect syslog from Cisco Meraki via file
- Paths
- Collect syslog from Cisco Meraki via Webhooks
- Listen Address - Bind address for the listener. Use 0.0.0.0 to listen on all interfaces.
- Listen Port
CISCO Secure Endpoint Integrations
Introduction
Secure Endpoint offers cloud-delivered, advanced endpoint detection and response across multidomain control points to rapidly detect, contain, and remediate advanced threats.
Assumptions
The procedures described in Section 3 assume that a Log Collector has already been setup.
Requirements
This integration is for Cisco Secure Endpoint logs. It includes the following datasets for receiving logs over syslog or read from a file:
-
event dataset: supports Cisco Secure Endpoint Event logs.
Generating Client ID and API Key:
- Log in to your AMP for Endpoints Console.
- Go to Accounts > Organization Settings.
- Click Configure API Credentials under Features to generate the Client ID and secure API Key.
Logs
Secure Endpoint
The event dataset collects Cisco Secure Endpoint logs.
What can the Secure Endpoint API be used for?
-
Generate a list of organizations a user has access to
-
Generate a list of policies for a specified organization
-
Generate specific information about a specified policy such as:
-
-
-
-
General policy data
-
Associated network control lists
-
Associated computers
-
Associated groups
-
Proxy settings
-
Policy XML
-
-
-
-
Generate all policy types and operating systems available for a specified organization
Top Use Cases
-
Generating reports on policy settings across an organization
-
Inspecting a particular policy's settings
-
Querying to find policies matching certain criteria in order to detect which policies should be edited
Response Format
-
Data
-
Meta
-
Errors
Cisco Secure Endpoint Integration Procedures
Please provide the following information to CyTech:
Collect logs from the Cisco Secure Endpoint API.
-
Client ID - Cisco Secure Endpoint Client ID
-
API Key - Cisco Secure Endpoint API Key
CISCO Umbrella Integrations
Introduction
Cisco Umbrella is a cloud security platform that provides an additional line of defense against malicious software and threats on the internet by using threat intelligence. That intelligence helps prevent adware, malware, botnets, phishing attacks, and other known bad Websites from being accessed.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Prerequisites
-
-
You must have Full Admin access to Umbrella to create and manage Umbrella API keys or Umbrella KeyAdmin API keys.
-
Requirements
This integration is for Cisco Umbrella. It includes the following datasets for receiving logs from an AWS S3 bucket using an SQS notification queue and Cisco Managed S3 bucket without SQS:
-
log dataset: supports Cisco Umbrella logs.
Logs
Umbrella
When using Cisco Managed S3 buckets that does not use SQS there is no load balancing possibilities for multiple agents, a single agent should be configured to poll the S3 bucket for new and updated files, and the number of workers can be configured to scale vertically.
The log dataset collects Cisco Umbrella logs.
Advantages of Integrating with the Umbrella API
The Umbrella API features a number of improvements over the Umbrella v1 APIs and the Umbrella Reporting v2 API.
-
Intuitive base URI
-
API paths defined by top-level scopes
-
Intent-based, granular API key scopes
-
API key expiration
-
Updated API key administration dashboard views
-
Programmatic API key administration
-
API authentication and authorization supported by OAuth 2.0 client credentials flow
-
Portable, programmable API interface for client integrations
Before you send a request to the Umbrella API, you must create new Umbrella API credentials and generate an API access token. For more information, see Umbrella API Authentication.
https://developer.cisco.com/docs/cloud-security/authentication/#authentication
Authentication
The Umbrella API provides a standard REST interface and supports the OAuth 2.0 client credentials flow. To get started, log in to Umbrella and create an Umbrella API key. Then, use your API credentials to generate an API access token.
Note: API keys, passwords, secrets, and tokens allow access to your private customer data. You should never share your credentials with another user or organization.
Log in to Umbrella
-
Log in to Umbrella with the following URL: https://dashboard.umbrella.com
-
You can find your username after Admin in the navigation tree. Confirm that your organization appears under your username.
Create Umbrella API Key
Create an Umbrella API key ID and key secret.
Note: You have only one opportunity to copy your API secret. Umbrella does not save your API secret and you cannot retrieve the secret after its initial creation.
Refresh Umbrella API Key
Refresh an Umbrella API key ID and key secret.
Note: You have only one opportunity to copy your API secret. Umbrella does not save your API secret and you cannot retrieve the secret after its initial creation.
Update Umbrella API Key
Update an Umbrella API key.
Cisco Secure Endpoint Integration Procedures
Please provide the following information to CyTech:
Collect logs from the Cisco Umbrella
-
Queue URL - URL of the AWS SQS queue that messages will be received from. For Cisco Managed S3 buckets or S3 without SQS, use Bucket ARN.
-
Bucket ARN - Required for Cisco Managed S3. If the S3 bucket does not use SQS, this is the address for the S3 bucket, one example is arn:aws:s3:::cisco-managed-eu-central-1 For a list of Cisco Managed buckets, please see https://docs.umbrella.com/mssp-deployment/docs/enable-logging-to-a-cisco-managed-s3-bucket.
-
Bucket Region - Required for Cisco Managed S3. The region the bucket is located in.
-
Bucket List Prefix - Required for Cisco Managed S3. This sets the root folder of the S3 bucket that should be monitored, found in the S3 Web UI. Example value: 1235_654vcasd23431e5dd6f7fsad457sdf1fd5.
-
Number of Workers - Required for Cisco Managed S3. Number of workers that will process the S3 objects listed. Minimum is 1.
-
Bucket List Interval - Time interval for polling listing of the S3 bucket. Defaults to 120s.
-
Access Key ID
-
Secret Access Key
If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.
Cloudflare Integration
Introduction
Cloudflare integration uses Cloudflare's API to retrieve audit logs and traffic logs from Cloudflare, for a particular zone, and ingest them into Elasticsearch. This allows you to search, observe and visualize the Cloudflare log events through Elasticsearch.
Users of Cloudflare use Cloudflare services to increase the security and performance of their web sites and services.
To enable the Cloudflare Logpush, please refer to Section 5. Currently, the procedures described is for the setup of Amazon S3.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
Configure Cloudflare audit logs data stream
Enter values "Auth Email", "Auth Key" and "Account ID".
-
-
-
Auth Email is the email address associated with your account.
-
Auth Key is the API key generated on the "My Account" page.
-
Account ID can be found on the Cloudflare dashboard. Follow the navigation documentation from here.
-
-
Configure Cloudflare logs
These logs contain data related to the connecting client, the request path through the Cloudflare network, and the response from the origin web server. For more information see here.
The integration can retrieve Cloudflare logs using -
-
-
-
Auth Email and Auth Key
-
API Token More information is available here.
-
-
CONFIGURE USING AUTH EMAIL AND AUTH KEY
Enter values "Auth Email", "Auth Key" and "Zone ID".
-
-
- Auth Email is the email address associated with your account.
- Auth Key is the API key generated on the "My Account" page.
- Zone ID can be found here.
-
CONFIGURE USING API TOKEN
Enter values "API Token" and "Zone ID".
For the Cloudflare integration to be able to successfully get logs the following permissions must be granted to the API token -
-
Account.Access: Audit Logs: Read
-
-
- API Tokens allow for more granular permission settings.
- Zone ID can be found here.
-
Logs
Audit
Audit logs summarize the history of changes made within your Cloudflare account. Audit logs include account-level actions like login and logout, as well as setting changes to DNS, Crypto, Firewall, Speed, Caching, Page Rules, Network, and Traffic features, etc.
Logpull
These logs contain data related to the connecting client, the request path through the Cloudflare network, and the response from the origin web server.
Cloudflare Integration Procedures
Please provide the following information to CyTech:
See the Screenshot Below
Audit Logs
-
Auth Email -
-
Auth Key
-
Account ID
Cloudflare Logs
-
Auth Token
-
Zone ID
Enable Logpush to Amazon S3
To enable the Cloudflare Logpush service:
-
Log in to the Cloudflare dashboard.
-
Select the Enterprise account or domain you want to use with Logpush.
-
Go to Analytics & Logs > Logs.
-
Select Add Logpush job. A modal window opens where you will need to complete several steps.
-
Select the dataset you want to push to a storage service.
-
Select the data fields to include in your logs. Add or remove fields later by modifying your settings in Logs > Logpush.
-
Select Amazon S3.
-
Enter or select the following destination information:
-
-
-
Bucket path
-
Daily subfolders
-
Bucket region
-
Encryption constraint in bucket policy
-
For Grant Cloudflare access to upload files to your bucket, make sure your bucket has a policy (if you did not add it already):
-
Copy the JSON policy, then go to your bucket in the Amazon S3 console and paste the policy in Permissions > Bucket Policy and click Save.
-
-
-
Click Validate access.
-
Enter the Ownership token (included in a file or log Cloudflare sends to your provider) and click Prove ownership. To find the ownership token, click the Open button in the Overview tab of the ownership challenge file.
-
Click Save and Start Pushing to finish enabling Logpush.
Once connected, Cloudflare lists Amazon S3 as a connected service under Logs > Logpush. Edit or remove connected services from here.
Crowdstrike Integrations
Introduction
This integration is for CrowdStrike products. It includes the following datasets for receiving logs:
falcon dataset consists of endpoint data and Falcon platform audit data forwarded from Falcon SIEM Connector.
fdr dataset consists of logs forwarded using the Falcon Data Replicator.
Assumptions
The procedures described in Section 3 assume that a Log Collector has already been setup.
Compatibility
This integration supports CrowdStrike Falcon SIEM-Connector-v2.0.
Requirements
Logs
Falcon
Contains endpoint data and CrowdStrike Falcon platform audit data forwarded from Falcon SIEM Connector.
FDR
The CrowdStrike Falcon Data Replicator (FDR) allows CrowdStrike users to replicate FDR data from CrowdStrike managed S3 buckets. CrowdStrike writes notification events to a CrowdStrike managed SQS queue when new data is available in S3.
This integration can be used in two ways. It can consume SQS notifications directly from the CrowdStrike managed SQS queue or it can be used in conjunction with the FDR tool that replicates the data to a self-managed S3 bucket and the integration can read from there.
In both cases SQS messages are deleted after they are processed. This allows you to operate more than one Elastic Agent with this integration if needed and not have duplicate events, but it means you cannot ingest the data a second time.
CrowdStrike Integration Procedures
Please provide the following information to CyTech:
Collect CrowdStrike Falcon Data Replicator logs (input: aws-s3) Option 1
-
AWS: Access Key ID
-
AWS: Secret Access Key
-
AWS: Queue URL - URL of the AWS SQS queue that messages will be received from.
Collect CrowdStrike logs via API. Option 2 (Recommended)
-
Client ID: Client ID for the CrowdStrike.
-
Client Secret: Client Secret for the CrowdStrike.
-
URL: Token URL of CrowdStrike.
Dropbox Integrations
Introduction
Connecting Dropbox
Use the Workplace Search Dropbox connector to automatically capture, sync and index the following items from your Dropbox service:
Stored Files
Including ID, File Metadata, File Content, Updated by, and timestamps.
Dropbox Paper
Including ID, Metadata, Content, Updated by, and timestamps.
This document helps you configure and connect your Dropbox service to Workplace Search. To do this, you must register your Elastic deployment in the Dropbox Developer platform, by creating an OAuth 2.0 app. This gives your app permission to access Dropbox data. You will need your Workplace Search OAuth redirect URL, so have that handy.
If you need a primer on OAuth, read the official OAuth 2.0 authorization flow RFC. The diagrams provide a good mental model of the protocol flow. Dropbox also has its own OAuth guide.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
Configuring the Dropbox Connector
-
To register your Elastic deployment with Dropbox, create a new OAuth app in your organization’s Dropbox developer platform. Make sure to use a trusted and stable Dropbox account.
-
Provide basic information about the app and define the access scopes, or specific permissions, the app needs to interact with Dropbox. A good rule of thumb is that these should be read-only permissions. Choose only the following permissions:
-
-
-
files.content.read
-
sharing.read
-
account_info.read
-
files.metadata.read
-
-
To fetch document-level permissions, you must create an OAuth App using a team-owned (Dropbox Business) account. Enable document-level permissions by adding the following permissions:
-
-
-
team_info.read
-
team_data.member
-
team_data.team_space
-
members.read
-
-
-
Register a redirect URL for the app to use. This is where the OAuth 2.0 service will return the user after they authorize the application. This is the Workplace Search OAuth redirect URL for your deployment. This must be a https endpoint for production use cases. Only use http for local development.
-
Find and record the client_id and client_secret for the app. Dropbox calls these App Key and App Secret.
-
Switch back to your organization’s Workplace Search administrative dashboard
-
In the Sources —> Add Source tab, add a new Dropbox source
-
Configure the service with Workplace Search using the App Key and App Secret.
Your Dropbox service is now configured, and you can connect it to Workplace Search.
Dropbox Integration Procedures
Connecting Dropbox to Workplace Search
Once the Dropbox connector is configured, you can connect a Dropbox instance to your organization’s Workplace Search deployment.
-
Follow the Dropbox authentication flow as presented in the Workplace Search administrative dashboard.
-
If the authentication flow succeeds, you will be redirected to Workplace Search.
Your Dropbox content becomes searchable as soon as syncing starts. Once configured and connected, Dropbox synchronizes automatically every 2 hours.
Limiting the content to be indexed
If you don’t need to index all available content, you can use the API to apply indexing rules. This reduces indexing load and overall index size. See Customizing indexing.
The path_template and file_extension rules are applicable for Dropbox.
Synchronized fields
The following table lists the fields synchronized from the connected source to Workplace Search. The attributes in the table apply to the default search application, as follows:
-
Display name - The label used when displayed in the UI
-
Field name - The name of the underlying field attribute
-
Faceted filter - whether the field is a faceted filter by default, or can be enabled (see also: Customizing filters)
-
Automatic query refinement preceding phrases - The default list of phrases that must precede a value of this field in a search query in order to automatically trigger query refinement. If "None," a value from this field may trigger refinement regardless of where it is found in the query string. If '', a value from this field must be the first token(s) in the query string. If N.A., automatic query refinement is not available for this field by default. All fields that have a faceted filter (default or configurable) can also be configured for automatic query refinement; see also Update a content source, Get a content source’s automatic query refinement details and Customizing filters.
F5 Integrations
Introduction
This document shows information related to F5 Integration.
The F5 BIG-IP integration allows users to monitor LTM, AFM, APM, ASM, and AVR activity. F5 BIG-IP covers software and hardware designed around application availability, access control, and security solutions.
The F5 BIG-IP integration can be used in three different modes to collect data:
HTTP Endpoint mode - F5 BIG-IP pushes logs directly to an HTTP endpoint hosted by users’ Elastic Agent.
AWS S3 polling mode - F5 BIG-IP writes data to S3 and Elastic Agent polls the S3 bucket by listing its contents and reading new files.
AWS S3 SQS mode - F5 BIG-IP writes data to S3, S3 pushes a new object notification to SQS, Elastic Agent receives the notification from SQS, and then reads the S3 object. Multiple Agents can be used in this mode.
For example, users can use the data from this integration to analyze the traffic that passes through their F5 BIG-IP network.
Data streams
The F5 BIG-IP integration collects one type of data stream: log.
Log help users to keep a record of events happening on the network using telemetry streaming. The log data stream collected by the F5 BIG-IP integration includes events that are related to network traffic. See more details in the Logs.
This integration targets the five types of events as mentioned below:
LTM provides the platform for creating virtual servers, performance, service, protocol, authentication, and security profiles to define and shape users’ application traffic. For more information, refer to the link here.
AFM is designed to reduce the hardware and extra hops required when ADC's are paired with traditional firewalls and helps to protect traffic destined for the user's data center. For more information, refer to the link here.
APM provides federation, SSO, application access policies, and secure web tunneling and allows granular access to users' various applications, virtualized desktop environments, or just go full VPN tunnel. For more information, refer to the link here.
ASM is F5's web application firewall (WAF) solution. It allows users to tailor acceptable and expected application behavior on a per-application basis. For more information, refer to the link here.
AVR provides detailed charts and graphs to give users more insight into the performance of web applications, with detailed views on HTTP and TCP stats, as well as system performance (CPU, memory, etc.). For more information, refer to the link here.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
Elasticsearch is needed to store and search data, and Kibana is needed for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your hardware.
The reference link for requirements of telemetry streaming is here.
-
https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/prereqs.html
The reference link for requirements of Application Services 3(AS3) Extension is here.
-
https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/userguide/prereqs.html
This module has been tested against F5 BIG-IP version 16.1.0, Telemetry Streaming version 1.32.0 and AS3 version 3.40.0.
Setup
To collect LTM, AFM, APM, ASM, and AVR data from F5 BIG-IP, the user has to configure modules in F5 BIG-IP as per the requirements.
To set up the F5 BIG-IP environment, users can use the BIG-IP system browser-based Configuration Utility or the command line tools that are provided. For more information related to the configuration of F5 BIG-IP servers, refer to F5 support website here.
https://support.f5.com/csp/knowledge-center/software
Configuration of Telemetry Streaming in F5
For downloading and installing Telemetry Streaming, refer to the link here.
https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/installation.html
Telemetry Streaming will send logs in the JSON format to the destination. Telemetry Streaming is compatible with BIG-IP versions 13.0 and later. Users have to prepare F5 servers for it and set up the Telemetry Streaming Consumer.
To use telemetry streaming, user have to send POST request on https://<BIG-IP>/mgmt/shared/telemetry/declare for declaration.
F5 BIG-IP modules named LTM, AFM, ASM, and APM are not configured by Telemetry Streaming, they must be configured with AS3 or another method. Reference link for setup AS3 extension in F5 BIG-IP is here.
To configure logging using AS3, refer to the link here.
To collect data from AWS S3 Bucket, follow the below steps:
- Create an Amazon S3 bucket. Refer to the link here.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html
- The default value of the "Bucket List Prefix" is listed below. However, the user can set the parameter "Bucket List Prefix" according to the requirement.
To collect data from AWS SQS, follow the below steps:
- If data forwarding to an AWS S3 Bucket hasn't been configured, then first set up an AWS S3 Bucket as mentioned in the above documentation.
- To set up an SQS queue, follow "Step 1: Create an Amazon SQS queue" mentioned in the Documentation. https://docs.aws.amazon.com/AmazonS3/latest/userguide/ways-to-add-notification-config-to-bucket.html
-
-
While creating an SQS Queue, please provide the same bucket ARN that has been generated after creating an AWS S3 Bucket.
-
- Set up event notifications for an S3 bucket. Follow this link. https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications.html
-
- Users have to set the prefix parameter the same as the S3 Bucket List Prefix as created earlier. (for example, log/ for a log data stream.)
- Select the event type as s3:ObjectCreated:*, select the destination type SQS Queue, and select the queue that has been created in Step 2.
Note:
- Credentials for the above AWS S3 and SQS input types should be configured using the link. https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-s3.html#aws-credentials-config
- Data collection via AWS S3 Bucket and AWS SQS are mutually exclusive in this case.
Enabling the integration in Elastic
-
In Kibana go to Management > Integrations.
-
In the "Search for integrations" search bar, type F5 BIG-IP.
-
Click on F5 BIG-IP integration from the search results.
-
Click on the Add F5 BIG-IP button to add F5 BIG-IP integration.
-
Enable the Integration to collect logs via AWS S3 or HTTP endpoint input.
F5 BIG-IP integration
The "Integration name" and either the " Description" and the following will need to be provided in the Configure integration when adding the F5 BIG-IP integration.
Procedures:
Please provide the following information to CyTech:
Collect F5 BIG-IP logs via HTTP Endpoint:
-
Listen Address - The bind address to listen for http endpoint connections. Set to 0.0.0.0 to bind to all available interfaces.
F5 BIG-IP logs via HTTP Endpoint:
-
Listen Port - The port number the listener binds to.
Fortinet-Fortigate Integrations
Introduction
This integration is for Fortinet FortiGate logs sent in the syslog format.
Pre-requisite:
Configure syslog on FortiGate
From the GUI:
- Log into FortiGate.
- Select Log & Report to expand the menu.
- Select Log Settings.
-
Toggle Send Logs to Syslog to Enabled.
-
Enter the Syslog Collector IP address. Note: IP Address must be host's IP Address where the Elastic-Agent is installed. (For example. 192.168.1.19 as shown below)
If it is necessary to customize the port or protocol or setup the Syslog from the CLI below are the commands:
config log syslogd setting
set status enable
set server "192.168.1.19" -- change IP Address to same as host's where Elastic Agent is installed
set mode udp
set port 514
end
To establish the connection to the Syslog Server using a specific Source IP Address, use the below CLI configuration:
config log syslogd setting
set status enable
set server "192.168.1.19" -- change ip address to match host's IP
set source-ip "172.16.1.1" -- change ip address to match host's source-ip address
set mode udp
set port 514
end
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Compatibility
This integration has been tested against FortiOS versions 6.x and 7.x up to 7.4.1. Newer versions are expected to work but have not been tested.
Note
- When using the TCP input, be careful with the configured TCP framing. According to the Fortigate reference, framing should be set to
rfc6587
when the syslog mode is reliable.
Fortinet FortiGate Integration Procedures
Please provide the following information to CyTech:
Collect Fortinet FortiGate logs (input: tcp)
-
Listen Address - The bind address to listen for TCP connections.
-
Listen Port - The TCP port number to listen on.
Collect Fortinet FortiGate logs (input: udp)
-
Listen Address - The bind address to listen for UDP connections.
-
Listen Port - The UDP port number to listen on.
If you need further assistance, kindly contact our support at info@cytechint.com for prompt assistance and guidance.
GCP Integrations
Introduction
This document shows information related to GCP Integration.
The Google Cloud integration collects and parses Google Cloud Audit Logs, VPC Flow Logs, Firewall Rules Logs and Cloud DNS Logs that have been exported from Cloud Logging to a Google Pub/Sub topic sink and collects Google Cloud metrics and metadata from Google Cloud Monitoring.
Requirements
To use this Google Cloud Platform (GCP) integration, the following needs to be set up:
-
Service Account with a Role. (Section 3.1.1)
-
Service Account Key to access data on your GCP project. (Section 3.1.2)
Service Accounts
First, please create a Service Account. A Service Account (SA) is a particular type of Google account intended to represent a non-human user who needs to access the GCP resources.
The Log Collector uses the SA (Service Account) to access data on Google Cloud Platform using the Google APIs.
Here is a reference from Google related to Service Accounts:
https://cloud.google.com/iam/docs/best-practices-for-securing-service-accounts
Service Account with a Role
You need to grant your Service Account (SA) access to Google Cloud Platform resources by assigning a role to the account. In order to assign minimal privileges, create a custom role that has only the privileges required by the Log Collector. Those privileges are below:
-
compute.instances.list (required for GCP Compute instance metadata collection) (**2)
-
monitoring.metricDescriptors.list
-
monitoring.timeSeries.list
-
pubsub.subscriptions.consume
-
pubsub.subscriptions.create (*1)
-
pubsub.subscriptions.get
-
pubsub.topics.attachSubscription (*1)
*1 Only required if Agent is expected to create a new subscription. If you create the subscriptions yourself, you may omit these privileges.
**2 Only required if corresponding collection will be enabled.
After you have created the custom role, assign the role to your service account.
Service Account Key
Next, with the Service Account (SA) with access to Google Cloud Platform (GCP) resources setup in Section 3.1.1, you need some credentials to associate with it: a Service Account Key.
From the list of SA (Service Accounts):
-
Click the one you just created to open the detailed view.
-
From the Keys section, click "Add key" > "Create new key" and select JSON as the type.
-
Download and store the generated private key securely (remember, the private key can't be recovered from GCP if lost).
GCP Integrations Procedures
-
GCP Audit Logs
The audit dataset collects audit logs of administrative activities and accesses within your Google Cloud resources.
Procedures
The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration.
Please provide the following information to CyTech:
-
Project ID
The Project ID is the Google Cloud project ID where your resources exist.
-
Credentials File vs JSON
Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field.
OPTION 1: CREDENTIALS FILE
Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file.
Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json.
OPTION 2: CREDENTIALS JSON
Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration.
-
GCP DNS Logs
The dns dataset collects queries that name servers resolve for your Virtual Private Cloud (VPC) networks, as well as queries from an external entity directly to a public zone.
Procedures
The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration.
Please provide the following information to CyTech:
- Project ID
The Project ID is the Google Cloud project ID where your resources exist.
-
Credentials File vs JSON
Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field.
OPTION 1: CREDENTIALS FILE
Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file.
Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json.
OPTION 2: CREDENTIALS JSON
Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration.
-
GCP Firewall Logs
The firewall dataset collects logs from Firewall Rules in your Virtual Private Cloud (VPC) networks.
Procedures
The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration.
Please provide the following information to CyTech:
-
Project ID
The Project ID is the Google Cloud project ID where your resources exist.
-
Credentials File vs JSON
Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field.
OPTION 1: CREDENTIALS FILE
Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file.
Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json.
OPTION 2: CREDENTIALS JSON
Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration.
-
GCP VPC Flow Logs
The vpcflow dataset collects logs sent from and received by VM instances, including instances used as GKE nodes.
Procedures
The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration.
Please provide the following information to CyTech:
-
Project ID
The Project ID is the Google Cloud project ID where your resources exist.
-
Credentials File vs JSON
Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field.
OPTION 1: CREDENTIALS FILE
Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file.
Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json.
OPTION 2: CREDENTIALS JSON
Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration.
GitLab Integrations
Introduction
Introduced in GitLab Starter 8.4. Support for Amazon Elasticsearch was introduced in GitLab Starter 9.0.
This document describes how to set up Elasticsearch with GitLab. Once enabled, you'll have the benefit of fast search response times and the advantage of two special searches:
-
Advance Global Search
-
Advanced Syntax Search
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
Elasticsearch 6.0+ is not supported currently. We will support 6.0+ in the future.
GitLabIntegration Procedures
Procedures:
Please provide the following information to CyTech:
Elasticsearch is not included in the Omnibus packages. You will have to install it yourself whether you are using the Omnibus package or installed GitLab from source. Providing detailed information on installing Elasticsearch is out of the scope of this document.
Once the data is added to the database or repository and Elasticsearch is enabled in the admin area the search index will be updated automatically. Elasticsearch can be installed on the same machine as GitLab, or on a separate server, or you can use the Amazon Elasticsearch service.
You can follow the steps as described in the official web site or use the packages that are available for your OS.
Google Workspace Integrations
Introduction
Google Workspace (formerly G Suite) is a suite of cloud computing, productivity and collaboration tools, software and products developed and marketed by Google. It allows users to create, edit, and share documents, spreadsheets, presentations, and more. It also includes email, calendar, chat, and video conferencing tools.
Google Workspace is designed for businesses of all sizes, from small businesses to large enterprises. It is a popular choice for businesses because it is affordable, easy to use, and secure.
The Google Workspace integration collects and parses data from the different Google Workspace audit reports APIs.
If you want to know more about how you can fully leverage the Google Workspace integration, there is a multipart blog from our Security Labs that will help you:
-
To understand what Google Workspace is in Part One - Surveying the Land
-
To set it up, step by step, in Part Two - Setup Threat Detection with Elastic
-
And to use the collected information to your advantage in Part Three - Detecting Common Threats
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Compatibility
It is compatible with a subset of applications under the Google Reports API v1.
Requirements
The procedures described in Section 3 assumes that a Log Collector has already been setup.
In order to ingest data from the Google Reports API you must:
-
Have an administrator account.
-
Set up a ServiceAccount using the administrator account.
-
Set up access to the Admin SDK API for the ServiceAccount.
-
Enable Domain-Wide Delegation for your ServiceAccount.
Create access credentials
Credentials are used to obtain an access token from Google's authorization servers so your app can call Google Workspace APIs. This guide describes how to choose and set up the credentials your app needs.
Choose the access credential that is right for you
The required credentials depends on the type of data, platform, and access methodology of your app. There are three types of credential types available:
-
API key credentials - An API key is a long string containing upper and lower case letters, numbers, underscores, and hyphens, such as AIzaSyDaGmWKa4JsXZ-HjGw7ISLn_3namBGewQe. This authentication method is used to anonymously access publicly-available data, such as Google Workspace files shared using the "Anyone on the Internet with this link" sharing setting. For more details, see Using API keys. To create an API key:
-
-
In the Google Cloud console, go to Menu > APIs & Services > Credentials.
-
Click Create credentials > API key.
-
Your new API key is displayed.
-
-
-
-
Click Copy to copy your API key for use in your app's code. The API key can also be found in the "API keys" section of your project's credentials.
-
Click Restrict key to update advanced settings and limit use of your API key. For more details, see Applying API key restrictions.
-
-
-
OAuth client ID credentials - To authenticate as an end user and access user data in your app, you need to create one or more OAuth 2.0 Client IDs. A client ID is used to identify a single app to Google's OAuth servers. If your app runs on multiple platforms, you must create a separate client ID for each platform. Choose your application type for specific instructions about how to create an OAuth client ID:
-
-
-
-
-
Web application
-
Android
-
iOS
-
Chrome app
-
Desktop app
-
TVs & limited-input devices
-
Universal Windows Platform (UWP)
-
-
-
-
-
Service account credentials - A service account is a special kind of account used by an application, rather than a person. You can use a service account to access data or perform actions by the robot account, or to access data on behalf of Google Workspace or Cloud Identity users. For more information, see Understanding service accounts. Create a service account:
-
-
-
-
-
In the Google Cloud console, go to Menu > IAM & Admin > Service Accounts.
-
Click Create service account.
-
Fill in the service account details, then click Create and continue.
-
Note: By default, Google creates a unique service account ID. If you would like to change the ID, modify the ID in the service account ID field.
-
Optional: Assign roles to your service account to grant access to your Google Cloud project's resources. For more details, refer to Granting, changing, and revoking access to resources.
-
Click Continue.
-
Optional: Enter users or groups that can manage and perform actions with this service account. For more details, refer to Managing service account impersonation.
-
Click Done. Make a note of the email address for the service account.
-
-
-
-
Assign a role to a service account:
You must assign a prebuilt or custom role to a service account by a super administrator account.
-
In the Google Admin console, go to Menu > Account > Admin roles.
-
Point to the role that you want to assign, and then click Assign admin.
-
Click Assign service accounts.
-
Enter the email address of the service account.
-
Click Add > Assign role.
Create credentials for a service account:
You need to obtain credentials in the form of a public/private key pair. These credentials are used by your code to authorize service account actions within your app.
To obtain credentials for your service account:
-
In the Google Cloud console, go to Menu > IAM & Admin > Service Accounts.
-
Select your service account.
-
Click Keys > Add key > Create new key.
-
Select JSON, then click Create.
Your new public/private key pair is generated and downloaded to your machine as a new file. Save the downloaded JSON file as credentials.json in your working directory. This file is the only copy of this key. For information about how to store your key securely, see Managing service account keys.
-
Click Close.
Optional: Set up domain-wide delegation for a service account
To call APIs on behalf of users in a Google Workspace organization, your service account needs to be granted domain-wide delegation of authority in the Google Workspace Admin console by a super administrator account. For more information, see Delegating domain-wide authority to a service account.
To set up domain-wide delegation of authority for a service account:
-
In the Google Cloud console, go to Menu > IAM & Admin > Service Accounts.
-
Select your service account.
-
Click Show advanced settings.
-
Under "Domain-wide delegation," find your service account's "Client ID." Click Copy to copy the client ID value to your clipboard.
-
If you have super administrator access to the relevant Google Workspace account, click View Google Workspace Admin Console, then sign in using a super administrator user account and continue following these steps.
If you don't have super administrator access to the relevant Google Workspace account, contact a super administrator for that account and send them your service account's Client ID and list of OAuth Scopes so they can complete the following steps in the Admin console.
-
-
-
-
In the Google Admin console, go to Menu > Security > Access and data control > API controls.
-
-
-
-
-
Click Manage Domain Wide Delegation.
-
-
-
Click Add new.
-
-
-
In the "Client ID" field, paste the client ID that you previously copied.
-
-
-
In the "OAuth Scopes" field, enter a comma-delimited list of the scopes required by your application. This is the same set of scopes you defined when configuring the OAuth consent screen.
-
-
-
Click Authorize.
-
This integration will make use of the following oauth2 scope:
-
https://www.googleapis.com/auth/admin.reports.audit.readonly
Once you have downloaded your service account credentials as a JSON file, you are ready to set up your integration.
Click the Advanced option of Google Workspace Audit Reports. The default value of "API Host" is https://www.googleapis.com. The API Host will be used for collecting access_transparency, admin, device, context_aware_access, drive, gcp, groups, group_enterprise, login, rules, saml, token and user accounts logs.
NOTE: The Delegated Account value in the configuration, is expected to be the email of the administrator account, and not the email of the ServiceAccount.
Google Workspace Integration Procedures
Please provide the following information to CyTech:
Collect access_transparency, admin, alert, context_aware_access, device, drive, gcp, groups, group_enterprise, login, rules, saml, token and user accounts logs (input: httpjson).
-
Jwt File - Specifies the path to the JWT credentials file. NOTE: Please use either JWT File or JWT JSON parameter.
-
Jwt JSON - Raw contents of the JWT file. Useful when hosting a file along with the agent is not possible. NOTE: Please use either JWT File or JWT JSON parameter.
-
Delegated Account - Delegated Account is required. Email of the admin user used to access the API.
Jumpcloud Integrations
Introduction
The JumpCloud integration allows you to monitor events related to the JumpCloud Directory as a Service via the Directory Insights API.
You can find out more about JumpCloud and JumpCloud Directory Insights here
Data streams
A single data stream named "jumpcloud.events" is used by this integration.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
An Elastic Stack with an Elastic Agent is a fundamental requirement.
An established JumpCloud tenancy with active users is the the other requirement. Basic Directory Insights API access is available to all subscription levels.
NOTE: The lowest level of subscription currently has retention limits, with access to Directory Insights events for the last 15 days at most. Other subscriptions levels provide 90 days or longer historical event access.
A JumpCloud API key is required, the JumpCloud documentation describing how to create one is here
This JumpCloud Directory Insights API is documented here
JumpCloud Integration Procedures
Procedures:
Please provide the following information to CyTech:
Enabling the integration in Elastic
-
In Kibana go to Management > Integrations
-
In "Search for integrations" search bar type JumpCloud
-
Click on "JumpCloud" integration from the search results.
-
Click on Add JumpCloud button to add the JumpCloud integration.
-
Configure the integration as appropriate
-
Assign the integration to a new Elastic Agent host, or an existing Elastic Agent host
Mimecast Integrations
Introduction
The Mimecast integration collects events from the Mimecast API.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
Configuration
Note: Rate limit quotas may require you to set up different credentials for the different available log types.
Logs
Audit Events
This is the mimecast.audit_events dataset. These logs contain Mimecast audit events with the following details: audit type, event category and detailed information about the event. More information about these logs.
DLP Logs
This is the mimecast.dlp_logs dataset. These logs contain information about messages that triggered a DLP or Content Examination policy. More information about these logs.
SIEM Logs
This is the mimecast.siem_logs dataset. These logs contain information about messages that contains MTA (message transfer agent) log – all inbound, outbound, and internal messages.
Threat Intel Feed Malware: Customer
This is the mimecast.threat_intel_malware_customer dataset. These logs contain information about messages that return identified malware threats at a customer level. Learn more about these logs.
Threat Intel Feed Malware: Grid
This is the mimecast.threat_intel_malware_grid dataset. These logs contain information about messages that return identified malware threats at a regional grid level. More about these logs.
TTP Attachment Logs
This is the mimecast.ttp_ap_logs dataset. These logs contain Mimecast TTP attachment protection logs with the following details: result of attachment analysis (if it is malicious or not etc.), date when file is released, sender and recipient address, filename and type, action triggered for the attachment, the route of the original email containing the attachment and details. Learn more about these logs.
TTP Impersonation Logs
This is the mimecast.ttp_ip_logs dataset. These logs contain information about messages containing information flagged by an Impersonation Protection configuration.
TTP URL Logs
This is the mimecast.ttp_url_logs dataset. These logs contain Mimecast TTP attachment protection logs with the following details: the category of the URL clicked, the email address of the user who clicked the link, the url clicked, the action taken by the user if user awareness was applied, the route of the email that contained the link, the action defined by the administrator for the URL, the date that the URL was clicked, url scan result, the action that was taken for the click, the description of the definition that triggered the URL to be rewritten by Mimecast, the action requested by the user, an array of components of the message where the URL was found. More about these logs.
Mimecast Integration Procedures
Please provide the following information to CyTech:
Mimecast API
-
Application Key - Specifies application key for user.
-
Application ID - Set the Application Id.
-
Access Key - Set Access Key.
-
Secret Key - Set Secret Key.
MongoDB Integrations
Introduction
This integration is used to fetch logs and metrics from MongoDB.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Compatibility
The log dataset is tested with logs from versions v3.2.11 and v4.4.4 in plaintext and json formats. The collstats, dbstats, metrics, replstatus and status datasets are tested with MongoDB 3.4 and 3.0 and are expected to work with all versions >= 2.8.
Requirements
MongoDB Privileges
In order to use the metrics datasets, the MongoDB user specified in the package configuration needs to have certain privileges.
We recommend using the clusterMonitor role to cover all the necessary privileges.
You can use the following command in Mongo shell to create the privileged user (make sure you are using the admin db by using db command in Mongo shell).
db.createUser(
{
user: "beats",
pwd: "pass",
roles: ["clusterMonitor"]
}
)
You can use the following command in Mongo shell to grant the role to an existing user (make sure you are using the admin db by using db command in Mongo shell).
db.grantRolesToUser("user", ["clusterMonitor"])
Logs
log
The log dataset collects the MongoDB logs.
Metrics
collstats
The collstats dataset uses the top administrative command to return usage statistics for each collection. It provides the amount of time, in microseconds, used and a count of operations for the following types: total, readLock, writeLock, queries, getmore, insert, update, remove, and commands.
It requires the following privileges, which is covered by the clusterMonitor role:
-
top action on cluster resource
dbstats
The dbstats dataset collects storage statistics for a given database.
It requires the following privileges, which is covered by the clusterMonitor role:
-
listDatabases
action on cluster resource
-
for each of the databases, also need dbStats
action on the database resource
metrics
It requires the following privileges, which is covered by the clusterMonitor role:
-
serverStatus
action on cluster resource
replstatus
The replstatus dataset collects status of the replica set. It requires the following privileges, which is covered by the clusterMonitor role:
-
find/listCollections action on the local database resource
-
collStats action on the local.oplog.rs collection resource
-
replSetGetStatus action on cluster resource
status
The status returns a document that provides an overview of the database's state.
It requires the following privileges, which is covered by the clusterMonitor role:
-
serverStatus
action on cluster resource
MongoDB Integration Procedures
Please provide the following information to CyTech:
Collect MongoDB application logs
- Paths
Collect MongoDB metrics
- Hosts
Ex: localhost:27017
OKTA Integrations
Introduction
The Okta integration collects events from the Okta API, specifically reading from the Okta System Log API.
Logs
System
The Okta System Log records system events related to your organization in order to provide an audit trail that can be used to understand platform activity and to diagnose problems. This module is implemented using the httpjson input and is configured to paginate through the logs while honoring any rate-limiting headers sent by Okta.
Okta Integration Procedures
Please provide the following information to CyTech:
Collect Okta logs via API
-
API Key - API Key is required.
-
Okta System Log API Url - Okta System Log API Url is required.
Pulse Connect Secure Integrations
Introduction
This integration is for Pulse Connect Secure.
Pulse Connect Secure Integration Procedures
Please provide the following information to CyTech:
Collect Pulse Connect Secure logs (input: udp)
-
Syslog Host
-
Syslog Port
Collect Pulse Connect Secure logs (input: tcp)
-
Syslog Host
-
Syslog Port
Slack Integrations
Introduction
Slack is used by numerous organizations as their primary chat and collaboration tool.
Please note the Audit Logs API is only available to Slack workspaces on an Enterprise Grid plan. These API methods will not work for workspaces on a Free, Standard, or Business+ plan.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
Configuration
Enabling the integration in Elastic
-
In Kibana go to Management > Integrations
-
In the "Search for integrations" search bar type Slack.
-
Click on "Slack" integration from the search results.
-
Click on Add Slack button to add Slack integration.
Configure Slack audit logs data stream
Enter values "OAuth API Token".
CONFIGURE USING API TOKEN
For the Slack integration to be able to successfully get logs the following "User Token Scopes"" must be granted to the Slack App:
-
auditlogs:read
Logs
Audit
Audit logs summarize the history of changes made within the Slack Enterprise.
SLACK Integration Procedures
Please provide the following information to CyTech:
Collect Slack logs via API
-
API URL - The root URL for the API endpoints.
Slack Audit logs
-
OAuth API Token - The OAuth API Token used to authenticate with the Slack API
System Integrations
Introduction
The System integration allows you to monitor servers, personal computers, and more.
Use the System integration to collect metrics and logs from your machines. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue.
For example, if you wanted to be notified when less than 10% of the disk space is still available, you could install the System integration to send file system metrics to Elastic. Then, you could view real-time updates to disk space used on your system in Kibana's [Metrics System] Overview dashboard. You could also set up a new rule in the Elastic Observability Metrics app to alert you when the percent free is less than 10% of the total disk space.
Data streams
The System integration collects two types of data: logs and metrics.
Logs help you keep a record of events that happen on your machine. Log data streams collected by the System integration include application, system, and security events on machines running Windows and auth and syslog events on machines running macOS or Linux. See more details in the Logs reference.
-
https://aquila-elk.kb.us-east-1.aws.found.io:9243/app/integrations/detail/system-1.20.4/overview#logs-reference
Metrics give you insight into the state of the machine. Metric data streams collected by the System integration include CPU usage, load statistics, memory usage, information on network behavior, and more. See more details in the Metrics reference.
-
https://aquila-elk.kb.us-east-1.aws.found.io:9243/app/integrations/detail/system-1.20.4/overview#metrics-reference
You can enable and disable individual data streams. If all data streams are disabled and the System integration is still enabled, Fleet uses the default data streams.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Requirements
You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
Each data stream collects different kinds of metric data, which may require dedicated permissions to be fetched and which may vary across operating systems. Details on the permissions needed for each data stream are available in the Metrics reference.
-
https://aquila-elk.kb.us-east-1.aws.found.io:9243/app/integrations/detail/system-1.20.4/overview#metrics-reference
Setup
For step-by-step instructions on how to set up an integration, see the Getting started guide.
-
https://www.elastic.co/guide/en/welcome-to-elastic/current/getting-started-observability.html
Troubleshooting
Note that certain data streams may access /proc to gather process information, and the resulting ptrace_may_access() call by the kernel to check for permissions can be blocked by AppArmor and other LSM software, even though the System module doesn't use ptrace directly.
-
https://gitlab.com/apparmor/apparmor/wikis/TechnicalDoc_Proc_and_ptrace
In addition, when running inside a container the proc filesystem directory of the host should be set using system.hostfs setting to /hostfs.
Logs reference
Application
The Windows application data stream provides events from the Windows Application event log.
SUPPORTED OPERATING SYSTEMS
-
Windows
System Integration Procedures
Please provide the following information to CyTech:
Collect logs from System instances (Enable Yes/No)
System auth logs (log) (Enable Yes/No)
-
-
-
Collect System auth logs using log input.
-
-
-
Paths
-
Preserve original event (Enable Yes/No)
-
-
-
Preserves a raw copy of the original event, added to the field event.original.
-
-
-
Ignore events older than
-
-
-
If this option is specified, events that are older than the specified amount of time are ignored. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
-
-
Tags
-
Processors (Optional)
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System syslog logs (log)
-
Paths
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Ignore events older than
-
-
-
If this option is specified, events that are older than the specified amount of time are ignored. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
-
-
Synthetic source (Enable Yes/No)
Collect events from the Windows event log (Enable Yes/No)
-
Application (Enable Yes/No)
-
-
-
Collect Windows application logs.
-
-
-
Preserve original event (Enable Yes/No)
-
-
-
Preserves a raw copy of the original XML event, added to the field event.original
-
-
-
Event ID
-
-
-
A list of included and excluded (blocked) event IDs. The value is a comma-separated list. The accepted values are single event IDs to include (e.g. 4624), a range of event IDs to include (e.g. 4700-4800), and single event IDs to exclude (e.g. -4735). Limit 22 IDs.
-
-
-
Ignore events older than
-
-
-
If this option is specified, events that are older than the specified amount of time are ignored. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
-
-
Language ID
-
-
-
The language ID the events will be rendered in. The language will be forced regardless of the system language. A complete list of language IDs can be found https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c[here]. It defaults to 0, which indicates to use the system language. E.g.: 0x0409 for en-US
-
-
-
Tags
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
Security (Enable Yes/No)
-
Preserve original event (Enable Yes/No)
-
-
-
Preserves a raw copy of the original XML event, added to the field event.original
-
-
-
Event ID
-
-
-
A list of included and excluded (blocked) event IDs. The value is a comma-separated list. The accepted values are single event IDs to include (e.g. 4624), a range of event IDs to include (e.g. 4700-4800), and single event IDs to exclude (e.g. -4735). Limit 22 IDs.
-
-
-
Ignore events older than
-
-
-
If this option is specified, events that are older than the specified amount of time are ignored. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
-
-
Language ID
-
-
-
The language ID the events will be rendered in. The language will be forced regardless of the system language. A complete list of language IDs can be found https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c[here]. It defaults to 0, which indicates to use the system language. E.g.: 0x0409 for en-US
-
-
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System (Enable Yes/No)
-
Preserve original event (Enable Yes/No)
-
-
-
Preserves a raw copy of the original XML event, added to the field event.original
-
-
-
Event ID
-
-
-
A list of included and excluded (blocked) event IDs. The value is a comma-separated list. The accepted values are single event IDs to include (e.g. 4624), a range of event IDs to include (e.g. 4700-4800), and single event IDs to exclude (e.g. -4735). Limit 22 IDs.
-
-
-
Ignore events older than
-
-
-
If this option is specified, events that are older than the specified amount of time are ignored. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
-
-
-
Language ID
-
-
-
The language ID the events will be rendered in. The language will be forced regardless of the system language. A complete list of language IDs can be found https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-lcid/a9eac961-e77d-41a6-90a5-ce1a8b0cdb9c[here]. It defaults to 0, which indicates to use the system language. E.g.: 0x0409 for en-US
-
-
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
Collect metrics from System instances (Enable Yes/No)
-
Proc Filesystem Directory
-
-
-
The proc filesystem base directory.
-
-
-
Period
-
Core Metrics
-
-
-
How to report core metrics. Can be "percentages" or "ticks"
-
-
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System CPU metrics (Enable Yes/No)
-
Period
-
Cpu Metrics
-
-
-
How to report CPU metrics. Can be "percentages", "normalized_percentages", or "ticks"
-
-
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System diskio metrics (Enable Yes/No)
-
Period
-
Include Devices
-
-
-
Provide a specific list of devices to monitor. By default, all devices are monitored.
-
-
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System filesystem metrics
-
Period
-
List of filesystem types to ignore
-
-
-
The filesystem datastream will ignore any filesystems with a matching type as specified here. By default, this will exclude any filesystems marked as "nodev" in /proc/filesystems on linux.
-
-
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with external metadata.
-
-
-
Indexing settings (experimental)
-
-
-
Select data streams to configure indexing options. This is an experimental feature and may have effects on other properties.
-
-
-
Synthetic source (Enable Yes/No)
System fsstat metrics (Enable Yes/No)
-
Period
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with external metadata.
-
-
-
Tags
-
Synthetic source (Enable Yes/No)
System load metrics (Enable Yes/No)
-
Period
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System memory metrics (Enable Yes/No)
-
Period
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System network metrics (Enable Yes/No)
-
Period
-
Interfaces
-
-
-
List of interfaces to monitor. Will monitor all by default.
-
-
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System process metrics
-
Period
-
Process Include Top N By Cpu
-
-
-
Include the top N processes by CPU usage.
-
-
-
Process Include Top N By Memory
-
-
-
Include the top N processes by memory usage.
-
-
-
Enable cmdline cache (Enable Yes/No)
-
-
-
If false, cmdline of a process is not cached
-
-
-
Enable cgroup reporting (Enable Yes/No)
-
-
-
Enable collection of cgroup metrics from processes on Linux.
-
-
-
Env whitelist
-
-
-
A list of regular expressions used to whitelist environment variables reported with the process metricset's events. Defaults to empty.
-
-
-
Include CPU Ticks (Enable Yes/No)
-
-
-
Include the cumulative CPU tick values with the process metrics.
-
-
-
Processes
-
-
-
A glob to match reported processes. By default all processes are reported.
-
-
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System process_summary metrics (Enable Yes/No)
-
Period
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System socket_summary metrics (Enable Yes/No)
-
Period
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
System uptime metrics (Enable Yes/No)
-
Period
-
Tags
-
Processors
-
-
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See Processors for details.
-
-
-
Synthetic source (Enable Yes/No)
Collect logs from third-party REST API (experimental) (Enable Yes/No)
-
URL of Splunk Enterprise Server
-
-
-
i.e. scheme://host:port, path is automatic
-
-
-
Splunk REST API Username
-
Splunk REST API Password
-
Splunk Authorization Token
-
-
-
Bearer Token or Session Key, e.g. "Bearer eyJFd3e46..." or "Splunk 192fd3e...". Cannot be used with username and password.
-
-
-
Preserve original event (Enable Yes/No)
-
-
-
Preserves a raw copy of the original event, added to the field event.original
-
-
-
SSL Configuration
-
-
-
i.e. certificate_authorities, supported_protocols, verification_mode etc.
-
-
Windows Application Events via Splunk Enterprise REST API (Enable Yes/No)
-
Interval to query Splunk Enterprise REST API
-
-
-
Go Duration syntax (eg. 10s)
-
-
-
Splunk search string
-
Tags
-
Synthetic source (Enable Yes/No)
Windows Security Events via Splunk Enterprise REST API (Enable Yes/No)
-
Interval to query Splunk Enterprise REST API
-
-
-
Go Duration syntax (eg. 10s)
-
-
-
Splunk search string
-
Tags
-
Synthetic source (Enable Yes/No)
Windows System Events via Splunk Enterprise REST API (Enable Yes/No)
-
Interval to query Splunk Enterprise REST API
-
-
-
Go Duration syntax (eg. 10s)
-
-
-
Splunk search string
-
Tags
-
Synthetic source (Enable Yes/No)
Team Viewer Integrations
Remote File Copy via TeamViewer
Identifies an executable or script file remotely downloaded via a TeamViewer transfer session.
Rule type: eql
Rule indices:
-
winlogbeat-*
-
logs-endpoint.events.*
-
logs-windows.*
Severity: medium
Risk score: 47
Runs every: 5m
Searches indices from: now-9m (Date Math format, see also Additional look-back time)
Maximum alerts per execution: 100
References:
-
https://blog.menasec.net/2019/11/hunting-for-suspicious-use-of.html
-
Elastic
-
Host
-
Windows
-
Threat Detection
-
Command and Control
Version: 5
-
Elastic
Rule license: Elastic License v2
Rule query
file where event.type == "creation" and process.name : "TeamViewer.exe" and
file.extension : ("exe", "dll", "scr", "com", "bat", "ps1", "vbs", "vbe", "js", "wsh", "hta")
Framework: MITRE ATT&CKTM
-
Tactic:
-
-
-
Name: Command and Control
-
ID: TA0011
-
Reference URL: https://attack.mitre.org/tactics/TA0011/
-
-
-
Technique:
-
-
-
Name: Ingress Tool Transfer
-
ID: T1105
-
Reference URL: https://attack.mitre.org/techniques/T1105/
-
-
-
Technique:
-
-
-
Name: Remote Access Software
-
ID: T1219
-
Reference URL: https://attack.mitre.org/techniques/T1219/
-
-
Source: https://www.elastic.co/guide/en/security/master/prebuilt-rule-0-14-2-remote-file-copy-via-teamviewer.html
TeamViewer Integration Procedure
-
Install the Elastic Stack (Elasticsearch, Kibana, and Logstash) on your Ubuntu machine by following the instructions provided on the Elastic website.
-
Once you have installed and configured the Elastic Stack, navigate to the Logstash directory and create a new configuration file for the TeamViewer logs by running the command:
Copy and paste the following Logstash configuration into the file:
-
Save and close the file.
-
Start Logstash by running the command:
-
Ensure that Logstash is properly reading and processing the TeamViewer logs by checking the Logstash logs in the /var/log/logstash/ directory.
-
Navigate to the Kibana web interface by opening a web browser and entering the URL: http://localhost:5601/.
-
In Kibana, click on the "Discover" tab to view your logs.
-
Click on the "Create index pattern" button and enter the name of the TeamViewer index pattern (e.g. teamviewer-*).
-
Select the time range for the logs you want to view, and click on the "Create index pattern" button.
-
You should now see a list of logs from your TeamViewer deployment. You can filter the logs based on various criteria like severity, source, or date.
-
You can also create custom dashboards or visualizations to monitor specific aspects of your TeamViewer deployment, such as usage patterns or connection quality.
-
If you encounter any issues with your TeamViewer deployment, you can use the logs to identify the root cause and take corrective action.
Source: ChatGPT
-
Elastic official documentation: https://www.elastic.co/guide/index.html
-
Logstash input plugin documentation: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html
-
Logstash output plugin documentation: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
-
Kibana official documentation: https://www.elastic.co/guide/en/kibana/current/index.html
Z Scaler Integrations
Introduction
This integration is for Zscaler Internet Access logs. It can be used to receive logs sent by NSS log server on respective TCP ports.
The log message is expected to be in JSON format. The data is mapped to ECS fields where applicable and the remaining fields are written under zscaler_zia.<data-stream-name>.*.
Assumptions
The procedures described in Section 3 assumes that a Log Collector has already been setup.
Compatibility
This package has been tested against Zscaler Internet Access version 6.1
Requirements
Steps for setting up NSS Feeds
-
Enable the integration with the TCP input.
-
Configure the Zscaler NSS Server and NSS Feeds to send logs to the Elastic Agent that is running this integration. See Add NSS Server and Add NSS Feeds. Use the IP address hostname of the Elastic Agent as the 'NSS Feed SIEM IP Address/FQDN', and use the listening port of the Elastic Agent as the 'SIEM TCP Port' on the Add NSS Feed configuration screen. To configure Zscaler NSS Server and NSS Feeds follow the following steps.
-
In the ZIA Admin Portal, add an NSS Server.
-
Log in to the ZIA Admin Portal using your admin account. If you're unable to log in, contact Support.
-
Add an NSS server. Refer to Adding NSS Servers to set up an Add NSS Server for Web and/or Firewall.
-
Verify that the state of the NSS Server is healthy.
-
In the ZIA Admin Portal, go to Administration > Nanolog Streaming Service > NSS Servers.
-
In the State column, confirm that the state of the NSS server is healthy.
-
In the ZIA Admin Portal, add an NSS Feed.
-
Refer to Add NSS Feeds and select the type of feed you want to configure. The following fields require specific inputs:
-
SIEM IP Address: Enter the IP address of the Elastic agent you’ll be assigning the Zscaler integration to.
-
SIEM TCP Port: Enter the port number, depending on the logs associated with the NSS Feed. You will need to create an NSS Feed for each log type.
-
Alerts: 9010
-
DNS: 9011
-
Firewall: 9012
-
Tunnel: 9013
-
Web: 9014
-
- Feed Output Type: Select Custom in Feed output type and paste the appropriate response format in Feed output format as follows:
Steps for setting up Cloud NSS Feeds
-
Enable the integration with the HTTP Endpoint input.
-
Configure the Zscaler Cloud NSS Feeds to send logs to the Elastic Agent that is running this integration. Provide API URL to send logs to the Elastic Agent. To configure Zscaler Cloud NSS Feeds follow the following steps.
-
In the ZIA Admin Portal, add a Cloud NSS Feed.
-
Log in to the ZIA Admin Portal using your admin account.
-
Add a Cloud NSS Feed. See to Add Cloud NSS Feed.
-
In the ZIA Admin Portal, go to Administration > Nanolog Streaming Service > Cloud NSS Feeds.
-
Give Feed Name, change status to Enabled.
-
Select NSS Type.
-
Change SIEM Type to other.
-
Add an API URL.
-
Default ports:
-
DNS: 9556
-
Firewall: 9557
-
Tunnel: 9558
-
Web: 9559
-
-
Select JSON as feed output type.
-
Add same custom header along with its value on both the side for additional security.
-
Repeat step 2 for each log type.
Please make sure to use the given response formats for NSS and Cloud NSS Feeds.
Note: Please make sure to use latest version of given response formats.
Zscaler Integration Procedures
Please provide the following information to CyTech:
Collect Zscaler Internet Access logs via TCP input
-
-
Listen Address - The bind address to listen for TCP connections.
-
Types:
-
-
-
-
-
-
-
TCP Listen Port for Zscaler Internet Access Alerts
-
-
-
-
-
-
-
TCP Listen Port for Zscaler Internet Access DNS logs
-
TCP Listen Port for Zscaler Internet Access Firewall Logs
-
TCP Listen Port for Zscaler Internet Access Tunnel Logs
-
TCP Listen Port for Zscaler Internet Access Web Logs
-
Collect Zscaler Internet Access logs via HTTP Endpoint
-
-
Listen Address - The bind address to listen for http endpoint connections.
-
Types:
-
-
-
TCP Listen Port for Zscaler Internet Access DNS logs
-
TCP Listen Port for Zscaler Internet Access Firewall Logs
-
TCP Listen Port for Zscaler Internet Access Tunnel Logs
-
TCP Listen Port for Zscaler Internet Access Web Logs
-
gcp
Google Cloud Platform
Google Cloud Platform Integration
The Google Cloud integration collects and parses Google Cloud Audit Logs, VPC Flow Logs, Firewall Rules Logs and Cloud DNS Logs that have been exported from Cloud Logging to a Google Pub/Sub topic sink and collects Google Cloud metrics and metadata from Google Cloud Monitoring.
Authentication
To use this Google Cloud Platform (GCP) integration, you need to set up a Service Account with a Role and a Service Account Key to access data on your GCP project.
Service Account
First, you need to create a Service Account. A Service Account (SA) is a particular type of Google account intended to represent a non-human user who needs to access the GCP resources.
The Elastic Agent uses the SA to access data on Google Cloud Platform using the Google APIs.
If you haven't already, this might be a good moment to check out the best practices for securing service accounts guide.
Role
You need to grant your Service Account (SA) access to Google Cloud Platform resources by assigning a role to the account. In order to assign minimal privileges, create a custom role that has only the privileges required by Agent. Those privileges are:
compute.instances.list
(required for GCP Compute instance metadata collection) **monitoring.metricDescriptors.list
monitoring.timeSeries.list
pubsub.subscriptions.consume
pubsub.subscriptions.create
*pubsub.subscriptions.get
pubsub.topics.attachSubscription
*
* Only required if Agent is expected to create a new subscription. If you create the subscriptions yourself you may omit these privileges. ** Only required if corresponding collection will be enabled.
After you have created the custom role, assign the role to your service account.
Service Account Keys
Now, with your brand new Service Account (SA) with access to Google Cloud Platform (GCP) resources, you need some credentials to associate with it: a Service Account Key.
From the list of SA:
- Click the one you just created to open the detailed view.
- From the Keys section, click "Add key" > "Create new key" and select JSON as the type.
- Download and store the generated private key securely (remember, the private key can't be recovered from GCP if lost).
Configure the Integration Settings
The next step is to configure the general integration settings used for all logs from the supported services (Audit, DNS, Firewall, and VPC Flow).
The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration.
Project Id
The Project Id is the Google Cloud project ID where your resources exist.
Credentials File vs Json
Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field.
Option 1: Credentials File
Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file.
Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json
.
Option 2: Credentials JSON
Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration.
Recommendations
Elastic recommends using Credentials File, as in this method the credential information doesn’t leave your Google Cloud Platform environment. When using Credentials JSON, the integration stores the info in Elasticsearch, and the access is controlled based on policy permissions or access to underlying Elasticsearch data.
Logs Collection Configuration
With a properly configured Service Account and the integration setting in place, it's time to start collecting some logs.
Requirements
You need to create a few dedicated Google Cloud resources before starting, in detail:
- Log Sink
- Pub/Sub Topic
- Subscription
Elastic recommends separate Pub/Sub topics for each of the log types so that they can be parsed and stored in a specific data stream.
Here's an example of collecting Audit Logs using a Pub/Sub topic, a subscription, and a Log Router. We will create the resources in the Google Cloud Console and then configure the Google Cloud Platform integration.
On the Google Cloud Console
At a high level, the steps required are:
- Visit "Logging" > "Log Router" > "Create Sink" and provide a sink name and description.
- In "Sink destination", select "Cloud Pub/Sub topic" as the sink service. Select an existing topic or "Create a topic". Note the topic name, as it will be provided in the Topic field in the Elastic agent configuration.
- If you created a new topic, you must remember to go to that topic and create a subscription for it. A subscription directs messages on a topic to subscribers. Note the "Subscription ID", as it will need to be entered in the "Subscription name" field in the integration settings.
- Under "Choose logs to include in sink", for example add
logName:"cloudaudit.googleapis.com"
in the "Inclusion filter" to include all audit logs.
This is just an example; you will need to create your filter expression to select the log types you want to export to the Pub/Sub topic.
More example filters for different log types:
#
# VPC Flow: logs for specific subnet
#
resource.type="gce_subnetwork" AND
log_id("compute.googleapis.com/vpc_flows") AND
resource.labels.subnetwork_name"=[SUBNET_NAME]"
#
# Audit: Google Compute Engine firewall rule deletion
#
resource.type="gce_firewall_rule" AND
log_id("cloudaudit.googleapis.com/activity") AND
protoPayload.methodName:"firewalls.delete"
#
# DNS: all DNS queries
#
resource.type="dns_query"
#
# Firewall: logs for a given country
#
resource.type="gce_subnetwork" AND
log_id("compute.googleapis.com/firewall") AND
jsonPayload.remote_location.country=[COUNTRY_ISO_ALPHA_3]
Start working on your query using the Google Cloud Logs Explorer, so you can preview and pinpoint the exact log types you want to forward to your Elastic Stack.
To learn more, please read how to Build queries in the Logs Explorer, and take a look at the Sample queries using the Logs Explorer page in the Google Cloud docs.
On Kibana
Visit "Management" > "Integrations" > "Installed Integrations" > "Google Cloud Platform" and select the "Integration Policies" tab. Select the integration policy you previously created.
From the list of services, select "Google Cloud Platform (GCP) audit logs (gcp-pubsub)" and:
- On the "Topic" field, specify the "topic name" you noted before on the Google Cloud Console.
- On the "Subscription Name", specify the short subscription name you noted before on the Google Cloud Console (note: do NOT use the full-blown subscription name made of project/PROJECT_ID/subscriptions/SUBSCRIPTION_ID). Just pick the Subscription ID from the Google Cloud Console).
- Click on "Save Integration", and make sure the Elastic Agent gets the updated policy.
Troubleshooting
If you don't see Audit logs showing up, check the Agent logs to see if there are errors.
Common error types:
- Missing roles in the Service Account
- Misconfigured settings, like "Project Id", "Topic" or "Subscription Name" fields
Missing Roles in the Service Account
If your Service Account (SA) does not have the required roles, you might find errors like this one in the elastic_agent.filebeat
dataset:
failed to subscribe to pub/sub topic: failed to check if subscription exists: rpc error: code = PermissionDenied desc = User not authorized to perform this action.
Solution: make sure your SA has all the required roles.
Misconfigured Settings
If you specify the wrong "Topic field" or "Subscription Name", you might find errors like this one in the elastic_agent.filebeat
dataset:
[elastic_agent.filebeat][error] failed to subscribe to pub/sub topic: failed to check if subscription exists: rpc error: code = InvalidArgument desc = Invalid resource name given (name=projects/project/subscriptions/projects/project/subscriptions/non-existent-sub). Refer to https://cloud.google.com/pubsub/docs/admin#resource_names for more information.
Solution: double check the integration settings.
Metrics Collection Configuration
With a properly configured Service Account and the integration setting in place, it's time to start collecting some metrics.
Requirements
No additional requirement is needed to collect metrics.
Troubleshooting
If you don't see metrics showing up, check the Agent logs to see if there are errors.
Common error types:
- Period is lower than 60 seconds
- Missing roles in the Service Account
- Misconfigured settings, like "Project Id"
Period is lower than 60 seconds
Usual minimum collection period for GCP metrics is 60 seconds. Any value lower than that cause an error when retrieving the metric metadata. If an error happens, the affected metric is skipped at the metric collection stage, resulting in no data being sent.
Missing Roles in the Service Account
If your Service Account (SA) does not have required roles, you might find errors related to accessing GCP resources.
To check you may add Monitoring Viewer
and Compute Viewer
roles (built-in GCP roles) to your SA. These roles contain the permission added in the previous step and expand them with additional permissions. You can analyze additional missing permissions from the GCP Console > IAM > clicking on the down arrow near the roles on the same line of your SA > View analyzed permissions. From the shown table you can check which permissions from the role the SA is actively using. They should match what you configured in your custom role.
Misconfigured Settings
If you specify a wrong setting you will probably find errors related to missing GCP resources.
Make sure the settings are correct and the SA has proper permissions for the given "Project Id".
Logs
Audit
The audit
dataset collects audit logs of administrative activities and accesses within your Google Cloud resources.
Exported fields
Field | Description | Type |
---|---|---|
@timestamp
|
Event timestamp.
|
date
|
client.user.email
|
User email address.
|
keyword
|
client.user.id
|
Unique identifier of the user.
|
keyword
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
cloud.availability_zone
|
Availability zone in which this host is running.
|
keyword
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
cloud.region
|
Region in which this host is running.
|
keyword
|
container.id
|
Unique container id.
|
keyword
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
container.labels
|
Image labels.
|
object
|
container.name
|
Container name.
|
keyword
|
container.runtime
|
Runtime managing this container.
|
keyword
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
error.code
|
Error code describing the error.
|
keyword
|
error.message
|
Error message.
|
match_only_text
|
event.action
|
The action captured by the event. This describes the information in the event. It is more specific than
event.category . Examples are group-add , process-started , file-created . The value is normally defined by the implementer. |
keyword
|
event.created
|
event.created contains the date/time when the event was first read by an agent, or by your pipeline. This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event. In most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source. In case the two timestamps are identical, @timestamp should be used.
|
date
|
event.dataset
|
Event dataset
|
constant_keyword
|
event.id
|
Unique ID to describe the event.
|
keyword
|
event.ingested
|
Timestamp when an event arrived in the central data store. This is different from
@timestamp , which is when the event originally occurred. It's also different from event.created , which is meant to capture the first time an agent saw the event. In normal conditions, assuming no tampering, the timestamps should chronologically look like this: @timestamp < event.created < event.ingested . |
date
|
event.kind
|
This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy.
event.kind gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. |
keyword
|
event.module
|
Event module
|
constant_keyword
|
event.original
|
Raw text message of entire event. Used to demonstrate log integrity or where the full log message (before splitting it up in multiple parts) may be required, e.g. for reindex. This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from
_source . If users wish to override this and index this field, please see Field data types in the Elasticsearch Reference . |
keyword
|
event.outcome
|
This is one of four ECS Categorization Fields, and indicates the lowest level in the ECS category hierarchy.
event.outcome simply denotes whether the event represents a success or a failure from the perspective of the entity that produced the event. Note that when a single transaction is described in multiple events, each event may populate different values of event.outcome , according to their perspective. Also note that in the case of a compound event (a single event that contains multiple logical events), this field should be populated with the value that best captures the overall success or failure from the perspective of the event producer. Further note that not all events will have an associated outcome. For example, this field is generally not populated for metric events, events with event.type:info , or any events for which an outcome does not make logical sense. |
keyword
|
gcp.audit.authentication_info.authority_selector
|
The authority selector specified by the requestor, if any. It is not guaranteed that the principal was allowed to use this authority.
|
keyword
|
gcp.audit.authentication_info.principal_email
|
The email address of the authenticated user making the request.
|
keyword
|
gcp.audit.authentication_info.principal_subject
|
String representation of identity of requesting party. Populated for both first and third party identities. Only present for APIs that support third-party identities.
|
keyword
|
gcp.audit.authorization_info.granted
|
Whether or not authorization for resource and permission was granted.
|
boolean
|
gcp.audit.authorization_info.permission
|
The required IAM permission.
|
keyword
|
gcp.audit.authorization_info.resource
|
The resource being accessed, as a REST-style string.
|
keyword
|
gcp.audit.authorization_info.resource_attributes.name
|
The name of the resource.
|
keyword
|
gcp.audit.authorization_info.resource_attributes.service
|
The name of the service.
|
keyword
|
gcp.audit.authorization_info.resource_attributes.type
|
The type of the resource.
|
keyword
|
gcp.audit.flattened
|
Contains the full audit document as sent by GCP.
|
flattened
|
gcp.audit.labels
|
A map of key, value pairs that provides additional information about the log entry. The labels can be user-defined or system-defined.
|
flattened
|
gcp.audit.logentry_operation.first
|
Optional. Set this to True if this is the first log entry in the operation.
|
boolean
|
gcp.audit.logentry_operation.id
|
Optional. An arbitrary operation identifier. Log entries with the same identifier are assumed to be part of the same operation.
|
keyword
|
gcp.audit.logentry_operation.last
|
Optional. Set this to True if this is the last log entry in the operation.
|
boolean
|
gcp.audit.logentry_operation.producer
|
Optional. An arbitrary producer identifier. The combination of id and producer must be globally unique.
|
keyword
|
gcp.audit.method_name
|
The name of the service method or operation. For API calls, this should be the name of the API method. For example, 'google.datastore.v1.Datastore.RunQuery'.
|
keyword
|
gcp.audit.num_response_items
|
The number of items returned from a List or Query API method, if applicable.
|
long
|
gcp.audit.request
|
flattened
|
|
gcp.audit.request_metadata.caller_ip
|
The IP address of the caller.
|
ip
|
gcp.audit.request_metadata.caller_supplied_user_agent
|
The user agent of the caller. This information is not authenticated and should be treated accordingly.
|
keyword
|
gcp.audit.request_metadata.raw.caller_ip
|
The raw IP address of the caller.
|
keyword
|
gcp.audit.resource_location.current_locations
|
Current locations of the resource.
|
keyword
|
gcp.audit.resource_name
|
The resource or collection that is the target of the operation. The name is a scheme-less URI, not including the API service name. For example, 'shelves/SHELF_ID/books'.
|
keyword
|
gcp.audit.response
|
flattened
|
|
gcp.audit.service_name
|
The name of the API service performing the operation. For example, datastore.googleapis.com.
|
keyword
|
gcp.audit.status.code
|
The status code, which should be an enum value of google.rpc.Code.
|
integer
|
gcp.audit.status.message
|
A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.
|
keyword
|
gcp.audit.type
|
Type property.
|
keyword
|
gcp.destination.instance.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.destination.instance.region
|
Region of the VM.
|
keyword
|
gcp.destination.instance.zone
|
Zone of the VM.
|
keyword
|
gcp.destination.vpc.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.destination.vpc.subnetwork_name
|
Subnetwork on which the VM is operating.
|
keyword
|
gcp.destination.vpc.vpc_name
|
VPC on which the VM is operating.
|
keyword
|
gcp.source.instance.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.source.instance.region
|
Region of the VM.
|
keyword
|
gcp.source.instance.zone
|
Zone of the VM.
|
keyword
|
gcp.source.vpc.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.source.vpc.subnetwork_name
|
Subnetwork on which the VM is operating.
|
keyword
|
gcp.source.vpc.vpc_name
|
VPC on which the VM is operating.
|
keyword
|
host.architecture
|
Operating system architecture.
|
keyword
|
host.containerized
|
If the host is a container.
|
boolean
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
host.ip
|
Host ip addresses.
|
ip
|
host.mac
|
Host mac addresses.
|
keyword
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
host.os.build
|
OS build information.
|
keyword
|
host.os.codename
|
OS codename, if any.
|
keyword
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
input.type
|
Input type
|
keyword
|
log.file.path
|
Full path to the log file this event came from, including the file name. It should include the drive letter, when appropriate. If the event wasn't read from a log file, do not populate this field.
|
keyword
|
log.level
|
Original log level of the log event. If the source of the event provides a log level or textual severity, this is the one that goes in
log.level . If your source doesn't specify one, you may put your event transport's severity here (e.g. Syslog severity). Some examples are warn , err , i , informational . |
keyword
|
log.logger
|
The name of the logger inside an application. This is usually the name of the class which initialized the logger, or can be a custom name.
|
keyword
|
log.offset
|
Log offset
|
long
|
message
|
For log events the message field contains the log message, optimized for viewing in a log viewer. For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event. If multiple messages exist, they can be combined into one message.
|
match_only_text
|
orchestrator.api_version
|
API version being used to carry out the action
|
keyword
|
orchestrator.cluster.name
|
Name of the cluster.
|
keyword
|
orchestrator.cluster.url
|
URL of the API used to manage the cluster.
|
keyword
|
orchestrator.cluster.version
|
The version of the cluster.
|
keyword
|
orchestrator.namespace
|
Namespace in which the action is taking place.
|
keyword
|
orchestrator.organization
|
Organization affected by the event (for multi-tenant orchestrator setups).
|
keyword
|
orchestrator.resource.name
|
Name of the resource being acted upon.
|
keyword
|
orchestrator.resource.type
|
Type of resource being acted upon.
|
keyword
|
orchestrator.type
|
Orchestrator cluster type (e.g. kubernetes, nomad or cloudfoundry).
|
keyword
|
service.name
|
Name of the service data is collected from. The name of the service is normally user given. This allows for distributed services that run on multiple hosts to correlate the related instances based on the name. In the case of Elasticsearch the
service.name could contain the cluster name. For Beats the service.name is by default a copy of the service.type field if no name is specified. |
keyword
|
source.address
|
Some event source addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the
.address field. Then it should be duplicated to .ip or .domain , depending on which one it is. |
keyword
|
source.as.number
|
Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
|
long
|
source.as.organization.name
|
Organization name.
|
keyword
|
source.as.organization.name.text
|
Multi-field of
source.as.organization.name . |
match_only_text
|
source.geo.city_name
|
City name.
|
keyword
|
source.geo.continent_name
|
Name of the continent.
|
keyword
|
source.geo.country_iso_code
|
Country ISO code.
|
keyword
|
source.geo.country_name
|
Country name.
|
keyword
|
source.geo.location
|
Longitude and latitude.
|
geo_point
|
source.geo.region_iso_code
|
Region ISO code.
|
keyword
|
source.geo.region_name
|
Region name.
|
keyword
|
source.ip
|
IP address of the source (IPv4 or IPv6).
|
ip
|
tags
|
List of keywords used to tag each event.
|
keyword
|
user.email
|
User email address.
|
keyword
|
user_agent.device.name
|
Name of the device.
|
keyword
|
user_agent.name
|
Name of the user agent.
|
keyword
|
user_agent.original
|
Unparsed user_agent string.
|
keyword
|
user_agent.original.text
|
Multi-field of
user_agent.original . |
match_only_text
|
user_agent.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
user_agent.os.full
|
Operating system name, including the version or code name.
|
keyword
|
user_agent.os.full.text
|
Multi-field of
user_agent.os.full . |
match_only_text
|
user_agent.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
user_agent.os.name
|
Operating system name, without the version.
|
keyword
|
user_agent.os.name.text
|
Multi-field of
user_agent.os.name . |
match_only_text
|
user_agent.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
user_agent.os.version
|
Operating system version as a raw string.
|
keyword
|
user_agent.version
|
Version of the user agent.
|
keyword
|
An example event for audit
looks as following:
{
"@timestamp": "2019-12-19T00:44:25.051Z",
"agent": {
"ephemeral_id": "a22278bb-5e1f-4ab7-b468-277c8c0b80a9",
"id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.7.1"
},
"client": {
"user": {
"email": "xxx@xxx.xxx"
}
},
"cloud": {
"project": {
"id": "elastic-beats"
},
"provider": "gcp"
},
"data_stream": {
"dataset": "gcp.audit",
"namespace": "ep",
"type": "logs"
},
"ecs": {
"version": "8.8.0"
},
"elastic_agent": {
"id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
"snapshot": false,
"version": "8.7.1"
},
"event": {
"action": "beta.compute.instances.aggregatedList",
"agent_id_status": "verified",
"category": [
"network",
"configuration"
],
"created": "2023-10-25T04:18:46.637Z",
"dataset": "gcp.audit",
"id": "yonau2dg2zi",
"ingested": "2023-10-25T04:18:47Z",
"kind": "event",
"outcome": "success",
"provider": "data_access",
"type": [
"access",
"allowed"
]
},
"gcp": {
"audit": {
"authorization_info": [
{
"granted": true,
"permission": "compute.instances.list",
"resource_attributes": {
"name": "projects/elastic-beats",
"service": "resourcemanager",
"type": "resourcemanager.projects"
}
}
],
"num_response_items": 61,
"request": {
"@type": "type.googleapis.com/compute.instances.aggregatedList"
},
"resource_location": {
"current_locations": [
"global"
]
},
"resource_name": "projects/elastic-beats/global/instances",
"response": {
"@type": "core.k8s.io/v1.Status",
"apiVersion": "v1",
"details": {
"group": "batch",
"kind": "jobs",
"name": "gsuite-exporter-1589294700",
"uid": "2beff34a-945f-11ea-bacf-42010a80007f"
},
"kind": "Status",
"status_value": "Success"
},
"type": "type.googleapis.com/google.cloud.audit.AuditLog"
}
},
"input": {
"type": "gcp-pubsub"
},
"log": {
"level": "INFO",
"logger": "projects/elastic-beats/logs/cloudaudit.googleapis.com%2Fdata_access"
},
"service": {
"name": "compute.googleapis.com"
},
"source": {
"ip": "192.168.1.1"
},
"tags": [
"forwarded",
"gcp-audit"
],
"user_agent": {
"device": {
"name": "Mac"
},
"name": "Firefox",
"original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:71.0) Gecko/20100101 Firefox/71.0,gzip(gfe),gzip(gfe)",
"os": {
"full": "Mac OS X 10.15",
"name": "Mac OS X",
"version": "10.15"
},
"version": "71.0."
}
}
Firewall
The firewall
dataset collects logs from Firewall Rules in your Virtual Private Cloud (VPC) networks.
Exported fields
Field | Description | Type |
---|---|---|
@timestamp
|
Event timestamp.
|
date
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
cloud.availability_zone
|
Availability zone in which this host is running.
|
keyword
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
cloud.region
|
Region in which this host is running.
|
keyword
|
container.id
|
Unique container id.
|
keyword
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
container.labels
|
Image labels.
|
object
|
container.name
|
Container name.
|
keyword
|
container.runtime
|
Runtime managing this container.
|
keyword
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
destination.address
|
Some event destination addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the
.address field. Then it should be duplicated to .ip or .domain , depending on which one it is. |
keyword
|
destination.as.number
|
Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
|
long
|
destination.as.organization.name
|
Organization name.
|
keyword
|
destination.as.organization.name.text
|
Multi-field of
destination.as.organization.name . |
match_only_text
|
destination.domain
|
The domain name of the destination system. This value may be a host name, a fully qualified domain name, or another host naming format. The value may derive from the original event or be added from enrichment.
|
keyword
|
destination.geo.city_name
|
City name.
|
keyword
|
destination.geo.continent_name
|
Name of the continent.
|
keyword
|
destination.geo.country_iso_code
|
Country ISO code.
|
keyword
|
destination.geo.country_name
|
Country name.
|
keyword
|
destination.geo.location
|
Longitude and latitude.
|
geo_point
|
destination.geo.name
|
User-defined description of a location, at the level of granularity they care about. Could be the name of their data centers, the floor number, if this describes a local physical entity, city names. Not typically used in automated geolocation.
|
keyword
|
destination.geo.region_iso_code
|
Region ISO code.
|
keyword
|
destination.geo.region_name
|
Region name.
|
keyword
|
destination.ip
|
IP address of the destination (IPv4 or IPv6).
|
ip
|
destination.port
|
Port of the destination.
|
long
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
event.action
|
The action captured by the event. This describes the information in the event. It is more specific than
event.category . Examples are group-add , process-started , file-created . The value is normally defined by the implementer. |
keyword
|
event.category
|
This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy.
event.category represents the "big buckets" of ECS categories. For example, filtering on event.category:process yields all events relating to process activity. This field is closely related to event.type , which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories. |
keyword
|
event.created
|
event.created contains the date/time when the event was first read by an agent, or by your pipeline. This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event. In most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source. In case the two timestamps are identical, @timestamp should be used.
|
date
|
event.dataset
|
Event dataset
|
constant_keyword
|
event.id
|
Unique ID to describe the event.
|
keyword
|
event.ingested
|
Timestamp when an event arrived in the central data store. This is different from
@timestamp , which is when the event originally occurred. It's also different from event.created , which is meant to capture the first time an agent saw the event. In normal conditions, assuming no tampering, the timestamps should chronologically look like this: @timestamp < event.created < event.ingested . |
date
|
event.kind
|
This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy.
event.kind gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. |
keyword
|
event.module
|
Event module
|
constant_keyword
|
event.original
|
Raw text message of entire event. Used to demonstrate log integrity or where the full log message (before splitting it up in multiple parts) may be required, e.g. for reindex. This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from
_source . If users wish to override this and index this field, please see Field data types in the Elasticsearch Reference . |
keyword
|
event.outcome
|
This is one of four ECS Categorization Fields, and indicates the lowest level in the ECS category hierarchy.
event.outcome simply denotes whether the event represents a success or a failure from the perspective of the entity that produced the event. Note that when a single transaction is described in multiple events, each event may populate different values of event.outcome , according to their perspective. Also note that in the case of a compound event (a single event that contains multiple logical events), this field should be populated with the value that best captures the overall success or failure from the perspective of the event producer. Further note that not all events will have an associated outcome. For example, this field is generally not populated for metric events, events with event.type:info , or any events for which an outcome does not make logical sense. |
keyword
|
event.type
|
This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy.
event.type represents a categorization "sub-bucket" that, when used along with the event.category field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. |
keyword
|
gcp.destination.instance.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.destination.instance.region
|
Region of the VM.
|
keyword
|
gcp.destination.instance.zone
|
Zone of the VM.
|
keyword
|
gcp.destination.vpc.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.destination.vpc.subnetwork_name
|
Subnetwork on which the VM is operating.
|
keyword
|
gcp.destination.vpc.vpc_name
|
VPC on which the VM is operating.
|
keyword
|
gcp.firewall.flattened
|
Contains the full firewall document as sent by GCP.
|
flattened
|
gcp.firewall.rule_details.action
|
Action that the rule performs on match.
|
keyword
|
gcp.firewall.rule_details.destination_range
|
List of destination ranges that the firewall applies to.
|
keyword
|
gcp.firewall.rule_details.direction
|
Direction of traffic that matches this rule.
|
keyword
|
gcp.firewall.rule_details.ip_port_info
|
List of ip protocols and applicable port ranges for rules.
|
nested
|
gcp.firewall.rule_details.priority
|
The priority for the firewall rule.
|
long
|
gcp.firewall.rule_details.reference
|
Reference to the firewall rule.
|
keyword
|
gcp.firewall.rule_details.source_range
|
List of source ranges that the firewall rule applies to.
|
keyword
|
gcp.firewall.rule_details.source_service_account
|
List of all the source service accounts that the firewall rule applies to.
|
keyword
|
gcp.firewall.rule_details.source_tag
|
List of all the source tags that the firewall rule applies to.
|
keyword
|
gcp.firewall.rule_details.target_service_account
|
List of all the target service accounts that the firewall rule applies to.
|
keyword
|
gcp.firewall.rule_details.target_tag
|
List of all the target tags that the firewall rule applies to.
|
keyword
|
gcp.source.instance.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.source.instance.region
|
Region of the VM.
|
keyword
|
gcp.source.instance.zone
|
Zone of the VM.
|
keyword
|
gcp.source.vpc.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.source.vpc.subnetwork_name
|
Subnetwork on which the VM is operating.
|
keyword
|
gcp.source.vpc.vpc_name
|
VPC on which the VM is operating.
|
keyword
|
host.architecture
|
Operating system architecture.
|
keyword
|
host.containerized
|
If the host is a container.
|
boolean
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
host.ip
|
Host ip addresses.
|
ip
|
host.mac
|
Host mac addresses.
|
keyword
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
host.os.build
|
OS build information.
|
keyword
|
host.os.codename
|
OS codename, if any.
|
keyword
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
input.type
|
Input type
|
keyword
|
log.file.path
|
Full path to the log file this event came from, including the file name. It should include the drive letter, when appropriate. If the event wasn't read from a log file, do not populate this field.
|
keyword
|
log.logger
|
The name of the logger inside an application. This is usually the name of the class which initialized the logger, or can be a custom name.
|
keyword
|
log.offset
|
Log offset
|
long
|
message
|
For log events the message field contains the log message, optimized for viewing in a log viewer. For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event. If multiple messages exist, they can be combined into one message.
|
match_only_text
|
network.community_id
|
A hash of source and destination IPs and ports, as well as the protocol used in a communication. This is a tool-agnostic standard to identify flows. Learn more at https://github.com/corelight/community-id-spec.
|
keyword
|
network.direction
|
Direction of the network traffic. When mapping events from a host-based monitoring context, populate this field from the host's point of view, using the values "ingress" or "egress". When mapping events from a network or perimeter-based monitoring context, populate this field from the point of view of the network perimeter, using the values "inbound", "outbound", "internal" or "external". Note that "internal" is not crossing perimeter boundaries, and is meant to describe communication between two hosts within the perimeter. Note also that "external" is meant to describe traffic between two hosts that are external to the perimeter. This could for example be useful for ISPs or VPN service providers.
|
keyword
|
network.iana_number
|
IANA Protocol Number (https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml). Standardized list of protocols. This aligns well with NetFlow and sFlow related logs which use the IANA Protocol Number.
|
keyword
|
network.name
|
Name given by operators to sections of their network.
|
keyword
|
network.transport
|
Same as network.iana_number, but instead using the Keyword name of the transport layer (udp, tcp, ipv6-icmp, etc.) The field value must be normalized to lowercase for querying.
|
keyword
|
network.type
|
In the OSI Model this would be the Network Layer. ipv4, ipv6, ipsec, pim, etc The field value must be normalized to lowercase for querying.
|
keyword
|
related.hash
|
All the hashes seen on your event. Populating this field, then using it to search for hashes can help in situations where you're unsure what the hash algorithm is (and therefore which key name to search).
|
keyword
|
related.hosts
|
All hostnames or other host identifiers seen on your event. Example identifiers include FQDNs, domain names, workstation names, or aliases.
|
keyword
|
related.ip
|
All of the IPs seen on your event.
|
ip
|
related.user
|
All the user names or other user identifiers seen on the event.
|
keyword
|
rule.name
|
The name of the rule or signature generating the event.
|
keyword
|
source.address
|
Some event source addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the
.address field. Then it should be duplicated to .ip or .domain , depending on which one it is. |
keyword
|
source.as.number
|
Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
|
long
|
source.as.organization.name
|
Organization name.
|
keyword
|
source.as.organization.name.text
|
Multi-field of
source.as.organization.name . |
match_only_text
|
source.domain
|
The domain name of the source system. This value may be a host name, a fully qualified domain name, or another host naming format. The value may derive from the original event or be added from enrichment.
|
keyword
|
source.geo.city_name
|
City name.
|
keyword
|
source.geo.continent_name
|
Name of the continent.
|
keyword
|
source.geo.country_iso_code
|
Country ISO code.
|
keyword
|
source.geo.country_name
|
Country name.
|
keyword
|
source.geo.location
|
Longitude and latitude.
|
geo_point
|
source.geo.name
|
User-defined description of a location, at the level of granularity they care about. Could be the name of their data centers, the floor number, if this describes a local physical entity, city names. Not typically used in automated geolocation.
|
keyword
|
source.geo.region_iso_code
|
Region ISO code.
|
keyword
|
source.geo.region_name
|
Region name.
|
keyword
|
source.ip
|
IP address of the source (IPv4 or IPv6).
|
ip
|
source.port
|
Port of the source.
|
long
|
tags
|
List of keywords used to tag each event.
|
keyword
|
An example event for firewall
looks as following:
{
"@timestamp": "2019-10-30T13:52:42.191Z",
"agent": {
"ephemeral_id": "175ae0b3-355c-4ca7-87ea-d5f1ee34102e",
"id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.7.1"
},
"cloud": {
"availability_zone": "us-east1-b",
"project": {
"id": "test-beats"
},
"provider": "gcp",
"region": "us-east1"
},
"data_stream": {
"dataset": "gcp.firewall",
"namespace": "ep",
"type": "logs"
},
"destination": {
"address": "10.42.0.2",
"domain": "test-windows",
"ip": "10.42.0.2",
"port": 3389
},
"ecs": {
"version": "8.8.0"
},
"elastic_agent": {
"id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
"snapshot": false,
"version": "8.7.1"
},
"event": {
"action": "firewall-rule",
"agent_id_status": "verified",
"category": [
"network"
],
"created": "2023-10-25T04:20:37.182Z",
"dataset": "gcp.firewall",
"id": "1f21ciqfpfssuo",
"ingested": "2023-10-25T04:20:41Z",
"kind": "event",
"type": [
"allowed",
"connection"
]
},
"gcp": {
"destination": {
"instance": {
"project_id": "test-beats",
"region": "us-east1",
"zone": "us-east1-b"
},
"vpc": {
"project_id": "test-beats",
"subnetwork_name": "windows-isolated",
"vpc_name": "windows-isolated"
}
},
"firewall": {
"rule_details": {
"action": "ALLOW",
"direction": "INGRESS",
"ip_port_info": [
{
"ip_protocol": "TCP",
"port_range": [
"3389"
]
}
],
"priority": 1000,
"source_range": [
"0.0.0.0/0"
],
"target_tag": [
"allow-rdp"
]
}
}
},
"input": {
"type": "gcp-pubsub"
},
"log": {
"logger": "projects/test-beats/logs/compute.googleapis.com%2Ffirewall"
},
"network": {
"community_id": "1:OdLB9eXsBDLz8m97ao4LepX6q+4=",
"direction": "inbound",
"iana_number": "6",
"name": "windows-isolated",
"transport": "tcp",
"type": "ipv4"
},
"related": {
"ip": [
"192.168.2.126",
"10.42.0.2"
]
},
"rule": {
"name": "network:windows-isolated/firewall:windows-isolated-allow-rdp"
},
"source": {
"address": "192.168.2.126",
"geo": {
"continent_name": "Asia",
"country_name": "omn"
},
"ip": "192.168.2.126",
"port": 64853
},
"tags": [
"forwarded",
"gcp-firewall"
]
}
VPC Flow
The vpcflow
dataset collects logs sent from and received by VM instances, including instances used as GKE nodes.
Exported fields
Field | Description | Type |
---|---|---|
@timestamp
|
Event timestamp.
|
date
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
cloud.availability_zone
|
Availability zone in which this host is running.
|
keyword
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
cloud.region
|
Region in which this host is running.
|
keyword
|
container.id
|
Unique container id.
|
keyword
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
container.labels
|
Image labels.
|
object
|
container.name
|
Container name.
|
keyword
|
container.runtime
|
Runtime managing this container.
|
keyword
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
destination.address
|
Some event destination addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the
.address field. Then it should be duplicated to .ip or .domain , depending on which one it is. |
keyword
|
destination.as.number
|
Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
|
long
|
destination.as.organization.name
|
Organization name.
|
keyword
|
destination.as.organization.name.text
|
Multi-field of
destination.as.organization.name . |
match_only_text
|
destination.domain
|
The domain name of the destination system. This value may be a host name, a fully qualified domain name, or another host naming format. The value may derive from the original event or be added from enrichment.
|
keyword
|
destination.geo.city_name
|
City name.
|
keyword
|
destination.geo.continent_name
|
Name of the continent.
|
keyword
|
destination.geo.country_iso_code
|
Country ISO code.
|
keyword
|
destination.geo.country_name
|
Country name.
|
keyword
|
destination.geo.location
|
Longitude and latitude.
|
geo_point
|
destination.geo.name
|
User-defined description of a location, at the level of granularity they care about. Could be the name of their data centers, the floor number, if this describes a local physical entity, city names. Not typically used in automated geolocation.
|
keyword
|
destination.geo.region_iso_code
|
Region ISO code.
|
keyword
|
destination.geo.region_name
|
Region name.
|
keyword
|
destination.ip
|
IP address of the destination (IPv4 or IPv6).
|
ip
|
destination.port
|
Port of the destination.
|
long
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
event.action
|
The action captured by the event. This describes the information in the event. It is more specific than
event.category . Examples are group-add , process-started , file-created . The value is normally defined by the implementer. |
keyword
|
event.category
|
This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy.
event.category represents the "big buckets" of ECS categories. For example, filtering on event.category:process yields all events relating to process activity. This field is closely related to event.type , which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories. |
keyword
|
event.created
|
event.created contains the date/time when the event was first read by an agent, or by your pipeline. This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event. In most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source. In case the two timestamps are identical, @timestamp should be used.
|
date
|
event.dataset
|
Event dataset
|
constant_keyword
|
event.end
|
event.end contains the date when the event ended or when the activity was last observed.
|
date
|
event.id
|
Unique ID to describe the event.
|
keyword
|
event.ingested
|
Timestamp when an event arrived in the central data store. This is different from
@timestamp , which is when the event originally occurred. It's also different from event.created , which is meant to capture the first time an agent saw the event. In normal conditions, assuming no tampering, the timestamps should chronologically look like this: @timestamp < event.created < event.ingested . |
date
|
event.kind
|
This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy.
event.kind gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. |
keyword
|
event.module
|
Event module
|
constant_keyword
|
event.original
|
Raw text message of entire event. Used to demonstrate log integrity or where the full log message (before splitting it up in multiple parts) may be required, e.g. for reindex. This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from
_source . If users wish to override this and index this field, please see Field data types in the Elasticsearch Reference . |
keyword
|
event.outcome
|
This is one of four ECS Categorization Fields, and indicates the lowest level in the ECS category hierarchy.
event.outcome simply denotes whether the event represents a success or a failure from the perspective of the entity that produced the event. Note that when a single transaction is described in multiple events, each event may populate different values of event.outcome , according to their perspective. Also note that in the case of a compound event (a single event that contains multiple logical events), this field should be populated with the value that best captures the overall success or failure from the perspective of the event producer. Further note that not all events will have an associated outcome. For example, this field is generally not populated for metric events, events with event.type:info , or any events for which an outcome does not make logical sense. |
keyword
|
event.start
|
event.start contains the date when the event started or when the activity was first observed.
|
date
|
event.type
|
This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy.
event.type represents a categorization "sub-bucket" that, when used along with the event.category field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. |
keyword
|
gcp.destination.instance.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.destination.instance.region
|
Region of the VM.
|
keyword
|
gcp.destination.instance.zone
|
Zone of the VM.
|
keyword
|
gcp.destination.vpc.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.destination.vpc.subnetwork_name
|
Subnetwork on which the VM is operating.
|
keyword
|
gcp.destination.vpc.vpc_name
|
VPC on which the VM is operating.
|
keyword
|
gcp.source.instance.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.source.instance.region
|
Region of the VM.
|
keyword
|
gcp.source.instance.zone
|
Zone of the VM.
|
keyword
|
gcp.source.vpc.project_id
|
ID of the project containing the VM.
|
keyword
|
gcp.source.vpc.subnetwork_name
|
Subnetwork on which the VM is operating.
|
keyword
|
gcp.source.vpc.vpc_name
|
VPC on which the VM is operating.
|
keyword
|
gcp.vpcflow.flattened
|
Contains the full vpcflow document as sent by GCP.
|
flattened
|
gcp.vpcflow.reporter
|
The side which reported the flow. Can be either 'SRC' or 'DEST'.
|
keyword
|
gcp.vpcflow.rtt.ms
|
Latency as measured (for TCP flows only) during the time interval. This is the time elapsed between sending a SEQ and receiving a corresponding ACK and it contains the network RTT as well as the application related delay.
|
long
|
host.architecture
|
Operating system architecture.
|
keyword
|
host.containerized
|
If the host is a container.
|
boolean
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
host.ip
|
Host ip addresses.
|
ip
|
host.mac
|
Host mac addresses.
|
keyword
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
host.os.build
|
OS build information.
|
keyword
|
host.os.codename
|
OS codename, if any.
|
keyword
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
input.type
|
Input type
|
keyword
|
log.file.path
|
Full path to the log file this event came from, including the file name. It should include the drive letter, when appropriate. If the event wasn't read from a log file, do not populate this field.
|
keyword
|
log.logger
|
The name of the logger inside an application. This is usually the name of the class which initialized the logger, or can be a custom name.
|
keyword
|
log.offset
|
Log offset
|
long
|
message
|
For log events the message field contains the log message, optimized for viewing in a log viewer. For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event. If multiple messages exist, they can be combined into one message.
|
match_only_text
|
network.bytes
|
Total bytes transferred in both directions. If
source.bytes and destination.bytes are known, network.bytes is their sum. |
long
|
network.community_id
|
A hash of source and destination IPs and ports, as well as the protocol used in a communication. This is a tool-agnostic standard to identify flows. Learn more at https://github.com/corelight/community-id-spec.
|
keyword
|
network.direction
|
Direction of the network traffic. When mapping events from a host-based monitoring context, populate this field from the host's point of view, using the values "ingress" or "egress". When mapping events from a network or perimeter-based monitoring context, populate this field from the point of view of the network perimeter, using the values "inbound", "outbound", "internal" or "external". Note that "internal" is not crossing perimeter boundaries, and is meant to describe communication between two hosts within the perimeter. Note also that "external" is meant to describe traffic between two hosts that are external to the perimeter. This could for example be useful for ISPs or VPN service providers.
|
keyword
|
network.iana_number
|
IANA Protocol Number (https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml). Standardized list of protocols. This aligns well with NetFlow and sFlow related logs which use the IANA Protocol Number.
|
keyword
|
network.name
|
Name given by operators to sections of their network.
|
keyword
|
network.packets
|
Total packets transferred in both directions. If
source.packets and destination.packets are known, network.packets is their sum. |
long
|
network.transport
|
Same as network.iana_number, but instead using the Keyword name of the transport layer (udp, tcp, ipv6-icmp, etc.) The field value must be normalized to lowercase for querying.
|
keyword
|
network.type
|
In the OSI Model this would be the Network Layer. ipv4, ipv6, ipsec, pim, etc The field value must be normalized to lowercase for querying.
|
keyword
|
related.hash
|
All the hashes seen on your event. Populating this field, then using it to search for hashes can help in situations where you're unsure what the hash algorithm is (and therefore which key name to search).
|
keyword
|
related.hosts
|
All hostnames or other host identifiers seen on your event. Example identifiers include FQDNs, domain names, workstation names, or aliases.
|
keyword
|
related.ip
|
All of the IPs seen on your event.
|
ip
|
related.user
|
All the user names or other user identifiers seen on the event.
|
keyword
|
rule.name
|
The name of the rule or signature generating the event.
|
keyword
|
source.address
|
Some event source addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the
.address field. Then it should be duplicated to .ip or .domain , depending on which one it is. |
keyword
|
source.as.number
|
Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
|
long
|
source.as.organization.name
|
Organization name.
|
keyword
|
source.as.organization.name.text
|
Multi-field of
source.as.organization.name . |
match_only_text
|
source.bytes
|
Bytes sent from the source to the destination.
|
long
|
source.domain
|
The domain name of the source system. This value may be a host name, a fully qualified domain name, or another host naming format. The value may derive from the original event or be added from enrichment.
|
keyword
|
source.geo.city_name
|
City name.
|
keyword
|
source.geo.continent_name
|
Name of the continent.
|
keyword
|
source.geo.country_iso_code
|
Country ISO code.
|
keyword
|
source.geo.country_name
|
Country name.
|
keyword
|
source.geo.location
|
Longitude and latitude.
|
geo_point
|
source.geo.name
|
User-defined description of a location, at the level of granularity they care about. Could be the name of their data centers, the floor number, if this describes a local physical entity, city names. Not typically used in automated geolocation.
|
keyword
|
source.geo.region_iso_code
|
Region ISO code.
|
keyword
|
source.geo.region_name
|
Region name.
|
keyword
|
source.ip
|
IP address of the source (IPv4 or IPv6).
|
ip
|
source.packets
|
Packets sent from the source to the destination.
|
long
|
source.port
|
Port of the source.
|
long
|
tags
|
List of keywords used to tag each event.
|
keyword
|
An example event for vpcflow
looks as following:
{
"@timestamp": "2019-06-14T03:50:10.845Z",
"agent": {
"ephemeral_id": "0b8165a2-0e25-4e9a-bb68-271697e0993f",
"id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.7.1"
},
"cloud": {
"availability_zone": "us-east1-b",
"instance": {
"name": "kibana"
},
"project": {
"id": "my-sample-project"
},
"provider": "gcp",
"region": "us-east1"
},
"data_stream": {
"dataset": "gcp.vpcflow",
"namespace": "ep",
"type": "logs"
},
"destination": {
"address": "10.139.99.242",
"domain": "elasticsearch",
"ip": "10.139.99.242",
"port": 9200
},
"ecs": {
"version": "8.8.0"
},
"elastic_agent": {
"id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
"snapshot": false,
"version": "8.7.1"
},
"event": {
"agent_id_status": "verified",
"category": [
"network"
],
"created": "2023-10-25T04:21:42.006Z",
"dataset": "gcp.vpcflow",
"end": "2019-06-14T03:49:51.821056075Z",
"id": "ut8lbrffooxz5",
"ingested": "2023-10-25T04:21:43Z",
"kind": "event",
"start": "2019-06-14T03:40:20.510622432Z",
"type": [
"connection"
]
},
"gcp": {
"destination": {
"instance": {
"project_id": "my-sample-project",
"region": "us-east1",
"zone": "us-east1-b"
},
"vpc": {
"project_id": "my-sample-project",
"subnetwork_name": "default",
"vpc_name": "default"
}
},
"source": {
"instance": {
"project_id": "my-sample-project",
"region": "us-east1",
"zone": "us-east1-b"
},
"vpc": {
"project_id": "my-sample-project",
"subnetwork_name": "default",
"vpc_name": "default"
}
},
"vpcflow": {
"reporter": "DEST",
"rtt": {
"ms": 201
}
}
},
"input": {
"type": "gcp-pubsub"
},
"log": {
"logger": "projects/my-sample-project/logs/compute.googleapis.com%2Fvpc_flows"
},
"network": {
"bytes": 11773,
"community_id": "1:FYaJFSEAKLcBCMFoT6sR5TMHf/s=",
"direction": "internal",
"iana_number": "6",
"name": "default",
"packets": 94,
"transport": "tcp",
"type": "ipv4"
},
"related": {
"ip": [
"67.43.156.13",
"10.139.99.242"
]
},
"source": {
"address": "67.43.156.13",
"as": {
"number": 35908
},
"bytes": 11773,
"domain": "kibana",
"geo": {
"continent_name": "Asia",
"country_iso_code": "BT",
"country_name": "Bhutan",
"location": {
"lat": 27.5,
"lon": 90.5
}
},
"ip": "67.43.156.13",
"packets": 94,
"port": 33576
},
"tags": [
"forwarded",
"gcp-vpcflow"
]
}
DNS
The dns
dataset collects queries that name servers resolve for your Virtual Private Cloud (VPC) networks, as well as queries from an external entity directly to a public zone.
Exported fields
Field | Description | Type |
---|---|---|
@timestamp
|
Event timestamp.
|
date
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
cloud.availability_zone
|
Availability zone in which this host is running.
|
keyword
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
cloud.region
|
Region in which this host is running.
|
keyword
|
container.id
|
Unique container id.
|
keyword
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
container.labels
|
Image labels.
|
object
|
container.name
|
Container name.
|
keyword
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
destination.address
|
Some event destination addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the
.address field. Then it should be duplicated to .ip or .domain , depending on which one it is. |
keyword
|
destination.ip
|
IP address of the destination (IPv4 or IPv6).
|
ip
|
dns.answers
|
An array containing an object for each answer section returned by the server. The main keys that should be present in these objects are defined by ECS. Records that have more information may contain more keys than what ECS defines. Not all DNS data sources give all details about DNS answers. At minimum, answer objects must contain the
data key. If more information is available, map as much of it to ECS as possible, and add any additional fields to the answer objects as custom fields. |
group
|
dns.answers.class
|
The class of DNS data contained in this resource record.
|
keyword
|
dns.answers.data
|
The data describing the resource. The meaning of this data depends on the type and class of the resource record.
|
keyword
|
dns.answers.name
|
The domain name to which this resource record pertains. If a chain of CNAME is being resolved, each answer's
name should be the one that corresponds with the answer's data . It should not simply be the original question.name repeated. |
keyword
|
dns.answers.ttl
|
The time interval in seconds that this resource record may be cached before it should be discarded. Zero values mean that the data should not be cached.
|
long
|
dns.answers.type
|
The type of data contained in this resource record.
|
keyword
|
dns.question.name
|
The name being queried. If the name field contains non-printable characters (below 32 or above 126), those characters should be represented as escaped base 10 integers (\DDD). Back slashes and quotes should be escaped. Tabs, carriage returns, and line feeds should be converted to \t, \r, and \n respectively.
|
keyword
|
dns.question.registered_domain
|
The highest registered domain, stripped of the subdomain. For example, the registered domain for "foo.example.com" is "example.com". This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last two labels will not work well for TLDs such as "co.uk".
|
keyword
|
dns.question.subdomain
|
The subdomain is all of the labels under the registered_domain. If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com", the subdomain field should contain "sub2.sub1", with no trailing period.
|
keyword
|
dns.question.top_level_domain
|
The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for example.com is "com". This value can be determined precisely with a list like the public suffix list (http://publicsuffix.org). Trying to approximate this by simply taking the last label will not work well for effective TLDs such as "co.uk".
|
keyword
|
dns.question.type
|
The type of record being queried.
|
keyword
|
dns.resolved_ip
|
Array containing all IPs seen in
answers.data . The answers array can be difficult to use, because of the variety of data formats it can contain. Extracting all IP addresses seen in there to dns.resolved_ip makes it possible to index them as IP addresses, and makes them easier to visualize and query for. |
ip
|
dns.response_code
|
The DNS response code.
|
keyword
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
event.created
|
event.created contains the date/time when the event was first read by an agent, or by your pipeline. This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event. In most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source. In case the two timestamps are identical, @timestamp should be used.
|
date
|
event.dataset
|
Event dataset
|
constant_keyword
|
event.id
|
Unique ID to describe the event.
|
keyword
|
event.ingested
|
Timestamp when an event arrived in the central data store. This is different from
@timestamp , which is when the event originally occurred. It's also different from event.created , which is meant to capture the first time an agent saw the event. In normal conditions, assuming no tampering, the timestamps should chronologically look like this: @timestamp < event.created < event.ingested . |
date
|
event.kind
|
This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy.
event.kind gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data coming in at a regular interval or not. |
keyword
|
event.module
|
Event module
|
constant_keyword
|
event.original
|
Raw text message of entire event. Used to demonstrate log integrity or where the full log message (before splitting it up in multiple parts) may be required, e.g. for reindex. This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from
_source . If users wish to override this and index this field, please see Field data types in the Elasticsearch Reference . |
keyword
|
event.outcome
|
This is one of four ECS Categorization Fields, and indicates the lowest level in the ECS category hierarchy.
event.outcome simply denotes whether the event represents a success or a failure from the perspective of the entity that produced the event. Note that when a single transaction is described in multiple events, each event may populate different values of event.outcome , according to their perspective. Also note that in the case of a compound event (a single event that contains multiple logical events), this field should be populated with the value that best captures the overall success or failure from the perspective of the event producer. Further note that not all events will have an associated outcome. For example, this field is generally not populated for metric events, events with event.type:info , or any events for which an outcome does not make logical sense. |
keyword
|
gcp.dns.auth_answer
|
Authoritative answer.
|
boolean
|
gcp.dns.destination_ip
|
Destination IP address, only applicable for forwarding cases.
|
ip
|
gcp.dns.egress_error
|
Egress proxy error.
|
keyword
|
gcp.dns.flattened
|
Contains the full dns document as sent by GCP.
|
flattened
|
gcp.dns.protocol
|
Protocol TCP or UDP.
|
keyword
|
gcp.dns.query_name
|
DNS query name.
|
keyword
|
gcp.dns.query_type
|
DNS query type.
|
keyword
|
gcp.dns.rdata
|
DNS answer in presentation format, truncated to 260 bytes.
|
keyword
|
gcp.dns.response_code
|
Response code.
|
keyword
|
gcp.dns.server_latency
|
Server latency.
|
integer
|
gcp.dns.source_ip
|
Source IP address of the query.
|
ip
|
gcp.dns.source_network
|
Source network of the query.
|
keyword
|
gcp.dns.source_type
|
Type of source generating the DNS query: private-zone, public-zone, forwarding-zone, forwarding-policy, peering-zone, internal, external, internet
|
keyword
|
gcp.dns.target_type
|
Type of target resolving the DNS query: private-zone, public-zone, forwarding-zone, forwarding-policy, peering-zone, internal, external, internet
|
keyword
|
gcp.dns.vm_instance_id
|
Compute Engine VM instance ID, only applicable to queries initiated by Compute Engine VMs.
|
keyword
|
gcp.dns.vm_instance_name
|
Compute Engine VM instance name, only applicable to queries initiated by Compute Engine VMs.
|
keyword
|
gcp.dns.vm_project_id
|
Google Cloud project ID, only applicable to queries initiated by Compute Engine VMs.
|
keyword
|
gcp.dns.vm_zone_name
|
Google Cloud VM zone, only applicable to queries initiated by Compute Engine VMs.
|
keyword
|
host.architecture
|
Operating system architecture.
|
keyword
|
host.containerized
|
If the host is a container.
|
boolean
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
host.ip
|
Host ip addresses.
|
ip
|
host.mac
|
Host mac addresses.
|
keyword
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
host.os.build
|
OS build information.
|
keyword
|
host.os.codename
|
OS codename, if any.
|
keyword
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
input.type
|
Input type
|
keyword
|
log.level
|
Original log level of the log event. If the source of the event provides a log level or textual severity, this is the one that goes in
log.level . If your source doesn't specify one, you may put your event transport's severity here (e.g. Syslog severity). Some examples are warn , err , i , informational . |
keyword
|
log.logger
|
The name of the logger inside an application. This is usually the name of the class which initialized the logger, or can be a custom name.
|
keyword
|
log.offset
|
Log offset
|
long
|
network.iana_number
|
IANA Protocol Number (https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml). Standardized list of protocols. This aligns well with NetFlow and sFlow related logs which use the IANA Protocol Number.
|
keyword
|
network.protocol
|
In the OSI Model this would be the Application Layer protocol. For example,
http , dns , or ssh . The field value must be normalized to lowercase for querying. |
keyword
|
network.transport
|
Same as network.iana_number, but instead using the Keyword name of the transport layer (udp, tcp, ipv6-icmp, etc.) The field value must be normalized to lowercase for querying.
|
keyword
|
related.hosts
|
All hostnames or other host identifiers seen on your event. Example identifiers include FQDNs, domain names, workstation names, or aliases.
|
keyword
|
related.ip
|
All of the IPs seen on your event.
|
ip
|
source.address
|
Some event source addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the
.address field. Then it should be duplicated to .ip or .domain , depending on which one it is. |
keyword
|
source.ip
|
IP address of the source (IPv4 or IPv6).
|
ip
|
tags
|
List of keywords used to tag each event.
|
keyword
|
An example event for dns
looks as following:
{
"@timestamp": "2021-12-12T15:59:40.446Z",
"agent": {
"ephemeral_id": "fd6c4189-cbc6-493a-acfb-c9e7b2b7588c",
"id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.7.1"
},
"cloud": {
"project": {
"id": "key-reference-123456"
},
"provider": "gcp",
"region": "global"
},
"data_stream": {
"dataset": "gcp.dns",
"namespace": "ep",
"type": "logs"
},
"destination": {
"address": "216.239.32.106",
"ip": "216.239.32.106"
},
"dns": {
"answers": [
{
"class": "IN",
"data": "67.43.156.13",
"name": "asdf.gcp.example.com.",
"ttl": 300,
"type": "A"
}
],
"question": {
"name": "asdf.gcp.example.com",
"registered_domain": "example.com",
"subdomain": "asdf.gcp",
"top_level_domain": "com",
"type": "A"
},
"resolved_ip": [
"67.43.156.13"
],
"response_code": "NOERROR"
},
"ecs": {
"version": "8.8.0"
},
"elastic_agent": {
"id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
"snapshot": false,
"version": "8.7.1"
},
"event": {
"action": "dns-query",
"agent_id_status": "verified",
"category": "network",
"created": "2023-10-25T04:19:40.300Z",
"dataset": "gcp.dns",
"id": "zir4wud11tm",
"ingested": "2023-10-25T04:19:41Z",
"kind": "event",
"outcome": "success"
},
"gcp": {
"dns": {
"auth_answer": true,
"destination_ip": "216.239.32.106",
"protocol": "UDP",
"query_name": "asdf.gcp.example.com.",
"query_type": "A",
"response_code": "NOERROR",
"server_latency": 0,
"source_type": "internet",
"target_type": "public-zone"
}
},
"input": {
"type": "gcp-pubsub"
},
"log": {
"level": "INFO",
"logger": "projects/key-reference-123456/logs/dns.googleapis.com%2Fdns_queries"
},
"network": {
"iana_number": "17",
"protocol": "dns",
"transport": "udp"
},
"related": {
"hosts": [
"asdf.gcp.example.com"
],
"ip": [
"67.43.156.13",
"216.239.32.106"
]
},
"tags": [
"forwarded",
"gcp-dns"
]
}
Loadbalancing Logs
The loadbalancing_logs
dataset collects logs of the requests sent to and handled by GCP Load Balancers.
Exported fields
Field | Description | Type |
---|---|---|
@timestamp
|
Event timestamp.
|
date
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
cloud.availability_zone
|
Availability zone in which this host is running.
|
keyword
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
cloud.region
|
Region in which this host is running.
|
keyword
|
container.id
|
Unique container id.
|
keyword
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
container.labels
|
Image labels.
|
object
|
container.name
|
Container name.
|
keyword
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
destination.address
|
Some event destination addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the
.address field. Then it should be duplicated to .ip or .domain , depending on which one it is. |
keyword
|
destination.domain
|
The domain name of the destination system. This value may be a host name, a fully qualified domain name, or another host naming format. The value may derive from the original event or be added from enrichment.
|
keyword
|
destination.ip
|
IP address of the destination (IPv4 or IPv6).
|
ip
|
destination.nat.ip
|
Translated ip of destination based NAT sessions (e.g. internet to private DMZ) Typically used with load balancers, firewalls, or routers.
|
ip
|
destination.nat.port
|
Port the source session is translated to by NAT Device. Typically used with load balancers, firewalls, or routers.
|
long
|
destination.port
|
Port of the destination.
|
long
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
event.created
|
event.created contains the date/time when the event was first read by an agent, or by your pipeline. This field is distinct from @timestamp in that @timestamp typically contain the time extracted from the original event. In most situations, these two timestamps will be slightly different. The difference can be used to calculate the delay between your source generating an event, and the time when your agent first processed it. This can be used to monitor your agent's or pipeline's ability to keep up with your event source. In case the two timestamps are identical, @timestamp should be used.
|
date
|
event.dataset
|
Event dataset
|
constant_keyword
|
event.module
|
Event module
|
constant_keyword
|
gcp.load_balancer.backend_service_name
|
The backend service to which the load balancer is sending traffic
|
keyword
|
gcp.load_balancer.cache_hit
|
Whether or not an entity was served from cache (with or without validation).
|
boolean
|
gcp.load_balancer.cache_id
|
Indicates the location and cache instance that the cache response was served from. For example, a cache response served from a cache in Amsterdam would have a cacheId value of AMS-85e2bd4b, where AMS is the IATA code, and 85e2bd4b is an opaque identifier of the cache instance (because some Cloud CDN locations have multiple discrete caches).
|
keyword
|
gcp.load_balancer.cache_lookup
|
Whether or not a cache lookup was attempted.
|
boolean
|
gcp.load_balancer.forwarding_rule_name
|
The name of the forwarding rule
|
keyword
|
gcp.load_balancer.status_details
|
Explains why the load balancer returned the HTTP status that it did. See https://cloud.google.com/cdn/docs/cdn-logging-monitoring#statusdetail_http_success_messages for specific messages.
|
keyword
|
gcp.load_balancer.target_proxy_name
|
The target proxy name
|
keyword
|
gcp.load_balancer.url_map_name
|
The URL map name
|
keyword
|
host.architecture
|
Operating system architecture.
|
keyword
|
host.containerized
|
If the host is a container.
|
boolean
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
host.ip
|
Host ip addresses.
|
ip
|
host.mac
|
Host mac addresses.
|
keyword
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
host.os.build
|
OS build information.
|
keyword
|
host.os.codename
|
OS codename, if any.
|
keyword
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
http.request.bytes
|
Total size in bytes of the request (body and headers).
|
long
|
http.request.method
|
HTTP request method. The value should retain its casing from the original event. For example,
GET , get , and GeT are all considered valid values for this field. |
keyword
|
http.request.referrer
|
Referrer for this HTTP request.
|
keyword
|
http.response.bytes
|
Total size in bytes of the response (body and headers).
|
long
|
http.response.status_code
|
HTTP response status code.
|
long
|
http.version
|
HTTP version.
|
keyword
|
input.type
|
Input type
|
keyword
|
log.level
|
Original log level of the log event. If the source of the event provides a log level or textual severity, this is the one that goes in
log.level . If your source doesn't specify one, you may put your event transport's severity here (e.g. Syslog severity). Some examples are warn , err , i , informational . |
keyword
|
log.logger
|
The name of the logger inside an application. This is usually the name of the class which initialized the logger, or can be a custom name.
|
keyword
|
log.offset
|
Log offset
|
long
|
network.protocol
|
In the OSI Model this would be the Application Layer protocol. For example,
http , dns , or ssh . The field value must be normalized to lowercase for querying. |
keyword
|
related.hosts
|
All hostnames or other host identifiers seen on your event. Example identifiers include FQDNs, domain names, workstation names, or aliases.
|
keyword
|
related.ip
|
All of the IPs seen on your event.
|
ip
|
source.address
|
Some event source addresses are defined ambiguously. The event will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the
.address field. Then it should be duplicated to .ip or .domain , depending on which one it is. |
keyword
|
source.as.number
|
Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.
|
long
|
source.as.organization.name
|
Organization name.
|
keyword
|
source.as.organization.name.text
|
Multi-field of
source.as.organization.name . |
match_only_text
|
source.geo.city_name
|
City name.
|
keyword
|
source.geo.continent_name
|
Name of the continent.
|
keyword
|
source.geo.country_iso_code
|
Country ISO code.
|
keyword
|
source.geo.country_name
|
Country name.
|
keyword
|
source.geo.location
|
Longitude and latitude.
|
geo_point
|
source.geo.region_iso_code
|
Region ISO code.
|
keyword
|
source.geo.region_name
|
Region name.
|
keyword
|
source.ip
|
IP address of the source (IPv4 or IPv6).
|
ip
|
source.port
|
Port of the source.
|
long
|
tags
|
List of keywords used to tag each event.
|
keyword
|
url.domain
|
Domain of the url, such as "www.elastic.co". In some cases a URL may refer to an IP and/or port directly, without a domain name. In this case, the IP address would go to the
domain field. If the URL contains a literal IPv6 address enclosed by [ and ] (IETF RFC 2732), the [ and ] characters should also be captured in the domain field. |
keyword
|
url.extension
|
The field contains the file extension from the original request url, excluding the leading dot. The file extension is only set if it exists, as not every url has a file extension. The leading period must not be included. For example, the value must be "png", not ".png". Note that when the file name has multiple extensions (example.tar.gz), only the last one should be captured ("gz", not "tar.gz").
|
keyword
|
url.original
|
Unmodified original url as seen in the event source. Note that in network monitoring, the observed URL may be a full URL, whereas in access logs, the URL is often just represented as a path. This field is meant to represent the URL as it was observed, complete or not.
|
wildcard
|
url.original.text
|
Multi-field of
url.original . |
match_only_text
|
url.path
|
Path of the request, such as "/search".
|
wildcard
|
url.port
|
Port of the request, such as 443.
|
long
|
url.query
|
The query field describes the query string of the request, such as "q=elasticsearch". The
? is excluded from the query string. If a URL contains no ? , there is no query field. If there is a ? but no query, the query field exists with an empty string. The exists query can be used to differentiate between the two cases. |
keyword
|
url.scheme
|
Scheme of the request, such as "https". Note: The
: is not part of the scheme. |
keyword
|
user_agent.device.name
|
Name of the device.
|
keyword
|
user_agent.name
|
Name of the user agent.
|
keyword
|
user_agent.original
|
Unparsed user_agent string.
|
keyword
|
user_agent.original.text
|
Multi-field of
user_agent.original . |
match_only_text
|
user_agent.os.full
|
Operating system name, including the version or code name.
|
keyword
|
user_agent.os.full.text
|
Multi-field of
user_agent.os.full . |
match_only_text
|
user_agent.os.name
|
Operating system name, without the version.
|
keyword
|
user_agent.os.name.text
|
Multi-field of
user_agent.os.name . |
match_only_text
|
user_agent.os.version
|
Operating system version as a raw string.
|
keyword
|
user_agent.version
|
Version of the user agent.
|
keyword
|
An example event for loadbalancing
looks as following:
{
"@timestamp": "2020-06-08T23:41:30.078Z",
"agent": {
"ephemeral_id": "f4dde373-2ff7-464b-afdb-da94763f219b",
"id": "5d3eee86-91a9-4afa-af92-c6b79bd866c0",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.6.0"
},
"cloud": {
"project": {
"id": "PROJECT_ID"
},
"region": "global"
},
"data_stream": {
"dataset": "gcp.loadbalancing_logs",
"namespace": "ep",
"type": "logs"
},
"destination": {
"address": "81.2.69.193",
"ip": "81.2.69.193",
"nat": {
"ip": "10.5.3.1",
"port": 9090
},
"port": 8080
},
"ecs": {
"version": "8.8.0"
},
"elastic_agent": {
"id": "5d3eee86-91a9-4afa-af92-c6b79bd866c0",
"snapshot": true,
"version": "8.6.0"
},
"event": {
"agent_id_status": "verified",
"category": "network",
"created": "2020-06-08T23:41:30.588Z",
"dataset": "gcp.loadbalancing_logs",
"id": "1oek5rg3l3fxj7",
"ingested": "2023-01-13T15:02:22Z",
"kind": "event",
"type": "info"
},
"gcp": {
"load_balancer": {
"backend_service_name": "",
"cache_hit": true,
"cache_id": "SFO-fbae48ad",
"cache_lookup": true,
"forwarding_rule_name": "FORWARDING_RULE_NAME",
"status_details": "response_from_cache",
"target_proxy_name": "TARGET_PROXY_NAME",
"url_map_name": "URL_MAP_NAME"
}
},
"http": {
"request": {
"bytes": 577,
"method": "GET",
"referrer": "https://developer.mozilla.org/en-US/docs/Web/JavaScript"
},
"response": {
"bytes": 157,
"status_code": 304
},
"version": "2.0"
},
"input": {
"type": "gcp-pubsub"
},
"log": {
"level": "INFO",
"logger": "projects/PROJECT_ID/logs/requests"
},
"network": {
"protocol": "http"
},
"related": {
"ip": [
"89.160.20.156",
"81.2.69.193",
"10.5.3.1"
]
},
"source": {
"address": "89.160.20.156",
"as": {
"number": 29518,
"organization": {
"name": "Bredband2 AB"
}
},
"geo": {
"city_name": "Linköping",
"continent_name": "Europe",
"country_iso_code": "SE",
"country_name": "Sweden",
"location": {
"lat": 58.4167,
"lon": 15.6167
},
"region_iso_code": "SE-E",
"region_name": "Östergötland County"
},
"ip": "89.160.20.156",
"port": 9989
},
"tags": [
"forwarded",
"gcp-loadbalancing_logs"
],
"url": {
"domain": "81.2.69.193",
"extension": "jpg",
"original": "http://81.2.69.193:8080/static/us/three-cats.jpg",
"path": "/static/us/three-cats.jpg",
"port": 8080,
"scheme": "http"
},
"user_agent": {
"device": {
"name": "Mac"
},
"name": "Chrome",
"original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36",
"os": {
"full": "Mac OS X 10.14.6",
"name": "Mac OS X",
"version": "10.14.6"
},
"version": "83.0.4103.61"
}
}
Metrics
Billing
The billing
dataset collects GCP Billing information from Google Cloud BigQuery daily cost detail table.
Exported fields
Field | Description | Type |
---|---|---|
@timestamp
|
Event timestamp.
|
date
|
cloud
|
Fields related to the cloud or infrastructure the events are coming from.
|
group
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
cloud.account.name
|
The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name.
|
keyword
|
cloud.availability_zone
|
Availability zone in which this host, resource, or service is located.
|
keyword
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
cloud.region
|
Region in which this host, resource, or service is located.
|
keyword
|
container.id
|
Unique container id.
|
keyword
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
container.labels
|
Image labels.
|
object
|
container.name
|
Container name.
|
keyword
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
error
|
These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error.
|
group
|
error.message
|
Error message.
|
match_only_text
|
event.dataset
|
Event dataset
|
constant_keyword
|
event.module
|
Event module
|
constant_keyword
|
gcp.billing.billing_account_id
|
Project Billing Account ID.
|
keyword
|
gcp.billing.cost_type
|
Cost types include regular, tax, adjustment, and rounding_error.
|
keyword
|
gcp.billing.effective_price
|
The charged price for usage of the Google Cloud SKUs and SKU tiers. Reflects contract pricing if applicable, otherwise, it's the list price.
|
float
|
gcp.billing.invoice_month
|
Billing report month.
|
keyword
|
gcp.billing.project_id
|
Project ID of the billing report belongs to.
|
keyword
|
gcp.billing.project_name
|
Project Name of the billing report belongs to.
|
keyword
|
gcp.billing.service_description
|
The Google Cloud service that reported the Cloud Billing data.
|
keyword
|
gcp.billing.service_id
|
The ID of the service that the usage is associated with.
|
keyword
|
gcp.billing.sku_description
|
A description of the resource type used by the service. For example, a resource type for Cloud Storage is Standard Storage US.
|
keyword
|
gcp.billing.sku_id
|
The ID of the resource used by the service.
|
keyword
|
gcp.billing.tags.key
|
keyword
|
|
gcp.billing.tags.value
|
keyword
|
|
gcp.billing.total
|
Total billing amount.
|
float
|
host.architecture
|
Operating system architecture.
|
keyword
|
host.containerized
|
If the host is a container.
|
boolean
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
host.ip
|
Host ip addresses.
|
ip
|
host.mac
|
Host mac addresses.
|
keyword
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
host.os.build
|
OS build information.
|
keyword
|
host.os.codename
|
OS codename, if any.
|
keyword
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
service.type
|
The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch,
service.type would be elasticsearch . |
keyword
|
An example event for billing
looks as following:
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"cloud": {
"account": {
"id": "01475F-5B1080-1137E7"
},
"project": {
"id": "elastic-bi",
"name": "elastic-containerlib-prod"
},
"provider": "gcp"
},
"event": {
"dataset": "gcp.billing",
"duration": 115000,
"module": "gcp"
},
"gcp": {
"billing": {
"billing_account_id": "01475F-5B1080-1137E7",
"cost_type": "regular",
"invoice_month": "202106",
"project_id": "containerlib-prod-12763",
"project_name": "elastic-containerlib-prod",
"total": 4717.170681,
"sku_id": "0D56-2F80-52A5",
"service_id": "6F81-5844-456A",
"sku_description": "Network Inter Region Ingress from Jakarta to Americas",
"service_description": "Compute Engine",
"effective_price": 0.00292353,
"tags": [
{
"key": "stage",
"value": "prod"
},
{
"key": "size",
"value": "standard"
}
]
}
},
"metricset": {
"name": "billing",
"period": 10000
},
"service": {
"type": "gcp"
}
}
Compute
The compute
dataset is designed to fetch metrics for Compute Engine Virtual Machines in Google Cloud Platform.
Exported fields
Field | Description | Type | Metric Type |
---|---|---|---|
@timestamp
|
Event timestamp.
|
date
|
|
agent.id
|
Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.
|
keyword
|
|
cloud
|
Fields related to the cloud or infrastructure the events are coming from.
|
group
|
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
|
cloud.account.name
|
The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name.
|
keyword
|
|
cloud.availability_zone
|
Availability zone in which this host, resource, or service is located.
|
keyword
|
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
|
cloud.region
|
Region in which this host, resource, or service is located.
|
keyword
|
|
container.id
|
Unique container id.
|
keyword
|
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
|
container.labels
|
Image labels.
|
object
|
|
container.name
|
Container name.
|
keyword
|
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
|
error
|
These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error.
|
group
|
|
error.message
|
Error message.
|
match_only_text
|
|
event.dataset
|
Event dataset
|
constant_keyword
|
|
event.module
|
Event module
|
constant_keyword
|
|
gcp.compute.firewall.dropped.bytes
|
Delta of incoming bytes dropped by the firewall
|
long
|
gauge
|
gcp.compute.firewall.dropped_packets_count.value
|
Delta of incoming packets dropped by the firewall
|
long
|
gauge
|
gcp.compute.instance.cpu.reserved_cores.value
|
Number of cores reserved on the host of the instance
|
double
|
gauge
|
gcp.compute.instance.cpu.usage.pct
|
The fraction of the allocated CPU that is currently in use on the instance
|
double
|
gauge
|
gcp.compute.instance.cpu.usage_time.sec
|
Delta of usage for all cores in seconds
|
double
|
gauge
|
gcp.compute.instance.disk.read.bytes
|
Delta of count of bytes read from disk
|
long
|
gauge
|
gcp.compute.instance.disk.read_ops_count.value
|
Delta of count of disk read IO operations
|
long
|
gauge
|
gcp.compute.instance.disk.write.bytes
|
Delta of count of bytes written to disk
|
long
|
gauge
|
gcp.compute.instance.disk.write_ops_count.value
|
Delta of count of disk write IO operations
|
long
|
gauge
|
gcp.compute.instance.memory.balloon.ram_size.value
|
The total amount of memory in the VM. This metric is only available for VMs that belong to the e2 family.
|
long
|
gauge
|
gcp.compute.instance.memory.balloon.ram_used.value
|
Memory currently used in the VM. This metric is only available for VMs that belong to the e2 family.
|
long
|
gauge
|
gcp.compute.instance.memory.balloon.swap_in.bytes
|
Delta of the amount of memory read into the guest from its own swap space. This metric is only available for VMs that belong to the e2 family.
|
long
|
gauge
|
gcp.compute.instance.memory.balloon.swap_out.bytes
|
Delta of the amount of memory written from the guest to its own swap space. This metric is only available for VMs that belong to the e2 family.
|
long
|
gauge
|
gcp.compute.instance.network.egress.bytes
|
Delta of count of bytes sent over the network
|
long
|
gauge
|
gcp.compute.instance.network.egress.packets.count
|
Delta of count of packets sent over the network
|
long
|
gauge
|
gcp.compute.instance.network.ingress.bytes
|
Delta of count of bytes received from the network
|
long
|
gauge
|
gcp.compute.instance.network.ingress.packets.count
|
Delta of count of packets received from the network
|
long
|
gauge
|
gcp.compute.instance.uptime.sec
|
Delta of number of seconds the VM has been running.
|
long
|
gauge
|
gcp.compute.instance.uptime_total.sec
|
Elapsed time since the VM was started, in seconds. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
|
long
|
gauge
|
gcp.labels.metadata.*
|
object
|
||
gcp.labels.metrics.*
|
object
|
||
gcp.labels.resource.*
|
object
|
||
gcp.labels.system.*
|
object
|
||
gcp.labels.user.*
|
object
|
||
gcp.labels_fingerprint
|
Hashed value of the labels field.
|
keyword
|
|
gcp.metrics.*.*.*.*
|
Metrics that returned from Google Cloud API query.
|
object
|
|
host.architecture
|
Operating system architecture.
|
keyword
|
|
host.containerized
|
If the host is a container.
|
boolean
|
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
|
host.ip
|
Host ip addresses.
|
ip
|
|
host.mac
|
Host mac addresses.
|
keyword
|
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
|
host.os.build
|
OS build information.
|
keyword
|
|
host.os.codename
|
OS codename, if any.
|
keyword
|
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
|
service.type
|
The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch,
service.type would be elasticsearch . |
keyword
|
An example event for compute
looks as following:
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"cloud": {
"account": {
"id": "elastic-obs-integrations-dev",
"name": "elastic-obs-integrations-dev"
},
"instance": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"machine": {
"type": "e2-medium"
},
"provider": "gcp",
"availability_zone": "us-central1-c",
"region": "us-central1"
},
"event": {
"dataset": "gcp.compute",
"duration": 115000,
"module": "gcp"
},
"gcp": {
"compute": {
"firewall": {
"dropped": {
"bytes": 421
},
"dropped_packets_count": {
"value": 4
}
},
"instance": {
"cpu": {
"reserved_cores": {
"value": 1
},
"usage": {
"pct": 0.07259952346383708
},
"usage_time": {
"sec": 4.355971407830225
}
},
"memory": {
"balloon": {
"ram_size": {
"value": 4128378880
},
"ram_used": {
"value": 2190848000
},
"swap_in": {
"bytes": 0
},
"swap_out": {
"bytes": 0
}
}
},
"uptime": {
"sec": 60.00000000000091
}
}
},
"labels": {
"user": {
"goog-gke-node": ""
}
}
},
"host": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"metricset": {
"name": "compute",
"period": 10000
},
"service": {
"type": "gcp"
}
}
Dataproc
The dataproc
dataset is designed to fetch metrics from Dataproc in Google Cloud Platform.
Exported fields
Field | Description | Type | Metric Type |
---|---|---|---|
@timestamp
|
Event timestamp.
|
date
|
|
agent.id
|
Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.
|
keyword
|
|
cloud
|
Fields related to the cloud or infrastructure the events are coming from.
|
group
|
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
|
cloud.account.name
|
The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name.
|
keyword
|
|
cloud.availability_zone
|
Availability zone in which this host, resource, or service is located.
|
keyword
|
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
|
cloud.region
|
Region in which this host, resource, or service is located.
|
keyword
|
|
container.id
|
Unique container id.
|
keyword
|
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
|
container.labels
|
Image labels.
|
object
|
|
container.name
|
Container name.
|
keyword
|
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
|
error
|
These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error.
|
group
|
|
error.message
|
Error message.
|
match_only_text
|
|
event.dataset
|
Event dataset
|
constant_keyword
|
|
event.module
|
Event module
|
constant_keyword
|
|
gcp.dataproc.batch.spark.executors.count
|
Indicates the number of Batch Spark executors.
|
long
|
gauge
|
gcp.dataproc.cluster.hdfs.datanodes.count
|
Indicates the number of HDFS DataNodes that are running inside a cluster.
|
long
|
gauge
|
gcp.dataproc.cluster.hdfs.storage_capacity.value
|
Indicates capacity of HDFS system running on cluster in GB.
|
double
|
gauge
|
gcp.dataproc.cluster.hdfs.storage_utilization.value
|
The percentage of HDFS storage currently used.
|
double
|
gauge
|
gcp.dataproc.cluster.hdfs.unhealthy_blocks.count
|
Indicates the number of unhealthy blocks inside the cluster.
|
long
|
gauge
|
gcp.dataproc.cluster.job.completion_time.value
|
The time jobs took to complete from the time the user submits a job to the time Dataproc reports it is completed.
|
object
|
|
gcp.dataproc.cluster.job.duration.value
|
The time jobs have spent in a given state.
|
object
|
|
gcp.dataproc.cluster.job.failed.count
|
Indicates the delta of the number of jobs that have failed on a cluster.
|
long
|
gauge
|
gcp.dataproc.cluster.job.running.count
|
Indicates the number of jobs that are running on a cluster.
|
long
|
gauge
|
gcp.dataproc.cluster.job.submitted.count
|
Indicates the delta of the number of jobs that have been submitted to a cluster.
|
long
|
gauge
|
gcp.dataproc.cluster.operation.completion_time.value
|
The time operations took to complete from the time the user submits a operation to the time Dataproc reports it is completed.
|
object
|
|
gcp.dataproc.cluster.operation.duration.value
|
The time operations have spent in a given state.
|
object
|
|
gcp.dataproc.cluster.operation.failed.count
|
Indicates the delta of the number of operations that have failed on a cluster.
|
long
|
gauge
|
gcp.dataproc.cluster.operation.running.count
|
Indicates the number of operations that are running on a cluster.
|
long
|
gauge
|
gcp.dataproc.cluster.operation.submitted.count
|
Indicates the delta of the number of operations that have been submitted to a cluster.
|
long
|
gauge
|
gcp.dataproc.cluster.yarn.allocated_memory_percentage.value
|
The percentage of YARN memory is allocated.
|
double
|
gauge
|
gcp.dataproc.cluster.yarn.apps.count
|
Indicates the number of active YARN applications.
|
long
|
gauge
|
gcp.dataproc.cluster.yarn.containers.count
|
Indicates the number of YARN containers.
|
long
|
gauge
|
gcp.dataproc.cluster.yarn.memory_size.value
|
Indicates the YARN memory size in GB.
|
double
|
gauge
|
gcp.dataproc.cluster.yarn.nodemanagers.count
|
Indicates the number of YARN NodeManagers running inside cluster.
|
long
|
gauge
|
gcp.dataproc.cluster.yarn.pending_memory_size.value
|
The current memory request, in GB, that is pending to be fulfilled by the scheduler.
|
double
|
gauge
|
gcp.dataproc.cluster.yarn.virtual_cores.count
|
Indicates the number of virtual cores in YARN.
|
long
|
gauge
|
gcp.labels.metadata.*
|
object
|
||
gcp.labels.metrics.*
|
object
|
||
gcp.labels.resource.*
|
object
|
||
gcp.labels.system.*
|
object
|
||
gcp.labels.user.*
|
object
|
||
gcp.labels_fingerprint
|
Hashed value of the labels field.
|
keyword
|
|
gcp.metrics.*.*.*.*
|
Metrics that returned from Google Cloud API query.
|
object
|
|
host.architecture
|
Operating system architecture.
|
keyword
|
|
host.containerized
|
If the host is a container.
|
boolean
|
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
|
host.ip
|
Host ip addresses.
|
ip
|
|
host.mac
|
Host mac addresses.
|
keyword
|
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
|
host.os.build
|
OS build information.
|
keyword
|
|
host.os.codename
|
OS codename, if any.
|
keyword
|
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
|
service.type
|
The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch,
service.type would be elasticsearch . |
keyword
|
An example event for dataproc
looks as following:
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"cloud": {
"account": {
"id": "elastic-obs-integrations-dev",
"name": "elastic-obs-integrations-dev"
},
"instance": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"machine": {
"type": "e2-medium"
},
"provider": "gcp",
"availability_zone": "us-central1-c",
"region": "us-central1"
},
"event": {
"dataset": "gcp.dataproc",
"duration": 115000,
"module": "gcp"
},
"gcp": {
"dataproc": {
"cluster": {
"hdfs": {
"datanodes": {
"count": 15
}
}
}
},
"labels": {
"user": {
"goog-gke-node": ""
}
}
},
"host": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"metricset": {
"name": "dataproc",
"period": 10000
},
"service": {
"type": "gcp"
}
}
Firestore
The firestore
dataset fetches metrics from Firestore in Google Cloud Platform.
Exported fields
Field | Description | Type | Metric Type |
---|---|---|---|
@timestamp
|
Event timestamp.
|
date
|
|
cloud
|
Fields related to the cloud or infrastructure the events are coming from.
|
group
|
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
|
cloud.account.name
|
The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name.
|
keyword
|
|
cloud.availability_zone
|
Availability zone in which this host, resource, or service is located.
|
keyword
|
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
|
cloud.region
|
Region in which this host, resource, or service is located.
|
keyword
|
|
container.id
|
Unique container id.
|
keyword
|
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
|
container.labels
|
Image labels.
|
object
|
|
container.name
|
Container name.
|
keyword
|
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
|
error
|
These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error.
|
group
|
|
error.message
|
Error message.
|
match_only_text
|
|
event.dataset
|
Event dataset
|
constant_keyword
|
|
event.module
|
Event module
|
constant_keyword
|
|
gcp.firestore.document.delete.count
|
Delta of the number of successful document deletes.
|
long
|
gauge
|
gcp.firestore.document.read.count
|
Delta of the number of successful document reads from queries or lookups.
|
long
|
gauge
|
gcp.firestore.document.write.count
|
Delta of the number of successful document writes.
|
long
|
gauge
|
gcp.labels.metadata.*
|
object
|
||
gcp.labels.metrics.*
|
object
|
||
gcp.labels.resource.*
|
object
|
||
gcp.labels.system.*
|
object
|
||
gcp.labels.user.*
|
object
|
||
gcp.metrics.*.*.*.*
|
Metrics that returned from Google Cloud API query.
|
object
|
|
host.architecture
|
Operating system architecture.
|
keyword
|
|
host.containerized
|
If the host is a container.
|
boolean
|
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
|
host.ip
|
Host ip addresses.
|
ip
|
|
host.mac
|
Host mac addresses.
|
keyword
|
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
|
host.os.build
|
OS build information.
|
keyword
|
|
host.os.codename
|
OS codename, if any.
|
keyword
|
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
|
service.type
|
The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch,
service.type would be elasticsearch . |
keyword
|
An example event for firestore
looks as following:
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"cloud": {
"account": {
"id": "elastic-obs-integrations-dev",
"name": "elastic-obs-integrations-dev"
},
"instance": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"machine": {
"type": "e2-medium"
},
"provider": "gcp",
"availability_zone": "us-central1-c",
"region": "us-central1"
},
"event": {
"dataset": "gcp.firestore",
"duration": 115000,
"module": "gcp"
},
"gcp": {
"firestore": {
"document": {
"delete": {
"count": 3
},
"read": {
"count": 10
},
"write": {
"count": 1
}
}
},
"labels": {
"user": {
"goog-gke-node": ""
}
}
},
"host": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"metricset": {
"name": "firestore",
"period": 10000
},
"service": {
"type": "gcp"
}
}
GKE
The gke
dataset is designed to fetch metrics from GKE in Google Cloud Platform.
Exported fields
Field | Description | Type | Metric Type |
---|---|---|---|
@timestamp
|
Event timestamp.
|
date
|
|
agent.id
|
Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.
|
keyword
|
|
cloud
|
Fields related to the cloud or infrastructure the events are coming from.
|
group
|
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
|
cloud.account.name
|
The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name.
|
keyword
|
|
cloud.availability_zone
|
Availability zone in which this host, resource, or service is located.
|
keyword
|
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
|
cloud.region
|
Region in which this host, resource, or service is located.
|
keyword
|
|
container.id
|
Unique container id.
|
keyword
|
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
|
container.labels
|
Image labels.
|
object
|
|
container.name
|
Container name.
|
keyword
|
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
|
error
|
These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error.
|
group
|
|
error.message
|
Error message.
|
match_only_text
|
|
event.dataset
|
Event dataset
|
constant_keyword
|
|
event.module
|
Event module
|
constant_keyword
|
|
gcp.gke.container.cpu.core_usage_time.sec
|
Cumulative CPU usage on all cores used by the container in seconds. Sampled every 60 seconds.
|
double
|
counter
|
gcp.gke.container.cpu.limit_cores.value
|
CPU cores limit of the container. Sampled every 60 seconds.
|
double
|
gauge
|
gcp.gke.container.cpu.limit_utilization.pct
|
The fraction of the CPU limit that is currently in use on the instance. This value cannot exceed 1 as usage cannot exceed the limit. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds.
|
double
|
gauge
|
gcp.gke.container.cpu.request_cores.value
|
Number of CPU cores requested by the container. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
|
double
|
gauge
|
gcp.gke.container.cpu.request_utilization.pct
|
The fraction of the requested CPU that is currently in use on the instance. This value can be greater than 1 as usage can exceed the request. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds.
|
double
|
gauge
|
gcp.gke.container.ephemeral_storage.limit.bytes
|
Local ephemeral storage limit in bytes. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.container.ephemeral_storage.request.bytes
|
Local ephemeral storage request in bytes. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.container.ephemeral_storage.used.bytes
|
Local ephemeral storage usage in bytes. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.container.memory.limit.bytes
|
Memory limit of the container in bytes. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.container.memory.limit_utilization.pct
|
The fraction of the memory limit that is currently in use on the instance. This value cannot exceed 1 as usage cannot exceed the limit. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
|
double
|
gauge
|
gcp.gke.container.memory.page_fault.count
|
Number of page faults, broken down by type, major and minor.
|
long
|
counter
|
gcp.gke.container.memory.request.bytes
|
Memory request of the container in bytes. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
|
long
|
gauge
|
gcp.gke.container.memory.request_utilization.pct
|
The fraction of the requested memory that is currently in use on the instance. This value can be greater than 1 as usage can exceed the request. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds.
|
double
|
gauge
|
gcp.gke.container.memory.used.bytes
|
Memory usage in bytes. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.container.restart.count
|
Number of times the container has restarted. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
|
long
|
counter
|
gcp.gke.container.uptime.sec
|
Time in seconds that the container has been running. Sampled every 60 seconds.
|
double
|
gauge
|
gcp.gke.node.cpu.allocatable_cores.value
|
Number of allocatable CPU cores on the node. Sampled every 60 seconds.
|
double
|
gauge
|
gcp.gke.node.cpu.allocatable_utilization.pct
|
The fraction of the allocatable CPU that is currently in use on the instance. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds.
|
double
|
gauge
|
gcp.gke.node.cpu.core_usage_time.sec
|
Cumulative CPU usage on all cores used on the node in seconds. Sampled every 60 seconds.
|
double
|
counter
|
gcp.gke.node.cpu.total_cores.value
|
Total number of CPU cores on the node. Sampled every 60 seconds.
|
double
|
gauge
|
gcp.gke.node.ephemeral_storage.allocatable.bytes
|
Local ephemeral storage bytes allocatable on the node. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node.ephemeral_storage.inodes_free.value
|
Free number of inodes on local ephemeral storage. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node.ephemeral_storage.inodes_total.value
|
Total number of inodes on local ephemeral storage. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node.ephemeral_storage.total.bytes
|
Total ephemeral storage bytes on the node. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node.ephemeral_storage.used.bytes
|
Local ephemeral storage bytes used by the node. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node.memory.allocatable.bytes
|
Cumulative memory bytes used by the node. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node.memory.allocatable_utilization.pct
|
The fraction of the allocatable memory that is currently in use on the instance. This value cannot exceed 1 as usage cannot exceed allocatable memory bytes. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
|
double
|
gauge
|
gcp.gke.node.memory.total.bytes
|
Number of bytes of memory allocatable on the node. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node.memory.used.bytes
|
Cumulative memory bytes used by the node. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node.network.received_bytes.count
|
Cumulative number of bytes received by the node over the network. Sampled every 60 seconds.
|
long
|
counter
|
gcp.gke.node.network.sent_bytes.count
|
Cumulative number of bytes transmitted by the node over the network. Sampled every 60 seconds.
|
long
|
counter
|
gcp.gke.node.pid_limit.value
|
The max PID of OS on the node. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node.pid_used.value
|
The number of running process in the OS on the node. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.node_daemon.cpu.core_usage_time.sec
|
Cumulative CPU usage on all cores used by the node level system daemon in seconds. Sampled every 60 seconds.
|
double
|
counter
|
gcp.gke.node_daemon.memory.used.bytes
|
Memory usage by the system daemon in bytes. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.pod.network.received.bytes
|
Cumulative number of bytes received by the pod over the network. Sampled every 60 seconds.
|
long
|
counter
|
gcp.gke.pod.network.sent.bytes
|
Cumulative number of bytes transmitted by the pod over the network. Sampled every 60 seconds.
|
long
|
counter
|
gcp.gke.pod.volume.total.bytes
|
Total number of disk bytes available to the pod. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
|
long
|
gauge
|
gcp.gke.pod.volume.used.bytes
|
Number of disk bytes used by the pod. Sampled every 60 seconds.
|
long
|
gauge
|
gcp.gke.pod.volume.utilization.pct
|
The fraction of the volume that is currently being used by the instance. This value cannot be greater than 1 as usage cannot exceed the total available volume space. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
|
double
|
gauge
|
gcp.labels.metadata.*
|
object
|
||
gcp.labels.metrics.*
|
object
|
||
gcp.labels.resource.*
|
object
|
||
gcp.labels.system.*
|
object
|
||
gcp.labels.user.*
|
object
|
||
gcp.labels_fingerprint
|
Hashed value of the labels field.
|
keyword
|
|
gcp.metrics.*.*.*.*
|
Metrics that returned from Google Cloud API query.
|
object
|
|
host.architecture
|
Operating system architecture.
|
keyword
|
|
host.containerized
|
If the host is a container.
|
boolean
|
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
|
host.ip
|
Host ip addresses.
|
ip
|
|
host.mac
|
Host mac addresses.
|
keyword
|
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
|
host.os.build
|
OS build information.
|
keyword
|
|
host.os.codename
|
OS codename, if any.
|
keyword
|
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
|
service.type
|
The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch,
service.type would be elasticsearch . |
keyword
|
An example event for gke
looks as following:
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"cloud": {
"account": {
"id": "elastic-obs-integrations-dev",
"name": "elastic-obs-integrations-dev"
},
"instance": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"machine": {
"type": "e2-medium"
},
"provider": "gcp",
"availability_zone": "us-central1-c",
"region": "us-central1"
},
"event": {
"dataset": "gcp.gke",
"duration": 115000,
"module": "gcp"
},
"gcp": {
"gke": {
"container": {
"cpu": {
"core_usage_time": {
"sec": 15
}
}
}
},
"labels": {
"user": {
"goog-gke-node": ""
}
}
},
"host": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"metricset": {
"name": "gke",
"period": 10000
},
"service": {
"type": "gcp"
}
}
Loadbalancing Metrics
The loadbalancing_metrics
dataset is designed to fetch HTTPS, HTTP, and Layer 3 metrics from Load Balancing in Google Cloud Platform.
Exported fields
Field | Description | Type | Metric Type |
---|---|---|---|
@timestamp
|
Event timestamp.
|
date
|
|
agent.id
|
Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.
|
keyword
|
|
cloud
|
Fields related to the cloud or infrastructure the events are coming from.
|
group
|
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
|
cloud.account.name
|
The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name.
|
keyword
|
|
cloud.availability_zone
|
Availability zone in which this host, resource, or service is located.
|
keyword
|
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
|
cloud.region
|
Region in which this host, resource, or service is located.
|
keyword
|
|
container.id
|
Unique container id.
|
keyword
|
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
|
container.labels
|
Image labels.
|
object
|
|
container.name
|
Container name.
|
keyword
|
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
|
error
|
These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error.
|
group
|
|
error.message
|
Error message.
|
match_only_text
|
|
event.dataset
|
Event dataset
|
constant_keyword
|
|
event.module
|
Event module
|
constant_keyword
|
|
gcp.labels.metadata.*
|
object
|
||
gcp.labels.metrics.*
|
object
|
||
gcp.labels.resource.*
|
object
|
||
gcp.labels.system.*
|
object
|
||
gcp.labels.user.*
|
object
|
||
gcp.labels_fingerprint
|
Hashed value of the labels field.
|
keyword
|
|
gcp.loadbalancing_metrics.https.backend_latencies.value
|
A distribution of the latency calculated from when the request was sent by the proxy to the backend until the proxy received from the backend the last byte of response.
|
object
|
|
gcp.loadbalancing_metrics.https.backend_request.bytes
|
Delta of the number of bytes sent as requests from HTTP/S load balancer to backends.
|
long
|
gauge
|
gcp.loadbalancing_metrics.https.backend_request.count
|
Delta of the number of requests served by backends of HTTP/S load balancer.
|
long
|
gauge
|
gcp.loadbalancing_metrics.https.backend_response.bytes
|
Delta of the number of bytes sent as responses from backends (or cache) to external HTTP(S) load balancer.
|
long
|
gauge
|
gcp.loadbalancing_metrics.https.external.regional.backend_latencies.value
|
A distribution of the latency calculated from when the request was sent by the proxy to the backend until the proxy received from the backend the last byte of response.
|
object
|
|
gcp.loadbalancing_metrics.https.external.regional.total_latencies.value
|
A distribution of the latency calculated from when the request was received by the proxy until the proxy got ACK from client on last response byte.
|
object
|
|
gcp.loadbalancing_metrics.https.frontend_tcp_rtt.value
|
A distribution of the RTT measured for each connection between client and proxy.
|
object
|
|
gcp.loadbalancing_metrics.https.internal.backend_latencies.value
|
A distribution of the latency calculated from when the request was sent by the internal HTTP/S load balancer proxy to the backend until the proxy received from the backend the last byte of response.
|
object
|
|
gcp.loadbalancing_metrics.https.internal.total_latencies.value
|
A distribution of the latency calculated from when the request was received by the internal HTTP/S load balancer proxy until the proxy got ACK from client on last response byte.
|
object
|
|
gcp.loadbalancing_metrics.https.request.bytes
|
Delta of the number of bytes sent as requests from clients to HTTP/S load balancer.
|
long
|
gauge
|
gcp.loadbalancing_metrics.https.request.count
|
Delta of the number of requests served by HTTP/S load balancer.
|
long
|
gauge
|
gcp.loadbalancing_metrics.https.response.bytes
|
Delta of the number of bytes sent as responses from HTTP/S load balancer to clients.
|
long
|
gauge
|
gcp.loadbalancing_metrics.https.total_latencies.value
|
A distribution of the latency calculated from when the request was received by the external HTTP/S load balancer proxy until the proxy got ACK from client on last response byte.
|
object
|
|
gcp.loadbalancing_metrics.l3.external.egress.bytes
|
Delta of the number of bytes sent from external TCP/UDP network load balancer backend to client of the flow. For TCP flows it's counting bytes on application stream only.
|
long
|
gauge
|
gcp.loadbalancing_metrics.l3.external.egress_packets.count
|
Delta of the number of packets sent from external TCP/UDP network load balancer backend to client of the flow.
|
long
|
gauge
|
gcp.loadbalancing_metrics.l3.external.ingress.bytes
|
Delta of the number of bytes sent from client to external TCP/UDP network load balancer backend. For TCP flows it's counting bytes on application stream only.
|
long
|
gauge
|
gcp.loadbalancing_metrics.l3.external.ingress_packets.count
|
Delta of the number of packets sent from client to external TCP/UDP network load balancer backend.
|
long
|
gauge
|
gcp.loadbalancing_metrics.l3.external.rtt_latencies.value
|
A distribution of the round trip time latency, measured over TCP connections for the external network load balancer.
|
object
|
|
gcp.loadbalancing_metrics.l3.internal.egress.bytes
|
Delta of the number of bytes sent from ILB backend to client (for TCP flows it's counting bytes on application stream only).
|
long
|
gauge
|
gcp.loadbalancing_metrics.l3.internal.egress_packets.count
|
Delta of the number of packets sent from ILB backend to client of the flow.
|
long
|
gauge
|
gcp.loadbalancing_metrics.l3.internal.ingress.bytes
|
Delta of the number of bytes sent from client to ILB backend (for TCP flows it's counting bytes on application stream only).
|
long
|
gauge
|
gcp.loadbalancing_metrics.l3.internal.ingress_packets.count
|
Delta of the number of packets sent from client to ILB backend.
|
long
|
gauge
|
gcp.loadbalancing_metrics.l3.internal.rtt_latencies.value
|
A distribution of RTT measured over TCP connections for internal TCP/UDP load balancer flows.
|
object
|
|
gcp.loadbalancing_metrics.tcp_ssl_proxy.closed_connections.value
|
Delta of the number of connections that were terminated over TCP/SSL proxy.
|
long
|
gauge
|
gcp.loadbalancing_metrics.tcp_ssl_proxy.egress.bytes
|
Delta of the number of bytes sent from VM to client using proxy.
|
long
|
gauge
|
gcp.loadbalancing_metrics.tcp_ssl_proxy.frontend_tcp_rtt.value
|
A distribution of the smoothed RTT (in ms) measured by the proxy's TCP stack, each minute application layer bytes pass from proxy to client.
|
object
|
|
gcp.loadbalancing_metrics.tcp_ssl_proxy.ingress.bytes
|
Delta of the number of bytes sent from client to VM using proxy.
|
long
|
gauge
|
gcp.loadbalancing_metrics.tcp_ssl_proxy.new_connections.value
|
Delta of the number of connections that were created over TCP/SSL proxy.
|
long
|
gauge
|
gcp.loadbalancing_metrics.tcp_ssl_proxy.open_connections.value
|
Current number of outstanding connections through the TCP/SSL proxy.
|
long
|
gauge
|
gcp.metrics.*.*.*.*
|
Metrics that returned from Google Cloud API query.
|
object
|
|
host.architecture
|
Operating system architecture.
|
keyword
|
|
host.containerized
|
If the host is a container.
|
boolean
|
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
|
host.ip
|
Host ip addresses.
|
ip
|
|
host.mac
|
Host mac addresses.
|
keyword
|
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
|
host.os.build
|
OS build information.
|
keyword
|
|
host.os.codename
|
OS codename, if any.
|
keyword
|
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
|
service.type
|
The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch,
service.type would be elasticsearch . |
keyword
|
An example event for loadbalancing
looks as following:
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"cloud": {
"account": {
"id": "elastic-observability"
},
"provider": "gcp",
"region": "us-central1",
"availability_zone": "us-central1-a"
},
"event": {
"dataset": "gcp.loadbalancing_metrics",
"duration": 115000,
"module": "gcp"
},
"gcp": {
"labels": {
"metrics": {
"client_network": "ocp-be-c5kjr-network",
"client_subnetwork": "ocp-be-c5kjr-worker-subnet",
"client_zone": "us-central1-a"
},
"resource": {
"backend_name": "ocp-be-c5kjr-master-us-central1-a",
"backend_scope": "us-central1-a",
"backend_scope_type": "ZONE",
"backend_subnetwork_name": "ocp-be-c5kjr-master-subnet",
"backend_target_name": "ocp-be-c5kjr-api-internal",
"backend_target_type": "BACKEND_SERVICE",
"backend_type": "INSTANCE_GROUP",
"forwarding_rule_name": "ocp-be-c5kjr-api-internal",
"load_balancer_name": "ocp-be-c5kjr-api-internal",
"network_name": "ocp-be-c5kjr-network",
"region": "us-central1"
}
},
"loadbalancing_metrics": {
"l3": {
"internal": {
"egress_packets": {
"count": 100
},
"egress": {
"bytes": 1247589
}
}
}
}
},
"metricset": {
"name": "loadbalancing",
"period": 10000
},
"service": {
"type": "gcp"
}
}
Redis
The redis
dataset is designed to fetch metrics from GCP Memorystore for Redis in Google Cloud Platform.
Exported fields
Field | Description | Type | Unit | Metric Type |
---|---|---|---|---|
@timestamp
|
Event timestamp.
|
date
|
||
agent.id
|
Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.
|
keyword
|
||
cloud
|
Fields related to the cloud or infrastructure the events are coming from.
|
group
|
||
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
||
cloud.account.name
|
The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name.
|
keyword
|
||
cloud.availability_zone
|
Availability zone in which this host, resource, or service is located.
|
keyword
|
||
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
||
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
||
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
||
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
||
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
||
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
||
cloud.region
|
Region in which this host, resource, or service is located.
|
keyword
|
||
container.id
|
Unique container id.
|
keyword
|
||
container.image.name
|
Name of the image the container was built on.
|
keyword
|
||
container.labels
|
Image labels.
|
object
|
||
container.name
|
Container name.
|
keyword
|
||
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
||
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
||
data_stream.type
|
Data stream type.
|
constant_keyword
|
||
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
||
error
|
These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error.
|
group
|
||
error.message
|
Error message.
|
match_only_text
|
||
event.dataset
|
Event dataset
|
constant_keyword
|
||
event.module
|
Event module
|
constant_keyword
|
||
gcp.labels.metadata.*
|
object
|
|||
gcp.labels.metrics.*
|
object
|
|||
gcp.labels.resource.*
|
object
|
|||
gcp.labels.system.*
|
object
|
|||
gcp.labels.user.*
|
object
|
|||
gcp.labels_fingerprint
|
Hashed value of the labels field.
|
keyword
|
||
gcp.metrics.*.*.*.*
|
Metrics that returned from Google Cloud API query.
|
object
|
||
gcp.redis.clients.blocked.count
|
Number of blocked clients.
|
long
|
gauge
|
|
gcp.redis.clients.connected.count
|
Number of client connections.
|
long
|
gauge
|
|
gcp.redis.commands.calls.count
|
Delta of the number of calls for this command in one minute.
|
long
|
gauge
|
|
gcp.redis.commands.total_time.us
|
Delta of the amount of time in microseconds that this command took in the last second.
|
long
|
micros
|
gauge
|
gcp.redis.commands.usec_per_call.sec
|
Average time per call over 1 minute by command.
|
double
|
s
|
gauge
|
gcp.redis.keyspace.avg_ttl.sec
|
Average TTL for keys in this database.
|
double
|
s
|
gauge
|
gcp.redis.keyspace.keys.count
|
Number of keys stored in this database.
|
long
|
gauge
|
|
gcp.redis.keyspace.keys_with_expiration.count
|
Number of keys with an expiration in this database.
|
long
|
gauge
|
|
gcp.redis.persistence.rdb.bgsave_in_progress
|
Flag indicating a RDB save is on-going.
|
long
|
gauge
|
|
gcp.redis.replication.master.slaves.lag.sec
|
The number of seconds that replica is lagging behind primary.
|
long
|
s
|
gauge
|
gcp.redis.replication.master.slaves.offset.bytes
|
The number of bytes that have been acknowledged by replicas.
|
long
|
byte
|
gauge
|
gcp.redis.replication.master_repl_offset.bytes
|
The number of bytes that master has produced and sent to replicas.
|
long
|
byte
|
gauge
|
gcp.redis.replication.offset_diff.bytes
|
The largest number of bytes that have not been replicated across all replicas. This is the biggest difference between replication byte offset (master) and replication byte offset (replica) of all replicas.
|
long
|
byte
|
gauge
|
gcp.redis.replication.role
|
Returns a value indicating the node role. 1 indicates primary and 0 indicates replica.
|
long
|
gauge
|
|
gcp.redis.server.uptime.sec
|
Uptime in seconds.
|
long
|
s
|
gauge
|
gcp.redis.stats.cache_hit_ratio
|
Cache Hit ratio as a fraction.
|
double
|
gauge
|
|
gcp.redis.stats.connections.total.count
|
Delta of the total number of connections accepted by the server.
|
long
|
gauge
|
|
gcp.redis.stats.cpu_utilization.sec
|
CPU-seconds consumed by the Redis server, broken down by system/user space and parent/child relationship.
|
double
|
s
|
gauge
|
gcp.redis.stats.evicted_keys.count
|
Delta of the number of evicted keys due to maxmemory limit.
|
long
|
gauge
|
|
gcp.redis.stats.expired_keys.count
|
Delta of the total number of key expiration events.
|
long
|
gauge
|
|
gcp.redis.stats.keyspace_hits.count
|
Delta of the number of successful lookup of keys in the main dictionary.
|
long
|
gauge
|
|
gcp.redis.stats.keyspace_misses.count
|
Delta of the number of failed lookup of keys in the main dictionary.
|
long
|
gauge
|
|
gcp.redis.stats.memory.maxmemory.mb
|
Maximum amount of memory Redis can consume.
|
long
|
m
|
gauge
|
gcp.redis.stats.memory.system_memory_overload_duration.us
|
The amount of time in microseconds the instance is in system memory overload mode.
|
long
|
micros
|
gauge
|
gcp.redis.stats.memory.system_memory_usage_ratio
|
Memory usage as a ratio of maximum system memory.
|
double
|
gauge
|
|
gcp.redis.stats.memory.usage.bytes
|
Total number of bytes allocated by Redis.
|
long
|
byte
|
gauge
|
gcp.redis.stats.memory.usage_ratio
|
Memory usage as a ratio of maximum memory.
|
double
|
gauge
|
|
gcp.redis.stats.network_traffic.bytes
|
Delta of the total number of bytes sent to/from redis (includes bytes from commands themselves, payload data, and delimiters).
|
long
|
byte
|
gauge
|
gcp.redis.stats.pubsub.channels.count
|
Global number of pub/sub channels with client subscriptions.
|
long
|
gauge
|
|
gcp.redis.stats.pubsub.patterns.count
|
Global number of pub/sub pattern with client subscriptions.
|
long
|
gauge
|
|
gcp.redis.stats.reject_connections.count
|
Number of connections rejected because of maxclients limit.
|
long
|
gauge
|
|
host.architecture
|
Operating system architecture.
|
keyword
|
||
host.containerized
|
If the host is a container.
|
boolean
|
||
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
||
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
||
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
||
host.ip
|
Host ip addresses.
|
ip
|
||
host.mac
|
Host mac addresses.
|
keyword
|
||
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
||
host.os.build
|
OS build information.
|
keyword
|
||
host.os.codename
|
OS codename, if any.
|
keyword
|
||
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
||
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
||
host.os.name
|
Operating system name, without the version.
|
keyword
|
||
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
||
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
||
host.os.version
|
Operating system version as a raw string.
|
keyword
|
||
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
||
service.type
|
The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch,
service.type would be elasticsearch . |
keyword
|
An example event for redis
looks as following:
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"cloud": {
"account": {
"id": "elastic-obs-integrations-dev",
"name": "elastic-obs-integrations-dev"
},
"instance": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"machine": {
"type": "e2-medium"
},
"provider": "gcp",
"availability_zone": "us-central1-c",
"region": "us-central1"
},
"event": {
"dataset": "gcp.redis",
"duration": 115000,
"module": "gcp"
},
"gcp": {
"redis": {
"clients": {
"blocked": {
"count": 4
}
}
},
"labels": {
"user": {
"goog-gke-node": ""
}
}
},
"host": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"metricset": {
"name": "metrics",
"period": 10000
},
"service": {
"type": "gcp"
}
}
Storage
The storage
dataset fetches metrics from Storage in Google Cloud Platform.
Exported fields
Field | Description | Type | Metric Type |
---|---|---|---|
@timestamp
|
Event timestamp.
|
date
|
|
agent.id
|
Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.
|
keyword
|
|
cloud
|
Fields related to the cloud or infrastructure the events are coming from.
|
group
|
|
cloud.account.id
|
The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.
|
keyword
|
|
cloud.account.name
|
The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name.
|
keyword
|
|
cloud.availability_zone
|
Availability zone in which this host, resource, or service is located.
|
keyword
|
|
cloud.image.id
|
Image ID for the cloud instance.
|
keyword
|
|
cloud.instance.id
|
Instance ID of the host machine.
|
keyword
|
|
cloud.instance.name
|
Instance name of the host machine.
|
keyword
|
|
cloud.machine.type
|
Machine type of the host machine.
|
keyword
|
|
cloud.project.id
|
Name of the project in Google Cloud.
|
keyword
|
|
cloud.provider
|
Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
|
keyword
|
|
cloud.region
|
Region in which this host, resource, or service is located.
|
keyword
|
|
container.id
|
Unique container id.
|
keyword
|
|
container.image.name
|
Name of the image the container was built on.
|
keyword
|
|
container.labels
|
Image labels.
|
object
|
|
container.name
|
Container name.
|
keyword
|
|
data_stream.dataset
|
Data stream dataset.
|
constant_keyword
|
|
data_stream.namespace
|
Data stream namespace.
|
constant_keyword
|
|
data_stream.type
|
Data stream type.
|
constant_keyword
|
|
ecs.version
|
ECS version this event conforms to.
ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. |
keyword
|
|
error
|
These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error.
|
group
|
|
error.message
|
Error message.
|
match_only_text
|
|
event.dataset
|
Event dataset
|
constant_keyword
|
|
event.module
|
Event module
|
constant_keyword
|
|
gcp.labels.metadata.*
|
object
|
||
gcp.labels.metrics.*
|
object
|
||
gcp.labels.resource.*
|
object
|
||
gcp.labels.system.*
|
object
|
||
gcp.labels.user.*
|
object
|
||
gcp.labels_fingerprint
|
Hashed value of the labels field.
|
keyword
|
|
gcp.metrics.*.*.*.*
|
Metrics that returned from Google Cloud API query.
|
object
|
|
gcp.storage.api.request.count
|
Delta count of API calls, grouped by the API method name and response code.
|
long
|
gauge
|
gcp.storage.authz.acl_based_object_access.count
|
Delta count of requests that result in an object being granted access solely due to object ACLs.
|
long
|
gauge
|
gcp.storage.authz.acl_operations.count
|
Usage of ACL operations broken down by type.
|
long
|
gauge
|
gcp.storage.authz.object_specific_acl_mutation.count
|
Delta count of changes made to object specific ACLs.
|
long
|
gauge
|
gcp.storage.network.received.bytes
|
Delta count of bytes received over the network, grouped by the API method name and response code.
|
long
|
gauge
|
gcp.storage.network.sent.bytes
|
Delta count of bytes sent over the network, grouped by the API method name and response code.
|
long
|
gauge
|
gcp.storage.storage.object.count
|
Total number of objects per bucket, grouped by storage class. This value is measured once per day, and the value is repeated at each sampling interval throughout the day.
|
long
|
gauge
|
gcp.storage.storage.total.bytes
|
Total size of all objects in the bucket, grouped by storage class. This value is measured once per day, and the value is repeated at each sampling interval throughout the day.
|
long
|
gauge
|
gcp.storage.storage.total_byte_seconds.bytes
|
Delta count of bytes received over the network, grouped by the API method name and response code.
|
long
|
gauge
|
host.architecture
|
Operating system architecture.
|
keyword
|
|
host.containerized
|
If the host is a container.
|
boolean
|
|
host.domain
|
Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider.
|
keyword
|
|
host.hostname
|
Hostname of the host. It normally contains what the
hostname command returns on the host machine. |
keyword
|
|
host.id
|
Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of
beat.name . |
keyword
|
|
host.ip
|
Host ip addresses.
|
ip
|
|
host.mac
|
Host mac addresses.
|
keyword
|
|
host.name
|
Name of the host. It can contain what
hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword
|
|
host.os.build
|
OS build information.
|
keyword
|
|
host.os.codename
|
OS codename, if any.
|
keyword
|
|
host.os.family
|
OS family (such as redhat, debian, freebsd, windows).
|
keyword
|
|
host.os.kernel
|
Operating system kernel version as a raw string.
|
keyword
|
|
host.os.name
|
Operating system name, without the version.
|
keyword
|
|
host.os.name.text
|
Multi-field of
host.os.name . |
text
|
|
host.os.platform
|
Operating system platform (such centos, ubuntu, windows).
|
keyword
|
|
host.os.version
|
Operating system version as a raw string.
|
keyword
|
|
host.type
|
Type of host. For Cloud providers this can be the machine type like
t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. |
keyword
|
|
service.type
|
The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch,
service.type would be elasticsearch . |
keyword
|
An example event for storage
looks as following:
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"cloud": {
"account": {
"id": "elastic-obs-integrations-dev",
"name": "elastic-obs-integrations-dev"
},
"instance": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"machine": {
"type": "e2-medium"
},
"provider": "gcp",
"availability_zone": "us-central1-c",
"region": "us-central1"
},
"event": {
"dataset": "gcp.storage",
"duration": 115000,
"module": "gcp"
},
"gcp": {
"storage": {
"storage": {
"total": {
"bytes": 4472520191
}
},
"network": {
"received": {
"bytes": 4472520191
}
}
},
"labels": {
"user": {
"goog-gke-node": ""
}
}
},
"host": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"metricset": {
"name": "storage",
"period": 10000
},
"service": {
"type": "gcp"
}
}
VMware vSphere Integration
This integration periodically fetches logs and metrics from vSphere vCenter servers.
Compatibility
The integration uses the Govmomi library to collect metrics and logs from any Vmware SDK URL (ESXi/VCenter). This library is built for and tested against ESXi and vCenter 6.5, 6.7 and 7.0.
Installation Guide:
VMware vSphere 7.0 Installation
Govmomi Library
Integration Process
Go> Cyber Incident Management (XDR and MDR)
Go> Cyber Incident Management (XDR and MDR)> Settings
Go> Cyber Incident Management (XDR and MDR)> Settings> Integration
Go> Cyber Incident Management (XDR and MDR)> Settings> Integration>
In search bar type “Vmware”
Click Add Agent
Choose your Log Collector
Click the vSphere logs and metrics
Keep it as is
Enter the IP address and port
Example: https://127.0.0.1:8989/sdk
127.0.0.1: This is the IP address of the local machine (localhost).
8989: This is the port number on which the SDK service is running. (Keep it as is)
/sdk: This indicates that the SDK is accessible at this path. (Keep it as is)
Notes: To add multiple hosts, enter each IP address following the same format (https://<IP_or_hostname>:port/sdk) and press enter.
Enter the Username and password of vSphere account
Notes: The insecure option bypasses the verification of the server's certificate chain, which can be useful in certain scenarios but comes with significant security risks. It is recommended to use this option only when necessary and in environments where security concerns are minimal.
Logs collection
Collect logs from vSphere via UDP
UDP host to listen on: This is the IP address of the machine where the log collector is running.
UDP port to listen on: This is the port on which the log collector will listen for incoming log data. (Keep it as is)
Notes: Enabling "Preserve original event" ensures raw log data is always available, crucial for troubleshooting, compliance, and verifying log accuracy. It adds raw data to event.original, doubling storage needs and potentially slowing processing if storage isn't scaled, impacting efficiency.
Collect logs from vSphere via TCP
TCP host to listen on: This is the IP address of the machine where the log collector is running.
TCP port to listen on: This is the port on which the log collector will listen for incoming log data. (Keep it as is)
Notes: Enabling "Preserve original event" ensures raw log data is always available, crucial for troubleshooting, compliance, and verifying log accuracy. It adds raw data to event.original, doubling storage needs and potentially slowing processing if storage isn't scaled, impacting efficiency.
Click Next to complete the integration.
SentinelOne Integrations
The SentinelOne integration collects and parses data from SentinelOne REST APIs. This integration also offers the capability to perform response actions on SentinelOne hosts directly through the Elastic Security interface
Compatibility
This module has been tested against SentinelOne Management Console API version 2.1
.
API token
To collect data from SentinelOne APIs, you must have an API token. To create an API token, follow these steps:
- Log in to the SentinelOne Management Console as an Admin.
3. Click My User.
4. In the API token section, click Generate.
Note
The API token generated by the user is time-limited. To rotate a new token, log in with the dedicated admin account.
Integrate on Elastic
1. Add SentinelOne console URL (https://<DomainName>.sentinelone.net/
, where <DomainName>
is the domain name of your SentinelOne account.)
2. Add API token
If you need further assistance, kindly contact our support at info@cytechint.com for prompt assistance and guidance.
Custom Windows Event Logs - Integration
Custom Windows Event Logs
Collect and parse logs from any Windows event log channel with Elastic Agent.
The custom Windows event log package allows you to ingest events from any Windows event log channel. You can get a list of available event log channels by running Get-WinEvent -ListLog * | Format-List -Property LogName
in PowerShell on Windows Vista or newer. If Get-WinEvent
is not available, Get-EventLog *
may be used.
By executing this command in the powershell(administrator), it will list the log names that is being used.
Add a channel name in the Channel Name text field (e.g Application).
Windows Event Forwarding to Linux server using Nxlog
Introduction
Windows Event Forwarding (WEF) allows the collection of event logs from multiple Windows machines and their forwarding to a centralized server. Using Nxlog, you can send these logs to a Linux server for storage and analysis. This documentation provides a step-by-step guide to set up Windows Event Forwarding using Nxlog to send logs to a Linux server.
Prerequisites
- Windows Server or Workstation: The machine that will send logs.
- Linux Server: The machine that will receive logs.
- Nxlog: Download the latest version of Nxlog for Windows from Nxlog's official website.
- Network Connectivity: Ensure both machines can communicate over the network.
- Rsyslog: Download the latest version of Rsyslog for Linux server or workstation.
Installing Nxlog on Windows
Configuring Nxlog on Windows
-
Open Configuration File:
- Edit the Nxlog configuration file located at
C:\Program Files\nxlog\conf\nxlog.conf
.
- Edit the Nxlog configuration file located at
-
Configure File:
- Add the following lines to capture Windows Event Logs and send the logs :
# Input Module<Input eventlog>Module im_msvistalogReadFromLast True<QueryXML><QueryList><Query Id='1'><Select Path='Application'>*</Select><Select Path='Security'>*</Select><Select Path='System'>*</Select></Query></QueryList></QueryXML></Input># Output Module<Output out>Module om_udpHost 192.168.20.24Port 514# Exec $raw_event = "<" + $syslog_severity + ">" + $time + " " + $hostname + " " + $procname + ": " + $raw_event;Exec parse_syslog_ietf();</Output># Route<Route r>Path eventlog => out</Route># Include any other necessary modules/extensions<Extension _syslog>Module xm_syslog</Extension>
- Add the following lines to capture Windows Event Logs and send the logs :
Installing Rsyslog on Linux
-
Install Rsyslog:
- For Ubuntu, run:
-
Enable Rsyslog:
- Ensure Rsyslog is enabled and started:
Configuring Rsyslog on Linux
-
Open Configuration File:
- Edit /etc/rsyslog.conf or create a new config file in /etc/rsyslog.d/.
-
Configure Rsyslog to Listen for UDP:module(load="imudp") # Load UDP listener input(type="imudp" port="514")
-
Define Output File:
- Specify where to store the incoming logs:
-
Save and Exit:
- Save the configuration file and restart Rsyslog:
Firewall Configuration
Windows Firewall
-
Open Windows Defender Firewall:
- Go to Control Panel > System and Security > Windows Defender Firewall.
-
Allow Port 514:
- In the left pane, click Advanced settings.
- Select Inbound Rules and click on New Rule.
- Choose Port, then click Next.
- Select UDP and enter 514 in the Specific local ports field.
- Allow the connection and complete the rule setup.
Firewalld Configuration on Linux
-
Open Port 514 for UDP:
-
Reload Firewalld:
-
Verify Open Ports:
Verifying Event Forwarding
-
Check Nxlog Status on Windows:
-
Monitor Logs on Linux:
- Use the following command to view the log file:
-
Review Rsyslog Logs:
- If issues arise, check Rsyslog logs located at /var/log/syslog or /var/log/messages.
Windows Event Forwarding to Linux server using Powershell script
Overview
This PowerShell script forwards Windows event logs to a Linux server using the syslog protocol. It captures specific event logs, sends them to the specified syslog server, and ensures that duplicate events are not sent.
Prerequisites
- PowerShell on Windows
- Syslog server running on Linux (e.g., Rocky Linux) with an accessible IP
- UDP port 514 open for communication
Powershell Script
Save the file as .ps1 (e.g sendlogs.ps1).
Script:
# Define the syslog server IP address and port
$syslogServerIP = "192.168.20.24" # Replace with your Rocky server's IP
$syslogPort = 514
# File to store last sent event info.
# Change this directory if necessary
$logFilePath = "C:\Users\Administrator\Desktop\lastEventInfo.txt"
# Initialize last sent events
$lastSentEvents = @{}
# Check if the file exists and read the last sent events
if (Test-Path $logFilePath) {
Write-Host "Loading last sent events from file."
$lastSentEvents = Get-Content $logFilePath | ConvertFrom-Json
}
# Loop for sending logs
while ($true) {
Write-Host "Checking for new logs..."
# Define the logs you want to forward
$logNames = @("Application", "Security", "Setup", "System")
# Loop through each log
foreach ($logName in $logNames) {
Write-Host "Processing log: $logName"
# Get new logs
$logs = Get-WinEvent -LogName $logName | Sort-Object TimeCreated
foreach ($log in $logs) {
# Create a unique key based on Event ID and TimeCreated
$eventKey = "$($log.Id)-$($log.TimeCreated.Ticks)"
# Check if the event has already been sent
if (-not $lastSentEvents.ContainsKey($eventKey)) {
# Create syslog message format
$message = "<134>" + $log.TimeCreated.ToString("yyyy-MM-dd HH:mm:ss") + " " + $log.ProviderName + ": " + $log.Message
# Send the message to the syslog server
$client = New-Object System.Net.Sockets.UdpClient
$client.Connect($syslogServerIP, $syslogPort)
$bytes = [System.Text.Encoding]::ASCII.GetBytes($message)
$client.Send($bytes, $bytes.Length)
$client.Close()
# Mark the event as sent
$lastSentEvents[$eventKey] = $true
Write-Host "Sent log: $message"
} else {
Write-Host "Log already sent: $eventKey"
}
}
}
# Save the last sent events to the file
$lastSentEvents | ConvertTo-Json | Set-Content -Path $logFilePath
Write-Host "Last sent events updated."
# Wait for 1 second before running again
Start-Sleep -Seconds 1
}
Script Components
-
Define Variables:
$syslogServerIP
: IP address of the Linux syslog server.$syslogPort
: Port number for syslog (default is 514).$logFilePath
: Path to store the last sent event information.
-
Initialize Last Sent Events:
- Loads previously sent events from a file, if it exists.
-
Main Loop:
- Continuously checks for new logs from specified log categories:
Application
,Security
,Setup
, andSystem
.
- Continuously checks for new logs from specified log categories:
-
Processing Logs:
- Retrieves new logs, sorts them, and creates a unique key based on the event ID and timestamp.
- Sends new logs to the syslog server in a specified message format.
- Updates the log file with the newly sent events.
-
Error Handling:
- Logs messages indicating whether an event has been sent or is a duplicate.
Usage
- Update the
$syslogServerIP
and$logFilePath
variables. - Run the script in PowerShell. It will run indefinitely, checking for new logs every second.
Task Scheduler
Make sure to enable the "Run with highest privileges"
Add a new action
1. Fill in the Program/Script Text Box with: powershell.exe
2. Fill in the Add arguments Text Box with: -ExecutionPolicy Bypass -File "C:\Users\Administrator\Desktop\sendlogs.ps1"
Settings Configuration
After the creating the task, you can enable the script by activating the task.
Sophos Integration
Overview
The Sophos Central integration allows you to monitor Alerts and Events logs. Sophos Central is a cloud-native application with high availability. It is a cybersecurity management platform hosted on public cloud platforms. Each Sophos Central account is hosted in a named region. Sophos Central uses well-known, widely used, and industry-standard software libraries to mitigate common vulnerabilities.
Use the Sophos Central integration to collect logs across Sophos Central managed by your Sophos account. Visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue.
Compatibility
The Sophos Central Application does not feature version numbers. This integration has been configured and tested against Sophos Central SIEM Integration API version v1.
Requirements
You need Elasticsearch for storing and searching your data, and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.
Setup
Elastic Integration for Sophos Central Settings
The Elastic Integration for Sophos Central requires the following Authentication Settings in order to connect to the Target service:
- Client ID
- Client Secret
- Grant Type
- Scope
- Tenant ID
- Token URL
NOTE: Sophos central supports logs only upto last 24 hrs.
Step 1 - Create a service principal
We will show you how you can sign in to Sophos Central Admin and create a service principal. You need to have the Super Admin role to do this.
Step 1a - Sophos Central Admin
Sign in to Sophos Central Admin. Go to https://central.sophos.com/manage.
Click 'Global Settings' and then click the "API Credentials" link.
Step 1b - Add a new set of credentials
Supply a name for your credential set and a description, then click 'Add' as shown in the example below.
Step 1c - Grab your client ID and secret
Click 'Copy' to note down the client ID. Also show the client secret.
Click 'Copy' to note down the client secret.
⚠️ WARNING: It is your responsibility to store your client ID and secret securely. If these are lost or stolen, an attacker will be able to call APIs on your behalf and steal your data or cause damage.
If you need further assistance, kindly contact our support at info@cytechint.com for prompt assistance and guidance.
Atlassian Bitbucket Integrations (New)
Introduction
The Bitbucket integration collects audit logs from the audit log files or the audit API.
Reference: https://developer.atlassian.com/server/bitbucket/reference/rest-api/
Assumptions
The procedures described in Section 3 assume that a Log Collector has already been set up.
Requirements
For more information on auditing in Bitbucket and how it can be configured, see View and configure the audit log on Atlassian's website.
Reference: https://confluence.atlassian.com/bitbucketserver/view-and-configure-the-audit-log-776640417.html
Logs
Audit
The Confluence integration collects audit logs from the audit log files or the audit API from self-hosted Confluence Data Center. It has been tested with Confluence 7.14.2 but is expected to work with newer versions. As of version 1.2.0, this integration added experimental support for Atlassian Confluence Cloud. JIRA Cloud only supports Basic Auth using username and a Personal Access Token.
Atlassian Bitbucket Integration Procedures
Please provide the following information to CyTech:
Collect Bitbucket audit logs via log files
-
Path
-
Preserve Original Event? (Enable Yes/No)
-
Preserves a raw copy of the original event, added to the field event.original
-
Tags
-
Processors (Optional)
-
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed.
-
Indexing settings (experimental) (Enable Yes/No)
-
Select data streams to configure indexing options. This is an experimental feature and may have effects on other properties.
Collect Bitbucket audit logs via API (Enable Yes/No)
-
API URL - The API URL without the path.
-
Bitbucket Username - JIRA Username. Needs to be used with a Password. Do not fill if you are using a personal access token.
-
Bitbucket Password - JIRA Password. Needs to be used with a Username. Do not fill if you are using a personal access token.
-
Personal Access Token - The Personal Access Token. If set, Username and Password will be ignored.
-
Initial Interval - Initial interval for the first API call. Defaults to 24 hours.
Create Access Token:
Access Tokens are single-purpose access tokens (or passwords) with access to a single workspace with limited permissions (specified at creation time). Use tokens for tasks such as scripting, CI/CD tools, and testing Bitbucket integrations or Marketplace apps while in development.
To create an Access Token:
-
At bitbucket.org, navigate to the target workspace for the Access Token. This workspace is the only one that the Workspace Access Token can access.
-
On the sidebar, select Settings.
-
On the sidebar, under Security, select Access tokens.
-
Select Create Workspace Access Token.
-
Give the Workspace Access Token a name, usually related to the app or task that will use the token.
-
Select the permissions the Access Token needs. Note give the Access Token admin permission.
-
Select the Create button. The page will display the Workspace Access Token created dialog.
-
Copy the generated token and either record or paste it into the app that requires access. The token is only displayed once and can't be retrieved later.
If you need further assistance, kindly contact our support at info@cytechint.com for prompt assistance and guidance.
Palo Alto Cortex XDR Integration
Palo Alto Cortex XDR Integration
Using the Cortex XDR APIs, you can integrate Cortex XDR with third-party apps or services to ingest alerts and to leverage alert stitching and investigation capabilities. The APIs allows you to manage incidents in a ticketing or automation system of your choice by reviewing and editing the incident's details, status, and assignee. Using the APIs, you can also retrieve information on the endpoints, create installation package, perform response actions directly on the endpoint and more.
Alerts
The Cortex XDR Alerts API is used to retrieve alerts generated by Cortex XDR based on raw endpoint data. A single alert might include one or more local endpoint events, each event generating its own document on Elasticsearch.
The Palo Alto XDR integration requires both an API key and API key ID, both which can be retrieved from the Cortex XDR UI.
API
Before you can begin using Cortex XDR APIs, you must generate the following items from the Cortex XDR app:
Value | Description |
---|---|
API Key | The API Key is your unique identifier used as the Authorization:{key} header required for authenticating API calls. Depending on your desired security level, you can generate two types of API keys, Advanced or Standard, from your Cortex XDR app. |
API Key ID | The API Key ID is your unique token used to authenticate the API Key. The header used when running an API call is x-xdr-auth-id:{key_id} . |
FQDN | The FQDN is a unique host and domain name associated with each tenant. When you generate the API Key and Key ID, you are assigned an individual FQDN. |
Create Cortex API Key:
The following steps describe how to generate the necessary key values:
-
Get your Cortex XDR API Key:
- In Cortex XDR, navigate to Settings > Configurations > Integrations > API Keys.
- Select + New Key.
- Choose the type of API Key you want to generate based on your desired security level: Advanced or Standard. The Advanced API key hashes the key useing a nonce, a random string, and a timestamp to prevent replay attacks. cURL does not support this but is suitable with scripts. Use the provided script to create the advanced API authentication token.
Note: To integrate with Cortex XSOAR you must generate an Advanced Key.
- If you want to define a time limit on the API key authentication, mark Enable Expiration Date and select the expiration date and time. Navigate to Settings > Configurations > Integrations > API Keys to track the Expiration Time field for each API key. In addition, Cortex XDR displays a API Key Expiration notification in the Notification Center one week and one day prior to the defined expiration date.
- Provide a comment that describes the purpose for the API key, if desired.
- Select the desired level of access for this key. You can select from the list of existing Roles, or you can select Custom to set the permissions on a more granular level. Roles are available according what was defined in the hub as described in the Manage Roles section of the Cortex XDR Administrator’s Guide.
- Generate the API Key.
- Copy the API key, and then click Done. This value represents your unique
Authorization:{key}
.
Note: You will not be able to view the API Key again after you complete this step. Ensure that you copy it before closing the notification.
Cortex XDR API Key ID
Get your Cortex XDR API Key ID.
-
- In the API Keys table, locate the ID field.
- Note your corresponding ID number. This value represents the
x-xdr-auth-id:{key_id}
token.
Note: This key id will be used for integrations with elastic.
Cortex XDR API FQDN
Get your FQDN.
- Right-click your API key and select View Examples.
- Copy the CURL Example URL. The example contains your unique FQDN:
https://api-{fqdn}/public_api/v1/{name of api}/{name of call}/
You can use the CURL Example URL to run the APIs.
If you need further assistance, kindly contact our support at info@cytechint.com for prompt assistance and guidance.
Active Directory Integrations
Introduction
Elastic Stack security features can be configured to authenticate users through Active Directory by using LDAP to communicate with the directory. Active Directory realms are similar to LDAP realms, as they both store users and groups in a hierarchical structure, which includes containers such as organizational units (OU), organizations (O), and domain components (DC).
The security features support authentication based on Active Directory security groups, but not distribution groups. When authenticating users, the username entered must match the sAMAccountName or userPrincipalName, not the common name (cn). The realm authenticates users via an LDAP bind request, searches for their entry in Active Directory, and retrieves their group memberships from the tokenGroups attribute to assign appropriate roles.
Requirements
Elastic Agent must be installed. For more details and installation instructions, please refer to the Elastic Agent Installation Guide.
Installing and managing an Elastic Agent:
There are several options for installing and managing Elastic Agent:
Install a Fleet-managed Elastic Agent (recommended):
With this approach, you install Elastic Agent and use Fleet in Kibana to define, configure, and manage your agents in a central location. We recommend using Fleet management because it makes the management and upgrade of your agents considerably easier.
Install Elastic Agent in standalone mode (advanced users):
With this approach, you install Elastic Agent and manually configure the agent locally on the system where it’s installed. You are responsible for managing and upgrading the agents. This approach is reserved for advanced users only.
Install Elastic Agent in a containerized environment:
You can run Elastic Agent inside a container, either with Fleet Server or standalone. Docker images for all versions of Elastic Agent are available from the Elastic Docker registry, and we provide deployment manifests for running on Kubernetes.
Please note, there are minimum requirements for running Elastic Agent. For more information, refer to the Elastic Agent Minimum Requirements.
How to add configurations to Elastic Integration
I. Active Directory Base DN
-
Definition: The Base DN (Distinguished Name) specifies the starting point in the Active Directory hierarchy for user and group searches.
-
Format: It typically represents the container or organizational unit (OU) where your user accounts are located.
-
Example: If your AD users are in the "Users" OU under the domain "example.com", the Base DN might look like:
-
CN=Users,DC=example,DC=com
-
Note: Refer to Step I. Active Directory Information Lookup for information on how to properly setup the configuration.
II. Active Directory URL
-
Definition: The URL of your Active Directory server, specifying either an unsecured LDAP or secure LDAPS connection.
-
Format:
-
LDAP (insecure):
ldap://your-ad-server.example.com:389
-
LDAPS (secure):
ldaps://your-ad-server.example.com:636
-
-
Example:
-
ldap://ad.example.com:389
-
Note: Refer to Step II. Finding Active Directory URL for information on how to properly setup the configuration.
III. Active Directory User
-
Definition: The username of the service account that Elastic Stack will use to authenticate and query AD. This account should have sufficient privileges to search for users and groups.
-
Format:
-
It can be in the form of a fully qualified domain username:
username@example.com
-
Or a Distinguished Name (DN):
CN=ServiceAccount,OU=ServiceAccounts,DC=example,DC=com
-
-
Example:
-
CN=serviceaccount,OU=ServiceAccounts,DC=example,DC=com
-
Note: Refer to Step III. Navigate to Users for information on how to properly setup the configuration.
IV. Active Directory User Password
-
Definition: The password for the AD user account used for the connection.
-
Example:
-
MySecurePassword123
-
I. Active Directory Information Lookup
Finding the Base DN (Distinguished Name)
Method 1: Using Active Directory GUI
- Open "Active Directory Users and Computers"
2. Right-click on your domain
3. Select "Properties"
4. Look for the "Distinguished Name" field
Method 2: Using PowerShell
- Open PowerShell with administrator privileges
- Run the command:
- The output will be in the format: "DC=company,DC=local"
II. Finding Active Directory URL
Method 1: PowerShell
- Open PowerShell as administrator
- Run the command:
- Take the DC name from the output
III. Navigate to Users
If you need further assistance, kindly contact our support at info@cytechint.com for prompt assistance and guidance.
Microsoft SQL Server Integration
The Microsoft SQL Server integration package allows you to search, observe, and visualize the SQL Server audit logs, as well as performance and transaction log metrics.
Requirements
Microsoft SQL Server is installed and has connectivity with the CyTech Log Collector.
Note. For more information regarding Microsoft SQL Server Installation, click the link (https://learn.microsoft.com/en-us/sql/database-engine/install-windows/install-sql-server?view=sql-server-ver16) for more info.
Microsoft SQL Server permissions
Before you can start sending data to the Log Collector, make sure you have the necessary Microsoft SQL Server permissions.
If you browse Microsoft Developer Network (MSDN) for the following tables, you will find a "Permissions" section that defines the permission needed for each table.
transaction_log
:- sys.databases
- sys.dm_db_log_space_usage
- sys.dm_db_log_stats (DB_ID) (Available on SQL Server (MSSQL) 2016 (13.x) SP 2 and later)
performance
:- sys.dm_os_performance_counters
Please make sure the user has the permissions to system as well as user-defined databases. For the particular user used in the integration, the following requirements are met:
User setup options:
- Grant specific permissions as mentioned in the MSDN pages above.
- Alteratively, use
sysadmin
role (includes all required permissions): This can be configured via SQL Server Management Studio (SSMS) inServer Roles
. Read more about joining a role in the SQL Server documentation.
User Mappings (using SQL Server Management Studio (SSMS)):
- Open SSMS and connect to your server.
- Navigate to "Object Explorer" > "Security" > "Logins".
- Right-click the user and select "Properties".
- In the "User Mapping" tab, select the appropriate database and grant the required permissions.
Setup
Below you'll find more specific details on setting up the Microsoft SQL Server integration.
Named Instance
Microsoft SQL Server has a feature that allows running multiple databases on the same host (or clustered hosts) with separate settings. Establish a named instance connection by using the instance name along with the hostname (e.g. host/instance_name
or host:named_instance_port
) to collect metrics. Details of the host configuration are provided below.
Host Configuration
As part of the input configuration, you need to provide the user name, password and host details. The host configuration supports both named instances or default (no-name) instances, using the syntax below.
Note: This integration supports collecting metrics from a single host. For multi-host metrics, each host can be run as a new integration.
Connecting to Default Instance (host):
host
(e.g.localhost
(Instance name is not needed when connecting to default instance))
Note. IP Address of the SQL Server will be needed for the integration
host:port
(e.g.localhost:1433
)
Note. Default port is 1433
Connecting to Named Instance (host):
host/instance_name
(e.g.localhost/namedinstance_01
)host:named_instance_port
(e.g.localhost:60873
)
If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.
Azure Logs Integration
Introduction
This document shows information related to Azure Active Directory Integration.
The Azure Logs integration retrieves different types of log data from Azure.
Assumptions
The procedures described in the Requirements section assumes that a Log Collector has already
been setup.
Requirements
Main Setup
- One or more event hub to store in-flight logs exported by Azure services and make them available to the Log Collector
- Example:
-
┌────────────────┐ ┌────────────┐ │ adlogs │ │ Log │ │ <<Event Hub>> │─────▶ │ Collector │ └────────────────┘ └────────────┘
- One or more diagnostic setting to export logs from Azure services to Event Hubs
- Example:
-
┌──────────────────┐ ┌──────────────┐ ┌─────────────────┐ │Microsoft Entra ID│ │ Diagnostic │ │ Event Hub │ │ <<source>> │─────▶│ settings │────▶│ <<destination>> │ └──────────────────┘ └──────────────┘ └─────────────────┘
- One Storage Account Container to store information about logs consumed by the Log Collector
-
- Example:
┌────────────────┐ ┌────────────┐ │ adlogs │ logs │ Log │ │ <<Event Hub>> │────────────────────▶│ Collector │ └────────────────┘ └────────────┘ │ consumer group info │ ┌────────────────┐ (state, position, or │ │ azurelogs │ offset) │ │ <<container>> │◀───────────────────────────┘ └────────────────┘
- Example:
-
This is the final diagram of the a setup for collecting Activity logs from the Azure Monitor service.
┌───────────────┐ ┌──────────────┐ ┌────────────────┐ ┌────────────┐
│ MS Entra ID │ │ Diagnostic │ │ adlogs │ logs │ Log │
│ <<service>> ├──▶│ Settings │──▶│ <<Event Hub>> │────────▶│ Collector │
└───────────────┘ └──────────────┘ └────────────────┘ └────────────┘
│
┌──────────────┐ consumer group info │
│ azurelogs │ (state, position, or │
│<<container>> │◀───────────────offset)──────────────┘
└──────────────┘
If the integration is running behind a firewall, please proceed here.
Here are several requirements before using the integration since the logs will
be read from azure event hubs.
- The logs have to be exported first to the event hub.
• Create an event hub using Azure portal.
• More information can be found on: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubscreate. - To export activity logs to event hubs users can follow the steps here.
• Legacy collection methods
• More information can be found on: https://learn.microsoft.com/en-us/azure/azuremonitor/essentials/activity-log?tabs=powershell#legacy-collectionmethods - To export audit and sign-in logs to event hubs users can follow the
steps here.
• Stream Azure Active Directory logs
• More information can be found on: https://learn.microsoft.com/en-us/azure/active-directory/reportsmonitoring/tutorial-azure-monitor-stream-logs-to-event-hub
Azure Active Directory Integration Procedures
Create a Resource Group
A resource group is a logical collection of Azure resources. All resources are
deployed and managed in a resource group. To create a resource group:
- Sign in to the Azure portal.
- In the left navigation, select Resource groups, and then
select Create a resource. - For Subscription, select the name of the Azure subscription in which
you want to create the resource group. For CyTech (Azure Active Directory) - Type a unique name for the resource group. The system
immediately checks to see if the name is available in the currently
selected Azure subscription. - Select a region for the resource group.
- Select Review + Create.
- Takes a few minutes to complete.
Create an Event Hubs Namespace
An Event Hubs namespace provides a unique scoping container, in which you create
one or more event hubs. To create a namespace in your resource group using the
portal, do the following actions:
- In the Azure portal, and select Create a resource at the top left of
the screen. - Select All services in the left menu, and select star (*) next to Event
Hubs in the Analytics category. Confirm that Event Hubs is added
to FAVORITES in the left navigational menu. - Select Event Hubs under FAVORITES in the left navigational menu, and
select Create on the toolbar. - On the Create namespace page, take the following steps:
a. Select the subscription in which you want to create the
namespace.
b. Select the resource group you created in the previous step.
c. Enter a name for the namespace. The system immediately checks to see if the name is available.d. Select a location for the namespace.
e. Choose Basic for the pricing tier. To learn about differences
between tiers, see Quotas and limits, Event Hubs Premium, and Event
Hubs Dedicated articles.f. Leave the throughput units (for standard tier) or processing
units (for premium tier) settings as it is. To learn about throughput units
or processing units: Event Hubs scalability.g. Select Review + Create at the bottom of the page.
h. On the Review + Create page, review the settings, and select Create.
Wait for the deployment to complete. - On the Deployment page, select Go to resource to navigate to the page for
your namespace.
Create an Event Hub
- To create an event hub within the namespace, do the following actions:
- On the Overview page, select + Event hub on the command bar.
- Type a name for your event hub, then select Review + create.
The partition count setting allows you to parallelize consumption across
many consumers. For more information, see Partitions.
The message retention setting specifies how long the Event Hubs service
keeps data. For more information, see Event retention. - On the Review + create page, select Create.
- You can check the status of the event hub creation in alerts. After the event
hub is created, you see it in the list of event hubs.
Create a Diagnostic Setting
The diagnostic settings export the logs from Azure services to a destination and in order to use Azure Logs integration, it must be an event hub.
To create a diagnostic settings to export logs:
- Locate the diagnostic settings for the service (for example, Microsoft Entra ID).
- Select diagnostic settings in the Monitoring section of the service. Note that different services may place the diagnostic settings in different positions.
- Select Add diagnostic settings.
In the diagnostic settings page you have to select the source log categories you want to export and then select their destination.
Select log categories
Each Azure services exports a well-defined list of log categories. Check the individual integration doc to learn which log categories are supported by the integration.
Select the destination
Select the subscription and the Event Hubs namespace you previously created. Select the event hub dedicated to this integration.
Example:
┌───────────────┐ ┌──────────────┐ ┌───────────────┐ ┌────────────┐
│ MS Entra ID │ │ Diagnostic │ │ adlogs │ │ Log │
│ <<service>> ├──▶│ Settings │──▶│ <<Event Hub>> │─────▶ │ Collector │
└───────────────┘ └──────────────┘ └───────────────┘ └────────────┘
Create a Storage Account
To create an Azure storage account with the Azure portal, follow these steps:
- From the left portal menu, select Storage accounts to display a list
of your storage accounts. If the portal menu isn't visible, click the
menu button to toggle it on. - On the Storage accounts page, select Create.
- The following image shows a standard configuration of the basic properties
- The following image shows a standard configuration of the advanced
properties for a new storage account. - The following image shows a standard configuration of the networking
properties for a new storage account. - The following image shows a standard configuration of the data protection
properties for a new storage account. - The following image shows a standard configuration of the encryption
properties for a new storage account. - Review + Create Tab
When you navigate to the Review + create tab, Azure runs
validation on the storage account settings that you have chosen. If
validation passes, you can proceed to create the storage account.
If validation fails, then the portal indicates which settings need to be
modified.
The following image shows the Review tab data prior to the creation
of a new storage account.
Resources needed for the integration of Azure Active Directory:
- Azure Diagnostics Settings
Create a Diagnostics Configuration and select which log from
Azure will send to the event hub.
Navigate to Microsoft Entra ID > Monitoring > Diagnostic settings - Event Hub Credentials
- Go to > EventHub Resources > Select Shared Access Policies
- Please provide CyTech the:
a. Event Hubs Name Not the Name Space:
b. Connection string-primary key: - Account Storage Credentials
- Please provide CyTech the:
a. Storage Account Name:
b. Key 1 Key
Running the integration behind a firewall:
When you run the Elastic Agent behind a firewall, to ensure proper communication with the necessary components, you need to allow traffic on port 5671
and 5672
for the event hub, and port 443
for the Storage Account container.
┌────────────────────────────────┐ ┌───────────────────┐ ┌───────────────────┐
│ │ │ │ │ │
│ ┌────────────┐ ┌───────────┐ │ │ ┌──────────────┐ │ │ ┌───────────────┐ │
│ │ diagnostic │ │ event hub │ │ │ │azure-eventhub│ │ │ │ activity logs │ │
│ │ setting │──▶│ │◀┼AMQP─│ <<input>> │─┼──┼▶│<<data stream>>│ │
│ └────────────┘ └───────────┘ │ │ └──────────────┘ │ │ └───────────────┘ │
│ │ │ │ │ │ │
│ │ │ │ │ │ │
│ │ │ │ │ │ │
│ ┌─────────────┬─────HTTPS─┼──────────┘ │ │ │
│ ┌───────┼─────────────┼──────┐ │ │ │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ ▼ ▼ │ │ └─Log Collector─────┘ └─Elastic Cloud─────┘
│ │ ┌──────────┐ ┌──────────┐ │ │
│ │ │ 0 │ │ 1 │ │ │
│ │ │ <<blob>> │ │ <<blob>> │ │ │
│ │ └──────────┘ └──────────┘ │ │
│ │ │ │
│ │ │ │
│ └─Storage Account Container──┘ │
│ │
│ │
└─Azure──────────────────────────┘
Event Hub
Port 5671
and 5672
are commonly used for secure communication with the event hub. These ports are used to receive events. By allowing traffic on these ports, the Elastic Agent can establish a secure connection with the event hub.
Storage Account Container
Port 443
is used for secure communication with the Storage Account container. This port is commonly used for HTTPS traffic. By allowing traffic on port 443, the Elastic Agent can securely access and interact with the Storage Account container, which is essential for storing and retrieving checkpoint data for each event hub partition.
DNS
Optionally, you can restrict the traffic to the following domain names:
*.servicebus.windows.net
*.blob.core.windows.net
*.cloudapp.net
Additional Information:
Azure Active Directory Logs contain
Sign-in logs – Information about sign-ins and how your users use your
resources.
- Retrieves Azure Active Directory sign-in logs. The sign-ins report provides
information about the usage of managed applications and user sign-in
activities.
Identity Protection logs - Information about user risk status and the events
that change it.
- Retrieves Azure AD Identity Protection logs. The Azure AD Identity
Protection service analyzes events from AD users' behavior, detects risk
situations, and can respond by reporting only or even blocking users at
risk, according to policy configurations.
Provisioning logs - Information about users and group synchronization to
and from external enterprise applications.
- Retrieves Azure Active Directory Provisioning logs. The Azure AD
Provisioning service syncs AD users and groups to and from external
enterprise applications. For example, you can configure the provisioning
service to replicate all existing AD users and groups to an external
Dropbox Business account or vice-versa.
The Provisioning Logs contain a lot of details about a inbound/outbound
sync activity, like:
- User or group details.
- Source and target systems (e.g., from Azure AD to Dropbox).
- Provisioning status.
- Provisioning steps (with details for each step).
Audit logs – Information about changes to your tenant, such as users and
group management, or updates to your tenant's resources.
- Retrieves Azure Active Directory audit logs. The audit logs provide
traceability through logs for all changes done by various features within
Azure AD. Examples of audit logs include changes made to any resources
within Azure AD like adding or removing users, apps, groups, roles and
policies.
If you need further assistance, kindly contact our support at info@cytechint.com for prompt assistance and guidance.
ESET Protect Integration
ESET PROTECT allows you to efficiently manage ESET products across workstations and servers within a networked environment, supporting up to 50,000 devices from a single centralized platform. Through the ESET PROTECT Web Console, you can seamlessly deploy ESET solutions, manage tasks, enforce security policies, monitor system health, and swiftly address any issues or threats on remote devices.
Data streams
The ESET PROTECT integration collects three types of logs: Detection, Device Task and Event.
Detection is used to retrieve detections via the ESET Connect - Incident Management.
Device Task is used to retrieve device tasks via the ESET Connect - Automation.
Event is used to retrieve Detection, Firewall, HIPS, Audit, and ESET Inspect logs using the Syslog Server.
Requirements:
- Elastic Agent must be installed
Setup
To collect data from ESET Connect, follow the below steps:
- Create API User Account (Refer to How to Create an API User Account below)
- Retrieve the username and password generated during the creation of an API user account.
- Retrieve the region from the ESET Web Console URL.
To collect data from ESET PROTECT via Syslog, follow the below steps:
- Follow the steps to configure syslog server (Refer to How to Configure Syslog Server).
- Set the format of the payload to JSON.
- Set the format of the envelope to Syslog.
- Set the minimal log level to Information to collect all data.
- Select all checkboxes to collect logs for all event types.
- Enter the IP Address or FQDN of the Elastic Agent that is running the integration in the Destination IP field.
How to Create an API User Account:
For ESET Business Account and ESET MSP Administrator 2
Follow the steps below to create the dedicated API user account:
- Log in as Superuser (or Root) to your ESET Business Account or ESET MSP Administrator 2.
- Navigate to User management and create a new user.
- Under the Access Rights section, enable the toggle next to Integrations.
- Click Create to apply the changes.
- The new user receives an invitation email and must finish the account activation process.
For ESET PROTECT Hub
Follow the steps below to create the dedicated API user account:
- Log in as a Superuser to your ESET PROTECT Hub account.
- Navigate to Users and add a new user.
- Under the Permissions section, enable the toggle next to Integrations.
- Click Next and then click Create to apply the changes.
- The new user receives an invitation email and must finish the account activation process.
How to Configure Syslog Server
If you have a Syslog server running in your network, you can Export logs to Syslog to receive certain events (Detection Event, Firewall Aggregated Event, HIPS Aggregated Event, etc.) from client computers running ESET Endpoint Security.
To enable the Syslog server:
- Click More > Settings > Syslog and click the toggle next to Enable Syslog sending.
- Specify the following mandatory settings:
- Format of payload: JSON, LEEF or CEF
- Format of envelope of the log: BSD (specification), Syslog (specification)
- Minimal log level: Information, Warning, Error or Critical
- Event type of logs: Select the type of logs you want to include (Antivirus, HIPS, Firewall, Web protection, Audit Log, Blocked files, ESET Inspect alerts).
- Destination IP or FQDN of TLS-compatible syslog server: IPv4 address or hostname of the destination for Syslog messages
- Validate CA Root certificates of TLS connections: Click the toggle to enable the certificate validation for the connection between your Syslog server and ESET PROTECT. After enabling the validation, a new text field will be displayed where you can copy and paste the required certificate chain. The server certificate must meet the following requirements:
- The whole certificate chain in PEM format is uploaded and saved in the Syslog export configuration (this includes root CA, as there are no built-in trusted certificates)
- Your Syslog server's certificate provides a Subject Alternative Name extension (DNS=/IP=), in which at least one record corresponds to the FQDN/IP hostname configuration.
You need the certification authority version 3 (and later) with the Basic Constraints certificate extension to pass the validation.
The validation of TLS connections applies only to the certificates. Disabling the validation does not affect the TLS settings of ESET PROTECT.
After making the applicable changes, click Apply settings. The configuration becomes effective in 10 minutes.
The regular application log file is constantly being written to. Syslog only serves as a medium to export certain asynchronous events, such as notifications or various client computer events.
ESET Threat Intelligence Integrations
ESET Threat Intelligence provides advanced, real-time insights into global cybersecurity threats, empowering you to proactively defend your network and systems. By leveraging a vast database of threat data, it enables you to detect and respond to emerging threats, track attack trends, and enhance your security posture with actionable intelligence. With ESET Threat Intelligence, you can make informed decisions to protect your organization from sophisticated cyber threats.
Setup:
1) Log Collector must be installed.
2) Prepare the information from the ESET Threat Intelligence Account:
- Ensure that you have access to ESET Threat Intelligence feeds (via ESET Threat Intelligence API or downloadable data).
- Please prepare the Username and Password that you have received from ESET during their onboarding process.
References Information:
Data streams
This integration connects with the ESET Threat Intelligence TAXII version 2 server. It includes the following datasets for retrieving logs:
Dataset | TAXII2 Collection name |
---|---|
apt
|
apt stix 2.1
|
botnet
|
botnet stix 2.1
|
cc
|
botnet.cc stix 2.1
|
domains
|
domain stix 2.1
|
files
|
file stix 2.1
|
ip
|
ip stix 2.1
|
url
|
url stix 2.1
|
Obtaining an API Key for ESET Threat Intelligence
Usage of the ESET Threat Intelligence (ETI) API
The ESET Threat Intelligence (ETI) API can be used directly in a web browser’s address bar as a REST API, meaning that it does not necessarily require implementation in a programming language. This allows for a straightforward integration of threat intelligence data without the need for additional software development.
Authentication
Authentication with the ETI API is managed via a token. This token can be generated in the profile section of the ESET Threat Intelligence portal. It is important to note that each token is valid for only one hour, ensuring secure access to the API.
To generate a token, users can either manually generate it through the portal interface or use a CURL request. This approach provides flexibility, allowing automated generation of tokens for integration or scheduled use.
Generate via CURL Request
Step 1: Open a Command-Line Interface (CLI)
- Windows: Open Command Prompt (cmd) or PowerShell.
- macOS/Linux: Open Terminal.
Step 2: Enter the CURL Command
In the command-line interface, use the following CURL command to generate an authentication token:
curl -F name="YOUR-USERNAME" -F pass="YOUR-PASSWORD" ETI_URL/auth/
Step 3: Copy and save the authentication token
Note.
After 10 failed login attempts within 5 minutes, the user will be blocked for 15 minutes.
After 20 failed attempts from a specific IP address within 5 minutes, all login attempts from that IP will be blocked for 15 minutes.
If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.
CSPM for Azure Integration
This manual explains how to get started monitoring the security posture of your Azure CSP using the Cloud Security Posture Management (CSPM) feature.
Requirements
- The user who gives the CSPM integration permissions in Azure must be an Azure subscription admin.
Setup
Option 1: Service principal with client secret (recommended)
Before using this method, you must have set up a Microsoft Entra application and service principal that can access resources. Please go here before following the steps below.
- The following information is required.
- Directory (tenant) ID and Application (client) ID
- To get these values:
- Go to the Registered apps section of Microsoft Entra ID.
- Click on New Registration, name your app and click Register.
- Copy your new app’s Directory (tenant) ID and Application (client) ID.
- To get these values:
- Client Secret
- In Azure portal, select Certificates & secrets, then go to the Client secrets tab. Click New client secret.
- Copy the new secret.
- Directory (tenant) ID and Application (client) ID
- Return to Azure. Go to your Azure subscription list and select the subscription or management group you want to monitor with CSPM.
- Go to Access control (IAM) and select Add Role Assignment.
- Select the Reader function role, assign access to User, group, or service principal, and select your new app.
Option 2: Managed identity (optional)
This method involves creating an Azure VM (or using an existing one), giving it read access to the resources you want to monitor with CSPM, and installing the Log Collector on the Azure VM.
- Go to the Azure portal to create a new Azure VM.
- Follow the setup process, and make sure you enable System assigned managed identity under the Management tab.
- Go to your Azure subscription list and select the subscription or management group you want to monitor with CSPM.
- Go to Access control (IAM) and select Add Role Assignment.
- Select the Reader role, assign access to Managed Identity, then select your VM.
If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.
Resource Manager Endpoint Integration
The Azure Resource Manager (ARM) endpoint is the primary entry point for interacting with the Azure platform's resource management services. It allows users to deploy, manage, and organize resources like virtual machines, storage accounts, and networks within a defined Azure subscription. The ARM endpoint serves as a REST API that facilitates the creation, modification, and deletion of resources, ensuring secure and scalable management of Azure resources. It also supports role-based access control (RBAC), policy enforcement, and resource tagging for effective governance. In essence, the ARM endpoint enables seamless communication between Azure services and clients for infrastructure management.
Requirements:
- Microsoft Azure account
Setup:
How to Create an Azure Resource Manager Service Endpoint
1. Create Service Principal
- Click on Applications of the selected Active directory
- Click the Add button at the button of the page
- Enter a name for your application and make sure Web Application and/or Web API is selected
- Enter two URLs based on your application name
They do not have to be real. I used the same value for both. - Once the application is created, click on Configure
- Make note of the Client ID because we will need it in a moment
- Select a key duration under the keys section and click Save at the bottom of the page
- Once the key is saved copy the value and place it with your Client ID
This will be your only chance to collect this value.
2. Find Tenant ID
With the Active Directory select on the Applications page, we can harvest the Tenant ID.
- Click View Endpoints at the bottom of the page
- Copy any of the URLS and paste into an editor
- The GUID in the URL is your Tenant ID
3. Find Subscription Name and ID
You will also need the subscription name and ID to complete the service endpoint. We can get them while we are in the old portal.
- Click Settings in the left vertical menu
- Copy the Subscription and Subscription ID values
4. Grant access
- Click Browse and select Subscriptions
- Select the subscription you are using
- Click the Access button
- Click Add
- Select Contributor as the roll
- Search and select the name of the application you just created
- Click OK to grant the service principal access to your subscription
There is a script that can do all of this for you here on GitHub.
5. Create Service Endpoint
With the Service Principal created, we can now create the Service Endpoint in VSTS.
- Log in to VSTS and select a project
- Click the manage project gear icon in the upper right hand corner of the page
- Select the Services tab
- Select Azure Resource Manager from the New Service Endpoint drop down
Field Value Connection Name {AnyValueYouLike} Subscription Id Subscription Id Subscription Name Subscription Name Service Principal Id Client Id Service Principal Key Key Tenant Id Tenant Id - Click OK
How to navigate and locate Resource Manager Endpoint.
1.Sign in to the Azure portal.
2.Select Resource groups from the left panel.
3.Select the resource group that you have already created specifically for Azure Integration.
4.Hover to "Overview".
5.Verify the following resources were created in the resource group: "Endpoint Name" and type "Endpoint".
6.Copy the Endpoint Name for integration requirement. Ex. contosocdn123(myCDNProfile/contosocdn123)
If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.
CISCO Secure Email Gateway Integrations
The Cisco Email Security Appliance (ESA) integration is a comprehensive solution for managing and securing email traffic within an organization's network. It provides various functionalities, such as spam filtering, virus scanning, policy enforcement, and data loss prevention. The integration collects and parses data from the Cisco Secure Email Gateway (formerly known as Cisco Email Security Appliance) to provide valuable insights into the email environment. The data collection occurs through multiple methods, primarily through TCP/UDP communication and log file analysis.
Requirements:
- Cisco account
- Elastic agent already installed
Compatibility
This module has been tested against Cisco Secure Email Gateway server version 14.0.0 Virtual Gateway C100V with the below given logs pattern.
Setup
Configurations
- Sign-in to Cisco Secure Email Gateway Portal and follow the below steps for configurations:
- In Cisco Secure Email Gateway Administrator Portal, go to System Administration > Log Subscriptions.
- Click Add Log Subscription.
- Enter all the Required Details.
- Set Log Name as below for the respective category:
- AMP Engine Logs -> amp
- Anti-Spam Logs -> antispam
- Antivirus Logs -> antivirus
- Authentication Logs -> authentication
- Bounce Logs -> bounces
- Consolidated Event Logs -> consolidated_event
- Content Scanner Logs -> content_scanner
- HTTP Logs -> gui_logs
- IronPort Text Mail Logs -> error_logs
- Text Mail Logs -> mail_logs
- Status Logs -> status
- System Logs -> system
- Select Log Level as Information.
- Select Retrieval Method.
- Click Submit and commit the Changes.
Note
- Retrieval Method Supported:
- FTP Push to Remote Server for the below categories: AMP Engine Logs, Anti-Spam Logs, Antivirus Logs, Authentication Logs, Bounce Logs, Consolidated Event Logs, Content Scanner Logs, HTTP Logs, IronPort Text Mail Logs, Text Mail Logs, Status Logs and System Logs.
- Syslog Push for the below categories: AMP Engine Logs, Anti-Spam Logs, Antivirus Logs, Consolidated Event Logs, Content Scanner Logs, HTTP Logs, IronPort Text Mail Logs, Text Mail Logs, Status Logs and System Logs.
If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.
CISCO Nexus Integrations
Overview
The Cisco Nexus integration allows users to monitor Errors and System Messages. The Cisco Nexus series switches are modular and fixed port network switches designed for the data center. All switches in the Nexus range run the modular NX-OS firmware/operating system on the fabric. NX-OS has some high-availability features compared to the well-known Cisco IOS. This platform is optimized for high-density 10 Gigabit Ethernet.
Use the Cisco Nexus integration to collect and parse data from Syslog and log files. Then visualize that data through search, correlation and visualization within Elastic Security.
Data streams
The Cisco Nexus integration collects one type of data: log.
Log consists of errors and system messages. See more details about errors and system messages
Requirements
Elastic Agent must be installed.
The minimum kibana.version required is 8.7.0.
This module has been tested against the Cisco Nexus Series 9000, 3172T and 3048 Switches.
Setup
To collect data from Cisco Nexus, follow the below steps:
Logging System Messages to a File
You can configure the device to log system messages to a file. By default, system messages are logged to the file /logflash/log/logfilename .
Command or Action | Purpose | |
---|---|---|
Step 1 |
configure terminal Example:
|
Enters global configuration mode. |
Step 2 |
[ no ] logging logfile logfile-name severity-level [ | size bytes ] Example:
|
Configures the nonpersistent log file parameters. logfile-name : Configures the name of the log file that is used to store system messages. Default filename is "message". severity-level : Configures the minimum severity level to log. A lower number indicates a higher severity level. Default is 5. Range is from 0 through 7:
size bytes : Optionally specify maximum file size. Range is from 4096 through 4194304 bytes. |
Step 3 |
logging event {link-status | trunk-status} {enable | default} Example:
|
Logs interface events.
|
Configuring Syslog Servers
Note: Cisco recommends that you configure the syslog server to use the management virtual routing and forwarding (VRF) instance. For more information on VRFs, see Cisco Nexus 9000 Series NX-OS Unicast Routing Configuration Guide.
You can configure up to eight syslog servers that reference remote systems where you want to log system messages.
Procedure
Command or Action | Purpose | |
---|---|---|
Step 1 |
configure terminal Example:
|
Enters global configuration mode. |
Step 2 |
[no] logging server host [severity-level [use-vrf vrf-name]] Example:
Example:
|
Configures a syslog server at the specified hostname, IPv4, or IPv6 address. You can specify logging of messages to a particular syslog server in a VRF by using the use-vrf keyword. The use-vrf vrf-name keyword identifies the default or management values for the VRF name. The default VRF is the management VRF, by default. However, the show-running command will not list the default VRF. Severity levels range from 0 to 7:
The default outgoing facility is local7. The no option removes the logging server for the specified host. The first example forwards all messages on facility local 7. The second example forwards messages with severity level 5 or lower to the specified IPv6 address in VRF red. |
Step 3 |
logging source-interface loopback virtual-interface Example:
|
Enables a source interface for the remote syslog server. The range for the virtual-interface argument is from 0 to 1023. |
NOTE:
- Use the Timezone Offset parameter, if the timezone is not present in the log messages.
If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.
BitDefender Integrations
BitDefender GravityZone supports SIEM integration using "push notifications", which are JSON messages sent via HTTP POST to a HTTP or HTTPS endpoint, which this integration can consume.
This integration additionally provides:
- Collection of push notification configuration via API polling, which includes the "state" of the push notification service on the BitDefender GravityZone server, e.g. indicating if it is currently enabled or disabled. This is useful as the state may change to disabled (value of 0) for unknown reasons and you may wish to alert on this event.
- Collection of push notification statistics via API polling, which includes the number of events sent, and counters for errors of different types, which you may wish to use to troubleshoot lost push notification events and for alerting purposes.
- Support for multiple instances of the integration, which may be needed for MSP/MSSP scenarios where multiple BitDefender GravityZone tenants exist.
- BitDefender company ID to your own company name/description mapping, in order to determine to which tenant the event relates to in a human friendly way. This is very useful for MSP/MSSP environments or for large organizations with multiple sub-organizations.
This allows you to search, observe and visualize the BitDefender GravityZone events through Elastic, trigger alerts and monitor the BitDefender GravityZone Push Notification service for state and errors.
Data Stream
Log Stream Push Notifications
The BitDefender GravityZone events dataset provides events from BitDefender GravityZone push notifications that have been received.
All BitDefender GravityZone log events are available in the bitdefender_gravityzone.events
field group.
Compatibility
This integration supports BitDefender GravityZone, which is the business-oriented product set sold by BitDefender.
BitDefender products for home users are not supported.
The package collects BitDefender GravityZone push notification transported events sent in jsonrpc
, qradar
, or splunk
format.
The jsonrpc
format is recommended default, but the ingest pipeline will attempt to detect if qradar
or splunk
format events have been received and process them accordingly.
The integration can also collect the push notification configuration and statistics by polling the BitDefender GravityZone API.
Configuration
Enabling the integration in Elastic
- In Kibana go to Management > Integrations
- In "Search for integrations" search bar type GravityZone
- Click on "BitDefender GravityZone" integration from the search results.
- Click on Add BitDefender GravityZone button to add BitDefender GravityZone integration.
Create a BitDefender GravityZone API key that can configure a push notification service
The API key needed to configure push notifications, and collection push notification configuration state and statistics, is typically configured within the BitDefender GravityZone cloud portal.
Bear in mind the API key will be associated to the account you create it from. A named human account may not be desirable, e.g. you may wish to (probably should) create API keys for functions such as push notifications under a non-human/software service account that will never retire or be made redundant.
Give the API key a description and tick the "Event Push Service API" box at minimum.
NOTE: If you intend to use the API key for other API calls you may need to tick other boxes.
Click the Key value that is shown in blue.
If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.
Bitwarden Integrations
Overview
The Bitwarden integration allows users to monitor collections, events, groups, members and policies. Bitwarden is a free and open-source password management service that stores sensitive information such as website credentials in an encrypted vault. The Bitwarden platform offers a variety of client applications including a web interface, desktop applications, browser extensions, mobile apps and a command-line interface. Bitwarden offers a cloud-hosted service as well as the ability to deploy the solution on-premises.
Use the Bitwarden integration to collect and parse data from the REST APIs. Then visualize that data in Kibana.
Data streams
The Bitwarden integration collects five types of data: Collections, Events, Groups, Members and Policies.
Collections returns a list of an organization's collections.
Events returns a list of an organization's event logs.
Groups returns a list of an organization's groups.
Members returns the details of an organization's members.
Policies returns a list of an organization's policies.
Reference for Rest APIs of Bitwarden.
Requirements
Elasticsearch is needed to store and search data and Kibana is needed for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your hardware.
This module has been tested against Bitwarden Version 2023.2.0.
Setup
To collect data from Bitwarden REST APIs, follow the below steps:
- Go to the Bitwarden console, enter an email address and master password.
- Click Organizations.
- Go to Settings → Organization info.
- Click View API Key from API key Section.
- Enter master password.
- Click View API Key.
- Copy client_id and client_secret.
If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.
Forwarding logs from rsyslog client to a remote rsyslogs server
Introduction
This guide will walk you through setting up Rsyslog for log forwarding between a client and a remote server using Linux.
Setup
Server: The machine which will send message
Client: The machine which will receive the message
Prerequisites
Software Requirements
-
- Linux operating system
- Rsyslog (version 5.0 or higher recommended)
- Root or sudo access
Network Requirements
-
- Network connectivity between client and remote server
- Known IP address of the remote Rsyslog server
- Open network ports (typically 514 for UDP or TCP)
Step-by-Step Configuration Guide
Preparation
Before beginning, ensure you have:
-
- Administrative (root) access
- Stable network connection
- IP address of the remote server
Step 1: Rsyslog Installation
1.1 Obtain Root Access
sudo -i
- Enter your root password when prompted
1.2 Update System Packages
If you are using DNF, use the command below:
sudo dnf update
If you are using YUM, use the command below:
sudo yum update
1.3 Install Rsyslog
If you are using YUM, use the command below:
sudo yum install rsyslog
If you using DNF, use the command below:
sudo dnf install rsyslog
Verification Tip: Confirm Rsyslog is installed successfully
1.4 Start and Enable Rsyslog Service
sudo systemctl enable rsyslog
sudo systemctl start rsyslog
1.5 Check Rsyslog Status
sudo systemctl status rsyslog
Expected Result: Service should be in an active state
Step 2: Rsyslog Server and Client Configuration
The following steps outline how to forward system logs to a remote server using either TCP or UDP ports. You can choose to use either TCP or UDP, but if both are enabled, ensure that each protocol uses a different port.
2.1 Edit Rsyslog Configuration. Open using a text editor such as "vi" or "nano".
vi /etc/rsyslog.conf
2.2 Enable UDP or TCP Modules. This should be done on the Client machine only.
- For UDP, locate and uncomment the following lines by removing the #
symbol. The default port is 514, but you can change it if necessary.
$Modload imudp
$UDPServerRun 514
- For TCP, locate and uncomment the following lines by removing the #
symbol. The default port is 10514, but you can change it if necessary.
$Modload imtcp
$inputTCPServerRun 10514
2.3 Configure Log Template
Add the following line to define log storage:
$template RemoteLogs,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
& ~
2.4 On Server
Add content below at the end of the file /etc/rsyslog.conf.
This will configure the log forwarding to the remote host. Please update the "target", "port" and "tcp" appropriately.
*.* action(type="omfwd"
queue.type="LinkedList"
action.resumeRetryCount="-1"
queue.size="10000"
queue.saveonshutdown="on"
target="10.43.138.1" Port="10514" Protocol="tcp")
queue.type enables a LinkedList in-memory queue, queue_type can be direct, linkedlist or fixedarray (which are in-memory queues), or disk.
enabled queue.saveonshutdown saves in-memory data if rsyslog shuts down,
action.resumeRetryCount= “-1” setting prevents rsyslog from dropping messages when retrying to connect if server is not responding,
queue.size where size represents the specified size of disk queue part. The defined size limit is not restrictive, rsyslog always writes one complete queue entry, even if it violates the size limit.
target is the IP Address of the remote machine
Port is the port of the remote machine
Protocol is the protocol to be used. Values can be udp or tcp.
2.5 Add port in the firewall rules
On client side
Add the provided port to the firewall
iptables -A INPUT -p tcp --dport 10514 -j ACCEPT
Next open the port using nc
nc -l -p 10514 -4
2.6 Apply Server Configuration
systemctl restart rsyslog
2.7 Verify Log Directory
Type : ls -1
Expected Result:
Should see a directory with the client's hostname
Contains files like `rsyslogd.log` and `systemd.log`
Troubleshooting Tips
Ensure firewall settings allow log forwarding
Verify network connectivity between client and server
Check Rsyslog service status if logs aren't forwarding
Security Considerations
- Configure firewall rules appropriately
- Use encrypted log transmission when possible
- Regularly review and rotate logs
Common Issues
1. Port Blocking: Ensure port 514 is open
2. Permission Errors Verify root/sudo access
3. Network Connectivity: Check server IP and network settings
Conclusion
By following these steps, you should have successfully configured Rsyslog for log forwarding between a client and a remote server.
**Note:** Always test in a controlled environment first and adapt instructions to your specific system configuration.
New script for logs forwarding
# Define the syslog server IP address and port
$syslogServerIP = "192.168.20.24" # Replace with your Rocky server's IP
$syslogPort = 514
# File to store last sent event info.
$logFilePath = "C:\Users\Administrator\Desktop\lastEventInfo.txt"
# Initialize last sent events as a hashtable
$lastSentEvents = @{}
# Check if the file exists and read the last sent events
if (Test-Path $logFilePath) {
Write-Host "Loading last sent events from file."
$fileContent = Get-Content $logFilePath | ConvertFrom-Json
# Convert the JSON object back to hashtable
$fileContent.PSObject.Properties | ForEach-Object {
$lastSentEvents[$_.Name] = $_.Value
}
}
# Loop for sending logs
while ($true) {
Write-Host "Checking for new logs..."
# Define the logs you want to forward
$logNames = @("Application", "Security", "Setup", "System")
# Loop through each log
foreach ($logName in $logNames) {
Write-Host "Processing log: $logName"
# Get new logs with error handling
try {
$logs = Get-WinEvent -LogName $logName -ErrorAction Stop | Sort-Object TimeCreated
}
catch [System.Diagnostics.Eventing.Reader.EventLogNotFoundException] {
Write-Host "Log $logName not found"
continue
}
catch {
if ($_.Exception.Message -match "No events were found") {
Write-Host "No events found in $logName"
continue
}
else {
Write-Host "Error accessing $logName`: $_"
continue
}
}
foreach ($log in $logs) {
# Create a unique key based on Event ID and TimeCreated
$eventKey = "$($log.Id)-$($log.TimeCreated.Ticks)"
# Check if the event has already been sent
if (-not $lastSentEvents.ContainsKey($eventKey)) {
# Create syslog message format
$message = "<134>" + $log.TimeCreated.ToString("yyyy-MM-dd HH:mm:ss") + " " + $log.ProviderName + ": " + $log.Message
try {
# Send the message to the syslog server
$client = New-Object System.Net.Sockets.UdpClient
$client.Connect($syslogServerIP, $syslogPort)
$bytes = [System.Text.Encoding]::ASCII.GetBytes($message)
$client.Send($bytes, $bytes.Length)
$client.Close()
# Mark the event as sent
$lastSentEvents[$eventKey] = $true
Write-Host "Sent log: $message"
}
catch {
Write-Host "Error sending log: $_"
}
finally {
if ($client) {
$client.Dispose()
}
}
} else {
Write-Host "Log already sent: $eventKey"
}
}
}
# Save the last sent events to the file
try {
$lastSentEvents | ConvertTo-Json | Set-Content -Path $logFilePath
Write-Host "Last sent events updated."
}
catch {
Write-Host "Error saving last sent events: $_"
}
# Wait for 1 second before running again
Start-Sleep -Seconds 1
}
=====================
# Define the syslog server IP address and port
$syslogServerIP = "192.168.20.24"
$syslogPort = 514
# Test connectivity first
Write-Host "Testing connection to syslog server..." -ForegroundColor Yellow
$testConnection = Test-NetConnection -ComputerName $syslogServerIP -Port $syslogPort -WarningAction SilentlyContinue
if ($testConnection.TcpTestSucceeded) {
Write-Host "Connection test successful!" -ForegroundColor Green
} else {
Write-Host "Connection test failed! Please check network connectivity and firewall rules." -ForegroundColor Red
exit
}
# Send a test message
try {
$testMessage = "<134>$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') TEST-MESSAGE: This is a test syslog message"
$client = New-Object System.Net.Sockets.UdpClient
$client.Connect($syslogServerIP, $syslogPort)
$bytes = [System.Text.Encoding]::ASCII.GetBytes($testMessage)
$client.Send($bytes, $bytes.Length)
Write-Host "Test message sent successfully!" -ForegroundColor Green
Write-Host "Message content: $testMessage" -ForegroundColor Gray
$client.Close()
} catch {
Write-Host "Error sending test message: $_" -ForegroundColor Red
}
# Now run the main script for 30 seconds as a test
$endTime = (Get-Date).AddSeconds(30)
$sentCount = 0
while ((Get-Date) -lt $endTime) {
$logNames = @("Application", "Security", "Setup", "System")
foreach ($logName in $logNames) {
Write-Host "Checking $logName log..." -ForegroundColor Yellow
try {
$logs = Get-WinEvent -LogName $logName -MaxEvents 5 -ErrorAction Stop
foreach ($log in $logs) {
$message = "<134>" + $log.TimeCreated.ToString("yyyy-MM-dd HH:mm:ss") + " " +
$log.ProviderName + ": " + $log.Message
try {
$client = New-Object System.Net.Sockets.UdpClient
$client.Connect($syslogServerIP, $syslogPort)
$bytes = [System.Text.Encoding]::ASCII.GetBytes($message)
$client.Send($bytes, $bytes.Length)
$client.Close()
$sentCount++
Write-Host "Successfully sent log entry $sentCount" -ForegroundColor Green
Write-Host "EventID: $($log.Id)" -ForegroundColor Gray
Write-Host "Source: $($log.ProviderName)" -ForegroundColor Gray
Write-Host "Time: $($log.TimeCreated)" -ForegroundColor Gray
Write-Host "------------------" -ForegroundColor Gray
} catch {
Write-Host "Failed to send log entry: $_" -ForegroundColor Red
}
}
} catch {
Write-Host "Error reading from $logName: $_" -ForegroundColor Red
}
}
Start-Sleep -Seconds 1
}
Write-Host "Test completed! Total messages sent: $sentCount" -ForegroundColor Green
Write-Host "Please check your syslog server to verify receipt of messages." -ForegroundColor Yellow
====== testing connection
$syslogServerIP = "192.168.20.24"
$syslogPort = 514
Write-Host "Testing basic connectivity..."
Test-NetConnection -ComputerName $syslogServerIP -InformationLevel Detailed
Write-Host "`nTesting specific port connectivity..."
Test-NetConnection -ComputerName $syslogServerIP -Port $syslogPort -WarningAction SilentlyContinue | Format-List *
====
# Windows Event Log Forwarder to RHEL Rsyslog with CPU Optimization
# Configuration Parameters
$RsyslogServer = "192.168.20.40"
$RsyslogPort = 514
$LogPath = "$env:USERPROFILE\Desktop\WindowsLogs"
$JsonPath = "$LogPath\JsonLogs"
$SummaryPath = "$LogPath\Summary"
$MaxCPUUsage = 30 # Maximum CPU percentage to use
$BatchSize = 10 # Number of events to process in batch
$SleepInterval = 2 # Seconds to sleep between batches
# Create necessary directories
$null = New-Item -ItemType Directory -Force -Path $LogPath
$null = New-Item -ItemType Directory -Force -Path $JsonPath
$null = New-Item -ItemType Directory -Force -Path $SummaryPath
# Function to check CPU usage and throttle if necessary
function Test-CPUThreshold {
$CPUUsage = (Get-Counter '\Processor(_Total)\% Processor Time').CounterSamples.CookedValue
if ($CPUUsage -gt $MaxCPUUsage) {
Start-Sleep -Seconds ($CPUUsage / 20) # Dynamic sleep based on CPU load
return $true
}
return $false
}
# Function to convert Windows event to syslog format
function Convert-ToSyslogFormat {
param (
[Parameter(Mandatory=$true)]
$Event
)
$Severity = switch ($Event.LevelDisplayName) {
"Error" { 3 } # Error
"Warning" { 4 } # Warning
"Information" { 6 } # Info
"Critical" { 2 } # Critical
default { 5 } # Notice
}
$Facility = 16 # local0
$Priority = ($Facility * 8) + $Severity
$Timestamp = $Event.TimeCreated.ToString("yyyy-MM-ddTHH:mm:ss.fffzzz")
$Hostname = $env:COMPUTERNAME
$Message = "$($Event.Message)".Replace("`n", " ").Replace("`r", " ")
"<$Priority>1 $Timestamp $Hostname $($Event.LogName)[$($Event.ProcessId)] $($Event.Id) [$($Event.ProviderName)] - $Message"
}
# Function to send syslog message with UDP client pooling
$UdpClientPool = [System.Collections.Concurrent.ConcurrentDictionary[string, System.Net.Sockets.UdpClient]]::new()
function Send-SyslogMessage {
param (
[Parameter(Mandatory=$true)]
[string]$Message
)
try {
$ClientKey = [System.Threading.Thread]::CurrentThread.ManagedThreadId.ToString()
$UdpClient = $UdpClientPool.GetOrAdd($ClientKey, {
$Client = New-Object System.Net.Sockets.UdpClient
$Client.Client.ReceiveTimeout = 1000
$Client.Client.SendTimeout = 1000
return $Client
})
$EncodedMessage = [System.Text.Encoding]::UTF8.GetBytes($Message)
$null = $UdpClient.Send($EncodedMessage, $EncodedMessage.Length, $RsyslogServer, $RsyslogPort)
}
catch {
Write-Warning "Failed to send message to rsyslog: $_"
$Message | Out-File -Append -FilePath "$LogPath\FailedMessages.log"
}
}
# Main event monitoring function with event batching
function Start-EventMonitoring {
$EventLogs = @("Application", "Security", "Setup", "System")
$LastCheckpoints = @{}
$EventLogs | ForEach-Object { $LastCheckpoints[$_] = [DateTime]::Now }
Write-Host "Starting continuous event monitoring..."
while ($true) {
if (Test-CPUThreshold) { continue }
foreach ($LogName in $EventLogs) {
try {
$Events = Get-WinEvent -LogName $LogName -MaxEvents $BatchSize `
-FilterXPath "*[System[TimeCreated[@SystemTime>='$($LastCheckpoints[$LogName].ToUniversalTime().ToString("o"))']]]" -ErrorAction SilentlyContinue
if ($Events) {
$LastCheckpoints[$LogName] = $Events[0].TimeCreated
foreach ($Event in $Events) {
# Convert and send in batches
$SyslogMessage = Convert-ToSyslogFormat -Event $Event
Send-SyslogMessage -Message $SyslogMessage
# Save JSON format
$JsonEvent = $Event | Select-Object * | ConvertTo-Json -Depth 5
$JsonEvent | Out-File -Append -FilePath "$JsonPath\$LogName-$(Get-Date -Format 'yyyyMMdd').json"
}
}
}
catch {
Write-Warning "Error processing $LogName events: $_"
Start-Sleep -Seconds 5 # Back off on error
}
}
Start-Sleep -Seconds $SleepInterval
}
}
# Cleanup function
function Cleanup {
$UdpClientPool.Values | ForEach-Object { $_.Dispose() }
$UdpClientPool.Clear()
}
# Start monitoring with error handling
try {
Write-Host "Press Ctrl+C to stop the monitoring..."
Start-EventMonitoring
}
catch {
Write-Warning "Error in main monitoring loop: $_"
}
finally {
Cleanup
}
=======latest script
# PowerShell script to export Windows Event Logs to JSON
param (
[string]$LogName = "System", # Default to System log
[int]$Hours = 24, # Default to last 24 hours
[string]$DesktopPath = [Environment]::GetFolderPath("Desktop")
)
# Create timestamp for filename
$timestamp = Get-Date -Format "yyyyMMdd_HHmmss"
$outputFile = Join-Path $DesktopPath "EventLogs_${LogName}_${timestamp}.json"
try {
# Calculate time range
$startTime = (Get-Date).AddHours(-$Hours)
# Get events from specified log
Write-Host "Collecting events from $LogName log for the past $Hours hours..."
Get-WinEvent -FilterHashtable @{
LogName = $LogName
StartTime = $startTime
} | Select-Object TimeCreated, Level, LevelDisplayName, LogName, Id,
ProviderName, Message | ConvertTo-Json -Depth 10 |
Out-File -FilePath $outputFile -Encoding UTF8
Write-Host "Events successfully exported to: $outputFile"
Write-Host "Total events exported: $((Get-Content $outputFile | ConvertFrom-Json).Count)"
}
catch {
Write-Error "Error occurred while exporting events: $_"
exit 1
}
===== user create script
# PowerShell script to find user creation events
param (
[int]$DaysBack = 30, # Default to last 30 days
[string]$DesktopPath = [Environment]::GetFolderPath("Desktop")
)
# Create timestamp for filename
$timestamp = Get-Date -Format "yyyyMMdd_HHmmss"
$outputFile = Join-Path $DesktopPath "UserCreationEvents_${timestamp}.json"
try {
# Calculate time range
$startTime = (Get-Date).AddDays(-$DaysBack)
Write-Host "Searching for user creation events in the past $DaysBack days..."
# Get user creation events (Event ID 4720)
$events = Get-WinEvent -FilterHashtable @{
LogName = 'Security'
ID = 4720 # User account created
StartTime = $startTime
} | ForEach-Object {
$xml = [xml]$_.ToXml()
# Extract relevant data from event
@{
TimeCreated = $_.TimeCreated
CreatedUser = $xml.Event.EventData.Data | Where-Object { $_.Name -eq 'TargetUserName' } | Select-Object -ExpandProperty '#text'
CreatedBy = $xml.Event.EventData.Data | Where-Object { $_.Name -eq 'SubjectUserName' } | Select-Object -ExpandProperty '#text'
AccountType = $xml.Event.EventData.Data | Where-Object { $_.Name -eq 'TargetUserName' } | Select-Object -ExpandProperty '#text'
Domain = $xml.Event.EventData.Data | Where-Object { $_.Name -eq 'TargetDomainName' } | Select-Object -ExpandProperty '#text'
SecurityID = $xml.Event.EventData.Data | Where-Object { $_.Name -eq 'TargetSid' } | Select-Object -ExpandProperty '#text'
WorkstationName = $xml.Event.EventData.Data | Where-Object { $_.Name -eq 'WorkstationName' } | Select-Object -ExpandProperty '#text'
}
} | ConvertTo-Json -Depth 10
# Save to file
$events | Out-File -FilePath $outputFile -Encoding UTF8
# Display summary
$eventCount = ($events | ConvertFrom-Json).Count
Write-Host "Found $eventCount user creation events"
Write-Host "Results exported to: $outputFile"
# Display recent creations on screen
if ($eventCount -gt 0) {
Write-Host "`nMost recent user creations:"
$events | ConvertFrom-Json | Select-Object TimeCreated, CreatedUser, CreatedBy, Domain | Format-Table -AutoSize
}
}
catch {
Write-Error "Error occurred while searching events: $_"
exit 1
}
========= find user
# Default usage (checks last 30 days)
.\find-user-creations.ps1
# Specify different time range (e.g., last 90 days)
.\find-user-creations.ps1 -DaysBack 90
======= INSTALLING SYSLOG FORWARDER
import socket
import json
import time
import os
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
import codecs
class JsonFileHandler(FileSystemEventHandler):
def __init__(self, syslog_server, syslog_port):
self.syslog_server = syslog_server
self.syslog_port = syslog_port
self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
def on_modified(self, event):
if not event.is_directory and event.src_path.endswith('.json'):
try:
# Wait for file to be completely written
time.sleep(1)
# Open file with UTF-8-SIG to handle BOM
with codecs.open(event.src_path, 'r', encoding='utf-8-sig') as file:
content = file.read().strip()
print(f"Reading file: {event.src_path}")
print(f"Content length: {len(content)}")
if not content:
print("Warning: File is empty")
return
# Parse JSON content
json_data = json.loads(content)
# Convert back to string for sending
message = json.dumps(json_data).encode('utf-8')
# Send to syslog server
self.sock.sendto(message, (self.syslog_server, self.syslog_port))
print(f"Successfully sent data to {self.syslog_server}:{self.syslog_port}")
except json.JSONDecodeError as e:
print(f"JSON Error: {str(e)}")
print(f"Error occurred at position: {e.pos}")
print(f"Line with error: {content[max(0, e.pos-50):e.pos+50]}")
except Exception as e:
print(f"Error processing file: {str(e)}")
def main():
# Configuration
WATCH_PATH = os.path.join(os.path.expanduser("~"), "Desktop")
SYSLOG_SERVER = "192.168.1.223" # Your Ubuntu server IP
SYSLOG_PORT = 514
print(f"Starting watcher on {WATCH_PATH}")
print(f"Forwarding to {SYSLOG_SERVER}:{SYSLOG_PORT}")
event_handler = JsonFileHandler(SYSLOG_SERVER, SYSLOG_PORT)
observer = Observer()
observer.schedule(event_handler, WATCH_PATH, recursive=False)
observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
print("\nStopping watcher...")
observer.join()
if __name__ == "__main__":
main()
=================