System Integrations

• Cyber Incident Monitoring Integration Procedure
• Microsoft 365
• GitHub
• Add Windows Integrations
• Sysmon for Linux
• 1 Password Integrations
• Atlassian Bitbucket Integrations
• AWS Cloudtrails Integrations
• AWS GuardDuty Integrations
• AWS Security Hub Integrations
• AWS Integrations
• CISCO Meraki Integrations
• CISCO Secure Endpoint Integrations
• CISCO Umbrella Integrations
• Cloudflare Integration
• Crowdstrike Integrations
• Dropbox Integrations
• F5 Integrations
• Fortinet-Fortigate Integrations
• GCP Integrations
• GitLab Integrations
• Google Workspace Integrations
• Jumpcloud Integrations
• Mimecast Integrations
• MongoDB Integrations
• OKTA Integrations
• Pulse Connect Secure Integrations
• Slack Integrations
• System Integrations
• Team Viewer Integrations
• Z Scaler Integrations
• gcp
• VMware vSphere Integration

Cyber Incident Monitoring Integration Procedure

Go to > Cyber Incident Monitoring

Microsoft 365

Microsoft Office 365 integration currently supports user, admin, system, and policy actions and events from Office 365 and Azure AD activity logs exposed by the Office 365 Management Activity API.

Procedures 

To perform the setup, please confirm that you have the following access: 

Register a new Office 365 web application To get started collecting Office 365 logs, register an Office 365 web application: 

Setup Active Directory security permissions

The Active Directory security permissions allow the application you created to read threat intelligence data and activity reports for your organization. 

To set up Active Directory permissions: 

GitHub

Introduction 

The GitHub integration collects events from the GitHub API. 

Logs

Audit 

The GitHub audit log records all events related to the GitHub organization.  

To use this integration, you must be an organization owner, and you must use an Personal Access Token with the admin:org scope. 

This integration is not compatible with GitHub Enterprise server. 

 

Code Scanning 

The Code Scanning lets you retrieve all security vulnerabilities and coding errors from a repository setup using Github Advanced Security Code Scanning feature.  

To use this integration, GitHub Apps must have the security_events read permission. Or use a personal access token with the security_events scope for private repos or public_repo scope for public repos.  

 

Secret Scanning 

The Github Secret Scanning lets you retrieve secret scanning for advanced security alerts from a repository setup using Github Advanced Security Secret Scanning feature.  

To use this integration, GitHub Apps must have the secret_scanning_alerts read permission. Or you must be an administrator for the repository or for the organization that owns the repository, and you must use a personal access token with the repo scope or security_events scope. For public repositories, you may instead use the public_repo scope.  

 

Dependabot 

The Github Dependabot lets you retrieve known vulnerabilities in dependencies from a repository setup using Github Advanced Security Dependabot feature.  

 To use this integration, you must be an administrator for the repository or for the organization that owns the repository, and you must use a personal access token with the repo scope or security_events scope. For public repositories, you may instead use the public_repo scope.  

 

Issues 

The Github Issues datastream lets you retrieve github issues, including pull requests, issue assignees, comments, labels, and milestones. See About Issues for more details. You can retrieve issues for specific repository or for entire organization. Since Github API considers pull requests as issues, users can use github.issues.is_pr field to filter for only pull requests.

All issues including closed are retrieved by default. If users want to retrieve only open requests, you need to change State parameter to open. 

To use this integration, users must use Github Apps or Personal Access Token with read permission to repositories or organization. Please refer to Github Apps Permissions Required and Personal Access Token Permissions Required for more details. 

GitHub Integration Procedures 

Please provide the following information to CyTech: 

1.Select Settings 

2. Select Developer Settings 

 

 

3. Select token (classic) 

4. Select scope admin:scope 

Collect GitHub logs via API 

GHAS Code Scanning 

GHAS Dependabot 

Github Issues 

1. Personal Access Token - the GitHub Personal Access Token. 

2. Repository owner - The owner of GitHub Repository. If repository belongs to an organization, owner is name of the organization.

GHAS Secret Scanning 

1. Personal Access Token - the GitHub Personal Access Token. Requires admin access to the repository or organization owning the repository along with a personal access token with 'public_repo' scope for public repositories and repo or security_events scope for private repositories. \nSee List secret scanning alerts for a repository 

2. Repository owner - The owner of GitHub Repository 

Add Windows Integrations

Introduction 

The Windows integration allows you to monitor the Windows OS, services, applications, and more. 

Use the Windows integration to collect metrics and logs from your machine. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue. 

For example, if you wanted to know if a Windows service unexpectedly stops running, you could install the Windows integration to send service metrics to Elastic. Then, you could view real-time changes to service status in Kibana's [Metrics Windows] Services dashboard. 

Data streams 

The Windows integration collects two types of data: logs and metrics. 

Logs help you keep a record of events that happen on your machine. Log data streams collected by the Windows integration include forwarded events, PowerShell events, and Sysmon events. Log collection for the Security, Application, and System event logs is handled by the System integration. See more details in the Logs reference. 

Metrics give you insight into the state of the machine. Metric data streams collected by the Windows integration include service details and performance counter values. See more details in the Metrics reference. 

Note: For 7.11, security, application and system logs have been moved to the system package. 

 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

Requirements 

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware. 

Each data stream collects different kinds of metric data, which may require dedicated permissions to be fetched and which may vary across operating systems. 

Setup 

For step-by-step instructions on how to set up an integration, see the Getting started guide. 

Note: Because the Windows integration always applies to the local server, the hosts config option is not needed. 

Ingesting Windows Events via Splunk 

This integration allows you to seamlessly ingest data from a Splunk Enterprise instance. The integration uses the httpjson input in Elastic Agent to run a Splunk search via the Splunk REST API and then extract the raw event from the results. The raw event is then processed via the Elastic Agent. You can customize both the Splunk search query and the interval between searches. For more information see Ingest data from Splunk. 

Note: This integration requires Windows Events from Splunk to be in XML format. To achieve this, renderXml needs to be set to 1 in your inputs.conf file. 

Logs reference 

Forwarded 

The Windows forwarded data stream provides events from the Windows ForwardedEvents event log. The fields will be the same as the channel specific data streams. 

Powershell 

The Windows powershell data stream provides events from the Windows Windows PowerShell event log. 

System Integration Procedures 

Collect events from the following Windows event log channels: (Enable Yes/No) 

Powershell (Enable Yes/No) 

Powershell Operational (Enable Yes/No) 

Sysmon Operational (Enable Yes/No) 

Collect Windows perfmon and service metrics (Enable Yes/No) 

Windows service metrics (Enable Yes/No) 

Collect logs from third-party REST API (experimental) (Enable Yes/No) 

Windows ForwardedEvents via Splunk Enterprise REST API (Enable Yes/No) 

Windows Powershell Events via Splunk Enterprise REST API (Enable Yes/No) 

Windows Powershell Operational Events via Splunk Enterprise REST API (Enable Yes/No) 

Windows Sysmon Operational Events via Splunk Enterprise REST API (Enable Yes/No) 

Sysmon for Linux

Introduction 

The Sysmon for Linux integration allows you to monitor the Sysmon for Linux, which is an open-source system monitor tool developed to collect security events from Linux environments. 

Use the Sysmon for Linux integration to collect logs from linux machine which has sysmon tool running. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue. 

NOTE: To collect Sysmon events from Windows event log, use Windows sysmon_operational data stream instead. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

Requirements 

Setup 

For step-by-step instructions on how to set up an integration, see the Getting started guide. 

Data streams 

The Sysmon for Linux log data stream provides events from logs produced by Sysmon tool running on Linux machine. 

Sysmon for Linux Integration 

Please provide the following information to CyTech:   

Collect Sysmon for Linux logs (Enable Yes/No) 

1 Password Integrations

Introduction 

With 1Password Business, you can send your account activity to your security information and event management (SIEM) system, using the 1Password Events API. 

Get reports about 1Password activity, such as sign-in attempts and item usage, while you manage all your company’s applications and services from a central location.  

With 1Password Events Reporting and Elastic SIEM, you can: 

Events 

Sign-in Attempts 

Use the 1Password Events API to retrieve information about sign-in attempts. Events include the name and IP address of the user who attempted to sign in to the account, when the attempt was made, and – for failed attempts – the cause of the failure. 

Item Usages 

This uses the 1Password Events API to retrieve information about items in shared vaults that have been modified, accessed, or used. Events include the name and IP address of the user who accessed the item, when it was accessed, and the vault where the item is stored. 

Requirements 

You can set up Events Reporting if you’re an owner or administrator. 

Ready to get started?  

1Password Integration Procedures 

Please provide the following information to CyTech: 

The 1Password Events API Beat returns information from 1Password through requests to the Events REST API and sends that data securely to Elasticsearch. Requests are authenticated with a bearer token. Issue a token for each application or service you use. 

To connect your 1Password account to Elastic: 

You can now use Elasticsearch with the 1Password Events API Beat to monitor events from your 1Password account. The returned data will follow the Elastic Common Schema (ECS) specifications.

Collect events from 1Password Events API 

Atlassian Bitbucket Integrations

Introduction 

The Bitbucket integration collects audit logs from the audit log files or the audit API. 

Reference:  https://developer.atlassian.com/server/bitbucket/reference/rest-api/  

Assumptions 

The procedures described in Section 3 assume that a Log Collector has already been set up.  

Requirements 

For more information on auditing in Bitbucket and how it can be configured, see View and configure the audit log on Atlassian's website. 

Reference: https://confluence.atlassian.com/bitbucketserver/view-and-configure-the-audit-log-776640417.html  

 

Logs 

Audit 

The Confluence integration collects audit logs from the audit log files or the audit API from self-hosted Confluence Data Center. It has been tested with Confluence 7.14.2 but is expected to work with newer versions. As of version 1.2.0, this integration added experimental support for Atlassian Confluence Cloud. JIRA Cloud only supports Basic Auth using username and a Personal Access Token. 

Atlassian Bitbucket Integration Procedures 

Please provide the following information to CyTech: 

Collect Bitbucket audit logs via log files 

Collect Bitbucket audit logs via API (Enable Yes/No) 

AWS Cloudtrails Integrations

Introduction 

The AWS CloudTrail integration allows you to monitor AWS CloudTrail 

Reference: https://aws.amazon.com/cloudtrail/  

Use the AWS CloudTrail integration to collect and parse logs related to account activity across your AWS infrastructure. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference logs when troubleshooting an issue. 

For example, you could use the data from this integration to spot unusual activity in your AWS accounts—like excessive failed AWS console sign in attempts. 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Data streams 

The AWS CloudTrail integration collects one type of data: logs. 

Logs help you keep a record of every event that CloudTrail receives. These logs are useful for many scenarios, including security and access audits. See more details in the Logs reference. 

Reference : https://aquila-elk.kb.us-east-1.aws.found.io:9243/app/integrations/detail/aws-1.28.3/overview?integration=cloudtrail - logs-reference 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

Requirements 

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware. 

Before using any AWS integration you will need: 

For more details about these requirements, see the AWS integration documentation. 

 

Setup 

Use this integration if you only need to collect data from the AWS CloudTrail service. 

If you want to collect data from two or more AWS services, consider using the AWS integration. When you configure the AWS integration, you can collect data from as many AWS services as you'd like. 

For step-by-step instructions on how to set up an integration, see the Getting started guide. 

 

Logs reference 

The cloudtrail data stream collects AWS CloudTrail logs. CloudTrail monitors events like user activity and API usage in AWS services. If a user creates a trail, it delivers those events as log files to a specific Amazon S3 bucket. 

AWS CloudTrail integration Procedures  

The following will need to be provided in the Configure integration when adding the AWS Audit CloudTrail integration.   

Procedures:   

Please provide the following information to CyTech:   

Configure Integration 

AWS GuardDuty Integrations

Introduction 

The Amazon GuardDuty integration collects and parses data from Amazon GuardDuty Findings REST APIs. 

The Amazon GuardDuty integration can be used in three different modes to collect data: 

Assumptions 

The procedures described in Section 3 assume that a Log Collector has already been setup.  

Requirements 

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware. 

Note: It is recommended to use AWS SQS for Amazon GuardDuty. 

Aws GuardDuty integration Procedures 

To collect data from AWS S3 Bucket, follow the steps below: 

To collect data from AWS SQS, follow the steps below: 

Note: 

To collect data from Amazon GuardDuty API, users must have an Access Key and a Secret Key. To create an API token follow the steps below: 

Note 

 

AWS Security Hub Integrations

Introduction 

The AWS Security Hub integration collects and parses data from AWS Security Hub REST APIs. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Compatibility 

Requirements 

To collect data from AWS Security Hub APIs, users must have an Access Key and a Secret Key. To create API token follow below steps: 

Note:

Logs:

AWS Security Hub Integration Procedures 

Please provide the following information to CyTech: 

Collect AWS Security Hub logs via API 

Collect AWS Security Hub Insights from AWS 

AWS Integrations

Introduction 

This document shows information related to AWS Integration.  

The AWS integration is used to fetch logs and metrics from Amazon Web Services. 

The usage of the AWS integration is to collect metrics and logs across many AWS services managed by your AWS account. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Requirements 

Before using the AWS integration you will need: 

AWS Credentials 

Use access keys directly (Option 1 Recommended) 

Access keys are long-term credentials for an IAM user or the AWS account root user. To use access keys as credentials, you need to provide: 

Use an IAM role Amazon Resource Name (ARN)  

To use an IAM role ARN, you need to provide either a credential profile or access keys along with the role_arn advanced option. role_arn is used to specify which AWS IAM role to assume for generating temporary credentials. 

Use a shared credentials file (Option 2) 

Instead of providing the access_key_id and secret_access_key directly to the integration, you will provide two advanced options to look up the access keys in the shared credentials file: 

shared_credential_file: The directory of the shared credentials file. 

Access keys are long-term credentials for an IAM user or the AWS account root user. To use access keys as credentials, you need to provide: 

AWS Permissions 

Specific AWS permissions are required for the IAM user to make specific AWS API calls. To enable the AWS integration to collect metrics and logs from all supported services, make sure to give necessary permissions which CyTech to monitor. 

Reference permissions: 

AWS Integrations Procedures 

Access Key ID and Secret Access Key:  

These are associated with AWS Identity and Access Management (IAM) users and are used for programmatic access to AWS services. To find them: 

S3 Bucket ARN:  

You can find the Amazon Resource Name (ARN) for an S3 bucket in the S3 Management Console or by using the AWS CLI. It typically looks like this: 

Log Group ARN: 

Log Groups are associated with AWS CloudWatch Logs. You can find the ARN for a log group as follows: 

SQS Queue URL: (Ignore if you're not using SQS Queue URL) 

To find the URL of an Amazon Simple Queue Service (SQS) queue: 

Please provide the following information to CyTech: 

CISCO Meraki Integrations

Introduction 

Cisco Meraki offers a centralized cloud management platform for all Meraki devices such as MX Security Appliances, MR Access Points and so on. Its out-of-band cloud architecture creates secure, scalable, and easy-to-deploy networks that can be managed from anywhere. This can be done from almost any device using web-based Meraki Dashboard and Meraki Mobile App. Each Meraki network generates its own events. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

Compatibility 

A syslog server can be configured to store messages for reporting purposes from MX Security Appliances, MR Access Points, and MS switches. This package collects events from the configured syslog server. The integration supports collection of events from "MX Security Appliances" and "MR Access Points". The "MS Switch" events are not recognized. 

Requirements 

Cisco Meraki Dashboard Configuration 

SYSLOG 

Cisco Meraki dashboard can be used to configure one or more syslog servers and Meraki message types to be sent to the syslog servers. Refer to Syslog Server Overview and Configuration page for more information on how to configure syslog server on Cisco Meraki. 

API ENDPOINT (WEBHOOKS) 

Cisco Meraki dashboard can be used to configure Meraki webhooks. Refer to the Webhooks Dashboard Setup section. 

Configure the Cisco Meraki integration 

SYSLOG 

Depending on the syslog server setup in your environment check one/more of the following options "Collect syslog from Cisco Meraki via UDP", "Collect syslog from Cisco Meraki via TCP", "Collect syslog from Cisco Meraki via file". 

Enter the values for syslog host and port OR file path based on the chosen configuration options. 

API Endpoint (Webhooks) 

Check the option "Collect events from Cisco Meraki via Webhooks" option. 

Log Events 

Enable to collect Cisco Meraki log events for all the applications configured for the chosen log stream. 

Logs 

Syslog 

The cisco_meraki.log dataset provides events from the configured syslog server. All Cisco Meraki syslog specific fields are available in the cisco_meraki.log field group. 

API Endpoint (Webhooks) 

 

Cisco Meraki Integration Procedures 

Please provide the following information to CyTech: 

CISCO Secure Endpoint Integrations

Introduction 

Secure Endpoint offers cloud-delivered, advanced endpoint detection and response across multidomain control points to rapidly detect, contain, and remediate advanced threats. 

Assumptions 

The procedures described in Section 3 assume that a Log Collector has already been setup.   

Requirements 

This integration is for Cisco Secure Endpoint logs. It includes the following datasets for receiving logs over syslog or read from a file: 

Logs 

Secure Endpoint 

The event dataset collects Cisco Secure Endpoint logs. 

What can the Secure Endpoint API be used for? 

Top Use Cases 

Response Format 

Cisco Secure Endpoint Integration Procedures 

Please provide the following information to CyTech: 

Collect logs from the Cisco Secure Endpoint API. 

 

CISCO Umbrella Integrations

Introduction 

Cisco Umbrella is a cloud security platform that provides an additional line of defense against malicious software and threats on the internet by using threat intelligence. That intelligence helps prevent adware, malware, botnets, phishing attacks, and other known bad Websites from being accessed. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

Prerequisites 

Requirements 

This integration is for Cisco Umbrella. It includes the following datasets for receiving logs from an AWS S3 bucket using an SQS notification queue and Cisco Managed S3 bucket without SQS: 

Logs 

Umbrella 

When using Cisco Managed S3 buckets that does not use SQS there is no load balancing possibilities for multiple agents, a single agent should be configured to poll the S3 bucket for new and updated files, and the number of workers can be configured to scale vertically. 

The log dataset collects Cisco Umbrella logs. 

Advantages of Integrating with the Umbrella API 

The Umbrella API features a number of improvements over the Umbrella v1 APIs and the Umbrella Reporting v2 API. 

Before you send a request to the Umbrella API, you must create new Umbrella API credentials and generate an API access token. For more information, see Umbrella API Authentication. 

Authentication 

The Umbrella API provides a standard REST interface and supports the OAuth 2.0 client credentials flow. To get started, log in to Umbrella and create an Umbrella API key. Then, use your API credentials to generate an API access token. 

Note: API keys, passwords, secrets, and tokens allow access to your private customer data. You should never share your credentials with another user or organization. 

Log in to Umbrella 

 

Cisco Secure Endpoint Integration Procedures 

Please provide the following information to CyTech: 

Collect logs from the Cisco Umbrella 

Cloudflare Integration

Introduction 

Cloudflare integration uses Cloudflare's API to retrieve audit logs and traffic logs from Cloudflare, for a particular zone, and ingest them into Elasticsearch. This allows you to search, observe and visualize the Cloudflare log events through Elasticsearch. 

Users of Cloudflare use Cloudflare services to increase the security and performance of their web sites and services. 

To enable the Cloudflare Logpush, please refer to Section 5. Currently, the procedures described is for the setup of Amazon S3.  

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Requirements 

Configure Cloudflare audit logs data stream 

Enter values "Auth Email", "Auth Key" and "Account ID". 

Configure Cloudflare logs 

These logs contain data related to the connecting client, the request path through the Cloudflare network, and the response from the origin web server. For more information see here. 

The integration can retrieve Cloudflare logs using - 

CONFIGURE USING AUTH EMAIL AND AUTH KEY 

Enter values "Auth Email", "Auth Key" and "Zone ID". 

CONFIGURE USING API TOKEN 

Enter values "API Token" and "Zone ID".

For the Cloudflare integration to be able to successfully get logs the following permissions must be granted to the API token - 

Logs 

Audit 

Audit logs summarize the history of changes made within your Cloudflare account. Audit logs include account-level actions like login and logout, as well as setting changes to DNS, Crypto, Firewall, Speed, Caching, Page Rules, Network, and Traffic features, etc. 

Logpull 

These logs contain data related to the connecting client, the request path through the Cloudflare network, and the response from the origin web server. 

Cloudflare Integration Procedures 

 Please provide the following information to CyTech: 

See the Screenshot Below 

 

 

 

 

 

Audit Logs 

Cloudflare Logs 

Enable Logpush to Amazon S3 

To enable the Cloudflare Logpush service: 

Once connected, Cloudflare lists Amazon S3 as a connected service under Logs > Logpush. Edit or remove connected services from here. 

Crowdstrike Integrations

Introduction 

This integration is for CrowdStrike products. It includes the following datasets for receiving logs: 

falcon dataset consists of endpoint data and Falcon platform audit data forwarded from Falcon SIEM Connector. 

fdr dataset consists of logs forwarded using the Falcon Data Replicator. 

Assumptions 

The procedures described in Section 3 assume that a Log Collector has already been setup.   

Compatibility 

This integration supports CrowdStrike Falcon SIEM-Connector-v2.0. 

Requirements 

Logs 

Falcon 

Contains endpoint data and CrowdStrike Falcon platform audit data forwarded from Falcon SIEM Connector. 

FDR 

The CrowdStrike Falcon Data Replicator (FDR) allows CrowdStrike users to replicate FDR data from CrowdStrike managed S3 buckets. CrowdStrike writes notification events to a CrowdStrike managed SQS queue when new data is available in S3. 

This integration can be used in two ways. It can consume SQS notifications directly from the CrowdStrike managed SQS queue or it can be used in conjunction with the FDR tool that replicates the data to a self-managed S3 bucket and the integration can read from there. 

In both cases SQS messages are deleted after they are processed. This allows you to operate more than one Elastic Agent with this integration if needed and not have duplicate events, but it means you cannot ingest the data a second time. 

CrowdStrike Integration Procedures 

Please provide the following information to CyTech: 

Collect CrowdStrike Falcon Data Replicator logs (input: aws-s3) Option 1 

Collect CrowdStrike logs via API. Option 2 (Recommended) 

Dropbox Integrations

Introduction 

Connecting Dropbox 

Use the Workplace Search Dropbox connector to automatically capture, sync and index the following items from your Dropbox service: 

Stored Files 

Including ID, File Metadata, File Content, Updated by, and timestamps. 

Dropbox Paper 

Including ID, Metadata, Content, Updated by, and timestamps. 

This document helps you configure and connect your Dropbox service to Workplace Search. To do this, you must register your Elastic deployment in the Dropbox Developer platform, by creating an OAuth 2.0 app. This gives your app permission to access Dropbox data. You will need your Workplace Search OAuth redirect URL, so have that handy. 

If you need a primer on OAuth, read the official OAuth 2.0 authorization flow RFC. The diagrams provide a good mental model of the protocol flow. Dropbox also has its own OAuth guide. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Requirements 

Configuring the Dropbox Connector 

To fetch document-level permissions, you must create an OAuth App using a team-owned (Dropbox Business) account. Enable document-level permissions by adding the following permissions: 

Your Dropbox service is now configured, and you can connect it to Workplace Search.  

 

Dropbox Integration Procedures 

Connecting Dropbox to Workplace Search 

Once the Dropbox connector is configured, you can connect a Dropbox instance to your organization’s Workplace Search deployment. 

Your Dropbox content becomes searchable as soon as syncing starts. Once configured and connected, Dropbox synchronizes automatically every 2 hours. 

Limiting the content to be indexed 

If you don’t need to index all available content, you can use the API to apply indexing rules. This reduces indexing load and overall index size. See Customizing indexing. 

The path_template and file_extension rules are applicable for Dropbox. 

Synchronized fields 

The following table lists the fields synchronized from the connected source to Workplace Search. The attributes in the table apply to the default search application, as follows: 

F5 Integrations

Introduction 

This document shows information related to F5 Integration.  

The F5 BIG-IP integration allows users to monitor LTM, AFM, APM, ASM, and AVR activity. F5 BIG-IP covers software and hardware designed around application availability, access control, and security solutions. 

 

The F5 BIG-IP integration can be used in three different modes to collect data: 

 

HTTP Endpoint mode - F5 BIG-IP pushes logs directly to an HTTP endpoint hosted by users’ Elastic Agent. 

AWS S3 polling mode - F5 BIG-IP writes data to S3 and Elastic Agent polls the S3 bucket by listing its contents and reading new files. 

AWS S3 SQS mode - F5 BIG-IP writes data to S3, S3 pushes a new object notification to SQS, Elastic Agent receives the notification from SQS, and then reads the S3 object. Multiple Agents can be used in this mode. 

For example, users can use the data from this integration to analyze the traffic that passes through their F5 BIG-IP network. 

 

Data streams 

The F5 BIG-IP integration collects one type of data stream: log. 

Log help users to keep a record of events happening on the network using telemetry streaming. The log data stream collected by the F5 BIG-IP integration includes events that are related to network traffic. See more details in the Logs. 

This integration targets the five types of events as mentioned below: 

LTM provides the platform for creating virtual servers, performance, service, protocol, authentication, and security profiles to define and shape users’ application traffic. For more information, refer to the link here. 

AFM is designed to reduce the hardware and extra hops required when ADC's are paired with traditional firewalls and helps to protect traffic destined for the user's data center. For more information, refer to the link here. 

APM provides federation, SSO, application access policies, and secure web tunneling and allows granular access to users' various applications, virtualized desktop environments, or just go full VPN tunnel. For more information, refer to the link here. 

ASM is F5's web application firewall (WAF) solution. It allows users to tailor acceptable and expected application behavior on a per-application basis. For more information, refer to the link here. 

AVR provides detailed charts and graphs to give users more insight into the performance of web applications, with detailed views on HTTP and TCP stats, as well as system performance (CPU, memory, etc.). For more information, refer to the link here. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Requirements 

Elasticsearch is needed to store and search data, and Kibana is needed for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your hardware. 

The reference link for requirements of telemetry streaming is here.  

The reference link for requirements of Application Services 3(AS3) Extension is here. 

This module has been tested against F5 BIG-IP version 16.1.0, Telemetry Streaming version 1.32.0 and AS3 version 3.40.0. 

 

Setup 

To collect LTM, AFM, APM, ASM, and AVR data from F5 BIG-IP, the user has to configure modules in F5 BIG-IP as per the requirements. 

To set up the F5 BIG-IP environment, users can use the BIG-IP system browser-based Configuration Utility or the command line tools that are provided. For more information related to the configuration of F5 BIG-IP servers, refer to F5 support website here. 

https://support.f5.com/csp/knowledge-center/software 

 

Configuration of Telemetry Streaming in F5 

For downloading and installing Telemetry Streaming, refer to the link here. 

https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/installation.html 

Telemetry Streaming will send logs in the JSON format to the destination. Telemetry Streaming is compatible with BIG-IP versions 13.0 and later. Users have to prepare F5 servers for it and set up the Telemetry Streaming Consumer.      

To use telemetry streaming, user have to send POST request on https://<BIG-IP>/mgmt/shared/telemetry/declare for declaration. 

F5 BIG-IP modules named LTM, AFM, ASM, and APM are not configured by Telemetry Streaming, they must be configured with AS3 or another method. Reference link for setup AS3 extension in F5 BIG-IP is here. 

To configure logging using AS3, refer to the link here. 

https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/event-listener.html?highlight=as3#configure-logging-using-as3 

To collect data from AWS S3 Bucket, follow the below steps: 

https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html 

To collect data from AWS SQS, follow the below steps: 

Note: 

Enabling the integration in Elastic 

F5 BIG-IP integration 

The "Integration name" and either the " Description" and the following will need to be provided in the Configure integration when adding the F5 BIG-IP integration.  

Procedures: 

Please provide the following information to CyTech: 

Collect F5 BIG-IP logs via HTTP Endpoint: 

F5 BIG-IP logs via HTTP Endpoint: 

Fortinet-Fortigate Integrations

Introduction 

This integration is for Fortinet FortiGate logs sent in the syslog format. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Compatibility

This integration has been tested against FortiOS version 6.0.x and 6.2.x. Versions above this are expected to work but have not been tested. 

Fortinet FortiGate Integration Procedures 

Please provide the following information to CyTech: 

Collect Fortinet FortiGate logs (input: tcp) 

Collect Fortinet FortiGate logs (input: udp) 

GCP Integrations

Introduction 

This document shows information related to GCP Integration.  

The Google Cloud integration collects and parses Google Cloud Audit Logs, VPC Flow Logs, Firewall Rules Logs and Cloud DNS Logs that have been exported from Cloud Logging to a Google Pub/Sub topic sink and collects Google Cloud metrics and metadata from Google Cloud Monitoring. 

Requirements 

To use this Google Cloud Platform (GCP) integration, the following needs to be set up: 

Service Accounts 

First, please create a Service Account. A Service Account (SA) is a particular type of Google account intended to represent a non-human user who needs to access the GCP resources. 

The Log Collector uses the SA (Service Account) to access data on Google Cloud Platform using the Google APIs. 

Here is a reference from Google related to Service Accounts: 

https://cloud.google.com/iam/docs/best-practices-for-securing-service-accounts 

Service Account with a Role  

You need to grant your Service Account (SA) access to Google Cloud Platform resources by assigning a role to the account. In order to assign minimal privileges, create a custom role that has only the privileges required by the Log Collector. Those privileges are below: 

*1 Only required if Agent is expected to create a new subscription. If you create the subscriptions yourself, you may omit these privileges.  

**2 Only required if corresponding collection will be enabled. 

After you have created the custom role, assign the role to your service account. 

Service Account Key 

Next, with the Service Account (SA) with access to Google Cloud Platform (GCP) resources setup in Section 3.1.1, you need some credentials to associate with it: a Service Account Key. 

From the list of SA (Service Accounts): 

 

GCP Integrations Procedures 

The audit dataset collects audit logs of administrative activities and accesses within your Google Cloud resources. 

 Procedures 

The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration. 

Please provide the following information to CyTech: 

The Project ID is the Google Cloud project ID where your resources exist. 

Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field. 

 

OPTION 1: CREDENTIALS FILE 

Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file. 

Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json. 

OPTION 2: CREDENTIALS JSON 

Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration. 

The dns dataset collects queries that name servers resolve for your Virtual Private Cloud (VPC) networks, as well as queries from an external entity directly to a public zone. 

Procedures 

The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration. 

Please provide the following information to CyTech: 

The Project ID is the Google Cloud project ID where your resources exist. 

Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field. 

 

OPTION 1: CREDENTIALS FILE 

Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file. 

Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json. 

OPTION 2: CREDENTIALS JSON 

Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration. 

The firewall dataset collects logs from Firewall Rules in your Virtual Private Cloud (VPC) networks. 

Procedures 

The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration. 

Please provide the following information to CyTech: 

The Project ID is the Google Cloud project ID where your resources exist. 

Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field. 

 

OPTION 1: CREDENTIALS FILE 

Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file. 

Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json. 

 

OPTION 2: CREDENTIALS JSON 

Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration. 

 

The vpcflow dataset collects logs sent from and received by VM instances, including instances used as GKE nodes. 

Procedures 

The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration. 

Please provide the following information to CyTech: 

The Project ID is the Google Cloud project ID where your resources exist. 

Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field. 

 

OPTION 1: CREDENTIALS FILE 

Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file. 

Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json. 

OPTION 2: CREDENTIALS JSON 

Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration. 

GitLab Integrations

Introduction 

Introduced in GitLab Starter 8.4. Support for Amazon Elasticsearch was introduced in GitLab Starter 9.0. 

This document describes how to set up Elasticsearch with GitLab. Once enabled, you'll have the benefit of fast search response times and the advantage of two special searches: 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Requirements 

Elasticsearch 6.0+ is not supported currently. We will support 6.0+ in the future.  

 

GitLabIntegration Procedures 

Procedures:  

Please provide the following information to CyTech:  

Elasticsearch is not included in the Omnibus packages. You will have to install it yourself whether you are using the Omnibus package or installed GitLab from source. Providing detailed information on installing Elasticsearch is out of the scope of this document. 

Once the data is added to the database or repository and Elasticsearch is enabled in the admin area the search index will be updated automatically. Elasticsearch can be installed on the same machine as GitLab, or on a separate server, or you can use the Amazon Elasticsearch service. 

You can follow the steps as described in the official web site or use the packages that are available for your OS. 

 

Google Workspace Integrations

Introduction 

Google Workspace (formerly G Suite) is a suite of cloud computing, productivity and collaboration tools, software and products developed and marketed by Google. It allows users to create, edit, and share documents, spreadsheets, presentations, and more. It also includes email, calendar, chat, and video conferencing tools. 

Google Workspace is designed for businesses of all sizes, from small businesses to large enterprises. It is a popular choice for businesses because it is affordable, easy to use, and secure. 

The Google Workspace integration collects and parses data from the different Google Workspace audit reports APIs. 

If you want to know more about how you can fully leverage the Google Workspace integration, there is a multipart blog from our Security Labs that will help you: 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

Compatibility 

It is compatible with a subset of applications under the Google Reports API v1. 

Requirements 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

In order to ingest data from the Google Reports API you must: 

Create access credentials 

Credentials are used to obtain an access token from Google's authorization servers so your app can call Google Workspace APIs. This guide describes how to choose and set up the credentials your app needs. 

Choose the access credential that is right for you 

The required credentials depends on the type of data, platform, and access methodology of your app. There are three types of credential types available: 

Assign a role to a service account: 

You must assign a prebuilt or custom role to a service account by a super administrator account. 

Create credentials for a service account: 

You need to obtain credentials in the form of a public/private key pair. These credentials are used by your code to authorize service account actions within your app. 

To obtain credentials for your service account: 

Your new public/private key pair is generated and downloaded to your machine as a new file. Save the downloaded JSON file as credentials.json in your working directory. This file is the only copy of this key. For information about how to store your key securely, see Managing service account keys. 

 

Optional: Set up domain-wide delegation for a service account 

To call APIs on behalf of users in a Google Workspace organization, your service account needs to be granted domain-wide delegation of authority in the Google Workspace Admin console by a super administrator account. For more information, see Delegating domain-wide authority to a service account. 

To set up domain-wide delegation of authority for a service account: 

If you don't have super administrator access to the relevant Google Workspace account, contact a super administrator for that account and send them your service account's Client ID and list of OAuth Scopes so they can complete the following steps in the Admin console. 

This integration will make use of the following oauth2 scope: 

Once you have downloaded your service account credentials as a JSON file, you are ready to set up your integration. 

Click the Advanced option of Google Workspace Audit Reports. The default value of "API Host" is https://www.googleapis.com. The API Host will be used for collecting access_transparency, admin, device, context_aware_access, drive, gcp, groups, group_enterprise, login, rules, saml, token and user accounts logs. 

NOTE: The Delegated Account value in the configuration, is expected to be the email of the administrator account, and not the email of the ServiceAccount. 

Google Workspace Integration Procedures 

Please provide the following information to CyTech: 

Collect access_transparency, admin, alert, context_aware_access, device, drive, gcp, groups, group_enterprise, login, rules, saml, token and user accounts logs (input: httpjson). 

Jumpcloud Integrations

Introduction 

The JumpCloud integration allows you to monitor events related to the JumpCloud Directory as a Service via the Directory Insights API. 

You can find out more about JumpCloud and JumpCloud Directory Insights here 

Data streams 

A single data stream named "jumpcloud.events" is used by this integration. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Requirements 

An Elastic Stack with an Elastic Agent is a fundamental requirement. 

An established JumpCloud tenancy with active users is the the other requirement. Basic Directory Insights API access is available to all subscription levels. 

NOTE: The lowest level of subscription currently has retention limits, with access to Directory Insights events for the last 15 days at most. Other subscriptions levels provide 90 days or longer historical event access. 

A JumpCloud API key is required, the JumpCloud documentation describing how to create one is here 

This JumpCloud Directory Insights API is documented here 

JumpCloud Integration Procedures 

Procedures:  

Please provide the following information to CyTech:  

Enabling the integration in Elastic 

 

Mimecast Integrations

Introduction 

The Mimecast integration collects events from the Mimecast API. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

Requirements 

Configuration 

Authorization parameters for the Mimecast API (Application Key, Application ID, Access Key, and Secret Key) should be provided by a Mimecast representative for this integration. Under Advanced options you can set the time interval between two API requests as well as the API URL. A Mimecast representative should also be able to give you this information in case you need to change the defaults. 

Note: Rate limit quotas may require you to set up different credentials for the different available log types. 

Logs 

Audit Events 

This is the mimecast.audit_events dataset. These logs contain Mimecast audit events with the following details: audit type, event category and detailed information about the event. More information about these logs. 

DLP Logs 

This is the mimecast.dlp_logs dataset. These logs contain information about messages that triggered a DLP or Content Examination policy. More information about these logs. 

SIEM Logs 

This is the mimecast.siem_logs dataset. These logs contain information about messages that contains MTA (message transfer agent) log – all inbound, outbound, and internal messages. 

Threat Intel Feed Malware: Customer 

This is the mimecast.threat_intel_malware_customer dataset. These logs contain information about messages that return identified malware threats at a customer level. Learn more about these logs. 

Threat Intel Feed Malware: Grid 

This is the mimecast.threat_intel_malware_grid dataset. These logs contain information about messages that return identified malware threats at a regional grid level. More about these logs. 

TTP Attachment Logs 

This is the mimecast.ttp_ap_logs dataset. These logs contain Mimecast TTP attachment protection logs with the following details: result of attachment analysis (if it is malicious or not etc.), date when file is released, sender and recipient address, filename and type, action triggered for the attachment, the route of the original email containing the attachment and details. Learn more about these logs. 

TTP Impersonation Logs 

This is the mimecast.ttp_ip_logs dataset. These logs contain information about messages containing information flagged by an Impersonation Protection configuration. 

TTP URL Logs 

This is the mimecast.ttp_url_logs dataset. These logs contain Mimecast TTP attachment protection logs with the following details: the category of the URL clicked, the email address of the user who clicked the link, the url clicked, the action taken by the user if user awareness was applied, the route of the email that contained the link, the action defined by the administrator for the URL, the date that the URL was clicked, url scan result, the action that was taken for the click, the description of the definition that triggered the URL to be rewritten by Mimecast, the action requested by the user, an array of components of the message where the URL was found. More about these logs. 

 

Mimecast Integration Procedures 

Please provide the following information to CyTech: 

Mimecast API 

 

MongoDB Integrations

Introduction 

This integration is used to fetch logs and metrics from MongoDB. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

Compatibility 

The log dataset is tested with logs from versions v3.2.11 and v4.4.4 in plaintext and json formats. The collstats, dbstats, metrics, replstatus and status datasets are tested with MongoDB 3.4 and 3.0 and are expected to work with all versions >= 2.8. 

Requirements 

MongoDB Privileges 

In order to use the metrics datasets, the MongoDB user specified in the package configuration needs to have certain privileges. 

We recommend using the clusterMonitor role to cover all the necessary privileges. 

You can use the following command in Mongo shell to create the privileged user (make sure you are using the admin db by using db command in Mongo shell). 

 

db.createUser( 

    { 

        user: "beats", 

        pwd: "pass", 

        roles: ["clusterMonitor"] 

    } 

) 

 

You can use the following command in Mongo shell to grant the role to an existing user (make sure you are using the admin db by using db command in Mongo shell). 

db.grantRolesToUser("user", ["clusterMonitor"]) 

Logs 

log 

The log dataset collects the MongoDB logs. 

 

Metrics 

collstats 

The collstats dataset uses the top administrative command to return usage statistics for each collection. It provides the amount of time, in microseconds, used and a count of operations for the following types: total, readLock, writeLock, queries, getmore, insert, update, remove, and commands. 

It requires the following privileges, which is covered by the clusterMonitor role: 

dbstats 

The dbstats dataset collects storage statistics for a given database. 

It requires the following privileges, which is covered by the clusterMonitor role: 

action on cluster resource 

action on the database resource 

metrics 

It requires the following privileges, which is covered by the clusterMonitor role: 

action on cluster resource 

replstatus 

The replstatus dataset collects status of the replica set. It requires the following privileges, which is covered by the clusterMonitor role: 

status 

The status returns a document that provides an overview of the database's state. 

It requires the following privileges, which is covered by the clusterMonitor role: 

action on cluster resource 

 

MongoDB Integration Procedures 

Please provide the following information to CyTech: 

Collect MongoDB application logs 

Collect MongoDB metrics 

Ex: localhost:27017 

OKTA Integrations

Introduction 

The Okta integration collects events from the Okta API, specifically reading from the Okta System Log API. 

Logs 

System 

The Okta System Log records system events related to your organization in order to provide an audit trail that can be used to understand platform activity and to diagnose problems. This module is implemented using the httpjson input and is configured to paginate through the logs while honoring any rate-limiting headers sent by Okta. 

Okta Integration Procedures 

Please provide the following information to CyTech: 

Collect Okta logs via API 

Pulse Connect Secure Integrations

Introduction 

This integration is for Pulse Connect Secure. 

Pulse Connect Secure Integration Procedures 

Please provide the following information to CyTech: 

Collect Pulse Connect Secure logs (input: udp) 

Collect Pulse Connect Secure logs (input: tcp) 

 

 

 

 

Slack Integrations

Introduction 

Slack is used by numerous organizations as their primary chat and collaboration tool. 

Please note the Audit Logs API is only available to Slack workspaces on an Enterprise Grid plan. These API methods will not work for workspaces on a Free, Standard, or Business+ plan. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Requirements 

Configuration 

Enabling the integration in Elastic 

Configure Slack audit logs data stream 

Enter values "OAuth API Token". 

CONFIGURE USING API TOKEN 

For the Slack integration to be able to successfully get logs the following "User Token Scopes"" must be granted to the Slack App: 

Logs 

Audit 

Audit logs summarize the history of changes made within the Slack Enterprise. 

 

SLACK Integration Procedures 

Please provide the following information to CyTech: 

Collect Slack logs via API 

Slack Audit logs 

System Integrations

Introduction 

The System integration allows you to monitor servers, personal computers, and more. 

Use the System integration to collect metrics and logs from your machines. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue. 

For example, if you wanted to be notified when less than 10% of the disk space is still available, you could install the System integration to send file system metrics to Elastic. Then, you could view real-time updates to disk space used on your system in Kibana's [Metrics System] Overview dashboard. You could also set up a new rule in the Elastic Observability Metrics app to alert you when the percent free is less than 10% of the total disk space. 

Data streams 

The System integration collects two types of data: logs and metrics. 

Logs help you keep a record of events that happen on your machine. Log data streams collected by the System integration include application, system, and security events on machines running Windows and auth and syslog events on machines running macOS or Linux. See more details in the Logs reference. 

Metrics give you insight into the state of the machine. Metric data streams collected by the System integration include CPU usage, load statistics, memory usage, information on network behavior, and more. See more details in the Metrics reference. 

You can enable and disable individual data streams. If all data streams are disabled and the System integration is still enabled, Fleet uses the default data streams. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

Requirements 

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware. 

Each data stream collects different kinds of metric data, which may require dedicated permissions to be fetched and which may vary across operating systems. Details on the permissions needed for each data stream are available in the Metrics reference. 

Setup 

For step-by-step instructions on how to set up an integration, see the Getting started guide. 

Troubleshooting 

Note that certain data streams may access /proc to gather process information, and the resulting ptrace_may_access() call by the kernel to check for permissions can be blocked by AppArmor and other LSM software, even though the System module doesn't use ptrace directly. 

In addition, when running inside a container the proc filesystem directory of the host should be set using system.hostfs setting to /hostfs. 

Logs reference 

Application 

The Windows application data stream provides events from the Windows Application event log. 

SUPPORTED OPERATING SYSTEMS 

System Integration Procedures 

Please provide the following information to CyTech:  

Collect logs from System instances (Enable Yes/No)  

System auth logs (log) (Enable Yes/No) 

System syslog logs (log) 

Collect events from the Windows event log (Enable Yes/No) 

Processors 

Security (Enable Yes/No) 

 

System (Enable Yes/No) 

Collect metrics from System instances (Enable Yes/No) 

System CPU metrics (Enable Yes/No) 

System diskio metrics (Enable Yes/No) 

System filesystem metrics 

System fsstat metrics (Enable Yes/No) 

System load metrics (Enable Yes/No) 

System memory metrics (Enable Yes/No) 

System network metrics (Enable Yes/No) 

System process metrics 

System process_summary metrics (Enable Yes/No) 

System socket_summary metrics (Enable Yes/No) 

System uptime metrics (Enable Yes/No) 

Collect logs from third-party REST API (experimental) (Enable Yes/No) 

Windows Application Events via Splunk Enterprise REST API (Enable Yes/No) 

Windows Security Events via Splunk Enterprise REST API (Enable Yes/No) 

Windows System Events via Splunk Enterprise REST API (Enable Yes/No) 

Team Viewer Integrations

Remote File Copy via TeamViewer  

Identifies an executable or script file remotely downloaded via a TeamViewer transfer session. 

Rule type: eql  

Rule indices:  

Severity: medium  

Risk score: 47  

Runs every: 5m  

Searches indices from: now-9m (Date Math format, see also Additional look-back time)  

Maximum alerts per execution: 100  

References:  

Tags:  

Version: 5  

Rule authors:  

Rule license: Elastic License v2 

Rule query 

file where event.type == "creation" and process.name : "TeamViewer.exe" and  

  file.extension : ("exe", "dll", "scr", "com", "bat", "ps1", "vbs", "vbe", "js", "wsh", "hta")  

Framework: MITRE ATT&CKTM 

 Source: https://www.elastic.co/guide/en/security/master/prebuilt-rule-0-14-2-remote-file-copy-via-teamviewer.html  

TeamViewer Integration Procedure 

 

Copy and paste the following Logstash configuration into the file:  

 

 
 

 

 

Source: ChatGPT  

Z Scaler Integrations

Introduction 

This integration is for Zscaler Internet Access logs. It can be used to receive logs sent by NSS log server on respective TCP ports. 

The log message is expected to be in JSON format. The data is mapped to ECS fields where applicable and the remaining fields are written under zscaler_zia.<data-stream-name>.*. 

 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Compatibility 

This package has been tested against Zscaler Internet Access version 6.1 

 

Requirements 

Steps for setting up NSS Feeds 

 

 

 

 

Steps for setting up Cloud NSS Feeds 

 

 

Please make sure to use the given response formats for NSS and Cloud NSS Feeds. 

Note: Please make sure to use latest version of given response formats. 

 

 

Zscaler Integration Procedures 

Please provide the following information to CyTech: 

Collect Zscaler Internet Access logs via TCP input 

Collect Zscaler Internet Access logs via HTTP Endpoint 

gcp

Google Cloud Platform

Google Cloud Platform Integration

The Google Cloud integration collects and parses Google Cloud Audit Logs, VPC Flow Logs, Firewall Rules Logs and Cloud DNS Logs that have been exported from Cloud Logging to a Google Pub/Sub topic sink and collects Google Cloud metrics and metadata from Google Cloud Monitoring.

Authentication

To use this Google Cloud Platform (GCP) integration, you need to set up a Service Account with a Role and a Service Account Key to access data on your GCP project.

Service Account

First, you need to create a Service Account. A Service Account (SA) is a particular type of Google account intended to represent a non-human user who needs to access the GCP resources.

The Elastic Agent uses the SA to access data on Google Cloud Platform using the Google APIs.

If you haven't already, this might be a good moment to check out the best practices for securing service accounts guide.

Role

You need to grant your Service Account (SA) access to Google Cloud Platform resources by assigning a role to the account. In order to assign minimal privileges, create a custom role that has only the privileges required by Agent. Those privileges are:

* Only required if Agent is expected to create a new subscription. If you create the subscriptions yourself you may omit these privileges. ** Only required if corresponding collection will be enabled.

After you have created the custom role, assign the role to your service account.

Service Account Keys

Now, with your brand new Service Account (SA) with access to Google Cloud Platform (GCP) resources, you need some credentials to associate with it: a Service Account Key.

From the list of SA:

Configure the Integration Settings

The next step is to configure the general integration settings used for all logs from the supported services (Audit, DNS, Firewall, and VPC Flow).

The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration.

Project Id

The Project Id is the Google Cloud project ID where your resources exist.

Credentials File vs Json

Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field.

Option 1: Credentials File

Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file.

Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json.

Option 2: Credentials JSON

Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration.

Recommendations

Elastic recommends using Credentials File, as in this method the credential information doesn’t leave your Google Cloud Platform environment. When using Credentials JSON, the integration stores the info in Elasticsearch, and the access is controlled based on policy permissions or access to underlying Elasticsearch data.

Logs Collection Configuration

With a properly configured Service Account and the integration setting in place, it's time to start collecting some logs.

Requirements

You need to create a few dedicated Google Cloud resources before starting, in detail:

Elastic recommends separate Pub/Sub topics for each of the log types so that they can be parsed and stored in a specific data stream.

Here's an example of collecting Audit Logs using a Pub/Sub topic, a subscription, and a Log Router. We will create the resources in the Google Cloud Console and then configure the Google Cloud Platform integration.

On the Google Cloud Console

At a high level, the steps required are:

This is just an example; you will need to create your filter expression to select the log types you want to export to the Pub/Sub topic.

More example filters for different log types:

#
# VPC Flow: logs for specific subnet
#
resource.type="gce_subnetwork" AND
log_id("compute.googleapis.com/vpc_flows") AND
resource.labels.subnetwork_name"=[SUBNET_NAME]"
#
# Audit: Google Compute Engine firewall rule deletion
#
resource.type="gce_firewall_rule" AND
log_id("cloudaudit.googleapis.com/activity") AND
protoPayload.methodName:"firewalls.delete"
#
# DNS: all DNS queries
#
resource.type="dns_query"
#
# Firewall: logs for a given country
#
resource.type="gce_subnetwork" AND
log_id("compute.googleapis.com/firewall") AND
jsonPayload.remote_location.country=[COUNTRY_ISO_ALPHA_3]

Start working on your query using the Google Cloud Logs Explorer, so you can preview and pinpoint the exact log types you want to forward to your Elastic Stack.

To learn more, please read how to Build queries in the Logs Explorer, and take a look at the Sample queries using the Logs Explorer page in the Google Cloud docs.

On Kibana

Visit "Management" > "Integrations" > "Installed Integrations" > "Google Cloud Platform" and select the "Integration Policies" tab. Select the integration policy you previously created.

From the list of services, select "Google Cloud Platform (GCP) audit logs (gcp-pubsub)" and:

Troubleshooting

If you don't see Audit logs showing up, check the Agent logs to see if there are errors.

Common error types:

Missing Roles in the Service Account

If your Service Account (SA) does not have the required roles, you might find errors like this one in the elastic_agent.filebeat dataset:

failed to subscribe to pub/sub topic: failed to check if subscription exists: rpc error: code = PermissionDenied desc = User not authorized to perform this action.

Solution: make sure your SA has all the required roles.

Misconfigured Settings

If you specify the wrong "Topic field" or "Subscription Name", you might find errors like this one in the elastic_agent.filebeat dataset:

[elastic_agent.filebeat][error] failed to subscribe to pub/sub topic: failed to check if subscription exists: rpc error: code = InvalidArgument desc = Invalid resource name given (name=projects/project/subscriptions/projects/project/subscriptions/non-existent-sub). Refer to https://cloud.google.com/pubsub/docs/admin#resource_names for more information.

Solution: double check the integration settings.

Metrics Collection Configuration

With a properly configured Service Account and the integration setting in place, it's time to start collecting some metrics.

Requirements

No additional requirement is needed to collect metrics.

Troubleshooting

If you don't see metrics showing up, check the Agent logs to see if there are errors.

Common error types:

Period is lower than 60 seconds

Usual minimum collection period for GCP metrics is 60 seconds. Any value lower than that cause an error when retrieving the metric metadata. If an error happens, the affected metric is skipped at the metric collection stage, resulting in no data being sent.

Missing Roles in the Service Account

If your Service Account (SA) does not have required roles, you might find errors related to accessing GCP resources.

To check you may add Monitoring Viewer and Compute Viewer roles (built-in GCP roles) to your SA. These roles contain the permission added in the previous step and expand them with additional permissions. You can analyze additional missing permissions from the GCP Console > IAM > clicking on the down arrow near the roles on the same line of your SA > View analyzed permissions. From the shown table you can check which permissions from the role the SA is actively using. They should match what you configured in your custom role.

Misconfigured Settings

If you specify a wrong setting you will probably find errors related to missing GCP resources.

Make sure the settings are correct and the SA has proper permissions for the given "Project Id".

Logs

Audit

The audit dataset collects audit logs of administrative activities and accesses within your Google Cloud resources.

Exported fields

An example event for audit looks as following:

{
    "@timestamp": "2019-12-19T00:44:25.051Z",
    "agent": {
        "ephemeral_id": "a22278bb-5e1f-4ab7-b468-277c8c0b80a9",
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.7.1"
    },
    "client": {
        "user": {
            "email": "xxx@xxx.xxx"
        }
    },
    "cloud": {
        "project": {
            "id": "elastic-beats"
        },
        "provider": "gcp"
    },
    "data_stream": {
        "dataset": "gcp.audit",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.8.0"
    },
    "elastic_agent": {
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "snapshot": false,
        "version": "8.7.1"
    },
    "event": {
        "action": "beta.compute.instances.aggregatedList",
        "agent_id_status": "verified",
        "category": [
            "network",
            "configuration"
        ],
        "created": "2023-10-25T04:18:46.637Z",
        "dataset": "gcp.audit",
        "id": "yonau2dg2zi",
        "ingested": "2023-10-25T04:18:47Z",
        "kind": "event",
        "outcome": "success",
        "provider": "data_access",
        "type": [
            "access",
            "allowed"
        ]
    },
    "gcp": {
        "audit": {
            "authorization_info": [
                {
                    "granted": true,
                    "permission": "compute.instances.list",
                    "resource_attributes": {
                        "name": "projects/elastic-beats",
                        "service": "resourcemanager",
                        "type": "resourcemanager.projects"
                    }
                }
            ],
            "num_response_items": 61,
            "request": {
                "@type": "type.googleapis.com/compute.instances.aggregatedList"
            },
            "resource_location": {
                "current_locations": [
                    "global"
                ]
            },
            "resource_name": "projects/elastic-beats/global/instances",
            "response": {
                "@type": "core.k8s.io/v1.Status",
                "apiVersion": "v1",
                "details": {
                    "group": "batch",
                    "kind": "jobs",
                    "name": "gsuite-exporter-1589294700",
                    "uid": "2beff34a-945f-11ea-bacf-42010a80007f"
                },
                "kind": "Status",
                "status_value": "Success"
            },
            "type": "type.googleapis.com/google.cloud.audit.AuditLog"
        }
    },
    "input": {
        "type": "gcp-pubsub"
    },
    "log": {
        "level": "INFO",
        "logger": "projects/elastic-beats/logs/cloudaudit.googleapis.com%2Fdata_access"
    },
    "service": {
        "name": "compute.googleapis.com"
    },
    "source": {
        "ip": "192.168.1.1"
    },
    "tags": [
        "forwarded",
        "gcp-audit"
    ],
    "user_agent": {
        "device": {
            "name": "Mac"
        },
        "name": "Firefox",
        "original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:71.0) Gecko/20100101 Firefox/71.0,gzip(gfe),gzip(gfe)",
        "os": {
            "full": "Mac OS X 10.15",
            "name": "Mac OS X",
            "version": "10.15"
        },
        "version": "71.0."
    }
}

Firewall

The firewall dataset collects logs from Firewall Rules in your Virtual Private Cloud (VPC) networks.

Exported fields

An example event for firewall looks as following:

{
    "@timestamp": "2019-10-30T13:52:42.191Z",
    "agent": {
        "ephemeral_id": "175ae0b3-355c-4ca7-87ea-d5f1ee34102e",
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.7.1"
    },
    "cloud": {
        "availability_zone": "us-east1-b",
        "project": {
            "id": "test-beats"
        },
        "provider": "gcp",
        "region": "us-east1"
    },
    "data_stream": {
        "dataset": "gcp.firewall",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "address": "10.42.0.2",
        "domain": "test-windows",
        "ip": "10.42.0.2",
        "port": 3389
    },
    "ecs": {
        "version": "8.8.0"
    },
    "elastic_agent": {
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "snapshot": false,
        "version": "8.7.1"
    },
    "event": {
        "action": "firewall-rule",
        "agent_id_status": "verified",
        "category": [
            "network"
        ],
        "created": "2023-10-25T04:20:37.182Z",
        "dataset": "gcp.firewall",
        "id": "1f21ciqfpfssuo",
        "ingested": "2023-10-25T04:20:41Z",
        "kind": "event",
        "type": [
            "allowed",
            "connection"
        ]
    },
    "gcp": {
        "destination": {
            "instance": {
                "project_id": "test-beats",
                "region": "us-east1",
                "zone": "us-east1-b"
            },
            "vpc": {
                "project_id": "test-beats",
                "subnetwork_name": "windows-isolated",
                "vpc_name": "windows-isolated"
            }
        },
        "firewall": {
            "rule_details": {
                "action": "ALLOW",
                "direction": "INGRESS",
                "ip_port_info": [
                    {
                        "ip_protocol": "TCP",
                        "port_range": [
                            "3389"
                        ]
                    }
                ],
                "priority": 1000,
                "source_range": [
                    "0.0.0.0/0"
                ],
                "target_tag": [
                    "allow-rdp"
                ]
            }
        }
    },
    "input": {
        "type": "gcp-pubsub"
    },
    "log": {
        "logger": "projects/test-beats/logs/compute.googleapis.com%2Ffirewall"
    },
    "network": {
        "community_id": "1:OdLB9eXsBDLz8m97ao4LepX6q+4=",
        "direction": "inbound",
        "iana_number": "6",
        "name": "windows-isolated",
        "transport": "tcp",
        "type": "ipv4"
    },
    "related": {
        "ip": [
            "192.168.2.126",
            "10.42.0.2"
        ]
    },
    "rule": {
        "name": "network:windows-isolated/firewall:windows-isolated-allow-rdp"
    },
    "source": {
        "address": "192.168.2.126",
        "geo": {
            "continent_name": "Asia",
            "country_name": "omn"
        },
        "ip": "192.168.2.126",
        "port": 64853
    },
    "tags": [
        "forwarded",
        "gcp-firewall"
    ]
}

VPC Flow

The vpcflow dataset collects logs sent from and received by VM instances, including instances used as GKE nodes.

Exported fields

An example event for vpcflow looks as following:

{
    "@timestamp": "2019-06-14T03:50:10.845Z",
    "agent": {
        "ephemeral_id": "0b8165a2-0e25-4e9a-bb68-271697e0993f",
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.7.1"
    },
    "cloud": {
        "availability_zone": "us-east1-b",
        "instance": {
            "name": "kibana"
        },
        "project": {
            "id": "my-sample-project"
        },
        "provider": "gcp",
        "region": "us-east1"
    },
    "data_stream": {
        "dataset": "gcp.vpcflow",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "address": "10.139.99.242",
        "domain": "elasticsearch",
        "ip": "10.139.99.242",
        "port": 9200
    },
    "ecs": {
        "version": "8.8.0"
    },
    "elastic_agent": {
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "snapshot": false,
        "version": "8.7.1"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "network"
        ],
        "created": "2023-10-25T04:21:42.006Z",
        "dataset": "gcp.vpcflow",
        "end": "2019-06-14T03:49:51.821056075Z",
        "id": "ut8lbrffooxz5",
        "ingested": "2023-10-25T04:21:43Z",
        "kind": "event",
        "start": "2019-06-14T03:40:20.510622432Z",
        "type": [
            "connection"
        ]
    },
    "gcp": {
        "destination": {
            "instance": {
                "project_id": "my-sample-project",
                "region": "us-east1",
                "zone": "us-east1-b"
            },
            "vpc": {
                "project_id": "my-sample-project",
                "subnetwork_name": "default",
                "vpc_name": "default"
            }
        },
        "source": {
            "instance": {
                "project_id": "my-sample-project",
                "region": "us-east1",
                "zone": "us-east1-b"
            },
            "vpc": {
                "project_id": "my-sample-project",
                "subnetwork_name": "default",
                "vpc_name": "default"
            }
        },
        "vpcflow": {
            "reporter": "DEST",
            "rtt": {
                "ms": 201
            }
        }
    },
    "input": {
        "type": "gcp-pubsub"
    },
    "log": {
        "logger": "projects/my-sample-project/logs/compute.googleapis.com%2Fvpc_flows"
    },
    "network": {
        "bytes": 11773,
        "community_id": "1:FYaJFSEAKLcBCMFoT6sR5TMHf/s=",
        "direction": "internal",
        "iana_number": "6",
        "name": "default",
        "packets": 94,
        "transport": "tcp",
        "type": "ipv4"
    },
    "related": {
        "ip": [
            "67.43.156.13",
            "10.139.99.242"
        ]
    },
    "source": {
        "address": "67.43.156.13",
        "as": {
            "number": 35908
        },
        "bytes": 11773,
        "domain": "kibana",
        "geo": {
            "continent_name": "Asia",
            "country_iso_code": "BT",
            "country_name": "Bhutan",
            "location": {
                "lat": 27.5,
                "lon": 90.5
            }
        },
        "ip": "67.43.156.13",
        "packets": 94,
        "port": 33576
    },
    "tags": [
        "forwarded",
        "gcp-vpcflow"
    ]
}

DNS

The dns dataset collects queries that name servers resolve for your Virtual Private Cloud (VPC) networks, as well as queries from an external entity directly to a public zone.

Exported fields

An example event for dns looks as following:

{
    "@timestamp": "2021-12-12T15:59:40.446Z",
    "agent": {
        "ephemeral_id": "fd6c4189-cbc6-493a-acfb-c9e7b2b7588c",
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.7.1"
    },
    "cloud": {
        "project": {
            "id": "key-reference-123456"
        },
        "provider": "gcp",
        "region": "global"
    },
    "data_stream": {
        "dataset": "gcp.dns",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "address": "216.239.32.106",
        "ip": "216.239.32.106"
    },
    "dns": {
        "answers": [
            {
                "class": "IN",
                "data": "67.43.156.13",
                "name": "asdf.gcp.example.com.",
                "ttl": 300,
                "type": "A"
            }
        ],
        "question": {
            "name": "asdf.gcp.example.com",
            "registered_domain": "example.com",
            "subdomain": "asdf.gcp",
            "top_level_domain": "com",
            "type": "A"
        },
        "resolved_ip": [
            "67.43.156.13"
        ],
        "response_code": "NOERROR"
    },
    "ecs": {
        "version": "8.8.0"
    },
    "elastic_agent": {
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "snapshot": false,
        "version": "8.7.1"
    },
    "event": {
        "action": "dns-query",
        "agent_id_status": "verified",
        "category": "network",
        "created": "2023-10-25T04:19:40.300Z",
        "dataset": "gcp.dns",
        "id": "zir4wud11tm",
        "ingested": "2023-10-25T04:19:41Z",
        "kind": "event",
        "outcome": "success"
    },
    "gcp": {
        "dns": {
            "auth_answer": true,
            "destination_ip": "216.239.32.106",
            "protocol": "UDP",
            "query_name": "asdf.gcp.example.com.",
            "query_type": "A",
            "response_code": "NOERROR",
            "server_latency": 0,
            "source_type": "internet",
            "target_type": "public-zone"
        }
    },
    "input": {
        "type": "gcp-pubsub"
    },
    "log": {
        "level": "INFO",
        "logger": "projects/key-reference-123456/logs/dns.googleapis.com%2Fdns_queries"
    },
    "network": {
        "iana_number": "17",
        "protocol": "dns",
        "transport": "udp"
    },
    "related": {
        "hosts": [
            "asdf.gcp.example.com"
        ],
        "ip": [
            "67.43.156.13",
            "216.239.32.106"
        ]
    },
    "tags": [
        "forwarded",
        "gcp-dns"
    ]
}

Loadbalancing Logs

The loadbalancing_logs dataset collects logs of the requests sent to and handled by GCP Load Balancers.

Exported fields

An example event for loadbalancing looks as following:

{
    "@timestamp": "2020-06-08T23:41:30.078Z",
    "agent": {
        "ephemeral_id": "f4dde373-2ff7-464b-afdb-da94763f219b",
        "id": "5d3eee86-91a9-4afa-af92-c6b79bd866c0",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.6.0"
    },
    "cloud": {
        "project": {
            "id": "PROJECT_ID"
        },
        "region": "global"
    },
    "data_stream": {
        "dataset": "gcp.loadbalancing_logs",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "address": "81.2.69.193",
        "ip": "81.2.69.193",
        "nat": {
            "ip": "10.5.3.1",
            "port": 9090
        },
        "port": 8080
    },
    "ecs": {
        "version": "8.8.0"
    },
    "elastic_agent": {
        "id": "5d3eee86-91a9-4afa-af92-c6b79bd866c0",
        "snapshot": true,
        "version": "8.6.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": "network",
        "created": "2020-06-08T23:41:30.588Z",
        "dataset": "gcp.loadbalancing_logs",
        "id": "1oek5rg3l3fxj7",
        "ingested": "2023-01-13T15:02:22Z",
        "kind": "event",
        "type": "info"
    },
    "gcp": {
        "load_balancer": {
            "backend_service_name": "",
            "cache_hit": true,
            "cache_id": "SFO-fbae48ad",
            "cache_lookup": true,
            "forwarding_rule_name": "FORWARDING_RULE_NAME",
            "status_details": "response_from_cache",
            "target_proxy_name": "TARGET_PROXY_NAME",
            "url_map_name": "URL_MAP_NAME"
        }
    },
    "http": {
        "request": {
            "bytes": 577,
            "method": "GET",
            "referrer": "https://developer.mozilla.org/en-US/docs/Web/JavaScript"
        },
        "response": {
            "bytes": 157,
            "status_code": 304
        },
        "version": "2.0"
    },
    "input": {
        "type": "gcp-pubsub"
    },
    "log": {
        "level": "INFO",
        "logger": "projects/PROJECT_ID/logs/requests"
    },
    "network": {
        "protocol": "http"
    },
    "related": {
        "ip": [
            "89.160.20.156",
            "81.2.69.193",
            "10.5.3.1"
        ]
    },
    "source": {
        "address": "89.160.20.156",
        "as": {
            "number": 29518,
            "organization": {
                "name": "Bredband2 AB"
            }
        },
        "geo": {
            "city_name": "Linköping",
            "continent_name": "Europe",
            "country_iso_code": "SE",
            "country_name": "Sweden",
            "location": {
                "lat": 58.4167,
                "lon": 15.6167
            },
            "region_iso_code": "SE-E",
            "region_name": "Östergötland County"
        },
        "ip": "89.160.20.156",
        "port": 9989
    },
    "tags": [
        "forwarded",
        "gcp-loadbalancing_logs"
    ],
    "url": {
        "domain": "81.2.69.193",
        "extension": "jpg",
        "original": "http://81.2.69.193:8080/static/us/three-cats.jpg",
        "path": "/static/us/three-cats.jpg",
        "port": 8080,
        "scheme": "http"
    },
    "user_agent": {
        "device": {
            "name": "Mac"
        },
        "name": "Chrome",
        "original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36",
        "os": {
            "full": "Mac OS X 10.14.6",
            "name": "Mac OS X",
            "version": "10.14.6"
        },
        "version": "83.0.4103.61"
    }
}

Metrics

Billing

The billing dataset collects GCP Billing information from Google Cloud BigQuery daily cost detail table.

Exported fields

An example event for billing looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "01475F-5B1080-1137E7"
        },
        "project": {
            "id": "elastic-bi",
            "name": "elastic-containerlib-prod"
        },
        "provider": "gcp"
    },
    "event": {
        "dataset": "gcp.billing",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "billing": {
            "billing_account_id": "01475F-5B1080-1137E7",
            "cost_type": "regular",
            "invoice_month": "202106",
            "project_id": "containerlib-prod-12763",
            "project_name": "elastic-containerlib-prod",
            "total": 4717.170681,
            "sku_id": "0D56-2F80-52A5",
            "service_id": "6F81-5844-456A",
            "sku_description": "Network Inter Region Ingress from Jakarta to Americas",
            "service_description": "Compute Engine",
            "effective_price": 0.00292353,
            "tags": [
                {
                    "key": "stage",
                    "value": "prod"
                },
                {
                    "key": "size",
                    "value": "standard"
                }
            ]
        }
    },
    "metricset": {
        "name": "billing",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

Compute

The compute dataset is designed to fetch metrics for Compute Engine Virtual Machines in Google Cloud Platform.

Exported fields

An example event for compute looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.compute",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "compute": {
            "firewall": {
                "dropped": {
                    "bytes": 421
                },
                "dropped_packets_count": {
                    "value": 4
                }
            },
            "instance": {
                "cpu": {
                    "reserved_cores": {
                        "value": 1
                    },
                    "usage": {
                        "pct": 0.07259952346383708
                    },
                    "usage_time": {
                        "sec": 4.355971407830225
                    }
                },
                "memory": {
                    "balloon": {
                        "ram_size": {
                            "value": 4128378880
                        },
                        "ram_used": {
                            "value": 2190848000
                        },
                        "swap_in": {
                            "bytes": 0
                        },
                        "swap_out": {
                            "bytes": 0
                        }
                    }
                },
                "uptime": {
                    "sec": 60.00000000000091
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "compute",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

Dataproc

The dataproc dataset is designed to fetch metrics from Dataproc in Google Cloud Platform.

Exported fields

An example event for dataproc looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.dataproc",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "dataproc": {
            "cluster": {
                "hdfs": {
                    "datanodes": {
                        "count": 15
                    }
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "dataproc",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

Firestore

The firestore dataset fetches metrics from Firestore in Google Cloud Platform.

Exported fields

An example event for firestore looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.firestore",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "firestore": {
            "document": {
                "delete": {
                    "count": 3
                },
                "read": {
                    "count": 10
                },
                "write": {
                    "count": 1
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "firestore",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

GKE

The gke dataset is designed to fetch metrics from GKE in Google Cloud Platform.

Exported fields

An example event for gke looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.gke",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "gke": {
            "container": {
                "cpu": {
                    "core_usage_time": {
                        "sec": 15
                    }
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "gke",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

Loadbalancing Metrics

The loadbalancing_metrics dataset is designed to fetch HTTPS, HTTP, and Layer 3 metrics from Load Balancing in Google Cloud Platform.

Exported fields

An example event for loadbalancing looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-observability"
        },
        "provider": "gcp",
        "region": "us-central1",
        "availability_zone": "us-central1-a"
    },
    "event": {
        "dataset": "gcp.loadbalancing_metrics",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "labels": {
            "metrics": {
                "client_network": "ocp-be-c5kjr-network",
                "client_subnetwork": "ocp-be-c5kjr-worker-subnet",
                "client_zone": "us-central1-a"
            },
            "resource": {
                "backend_name": "ocp-be-c5kjr-master-us-central1-a",
                "backend_scope": "us-central1-a",
                "backend_scope_type": "ZONE",
                "backend_subnetwork_name": "ocp-be-c5kjr-master-subnet",
                "backend_target_name": "ocp-be-c5kjr-api-internal",
                "backend_target_type": "BACKEND_SERVICE",
                "backend_type": "INSTANCE_GROUP",
                "forwarding_rule_name": "ocp-be-c5kjr-api-internal",
                "load_balancer_name": "ocp-be-c5kjr-api-internal",
                "network_name": "ocp-be-c5kjr-network",
                "region": "us-central1"
            }
        },
        "loadbalancing_metrics": {
            "l3": {
                "internal": {
                    "egress_packets": {
                        "count": 100
                    },
                    "egress": {
                        "bytes": 1247589
                    }
                }
            }
        }
    },
    "metricset": {
        "name": "loadbalancing",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

Redis

The redis dataset is designed to fetch metrics from GCP Memorystore for Redis in Google Cloud Platform.

Exported fields

An example event for redis looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.redis",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "redis": {
            "clients": {
                "blocked": {
                    "count": 4
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "metrics",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

Storage

The storage dataset fetches metrics from Storage in Google Cloud Platform.

Exported fields

An example event for storage looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.storage",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "storage": {
            "storage": {
                "total": {
                    "bytes": 4472520191
                }
            },
            "network": {
                "received": {
                    "bytes": 4472520191
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "storage",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

VMware vSphere Integration

This integration periodically fetches logs and metrics from vSphere vCenter servers. 

 Compatibility 
The integration uses the Govmomi library to collect metrics and logs from any Vmware SDK URL (ESXi/VCenter). This library is built for and tested against ESXi and vCenter 6.5, 6.7 and 7.0. 

 Installation Guide:  

VMware vSphere 7.0 Installation 
Govmomi Library 

 Integration Process 

Go> Cyber Incident Management (XDR and MDR) 

 

 Go> Cyber Incident Management (XDR and MDR)> Settings 

 

 Go> Cyber Incident Management (XDR and MDR)> Settings> Integration 

 

Go> Cyber Incident Management (XDR and MDR)> Settings> Integration> 
In search bar type “Vmware” 

 

 Click Add Agent 

 

 Choose your Log Collector 

 

 

Click the vSphere logs and metrics 

 

 

  Keep it as is  

 

 

Enter the IP address and port 

 

Example: https://127.0.0.1:8989/sdk 
127.0.0.1: This is the IP address of the local machine (localhost). 
8989: This is the port number on which the SDK service is running. (Keep it as is) 
/sdk: This indicates that the SDK is accessible at this path. (Keep it as is) 

Notes: To add multiple hosts, enter each IP address following the same format (https://<IP_or_hostname>:port/sdk) and press enter. 

 

Enter the Username and password of vSphere account 

 

Notes: The insecure option bypasses the verification of the server's certificate chain, which can be useful in certain scenarios but comes with significant security risks. It is recommended to use this option only when necessary and in environments where security concerns are minimal. 

 Logs collection 

 

 

 Collect logs from vSphere via UDP 

Tags: Click the given tags  

UDP host to listen on: This is the IP address of the machine where the log collector is running. 

UDP port to listen on: This is the port on which the log collector will listen for incoming log data. (Keep it as is) 

Notes: Enabling "Preserve original event" ensures raw log data is always available, crucial for troubleshooting, compliance, and verifying log accuracy. It adds raw data to event.original, doubling storage needs and potentially slowing processing if storage isn't scaled, impacting efficiency. 

 

 Collect logs from vSphere via TCP 

 

 

Tags: Click the given tags  

TCP host to listen on: This is the IP address of the machine where the log collector is running. 

TCP port to listen on: This is the port on which the log collector will listen for incoming log data. (Keep it as is) 

Notes: Enabling "Preserve original event" ensures raw log data is always available, crucial for troubleshooting, compliance, and verifying log accuracy. It adds raw data to event.original, doubling storage needs and potentially slowing processing if storage isn't scaled, impacting efficiency. 

 
Click Next to complete the integration.