System Integrations

• Cyber Incident Monitoring Integration Procedure
• Microsoft 365
• GitHub
• Add Windows Integrations
• Sysmon for Linux
• 1 Password Integrations
• Atlassian Bitbucket Integrations
• AWS Cloudtrails Integrations
• AWS GuardDuty Integrations
• AWS Security Hub Integrations
• AWS Integrations
• CISCO Meraki Integrations
• CISCO Secure Endpoint Integrations
• CISCO Umbrella Integrations
• Cloudflare Integration
• Crowdstrike Integrations
• Dropbox Integrations
• F5 Integrations
• Fortinet-Fortigate Integrations
• GCP Integrations
• GitLab Integrations
• Google Workspace Integrations
• Jumpcloud Integrations
• Mimecast Integrations
• MongoDB Integrations
• OKTA Integrations
• Pulse Connect Secure Integrations
• Slack Integrations
• System Integrations
• Team Viewer Integrations
• Z Scaler Integrations
• gcp
• VMware vSphere Integration
• SentinelOne Integrations
• Custom Windows Event Logs - Integration
• Windows Event Forwarding to Linux server using Nxlog
• Windows Event Forwarding to Linux server using Powershell script
• Sophos Integration
• Atlassian Bitbucket Integrations (New)
• Palo Alto Cortex XDR Integration
• Active Directory Integrations
• Microsoft SQL Server Integration
• Azure Logs Integration
• ESET Protect Integration
• ESET Threat Intelligence Integrations
• CSPM for Azure Integration
• Resource Manager Endpoint Integration
• CISCO Secure Email Gateway Integrations
• CISCO Nexus Integrations
• BitDefender Integrations
• Bitwarden Integrations
• Forwarding logs from rsyslog client to a remote rsyslogs server
• New script for logs forwarding

Cyber Incident Monitoring Integration Procedure

Go to > Cyber Incident Monitoring

Microsoft 365

Microsoft Office 365 integration currently supports user, admin, system, and policy actions and events from Office 365 and Azure AD activity logs exposed by the Office 365 Management Activity API.

Procedures 

To perform the setup, please confirm that you have the following access: 

Register a new Office 365 web application To get started collecting Office 365 logs, register an Office 365 web application: 

Setup Active Directory security permissions

The Active Directory security permissions allow the application you created to read threat intelligence data and activity reports for your organization. 

To set up Active Directory permissions: 

Steps to Renew the Client Secret (API Key):

1. Log into the Azure Portal:

   • Go to the Azure Portal and log in using an account with administrative privileges.

2. Navigate to Azure Active Directory:

   • In the left navigation pane, select Azure Active Directory.
   • If it's not visible, click Show all to expand the list and find it.

3. Go to App Registrations:

   • Under Azure Active Directory, select App Registrations.
   • Find your registered application (e.g., "o365cytech") in the list, or use the search bar to locate it.

4. Open Certificates & Secrets:

   • Click on the registered app to open its details page.
   • In the left-hand menu, select Certificates & Secrets.

5. Generate a New Client Secret:

   • Under Client Secrets, you'll see a list of previously created secrets, along with their expiration dates.
   • Click + New client secret to create a new one.

6. Configure the New Secret:

   • Enter a description for the new key (e.g., "Renewed Key for o365cytech").
   • Set the duration for the new client secret:

7. Save and Copy the New Secret:

   • Click Add.
   • Once the new secret is generated, copy the value immediately. This is your new client secret (API key). The secret value will be hidden after you leave this page, so make sure to store it securely.

8. Update Any Services Using the Key:

   • If any services or scripts are using the previous client secret, you'll need to update them with the new one.

9. Remove the Old Secret (Optional):

   • If the old client secret is no longer needed, you can delete it to avoid confusion. Simply click the trash icon next to the old key under Client Secrets.

GitHub

Introduction 

The GitHub integration collects events from the GitHub API. 

Logs

Audit 

The GitHub audit log records all events related to the GitHub organization.  

To use this integration, you must be an organization owner, and you must use an Personal Access Token with the admin:org scope. 

This integration is not compatible with GitHub Enterprise server. 

 

Code Scanning 

The Code Scanning lets you retrieve all security vulnerabilities and coding errors from a repository setup using Github Advanced Security Code Scanning feature.  

To use this integration, GitHub Apps must have the security_events read permission. Or use a personal access token with the security_events scope for private repos or public_repo scope for public repos.  

 

Secret Scanning 

The Github Secret Scanning lets you retrieve secret scanning for advanced security alerts from a repository setup using Github Advanced Security Secret Scanning feature.  

To use this integration, GitHub Apps must have the secret_scanning_alerts read permission. Or you must be an administrator for the repository or for the organization that owns the repository, and you must use a personal access token with the repo scope or security_events scope. For public repositories, you may instead use the public_repo scope.  

 

Dependabot 

The Github Dependabot lets you retrieve known vulnerabilities in dependencies from a repository setup using Github Advanced Security Dependabot feature.  

 To use this integration, you must be an administrator for the repository or for the organization that owns the repository, and you must use a personal access token with the repo scope or security_events scope. For public repositories, you may instead use the public_repo scope.  

 

Issues 

The Github Issues datastream lets you retrieve github issues, including pull requests, issue assignees, comments, labels, and milestones. See About Issues for more details. You can retrieve issues for specific repository or for entire organization. Since Github API considers pull requests as issues, users can use github.issues.is_pr field to filter for only pull requests.

All issues including closed are retrieved by default. If users want to retrieve only open requests, you need to change State parameter to open. 

To use this integration, users must use Github Apps or Personal Access Token with read permission to repositories or organization. Please refer to Github Apps Permissions Required and Personal Access Token Permissions Required for more details. 

GitHub Integration Procedures 

Please provide the following information to CyTech: 

1.Select Settings 

2. Select Developer Settings 

 

 

3. Select token (classic) 

4. Select scope admin:scope 

Collect GitHub logs via API 

GHAS Code Scanning 

GHAS Dependabot 

Github Issues 

1. Personal Access Token - the GitHub Personal Access Token. 

2. Repository owner - The owner of GitHub Repository. If repository belongs to an organization, owner is name of the organization.

GHAS Secret Scanning 

1. Personal Access Token - the GitHub Personal Access Token. Requires admin access to the repository or organization owning the repository along with a personal access token with 'public_repo' scope for public repositories and repo or security_events scope for private repositories. \nSee List secret scanning alerts for a repository 

2. Repository owner - The owner of GitHub Repository 

Add Windows Integrations

Introduction 

The Windows integration allows you to monitor the Windows OS, services, applications, and more. 

Use the Windows integration to collect metrics and logs from your machine. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue. 

For example, if you wanted to know if a Windows service unexpectedly stops running, you could install the Windows integration to send service metrics to Elastic. Then, you could view real-time changes to service status in Kibana's [Metrics Windows] Services dashboard. 

Data streams 

The Windows integration collects two types of data: logs and metrics. 

Logs help you keep a record of events that happen on your machine. Log data streams collected by the Windows integration include forwarded events, PowerShell events, and Sysmon events. Log collection for the Security, Application, and System event logs is handled by the System integration. See more details in the Logs reference. 

Metrics give you insight into the state of the machine. Metric data streams collected by the Windows integration include service details and performance counter values. See more details in the Metrics reference. 

Note: For 7.11, security, application and system logs have been moved to the system package. 

 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

Requirements 

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware. 

Each data stream collects different kinds of metric data, which may require dedicated permissions to be fetched and which may vary across operating systems. 

Setup 

For step-by-step instructions on how to set up an integration, see the Getting started guide. 

Note: Because the Windows integration always applies to the local server, the hosts config option is not needed. 

Ingesting Windows Events via Splunk 

This integration allows you to seamlessly ingest data from a Splunk Enterprise instance. The integration uses the httpjson input in Elastic Agent to run a Splunk search via the Splunk REST API and then extract the raw event from the results. The raw event is then processed via the Elastic Agent. You can customize both the Splunk search query and the interval between searches. For more information see Ingest data from Splunk. 

Note: This integration requires Windows Events from Splunk to be in XML format. To achieve this, renderXml needs to be set to 1 in your inputs.conf file. 

Logs reference 

Forwarded 

The Windows forwarded data stream provides events from the Windows ForwardedEvents event log. The fields will be the same as the channel specific data streams. 

Powershell 

The Windows powershell data stream provides events from the Windows Windows PowerShell event log. 

System Integration Procedures 

Collect events from the following Windows event log channels: (Enable Yes/No) 

Powershell (Enable Yes/No) 

Powershell Operational (Enable Yes/No) 

Sysmon Operational (Enable Yes/No) 

Collect Windows perfmon and service metrics (Enable Yes/No) 

Windows service metrics (Enable Yes/No) 

Collect logs from third-party REST API (experimental) (Enable Yes/No) 

Windows ForwardedEvents via Splunk Enterprise REST API (Enable Yes/No) 

Windows Powershell Events via Splunk Enterprise REST API (Enable Yes/No) 

Windows Powershell Operational Events via Splunk Enterprise REST API (Enable Yes/No) 

Windows Sysmon Operational Events via Splunk Enterprise REST API (Enable Yes/No) 

Sysmon for Linux

Introduction 

The Sysmon for Linux integration allows you to monitor the Sysmon for Linux, which is an open-source system monitor tool developed to collect security events from Linux environments. 

Use the Sysmon for Linux integration to collect logs from linux machine which has sysmon tool running. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue. 

NOTE: To collect Sysmon events from Windows event log, use Windows sysmon_operational data stream instead. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

Requirements 

Setup 

For step-by-step instructions on how to set up an integration, see the Getting started guide. 

Data streams 

The Sysmon for Linux log data stream provides events from logs produced by Sysmon tool running on Linux machine. 

Sysmon for Linux Integration 

Please provide the following information to CyTech:   

Collect Sysmon for Linux logs (Enable Yes/No) 

1 Password Integrations

Introduction 

With 1Password Business, you can send your account activity to your security information and event management (SIEM) system, using the 1Password Events API. 

Get reports about 1Password activity, such as sign-in attempts and item usage, while you manage all your company’s applications and services from a central location.  

With 1Password Events Reporting and Elastic SIEM, you can: 

Events 

Sign-in Attempts 

Use the 1Password Events API to retrieve information about sign-in attempts. Events include the name and IP address of the user who attempted to sign in to the account, when the attempt was made, and – for failed attempts – the cause of the failure. 

Item Usages 

This uses the 1Password Events API to retrieve information about items in shared vaults that have been modified, accessed, or used. Events include the name and IP address of the user who accessed the item, when it was accessed, and the vault where the item is stored. 

Requirements 

You can set up Events Reporting if you’re an owner or administrator. 

Ready to get started?  

1Password Integration Procedures 

Please provide the following information to CyTech: 

The 1Password Events API Beat returns information from 1Password through requests to the Events REST API and sends that data securely to Elasticsearch. Requests are authenticated with a bearer token. Issue a token for each application or service you use. 

To connect your 1Password account to Elastic: 

You can now use Elasticsearch with the 1Password Events API Beat to monitor events from your 1Password account. The returned data will follow the Elastic Common Schema (ECS) specifications.

Collect events from 1Password Events API 

Atlassian Bitbucket Integrations

Introduction 

The Bitbucket integration collects audit logs from the audit log files or the audit API. 

Reference:  https://developer.atlassian.com/server/bitbucket/reference/rest-api/  

Assumptions 

The procedures described in Section 3 assume that a Log Collector has already been set up.  

Requirements 

For more information on auditing in Bitbucket and how it can be configured, see View and configure the audit log on Atlassian's website. 

Reference: https://confluence.atlassian.com/bitbucketserver/view-and-configure-the-audit-log-776640417.html  

 

Logs 

Audit 

The Confluence integration collects audit logs from the audit log files or the audit API from self-hosted Confluence Data Center. It has been tested with Confluence 7.14.2 but is expected to work with newer versions. As of version 1.2.0, this integration added experimental support for Atlassian Confluence Cloud. JIRA Cloud only supports Basic Auth using username and a Personal Access Token. 

Atlassian Bitbucket Integration Procedures 

Please provide the following information to CyTech: 

Collect Bitbucket audit logs via log files 

Collect Bitbucket audit logs via API (Enable Yes/No) 

AWS Cloudtrails Integrations

Introduction 

The AWS CloudTrail integration allows you to monitor AWS CloudTrail 

Reference: https://aws.amazon.com/cloudtrail/  

Use the AWS CloudTrail integration to collect and parse logs related to account activity across your AWS infrastructure. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference logs when troubleshooting an issue. 

For example, you could use the data from this integration to spot unusual activity in your AWS accounts—like excessive failed AWS console sign in attempts. 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Data streams 

The AWS CloudTrail integration collects one type of data: logs. 

Logs help you keep a record of every event that CloudTrail receives. These logs are useful for many scenarios, including security and access audits. See more details in the Logs reference. 

Reference : https://aquila-elk.kb.us-east-1.aws.found.io:9243/app/integrations/detail/aws-1.28.3/overview?integration=cloudtrail - logs-reference 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

Requirements 

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware. 

Before using any AWS integration you will need: 

For more details about these requirements, see the AWS integration documentation. 

 

Setup 

Use this integration if you only need to collect data from the AWS CloudTrail service. 

If you want to collect data from two or more AWS services, consider using the AWS integration. When you configure the AWS integration, you can collect data from as many AWS services as you'd like. 

For step-by-step instructions on how to set up an integration, see the Getting started guide. 

 

Logs reference 

The cloudtrail data stream collects AWS CloudTrail logs. CloudTrail monitors events like user activity and API usage in AWS services. If a user creates a trail, it delivers those events as log files to a specific Amazon S3 bucket. 

AWS CloudTrail integration Procedures  

The following will need to be provided in the Configure integration when adding the AWS Audit CloudTrail integration.   

Procedures:   

Please provide the following information to CyTech:   

Configure Integration 

AWS GuardDuty Integrations

Introduction 

The Amazon GuardDuty integration collects and parses data from Amazon GuardDuty Findings REST APIs. 

The Amazon GuardDuty integration can be used in three different modes to collect data: 

Assumptions 

The procedures described in Section 3 assume that a Log Collector has already been setup.  

Requirements 

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware. 

Note: It is recommended to use AWS SQS for Amazon GuardDuty. 

Aws GuardDuty integration Procedures 

To collect data from AWS S3 Bucket, follow the steps below: 

To collect data from AWS SQS, follow the steps below: 

Note: 

To collect data from Amazon GuardDuty API, users must have an Access Key and a Secret Key. To create an API token follow the steps below: 

Note 

 

AWS Security Hub Integrations

Introduction 

The AWS Security Hub integration collects and parses data from AWS Security Hub REST APIs. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Compatibility 

Requirements 

To collect data from AWS Security Hub APIs, users must have an Access Key and a Secret Key. To create API token follow below steps: 

Note:

Logs:

AWS Security Hub Integration Procedures 

Please provide the following information to CyTech: 

Collect AWS Security Hub logs via API 

Collect AWS Security Hub Insights from AWS 

AWS Integrations

Introduction 

This document shows information related to AWS Integration.  

The AWS integration is used to fetch logs and metrics from Amazon Web Services. 

The usage of the AWS integration is to collect metrics and logs across many AWS services managed by your AWS account. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Requirements 

Before using the AWS integration you will need: 

AWS Credentials 

Use access keys directly (Option 1 Recommended) 

Access keys are long-term credentials for an IAM user or the AWS account root user. To use access keys as credentials, you need to provide: 

Use an IAM role Amazon Resource Name (ARN)  

To use an IAM role ARN, you need to provide either a credential profile or access keys along with the role_arn advanced option. role_arn is used to specify which AWS IAM role to assume for generating temporary credentials. 

Use a shared credentials file (Option 2) 

Instead of providing the access_key_id and secret_access_key directly to the integration, you will provide two advanced options to look up the access keys in the shared credentials file: 

shared_credential_file: The directory of the shared credentials file. 

Access keys are long-term credentials for an IAM user or the AWS account root user. To use access keys as credentials, you need to provide: 

AWS Permissions 

Specific AWS permissions are required for the IAM user to make specific AWS API calls. To enable the AWS integration to collect metrics and logs from all supported services, make sure to give necessary permissions which CyTech to monitor. 

Reference permissions: 

AWS Integrations Procedures 

Access Key ID and Secret Access Key:  

These are associated with AWS Identity and Access Management (IAM) users and are used for programmatic access to AWS services. To find them: 

S3 Bucket ARN:  

You can find the Amazon Resource Name (ARN) for an S3 bucket in the S3 Management Console or by using the AWS CLI. It typically looks like this: 

Log Group ARN: 

Log Groups are associated with AWS CloudWatch Logs. You can find the ARN for a log group as follows: 

SQS Queue URL: (Ignore if you're not using SQS Queue URL) 

To find the URL of an Amazon Simple Queue Service (SQS) queue: 

Please provide the following information to CyTech: 

CISCO Meraki Integrations

Introduction 

Cisco Meraki offers a centralized cloud management platform for all Meraki devices such as MX Security Appliances, MR Access Points and so on. Its out-of-band cloud architecture creates secure, scalable, and easy-to-deploy networks that can be managed from anywhere. This can be done from almost any device using web-based Meraki Dashboard and Meraki Mobile App. Each Meraki network generates its own events. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

Compatibility 

A syslog server can be configured to store messages for reporting purposes from MX Security Appliances, MR Access Points, and MS switches. This package collects events from the configured syslog server. The integration supports collection of events from "MX Security Appliances" and "MR Access Points". The "MS Switch" events are not recognized. 

Requirements 

Cisco Meraki Dashboard Configuration 

SYSLOG 

Cisco Meraki dashboard can be used to configure one or more syslog servers and Meraki message types to be sent to the syslog servers. Refer to Syslog Server Overview and Configuration page for more information on how to configure syslog server on Cisco Meraki. 

API ENDPOINT (WEBHOOKS) 

Cisco Meraki dashboard can be used to configure Meraki webhooks. Refer to the Webhooks Dashboard Setup section. 

Configure the Cisco Meraki integration 

SYSLOG 

Depending on the syslog server setup in your environment check one/more of the following options "Collect syslog from Cisco Meraki via UDP", "Collect syslog from Cisco Meraki via TCP", "Collect syslog from Cisco Meraki via file". 

Enter the values for syslog host and port OR file path based on the chosen configuration options. 

API Endpoint (Webhooks) 

Check the option "Collect events from Cisco Meraki via Webhooks" option. 

Log Events 

Enable to collect Cisco Meraki log events for all the applications configured for the chosen log stream. 

Logs 

Syslog 

The cisco_meraki.log dataset provides events from the configured syslog server. All Cisco Meraki syslog specific fields are available in the cisco_meraki.log field group. 

API Endpoint (Webhooks) 

 

Cisco Meraki Integration Procedures 

Please provide the following information to CyTech: 

CISCO Secure Endpoint Integrations

Introduction 

Secure Endpoint offers cloud-delivered, advanced endpoint detection and response across multidomain control points to rapidly detect, contain, and remediate advanced threats. 

Assumptions 

The procedures described in Section 3 assume that a Log Collector has already been setup.   

Requirements 

This integration is for Cisco Secure Endpoint logs. It includes the following datasets for receiving logs over syslog or read from a file: 

Generating Client ID and API Key:

Logs 

Secure Endpoint 

The event dataset collects Cisco Secure Endpoint logs. 

What can the Secure Endpoint API be used for? 

Top Use Cases 

Response Format 

Cisco Secure Endpoint Integration Procedures 

Please provide the following information to CyTech: 

Collect logs from the Cisco Secure Endpoint API. 

 

CISCO Umbrella Integrations

Introduction 

Cisco Umbrella is a cloud security platform that provides an additional line of defense against malicious software and threats on the internet by using threat intelligence. That intelligence helps prevent adware, malware, botnets, phishing attacks, and other known bad Websites from being accessed. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

Prerequisites 

Requirements 

This integration is for Cisco Umbrella. It includes the following datasets for receiving logs from an AWS S3 bucket using an SQS notification queue and Cisco Managed S3 bucket without SQS: 

Logs 

Umbrella 

When using Cisco Managed S3 buckets that does not use SQS there is no load balancing possibilities for multiple agents, a single agent should be configured to poll the S3 bucket for new and updated files, and the number of workers can be configured to scale vertically. 

The log dataset collects Cisco Umbrella logs. 

Advantages of Integrating with the Umbrella API 

The Umbrella API features a number of improvements over the Umbrella v1 APIs and the Umbrella Reporting v2 API. 

Before you send a request to the Umbrella API, you must create new Umbrella API credentials and generate an API access token. For more information, see Umbrella API Authentication.

 https://developer.cisco.com/docs/cloud-security/authentication/#authentication

Authentication 

The Umbrella API provides a standard REST interface and supports the OAuth 2.0 client credentials flow. To get started, log in to Umbrella and create an Umbrella API key. Then, use your API credentials to generate an API access token. 

Note: API keys, passwords, secrets, and tokens allow access to your private customer data. You should never share your credentials with another user or organization. 

Log in to Umbrella 

Create Umbrella API Key

Create an Umbrella API key ID and key secret.

Note: You have only one opportunity to copy your API secret. Umbrella does not save your API secret and you cannot retrieve the secret after its initial creation.

1. Navigate to Admin > API Keys or in a Multi-org, Managed Service Provider (MSP), or Managed Secure Service Provider (MSSP) console navigate to Console Settings > API Keys.

2. Click API Keys and then click Add.

   • The number of expired API keys appears next to the red triangle.
   • The number of API keys that expire within 30 days appears next to the yellow triangle.

3. Enter a name and description for the key. A name must contain fewer than 256 characters. The description is optional.

   

4. Check the key scopes and expand a key scope to view the scope categories. Check each scope category in a key scope to enable access to the API endpoints.

   

5. Choose Read-Only or Read / Write for the selected scope and resource.

   

6. For Expiry Date, choose the expiration date for the key, or choose Never expire.

   

7. (Optional) For Network Restrictions, enter a comma-separated list of public IP addresses or CIDRs, then click ADD.

   Note: You can add up to ten networks to your API key. You can only use your API key to authenticate requests for clients on the selected networks.

   

8. Click Create Key.

9. Copy and save your API Key and Key Secret.

10. Click Accept And Close.

Refresh Umbrella API Key

Refresh an Umbrella API key ID and key secret.

Note: You have only one opportunity to copy your API secret. Umbrella does not save your API secret and you cannot retrieve the secret after its initial creation.

1. Navigate to Admin > API Keys or in a Multi-org, Managed Service Provider (MSP), or Managed Secure Service Provider (MSSP) console, navigate to Console Settings > API Keys.

2. Click API Keys, and then expand an API key.

3. Click Refresh Key.

   

4. Copy and save your API Key and Key Secret.

5. Click Accept and Close.

Update Umbrella API Key

Update an Umbrella API key.

1. Navigate to Admin > API Keys or in a Multi-org, Managed Service Provider (MSP), or Managed Secure Service Provider (MSSP) console, navigate to Console Settings > API Keys.

2. Click API Keys, and then expand an API key. You can modify the API Key Name, Description, selected scopes and permissions in Key Scope, and Expiry Date.

   

3. For Network Restrictions, update the list of IP addresses and CIDRs. Click on the X to remove a network address.

4. Click Save.

Cisco Secure Endpoint Integration Procedures 

Please provide the following information to CyTech: 

Collect logs from the Cisco Umbrella 

If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.

Cloudflare Integration

Introduction 

Cloudflare integration uses Cloudflare's API to retrieve audit logs and traffic logs from Cloudflare, for a particular zone, and ingest them into Elasticsearch. This allows you to search, observe and visualize the Cloudflare log events through Elasticsearch. 

Users of Cloudflare use Cloudflare services to increase the security and performance of their web sites and services. 

To enable the Cloudflare Logpush, please refer to Section 5. Currently, the procedures described is for the setup of Amazon S3.  

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Requirements 

Configure Cloudflare audit logs data stream 

Enter values "Auth Email", "Auth Key" and "Account ID". 

Configure Cloudflare logs 

These logs contain data related to the connecting client, the request path through the Cloudflare network, and the response from the origin web server. For more information see here. 

The integration can retrieve Cloudflare logs using - 

CONFIGURE USING AUTH EMAIL AND AUTH KEY 

Enter values "Auth Email", "Auth Key" and "Zone ID". 

CONFIGURE USING API TOKEN 

Enter values "API Token" and "Zone ID".

For the Cloudflare integration to be able to successfully get logs the following permissions must be granted to the API token - 

Logs 

Audit 

Audit logs summarize the history of changes made within your Cloudflare account. Audit logs include account-level actions like login and logout, as well as setting changes to DNS, Crypto, Firewall, Speed, Caching, Page Rules, Network, and Traffic features, etc. 

Logpull 

These logs contain data related to the connecting client, the request path through the Cloudflare network, and the response from the origin web server. 

Cloudflare Integration Procedures 

 Please provide the following information to CyTech: 

See the Screenshot Below 

 

 

 

 

 

Audit Logs 

Cloudflare Logs 

Enable Logpush to Amazon S3 

To enable the Cloudflare Logpush service: 

Once connected, Cloudflare lists Amazon S3 as a connected service under Logs > Logpush. Edit or remove connected services from here. 

Crowdstrike Integrations

Introduction 

This integration is for CrowdStrike products. It includes the following datasets for receiving logs: 

falcon dataset consists of endpoint data and Falcon platform audit data forwarded from Falcon SIEM Connector. 

fdr dataset consists of logs forwarded using the Falcon Data Replicator. 

Assumptions 

The procedures described in Section 3 assume that a Log Collector has already been setup.   

Compatibility 

This integration supports CrowdStrike Falcon SIEM-Connector-v2.0. 

Requirements 

Logs 

Falcon 

Contains endpoint data and CrowdStrike Falcon platform audit data forwarded from Falcon SIEM Connector. 

FDR 

The CrowdStrike Falcon Data Replicator (FDR) allows CrowdStrike users to replicate FDR data from CrowdStrike managed S3 buckets. CrowdStrike writes notification events to a CrowdStrike managed SQS queue when new data is available in S3. 

This integration can be used in two ways. It can consume SQS notifications directly from the CrowdStrike managed SQS queue or it can be used in conjunction with the FDR tool that replicates the data to a self-managed S3 bucket and the integration can read from there. 

In both cases SQS messages are deleted after they are processed. This allows you to operate more than one Elastic Agent with this integration if needed and not have duplicate events, but it means you cannot ingest the data a second time. 

CrowdStrike Integration Procedures 

Please provide the following information to CyTech: 

Collect CrowdStrike Falcon Data Replicator logs (input: aws-s3) Option 1 

Collect CrowdStrike logs via API. Option 2 (Recommended) 

Dropbox Integrations

Introduction 

Connecting Dropbox 

Use the Workplace Search Dropbox connector to automatically capture, sync and index the following items from your Dropbox service: 

Stored Files 

Including ID, File Metadata, File Content, Updated by, and timestamps. 

Dropbox Paper 

Including ID, Metadata, Content, Updated by, and timestamps. 

This document helps you configure and connect your Dropbox service to Workplace Search. To do this, you must register your Elastic deployment in the Dropbox Developer platform, by creating an OAuth 2.0 app. This gives your app permission to access Dropbox data. You will need your Workplace Search OAuth redirect URL, so have that handy. 

If you need a primer on OAuth, read the official OAuth 2.0 authorization flow RFC. The diagrams provide a good mental model of the protocol flow. Dropbox also has its own OAuth guide. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Requirements 

Configuring the Dropbox Connector 

To fetch document-level permissions, you must create an OAuth App using a team-owned (Dropbox Business) account. Enable document-level permissions by adding the following permissions: 

Your Dropbox service is now configured, and you can connect it to Workplace Search.  

 

Dropbox Integration Procedures 

Connecting Dropbox to Workplace Search 

Once the Dropbox connector is configured, you can connect a Dropbox instance to your organization’s Workplace Search deployment. 

Your Dropbox content becomes searchable as soon as syncing starts. Once configured and connected, Dropbox synchronizes automatically every 2 hours. 

Limiting the content to be indexed 

If you don’t need to index all available content, you can use the API to apply indexing rules. This reduces indexing load and overall index size. See Customizing indexing. 

The path_template and file_extension rules are applicable for Dropbox. 

Synchronized fields 

The following table lists the fields synchronized from the connected source to Workplace Search. The attributes in the table apply to the default search application, as follows: 

F5 Integrations

Introduction 

This document shows information related to F5 Integration.  

The F5 BIG-IP integration allows users to monitor LTM, AFM, APM, ASM, and AVR activity. F5 BIG-IP covers software and hardware designed around application availability, access control, and security solutions. 

 

The F5 BIG-IP integration can be used in three different modes to collect data: 

 

HTTP Endpoint mode - F5 BIG-IP pushes logs directly to an HTTP endpoint hosted by users’ Elastic Agent. 

AWS S3 polling mode - F5 BIG-IP writes data to S3 and Elastic Agent polls the S3 bucket by listing its contents and reading new files. 

AWS S3 SQS mode - F5 BIG-IP writes data to S3, S3 pushes a new object notification to SQS, Elastic Agent receives the notification from SQS, and then reads the S3 object. Multiple Agents can be used in this mode. 

For example, users can use the data from this integration to analyze the traffic that passes through their F5 BIG-IP network. 

 

Data streams 

The F5 BIG-IP integration collects one type of data stream: log. 

Log help users to keep a record of events happening on the network using telemetry streaming. The log data stream collected by the F5 BIG-IP integration includes events that are related to network traffic. See more details in the Logs. 

This integration targets the five types of events as mentioned below: 

LTM provides the platform for creating virtual servers, performance, service, protocol, authentication, and security profiles to define and shape users’ application traffic. For more information, refer to the link here. 

AFM is designed to reduce the hardware and extra hops required when ADC's are paired with traditional firewalls and helps to protect traffic destined for the user's data center. For more information, refer to the link here. 

APM provides federation, SSO, application access policies, and secure web tunneling and allows granular access to users' various applications, virtualized desktop environments, or just go full VPN tunnel. For more information, refer to the link here. 

ASM is F5's web application firewall (WAF) solution. It allows users to tailor acceptable and expected application behavior on a per-application basis. For more information, refer to the link here. 

AVR provides detailed charts and graphs to give users more insight into the performance of web applications, with detailed views on HTTP and TCP stats, as well as system performance (CPU, memory, etc.). For more information, refer to the link here. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Requirements 

Elasticsearch is needed to store and search data, and Kibana is needed for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your hardware. 

The reference link for requirements of telemetry streaming is here.  

The reference link for requirements of Application Services 3(AS3) Extension is here. 

This module has been tested against F5 BIG-IP version 16.1.0, Telemetry Streaming version 1.32.0 and AS3 version 3.40.0. 

 

Setup 

To collect LTM, AFM, APM, ASM, and AVR data from F5 BIG-IP, the user has to configure modules in F5 BIG-IP as per the requirements. 

To set up the F5 BIG-IP environment, users can use the BIG-IP system browser-based Configuration Utility or the command line tools that are provided. For more information related to the configuration of F5 BIG-IP servers, refer to F5 support website here. 

https://support.f5.com/csp/knowledge-center/software 

 

Configuration of Telemetry Streaming in F5 

For downloading and installing Telemetry Streaming, refer to the link here. 

https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/installation.html 

Telemetry Streaming will send logs in the JSON format to the destination. Telemetry Streaming is compatible with BIG-IP versions 13.0 and later. Users have to prepare F5 servers for it and set up the Telemetry Streaming Consumer.      

To use telemetry streaming, user have to send POST request on https://<BIG-IP>/mgmt/shared/telemetry/declare for declaration. 

F5 BIG-IP modules named LTM, AFM, ASM, and APM are not configured by Telemetry Streaming, they must be configured with AS3 or another method. Reference link for setup AS3 extension in F5 BIG-IP is here. 

To configure logging using AS3, refer to the link here. 

https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/event-listener.html?highlight=as3#configure-logging-using-as3 

To collect data from AWS S3 Bucket, follow the below steps: 

https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html 

To collect data from AWS SQS, follow the below steps: 

Note: 

Enabling the integration in Elastic 

F5 BIG-IP integration 

The "Integration name" and either the " Description" and the following will need to be provided in the Configure integration when adding the F5 BIG-IP integration.  

Procedures: 

Please provide the following information to CyTech: 

Collect F5 BIG-IP logs via HTTP Endpoint: 

F5 BIG-IP logs via HTTP Endpoint: 

Fortinet-Fortigate Integrations

Introduction 

This integration is for Fortinet FortiGate logs sent in the syslog format. 

---

Pre-requisite:

Configure syslog on FortiGate

From the GUI: 

1. Log into FortiGate. 
2. Select Log & Report to expand the menu. 
3. Select Log Settings. 

4. Toggle Send Logs to Syslog to Enabled. 

    

5. Enter the Syslog Collector IP address. Note: IP Address must be host's IP Address where the Elastic-Agent is installed. (For example. 192.168.1.19 as shown below)

If it is necessary to customize the port or protocol or setup the Syslog from the CLI below are the commands: 

config log syslogd setting 

    set status enable 

    set server "192.168.1.19" -- change IP Address to same as host's where Elastic Agent is installed  

    set mode udp 

    set port 514 

end 

To establish the connection to the Syslog Server using a specific Source IP Address, use the below CLI configuration: 

config log syslogd setting
    set status enable
    set server "192.168.1.19" -- change ip address to match host's IP
    set source-ip "172.16.1.1" -- change ip address to match host's source-ip address

    set mode udp

    set port 514
end

---

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Compatibility

This integration has been tested against FortiOS versions 6.x and 7.x up to 7.4.1. Newer versions are expected to work but have not been tested. 

Note

• When using the TCP input, be careful with the configured TCP framing. According to the Fortigate reference, framing should be set to rfc6587 when the syslog mode is reliable.

---

Fortinet FortiGate Integration Procedures 

Please provide the following information to CyTech: 

Collect Fortinet FortiGate logs (input: tcp) 

Collect Fortinet FortiGate logs (input: udp) 

If you need further assistance, kindly contact our support at info@cytechint.com for prompt assistance and guidance.

GCP Integrations

Introduction 

This document shows information related to GCP Integration.  

The Google Cloud integration collects and parses Google Cloud Audit Logs, VPC Flow Logs, Firewall Rules Logs and Cloud DNS Logs that have been exported from Cloud Logging to a Google Pub/Sub topic sink and collects Google Cloud metrics and metadata from Google Cloud Monitoring. 

Requirements 

To use this Google Cloud Platform (GCP) integration, the following needs to be set up: 

Service Accounts 

First, please create a Service Account. A Service Account (SA) is a particular type of Google account intended to represent a non-human user who needs to access the GCP resources. 

The Log Collector uses the SA (Service Account) to access data on Google Cloud Platform using the Google APIs. 

Here is a reference from Google related to Service Accounts: 

https://cloud.google.com/iam/docs/best-practices-for-securing-service-accounts 

Service Account with a Role  

You need to grant your Service Account (SA) access to Google Cloud Platform resources by assigning a role to the account. In order to assign minimal privileges, create a custom role that has only the privileges required by the Log Collector. Those privileges are below: 

*1 Only required if Agent is expected to create a new subscription. If you create the subscriptions yourself, you may omit these privileges.  

**2 Only required if corresponding collection will be enabled. 

After you have created the custom role, assign the role to your service account. 

Service Account Key 

Next, with the Service Account (SA) with access to Google Cloud Platform (GCP) resources setup in Section 3.1.1, you need some credentials to associate with it: a Service Account Key. 

From the list of SA (Service Accounts): 

 

GCP Integrations Procedures 

The audit dataset collects audit logs of administrative activities and accesses within your Google Cloud resources. 

 Procedures 

The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration. 

Please provide the following information to CyTech: 

The Project ID is the Google Cloud project ID where your resources exist. 

Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field. 

 

OPTION 1: CREDENTIALS FILE 

Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file. 

Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json. 

OPTION 2: CREDENTIALS JSON 

Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration. 

The dns dataset collects queries that name servers resolve for your Virtual Private Cloud (VPC) networks, as well as queries from an external entity directly to a public zone. 

Procedures 

The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration. 

Please provide the following information to CyTech: 

The Project ID is the Google Cloud project ID where your resources exist. 

Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field. 

 

OPTION 1: CREDENTIALS FILE 

Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file. 

Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json. 

OPTION 2: CREDENTIALS JSON 

Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration. 

The firewall dataset collects logs from Firewall Rules in your Virtual Private Cloud (VPC) networks. 

Procedures 

The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration. 

Please provide the following information to CyTech: 

The Project ID is the Google Cloud project ID where your resources exist. 

Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field. 

 

OPTION 1: CREDENTIALS FILE 

Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file. 

Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json. 

 

OPTION 2: CREDENTIALS JSON 

Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration. 

 

The vpcflow dataset collects logs sent from and received by VM instances, including instances used as GKE nodes. 

Procedures 

The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration. 

Please provide the following information to CyTech: 

The Project ID is the Google Cloud project ID where your resources exist. 

Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field. 

 

OPTION 1: CREDENTIALS FILE 

Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file. 

Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json. 

OPTION 2: CREDENTIALS JSON 

Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration. 

GitLab Integrations

Introduction 

Introduced in GitLab Starter 8.4. Support for Amazon Elasticsearch was introduced in GitLab Starter 9.0. 

This document describes how to set up Elasticsearch with GitLab. Once enabled, you'll have the benefit of fast search response times and the advantage of two special searches: 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Requirements 

Elasticsearch 6.0+ is not supported currently. We will support 6.0+ in the future.  

 

GitLabIntegration Procedures 

Procedures:  

Please provide the following information to CyTech:  

Elasticsearch is not included in the Omnibus packages. You will have to install it yourself whether you are using the Omnibus package or installed GitLab from source. Providing detailed information on installing Elasticsearch is out of the scope of this document. 

Once the data is added to the database or repository and Elasticsearch is enabled in the admin area the search index will be updated automatically. Elasticsearch can be installed on the same machine as GitLab, or on a separate server, or you can use the Amazon Elasticsearch service. 

You can follow the steps as described in the official web site or use the packages that are available for your OS. 

 

Google Workspace Integrations

Introduction 

Google Workspace (formerly G Suite) is a suite of cloud computing, productivity and collaboration tools, software and products developed and marketed by Google. It allows users to create, edit, and share documents, spreadsheets, presentations, and more. It also includes email, calendar, chat, and video conferencing tools. 

Google Workspace is designed for businesses of all sizes, from small businesses to large enterprises. It is a popular choice for businesses because it is affordable, easy to use, and secure. 

The Google Workspace integration collects and parses data from the different Google Workspace audit reports APIs. 

If you want to know more about how you can fully leverage the Google Workspace integration, there is a multipart blog from our Security Labs that will help you: 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

Compatibility 

It is compatible with a subset of applications under the Google Reports API v1. 

Requirements 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

In order to ingest data from the Google Reports API you must: 

Create access credentials 

Credentials are used to obtain an access token from Google's authorization servers so your app can call Google Workspace APIs. This guide describes how to choose and set up the credentials your app needs. 

Choose the access credential that is right for you 

The required credentials depends on the type of data, platform, and access methodology of your app. There are three types of credential types available: 

Assign a role to a service account: 

You must assign a prebuilt or custom role to a service account by a super administrator account. 

Create credentials for a service account: 

You need to obtain credentials in the form of a public/private key pair. These credentials are used by your code to authorize service account actions within your app. 

To obtain credentials for your service account: 

Your new public/private key pair is generated and downloaded to your machine as a new file. Save the downloaded JSON file as credentials.json in your working directory. This file is the only copy of this key. For information about how to store your key securely, see Managing service account keys. 

 

Optional: Set up domain-wide delegation for a service account 

To call APIs on behalf of users in a Google Workspace organization, your service account needs to be granted domain-wide delegation of authority in the Google Workspace Admin console by a super administrator account. For more information, see Delegating domain-wide authority to a service account. 

To set up domain-wide delegation of authority for a service account: 

If you don't have super administrator access to the relevant Google Workspace account, contact a super administrator for that account and send them your service account's Client ID and list of OAuth Scopes so they can complete the following steps in the Admin console. 

This integration will make use of the following oauth2 scope: 

Once you have downloaded your service account credentials as a JSON file, you are ready to set up your integration. 

Click the Advanced option of Google Workspace Audit Reports. The default value of "API Host" is https://www.googleapis.com. The API Host will be used for collecting access_transparency, admin, device, context_aware_access, drive, gcp, groups, group_enterprise, login, rules, saml, token and user accounts logs. 

NOTE: The Delegated Account value in the configuration, is expected to be the email of the administrator account, and not the email of the ServiceAccount. 

Google Workspace Integration Procedures 

Please provide the following information to CyTech: 

Collect access_transparency, admin, alert, context_aware_access, device, drive, gcp, groups, group_enterprise, login, rules, saml, token and user accounts logs (input: httpjson). 

Jumpcloud Integrations

Introduction 

The JumpCloud integration allows you to monitor events related to the JumpCloud Directory as a Service via the Directory Insights API. 

You can find out more about JumpCloud and JumpCloud Directory Insights here 

Data streams 

A single data stream named "jumpcloud.events" is used by this integration. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Requirements 

An Elastic Stack with an Elastic Agent is a fundamental requirement. 

An established JumpCloud tenancy with active users is the the other requirement. Basic Directory Insights API access is available to all subscription levels. 

NOTE: The lowest level of subscription currently has retention limits, with access to Directory Insights events for the last 15 days at most. Other subscriptions levels provide 90 days or longer historical event access. 

A JumpCloud API key is required, the JumpCloud documentation describing how to create one is here 

This JumpCloud Directory Insights API is documented here 

JumpCloud Integration Procedures 

Procedures:  

Please provide the following information to CyTech:  

Enabling the integration in Elastic 

 

Mimecast Integrations

Introduction 

The Mimecast integration collects events from the Mimecast API. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

Requirements 

Configuration 

Authorization parameters for the Mimecast API (Application Key, Application ID, Access Key, and Secret Key) should be provided by a Mimecast representative for this integration. Under Advanced options you can set the time interval between two API requests as well as the API URL. A Mimecast representative should also be able to give you this information in case you need to change the defaults. 

Note: Rate limit quotas may require you to set up different credentials for the different available log types. 

Logs 

Audit Events 

This is the mimecast.audit_events dataset. These logs contain Mimecast audit events with the following details: audit type, event category and detailed information about the event. More information about these logs. 

DLP Logs 

This is the mimecast.dlp_logs dataset. These logs contain information about messages that triggered a DLP or Content Examination policy. More information about these logs. 

SIEM Logs 

This is the mimecast.siem_logs dataset. These logs contain information about messages that contains MTA (message transfer agent) log – all inbound, outbound, and internal messages. 

Threat Intel Feed Malware: Customer 

This is the mimecast.threat_intel_malware_customer dataset. These logs contain information about messages that return identified malware threats at a customer level. Learn more about these logs. 

Threat Intel Feed Malware: Grid 

This is the mimecast.threat_intel_malware_grid dataset. These logs contain information about messages that return identified malware threats at a regional grid level. More about these logs. 

TTP Attachment Logs 

This is the mimecast.ttp_ap_logs dataset. These logs contain Mimecast TTP attachment protection logs with the following details: result of attachment analysis (if it is malicious or not etc.), date when file is released, sender and recipient address, filename and type, action triggered for the attachment, the route of the original email containing the attachment and details. Learn more about these logs. 

TTP Impersonation Logs 

This is the mimecast.ttp_ip_logs dataset. These logs contain information about messages containing information flagged by an Impersonation Protection configuration. 

TTP URL Logs 

This is the mimecast.ttp_url_logs dataset. These logs contain Mimecast TTP attachment protection logs with the following details: the category of the URL clicked, the email address of the user who clicked the link, the url clicked, the action taken by the user if user awareness was applied, the route of the email that contained the link, the action defined by the administrator for the URL, the date that the URL was clicked, url scan result, the action that was taken for the click, the description of the definition that triggered the URL to be rewritten by Mimecast, the action requested by the user, an array of components of the message where the URL was found. More about these logs. 

 

Mimecast Integration Procedures 

Please provide the following information to CyTech: 

Mimecast API 

 

MongoDB Integrations

Introduction 

This integration is used to fetch logs and metrics from MongoDB. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

Compatibility 

The log dataset is tested with logs from versions v3.2.11 and v4.4.4 in plaintext and json formats. The collstats, dbstats, metrics, replstatus and status datasets are tested with MongoDB 3.4 and 3.0 and are expected to work with all versions >= 2.8. 

Requirements 

MongoDB Privileges 

In order to use the metrics datasets, the MongoDB user specified in the package configuration needs to have certain privileges. 

We recommend using the clusterMonitor role to cover all the necessary privileges. 

You can use the following command in Mongo shell to create the privileged user (make sure you are using the admin db by using db command in Mongo shell). 

 

db.createUser( 

    { 

        user: "beats", 

        pwd: "pass", 

        roles: ["clusterMonitor"] 

    } 

) 

 

You can use the following command in Mongo shell to grant the role to an existing user (make sure you are using the admin db by using db command in Mongo shell). 

db.grantRolesToUser("user", ["clusterMonitor"]) 

Logs 

log 

The log dataset collects the MongoDB logs. 

 

Metrics 

collstats 

The collstats dataset uses the top administrative command to return usage statistics for each collection. It provides the amount of time, in microseconds, used and a count of operations for the following types: total, readLock, writeLock, queries, getmore, insert, update, remove, and commands. 

It requires the following privileges, which is covered by the clusterMonitor role: 

dbstats 

The dbstats dataset collects storage statistics for a given database. 

It requires the following privileges, which is covered by the clusterMonitor role: 

action on cluster resource 

action on the database resource 

metrics 

It requires the following privileges, which is covered by the clusterMonitor role: 

action on cluster resource 

replstatus 

The replstatus dataset collects status of the replica set. It requires the following privileges, which is covered by the clusterMonitor role: 

status 

The status returns a document that provides an overview of the database's state. 

It requires the following privileges, which is covered by the clusterMonitor role: 

action on cluster resource 

 

MongoDB Integration Procedures 

Please provide the following information to CyTech: 

Collect MongoDB application logs 

Collect MongoDB metrics 

Ex: localhost:27017 

OKTA Integrations

Introduction 

The Okta integration collects events from the Okta API, specifically reading from the Okta System Log API. 

Logs 

System 

The Okta System Log records system events related to your organization in order to provide an audit trail that can be used to understand platform activity and to diagnose problems. This module is implemented using the httpjson input and is configured to paginate through the logs while honoring any rate-limiting headers sent by Okta. 

Okta Integration Procedures 

Please provide the following information to CyTech: 

Collect Okta logs via API 

Pulse Connect Secure Integrations

Introduction 

This integration is for Pulse Connect Secure. 

Pulse Connect Secure Integration Procedures 

Please provide the following information to CyTech: 

Collect Pulse Connect Secure logs (input: udp) 

Collect Pulse Connect Secure logs (input: tcp) 

 

 

 

 

Slack Integrations

Introduction 

Slack is used by numerous organizations as their primary chat and collaboration tool. 

Please note the Audit Logs API is only available to Slack workspaces on an Enterprise Grid plan. These API methods will not work for workspaces on a Free, Standard, or Business+ plan. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Requirements 

Configuration 

Enabling the integration in Elastic 

Configure Slack audit logs data stream 

Enter values "OAuth API Token". 

CONFIGURE USING API TOKEN 

For the Slack integration to be able to successfully get logs the following "User Token Scopes"" must be granted to the Slack App: 

Logs 

Audit 

Audit logs summarize the history of changes made within the Slack Enterprise. 

 

SLACK Integration Procedures 

Please provide the following information to CyTech: 

Collect Slack logs via API 

Slack Audit logs 

System Integrations

Introduction 

The System integration allows you to monitor servers, personal computers, and more. 

Use the System integration to collect metrics and logs from your machines. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue. 

For example, if you wanted to be notified when less than 10% of the disk space is still available, you could install the System integration to send file system metrics to Elastic. Then, you could view real-time updates to disk space used on your system in Kibana's [Metrics System] Overview dashboard. You could also set up a new rule in the Elastic Observability Metrics app to alert you when the percent free is less than 10% of the total disk space. 

Data streams 

The System integration collects two types of data: logs and metrics. 

Logs help you keep a record of events that happen on your machine. Log data streams collected by the System integration include application, system, and security events on machines running Windows and auth and syslog events on machines running macOS or Linux. See more details in the Logs reference. 

Metrics give you insight into the state of the machine. Metric data streams collected by the System integration include CPU usage, load statistics, memory usage, information on network behavior, and more. See more details in the Metrics reference. 

You can enable and disable individual data streams. If all data streams are disabled and the System integration is still enabled, Fleet uses the default data streams. 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.   

Requirements 

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware. 

Each data stream collects different kinds of metric data, which may require dedicated permissions to be fetched and which may vary across operating systems. Details on the permissions needed for each data stream are available in the Metrics reference. 

Setup 

For step-by-step instructions on how to set up an integration, see the Getting started guide. 

Troubleshooting 

Note that certain data streams may access /proc to gather process information, and the resulting ptrace_may_access() call by the kernel to check for permissions can be blocked by AppArmor and other LSM software, even though the System module doesn't use ptrace directly. 

In addition, when running inside a container the proc filesystem directory of the host should be set using system.hostfs setting to /hostfs. 

Logs reference 

Application 

The Windows application data stream provides events from the Windows Application event log. 

SUPPORTED OPERATING SYSTEMS 

System Integration Procedures 

Please provide the following information to CyTech:  

Collect logs from System instances (Enable Yes/No)  

System auth logs (log) (Enable Yes/No) 

System syslog logs (log) 

Collect events from the Windows event log (Enable Yes/No) 

Processors 

Security (Enable Yes/No) 

 

System (Enable Yes/No) 

Collect metrics from System instances (Enable Yes/No) 

System CPU metrics (Enable Yes/No) 

System diskio metrics (Enable Yes/No) 

System filesystem metrics 

System fsstat metrics (Enable Yes/No) 

System load metrics (Enable Yes/No) 

System memory metrics (Enable Yes/No) 

System network metrics (Enable Yes/No) 

System process metrics 

System process_summary metrics (Enable Yes/No) 

System socket_summary metrics (Enable Yes/No) 

System uptime metrics (Enable Yes/No) 

Collect logs from third-party REST API (experimental) (Enable Yes/No) 

Windows Application Events via Splunk Enterprise REST API (Enable Yes/No) 

Windows Security Events via Splunk Enterprise REST API (Enable Yes/No) 

Windows System Events via Splunk Enterprise REST API (Enable Yes/No) 

Team Viewer Integrations

Remote File Copy via TeamViewer  

Identifies an executable or script file remotely downloaded via a TeamViewer transfer session. 

Rule type: eql  

Rule indices:  

Severity: medium  

Risk score: 47  

Runs every: 5m  

Searches indices from: now-9m (Date Math format, see also Additional look-back time)  

Maximum alerts per execution: 100  

References:  

Tags:  

Version: 5  

Rule authors:  

Rule license: Elastic License v2 

Rule query 

file where event.type == "creation" and process.name : "TeamViewer.exe" and  

  file.extension : ("exe", "dll", "scr", "com", "bat", "ps1", "vbs", "vbe", "js", "wsh", "hta")  

Framework: MITRE ATT&CKTM 

 Source: https://www.elastic.co/guide/en/security/master/prebuilt-rule-0-14-2-remote-file-copy-via-teamviewer.html  

TeamViewer Integration Procedure 

 

Copy and paste the following Logstash configuration into the file:  

 

 
 

 

 

Source: ChatGPT  

Z Scaler Integrations

Introduction 

This integration is for Zscaler Internet Access logs. It can be used to receive logs sent by NSS log server on respective TCP ports. 

The log message is expected to be in JSON format. The data is mapped to ECS fields where applicable and the remaining fields are written under zscaler_zia.<data-stream-name>.*. 

 

Assumptions 

The procedures described in Section 3 assumes that a Log Collector has already been setup.  

Compatibility 

This package has been tested against Zscaler Internet Access version 6.1 

 

Requirements 

Steps for setting up NSS Feeds 

 

 

 

 

Steps for setting up Cloud NSS Feeds 

 

 

Please make sure to use the given response formats for NSS and Cloud NSS Feeds. 

Note: Please make sure to use latest version of given response formats. 

 

 

Zscaler Integration Procedures 

Please provide the following information to CyTech: 

Collect Zscaler Internet Access logs via TCP input 

Collect Zscaler Internet Access logs via HTTP Endpoint 

gcp

Google Cloud Platform

Google Cloud Platform Integration

The Google Cloud integration collects and parses Google Cloud Audit Logs, VPC Flow Logs, Firewall Rules Logs and Cloud DNS Logs that have been exported from Cloud Logging to a Google Pub/Sub topic sink and collects Google Cloud metrics and metadata from Google Cloud Monitoring.

Authentication

To use this Google Cloud Platform (GCP) integration, you need to set up a Service Account with a Role and a Service Account Key to access data on your GCP project.

Service Account

First, you need to create a Service Account. A Service Account (SA) is a particular type of Google account intended to represent a non-human user who needs to access the GCP resources.

The Elastic Agent uses the SA to access data on Google Cloud Platform using the Google APIs.

If you haven't already, this might be a good moment to check out the best practices for securing service accounts guide.

Role

You need to grant your Service Account (SA) access to Google Cloud Platform resources by assigning a role to the account. In order to assign minimal privileges, create a custom role that has only the privileges required by Agent. Those privileges are:

* Only required if Agent is expected to create a new subscription. If you create the subscriptions yourself you may omit these privileges. ** Only required if corresponding collection will be enabled.

After you have created the custom role, assign the role to your service account.

Service Account Keys

Now, with your brand new Service Account (SA) with access to Google Cloud Platform (GCP) resources, you need some credentials to associate with it: a Service Account Key.

From the list of SA:

Configure the Integration Settings

The next step is to configure the general integration settings used for all logs from the supported services (Audit, DNS, Firewall, and VPC Flow).

The "Project Id" and either the "Credentials File" or "Credentials JSON" will need to be provided in the integration UI when adding the Google Cloud Platform integration.

Project Id

The Project Id is the Google Cloud project ID where your resources exist.

Credentials File vs Json

Based on your preference, specify the information in either the Credentials File OR the Credentials JSON field.

Option 1: Credentials File

Save the JSON file with the private key in a secure location of the file system, and make sure that the Elastic Agent has at least read-only privileges to this file.

Specify the file path in the Elastic Agent integration UI in the "Credentials File" field. For example: /home/ubuntu/credentials.json.

Option 2: Credentials JSON

Specify the content of the JSON file you downloaded from Google Cloud Platform directly in the Credentials JSON field in the Elastic Agent integration.

Recommendations

Elastic recommends using Credentials File, as in this method the credential information doesn’t leave your Google Cloud Platform environment. When using Credentials JSON, the integration stores the info in Elasticsearch, and the access is controlled based on policy permissions or access to underlying Elasticsearch data.

Logs Collection Configuration

With a properly configured Service Account and the integration setting in place, it's time to start collecting some logs.

Requirements

You need to create a few dedicated Google Cloud resources before starting, in detail:

Elastic recommends separate Pub/Sub topics for each of the log types so that they can be parsed and stored in a specific data stream.

Here's an example of collecting Audit Logs using a Pub/Sub topic, a subscription, and a Log Router. We will create the resources in the Google Cloud Console and then configure the Google Cloud Platform integration.

On the Google Cloud Console

At a high level, the steps required are:

This is just an example; you will need to create your filter expression to select the log types you want to export to the Pub/Sub topic.

More example filters for different log types:

#
# VPC Flow: logs for specific subnet
#
resource.type="gce_subnetwork" AND
log_id("compute.googleapis.com/vpc_flows") AND
resource.labels.subnetwork_name"=[SUBNET_NAME]"
#
# Audit: Google Compute Engine firewall rule deletion
#
resource.type="gce_firewall_rule" AND
log_id("cloudaudit.googleapis.com/activity") AND
protoPayload.methodName:"firewalls.delete"
#
# DNS: all DNS queries
#
resource.type="dns_query"
#
# Firewall: logs for a given country
#
resource.type="gce_subnetwork" AND
log_id("compute.googleapis.com/firewall") AND
jsonPayload.remote_location.country=[COUNTRY_ISO_ALPHA_3]

Start working on your query using the Google Cloud Logs Explorer, so you can preview and pinpoint the exact log types you want to forward to your Elastic Stack.

To learn more, please read how to Build queries in the Logs Explorer, and take a look at the Sample queries using the Logs Explorer page in the Google Cloud docs.

On Kibana

Visit "Management" > "Integrations" > "Installed Integrations" > "Google Cloud Platform" and select the "Integration Policies" tab. Select the integration policy you previously created.

From the list of services, select "Google Cloud Platform (GCP) audit logs (gcp-pubsub)" and:

Troubleshooting

If you don't see Audit logs showing up, check the Agent logs to see if there are errors.

Common error types:

Missing Roles in the Service Account

If your Service Account (SA) does not have the required roles, you might find errors like this one in the elastic_agent.filebeat dataset:

failed to subscribe to pub/sub topic: failed to check if subscription exists: rpc error: code = PermissionDenied desc = User not authorized to perform this action.

Solution: make sure your SA has all the required roles.

Misconfigured Settings

If you specify the wrong "Topic field" or "Subscription Name", you might find errors like this one in the elastic_agent.filebeat dataset:

[elastic_agent.filebeat][error] failed to subscribe to pub/sub topic: failed to check if subscription exists: rpc error: code = InvalidArgument desc = Invalid resource name given (name=projects/project/subscriptions/projects/project/subscriptions/non-existent-sub). Refer to https://cloud.google.com/pubsub/docs/admin#resource_names for more information.

Solution: double check the integration settings.

Metrics Collection Configuration

With a properly configured Service Account and the integration setting in place, it's time to start collecting some metrics.

Requirements

No additional requirement is needed to collect metrics.

Troubleshooting

If you don't see metrics showing up, check the Agent logs to see if there are errors.

Common error types:

Period is lower than 60 seconds

Usual minimum collection period for GCP metrics is 60 seconds. Any value lower than that cause an error when retrieving the metric metadata. If an error happens, the affected metric is skipped at the metric collection stage, resulting in no data being sent.

Missing Roles in the Service Account

If your Service Account (SA) does not have required roles, you might find errors related to accessing GCP resources.

To check you may add Monitoring Viewer and Compute Viewer roles (built-in GCP roles) to your SA. These roles contain the permission added in the previous step and expand them with additional permissions. You can analyze additional missing permissions from the GCP Console > IAM > clicking on the down arrow near the roles on the same line of your SA > View analyzed permissions. From the shown table you can check which permissions from the role the SA is actively using. They should match what you configured in your custom role.

Misconfigured Settings

If you specify a wrong setting you will probably find errors related to missing GCP resources.

Make sure the settings are correct and the SA has proper permissions for the given "Project Id".

Logs

Audit

The audit dataset collects audit logs of administrative activities and accesses within your Google Cloud resources.

Exported fields

An example event for audit looks as following:

{
    "@timestamp": "2019-12-19T00:44:25.051Z",
    "agent": {
        "ephemeral_id": "a22278bb-5e1f-4ab7-b468-277c8c0b80a9",
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.7.1"
    },
    "client": {
        "user": {
            "email": "xxx@xxx.xxx"
        }
    },
    "cloud": {
        "project": {
            "id": "elastic-beats"
        },
        "provider": "gcp"
    },
    "data_stream": {
        "dataset": "gcp.audit",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.8.0"
    },
    "elastic_agent": {
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "snapshot": false,
        "version": "8.7.1"
    },
    "event": {
        "action": "beta.compute.instances.aggregatedList",
        "agent_id_status": "verified",
        "category": [
            "network",
            "configuration"
        ],
        "created": "2023-10-25T04:18:46.637Z",
        "dataset": "gcp.audit",
        "id": "yonau2dg2zi",
        "ingested": "2023-10-25T04:18:47Z",
        "kind": "event",
        "outcome": "success",
        "provider": "data_access",
        "type": [
            "access",
            "allowed"
        ]
    },
    "gcp": {
        "audit": {
            "authorization_info": [
                {
                    "granted": true,
                    "permission": "compute.instances.list",
                    "resource_attributes": {
                        "name": "projects/elastic-beats",
                        "service": "resourcemanager",
                        "type": "resourcemanager.projects"
                    }
                }
            ],
            "num_response_items": 61,
            "request": {
                "@type": "type.googleapis.com/compute.instances.aggregatedList"
            },
            "resource_location": {
                "current_locations": [
                    "global"
                ]
            },
            "resource_name": "projects/elastic-beats/global/instances",
            "response": {
                "@type": "core.k8s.io/v1.Status",
                "apiVersion": "v1",
                "details": {
                    "group": "batch",
                    "kind": "jobs",
                    "name": "gsuite-exporter-1589294700",
                    "uid": "2beff34a-945f-11ea-bacf-42010a80007f"
                },
                "kind": "Status",
                "status_value": "Success"
            },
            "type": "type.googleapis.com/google.cloud.audit.AuditLog"
        }
    },
    "input": {
        "type": "gcp-pubsub"
    },
    "log": {
        "level": "INFO",
        "logger": "projects/elastic-beats/logs/cloudaudit.googleapis.com%2Fdata_access"
    },
    "service": {
        "name": "compute.googleapis.com"
    },
    "source": {
        "ip": "192.168.1.1"
    },
    "tags": [
        "forwarded",
        "gcp-audit"
    ],
    "user_agent": {
        "device": {
            "name": "Mac"
        },
        "name": "Firefox",
        "original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:71.0) Gecko/20100101 Firefox/71.0,gzip(gfe),gzip(gfe)",
        "os": {
            "full": "Mac OS X 10.15",
            "name": "Mac OS X",
            "version": "10.15"
        },
        "version": "71.0."
    }
}

Firewall

The firewall dataset collects logs from Firewall Rules in your Virtual Private Cloud (VPC) networks.

Exported fields

An example event for firewall looks as following:

{
    "@timestamp": "2019-10-30T13:52:42.191Z",
    "agent": {
        "ephemeral_id": "175ae0b3-355c-4ca7-87ea-d5f1ee34102e",
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.7.1"
    },
    "cloud": {
        "availability_zone": "us-east1-b",
        "project": {
            "id": "test-beats"
        },
        "provider": "gcp",
        "region": "us-east1"
    },
    "data_stream": {
        "dataset": "gcp.firewall",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "address": "10.42.0.2",
        "domain": "test-windows",
        "ip": "10.42.0.2",
        "port": 3389
    },
    "ecs": {
        "version": "8.8.0"
    },
    "elastic_agent": {
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "snapshot": false,
        "version": "8.7.1"
    },
    "event": {
        "action": "firewall-rule",
        "agent_id_status": "verified",
        "category": [
            "network"
        ],
        "created": "2023-10-25T04:20:37.182Z",
        "dataset": "gcp.firewall",
        "id": "1f21ciqfpfssuo",
        "ingested": "2023-10-25T04:20:41Z",
        "kind": "event",
        "type": [
            "allowed",
            "connection"
        ]
    },
    "gcp": {
        "destination": {
            "instance": {
                "project_id": "test-beats",
                "region": "us-east1",
                "zone": "us-east1-b"
            },
            "vpc": {
                "project_id": "test-beats",
                "subnetwork_name": "windows-isolated",
                "vpc_name": "windows-isolated"
            }
        },
        "firewall": {
            "rule_details": {
                "action": "ALLOW",
                "direction": "INGRESS",
                "ip_port_info": [
                    {
                        "ip_protocol": "TCP",
                        "port_range": [
                            "3389"
                        ]
                    }
                ],
                "priority": 1000,
                "source_range": [
                    "0.0.0.0/0"
                ],
                "target_tag": [
                    "allow-rdp"
                ]
            }
        }
    },
    "input": {
        "type": "gcp-pubsub"
    },
    "log": {
        "logger": "projects/test-beats/logs/compute.googleapis.com%2Ffirewall"
    },
    "network": {
        "community_id": "1:OdLB9eXsBDLz8m97ao4LepX6q+4=",
        "direction": "inbound",
        "iana_number": "6",
        "name": "windows-isolated",
        "transport": "tcp",
        "type": "ipv4"
    },
    "related": {
        "ip": [
            "192.168.2.126",
            "10.42.0.2"
        ]
    },
    "rule": {
        "name": "network:windows-isolated/firewall:windows-isolated-allow-rdp"
    },
    "source": {
        "address": "192.168.2.126",
        "geo": {
            "continent_name": "Asia",
            "country_name": "omn"
        },
        "ip": "192.168.2.126",
        "port": 64853
    },
    "tags": [
        "forwarded",
        "gcp-firewall"
    ]
}

VPC Flow

The vpcflow dataset collects logs sent from and received by VM instances, including instances used as GKE nodes.

Exported fields

An example event for vpcflow looks as following:

{
    "@timestamp": "2019-06-14T03:50:10.845Z",
    "agent": {
        "ephemeral_id": "0b8165a2-0e25-4e9a-bb68-271697e0993f",
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.7.1"
    },
    "cloud": {
        "availability_zone": "us-east1-b",
        "instance": {
            "name": "kibana"
        },
        "project": {
            "id": "my-sample-project"
        },
        "provider": "gcp",
        "region": "us-east1"
    },
    "data_stream": {
        "dataset": "gcp.vpcflow",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "address": "10.139.99.242",
        "domain": "elasticsearch",
        "ip": "10.139.99.242",
        "port": 9200
    },
    "ecs": {
        "version": "8.8.0"
    },
    "elastic_agent": {
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "snapshot": false,
        "version": "8.7.1"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "network"
        ],
        "created": "2023-10-25T04:21:42.006Z",
        "dataset": "gcp.vpcflow",
        "end": "2019-06-14T03:49:51.821056075Z",
        "id": "ut8lbrffooxz5",
        "ingested": "2023-10-25T04:21:43Z",
        "kind": "event",
        "start": "2019-06-14T03:40:20.510622432Z",
        "type": [
            "connection"
        ]
    },
    "gcp": {
        "destination": {
            "instance": {
                "project_id": "my-sample-project",
                "region": "us-east1",
                "zone": "us-east1-b"
            },
            "vpc": {
                "project_id": "my-sample-project",
                "subnetwork_name": "default",
                "vpc_name": "default"
            }
        },
        "source": {
            "instance": {
                "project_id": "my-sample-project",
                "region": "us-east1",
                "zone": "us-east1-b"
            },
            "vpc": {
                "project_id": "my-sample-project",
                "subnetwork_name": "default",
                "vpc_name": "default"
            }
        },
        "vpcflow": {
            "reporter": "DEST",
            "rtt": {
                "ms": 201
            }
        }
    },
    "input": {
        "type": "gcp-pubsub"
    },
    "log": {
        "logger": "projects/my-sample-project/logs/compute.googleapis.com%2Fvpc_flows"
    },
    "network": {
        "bytes": 11773,
        "community_id": "1:FYaJFSEAKLcBCMFoT6sR5TMHf/s=",
        "direction": "internal",
        "iana_number": "6",
        "name": "default",
        "packets": 94,
        "transport": "tcp",
        "type": "ipv4"
    },
    "related": {
        "ip": [
            "67.43.156.13",
            "10.139.99.242"
        ]
    },
    "source": {
        "address": "67.43.156.13",
        "as": {
            "number": 35908
        },
        "bytes": 11773,
        "domain": "kibana",
        "geo": {
            "continent_name": "Asia",
            "country_iso_code": "BT",
            "country_name": "Bhutan",
            "location": {
                "lat": 27.5,
                "lon": 90.5
            }
        },
        "ip": "67.43.156.13",
        "packets": 94,
        "port": 33576
    },
    "tags": [
        "forwarded",
        "gcp-vpcflow"
    ]
}

DNS

The dns dataset collects queries that name servers resolve for your Virtual Private Cloud (VPC) networks, as well as queries from an external entity directly to a public zone.

Exported fields

An example event for dns looks as following:

{
    "@timestamp": "2021-12-12T15:59:40.446Z",
    "agent": {
        "ephemeral_id": "fd6c4189-cbc6-493a-acfb-c9e7b2b7588c",
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.7.1"
    },
    "cloud": {
        "project": {
            "id": "key-reference-123456"
        },
        "provider": "gcp",
        "region": "global"
    },
    "data_stream": {
        "dataset": "gcp.dns",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "address": "216.239.32.106",
        "ip": "216.239.32.106"
    },
    "dns": {
        "answers": [
            {
                "class": "IN",
                "data": "67.43.156.13",
                "name": "asdf.gcp.example.com.",
                "ttl": 300,
                "type": "A"
            }
        ],
        "question": {
            "name": "asdf.gcp.example.com",
            "registered_domain": "example.com",
            "subdomain": "asdf.gcp",
            "top_level_domain": "com",
            "type": "A"
        },
        "resolved_ip": [
            "67.43.156.13"
        ],
        "response_code": "NOERROR"
    },
    "ecs": {
        "version": "8.8.0"
    },
    "elastic_agent": {
        "id": "c6b95057-2f5d-4b8f-b4b5-37cbdb995dec",
        "snapshot": false,
        "version": "8.7.1"
    },
    "event": {
        "action": "dns-query",
        "agent_id_status": "verified",
        "category": "network",
        "created": "2023-10-25T04:19:40.300Z",
        "dataset": "gcp.dns",
        "id": "zir4wud11tm",
        "ingested": "2023-10-25T04:19:41Z",
        "kind": "event",
        "outcome": "success"
    },
    "gcp": {
        "dns": {
            "auth_answer": true,
            "destination_ip": "216.239.32.106",
            "protocol": "UDP",
            "query_name": "asdf.gcp.example.com.",
            "query_type": "A",
            "response_code": "NOERROR",
            "server_latency": 0,
            "source_type": "internet",
            "target_type": "public-zone"
        }
    },
    "input": {
        "type": "gcp-pubsub"
    },
    "log": {
        "level": "INFO",
        "logger": "projects/key-reference-123456/logs/dns.googleapis.com%2Fdns_queries"
    },
    "network": {
        "iana_number": "17",
        "protocol": "dns",
        "transport": "udp"
    },
    "related": {
        "hosts": [
            "asdf.gcp.example.com"
        ],
        "ip": [
            "67.43.156.13",
            "216.239.32.106"
        ]
    },
    "tags": [
        "forwarded",
        "gcp-dns"
    ]
}

Loadbalancing Logs

The loadbalancing_logs dataset collects logs of the requests sent to and handled by GCP Load Balancers.

Exported fields

An example event for loadbalancing looks as following:

{
    "@timestamp": "2020-06-08T23:41:30.078Z",
    "agent": {
        "ephemeral_id": "f4dde373-2ff7-464b-afdb-da94763f219b",
        "id": "5d3eee86-91a9-4afa-af92-c6b79bd866c0",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.6.0"
    },
    "cloud": {
        "project": {
            "id": "PROJECT_ID"
        },
        "region": "global"
    },
    "data_stream": {
        "dataset": "gcp.loadbalancing_logs",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "address": "81.2.69.193",
        "ip": "81.2.69.193",
        "nat": {
            "ip": "10.5.3.1",
            "port": 9090
        },
        "port": 8080
    },
    "ecs": {
        "version": "8.8.0"
    },
    "elastic_agent": {
        "id": "5d3eee86-91a9-4afa-af92-c6b79bd866c0",
        "snapshot": true,
        "version": "8.6.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": "network",
        "created": "2020-06-08T23:41:30.588Z",
        "dataset": "gcp.loadbalancing_logs",
        "id": "1oek5rg3l3fxj7",
        "ingested": "2023-01-13T15:02:22Z",
        "kind": "event",
        "type": "info"
    },
    "gcp": {
        "load_balancer": {
            "backend_service_name": "",
            "cache_hit": true,
            "cache_id": "SFO-fbae48ad",
            "cache_lookup": true,
            "forwarding_rule_name": "FORWARDING_RULE_NAME",
            "status_details": "response_from_cache",
            "target_proxy_name": "TARGET_PROXY_NAME",
            "url_map_name": "URL_MAP_NAME"
        }
    },
    "http": {
        "request": {
            "bytes": 577,
            "method": "GET",
            "referrer": "https://developer.mozilla.org/en-US/docs/Web/JavaScript"
        },
        "response": {
            "bytes": 157,
            "status_code": 304
        },
        "version": "2.0"
    },
    "input": {
        "type": "gcp-pubsub"
    },
    "log": {
        "level": "INFO",
        "logger": "projects/PROJECT_ID/logs/requests"
    },
    "network": {
        "protocol": "http"
    },
    "related": {
        "ip": [
            "89.160.20.156",
            "81.2.69.193",
            "10.5.3.1"
        ]
    },
    "source": {
        "address": "89.160.20.156",
        "as": {
            "number": 29518,
            "organization": {
                "name": "Bredband2 AB"
            }
        },
        "geo": {
            "city_name": "Linköping",
            "continent_name": "Europe",
            "country_iso_code": "SE",
            "country_name": "Sweden",
            "location": {
                "lat": 58.4167,
                "lon": 15.6167
            },
            "region_iso_code": "SE-E",
            "region_name": "Östergötland County"
        },
        "ip": "89.160.20.156",
        "port": 9989
    },
    "tags": [
        "forwarded",
        "gcp-loadbalancing_logs"
    ],
    "url": {
        "domain": "81.2.69.193",
        "extension": "jpg",
        "original": "http://81.2.69.193:8080/static/us/three-cats.jpg",
        "path": "/static/us/three-cats.jpg",
        "port": 8080,
        "scheme": "http"
    },
    "user_agent": {
        "device": {
            "name": "Mac"
        },
        "name": "Chrome",
        "original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36",
        "os": {
            "full": "Mac OS X 10.14.6",
            "name": "Mac OS X",
            "version": "10.14.6"
        },
        "version": "83.0.4103.61"
    }
}

Metrics

Billing

The billing dataset collects GCP Billing information from Google Cloud BigQuery daily cost detail table.

Exported fields

An example event for billing looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "01475F-5B1080-1137E7"
        },
        "project": {
            "id": "elastic-bi",
            "name": "elastic-containerlib-prod"
        },
        "provider": "gcp"
    },
    "event": {
        "dataset": "gcp.billing",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "billing": {
            "billing_account_id": "01475F-5B1080-1137E7",
            "cost_type": "regular",
            "invoice_month": "202106",
            "project_id": "containerlib-prod-12763",
            "project_name": "elastic-containerlib-prod",
            "total": 4717.170681,
            "sku_id": "0D56-2F80-52A5",
            "service_id": "6F81-5844-456A",
            "sku_description": "Network Inter Region Ingress from Jakarta to Americas",
            "service_description": "Compute Engine",
            "effective_price": 0.00292353,
            "tags": [
                {
                    "key": "stage",
                    "value": "prod"
                },
                {
                    "key": "size",
                    "value": "standard"
                }
            ]
        }
    },
    "metricset": {
        "name": "billing",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

Compute

The compute dataset is designed to fetch metrics for Compute Engine Virtual Machines in Google Cloud Platform.

Exported fields

An example event for compute looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.compute",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "compute": {
            "firewall": {
                "dropped": {
                    "bytes": 421
                },
                "dropped_packets_count": {
                    "value": 4
                }
            },
            "instance": {
                "cpu": {
                    "reserved_cores": {
                        "value": 1
                    },
                    "usage": {
                        "pct": 0.07259952346383708
                    },
                    "usage_time": {
                        "sec": 4.355971407830225
                    }
                },
                "memory": {
                    "balloon": {
                        "ram_size": {
                            "value": 4128378880
                        },
                        "ram_used": {
                            "value": 2190848000
                        },
                        "swap_in": {
                            "bytes": 0
                        },
                        "swap_out": {
                            "bytes": 0
                        }
                    }
                },
                "uptime": {
                    "sec": 60.00000000000091
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "compute",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

Dataproc

The dataproc dataset is designed to fetch metrics from Dataproc in Google Cloud Platform.

Exported fields

An example event for dataproc looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.dataproc",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "dataproc": {
            "cluster": {
                "hdfs": {
                    "datanodes": {
                        "count": 15
                    }
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "dataproc",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

Firestore

The firestore dataset fetches metrics from Firestore in Google Cloud Platform.

Exported fields

An example event for firestore looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.firestore",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "firestore": {
            "document": {
                "delete": {
                    "count": 3
                },
                "read": {
                    "count": 10
                },
                "write": {
                    "count": 1
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "firestore",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

GKE

The gke dataset is designed to fetch metrics from GKE in Google Cloud Platform.

Exported fields

An example event for gke looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.gke",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "gke": {
            "container": {
                "cpu": {
                    "core_usage_time": {
                        "sec": 15
                    }
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "gke",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

Loadbalancing Metrics

The loadbalancing_metrics dataset is designed to fetch HTTPS, HTTP, and Layer 3 metrics from Load Balancing in Google Cloud Platform.

Exported fields

An example event for loadbalancing looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-observability"
        },
        "provider": "gcp",
        "region": "us-central1",
        "availability_zone": "us-central1-a"
    },
    "event": {
        "dataset": "gcp.loadbalancing_metrics",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "labels": {
            "metrics": {
                "client_network": "ocp-be-c5kjr-network",
                "client_subnetwork": "ocp-be-c5kjr-worker-subnet",
                "client_zone": "us-central1-a"
            },
            "resource": {
                "backend_name": "ocp-be-c5kjr-master-us-central1-a",
                "backend_scope": "us-central1-a",
                "backend_scope_type": "ZONE",
                "backend_subnetwork_name": "ocp-be-c5kjr-master-subnet",
                "backend_target_name": "ocp-be-c5kjr-api-internal",
                "backend_target_type": "BACKEND_SERVICE",
                "backend_type": "INSTANCE_GROUP",
                "forwarding_rule_name": "ocp-be-c5kjr-api-internal",
                "load_balancer_name": "ocp-be-c5kjr-api-internal",
                "network_name": "ocp-be-c5kjr-network",
                "region": "us-central1"
            }
        },
        "loadbalancing_metrics": {
            "l3": {
                "internal": {
                    "egress_packets": {
                        "count": 100
                    },
                    "egress": {
                        "bytes": 1247589
                    }
                }
            }
        }
    },
    "metricset": {
        "name": "loadbalancing",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

Redis

The redis dataset is designed to fetch metrics from GCP Memorystore for Redis in Google Cloud Platform.

Exported fields

An example event for redis looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.redis",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "redis": {
            "clients": {
                "blocked": {
                    "count": 4
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "metrics",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

Storage

The storage dataset fetches metrics from Storage in Google Cloud Platform.

Exported fields

An example event for storage looks as following:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "cloud": {
        "account": {
            "id": "elastic-obs-integrations-dev",
            "name": "elastic-obs-integrations-dev"
        },
        "instance": {
            "id": "4751091017865185079",
            "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
        },
        "machine": {
            "type": "e2-medium"
        },
        "provider": "gcp",
        "availability_zone": "us-central1-c",
        "region": "us-central1"
    },
    "event": {
        "dataset": "gcp.storage",
        "duration": 115000,
        "module": "gcp"
    },
    "gcp": {
        "storage": {
            "storage": {
                "total": {
                    "bytes": 4472520191
                }
            },
            "network": {
                "received": {
                    "bytes": 4472520191
                }
            }
        },
        "labels": {
            "user": {
                "goog-gke-node": ""
            }
        }
    },
    "host": {
        "id": "4751091017865185079",
        "name": "gke-cluster-1-default-pool-6617a8aa-5clh"
    },
    "metricset": {
        "name": "storage",
        "period": 10000
    },
    "service": {
        "type": "gcp"
    }
}

VMware vSphere Integration

This integration periodically fetches logs and metrics from vSphere vCenter servers. 

 Compatibility 
The integration uses the Govmomi library to collect metrics and logs from any Vmware SDK URL (ESXi/VCenter). This library is built for and tested against ESXi and vCenter 6.5, 6.7 and 7.0. 

 Installation Guide:  

VMware vSphere 7.0 Installation 
Govmomi Library 

 Integration Process 

Go> Cyber Incident Management (XDR and MDR) 

 

 Go> Cyber Incident Management (XDR and MDR)> Settings 

 

 Go> Cyber Incident Management (XDR and MDR)> Settings> Integration 

 

Go> Cyber Incident Management (XDR and MDR)> Settings> Integration> 
In search bar type “Vmware” 

 

 Click Add Agent 

 

 Choose your Log Collector 

 

 

Click the vSphere logs and metrics 

 

 

  Keep it as is  

 

 

Enter the IP address and port 

 

Example: https://127.0.0.1:8989/sdk 
127.0.0.1: This is the IP address of the local machine (localhost). 
8989: This is the port number on which the SDK service is running. (Keep it as is) 
/sdk: This indicates that the SDK is accessible at this path. (Keep it as is) 

Notes: To add multiple hosts, enter each IP address following the same format (https://<IP_or_hostname>:port/sdk) and press enter. 

 

Enter the Username and password of vSphere account 

 

Notes: The insecure option bypasses the verification of the server's certificate chain, which can be useful in certain scenarios but comes with significant security risks. It is recommended to use this option only when necessary and in environments where security concerns are minimal. 

 Logs collection 

 

 

 Collect logs from vSphere via UDP 

Tags: Click the given tags  

UDP host to listen on: This is the IP address of the machine where the log collector is running. 

UDP port to listen on: This is the port on which the log collector will listen for incoming log data. (Keep it as is) 

Notes: Enabling "Preserve original event" ensures raw log data is always available, crucial for troubleshooting, compliance, and verifying log accuracy. It adds raw data to event.original, doubling storage needs and potentially slowing processing if storage isn't scaled, impacting efficiency. 

 

 Collect logs from vSphere via TCP 

 

 

Tags: Click the given tags  

TCP host to listen on: This is the IP address of the machine where the log collector is running. 

TCP port to listen on: This is the port on which the log collector will listen for incoming log data. (Keep it as is) 

Notes: Enabling "Preserve original event" ensures raw log data is always available, crucial for troubleshooting, compliance, and verifying log accuracy. It adds raw data to event.original, doubling storage needs and potentially slowing processing if storage isn't scaled, impacting efficiency. 

 
Click Next to complete the integration. 

SentinelOne Integrations

The SentinelOne integration collects and parses data from SentinelOne REST APIs. This integration also offers the capability to perform response actions on SentinelOne hosts directly through the Elastic Security interface 

Compatibility

This module has been tested against SentinelOne Management Console API version 2.1.

API token

To collect data from SentinelOne APIs, you must have an API token. To create an API token, follow these steps:

1. Log in to the SentinelOne Management Console as an Admin.

2. Navigate to Logged User Account from top right panel in the navigation bar.

3. Click My User.

4. In the API token section, click Generate.

Note

The API token generated by the user is time-limited. To rotate a new token, log in with the dedicated admin account.

Integrate on Elastic

1. Add SentinelOne console URL (https://<DomainName>.sentinelone.net/, where <DomainName> is the domain name of your SentinelOne account.)

2. Add API token

If you need further assistance, kindly contact our support at info@cytechint.com for prompt assistance and guidance.

Custom Windows Event Logs - Integration

Custom Windows Event Logs

Collect and parse logs from any Windows event log channel with Elastic Agent.

The custom Windows event log package allows you to ingest events from any Windows event log channel. You can get a list of available event log channels by running Get-WinEvent -ListLog * | Format-List -Property LogName in PowerShell on Windows Vista or newer. If Get-WinEvent is not available, Get-EventLog * may be used. 

By executing this command in the powershell(administrator), it will list the log names that is being used.

 

Add a channel name in the Channel Name text field (e.g Application). 

 

 

Windows Event Forwarding to Linux server using Nxlog

Introduction

Windows Event Forwarding (WEF) allows the collection of event logs from multiple Windows machines and their forwarding to a centralized server. Using Nxlog, you can send these logs to a Linux server for storage and analysis. This documentation provides a step-by-step guide to set up Windows Event Forwarding using Nxlog to send logs to a Linux server.

Prerequisites

• Windows Server or Workstation: The machine that will send logs.
• Linux Server: The machine that will receive logs.
• Nxlog: Download the latest version of Nxlog for Windows from Nxlog's official website.
• Network Connectivity: Ensure both machines can communicate over the network.
• Rsyslog: Download the latest version of Rsyslog for Linux server or workstation.
  

Installing Nxlog on Windows

Configuring Nxlog on Windows

1. Open Configuration File:

   • Edit the Nxlog configuration file located at C:\Program Files\nxlog\conf\nxlog.conf.

2. Configure File:

   • Add the following lines to capture Windows Event Logs and send the logs : 
     # Input Module

     <Input eventlog>

         Module im_msvistalog

         ReadFromLast True

         <QueryXML>

     <QueryList>

     <Query Id='1'>

     <Select Path='Application'>*</Select>

     <Select Path='Security'>*</Select>

     <Select Path='System'>*</Select>

     </Query>

     </QueryList>

          </QueryXML>

     </Input>

     
     

     # Output Module

     <Output out>

         Module om_udp

         Host 192.168.20.24  

         Port 514 

        # Exec $raw_event = "<" + $syslog_severity + ">" + $time + " " + $hostname + " " + $procname + ": " + $raw_event; 

         Exec parse_syslog_ietf();

     </Output>

     
     

     # Route

     <Route r>

         Path eventlog => out

     </Route>

     
     

     # Include any other necessary modules/extensions

     <Extension _syslog>

         Module      xm_syslog

     </Extension>

      

Installing Rsyslog on Linux

• Install Rsyslog:

  • For Ubuntu, run:
    sudo apt update sudo apt install rsyslog

• Enable Rsyslog:

  • Ensure Rsyslog is enabled and started:
    sudo systemctl enable rsyslog sudo systemctl start rsyslog

Configuring Rsyslog on Linux

1. Open Configuration File:

   • Edit /etc/rsyslog.conf or create a new config file in /etc/rsyslog.d/.

2. Configure Rsyslog to Listen for UDP:module(load="imudp") # Load UDP listener input(type="imudp" port="514")
   

3. Define Output File:

   • Specify where to store the incoming logs:
   *.* /var/log/windows_events.log

4. Save and Exit:

   • Save the configuration file and restart Rsyslog:
     sudo systemctl restart rsyslog

Firewall Configuration

Windows Firewall

1. Open Windows Defender Firewall:

   • Go to Control Panel > System and Security > Windows Defender Firewall.

2. Allow Port 514:

   • In the left pane, click Advanced settings.
   • Select Inbound Rules and click on New Rule.
   • Choose Port, then click Next.
   • Select UDP and enter 514 in the Specific local ports field.
   • Allow the connection and complete the rule setup.

Firewalld Configuration on Linux

1. Open Port 514 for UDP:

   sudo firewall-cmd --permanent --add-port=514/udp

2. Reload Firewalld:

   sudo firewall-cmd --reload

3. Verify Open Ports:

   sudo firewall-cmd --list-all

Verifying Event Forwarding

1. Check Nxlog Status on Windows:

   nxlog -v
   

2. Monitor Logs on Linux:

   • Use the following command to view the log file:
   tail -f /var/log/windows_events.log

3. Review Rsyslog Logs:

   • If issues arise, check Rsyslog logs located at /var/log/syslog or /var/log/messages.

Windows Event Forwarding to Linux server using Powershell script

Overview

This PowerShell script forwards Windows event logs to a Linux server using the syslog protocol. It captures specific event logs, sends them to the specified syslog server, and ensures that duplicate events are not sent.

Prerequisites

• PowerShell on Windows
• Syslog server running on Linux (e.g., Rocky Linux) with an accessible IP
• UDP port 514 open for communication

Powershell Script

Save the file as .ps1 (e.g sendlogs.ps1).

Script: 

# Define the syslog server IP address and port
$syslogServerIP = "192.168.20.24"  # Replace with your Rocky server's IP
$syslogPort = 514

# File to store last sent event info.
# Change this directory if necessary
$logFilePath = "C:\Users\Administrator\Desktop\lastEventInfo.txt"   

# Initialize last sent events
$lastSentEvents = @{}

# Check if the file exists and read the last sent events
if (Test-Path $logFilePath) {
    Write-Host "Loading last sent events from file."
    $lastSentEvents = Get-Content $logFilePath | ConvertFrom-Json
}

# Loop for sending logs
while ($true) {
    Write-Host "Checking for new logs..."

    # Define the logs you want to forward
    $logNames = @("Application", "Security", "Setup", "System")

    # Loop through each log
    foreach ($logName in $logNames) {
        Write-Host "Processing log: $logName"

        # Get new logs
        $logs = Get-WinEvent -LogName $logName | Sort-Object TimeCreated

        foreach ($log in $logs) {
            # Create a unique key based on Event ID and TimeCreated
            $eventKey = "$($log.Id)-$($log.TimeCreated.Ticks)"

            # Check if the event has already been sent
            if (-not $lastSentEvents.ContainsKey($eventKey)) {
                # Create syslog message format
                $message = "<134>" + $log.TimeCreated.ToString("yyyy-MM-dd HH:mm:ss") + " " + $log.ProviderName + ": " + $log.Message

                # Send the message to the syslog server
                $client = New-Object System.Net.Sockets.UdpClient
                $client.Connect($syslogServerIP, $syslogPort)
                $bytes = [System.Text.Encoding]::ASCII.GetBytes($message)
                $client.Send($bytes, $bytes.Length)
                $client.Close()

                # Mark the event as sent
                $lastSentEvents[$eventKey] = $true
                Write-Host "Sent log: $message"
            } else {
                Write-Host "Log already sent: $eventKey"
            }
        }
    }

    # Save the last sent events to the file
    $lastSentEvents | ConvertTo-Json | Set-Content -Path $logFilePath
    Write-Host "Last sent events updated."

    # Wait for 1 second before running again
    Start-Sleep -Seconds 1
}

 

Script Components

1. Define Variables:

   • $syslogServerIP: IP address of the Linux syslog server.
   • $syslogPort: Port number for syslog (default is 514).
   • $logFilePath: Path to store the last sent event information.

2. Initialize Last Sent Events:

   • Loads previously sent events from a file, if it exists.

3. Main Loop:

   • Continuously checks for new logs from specified log categories: Application, Security, Setup, and System.

4. Processing Logs:

   • Retrieves new logs, sorts them, and creates a unique key based on the event ID and timestamp.
   • Sends new logs to the syslog server in a specified message format.
   • Updates the log file with the newly sent events.

5. Error Handling:

   • Logs messages indicating whether an event has been sent or is a duplicate.

Usage

1. Update the $syslogServerIP and $logFilePath variables.
2. Run the script in PowerShell. It will run indefinitely, checking for new logs every second.

Task Scheduler

Make sure to enable the "Run with highest  privileges"

Add a new action

1. Fill in the Program/Script Text Box with: powershell.exe

2. Fill in the Add arguments Text Box with: -ExecutionPolicy Bypass -File "C:\Users\Administrator\Desktop\sendlogs.ps1" 

Settings Configuration

After the creating the task, you can enable the script by activating the task.

Sophos Integration

Overview

The Sophos Central integration allows you to monitor Alerts and Events logs. Sophos Central is a cloud-native application with high availability. It is a cybersecurity management platform hosted on public cloud platforms. Each Sophos Central account is hosted in a named region. Sophos Central uses well-known, widely used, and industry-standard software libraries to mitigate common vulnerabilities.

Use the Sophos Central integration to collect logs across Sophos Central managed by your Sophos account. Visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference data when troubleshooting an issue.

Compatibility

The Sophos Central Application does not feature version numbers. This integration has been configured and tested against Sophos Central SIEM Integration API version v1.

Requirements

You need Elasticsearch for storing and searching your data, and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.

Setup

Elastic Integration for Sophos Central Settings

The Elastic Integration for Sophos Central requires the following Authentication Settings in order to connect to the Target service:

• Client ID
• Client Secret
• Grant Type
• Scope
• Tenant ID
• Token URL

NOTE: Sophos central supports logs only upto last 24 hrs.

Step 1 - Create a service principal

We will show you how you can sign in to Sophos Central Admin and create a service principal. You need to have the Super Admin role to do this.

Step 1a - Sophos Central Admin

Sign in to Sophos Central Admin. Go to https://central.sophos.com/manage.

Click 'Global Settings' and then click the "API Credentials" link.

Step 1b - Add a new set of credentials

Supply a name for your credential set and a description, then click 'Add' as shown in the example below. 

Step 1c - Grab your client ID and secret

Click 'Copy' to note down the client ID. Also show the client secret.

Click 'Copy' to note down the client secret.

⚠️ WARNING: It is your responsibility to store your client ID and secret securely. If these are lost or stolen, an attacker will be able to call APIs on your behalf and steal your data or cause damage.

 

If you need further assistance, kindly contact our support at info@cytechint.com for prompt assistance and guidance.

Atlassian Bitbucket Integrations (New)

Introduction 

The Bitbucket integration collects audit logs from the audit log files or the audit API. 

Reference:  https://developer.atlassian.com/server/bitbucket/reference/rest-api/  

Assumptions 

The procedures described in Section 3 assume that a Log Collector has already been set up.  

Requirements 

For more information on auditing in Bitbucket and how it can be configured, see View and configure the audit log on Atlassian's website. 

Reference: https://confluence.atlassian.com/bitbucketserver/view-and-configure-the-audit-log-776640417.html  

Logs 

Audit 

The Confluence integration collects audit logs from the audit log files or the audit API from self-hosted Confluence Data Center. It has been tested with Confluence 7.14.2 but is expected to work with newer versions. As of version 1.2.0, this integration added experimental support for Atlassian Confluence Cloud. JIRA Cloud only supports Basic Auth using username and a Personal Access Token. 

Atlassian Bitbucket Integration Procedures 

Please provide the following information to CyTech: 

Collect Bitbucket audit logs via log files 

Collect Bitbucket audit logs via API (Enable Yes/No) 

 

 

Create Access Token:

Access Tokens are single-purpose access tokens (or passwords) with access to a single workspace with limited permissions (specified at creation time). Use tokens for tasks such as scripting, CI/CD tools, and testing Bitbucket integrations or Marketplace apps while in development.

To create an Access Token:

1. At bitbucket.org, navigate to the target workspace for the Access Token. This workspace is the only one that the Workspace Access Token can access.

2. On the sidebar, select Settings.

3. On the sidebar, under Security, select Access tokens.

4. Select Create Workspace Access Token.

5. Give the Workspace Access Token a name, usually related to the app or task that will use the token.

6. Select the permissions the Access Token needs. Note give the Access Token admin permission.

7. Select the Create button. The page will display the Workspace Access Token created dialog.

8. Copy the generated token and either record or paste it into the app that requires access. The token is only displayed once and can't be retrieved later. 

If you need further assistance, kindly contact our support at info@cytechint.com for prompt assistance and guidance.

Palo Alto Cortex XDR Integration

Palo Alto Cortex XDR Integration

Using the Cortex XDR APIs, you can integrate Cortex XDR with third-party apps or services to ingest alerts and to leverage alert stitching and investigation capabilities. The APIs allows you to manage incidents in a ticketing or automation system of your choice by reviewing and editing the incident's details, status, and assignee. Using the APIs, you can also retrieve information on the endpoints, create installation package, perform response actions directly on the endpoint and more.

Alerts

The Cortex XDR Alerts API is used to retrieve alerts generated by Cortex XDR based on raw endpoint data. A single alert might include one or more local endpoint events, each event generating its own document on Elasticsearch.

The Palo Alto XDR integration requires both an API key and API key ID, both which can be retrieved from the Cortex XDR UI.

API

Before you can begin using Cortex XDR APIs, you must generate the following items from the Cortex XDR app:

Value	Description
API Key	The API Key is your unique identifier used as the Authorization:{key} header required for authenticating API calls. Depending on your desired security level, you can generate two types of API keys, Advanced or Standard, from your Cortex XDR app.
API Key ID	The API Key ID is your unique token used to authenticate the API Key. The header used when running an API call is x-xdr-auth-id:{key_id}.
FQDN	The FQDN is a unique host and domain name associated with each tenant. When you generate the API Key and Key ID, you are assigned an individual FQDN.

Create Cortex API Key:

The following steps describe how to generate the necessary key values:

1. Get your Cortex XDR API Key:

   1. In Cortex XDR, navigate to Settings > Configurations > Integrations > API Keys.
   2. Select + New Key.
   3. Choose the type of API Key you want to generate based on your desired security level: Advanced or Standard. The Advanced API key hashes the key useing a nonce, a random string, and a timestamp to prevent replay attacks. cURL does not support this but is suitable with scripts. Use the provided script to create the advanced API authentication token.

Note: To integrate with Cortex XSOAR you must generate an Advanced Key.

4. If you want to define a time limit on the API key authentication, mark Enable Expiration Date and select the expiration date and time. Navigate to Settings > Configurations > Integrations > API Keys to track the Expiration Time field for each API key. In addition, Cortex XDR displays a API Key Expiration notification in the Notification Center one week and one day prior to the defined expiration date.
5. Provide a comment that describes the purpose for the API key, if desired.
6. Select the desired level of access for this key. You can select from the list of existing Roles, or you can select Custom to set the permissions on a more granular level. Roles are available according what was defined in the hub as described in the Manage Roles section of the Cortex XDR Administrator’s Guide.
7. Generate the API Key.
8. Copy the API key, and then click Done. This value represents your unique Authorization:{key}.

Note: You will not be able to view the API Key again after you complete this step. Ensure that you copy it before closing the notification.

Cortex XDR API Key ID

Get your Cortex XDR API Key ID.

1. 1. In the API Keys table, locate the ID field.
   2. Note your corresponding ID number. This value represents the x-xdr-auth-id:{key_id} token.

Note: This key id will be used for integrations with elastic.

Cortex XDR API FQDN

Get your FQDN.

1. Right-click your API key and select View Examples.
2. Copy the CURL Example URL. The example contains your unique FQDN:

https://api-{fqdn}/public_api/v1/{name of api}/{name of call}/ You can use the CURL Example URL to run the APIs.

 

If you need further assistance, kindly contact our support at info@cytechint.com for prompt assistance and guidance.

Active Directory Integrations

Introduction 

Elastic Stack security features can be configured to authenticate users through Active Directory by using LDAP to communicate with the directory. Active Directory realms are similar to LDAP realms, as they both store users and groups in a hierarchical structure, which includes containers such as organizational units (OU), organizations (O), and domain components (DC). 

The security features support authentication based on Active Directory security groups, but not distribution groups. When authenticating users, the username entered must match the sAMAccountName or userPrincipalName, not the common name (cn). The realm authenticates users via an LDAP bind request, searches for their entry in Active Directory, and retrieves their group memberships from the tokenGroups attribute to assign appropriate roles.

---

Requirements

Elastic Agent must be installed. For more details and installation instructions, please refer to the Elastic Agent Installation Guide.

Installing and managing an Elastic Agent:

There are several options for installing and managing Elastic Agent:

Install a Fleet-managed Elastic Agent (recommended):

With this approach, you install Elastic Agent and use Fleet in Kibana to define, configure, and manage your agents in a central location. We recommend using Fleet management because it makes the management and upgrade of your agents considerably easier.

Install Elastic Agent in standalone mode (advanced users):

With this approach, you install Elastic Agent and manually configure the agent locally on the system where it’s installed. You are responsible for managing and upgrading the agents. This approach is reserved for advanced users only.

Install Elastic Agent in a containerized environment:

You can run Elastic Agent inside a container, either with Fleet Server or standalone. Docker images for all versions of Elastic Agent are available from the Elastic Docker registry, and we provide deployment manifests for running on Kubernetes.

Please note, there are minimum requirements for running Elastic Agent. For more information, refer to the Elastic Agent Minimum Requirements.

---

How to add configurations to Elastic Integration 

I. Active Directory Base DN

• Definition: The Base DN (Distinguished Name) specifies the starting point in the Active Directory hierarchy for user and group searches.

• Format: It typically represents the container or organizational unit (OU) where your user accounts are located.

• Example: If your AD users are in the "Users" OU under the domain "example.com", the Base DN might look like:

  • CN=Users,DC=example,DC=com

Note: Refer to Step I. Active Directory Information Lookup for information on how to properly setup the configuration.

II. Active Directory URL

• Definition: The URL of your Active Directory server, specifying either an unsecured LDAP or secure LDAPS connection.

• Format:

  • LDAP (insecure): ldap://your-ad-server.example.com:389

  • LDAPS (secure): ldaps://your-ad-server.example.com:636

• Example:

  • ldap://ad.example.com:389

Note: Refer to Step II. Finding Active Directory URL for information on how to properly setup the configuration.

III. Active Directory User

• Definition: The username of the service account that Elastic Stack will use to authenticate and query AD. This account should have sufficient privileges to search for users and groups.

• Format:

  • It can be in the form of a fully qualified domain username: username@example.com

  • Or a Distinguished Name (DN): CN=ServiceAccount,OU=ServiceAccounts,DC=example,DC=com

• Example:

  • CN=serviceaccount,OU=ServiceAccounts,DC=example,DC=com

Note: Refer to Step III. Navigate to Users for information on how to properly setup the configuration.

IV. Active Directory User Password

• Definition: The password for the AD user account used for the connection.

• Example:

  • MySecurePassword123

---

I. Active Directory Information Lookup 

Finding the Base DN (Distinguished Name)

Method 1: Using Active Directory GUI

1. Open "Active Directory Users and Computers"

2. Right-click on your domain

3. Select "Properties"

4. Look for the "Distinguished Name" field

Method 2: Using PowerShell

1. Open PowerShell with administrator privileges
2. Run the command:
3. The output will be in the format: "DC=company,DC=local"

---

II. Finding Active Directory URL

Method 1: PowerShell

1. Open PowerShell as administrator
2. Run the command:
   

   

    

3. Take the DC name from the output
   • LDAP://servername.domain.com
   • LDAP://server-IP
     

---

III. Navigate to Users

If you need further assistance, kindly contact our support at info@cytechint.com for prompt assistance and guidance.

Microsoft SQL Server Integration

The Microsoft SQL Server integration package allows you to search, observe, and visualize the SQL Server audit logs, as well as performance and transaction log metrics.

---

Requirements

Microsoft SQL Server is installed and has connectivity with the CyTech Log Collector.

Note. For more information regarding Microsoft SQL Server Installation, click the link (https://learn.microsoft.com/en-us/sql/database-engine/install-windows/install-sql-server?view=sql-server-ver16) for more info.

---

Microsoft SQL Server permissions

Before you can start sending data to the Log Collector, make sure you have the necessary Microsoft SQL Server permissions.

If you browse Microsoft Developer Network (MSDN) for the following tables, you will find a "Permissions" section that defines the permission needed for each table.

1. transaction_log:
   • sys.databases
   • sys.dm_db_log_space_usage
   • sys.dm_db_log_stats (DB_ID) (Available on SQL Server (MSSQL) 2016 (13.x) SP 2 and later)
2. performance:
   • sys.dm_os_performance_counters

Please make sure the user has the permissions to system as well as user-defined databases. For the particular user used in the integration, the following requirements are met:

User setup options:

• Grant specific permissions as mentioned in the MSDN pages above.
• Alteratively, use sysadmin role (includes all required permissions): This can be configured via SQL Server Management Studio (SSMS) in Server Roles. Read more about joining a role in the SQL Server documentation.

User Mappings (using SQL Server Management Studio (SSMS)):

• Open SSMS and connect to your server.
• Navigate to "Object Explorer" > "Security" > "Logins".
• Right-click the user and select "Properties".
• In the "User Mapping" tab, select the appropriate database and grant the required permissions.

---

Setup

Below you'll find more specific details on setting up the Microsoft SQL Server integration.

Named Instance

Microsoft SQL Server has a feature that allows running multiple databases on the same host (or clustered hosts) with separate settings. Establish a named instance connection by using the instance name along with the hostname (e.g. host/instance_name or host:named_instance_port) to collect metrics. Details of the host configuration are provided below.

Host Configuration

As part of the input configuration, you need to provide the user name, password and host details. The host configuration supports both named instances or default (no-name) instances, using the syntax below.

Note: This integration supports collecting metrics from a single host. For multi-host metrics, each host can be run as a new integration.

Connecting to Default Instance (host):

• host (e.g. localhost (Instance name is not needed when connecting to default instance))

Note. IP Address of the SQL Server will be needed for the integration

• host:port (e.g. localhost:1433)

Note. Default port is 1433

Connecting to Named Instance (host):

• host/instance_name (e.g. localhost/namedinstance_01)
• host:named_instance_port (e.g. localhost:60873)

If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.

Azure Logs Integration

Introduction

This document shows information related to Azure Active Directory Integration.
The Azure Logs integration retrieves different types of log data from Azure.

---

Assumptions

The procedures described in the Requirements section assumes that a Log Collector has already
been setup.

---

Requirements

Main Setup

• One or more event hub to store in-flight logs exported by Azure services and make them available to the Log Collector
  • Example:

  •   ┌────────────────┐       ┌────────────┐
      │     adlogs     │       │  Log       │
      │ <<Event Hub>>  │─────▶ │  Collector │
      └────────────────┘       └────────────┘
• One or more diagnostic setting to export logs from Azure services to Event Hubs
  • Example:   

  • ┌──────────────────┐      ┌──────────────┐     ┌─────────────────┐
    │Microsoft Entra ID│      │  Diagnostic  │     │    Event Hub    │
    │    <<source>>    │─────▶│   settings   │────▶│ <<destination>> │
    └──────────────────┘      └──────────────┘     └─────────────────┘

• One Storage Account Container to store information about logs consumed by the Log Collector
  • • Example:

        ┌────────────────┐                     ┌────────────┐
        │     adlogs     │        logs         │  Log       │
        │ <<Event Hub>>  │────────────────────▶│  Collector │
        └────────────────┘                     └────────────┘
                                                      │
                             consumer group info      │
        ┌────────────────┐   (state, position, or     │
        │   azurelogs    │         offset)            │
        │ <<container>>  │◀───────────────────────────┘
        └────────────────┘

This is the final diagram of the a setup for collecting Activity logs from the Azure Monitor service.

 ┌───────────────┐   ┌──────────────┐   ┌────────────────┐         ┌────────────┐
 │  MS Entra ID  │   │  Diagnostic  │   │     adlogs     │  logs   │  Log       │
 │  <<service>>  ├──▶│   Settings   │──▶│ <<Event Hub>>  │────────▶│ Collector │
 └───────────────┘   └──────────────┘   └────────────────┘         └────────────┘
                                                                          │
                     ┌──────────────┐          consumer group info        │
                     │  azurelogs   │          (state, position, or       │
                     │<<container>> │◀───────────────offset)──────────────┘
                     └──────────────┘

If the integration is running behind a firewall, please proceed here.

Here are several requirements before using the integration since the logs will
be read from azure event hubs.

1. The logs have to be exported first to the event hub.
   • Create an event hub using Azure portal.
   • More information can be found on: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubscreate.
2. To export activity logs to event hubs users can follow the steps here.
   • Legacy collection methods
   • More information can be found on: https://learn.microsoft.com/en-us/azure/azuremonitor/essentials/activity-log?tabs=powershell#legacy-collectionmethods
3. To export audit and sign-in logs to event hubs users can follow the
   steps here.
   • Stream Azure Active Directory logs
   • More information can be found on: https://learn.microsoft.com/en-us/azure/active-directory/reportsmonitoring/tutorial-azure-monitor-stream-logs-to-event-hub

---

Azure Active Directory Integration Procedures

Create a Resource Group
A resource group is a logical collection of Azure resources. All resources are
deployed and managed in a resource group. To create a resource group:

1. Sign in to the Azure portal.
2. In the left navigation, select Resource groups, and then
   select Create a resource.
3. For Subscription, select the name of the Azure subscription in which
   you want to create the resource group. For CyTech (Azure Active Directory) 
4. Type a unique name for the resource group. The system
   immediately checks to see if the name is available in the currently
   selected Azure subscription.
5. Select a region for the resource group.
6. Select Review + Create.
7. Takes a few minutes to complete.

---

Create an Event Hubs Namespace

An Event Hubs namespace provides a unique scoping container, in which you create
one or more event hubs. To create a namespace in your resource group using the
portal, do the following actions:

1. In the Azure portal, and select Create a resource at the top left of
   the screen.
2. Select All services in the left menu, and select star (*) next to Event
   Hubs in the Analytics category. Confirm that Event Hubs is added
   to FAVORITES in the left navigational menu.
3. Select Event Hubs under FAVORITES in the left navigational menu, and
   select Create on the toolbar.
4. On the Create namespace page, take the following steps:
   a. Select the subscription in which you want to create the
   namespace.
   b. Select the resource group you created in the previous step.
   c. Enter a name for the namespace. The system immediately checks to see if the name is available.                                                                         d. Select a location for the namespace.
   e. Choose Basic for the pricing tier. To learn about differences
   between tiers, see Quotas and limits, Event Hubs Premium, and Event
   Hubs Dedicated articles.f. Leave the throughput units (for standard tier) or processing
   units (for premium tier) settings as it is. To learn about throughput units
   or processing units: Event Hubs scalability.                                                                                                                                             g. Select Review + Create at the bottom of the page.
   h. On the Review + Create page, review the settings, and select Create.
   Wait for the deployment to complete.
5. On the Deployment page, select Go to resource to navigate to the page for
   your namespace.

---

Create an Event Hub

1. To create an event hub within the namespace, do the following actions:
2. On the Overview page, select + Event hub on the command bar.
3. Type a name for your event hub, then select Review + create.                                                                        The partition count setting allows you to parallelize consumption across
   many consumers. For more information, see Partitions.
   The message retention setting specifies how long the Event Hubs service
   keeps data. For more information, see Event retention.
4. On the Review + create page, select Create.
5. You can check the status of the event hub creation in alerts. After the event
   hub is created, you see it in the list of event hubs.

---

Create a Diagnostic Setting

The diagnostic settings export the logs from Azure services to a destination and in order to use Azure Logs integration, it must be an event hub.

To create a diagnostic settings to export logs:

In the diagnostic settings page you have to select the source log categories you want to export and then select their destination.

Select log categories

Each Azure services exports a well-defined list of log categories. Check the individual integration doc to learn which log categories are supported by the integration.

Select the destination

Select the subscription and the Event Hubs namespace you previously created. Select the event hub dedicated to this integration.

Example:

  ┌───────────────┐   ┌──────────────┐    ┌───────────────┐       ┌────────────┐
  │  MS Entra ID  │   │  Diagnostic  │    │     adlogs    │       │  Log       │
  │  <<service>>  ├──▶│   Settings   │──▶│ <<Event Hub>> │─────▶ │ Collector │
  └───────────────┘   └──────────────┘    └───────────────┘       └────────────┘

---

Create a Storage Account

To create an Azure storage account with the Azure portal, follow these steps:

1. From the left portal menu, select Storage accounts to display a list
   of your storage accounts. If the portal menu isn't visible, click the
   menu button to toggle it on.
2. On the Storage accounts page, select Create.
3. The following image shows a standard configuration of the basic properties
4. The following image shows a standard configuration of the advanced
   properties for a new storage account.                                          
5. The following image shows a standard configuration of the networking
   properties for a new storage account.                            
6. The following image shows a standard configuration of the data protection
   properties for a new storage account.
7. The following image shows a standard configuration of the encryption
   properties for a new storage account.                            
8. Review + Create Tab
   When you navigate to the Review + create tab, Azure runs
   validation on the storage account settings that you have chosen. If
   validation passes, you can proceed to create the storage account.
   If validation fails, then the portal indicates which settings need to be
   modified.

The following image shows the Review tab data prior to the creation
of a new storage account.

---

Resources needed for the integration of Azure Active Directory:

1. Azure Diagnostics Settings
   Create a Diagnostics Configuration and select which log from
   Azure will send to the event hub.
   Navigate to Microsoft Entra ID > Monitoring > Diagnostic settings
2. Event Hub Credentials
3. Go to > EventHub Resources > Select Shared Access Policies
4. Please provide CyTech the:
   a. Event Hubs Name Not the Name Space:
   b. Connection string-primary key:
5. Account Storage Credentials
6. Please provide CyTech the:
   a. Storage Account Name:
   b. Key 1 Key

---

Running the integration behind a firewall:

When you run the Elastic Agent behind a firewall, to ensure proper communication with the necessary components, you need to allow traffic on port 5671 and 5672 for the event hub, and port 443 for the Storage Account container.

┌────────────────────────────────┐  ┌───────────────────┐  ┌───────────────────┐
│                                │  │                   │  │                   │
│ ┌────────────┐   ┌───────────┐ │  │  ┌──────────────┐ │  │ ┌───────────────┐ │
│ │ diagnostic │   │ event hub │ │  │  │azure-eventhub│ │  │ │ activity logs │ │
│ │  setting   │──▶│           │◀┼AMQP─│  <<input>>   │─┼──┼▶│<<data stream>>│ │
│ └────────────┘   └───────────┘ │  │  └──────────────┘ │  │ └───────────────┘ │
│                                │  │          │        │  │                   │
│                                │  │          │        │  │                   │
│                                │  │          │        │  │                   │
│         ┌─────────────┬─────HTTPS─┼──────────┘        │  │                   │
│ ┌───────┼─────────────┼──────┐ │  │                   │  │                   │
│ │       │             │      │ │  │                   │  │                   │
│ │       ▼             ▼      │ │  └─Log Collector─────┘  └─Elastic Cloud─────┘
│ │ ┌──────────┐  ┌──────────┐ │ │
│ │ │    0     │  │    1     │ │ │
│ │ │ <<blob>> │  │ <<blob>> │ │ │
│ │ └──────────┘  └──────────┘ │ │
│ │                            │ │
│ │                            │ │
│ └─Storage Account Container──┘ │
│                                │
│                                │
└─Azure──────────────────────────┘

Event Hub

Port 5671 and 5672 are commonly used for secure communication with the event hub. These ports are used to receive events. By allowing traffic on these ports, the Elastic Agent can establish a secure connection with the event hub.

Storage Account Container

Port 443 is used for secure communication with the Storage Account container. This port is commonly used for HTTPS traffic. By allowing traffic on port 443, the Elastic Agent can securely access and interact with the Storage Account container, which is essential for storing and retrieving checkpoint data for each event hub partition.

DNS

Optionally, you can restrict the traffic to the following domain names:

*.servicebus.windows.net
*.blob.core.windows.net
*.cloudapp.net

---

Additional Information:

Azure Active Directory Logs contain

Sign-in logs – Information about sign-ins and how your users use your
resources.

• Retrieves Azure Active Directory sign-in logs. The sign-ins report provides
  information about the usage of managed applications and user sign-in
  activities.

Identity Protection logs - Information about user risk status and the events
that change it.

• Retrieves Azure AD Identity Protection logs. The Azure AD Identity
  Protection service analyzes events from AD users' behavior, detects risk
  situations, and can respond by reporting only or even blocking users at
  risk, according to policy configurations.

Provisioning logs - Information about users and group synchronization to
and from external enterprise applications.

• Retrieves Azure Active Directory Provisioning logs. The Azure AD
  Provisioning service syncs AD users and groups to and from external
  enterprise applications. For example, you can configure the provisioning
  service to replicate all existing AD users and groups to an external
  Dropbox Business account or vice-versa.

The Provisioning Logs contain a lot of details about a inbound/outbound
sync activity, like:

• User or group details.
• Source and target systems (e.g., from Azure AD to Dropbox).
• Provisioning status.
• Provisioning steps (with details for each step).

Audit logs – Information about changes to your tenant, such as users and
group management, or updates to your tenant's resources.

• Retrieves Azure Active Directory audit logs. The audit logs provide
  traceability through logs for all changes done by various features within
  Azure AD. Examples of audit logs include changes made to any resources
  within Azure AD like adding or removing users, apps, groups, roles and
  policies.

If you need further assistance, kindly contact our support at info@cytechint.com for prompt assistance and guidance.

ESET Protect Integration

ESET PROTECT allows you to efficiently manage ESET products across workstations and servers within a networked environment, supporting up to 50,000 devices from a single centralized platform. Through the ESET PROTECT Web Console, you can seamlessly deploy ESET solutions, manage tasks, enforce security policies, monitor system health, and swiftly address any issues or threats on remote devices.

---

Data streams

The ESET PROTECT integration collects three types of logs: Detection, Device Task and Event.

Detection is used to retrieve detections via the ESET Connect - Incident Management.

Device Task is used to retrieve device tasks via the ESET Connect - Automation.

Event is used to retrieve Detection, Firewall, HIPS, Audit, and ESET Inspect logs using the Syslog Server.

---

Requirements:

• Elastic Agent must be installed

---

Setup

To collect data from ESET Connect, follow the below steps:

1. Create API User Account (Refer to How to Create an API User Account below)
2. Retrieve the username and password generated during the creation of an API user account.
3. Retrieve the region from the ESET Web Console URL.

To collect data from ESET PROTECT via Syslog, follow the below steps:

1. Follow the steps to configure syslog server (Refer to How to Configure Syslog Server).
   • Set the format of the payload to JSON.
   • Set the format of the envelope to Syslog.
   • Set the minimal log level to Information to collect all data.
   • Select all checkboxes to collect logs for all event types.
   • Enter the IP Address or FQDN of the Elastic Agent that is running the integration in the Destination IP field.

---

How to Create an API User Account:

For ESET Business Account and ESET MSP Administrator 2

Follow the steps below to create the dedicated API user account:

1. Log in as Superuser (or Root) to your ESET Business Account or ESET MSP Administrator 2.
2. Navigate to User management and create a new user.
3. Under the Access Rights section, enable the toggle next to Integrations.
4. Click Create to apply the changes.
5. The new user receives an invitation email and must finish the account activation process.

For ESET PROTECT Hub

Follow the steps below to create the dedicated API user account:

1. Log in as a Superuser to your ESET PROTECT Hub account.
2. Navigate to Users and add a new user.
3. Under the Permissions section, enable the toggle next to Integrations.
4. Click Next and then click Create to apply the changes.
5. The new user receives an invitation email and must finish the account activation process.

---

How to Configure Syslog Server

If you have a Syslog server running in your network, you can Export logs to Syslog to receive certain events (Detection Event, Firewall Aggregated Event, HIPS Aggregated Event, etc.) from client computers running ESET Endpoint Security.

To enable the Syslog server:

1. Click More > Settings > Syslog and click the toggle next to Enable Syslog sending.
2. Specify the following mandatory settings:

• Format of payload: JSON, LEEF or CEF
• Format of envelope of the log: BSD (specification), Syslog (specification)
• Minimal log level: Information, Warning, Error or Critical
• Event type of logs: Select the type of logs you want to include (Antivirus, HIPS, Firewall, Web protection, Audit Log, Blocked files, ESET Inspect alerts).
• Destination IP or FQDN of TLS-compatible syslog server: IPv4 address or hostname of the destination for Syslog messages
• Validate CA Root certificates of TLS connections: Click the toggle to enable the certificate validation for the connection between your Syslog server and ESET PROTECT. After enabling the validation, a new text field will be displayed where you can copy and paste the required certificate chain. The server certificate must meet the following requirements:
  • The whole certificate chain in PEM format is uploaded and saved in the Syslog export configuration (this includes root CA, as there are no built-in trusted certificates)
  • Your Syslog server's certificate provides a Subject Alternative Name extension (DNS=/IP=), in which at least one record corresponds to the FQDN/IP hostname configuration.

You need the certification authority version 3 (and later) with the Basic Constraints certificate extension to pass the validation.

The validation of TLS connections applies only to the certificates. Disabling the validation does not affect the TLS settings of ESET PROTECT.

After making the applicable changes, click Apply settings. The configuration becomes effective in 10 minutes.

The regular application log file is constantly being written to. Syslog only serves as a medium to export certain asynchronous events, such as notifications or various client computer events.

ESET Threat Intelligence Integrations

ESET Threat Intelligence provides advanced, real-time insights into global cybersecurity threats, empowering you to proactively defend your network and systems. By leveraging a vast database of threat data, it enables you to detect and respond to emerging threats, track attack trends, and enhance your security posture with actionable intelligence. With ESET Threat Intelligence, you can make informed decisions to protect your organization from sophisticated cyber threats.

---

Setup:

1) Log Collector must be installed.

2) Prepare the information from the ESET Threat Intelligence Account:

• Ensure that you have access to ESET Threat Intelligence feeds (via ESET Threat Intelligence API or downloadable data).
• Please prepare the Username and Password that you have received from ESET during their onboarding process.

---

References Information: 

Data streams

This integration connects with the ESET Threat Intelligence TAXII version 2 server. It includes the following datasets for retrieving logs:

Dataset	TAXII2 Collection name
apt	apt stix 2.1
botnet	botnet stix 2.1
cc	botnet.cc stix 2.1
domains	domain stix 2.1
files	file stix 2.1
ip	ip stix 2.1
url	url stix 2.1

Obtaining an API Key for ESET Threat Intelligence

Usage of the ESET Threat Intelligence (ETI) API

The ESET Threat Intelligence (ETI) API can be used directly in a web browser’s address bar as a REST API, meaning that it does not necessarily require implementation in a programming language. This allows for a straightforward integration of threat intelligence data without the need for additional software development.

Authentication

Authentication with the ETI API is managed via a token. This token can be generated in the profile section of the ESET Threat Intelligence portal. It is important to note that each token is valid for only one hour, ensuring secure access to the API.

To generate a token, users can either manually generate it through the portal interface or use a CURL request. This approach provides flexibility, allowing automated generation of tokens for integration or scheduled use.

Generate via CURL Request

Step 1: Open a Command-Line Interface (CLI)

• Windows: Open Command Prompt (cmd) or PowerShell.
• macOS/Linux: Open Terminal.

Step 2: Enter the CURL Command

In the command-line interface, use the following CURL command to generate an authentication token:

curl -F name="YOUR-USERNAME" -F pass="YOUR-PASSWORD" ETI_URL/auth/

Step 3: Copy and save the authentication token

Note. 
After 10 failed login attempts within 5 minutes, the user will be blocked for 15 minutes.
After 20 failed attempts from a specific IP address within 5 minutes, all login attempts from that IP will be blocked for 15 minutes.

If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.

CSPM for Azure Integration

This manual explains how to get started monitoring the security posture of your Azure CSP using the Cloud Security Posture Management (CSPM) feature.

Requirements

Setup

Option 1: Service principal with client secret  (recommended)

Before using this method, you must have set up a Microsoft Entra application and service principal that can access resources. Please go here before following the steps below.

Option 2: Managed identity (optional)

This method involves creating an Azure VM (or using an existing one), giving it read access to the resources you want to monitor with CSPM, and installing the Log Collector on the Azure VM.

If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.

Resource Manager Endpoint Integration

The Azure Resource Manager (ARM) endpoint is the primary entry point for interacting with the Azure platform's resource management services. It allows users to deploy, manage, and organize resources like virtual machines, storage accounts, and networks within a defined Azure subscription. The ARM endpoint serves as a REST API that facilitates the creation, modification, and deletion of resources, ensuring secure and scalable management of Azure resources. It also supports role-based access control (RBAC), policy enforcement, and resource tagging for effective governance. In essence, the ARM endpoint enables seamless communication between Azure services and clients for infrastructure management.

---

Requirements:

• Microsoft Azure account

---

Setup:

How to Create an Azure Resource Manager Service Endpoint

1. Create Service Principal

1. Click on Applications of the selected Active directory
2. Click the Add button at the button of the page
3. Enter a name for your application and make sure Web Application and/or Web API is selected
4. Enter two URLs based on your application name
   They do not have to be real. I used the same value for both.
5. Once the application is created, click on Configure
6. Make note of the Client ID because we will need it in a moment
7. Select a key duration under the keys section and click Save at the bottom of the page
8. Once the key is saved copy the value and place it with your Client ID
   This will be your only chance to collect this value.

2. Find Tenant ID

With the Active Directory select on the Applications page, we can harvest the Tenant ID.

1. Click View Endpoints at the bottom of the page
2. Copy any of the URLS and paste into an editor
3. The GUID in the URL is your Tenant ID

3. Find Subscription Name and ID

You will also need the subscription name and ID to complete the service endpoint. We can get them while we are in the old portal.

1. Click Settings in the left vertical menu
2. Copy the Subscription and Subscription ID values

4. Grant access

There is a script that can do all of this for you here on GitHub.

5. Create Service Endpoint

With the Service Principal created, we can now create the Service Endpoint in VSTS.

1. Log in to VSTS and select a project
2. Click the manage project gear icon in the upper right hand corner of the page
3. Select the Services tab
4. Select Azure Resource Manager from the New Service Endpoint drop down
   

   Field	Value
   Connection Name	{AnyValueYouLike}
   Subscription Id	Subscription Id
   Subscription Name	Subscription Name
   Service Principal Id	Client Id
   Service Principal Key	Key
   Tenant Id	Tenant Id

5. Click OK

How to navigate and locate Resource Manager Endpoint.

1.Sign in to the Azure portal.

2.Select Resource groups from the left panel.

3.Select the resource group that you have already created specifically for Azure Integration. 

4.Hover to "Overview".

5.Verify the following resources were created in the resource group: "Endpoint Name" and type "Endpoint".

6.Copy the Endpoint Name for integration requirement. Ex. contosocdn123(myCDNProfile/contosocdn123)

If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.

CISCO Secure Email Gateway Integrations

The Cisco Email Security Appliance (ESA) integration is a comprehensive solution for managing and securing email traffic within an organization's network. It provides various functionalities, such as spam filtering, virus scanning, policy enforcement, and data loss prevention. The integration collects and parses data from the Cisco Secure Email Gateway (formerly known as Cisco Email Security Appliance) to provide valuable insights into the email environment. The data collection occurs through multiple methods, primarily through TCP/UDP communication and log file analysis.

---

Requirements:

• Cisco account
• Elastic agent already installed

---

Compatibility

This module has been tested against Cisco Secure Email Gateway server version 14.0.0 Virtual Gateway C100V with the below given logs pattern.

---

Setup

Configurations

• Sign-in to Cisco Secure Email Gateway Portal and follow the below steps for configurations:
  1. In Cisco Secure Email Gateway Administrator Portal, go to System Administration > Log Subscriptions.
  2. Click Add Log Subscription.
  3. Enter all the Required Details.
  4. Set Log Name as below for the respective category:
     • AMP Engine Logs -> amp
     • Anti-Spam Logs -> antispam
     • Antivirus Logs -> antivirus
     • Authentication Logs -> authentication
     • Bounce Logs -> bounces
     • Consolidated Event Logs -> consolidated_event
     • Content Scanner Logs -> content_scanner
     • HTTP Logs -> gui_logs
     • IronPort Text Mail Logs -> error_logs
     • Text Mail Logs -> mail_logs
     • Status Logs -> status
     • System Logs -> system
  5. Select Log Level as Information.
  6. Select Retrieval Method.
  7. Click Submit and commit the Changes.

Note

• Retrieval Method Supported:
  • FTP Push to Remote Server for the below categories: AMP Engine Logs, Anti-Spam Logs, Antivirus Logs, Authentication Logs, Bounce Logs, Consolidated Event Logs, Content Scanner Logs, HTTP Logs, IronPort Text Mail Logs, Text Mail Logs, Status Logs and System Logs.
  • Syslog Push for the below categories: AMP Engine Logs, Anti-Spam Logs, Antivirus Logs, Consolidated Event Logs, Content Scanner Logs, HTTP Logs, IronPort Text Mail Logs, Text Mail Logs, Status Logs and System Logs.

If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.

CISCO Nexus Integrations

Overview

The Cisco Nexus integration allows users to monitor Errors and System Messages. The Cisco Nexus series switches are modular and fixed port network switches designed for the data center. All switches in the Nexus range run the modular NX-OS firmware/operating system on the fabric. NX-OS has some high-availability features compared to the well-known Cisco IOS. This platform is optimized for high-density 10 Gigabit Ethernet.

Use the Cisco Nexus integration to collect and parse data from Syslog and log files. Then visualize that data through search, correlation and visualization within Elastic Security.

---

Data streams

The Cisco Nexus integration collects one type of data: log.

Log consists of errors and system messages. See more details about errors and system messages

---

Requirements

Elastic Agent must be installed. 

The minimum kibana.version required is 8.7.0.

This module has been tested against the Cisco Nexus Series 9000, 3172T and 3048 Switches.

---

Setup

To collect data from Cisco Nexus, follow the below steps:

Logging System Messages to a File

You can configure the device to log system messages to a file. By default, system messages are logged to the file /logflash/log/logfilename .

	Command or Action	Purpose
Step 1	configure terminal Example: switch# configure terminal switch(config)#	Enters global configuration mode.
Step 2	[ no ] logging logfile logfile-name severity-level [ | size bytes ] Example: switch(config)# logging logfile my_log 6	Configures the nonpersistent log file parameters. logfile-name : Configures the name of the log file that is used to store system messages. Default filename is "message". severity-level : Configures the minimum severity level to log. A lower number indicates a higher severity level. Default is 5. Range is from 0 through 7: 0 – emergency 1 – alert 2 – critical 3 – error 4 – warning 5 – notification 6 – informational 7 – debugging size bytes : Optionally specify maximum file size. Range is from 4096 through 4194304 bytes.
Step 3	logging event {link-status | trunk-status} {enable | default} Example: switch(config)# logging event link-status default	Logs interface events. link-status —Logs all UP/DOWN and CHANGE messages. trunk-status —Logs all TRUNK status messages. enable —Specifies to enable logging to override the port level configuration. default —Specifies that the default logging configuration is used by interfaces that are not explicitly configured.

---

Configuring Syslog Servers

Note: Cisco recommends that you configure the syslog server to use the management virtual routing and forwarding (VRF) instance. For more information on VRFs, see Cisco Nexus 9000 Series NX-OS Unicast Routing Configuration Guide.

You can configure up to eight syslog servers that reference remote systems where you want to log system messages.

Procedure

	Command or Action	Purpose
Step 1	configure terminal Example: switch# configure terminal switch(config)#	Enters global configuration mode.
Step 2	[no] logging server host [severity-level [use-vrf vrf-name]] Example: switch(config)# logging server 192.0.2.253 Example: switch(config)# logging server 2001::3 5 use-vrf red	Configures a syslog server at the specified hostname, IPv4, or IPv6 address. You can specify logging of messages to a particular syslog server in a VRF by using the use-vrf keyword. The use-vrf vrf-name keyword identifies the default or management values for the VRF name. The default VRF is the management VRF, by default. However, the show-running command will not list the default VRF. Severity levels range from 0 to 7: 0 – emergency 1 – alert 2 – critical 3 – error 4 – warning 5 – notification 6 – informational 7 – debugging The default outgoing facility is local7. The no option removes the logging server for the specified host. The first example forwards all messages on facility local 7. The second example forwards messages with severity level 5 or lower to the specified IPv6 address in VRF red.
Step 3	logging source-interface loopback virtual-interface Example: switch(config)# logging source-interface loopback 5	Enables a source interface for the remote syslog server. The range for the virtual-interface argument is from 0 to 1023.

NOTE:

• Use the Timezone Offset parameter, if the timezone is not present in the log messages.

If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.

BitDefender Integrations

BitDefender GravityZone supports SIEM integration using "push notifications", which are JSON messages sent via HTTP POST to a HTTP or HTTPS endpoint, which this integration can consume.

This integration additionally provides:

1. Collection of push notification configuration via API polling, which includes the "state" of the push notification service on the BitDefender GravityZone server, e.g. indicating if it is currently enabled or disabled. This is useful as the state may change to disabled (value of 0) for unknown reasons and you may wish to alert on this event.
2. Collection of push notification statistics via API polling, which includes the number of events sent, and counters for errors of different types, which you may wish to use to troubleshoot lost push notification events and for alerting purposes.
3. Support for multiple instances of the integration, which may be needed for MSP/MSSP scenarios where multiple BitDefender GravityZone tenants exist.
4. BitDefender company ID to your own company name/description mapping, in order to determine to which tenant the event relates to in a human friendly way. This is very useful for MSP/MSSP environments or for large organizations with multiple sub-organizations.

This allows you to search, observe and visualize the BitDefender GravityZone events through Elastic, trigger alerts and monitor the BitDefender GravityZone Push Notification service for state and errors.

---

Data Stream

Log Stream Push Notifications

The BitDefender GravityZone events dataset provides events from BitDefender GravityZone push notifications that have been received.

All BitDefender GravityZone log events are available in the bitdefender_gravityzone.events field group.

---

Compatibility

This integration supports BitDefender GravityZone, which is the business-oriented product set sold by BitDefender.

BitDefender products for home users are not supported.

The package collects BitDefender GravityZone push notification transported events sent in jsonrpc, qradar, or splunk format.

The jsonrpc format is recommended default, but the ingest pipeline will attempt to detect if qradar or splunk format events have been received and process them accordingly.

The integration can also collect the push notification configuration and statistics by polling the BitDefender GravityZone API.

---

Configuration

Enabling the integration in Elastic

1. In Kibana go to Management > Integrations
2. In "Search for integrations" search bar type GravityZone
3. Click on "BitDefender GravityZone" integration from the search results.
4. Click on Add BitDefender GravityZone button to add BitDefender GravityZone integration.

Create a BitDefender GravityZone API key that can configure a push notification service

The API key needed to configure push notifications, and collection push notification configuration state and statistics, is typically configured within the BitDefender GravityZone cloud portal.

Bear in mind the API key will be associated to the account you create it from. A named human account may not be desirable, e.g. you may wish to (probably should) create API keys for functions such as push notifications under a non-human/software service account that will never retire or be made redundant.

Navigate to your account details within the GravityZone portal. If you have sufficient privileges, you will see the "API keys" section near the bottom of the page. Click "Add" here.

Give the API key a description and tick the "Event Push Service API" box at minimum.

NOTE: If you intend to use the API key for other API calls you may need to tick other boxes.

Click the Key value that is shown in blue.

Click the clipboard icon to copy the API key to your PC's clipboard.

 

If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.

Bitwarden Integrations

Overview

The Bitwarden integration allows users to monitor collections, events, groups, members and policies. Bitwarden is a free and open-source password management service that stores sensitive information such as website credentials in an encrypted vault. The Bitwarden platform offers a variety of client applications including a web interface, desktop applications, browser extensions, mobile apps and a command-line interface. Bitwarden offers a cloud-hosted service as well as the ability to deploy the solution on-premises.

Use the Bitwarden integration to collect and parse data from the REST APIs. Then visualize that data in Kibana.

---

Data streams

The Bitwarden integration collects five types of data: Collections, Events, Groups, Members and Policies.

Collections returns a list of an organization's collections.

Events returns a list of an organization's event logs.

Groups returns a list of an organization's groups.

Members returns the details of an organization's members.

Policies returns a list of an organization's policies.

Reference for Rest APIs of Bitwarden.

---

Requirements

Elasticsearch is needed to store and search data and Kibana is needed for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your hardware.

This module has been tested against Bitwarden Version 2023.2.0.

---

Setup

To collect data from Bitwarden REST APIs, follow the below steps:

1. Go to the Bitwarden console, enter an email address and master password.
2. Click Organizations.
3. Go to Settings → Organization info.
4. Click View API Key from API key Section.
5. Enter master password.
6. Click View API Key.
7. Copy client_id and client_secret.

If you need further assistance, kindly contact our support at support@cytechint.com for prompt assistance and guidance.

Forwarding logs from rsyslog client to a remote rsyslogs server

Introduction

This guide will walk you through setting up Rsyslog for log forwarding between a client and a remote server using Linux.

Setup

Server: The machine which will send message
Client:  The machine which will receive the message

 Prerequisites

 Software Requirements

• •  Linux operating system
  • Rsyslog (version 5.0 or higher recommended)
  • Root or sudo access

Network Requirements

• • Network connectivity between client and remote server
  • Known IP address of the remote Rsyslog server
  • Open network ports (typically 514 for UDP or TCP)

Step-by-Step Configuration Guide

Preparation
Before beginning, ensure you have:

• • Administrative (root) access
  • Stable network connection
  • IP address of the remote server

Step 1: Rsyslog Installation

 1.1 Obtain Root Access

sudo -i

• Enter your root password when prompted

 1.2 Update System Packages

If you are using DNF, use the command below:

sudo dnf update

If you are using YUM, use the command below:

sudo yum update

 1.3 Install Rsyslog

If you are using YUM, use the command below:

sudo yum install rsyslog

If you using DNF, use the command below:

sudo dnf install rsyslog

Verification Tip: Confirm Rsyslog is installed successfully

 1.4 Start and Enable Rsyslog Service

sudo systemctl enable rsyslog
sudo systemctl start rsyslog

 1.5 Check Rsyslog Status

sudo systemctl status rsyslog

Expected Result:  Service should be in an active state

Step 2: Rsyslog Server and Client Configuration

The following steps outline how to forward system logs to a remote server using either TCP or UDP ports. You can choose to use either TCP or UDP, but if both are enabled, ensure that each protocol uses a different port.

 2.1 Edit Rsyslog Configuration. Open using a text editor such as "vi" or "nano". 

vi /etc/rsyslog.conf

 2.2 Enable UDP or TCP Modules. This should be done on the Client machine only.

- For UDP, locate and uncomment the following lines by removing the # symbol. The default port is 514, but you can change it if necessary.

$Modload imudp
$UDPServerRun 514

- For TCP, locate and uncomment the following lines by removing the # symbol. The default port is 10514, but you can change it if necessary.

$Modload imtcp
$inputTCPServerRun 10514

2.3 Configure Log Template
Add the following line to define log storage:

$template RemoteLogs,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
& ~

2.4 On Server
Add content below at the end of the file /etc/rsyslog.conf. 
This will configure the log forwarding to the remote host. Please update the "target", "port" and "tcp" appropriately.

*.* action(type="omfwd"
queue.type="LinkedList"
action.resumeRetryCount="-1"
queue.size="10000"
queue.saveonshutdown="on"
target="10.43.138.1" Port="10514" Protocol="tcp")

queue.type enables a LinkedList in-memory queue, queue_type can be direct, linkedlist or fixedarray (which are in-memory queues), or disk.
enabled queue.saveonshutdown saves in-memory data if rsyslog shuts down,
action.resumeRetryCount= “-1” setting prevents rsyslog from dropping messages when retrying to connect if server is not responding,
queue.size where size represents the specified size of disk queue part. The defined size limit is not restrictive, rsyslog always writes one complete queue entry, even if it violates the size limit.
target is the IP Address of the remote machine
Port is the port of the remote machine
Protocol is the protocol to be used. Values can be udp or tcp.

2.5 Add port in the firewall rules

On client side
Add the provided port to the firewall

iptables -A INPUT -p tcp --dport 10514  -j ACCEPT

Next open the port using nc

nc -l -p 10514 -4

2.6 Apply Server Configuration

systemctl restart rsyslog

2.7 Verify Log Directory
Type : ls -1
Expected Result: 
Should see a directory with the client's hostname
Contains files like `rsyslogd.log` and `systemd.log`

Troubleshooting Tips
Ensure firewall settings allow log forwarding
Verify network connectivity between client and server
Check Rsyslog service status if logs aren't forwarding

Security Considerations
- Configure firewall rules appropriately
- Use encrypted log transmission when possible
- Regularly review and rotate logs

Common Issues
1. Port Blocking: Ensure port 514 is open
2. Permission Errors Verify root/sudo access
3. Network Connectivity: Check server IP and network settings

Conclusion
By following these steps, you should have successfully configured Rsyslog for log forwarding between a client and a remote server.

**Note:** Always test in a controlled environment first and adapt instructions to your specific system configuration.

New script for logs forwarding

# Define the syslog server IP address and port
$syslogServerIP = "192.168.20.24"  # Replace with your Rocky server's IP
$syslogPort = 514

# File to store last sent event info.
$logFilePath = "C:\Users\Administrator\Desktop\lastEventInfo.txt"

# Initialize last sent events as a hashtable
$lastSentEvents = @{}

# Check if the file exists and read the last sent events
if (Test-Path $logFilePath) {
    Write-Host "Loading last sent events from file."
    $fileContent = Get-Content $logFilePath | ConvertFrom-Json
    # Convert the JSON object back to hashtable
    $fileContent.PSObject.Properties | ForEach-Object {
        $lastSentEvents[$_.Name] = $_.Value
    }
}

# Loop for sending logs
while ($true) {
    Write-Host "Checking for new logs..."
    # Define the logs you want to forward
    $logNames = @("Application", "Security", "Setup", "System")
    
    # Loop through each log
    foreach ($logName in $logNames) {
        Write-Host "Processing log: $logName"
        # Get new logs with error handling
        try {
            $logs = Get-WinEvent -LogName $logName -ErrorAction Stop | Sort-Object TimeCreated
        }
        catch [System.Diagnostics.Eventing.Reader.EventLogNotFoundException] {
            Write-Host "Log $logName not found"
            continue
        }
        catch {
            if ($_.Exception.Message -match "No events were found") {
                Write-Host "No events found in $logName"
                continue
            }
            else {
                Write-Host "Error accessing $logName`: $_"
                continue
            }
        }

        foreach ($log in $logs) {
            # Create a unique key based on Event ID and TimeCreated
            $eventKey = "$($log.Id)-$($log.TimeCreated.Ticks)"
            
            # Check if the event has already been sent
            if (-not $lastSentEvents.ContainsKey($eventKey)) {
                # Create syslog message format
                $message = "<134>" + $log.TimeCreated.ToString("yyyy-MM-dd HH:mm:ss") + " " + $log.ProviderName + ": " + $log.Message
                
                try {
                    # Send the message to the syslog server
                    $client = New-Object System.Net.Sockets.UdpClient
                    $client.Connect($syslogServerIP, $syslogPort)
                    $bytes = [System.Text.Encoding]::ASCII.GetBytes($message)
                    $client.Send($bytes, $bytes.Length)
                    $client.Close()
                    
                    # Mark the event as sent
                    $lastSentEvents[$eventKey] = $true
                    Write-Host "Sent log: $message"
                }
                catch {
                    Write-Host "Error sending log: $_"
                }
                finally {
                    if ($client) {
                        $client.Dispose()
                    }
                }
            } else {
                Write-Host "Log already sent: $eventKey"
            }
        }
    }
    
    # Save the last sent events to the file
    try {
        $lastSentEvents | ConvertTo-Json | Set-Content -Path $logFilePath
        Write-Host "Last sent events updated."
    }
    catch {
        Write-Host "Error saving last sent events: $_"
    }
    
    # Wait for 1 second before running again
    Start-Sleep -Seconds 1
}

=====================

# Define the syslog server IP address and port
$syslogServerIP = "192.168.20.24"
$syslogPort = 514

# Test connectivity first
Write-Host "Testing connection to syslog server..." -ForegroundColor Yellow
$testConnection = Test-NetConnection -ComputerName $syslogServerIP -Port $syslogPort -WarningAction SilentlyContinue

if ($testConnection.TcpTestSucceeded) {
    Write-Host "Connection test successful!" -ForegroundColor Green
} else {
    Write-Host "Connection test failed! Please check network connectivity and firewall rules." -ForegroundColor Red
    exit
}

# Send a test message
try {
    $testMessage = "<134>$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') TEST-MESSAGE: This is a test syslog message"
    $client = New-Object System.Net.Sockets.UdpClient
    $client.Connect($syslogServerIP, $syslogPort)
    $bytes = [System.Text.Encoding]::ASCII.GetBytes($testMessage)
    $client.Send($bytes, $bytes.Length)
    Write-Host "Test message sent successfully!" -ForegroundColor Green
    Write-Host "Message content: $testMessage" -ForegroundColor Gray
    $client.Close()
} catch {
    Write-Host "Error sending test message: $_" -ForegroundColor Red
}

# Now run the main script for 30 seconds as a test
$endTime = (Get-Date).AddSeconds(30)
$sentCount = 0

while ((Get-Date) -lt $endTime) {
    $logNames = @("Application", "Security", "Setup", "System")
    
    foreach ($logName in $logNames) {
        Write-Host "Checking $logName log..." -ForegroundColor Yellow
        
        try {
            $logs = Get-WinEvent -LogName $logName -MaxEvents 5 -ErrorAction Stop
            
            foreach ($log in $logs) {
                $message = "<134>" + $log.TimeCreated.ToString("yyyy-MM-dd HH:mm:ss") + " " +
                          $log.ProviderName + ": " + $log.Message
                
                try {
                    $client = New-Object System.Net.Sockets.UdpClient
                    $client.Connect($syslogServerIP, $syslogPort)
                    $bytes = [System.Text.Encoding]::ASCII.GetBytes($message)
                    $client.Send($bytes, $bytes.Length)
                    $client.Close()
                    
                    $sentCount++
                    Write-Host "Successfully sent log entry $sentCount" -ForegroundColor Green
                    Write-Host "EventID: $($log.Id)" -ForegroundColor Gray
                    Write-Host "Source: $($log.ProviderName)" -ForegroundColor Gray
                    Write-Host "Time: $($log.TimeCreated)" -ForegroundColor Gray
                    Write-Host "------------------" -ForegroundColor Gray
                } catch {
                    Write-Host "Failed to send log entry: $_" -ForegroundColor Red
                }
            }
        } catch {
            Write-Host "Error reading from $logName: $_" -ForegroundColor Red
        }
    }
    
    Start-Sleep -Seconds 1
}

Write-Host "Test completed! Total messages sent: $sentCount" -ForegroundColor Green
Write-Host "Please check your syslog server to verify receipt of messages." -ForegroundColor Yellow

====== testing connection

$syslogServerIP = "192.168.20.24"
$syslogPort = 514

Write-Host "Testing basic connectivity..."
Test-NetConnection -ComputerName $syslogServerIP -InformationLevel Detailed
Write-Host "`nTesting specific port connectivity..."
Test-NetConnection -ComputerName $syslogServerIP -Port $syslogPort -WarningAction SilentlyContinue | Format-List *

====

# Windows Event Log Forwarder to RHEL Rsyslog with CPU Optimization
# Configuration Parameters
$RsyslogServer = "192.168.20.40"
$RsyslogPort = 514
$LogPath = "$env:USERPROFILE\Desktop\WindowsLogs"
$JsonPath = "$LogPath\JsonLogs"
$SummaryPath = "$LogPath\Summary"
$MaxCPUUsage = 30  # Maximum CPU percentage to use
$BatchSize = 10    # Number of events to process in batch
$SleepInterval = 2 # Seconds to sleep between batches

# Create necessary directories
$null = New-Item -ItemType Directory -Force -Path $LogPath
$null = New-Item -ItemType Directory -Force -Path $JsonPath
$null = New-Item -ItemType Directory -Force -Path $SummaryPath

# Function to check CPU usage and throttle if necessary
function Test-CPUThreshold {
    $CPUUsage = (Get-Counter '\Processor(_Total)\% Processor Time').CounterSamples.CookedValue
    if ($CPUUsage -gt $MaxCPUUsage) {
        Start-Sleep -Seconds ($CPUUsage / 20)  # Dynamic sleep based on CPU load
        return $true
    }
    return $false
}

# Function to convert Windows event to syslog format
function Convert-ToSyslogFormat {
    param (
        [Parameter(Mandatory=$true)]
        $Event
    )
    
    $Severity = switch ($Event.LevelDisplayName) {
        "Error" { 3 }      # Error
        "Warning" { 4 }    # Warning
        "Information" { 6 } # Info
        "Critical" { 2 }   # Critical
        default { 5 }      # Notice
    }
    
    $Facility = 16  # local0
    $Priority = ($Facility * 8) + $Severity
    
    $Timestamp = $Event.TimeCreated.ToString("yyyy-MM-ddTHH:mm:ss.fffzzz")
    $Hostname = $env:COMPUTERNAME
    $Message = "$($Event.Message)".Replace("`n", " ").Replace("`r", " ")
    
    "<$Priority>1 $Timestamp $Hostname $($Event.LogName)[$($Event.ProcessId)] $($Event.Id) [$($Event.ProviderName)] - $Message"
}

# Function to send syslog message with UDP client pooling
$UdpClientPool = [System.Collections.Concurrent.ConcurrentDictionary[string, System.Net.Sockets.UdpClient]]::new()

function Send-SyslogMessage {
    param (
        [Parameter(Mandatory=$true)]
        [string]$Message
    )
    
    try {
        $ClientKey = [System.Threading.Thread]::CurrentThread.ManagedThreadId.ToString()
        
        $UdpClient = $UdpClientPool.GetOrAdd($ClientKey, {
            $Client = New-Object System.Net.Sockets.UdpClient
            $Client.Client.ReceiveTimeout = 1000
            $Client.Client.SendTimeout = 1000
            return $Client
        })
        
        $EncodedMessage = [System.Text.Encoding]::UTF8.GetBytes($Message)
        $null = $UdpClient.Send($EncodedMessage, $EncodedMessage.Length, $RsyslogServer, $RsyslogPort)
    }
    catch {
        Write-Warning "Failed to send message to rsyslog: $_"
        $Message | Out-File -Append -FilePath "$LogPath\FailedMessages.log"
    }
}

# Main event monitoring function with event batching
function Start-EventMonitoring {
    $EventLogs = @("Application", "Security", "Setup", "System")
    $LastCheckpoints = @{}
    $EventLogs | ForEach-Object { $LastCheckpoints[$_] = [DateTime]::Now }
    
    Write-Host "Starting continuous event monitoring..."
    
    while ($true) {
        if (Test-CPUThreshold) { continue }
        
        foreach ($LogName in $EventLogs) {
            try {
                $Events = Get-WinEvent -LogName $LogName -MaxEvents $BatchSize `
                    -FilterXPath "*[System[TimeCreated[@SystemTime>='$($LastCheckpoints[$LogName].ToUniversalTime().ToString("o"))']]]" -ErrorAction SilentlyContinue
                
                if ($Events) {
                    $LastCheckpoints[$LogName] = $Events[0].TimeCreated
                    
                    foreach ($Event in $Events) {
                        # Convert and send in batches
                        $SyslogMessage = Convert-ToSyslogFormat -Event $Event
                        Send-SyslogMessage -Message $SyslogMessage
                        
                        # Save JSON format
                        $JsonEvent = $Event | Select-Object * | ConvertTo-Json -Depth 5
                        $JsonEvent | Out-File -Append -FilePath "$JsonPath\$LogName-$(Get-Date -Format 'yyyyMMdd').json"
                    }
                }
            }
            catch {
                Write-Warning "Error processing $LogName events: $_"
                Start-Sleep -Seconds 5  # Back off on error
            }
        }
        
        Start-Sleep -Seconds $SleepInterval
    }
}

# Cleanup function
function Cleanup {
    $UdpClientPool.Values | ForEach-Object { $_.Dispose() }
    $UdpClientPool.Clear()
}

# Start monitoring with error handling
try {
    Write-Host "Press Ctrl+C to stop the monitoring..."
    Start-EventMonitoring
}
catch {
    Write-Warning "Error in main monitoring loop: $_"
}
finally {
    Cleanup
}

=======latest script

# PowerShell script to export Windows Event Logs to JSON
param (
    [string]$LogName = "System",  # Default to System log
    [int]$Hours = 24,            # Default to last 24 hours
    [string]$DesktopPath = [Environment]::GetFolderPath("Desktop")
)

# Create timestamp for filename
$timestamp = Get-Date -Format "yyyyMMdd_HHmmss"
$outputFile = Join-Path $DesktopPath "EventLogs_${LogName}_${timestamp}.json"

try {
    # Calculate time range
    $startTime = (Get-Date).AddHours(-$Hours)
    
    # Get events from specified log
    Write-Host "Collecting events from $LogName log for the past $Hours hours..."
    
    Get-WinEvent -FilterHashtable @{
        LogName = $LogName
        StartTime = $startTime
    } | Select-Object TimeCreated, Level, LevelDisplayName, LogName, Id, 
                      ProviderName, Message | ConvertTo-Json -Depth 10 |
    Out-File -FilePath $outputFile -Encoding UTF8

    Write-Host "Events successfully exported to: $outputFile"
    Write-Host "Total events exported: $((Get-Content $outputFile | ConvertFrom-Json).Count)"
}
catch {
    Write-Error "Error occurred while exporting events: $_"
    exit 1
}

===== user create script

# PowerShell script to find user creation events
param (
    [int]$DaysBack = 30,  # Default to last 30 days
    [string]$DesktopPath = [Environment]::GetFolderPath("Desktop")
)

# Create timestamp for filename
$timestamp = Get-Date -Format "yyyyMMdd_HHmmss"
$outputFile = Join-Path $DesktopPath "UserCreationEvents_${timestamp}.json"

try {
    # Calculate time range
    $startTime = (Get-Date).AddDays(-$DaysBack)
    
    Write-Host "Searching for user creation events in the past $DaysBack days..."
    
    # Get user creation events (Event ID 4720)
    $events = Get-WinEvent -FilterHashtable @{
        LogName = 'Security'
        ID = 4720  # User account created
        StartTime = $startTime
    } | ForEach-Object {
        $xml = [xml]$_.ToXml()
        
        # Extract relevant data from event
        @{
            TimeCreated = $_.TimeCreated
            CreatedUser = $xml.Event.EventData.Data | Where-Object { $_.Name -eq 'TargetUserName' } | Select-Object -ExpandProperty '#text'
            CreatedBy = $xml.Event.EventData.Data | Where-Object { $_.Name -eq 'SubjectUserName' } | Select-Object -ExpandProperty '#text'
            AccountType = $xml.Event.EventData.Data | Where-Object { $_.Name -eq 'TargetUserName' } | Select-Object -ExpandProperty '#text'
            Domain = $xml.Event.EventData.Data | Where-Object { $_.Name -eq 'TargetDomainName' } | Select-Object -ExpandProperty '#text'
            SecurityID = $xml.Event.EventData.Data | Where-Object { $_.Name -eq 'TargetSid' } | Select-Object -ExpandProperty '#text'
            WorkstationName = $xml.Event.EventData.Data | Where-Object { $_.Name -eq 'WorkstationName' } | Select-Object -ExpandProperty '#text'
        }
    } | ConvertTo-Json -Depth 10

    # Save to file
    $events | Out-File -FilePath $outputFile -Encoding UTF8

    # Display summary
    $eventCount = ($events | ConvertFrom-Json).Count
    Write-Host "Found $eventCount user creation events"
    Write-Host "Results exported to: $outputFile"
    
    # Display recent creations on screen
    if ($eventCount -gt 0) {
        Write-Host "`nMost recent user creations:"
        $events | ConvertFrom-Json | Select-Object TimeCreated, CreatedUser, CreatedBy, Domain | Format-Table -AutoSize
    }
}
catch {
    Write-Error "Error occurred while searching events: $_"
    exit 1
}

========= find user

# Default usage (checks last 30 days)
.\find-user-creations.ps1

# Specify different time range (e.g., last 90 days)
.\find-user-creations.ps1 -DaysBack 90

======= INSTALLING SYSLOG FORWARDER

import socket
import json
import time
import os
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
import codecs

class JsonFileHandler(FileSystemEventHandler):
    def __init__(self, syslog_server, syslog_port):
        self.syslog_server = syslog_server
        self.syslog_port = syslog_port
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

    def on_modified(self, event):
        if not event.is_directory and event.src_path.endswith('.json'):
            try:
                # Wait for file to be completely written
                time.sleep(1)
                
                # Open file with UTF-8-SIG to handle BOM
                with codecs.open(event.src_path, 'r', encoding='utf-8-sig') as file:
                    content = file.read().strip()
                    print(f"Reading file: {event.src_path}")
                    print(f"Content length: {len(content)}")
                    
                    if not content:
                        print("Warning: File is empty")
                        return
                    
                    # Parse JSON content
                    json_data = json.loads(content)
                    
                    # Convert back to string for sending
                    message = json.dumps(json_data).encode('utf-8')
                    
                    # Send to syslog server
                    self.sock.sendto(message, (self.syslog_server, self.syslog_port))
                    print(f"Successfully sent data to {self.syslog_server}:{self.syslog_port}")
                    
            except json.JSONDecodeError as e:
                print(f"JSON Error: {str(e)}")
                print(f"Error occurred at position: {e.pos}")
                print(f"Line with error: {content[max(0, e.pos-50):e.pos+50]}")
            except Exception as e:
                print(f"Error processing file: {str(e)}")

def main():
    # Configuration
    WATCH_PATH = os.path.join(os.path.expanduser("~"), "Desktop")
    SYSLOG_SERVER = "192.168.1.223"  # Your Ubuntu server IP
    SYSLOG_PORT = 514

    print(f"Starting watcher on {WATCH_PATH}")
    print(f"Forwarding to {SYSLOG_SERVER}:{SYSLOG_PORT}")

    event_handler = JsonFileHandler(SYSLOG_SERVER, SYSLOG_PORT)
    observer = Observer()
    observer.schedule(event_handler, WATCH_PATH, recursive=False)
    observer.start()

    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()
        print("\nStopping watcher...")
    observer.join()

if __name__ == "__main__":
    main()

=================