IBM QRadar 5) Collecting File Logs
We will see how to collect file logs in this section.
Basically, we have to use FTP to collect file logs.
It has interval time, and logs are collected with interval time.
For security reason, many company don't allow FTP but we can use SFTP(Secure File Transfer Protocol).
SFTP is using SSH(Secure Shell) protocl to get file logs.
I am seeing most of servers allow FTP or SFTP, if there are some server that don't allow any connection you have to use tail2syslog that be used for forward file logs as syslog type.
I will show you about tail2syslog later.
FTP and SFTP's configuration are same without protocol selection.
Officially, let me show you how to configure to collect file logs.
1. Run Log Sources Menu
Move 'Admin' tab and click 'Log Sources' icon.
2. Add Log Source
Click 'Add' to add log list
3. Defualt view of Log Sources menu
4. Select file protocol (FTP or SFTP)
Select file protocol to collect file logs.
● Choose 'Universal DSM' in 'Log Source Type'
● Choose 'Log File' in 'Protocol Configuration'
● Choose 'FTP' or 'SFTP' in 'Service Type'
5. Fill information about detail information of log
In this section, we will type ip address, file location, file name, interval time and othters.
You don't need to fill all blanks that you saw as above picture.
'Log Source Name' and 'Log Source Description' don't give any problem to log. so you can fill this field as you want to do.
And other information like server ip address and port, authentication must be corrent to collect logs.
Just you want simple, you can fill these blank as number 5 picture.
Please see below information if you want to know every options.
● 'Log Source Name' : log's name
● 'Log Source Description' : log's description
● 'Log Source Type' : log's type
● 'Protocol Configuration' : form of logs
● 'Service Type' : log's protocol
● 'Remote IP or Hostname' : log server's ip address or hostname
● 'Remote Port' : FTP / SSH port
● 'Remote User' : account that have read permission
● 'Remote Password' : account's password
● 'Confirm Password' : repeat password
● 'SSH Key File' : SSH Key File informatin(it is not necessary)
● 'Remote Directory' : log file location
● 'Recursive' : collecting schedule
● 'FTP File Pattern' : log file name(here, regex can be used.)
● 'Start Time' : first collecting time
● 'Recurrence' : collecting interval time
● 'Run on Save' : auto save logs.
● 'EPS Throttle' : maximum of EPS
● 'Processor' : if log files are compressed, check it
● 'Ignore Previously Processed File(s)' : don't collect file that be processed
● 'Change Local Directory' : change log location in QRadar
● 'Event Generator' : form of logs
● 'File Encoding' : setting encoding of logs
● 'Folder Separator' : it is used if log directories are diffrent
● 'Enabled' : enable collecting log after configure
● 'Credibility' : important asset
● 'Target Event Collector' : event processor that collect logs
● 'Coalescing Payload' : many logs that have same information will be shown one log
● 'Store Event Payload' : save original log payload
● 'Log Source Language' : set language of log
Basically, we have to use FTP to collect file logs.
It has interval time, and logs are collected with interval time.
For security reason, many company don't allow FTP but we can use SFTP(Secure File Transfer Protocol).
SFTP is using SSH(Secure Shell) protocl to get file logs.
I am seeing most of servers allow FTP or SFTP, if there are some server that don't allow any connection you have to use tail2syslog that be used for forward file logs as syslog type.
I will show you about tail2syslog later.
FTP and SFTP's configuration are same without protocol selection.
Officially, let me show you how to configure to collect file logs.
1. Run Log Sources Menu
Move 'Admin' tab and click 'Log Sources' icon.
2. Add Log Source
Click 'Add' to add log list
3. Defualt view of Log Sources menu
4. Select file protocol (FTP or SFTP)
Select file protocol to collect file logs.
● Choose 'Universal DSM' in 'Log Source Type'
● Choose 'Log File' in 'Protocol Configuration'
● Choose 'FTP' or 'SFTP' in 'Service Type'
5. Fill information about detail information of log
In this section, we will type ip address, file location, file name, interval time and othters.
You don't need to fill all blanks that you saw as above picture.
'Log Source Name' and 'Log Source Description' don't give any problem to log. so you can fill this field as you want to do.
And other information like server ip address and port, authentication must be corrent to collect logs.
Just you want simple, you can fill these blank as number 5 picture.
Please see below information if you want to know every options.
● 'Log Source Name' : log's name
● 'Log Source Description' : log's description
● 'Log Source Type' : log's type
● 'Protocol Configuration' : form of logs
● 'Service Type' : log's protocol
● 'Remote IP or Hostname' : log server's ip address or hostname
● 'Remote Port' : FTP / SSH port
● 'Remote User' : account that have read permission
● 'Remote Password' : account's password
● 'Confirm Password' : repeat password
● 'SSH Key File' : SSH Key File informatin(it is not necessary)
● 'Remote Directory' : log file location
● 'Recursive' : collecting schedule
● 'FTP File Pattern' : log file name(here, regex can be used.)
● 'Start Time' : first collecting time
● 'Recurrence' : collecting interval time
● 'Run on Save' : auto save logs.
● 'EPS Throttle' : maximum of EPS
● 'Processor' : if log files are compressed, check it
● 'Ignore Previously Processed File(s)' : don't collect file that be processed
● 'Change Local Directory' : change log location in QRadar
● 'Event Generator' : form of logs
● 'File Encoding' : setting encoding of logs
● 'Folder Separator' : it is used if log directories are diffrent
● 'Enabled' : enable collecting log after configure
● 'Credibility' : important asset
● 'Target Event Collector' : event processor that collect logs
● 'Coalescing Payload' : many logs that have same information will be shown one log
● 'Store Event Payload' : save original log payload
● 'Log Source Language' : set language of log