Linux Log Management and Analysis — Cybersecurity

Objective. Students locate, read, and analyze key Linux log files to identify security events and indicators of compromise.

Why Log Analysis Matters in Cybersecurity

Every meaningful event on a Linux system — a user login, a failed authentication attempt, a kernel error, a package installation — is recorded in a log file. For a security analyst, logs are the primary evidence trail. The CompTIA CySA+ exam tests your ability to analyze log data under the Security Operations and Monitoring domain, and the ISC2 CC credential expects you to understand incident response fundamentals, which depend heavily on reading logs accurately.

Log analysis is a core daily task in Security Operations Centers (SOCs). Learning to extract, filter, and correlate log data by hand builds the mental model you need to operate SIEM tools effectively.

Key Log Files in /var/log

On Debian and Ubuntu-based Linux systems, most log files live in /var/log. Know what each file records:

/var/log/auth.log — Authentication events: SSH logins (successful and failed), sudo usage, su commands, PAM module activity. This is your primary source for detecting brute-force attacks and unauthorized access.
/var/log/syslog — General system messages from the kernel and services. Broad scope; useful for correlating events across components.
/var/log/kern.log — Kernel-specific messages including hardware events and driver errors. Useful when investigating rootkits or unusual kernel module activity.
/var/log/dpkg.log — Records every package installation, removal, or upgrade. An attacker who installs a backdoor tool will leave a trace here if they use the package manager.
/var/log/fail2ban.log — Records which IP addresses fail2ban has detected and banned. Comparing this file with auth.log helps confirm whether automated blocking is functioning.

On systems using systemd, the journalctl command queries the binary journal, which can include all of the above plus boot messages, service start/stop events, and more. Always check both traditional text logs and the journal during an investigation.

Essential Command-Line Tools

grep filters lines by pattern. To find all failed SSH login attempts in auth.log:

```

grep "Failed password" /var/log/auth.log

```

awk lets you extract specific fields. SSH failure lines contain the source IP in a predictable position. For example:

```

grep "Failed password" /var/log/auth.log | awk '{print $11}'

```

cut extracts columns by delimiter. Combine it with sort and uniq to count unique IPs:

```

grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -rn

```

This pipeline prints each unique source IP alongside the count of failures, sorted highest to lowest — a fast way to spot a brute-force source.

journalctl filters the systemd journal. To view SSH-related events since midnight:

```

journalctl -u ssh --since today

```

Add -r to reverse chronological order (newest first) and -n 50 to limit output to 50 lines.

Reconstructing a Timeline

When an incident is suspected, your job is to build a timestamped narrative of what happened and in what order. Linux log timestamps are typically in the format Month Day HH:MM:SS. Steps for timeline reconstruction:

1. Identify the approximate window of interest (when did the anomaly begin?).

2. Use grep with a time-range pattern or journalctl's --since and --until flags to narrow scope.

3. Collect relevant lines from multiple log files into a single working file using redirection and append (>>).

4. Sort all lines by timestamp with sort.

5. Read the sorted output and write your narrative, linking each event to its log source.

A well-written incident narrative includes: who (account name or IP), what (action taken), when (timestamp), where (log file and system), and outcome (success or failure).

Centralized Logging with rsyslog

Collecting logs only on the host being attacked creates a risk: an attacker with root access can modify or delete local log files. Forwarding logs to a separate, hardened syslog server addresses this problem. This is the foundation of SIEM log pipelines.

rsyslog is the standard log daemon on most Debian/Ubuntu systems. To forward auth.log entries to a remote server, you edit /etc/rsyslog.conf or add a file in /etc/rsyslog.d/ on the sending host. A basic forwarding rule uses the auth facility and sends over UDP or TCP to the receiving server's IP. On the receiving server, rsyslog must be configured to listen on the appropriate port.

Verify log receipt in real time with:

```

tail -f /var/log/syslog

```

on the receiving server while generating login events on the sending VM.

Log integrity means ensuring logs have not been altered after the fact. Forwarding to a remote server immediately upon generation is a practical first layer. In full SIEM pipelines, cryptographic hashing and chain-of-custody controls add further assurance.

Safety and Best Practices

Perform all lab exercises on isolated VMs only. Never run brute-force simulations, log injections, or network forwarding exercises against production systems or systems you do not own and have explicit permission to test.
Treat log data as sensitive. Even in a lab, logs may contain usernames and IP addresses. Handle them according to your program's data-handling policy.
When writing incident narratives, report only what the evidence shows. Avoid speculation; distinguish between confirmed events and inferences.
Maintain a clean audit trail of your own commands by keeping a terminal log or notes — this models professional incident-response discipline.

Key terms9

auth.logA Linux log file (typically at /var/log/auth.log) that records authentication events including SSH logins, sudo use, and PAM activity; a primary source for detecting unauthorized access.

journalctlA command-line utility for querying the systemd binary journal, which aggregates log messages from the kernel, services, and applications with structured metadata.

grepA Linux command that searches text input or files for lines matching a specified pattern, used extensively in log analysis to filter relevant events.

rsyslogThe Reliable System Logger daemon used on many Linux distributions to collect, process, and optionally forward log messages to local files or remote syslog servers.

Brute-force attackAn attack method in which an adversary systematically tries many passwords or credentials against an account, typically visible in auth.log as a high volume of 'Failed password' entries from one source IP.

Privilege escalationAn attack technique in which a user or process gains a higher level of access than intended, often evidenced in auth.log by sudo or su activity following an initial compromise.

Log integrityThe assurance that log records have not been altered or deleted after generation; achieved in practice by forwarding logs to remote servers and applying cryptographic controls in SIEM pipelines.

Indicator of Compromise (IoC)A piece of evidence in logs, network traffic, or files that suggests a system may have been breached, such as repeated failed logins from an unfamiliar IP followed by a successful login.

fail2banAn intrusion prevention service that monitors log files (such as auth.log) for repeated failure patterns and automatically adds firewall rules to block offending IP addresses; activity is recorded in fail2ban.log.

In class: Map the major log files (/var/log/auth.log, syslog, kern.log, dpkg.log, fail2ban.log) to the type of security events each records; perform grep, awk, and cut exercises to extract all failed login attempts from auth.log and count unique source IPs. · Instructor injects a simulated attack scenario into the VM's log files (failed SSH brute force followed by a successful login and privilege escalation); students use journalctl, grep, and sort to reconstruct the timeline of events and write a brief incident narrative. · Configure rsyslog to forward a copy of auth.log entries to a second VM acting as a central syslog server; verify receipt on the server using tail -f and confirm the forwarded messages match the originals — introduce the concept of log integrity for SIEM pipelines (previewing Course 6).
Materials: Two Linux VMs, rsyslog, journalctl, grep, awk, cut, sort, CompTIA CySA+ Study Guide (Security Operations and Monitoring domain), ISC2 CC domain notes on incident response
Check: Students submit the incident narrative with a timestamped event timeline; pass = at least four distinct events correctly identified in sequence, each linked to the correct log source.

Practice check8

Answer each question, then check it. Nothing is recorded; this is your own practice.