Advanced `ps` Command Usage for Linux Process Investigation

Table of Contents

• Introduction
• The ps Command: An Overview
• Basic ps Usage and Common Flags
• Filtering and Refining ps Output
  • Excluding Kernel Threads
  • Monitoring Specific Users
  • Searching for Specific Commands
  • Inverting Selections with deselect
• Advanced Investigation Techniques
  • Identifying Processes Not from init/systemd
  • Correlating Processes with Start Times
• Conclusion

Introduction

The ps (process status) command is a fundamental utility in Linux systems, providing a snapshot of current processes. While its basic usage is common, the sheer volume of information in its default output often makes in-depth investigation challenging. This article delves into advanced techniques for leveraging the ps command to efficiently identify suspicious processes, troubleshoot system issues, and gain clearer insights into system activity. We will explore various flags, environment variables, and combinations to refine output and pinpoint critical information, moving from basic usage to sophisticated investigative methods.

The ps Command: An Overview

The ps command displays information about running processes. It’s an indispensable tool for system administrators, security analysts, and developers alike. Historically, ps has evolved with two main syntaxes:

• Unix-style options: Preceded by a dash (e.g., -ef).
• BSD-style options: Without a dash (e.g., aux).

Both styles achieve similar results, but their specific flags and behaviors can differ. This guide will primarily focus on Unix-style options due to their commonality in many modern Linux environments.

Basic ps Usage and Common Flags

Running ps without any parameters will only display processes owned by the current user in the current terminal session.

ps

To see a more comprehensive list, including processes from all users and a full listing of columns, the -ef flags are commonly used.

• -e: Selects all processes.
• -f: Displays a full-format listing, showing more columns like UID, PID, PPID, C, STIME, TTY, TIME, and CMD.

ps -ef

The output of ps -ef can still be overwhelming. To focus on specific details, we can customize the output columns using the -o flag. For example, to view only the user, PID, parent PID (PPID), and the command:

ps -o user,pid,ppid,cmd

For better readability and syntax highlighting, especially when dealing with extensive output, piping ps to tools like bat can be very useful.

ps -o user,pid,ppid,cmd | bat -l sh --style=full

If line numbers are not desired, they can be removed from bat’s style parameters:

ps -o user,pid,ppid,cmd | bat -l sh --style=header,grid

Filtering and Refining ps Output

Excluding Kernel Threads

One common source of “noise” in ps output is the presence of kernel threads, often appearing in square brackets (e.g., [kworker/u16:0]). These are internal kernel processes and are usually not relevant for user-level process investigation. They can significantly inflate the output.

To exclude these kernel threads, you can set the PS_UNIXWARE environment variable:

export PS_UNIXWARE=1
ps -ef

Comparing the count of processes with and without this variable demonstrates its effectiveness:

# Count processes including kernel threads
ps -ef | wc -l

# Set environment variable and count again
export PS_UNIXWARE=1
ps -ef | wc -l

The difference can be substantial, making the output much cleaner and easier to analyze. For the remainder of this guide, we will assume PS_UNIXWARE=1 is set to improve readability.

Monitoring Specific Users

When investigating user-specific activity, simply greping the ps -ef output for a username can lead to false positives, as the username might appear within a command’s arguments or path. A more precise method is to use the -u flag.

To view all processes for a specific user, use the -fu flags followed by the username:

ps -fu <username>

For example, to check processes run by the root user:

ps -fu root

This syntax ensures that only processes actually owned by the specified user are displayed, making it ideal for monitoring suspicious activity related to a particular account.

Searching for Specific Commands

Similar to user monitoring, greping for a command name can also yield noisy results. The -C flag allows you to search for processes by their command name directly, providing a much cleaner output.

To check if a specific command, like cron, is running:

ps -C cron

If you need more details about the process, such as the user running it, you can combine -C with the full format flag -f:

ps -fC cron

The -C flag automatically searches across all users and processes in the system, so there’s no need to include -e. You can also search for multiple commands by separating them with commas:

ps -C cron,xorg

This is particularly useful when trying to ascertain if a set of expected services or applications are active.

Inverting Selections with deselect

Sometimes, it’s more efficient to identify what’s not on a whitelist of known processes rather than trying to define a blacklist. The --deselect (or -N) flag allows you to invert the selection criteria. This is powerful for quickly spotting unknown or potentially malicious processes.

To exclude a specific command, such as your display manager (gdm3 or sddm), from the ps -ef output:

ps -ef -C gdm3 --deselect

While this example excludes only one command, in a real-world scenario, you might have a list of known, benign processes. This list can be stored in a variable for convenience:

KNOWN_COMMANDS="systemd,gnome-shell,firefox,code" # Example list
ps -ef -C $KNOWN_COMMANDS --deselect

This technique can quickly reveal processes that are not part of your expected system operations, potentially highlighting malware, reverse shells, or other unauthorized software.

Advanced Investigation Techniques

Identifying Processes Not from init/systemd

The init process (PID 1, often systemd on modern Linux systems) is the first process started during boot and is the parent of almost all other processes. Processes with a PPID (Parent Process ID) of 1 are typically core system services or daemons launched directly by init/systemd.

To identify processes that were launched by other means—i.e., those whose parent is not init/systemd—you can filter the output. This can be a strong indicator of user-launched scripts, lateral movement tools, or user-space implants that might bypass typical system startup mechanisms.

ps -eo pid,ppid,user,cmd | awk '$2 != 1 {print}'

This command lists processes and their parent PIDs, then filters out any process whose PPID is 1. The remaining processes might warrant closer inspection. However, be aware that sophisticated malware might attempt to hijack systemd itself, in which case this technique alone might not suffice.

Correlating Processes with Start Times

The process start time is a crucial piece of information for forensic analysis. If a system suddenly becomes sluggish after a software installation, examining recently started processes can help correlate events.

You can sort ps output by start time to quickly identify the most recent processes. The start_time field can be used for this purpose.

ps -eo pid,ppid,user,cmd,start_time --sort=start_time

The output will show processes sorted by their start time, with the most recently launched processes appearing at the bottom. This allows you to quickly see what has started recently, which can be invaluable for:

• Troubleshooting: Identifying processes that started around the time a problem occurred.
• Security Investigations: Spotting suspicious binaries or scripts that began execution after a compromise or a questionable download. For instance, a crypto miner might start shortly after installing an untrusted application.

Conclusion

The ps command, when used effectively, transcends its basic function to become a powerful tool for system monitoring, security analysis, and troubleshooting. By mastering its various flags, filtering capabilities, and output customization, you can transform a noisy list of processes into actionable intelligence. From excluding kernel threads and targeting specific users or commands to inverting selections and correlating events with process start times, the techniques outlined in this guide empower you to gain a much clearer understanding of your Linux system’s activity. Always remember to consult the man ps page for a comprehensive list of all available options and their combinations, as the flexibility of ps is vast.