Category filter
Analyzing Linux System Logs Remotely via Custom Scripting
With increasing cyberthreats and safety concerns, system admins are finding it risky to remotely access a system endpoint for troubleshooting through the conventional method of manual SSHing (secure shelling).
This guide enables IT administrators to remotely assess Linux device health with scripts executed via Execute Custom Script feature. By automating the retrieval of critical system logs, you can perform non-invasive forensic analysis across Ubuntu, RHEL, CentOS, and Debian fleets without requiring direct SSH access.
Scripting Objective
The primary goal of this guide is to enable IT Administrators to remotely access critical system data logs from Linux endpoints. In an enterprise environment, logging into every machine via SSH to check out a crash or unauthorized access is inefficient and poses major security risks.
With the help of this lightweight Bash script that retrieves the system, security, and hardware logs, system admins can perform rapid, centralized diagnostics directly within the Hexnode UEM portal.
The script given here targets four specific areas:
- System Health: Checks if services are running or failing.
- Security: Identifies failed login attempts or unauthorized sudo usage.
- Hardware: Detects kernel errors or disk issues.
- Modern Logs: Pulls data from journalctl (the modern Linux logging standard).
Script Characteristics:
- Idempotency: The script performs strictly read-only operations, ensuring no modification to the system’s state.
- Minimal Footprint: By utilizing tail -n 50, the script ensures minimal CPU cycles and network bandwidth consumption.
- Cross-Distribution Compatibility: The logic includes conditional checks to identify log path variations across major Linux distributions.
System Compatibility
To ensure high-fidelity data retrieval across a diverse Linux fleet, the managed endpoints must meet the following environmental specifications.
| Category | Supported Specifications |
|---|---|
| Operating Systems |
|
| Hexnode Agent | Must be installed and active on the endpoint. |
| Shell Environment | Bash 4.2+ (Standard on almost all Linux distros released after 2012). |
| Logging Systems |
|
Bash Diagnostics Script Snippet
This script automates the extraction of critical system events across your remote Linux environment.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 |
#!/bin/bash # Objective: Remote Log Retrieval with Rotation Awareness for Hexnode UEM echo "=== System Log (syslog / messages) ===" # Checks current and the most recent rotated system log if [ -f /var/log/syslog ]; then echo "[Current syslog]" && tail -n 50 /var/log/syslog [ -f /var/log/syslog.1 ] && echo -e "\n[Recent Archive - syslog.1]" && tail -n 20 /var/log/syslog.1 elif [ -f /var/log/messages ]; then echo "[Current messages]" && tail -n 50 /var/log/messages [ -f /var/log/messages.1 ] && echo -e "\n[Recent Archive - messages.1]" && tail -n 20 /var/log/messages.1 else echo "System log not found." fi echo "" echo "=== Authentication Log (auth.log / secure) ===" # Checks current and rotated authentication logs for security auditing if [ -f /var/log/auth.log ]; then echo "[Current auth.log]" && tail -n 50 /var/log/auth.log [ -f /var/log/auth.log.1 ] && echo -e "\n[Recent Archive - auth.log.1]" && tail -n 20 /var/log/auth.log.1 elif [ -f /var/log/secure ]; then echo "[Current secure]" && tail -n 50 /var/log/secure [ -f /var/log/secure.1 ] && echo -e "\n[Recent Archive - secure.1]" && tail -n 20 /var/log/secure.1 else echo "Authentication log not found." fi echo "" echo "=== Kernel Log ===" if [ -f /var/log/kern.log ]; then tail -n 50 /var/log/kern.log else echo "Kernel log not found. Alternative: use dmesg" dmesg | tail -n 50 fi echo "" echo "=== Journalctl (systemd logs) ===" # Journalctl automatically handles rotation and archives internally [cite: 55] journalctl -n 50 --no-pager |
Script Execution steps
Save the above code as a .sh file and deploy it using the Execute Custom Script action available within the Hexnode UEM portal.
- Go to Manage > Devices > Select your Linux device.
- Select Actions > Execute Custom Script.
- Upload the .sh file and click Execute.
Verifying the results
Once the script is executed, the results are captured and displayed within the Hexnode console. This allows you to review the device’s status without ever leaving your browser.
To access the retrieved logs, follow these steps within your Hexnode portal:
- Navigate to the Action History tab under the specific device.
- Locate the entry for the log retrieval script (e.g. View logs.sh) in the Subject column.
- Click the Show Output button located next to the status field.
The output window will display a text-based summary of the retrieved logs. This data is categorized by the headers defined in the script (e.g., === System Log ===), making it easy to identify specific issues.
Result Interpretation
When reviewing the output, keep the following logic in mind:
- Current Logs: This is the most recent data captured by the system.
- Unified Journal: The journalctl section provides a continuous list of background service activities. It automatically combines old and new records into a single view, making it the best place to track the timeline of a specific app or service failure.
Admin Verification
- Success Confirmation: A successful run will display the logs as plain text.
- Error Handling: If a specific log file is not found (e.g., if the OS uses secure instead of auth.log), the output will explicitly state “Log not found” for that category, allowing the admin to quickly identify the distribution type.
Use the Ctrl+F search command while viewing the script output to efficiently locate error logs or specific activity markers like “Failed” or “Panic.”
Log Source Mapping and Metadata
The following table outlines the technical sources targeted by this script:
| Log Category | Primary Distributions | Standard Locations | Primary Tool | Diagnostic Use Case |
|---|---|---|---|---|
| System/General | Ubuntu, Debian, RHEL, CentOS, Fedora | /var/log/syslog, /var/log/messages | tail, cat | Monitoring general service status and non-critical system events. |
| Authentication | Ubuntu, Debian, RHEL, CentOS, Fedora | /var/log/auth.log, /var/log/secure | tail | Auditing user logins, sudo elevations, and SSH access activity. |
| Kernel | All Linux Distributions | /var/log/kern.log | tail, dmesg | Identifying hardware failures, driver conflicts, and critical system crashes. |
| Systemd Journal | All modern Linux (2015+) | Managed by systemd | journalctl | Unified log storage for services and applications on modern systems. |
Strategic Use Cases
With Hexnode UEM’s Execution Custom Script feature, transform simple scripting into a scalable infrastructure management tool. The following matrix outlines the strategic scenarios where this script provides maximum business impact across your infrastructure.
| Scenario | Hexnode Action/Feature | Business Impact |
|---|---|---|
| Post-Update Validation | Deploy script to a test group following a kernel or OS patch. | Infrastructure Stability: Verify that all critical services started correctly, and no new kernel errors were introduced by the update, ensuring a stable environment before a full rollout. |
| App Troubleshooting | Use Execute Custom Script to pull journalctl and syslog output for a specific service tag. | Rapid Root Cause Analysis: Identify why an enterprise application failed to launch by viewing the exact error code (e.g., “Permission Denied” or “Port in Use”) directly in the Hexnode portal, eliminating the need for time-consuming user screen-sharing sessions. |
| Remote Troubleshooting | Execute script on a single, reporting-issue device. | Reduced Downtime: Instantly retrieve kernel errors or failed service startup logs to diagnose problems like a crashed application or driver conflict without physical device access. |
| Security Audit & Compliance | Schedule script execution across all Linux endpoints or specific device groups. | Enhanced Security Posture: Rapidly scan auth.log (secure) across the fleet for unauthorized login attempts (e.g., brute-force via SSH), fulfilling compliance requirements for log review. |
| User Activity Verification | Execute script after a policy violation report or even a suspicious activity alert. | Forensic Analysis: Check systemd journal or auth.log for details on when a specific service was stopped or if a restricted user elevated privileges via sudo. |
Operational Notes for Administrators
- Permissions: The script, when executed via the Hexnode agent, typically runs with elevated privileges, ensuring access to sensitive log files like /var/log/secure.
- Verification & Validation: It is highly recommended to manually validate script execution on a single test system before initiating bulk actions.
- Liability Disclaimer: While this script is based on vetted open-source logic, it is provided ‘as-is’. Hexnode assumes no liability for unintended system behavior or data loss.
Hexnode UEM’s robust remote script execution capabilities allow administrators to unify the management of disparate Linux environments. This framework replaces manual, fragmented diagnostics with a centralized, real-time audit of infrastructure health. By automating log retrieval and system analysis, organizations can move towards a proactive security posture, ensuring consistent reliability across the managed fleet.
