SPLUNK: The Ultimate SIEM for Control

1. Introduction and Objectives

In this article, we will step into the shoes of a security analyst whose task involves detecting intrusions by analyzing the various log files generated by machines and devices, searching for abnormal behaviors. This is done to respond to potential incidents, establish appropriate policies and rules, and ensure their compliance. Additionally, this role includes the implementation of Security Information and Event Management (SIEM) systems.

For this exercise, we will use the tool “Splunk Enterprise.” In the following sections, we will define this tool and its functionality, as well as briefly explore some use cases that can prove to be highly beneficial in our daily operations.

2. What is Splunk?

Essentially, we can define “Splunk” as a tool responsible for collecting, parsing, storing, and correlating data in real-time.

The sources of this collected data are highly diverse, as they can include data from enterprise networks, systems and infrastructure, cloud services, and more. In essence, any machine or device that generates event logs can be monitored.

Data is typically collected and stored in Splunk using a “syslog” or one of its agents, known as a “forwarder.” Later on, we will discuss these concepts in more detail. Once collected, this data is sent to the main server, known as the “indexer,” and stored in data storage units called “indexes.”

This data can be processed to perform various tasks, such as generating intelligence, detecting potential attacks, or even generating “Big Data” insights.

The real-time collection and storage of this data enable early detection, allowing for necessary corrections to be applied to prevent an attack from becoming effective, or at least minimizing its impact on the organization.

Splunk features a user-friendly web interface, allowing users to review logs, create dashboards, generate reports, or schedule alerts.

We could assert that Splunk is indeed a powerful, flexible, and scalable tool. It empowers users to create their own applications or download them from its repository called “splunkbase.” In this repository, the majority of applications are free, with some being developed by Splunk itself and others by renowned security companies like Fortinet, CheckPoint, or Cisco.

There is the possibility to add additional paid modules to Splunk, which can transform Splunk into one of the most comprehensive SIEM solutions on the market. Examples include Splunk Enterprise Security, Splunk Phantom, and Splunk User Behavior Analytics, among others. These security modules enhance Splunk’s capabilities by providing predefined tools for detecting malicious behaviors through key security indicators, event correlation, and offering dashboards, lookups, and more.

3. Splunk instances

The core of Splunk offers several solutions that can perfectly adapt to our needs, based on the available resources and the infrastructure to be monitored.

• Splunk Enterprise: An “on-premise” instance designed for installations on physical servers. It allows gaining insights into everything happening in the organization and enables quick decision-making if necessary.
• Splunk Cloud: A cloud-based service that provides greater flexibility to scale the infrastructure and reduce costs, eliminating the need to invest in physical servers and related maintenance.
• Splunk Data Stream Processor: Collects, processes, and distributes data to Splunk and other instances in milliseconds and real-time.

There is a free 60-day trial version of Splunk Enterprise available, with a limitation of indexing 500 MB/day. Furthermore, after the trial period, you can convert the license to a perpetual one. This makes it suitable for testing and getting acquainted with Splunk, and it can also be used as a testing and development instance. Once registered on their official website, you can download it from: https://www.splunk.com

4. Components of Splunk Enterprise

Splunk is composed of several components that help us deploy and collect data in an easy and efficient manner. The main components of Splunk are three:

• Splunk Enterprise: A complete instance of Splunk with all its features. It enables the collection, parsing, indexing, and subsequent processing of data through searches, dashboards, etc.
• Heavy Forwarder (HF): It is a forwarder, an instance of Splunk Enterprise that sends data to another Splunk instance. HF retains most of the features of an indexer but cannot perform distributed searches, among other limitations.
• Universal Forwarder (UF): It is the lightest and easiest-to-implement Splunk agent, serving as a simple data collector-forwarder. It gathers data from various systems, applications, networks, etc., and forwards it to another Splunk instance. It doesn’t have the ability to parse, search, or index data.

Each Splunk component serves one or more functions and is assigned one or more roles. The most notable functions include:

• Forwarder: Used for data collection, forwarding, and parsing.s.
• Indexer: Used for parsing and storing data..
• Search head: This is a graphical interface (Splunk Web) used for data manipulation, searching, analysis, and reporting.

5. Detecting Brute Force Attacks

Next, we are going to look at the case of a brute force attack on a Windows system. It should be noted that any system with an authentication module can be monitored, such as a web panel or the entry to an application that requires credentials.

Before conducting the analysis, we had to properly configure the reception of Windows security logs by installing a “Universal Forwarder” on the machine to be monitored. It could be installed on an Active Directory (AD) to monitor all machines associated with that AD.

Using the search feature provided by the Splunk web interface, we can perform searches in SPL language for the data we are indexing.

The search to be performed would be as follows:

*Important: The parameters entered in “index” and “sourcetype” should be adapted to the environment.

6. Detecting Port and Service Scanning

In these times, it is almost inevitable to continuously receive port and service scanning on exposed network applications.

Detecting this type of attack is very important because it allows us to anticipate other possible more elaborate attacks. As we all know, scanning is one of the first phases of an attack. This way, we can implement measures such as blocking IP addresses based on the number of requests, among others.

How could we detect this type of attack in real-time?

Once we are receiving logs from our network devices (firewall, IPS, router, etc.), we can perform the following search:

• Index: It’s the “bucket” where we store network logs.
• Sourcetype: It’s the type of data that we store, in this case, IP address events.
• Stats count dc: It performs a count of all the different destination ports and all the different destination IP addresses.
• Where: Condition that requires the results for destination ports to be greater than 500 and the results for destination IP addresses to be greater than 500.

7. Detecting New Administrator Accounts

Another important use case is detecting new user accounts in the administrator group.

When cybercriminals intrude into a victim machine, one of the first Tactics, Techniques, and Procedures (TTPs) they often execute is gaining access to other user accounts for lateral movement within the organization (pivoting, privilege escalation), or creating new user accounts to ensure persistence on the victim machine. If you’re interested in learning more about offensive cybersecurity, you can take a look at the Advanced Professional Course in Offensive Cybersecurity.

From our search interface analyzing Windows security logs, we can find information about user accounts.

• Index: It is the “bucket” where we store Windows logs in this case.
• Sourcetype: It is the type of data we store, in this case, Windows security events.
• “EventCode= 4720 Administrators”: New user account created with local administrator privileges.
• Transaction Security_ID maxspan=180: Search for all events with the same identification number that have occurred within a maximum of 180 minutes.
• Search EventCode=4720 EventCode=4732: Within the result of the previous search, filter only the events with codes 4720 and 4732.
• Table: Create a data table with the specified fields.

8. Monitoring our website

With Splunk, we can also create a dashboard for our website, allowing us to monitor it in real-time. This helps us check if the website is operational, detect directory discovery attacks, identify access attempts from different locations around the world, and more.

To achieve this, we need to install the Splunk Universal Forwarder (UF) on the server hosting the website. Then, we can access the Splunk Web interface and perform the desired searches to gather the necessary information for our dashboard.

• index=jms_web.
• status!=200: Search for all results where the status is not equal to 200 (OK).
• stats count by status: It counts the results and groups them by status.
• sort status: It displays the result sorted by status.

Now we are going to turn this search into a dashboard panel with charts to make it visually appealing and easier to manage.

Select “Visualization” and “Format” to choose the type of chart you want to use. Then select “Save as” > “Dashboard Panel,” and the result will be as follows:

Needless to say, both the colors of the graphs and the dark theme that we’ve chosen in this case can be customized and changed according to individual preferences.

9. Controlling the resource consumption of our infrastructure

Another very interesting use we can give to Splunk is to monitor the status of our infrastructure and create alerts in case something is not going well.

For example, we can monitor the consumption of RAM memory or storage disks. And if deemed appropriate, we can create alerts to trigger a notification or send an email with information that we are at 80% capacity of the disk or that more than 90% of RAM memory is being consumed.

To do this, we need to install the Splunk agent (UF) on the servers to be monitored and indicate which counters we want to collect logs from. Once this is done, we only need to go to Splunk Web and perform the appropriate searches or create panels (dashboards) with graphics to make the data more visual in our monitoring center.

In the images above, we can see a dashboard with metrics collected from Windows perfmon.

In the first image, it shows an average of available memory in MB, along with the highest peak and the current availability at the time of measurement, along with its last update. In the second image, we can observe the same metrics as in the first one, but with the addition of CPU usage.

As we have seen, this is a very quick and convenient way to keep our environment under control. We can add the most critical machines of our organization to monitor them in real-time and proactively address potential availability issues or possible attacks. For instance, if we observe a sudden and high utilization of resources such as memory or CPU, we might be facing a potential malware operating in the background, engaging in malicious activities, such as the current and infamous ransomware threats.

10. Conclusions

In this article, we have taken a brief look at what Splunk is capable of and have witnessed the potential this application holds, making it a global benchmark in Security Operations Centers (SOCs).

Splunk is a comprehensive tool that enables us to stay informed about everything happening within our organization in real-time. It provides us with the ability to respond promptly in case of incidents or malicious behavior, which is crucial in cases of malware infections like ransomware.

Moreover, Splunk is known for its versatility and ease of scalability, allowing for expansion in a secure and straightforward manner. It is compatible with all systems and architectures, including Android systems, allowing us to monitor our company’s smartphones and even geolocate them.

To conclude, it’s worth noting that other cybersecurity-oriented uses of Splunk include monitoring activity on TOR networks, detecting ransomware (such as the infamous “WannaCry”), and many other highly relevant applications.

Until next time! Wishing you successful Splunking 😉