St0pp3r's Blog
Posts Categories Tags Projects Other Work About
St0pp3r's Blog

Threat Hunting Using Pair Probabilities

st0pp3r  included in  category ThreatHunting
   15 minutes 

Upon digging around Microsoft’s documentation for user-defined functions I stumbled on the function pair_probabilities_fl(). This function calculates probabilities and some additional metrics for a pair of categorical variables, A and B. In this blog post, we are exploring the possibility of using the pair probabilities function for threat hunting.

The documentation explains the output metrics in detail, but I am going to briefly go through the key points to help provide some context for the rest of the blog. The original function is the following:

kusto

let pair_probabilities_fl = (tbl:(*), A_col:string, B_col:string, scope_col:string)
{
let T = materialize(tbl | extend _A = column_ifexists(A_col, ''), _B = column_ifexists(B_col, ''), _scope = column_ifexists(scope_col, ''));
let countOnScope = T | summarize countAllOnScope = count() by _scope;
let probAB = T | summarize countAB = count() by _A, _B, _scope | join kind = leftouter (countOnScope) on _scope | extend P_AB = todouble(countAB)/countAllOnScope;
let probA  = probAB | summarize countA = sum(countAB), countAllOnScope = max(countAllOnScope) by _A, _scope | extend P_A = todouble(countA)/countAllOnScope;
let probB  = probAB | summarize countB = sum(countAB), countAllOnScope = max(countAllOnScope) by _B, _scope | extend P_B = todouble(countB)/countAllOnScope;
probAB
| join kind = leftouter (probA) on _A, _scope           // probability for each value of A
| join kind = leftouter (probB) on _B, _scope           // probability for each value of B
| extend P_AUB = P_A + P_B - P_AB                       // union probability
       , P_AIB = P_AB/P_B                               // conditional probability of A on B
       , P_BIA = P_AB/P_A                               // conditional probability of B on A
| extend Lift_AB = P_AB/(P_A * P_B)                     // lift metric
       , Jaccard_AB = P_AB/P_AUB                        // Jaccard similarity index
| project _A, _B, _scope, bin(P_A, 0.00001), bin(P_B, 0.00001), bin(P_AB, 0.00001), bin(P_AUB, 0.00001), bin(P_AIB, 0.00001)
, bin(P_BIA, 0.00001), bin(Lift_AB, 0.00001), bin(Jaccard_AB, 0.00001)
| sort by _scope, _A, _B
};

The function takes four arguments: the input table, column A, column B, and an optional scope column. Columns A and B represent the two categorical variables whose relationship we want to analyse, while the scope column can be used to calculate the probabilities within a specific context or population. For example, A could represent the parent process and B the child process, allowing us to calculate which child processes are commonly or unusually spawned by a given parent. In that context, the scope column could then be the OS version, allowing us to calculate these probabilities separately for different operating system versions.

The function returns several metrics that describe how often two categorical values occur individually and together, and how strongly they are related:

  • P_A: How often a particular value of A occurs within the dataset.
  • P_B: How often a particular value of B occurs within the dataset.
  • P_AB: How often A and B occur together within the dataset.
  • P_AUB: How often A or B occurs within the dataset, including cases where they occur together.
  • P_AIB: How often A occurs when B is present within the dataset. A low value means A is uncommon when B occurs.
  • P_BIA: How often B occurs when A is present within the dataset. A low value means B is uncommon when A occurs.
  • Lift_AB: Measures how strongly A and B are related compared with what we would expect if they were independent of each other. A Lift_AB value close to 1 means they occur together about as often as we would expect if there is no relationship between them. A value greater than 1 means they occur together more often than expected, while a value less than 1 means they occur together less often than expected.
  • Jaccard_AB: Shows how much A and B overlap within the dataset. In simple terms, it looks at how frequently A and B occur together compared with how frequently either one occurs. A value close to 1 means A and B are strongly shared, while a value close to 0 means they rarely occur together.

To make the results more actionable for threat hunting, the function was modified to provide additional context around the relationships it identifies. The original function tells us how common or unusual a pair of values, A and B, is, but when evaluating the results, additional context is useful, such as understanding the most common relationships associated with each variable (A or B), identifying devices where the pair has been observed, or collecting additional field values for context.

The modified function is provided below:

kusto

let pair_probabilities_ext_fl = (tbl:(*), A_col:string, B_col:string, scope_col:string, cxt_col_C:string, cxt_col_D:string)
{
let T = materialize(tbl | extend _A = column_ifexists(A_col, ""), _B = column_ifexists(B_col, ""), _scope = column_ifexists(scope_col, ""), cxt_col_C = column_ifexists(cxt_col_C, ""), cxt_col_D = column_ifexists(cxt_col_D, "") | project _A, _B, _scope, cxt_col_C, cxt_col_D);
let countOnScope = T | summarize countAllOnScope = count() by _scope;
let countABOnScope = T | summarize c = count() by _A, _B, _scope | extend Occurrences = bag_pack(_B, c) | summarize A_B_Occurrences = make_bag(Occurrences) by _A, _scope | mv-apply A_B_Occurrences on (extend bag_as_array = bag_keys(A_B_Occurrences) | mv-expand key = bag_as_array | extend value = toint(A_B_Occurrences[tostring(key)]) | top 10 by value desc | summarize A_B_Occurrences = make_bag(pack(tostring(key), value)));
let countBAOnScope = T | summarize c = count() by _A, _B, _scope | extend Occurrences = bag_pack(_A, c) | summarize B_A_Occurrences = make_bag(Occurrences) by _B, _scope | mv-apply B_A_Occurrences on (extend bag_as_array = bag_keys(B_A_Occurrences) | mv-expand key = bag_as_array | extend value = toint(B_A_Occurrences[tostring(key)]) | top 10 by value desc | summarize B_A_Occurrences = make_bag(pack(tostring(key), value)));
let cxtCVals = T | summarize cxtCVals = make_set(cxt_col_C) by _A, _B, _scope;
let cxtDVals = T | summarize cxtDVals = make_set(cxt_col_D) by _A, _B, _scope;
let probAB = T | summarize countAB = count() by _A, _B, _scope | join kind = leftouter (countOnScope) on _scope | extend P_AB = todouble(countAB)/countAllOnScope;
let probA  = probAB | summarize countA = sum(countAB), countAllOnScope = max(countAllOnScope) by _A, _scope | extend P_A = todouble(countA)/countAllOnScope;
let probB  = probAB | summarize countB = sum(countAB), countAllOnScope = max(countAllOnScope) by _B, _scope | extend P_B = todouble(countB)/countAllOnScope;
probAB
| join kind = leftouter (probA) on _A, _scope
| join kind = leftouter (probB) on _B, _scope
| join kind = leftouter (countABOnScope) on _A, _scope
| join kind = leftouter (countBAOnScope) on _B, _scope
| join kind = leftouter (cxtCVals) on _A, _B, _scope
| join kind = leftouter (cxtDVals) on _A, _B, _scope
| extend P_AUB = P_A + P_B - P_AB
       , P_AIB = P_AB/P_B
       , P_BIA = P_AB/P_A
| extend Lift_AB = P_AB/(P_A * P_B)
       , Jaccard_AB = P_AB/P_AUB
| project _A, _B, _scope, bin(P_A, 0.00001), bin(P_B, 0.00001), bin(P_AB, 0.00001), bin(P_AUB, 0.00001), bin(P_AIB, 0.00001)
, bin(P_BIA, 0.00001), bin(Lift_AB, 0.00001), bin(Jaccard_AB, 0.00001), A_B_Occurrences, B_A_Occurrences, countA, countB, cxtCVals, cxtDVals
| sort by _scope, _A, _B
};

The following fields were added:

  • A_B_Occurrences: Shows the top 10 most frequent B values associated with each A value within the specified scope, along with their occurrence counts.
  • B_A_Occurrences: Shows the top 10 most frequent A values associated with each B value within the specified scope, along with their occurrence counts.
  • countA: Provides the total number of occurrences of each A value within the specified scope.
  • countB: Provides the total number of occurrences of each B value within the specified scope.
  • cxtCVals: Provides the distinct values from context column C observed for the specific A/B pair within the specified scope.
  • cxtDVals: Provides the distinct values from context column D observed for the specific A/B pair within the specified scope.

An example of how the function can be used for threat hunting is provided in the next section.

The first example applies the extended pair probability function to identify known Windows binaries that are executing from directories that are unusual for that particular binary. Before calculating the probabilities, the FolderPath is normalized to reduce noise caused by values that are expected to vary between events. User directories are replaced with <user>, GUIDs with <uuid>, temporary filenames with <random>.tmp, numeric values with <number> and drive letters are stripped. This allows paths that are structurally the same to be treated as the same categorical value rather than as separate values. Common directories such as System32, SysWOW64, and Program Files are excluded to limit the results on executions from less expected locations.

The query for this hunt is:

kusto

let pair_probabilities_ext_fl = (tbl:(*), A_col:string, B_col:string, scope_col:string, cxt_col_C:string, cxt_col_D:string)
{
let T = materialize(tbl | extend _A = column_ifexists(A_col, ""), _B = column_ifexists(B_col, ""), _scope = column_ifexists(scope_col, ""), cxt_col_C = column_ifexists(cxt_col_C, ""), cxt_col_D = column_ifexists(cxt_col_D, "") | project _A, _B, _scope, cxt_col_C, cxt_col_D);
let countOnScope = T | summarize countAllOnScope = count() by _scope;
let countABOnScope = T | summarize c = count() by _A, _B, _scope | extend Occurrences = bag_pack(_B, c) | summarize A_B_Occurrences = make_bag(Occurrences) by _A, _scope | mv-apply A_B_Occurrences on (extend bag_as_array = bag_keys(A_B_Occurrences) | mv-expand key = bag_as_array | extend value = toint(A_B_Occurrences[tostring(key)]) | top 10 by value desc | summarize A_B_Occurrences = make_bag(pack(tostring(key), value)));
let countBAOnScope = T | summarize c = count() by _A, _B, _scope | extend Occurrences = bag_pack(_A, c) | summarize B_A_Occurrences = make_bag(Occurrences) by _B, _scope | mv-apply B_A_Occurrences on (extend bag_as_array = bag_keys(B_A_Occurrences) | mv-expand key = bag_as_array | extend value = toint(B_A_Occurrences[tostring(key)]) | top 10 by value desc | summarize B_A_Occurrences = make_bag(pack(tostring(key), value)));
let cxtCVals = T | summarize cxtCVals = make_set(cxt_col_C) by _A, _B, _scope;
let cxtDVals = T | summarize cxtDVals = make_set(cxt_col_D) by _A, _B, _scope;
let probAB = T | summarize countAB = count() by _A, _B, _scope | join kind = leftouter (countOnScope) on _scope | extend P_AB = todouble(countAB)/countAllOnScope;
let probA  = probAB | summarize countA = sum(countAB), countAllOnScope = max(countAllOnScope) by _A, _scope | extend P_A = todouble(countA)/countAllOnScope;
let probB  = probAB | summarize countB = sum(countAB), countAllOnScope = max(countAllOnScope) by _B, _scope | extend P_B = todouble(countB)/countAllOnScope;
probAB
| join kind = leftouter (probA) on _A, _scope
| join kind = leftouter (probB) on _B, _scope
| join kind = leftouter (countABOnScope) on _A, _scope
| join kind = leftouter (countBAOnScope) on _B, _scope
| join kind = leftouter (cxtCVals) on _A, _B, _scope
| join kind = leftouter (cxtDVals) on _A, _B, _scope
| extend P_AUB = P_A + P_B - P_AB
       , P_AIB = P_AB/P_B
       , P_BIA = P_AB/P_A
| extend Lift_AB = P_AB/(P_A * P_B)
       , Jaccard_AB = P_AB/P_AUB
| project _A, _B, _scope, bin(P_A, 0.00001), bin(P_B, 0.00001), bin(P_AB, 0.00001), bin(P_AUB, 0.00001), bin(P_AIB, 0.00001)
, bin(P_BIA, 0.00001), bin(Lift_AB, 0.00001), bin(Jaccard_AB, 0.00001), A_B_Occurrences, B_A_Occurrences, countA, countB, cxtCVals, cxtDVals
| sort by _scope, _A, _B
};
let SuspiciousBinaries = dynamic([
    "net1.exe","net.exe","whoami.exe","arp.exe","netstat.exe","nltest.exe","query.exe","tasklist.exe","hostname.exe","systeminfo.exe","qprocess.exe","ping.exe","qwinsta.exe","nslookup.exe","nbtstat.exe",
    "certutil.exe","cmd.exe","powershell.exe","powershell_ise.exe","pwsh.exe","mshta.exe","rundll32.exe","regsvr32.exe","wscript.exe","cscript.exe","bitsadmin.exe","msiexec.exe","schtasks.exe","wmic.exe","msbuild.exe","reg.exe","netsh.exe",
    "svchost.exe","explorer.exe","lsass.exe","msdtc.exe","wuauclt.exe","trustedinstaller.exe","audiodg.exe","dllhost.exe","w3wp.exe","conhost.exe",
    "microsoftedge.exe","winword.exe","excel.exe","powerpnt.exe"
]);
DeviceProcessEvents
| where ActionType == "ProcessCreated"
| where FileName in~ (SuspiciousBinaries)
| extend FileName = tolower(FileName), FolderPath = tolower(FolderPath)
| where FolderPath != ""
| extend FolderPath=replace_regex(FolderPath, @"^(\S:\\users\\)(.+?)\\(.*)$", @"\1<user>\\\3")
| extend FolderPath=replace_regex(FolderPath, @"^(\\device\\.+?)(\\windows.+)$", @"\device\2")
| extend FolderPath=replace_regex(FolderPath, @"^(.+?)(\\[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}\\)(.*)$", @"\1\\<uuid>\\\3")
| extend FolderPath = replace_regex(FolderPath, @"\\[A-Za-z0-9]+\.tmp", @"\\<random>.tmp")
| extend FolderPath = replace_regex(FolderPath, @"[0-9]{3,}", @"<random_num>")
| extend FolderPath = replace_regex(FolderPath, @"^(?:\S:|\\Device(?:\\HarddiskVolume\d+)?)", "")
| project DeviceName, FileName, FolderPath = tostring(parse_path(FolderPath).DirectoryPath)
| invoke pair_probabilities_ext_fl("FolderPath", "FileName", "", "DeviceName", "")
| project _A, _B, P_AIB, P_BIA, A_B_Occurrences, B_A_Occurrences, cxtCVals
| where not (_A startswith @"\windows\syswow64") and not (_A startswith @"\windows\system32") and not (_A startswith @"\program files")
| sort by P_AIB asc

The query returns a row for each unique A/B pair identified in the dataset. In this example, A represents the execution directory and B represents the binary name. Before looking at the results, it is useful to briefly explain the output columns.

  • _A: The first categorical value. In this case, it’s the directory from which the binary was executed.
  • _B: The second categorical value. In this case, it’s the name of the executed binary.
  • P_AIB: Represents P(A|B), the probability of observing _A given _B. In this case, it answers the question: how commonly does this binary execute from this directory? A low value indicates that the directory is unusual for that binary.
  • P_BIA: Represents P(B|A), the probability of observing _B given _A. In this case, it answers the question: given this directory, how often is this binary observed to execute from it? A low value indicates that the binary is uncommon for that directory.
  • A_B_Occurrences: Shows the most common (top 10) _B values observed for a particular _A. In this case, it shows which binaries have executed from the directory.
  • B_A_Occurrences: Shows the most common (top 10) _A values observed for a particular _B. In this case, it shows the directories from which the binary has executed.
  • cxtCVals: Lists the devices where the specific _A and _B were observed.

Results

In these results, _A is \users\<user>\desktop and _B is cmd.exe, meaning the pair represents cmd.exe executing from a user’s Desktop directory.

The P_AIB value of 0 indicates that this execution directory is extremely uncommon for cmd.exe in the dataset. In contrast, the P_BIA value of 0.99999 shows that, among the process executions observed from this particular directory, all were cmd.exe.

The A_B_Occurrences shows us that for \users\<user>\desktop only cmd.exe was observed to be executed from this directory. Looking at it from the opposite direction, the B_A_Occurrences shows us that cmd.exe was observed 133607 times from \windows\system32, 1886 times from \windows\syswow64, and only once from \users\<user>\desktop. This tells us that the Desktop execution is unusual for cmd.exe.

Finally, cxtCvals shows the device names where the pair \users\<user>\desktop and cmd.exe have been observed.

A variation of the query above identifies Microsoft binaries that are normally observed running from System32 or SysWOW64 and uses the pair probability function to identify unusual FolderPath/FileName relationships. This approach is more environment-driven, making the hunt more dynamic and adaptable to the environment.

kusto

let pair_probabilities_ext_fl = (tbl:(*), A_col:string, B_col:string, scope_col:string, cxt_col_C:string, cxt_col_D:string)
{
let T = materialize(tbl | extend _A = column_ifexists(A_col, ""), _B = column_ifexists(B_col, ""), _scope = column_ifexists(scope_col, ""), cxt_col_C = column_ifexists(cxt_col_C, ""), cxt_col_D = column_ifexists(cxt_col_D, "") | project _A, _B, _scope, cxt_col_C, cxt_col_D);
let countOnScope = T | summarize countAllOnScope = count() by _scope;
let countABOnScope = T | summarize c = count() by _A, _B, _scope | extend Occurrences = bag_pack(_B, c) | summarize A_B_Occurrences = make_bag(Occurrences) by _A, _scope | mv-apply A_B_Occurrences on (extend bag_as_array = bag_keys(A_B_Occurrences) | mv-expand key = bag_as_array | extend value = toint(A_B_Occurrences[tostring(key)]) | top 10 by value desc | summarize A_B_Occurrences = make_bag(pack(tostring(key), value)));
let countBAOnScope = T | summarize c = count() by _A, _B, _scope | extend Occurrences = bag_pack(_A, c) | summarize B_A_Occurrences = make_bag(Occurrences) by _B, _scope | mv-apply B_A_Occurrences on (extend bag_as_array = bag_keys(B_A_Occurrences) | mv-expand key = bag_as_array | extend value = toint(B_A_Occurrences[tostring(key)]) | top 10 by value desc | summarize B_A_Occurrences = make_bag(pack(tostring(key), value)));
let cxtCVals = T | summarize cxtCVals = make_set(cxt_col_C) by _A, _B, _scope;
let cxtDVals = T | summarize cxtDVals = make_set(cxt_col_D) by _A, _B, _scope;
let probAB = T | summarize countAB = count() by _A, _B, _scope | join kind = leftouter (countOnScope) on _scope | extend P_AB = todouble(countAB)/countAllOnScope;
let probA  = probAB | summarize countA = sum(countAB), countAllOnScope = max(countAllOnScope) by _A, _scope | extend P_A = todouble(countA)/countAllOnScope;
let probB  = probAB | summarize countB = sum(countAB), countAllOnScope = max(countAllOnScope) by _B, _scope | extend P_B = todouble(countB)/countAllOnScope;
probAB
| join kind = leftouter (probA) on _A, _scope
| join kind = leftouter (probB) on _B, _scope
| join kind = leftouter (countABOnScope) on _A, _scope
| join kind = leftouter (countBAOnScope) on _B, _scope
| join kind = leftouter (cxtCVals) on _A, _B, _scope
| join kind = leftouter (cxtDVals) on _A, _B, _scope
| extend P_AUB = P_A + P_B - P_AB
       , P_AIB = P_AB/P_B
       , P_BIA = P_AB/P_A
| extend Lift_AB = P_AB/(P_A * P_B)
       , Jaccard_AB = P_AB/P_AUB
| project _A, _B, _scope, bin(P_A, 0.00001), bin(P_B, 0.00001), bin(P_AB, 0.00001), bin(P_AUB, 0.00001), bin(P_AIB, 0.00001)
, bin(P_BIA, 0.00001), bin(Lift_AB, 0.00001), bin(Jaccard_AB, 0.00001), A_B_Occurrences, B_A_Occurrences, countA, countB, cxtCVals, cxtDVals
| sort by _scope, _A, _B
};
let SuspiciousBinaries = dynamic([
    "net1.exe","net.exe","whoami.exe","arp.exe","netstat.exe","nltest.exe","query.exe","tasklist.exe","hostname.exe","systeminfo.exe","qprocess.exe","ping.exe","qwinsta.exe","nslookup.exe","nbtstat.exe",
    "certutil.exe","cmd.exe","powershell.exe","powershell_ise.exe","pwsh.exe","mshta.exe","rundll32.exe","regsvr32.exe","wscript.exe","cscript.exe","bitsadmin.exe","msiexec.exe","schtasks.exe","wmic.exe","msbuild.exe","reg.exe","netsh.exe",
    "svchost.exe","explorer.exe","lsass.exe","msdtc.exe","wuauclt.exe","trustedinstaller.exe","audiodg.exe","dllhost.exe","w3wp.exe","conhost.exe",
    "microsoftedge.exe","winword.exe","excel.exe","powerpnt.exe"
]);
let FileNamesOfIntrest = DeviceProcessEvents
| where ActionType == "ProcessCreated"
| extend FileName = tolower(FileName), FolderPath = tolower(FolderPath)
| where FolderPath startswith @"c:\windows\system32" or FolderPath startswith @"c:\windows\syswow64"
| where ProcessVersionInfoCompanyName == "Microsoft Corporation"
| summarize by FileName;
DeviceProcessEvents
| where ActionType == "ProcessCreated"
| extend FileName = tolower(FileName), FolderPath = tolower(FolderPath)
| where FileName in (FileNamesOfIntrest)
| extend FolderPath=replace_regex(FolderPath, @"^(\S:\\users\\)(.+?)\\(.*)$", @"\1<user>\\\3")
| extend FolderPath=replace_regex(FolderPath, @"^(\\device\\.+?)(\\windows.+)$", @"\device\2")
| extend FolderPath=replace_regex(FolderPath, @"^(.+?)(\\[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}\\)(.*)$", @"\1\\<uuid>\\\3")
| extend FolderPath = replace_regex(FolderPath, @"\\[A-Za-z0-9]+\.tmp", @"\\<random>.tmp")
| extend FolderPath = replace_regex(FolderPath, @"[0-9]{3,}", @"<random_num>")
| extend FolderPath = replace_regex(FolderPath, @"^(?:\S:|\\Device(?:\\HarddiskVolume\d+)?)", "")
| project DeviceName, FileName, FolderPath = tostring(parse_path(FolderPath).DirectoryPath)
| invoke pair_probabilities_ext_fl("FolderPath", "FileName", "", "DeviceName", "ProcessCommandLine")
| project _A, _B, P_AIB, P_BIA, A_B_Occurrences, B_A_Occurrences, cxtCVals, cxtDVals
| where not (_A startswith @"c:\windows\syswow64") and not (_A startswith @"c:\windows\system32") and not (_A startswith @"c:\program files") and not (_A startswith @"\device")
| sort by P_AIB asc

While pair probabilities can be useful for identifying unusual relationships, there are several important considerations when using the function for threat hunting:

  • It detects statistical rarity, not maliciousness. A low probability indicates that a pair is unusual within the dataset, but it does not indicate that the activity is malicious.
  • The dataset defines what is considered unusual. The probabilities are entirely dependent on the dataset used to build the baseline. Differences between servers, workstations, users, applications, or other factors can significantly affect what is considered normal.
  • Rare values create unreliable probabilities. When either A, B, or the A/B pair has very few observations, probability estimates can change significantly based on only one or two additional events. Extremely high or low values should also be taken into consideration when analysing the results and interpreted alongside the number of observations for the pair. The size of the dataset also matters. A small dataset may not contain enough values to establish a reliable baseline.
  • Existing malicious activity can become part of the baseline. If the dataset used to calculate the probabilities already contains repetitive malicious behaviour, that behaviour may appear normal simply because it occurs frequently enough.

Overall, pair probabilities should be best viewed as a way of identifying unusual behaviour for further investigation, rather than as a standalone detection mechanism. The strongest results are likely to come from using the probability metrics as a baseline and in combination with occurrence counts, and additional security context.

Updated on 2026-08-16
 KQL, ThreatHunting
 | Home