Rule organization for multiple file types PE and ELF

[mr-tz]

Through version 2.0, capa only supported Windows PE files and shellcode. In #700 and mandiant/capa-rules#442 @Adir-Shemesh and @TcM1911 add support for ELF files. This means that capa can now match executables that target non-Windows operating systems!

Obviously, logic that matches on one operating system is (probably) not relevant to other operating systems. For example, the APIs provided by Microsoft Windows are different than those provided by libc on Linux. We expect this to introduce a few problems:

1. capa may match logic meant for one OS against files for another OS, potentially resulting in false positives
2. capa does extra work when to attempt to match irrelevant OS rules, resulting in worse performance
3. multi-OS logic in the same rule may be difficult to differentiate, leading to confusion in human readers

(3) is of particular note for us - we consider capa's rule set to be consumed as much by humans as by computers.

In this thread we want to document various strategies of organizing multi-OS rule logic and discuss the pros and cons.

Some points to consider:

• avoiding cross-OS matches, e.g. a Linux rule match on Windows
• intuitive capability documentation for humans
• duplication of meta data and file directory structure
• possibilities to leverage higher-level rules
• backwards compatibility and/or breaking changes requiring a new major release (this is fine!)

Note: PE vs ELF don't necessarily correspond directly to an underlying architecture or operating system. If we propose to extend capa rules with OS-specific features, then we'll need to also extend the feature extractors to provide the detected OS.

---

avoiding cross-OS matches

capa may match logic meant for one OS against files for another OS, potentially resulting in false positives. I'm not sure how prevalent this will be in practice. We should be aware of this possibility and consider its impact.

performance

With more and more rules we should keep an eye on performance. capa will perform better if we only apply targeted rules vs. everything and all features.

intuitive capability documentation for humans

We consider capa's rule set to be consumed as much by humans as by computers. We should design a strategy that makes it easy for a human to recognize which logic applies to which OS, when relevant.

For example, consider a rule that fetches system information:

rule:
  meta:
    name: get system information
  features:
    - or:
      - api: ZwQuerySystemInformationEx
      - api: sysconf
      - and:
        - api: system
        - string: "lshw"

The API ZwQuerySystemInformationEx is a Windows API, while sysconf and system are Linux/POSIX APIs. A human reader of these rules may not have the experience to recognize this. So, our strategy should somehow indicate to humans which logic is relevant to which OS.

duplication of meta data and file directory structure

capa rules have a lot of metadata associated with them, like ATT&CK mappings. We should design strategies that avoid duplicating metadata in many places (e.g. if we suggest duplicating rules by OS then the metadata is duplicated within the rules, too) because this information can fall out of date easily.

possibilities to leverage higher-level rules

capa already supports rules matching other rules and namespaces. This means that its feasible to split up logic across various rules and recognize capabilities in "higher level" rules.

For example, the following rule relies on another rule (read file) to define the various ways to read a file (or could use the namespace host-interaction/file-system/read):

rule:
  meta:
    name: collect ssh keys
  features:
    - and:
      - match: read file
      - or:
        - string: /id_rsa/

backwards compatibility and/or breaking changes requiring a new major release

capa releases follow semantic versioning: we avoid breaking changes in minor and patch releases. Furthermore, despite our recent publicity around v2.0, we don't consider major releases to necessarily be "major". They just signify breaking changes. So, if we need to change capa behavior in a breaking way, we'll need to coordinate a release of capa v3.0.

Its unclear if our semantic versioning applies to capa rules, too. That is, if we introduce a new capa rule feature that cannot be used in old capa versions, is this a breaking change that requires a new major release of capa? Probably, but we haven't documented this, yet.

[mr-tz]

Multiple OSes in one rule

This adds a few extra fields to the initially proposed rules.
Split up features block by OS, example based on get CPU information from mandiant/capa-rules#442

  meta:
    name: get CPU information
    namespace: host-interaction/hardware/cpu
    author:
      - moritz.raabe@fireeye.com
      - joakim@intezer.com
    scope: function
    att&ck:
      - Discovery::System Information Discovery [T1082]
    examples:
      - BFB9B5391A13D0AFD787E87AB90F14F5:0x13145B5A
  features:
    - windows:
      - or:
        - and:
          - match: query or enumerate registry value
          - string: /Hardware\\Description\\System\\CentralProcessor/i
    - linux:
      - or:
        - and:
          - match: read file
          - string: "/proc/cpuinfo"
        - and:
          - api: system
          - string: /\/proc\/cpuinfo/

Maybe adding an additional meta data fields makes sense as well, e.g.

  meta:
...
    os:
      - linux
      - freebsd
...

Pros

• documents features by OS
• can match according to OS
  • no FPs
  • better performance
• only one rule file per capability
  • can reuse meta data
• still allows for higher level matches

Cons

• breaking change to rule format
• may encourage large rules with lot of logic, vs splitting rules into smaller related parts

[mike-hunhoff]

I rank this idea the highest. I like that it provides a means to group features by OS which IMO makes it easiest for humans to interpret. capa can also skip entire rules and feature blocks simply based on the OS without needing to emit additional features with more suffixes as needed by the emit features with OS suffix idea.

One option is to require a new os meta field for all rules and only require "os-specific feature blocks" when more than one OS is represented in a rule e.g. if

- os:
   - windows
   - linux

then

features:
 - windows:
    ...
 - linux:
    ...

else if

- os: windows

then

features:
 - or:
   ...

Pros

• allows rules that only represent one OS to avoid unnecessary clutter in the features block
• easy for a human to look at the meta block and know right away what OS is supported without needing to dig through the features block or the rule name
• fairly simple to enforce with the linter

Cons

• may run into cases where a rule that represents multiple OSes has conflicting metadata e.g. ATT&CK as identified by @TcM1911

[recvfrom]

This is my favorite option as well. I like the idea of keeping everything together in one file. Is it correct to assume you could have something like the following, to mitigate concerns of duplication?:

  features:
    - and:
      - number: 6 = IPPROTO_TCP
      - number: 1 = SOCK_STREAM
      - number: 2 = AF_INET
      - windows:
        - or:
          - api: ws2_32.socket
          - api: ws2_32.WSASocket
      - linux:
        - or:
          - api: socket

[re-fox]

This is my preferred option.

Regarding rules that search for OS independent features (example: crypto constants) would the guidance be to leave as-is and only update rules where separate OS-specific features exist? (If I'm reading @mike-hunhoff's comment correctly, I think that is what he's recommending in the if/than/elseif/than condition)

[williballenthin]

would the guidance be to leave as-is and only update rules where separate platform-specific features exist?

I like this. Ideally, we don't add any additional syntax or overhead unless necessary.

To re-phrase what @mike-hunhoff said (and demonstrate that I do/don't get what he's saying), we'd propose:

1. Add a new optional meta.os field.

• if present, only match the rule when the target sample's OS matches one of the values here.
• not present, rule is assumed to match against all OSes. crypto constant rules would fit here.

2. Add new logic nodes linux, windows, etc.

• only match the logic underneath such a node when the target sample's OS matches the node name

Regarding (2), see also the proposal by @recvfrom here to use characteristic for this.

Questions:

• what are all the OSes that we recognize?
  • adding new logic nodes for a new OS is technically a breaking change (can't use such a rule on an older capa version). are we ok with that?
  • windows and linux are obvious. do we add macos to future proof the list?
• is there any interdependence between (1) and (2)?
  • maybe add a linter to suggest "if top level windows node then use a separate rule with meta.os"?
  • is meta.os required with a windows block is found within the rule logic? why?
• do we expect any blocks to be shared across some OSes but not others? like: "this works on linux and macos but not windows".
  • should the feature look something like:

rule:
  features:
    or:
      os: linux, macos
        offset: 0x10
      os: windows
        offset: 0x8

• if so, then we maybe avoid the "new OS logic node is a breaking change" problem because we can ignore unrecognized terms on the RHS of the os: . see characteristic proposal here, too.
• do we want something like posix to mean linux+macos+bsd+... ?

[williballenthin]

how much do we want the meta.os field if we have logic nodes like os: windows?

pro:

• (maybe) slightly better performance
• the OS might be more clear to a reader (but could put it in the name, too)

cons:

• reader has to know to look in two potential places for OS specifics (the header and the logic nodes)
• more code

[williballenthin]

emit features with OS suffix

Today, we emit number features with suffixes that reflects their architecture, like number/x32 for 32-bit PE files. This lets rule authors concisely match behaviors that are specific to an architecture/bitness. We could do the same with OSes: api/linux: system.

In this case, we'd emit both api/linux: system and api: system and rule authors could pick between either to restrict or relax where they match.

I think these suffixes would apply to most/all features.

Pros

• concise and explicit
• very fine grained

Cons

• how does this mix with existing arch suffix? number/linux/x32: 0x10 = FOO
• doesn't group related OS-specifics together
• performance: many more features emitted

[williballenthin]

how does this mix with existing arch suffix? number/linux/x32: 0x10 = FOO

here we'd emit all of the following:

• number: 0x10
• number/linux: 0x10
• number/x32: 0x10
• number/linux/x32: 0x10

maybe in practice this level of detail doesn't come up too often. probably just platform + arch specific structure layouts.

[mr-tz]

I think this allows for some elegant solutions and I currently would rank this idea and the multiple OSs in one rule as my favorites.

[mike-hunhoff]

I like how elegant this solution is but we could avoid emitting the additional features by opting for os-specific rules/feature blocks as discussed in multiple oses in one rule. I rank this idea my second highest.

[williballenthin]

one OS per file

We could limit the rule OS scope at the file-level via a new metadata field, like so:

rule:
  meta:
    name: invoke system
    os:
      - linux
      - freebsd
...

If os is not specified, then it would be assumed to apply to all OSes.

Rule names should probably be extended when they're OS specific, like create socket on Linux.

Then, we can encourage higher-level rules like so:

rule:
  meta:
    name: read file
  features:
    matches: host-interaction/file-system/read  # matches any of the "...on [OS]" rules here

or:

rule:
  meta:
    name: read file
  features:
    or:
      matches: read file on Linux
      matches: read file on Windows

Pro

• matching engine can skip rules that don't apply to a file
• its always clear how the OS logic is grouped together, just like matching scope

Con

• may result in fragmentation of rules, esp when its just a single line of logic per OS
• duplication of metadata that might be shared across OSes

[TcM1911]

Maybe this approach can be used togehter with the "do nothing" approach. This way, there is no need to "update" all rules to the new format. The field can be used to handle false positives on other operating systems, either restrict it one OS or break the rule up into "lower level" rules.

[williballenthin]

Good point. And I think this strategy would be backwards compatible: old versions of capa don't inspect the meta.os field so the worst that would happen would be FPs in old versions (but not exceptions).

[williballenthin]

may result in fragmentation of rules, esp when its just a single line of logic per OS

by this I mean I something like: given the proposed "get networking interfaces" rule:

  meta:
    name: get networking interfaces
  features:
    - or:
      - api: iphlpapi.GetIfTable        # windows
      - api: iphlpapi.GetAdaptersInfo   # windows
      - api: getifaddrs                 # linux

we'd have to split this up into two files: a windows rule with 2 lines and a linux rule with 1 line. Seems like a bit of overhead. But, maybe this is not that big of a deal because its probably not usually this extreme.

[TcM1911]

With "small" rules, could we only split it up when it causes FPs?

[williballenthin]

I think functionally that would work; however, I think its pretty important that capa rules be clear for a human too, so we'd want some way to teach human readers about the OS specifics.

[williballenthin]

one OS per root rule directory

Separate rules by OS at the top-level directory structure. Define OSes at the root of capa-rules, like:

windows
windows/host-interaction
windows/impact
linux
linux/host-interaction
linux/impact
...

Pro

• OS-specific rules are emphasized

Con

• capa engine needs more implicit knowledge of the dir layout
• how to handle multi-OS rules?
• related capabilities are not grouped together, e.g. user wants to see all the ways to read a file

[williballenthin]

i don't really like this strategy but its a possibility

[mr-tz]

I don't like this approach either. There's a lot of duplication.

[mike-hunhoff]

Not a fan either.

[re-fox]

This does mirror what Mitre is doing with Att&ck.

Separate matrices exist for each platform

• https://attack.mitre.org/matrices/enterprise/macos/
• https://attack.mitre.org/matrices/enterprise/linux/
• https://attack.mitre.org/matrices/enterprise/windows/
• ..etc...

Personally, I'm not a fan of having to manage separate rulesets. Thinking of how Yara rules are managed, I don't have duplicate rules per OS. The rules themselves describe the file/platform that they are intended to be run against and not the directory hierarchy.

[williballenthin]

do nothing

Don't introduce any further organization of rules or logic. Maybe its not yet a problem so we shouldn't over-engineer a solution before necessary.

Pro

• less work, simpler

Con

• potential for cross-OS FP
• confusing to users that aren't familiar with OS-specifics

[williballenthin]

i don't really like this strategy but its a possibility

[mr-tz]

Not a fan of this either. The cons here are major.

[mike-hunhoff]

Not a fan either.

[re-fox]

Agreed with the general consensus. The large concern for me would be tuning a rule on some Linux specific implementation and causing a regression/false negative on the Windows implementation. Trying to account for multiple conditions in a capa rule could result in a rule that is a bit of nightmare to debug. In most cases I imagine, having a Windows/Linux/...Other... section would be much easier to read and follow.

This approach may be fine if the rule is purely of features that are natively cross-OS (loops, mnemonics, constants), but I think that most rules have an API or OS specific feature where this would be a concern.

[mr-tz]

file "subclasses"

If a rule has an OS specific version add a file/reference that only updates what's listed in the new file

rule:
  meta:
    name: create reverse shell
    namespace: c2/shell
    author: moritz.raabe@fireeye.com
    scope: function
    ...
    os:
      - linux: create unix reverse shell

Alternatively, users add a file to the same path with a suffix, e.g. create-reverse-shell-linux.yml.

Then create unix reverse shell contains only what's unique to this OS. These fields override the parent "class"

rule:
  meta:
    name: create unix reverse shell
    author: joakim@intezer.com
    att&ck:
      - Execution::Command and Scripting Interpreter::Unix Shell [T1059.004]
    examples:
      - 7351f8a40c5450557b24622417fc478d:0x40231E
  features:
    - or:
      - and:
        - match: duplicate stdin and stdout
        - match: create process

Pro

• separation between OS
• easy for small updates, e.g. only update the feature block with a OS specific api

Con

• a lot of implicit data
• child rules may not be clear on their own

As I'm writing this I'm not really convinced by this approach, but documenting it regardless.

[mr-tz]

Maybe the other way around where a rule can indicate to take the meta data from another rule?

rule:
  meta:
    name: create unix reverse shell
    extends: create reverse shell
  features:
    <os specifics here>

[mike-hunhoff]

this is an interesting idea. I like the extends meta tag. The logic used to parse and display metadata might become complicated here e.g. how do you handle a linux nursery rule with minimal metadata that extends a windows rule with complete metadata including examples, att&ck, etc.?

[williballenthin]

I like how the logic is separated by OS; however, I strongly dislike how implicit the representation is. Also, it splits logic among multiple files, which is harder for humans to review.

[recvfrom]

Thinking about this more, one option that I haven't seen mentioned is adding new file scope characteristics corresponding to PE files or ELF files. This would be backwards-compatible AFAICT, and could look something like:

features:
  - and:
    - number: 6 = IPPROTO_TCP
    - number: 1 = SOCK_STREAM
    - number: 2 = AF_INET
    - or:
      - and:
        - characteristic: windows
        - or:
          - api: ws2_32.socket
          - api: ws2_32.WSASocket
      - and:
        - characteristic: linux
        - api: socket

[williballenthin]

I've have a draft PR here #723 that implements characteristics for OS and file format at every scope.

you can see it working against a linux rule here:

[williballenthin]

hybrid idea: introduce a new feature like os: windows, but rather than making it a statement with children nodes, make it a leaf node like characteristics.

this enables the expressive logic like the characteristics described above without "abusing" characteristics.

- or:
  - and:
    - os: windows
    - api: CreateFile
  - and:
    - or:
      - os: linux
      - os: macos
    - api: fopen

[mike-hunhoff]

I really like your hybrid idea @williballenthin and prefer it versus overloading characteristic.

[mike-hunhoff]

We could also add new features for the various file formats e.g. format: elf.

[mr-tz]

yes, this is a great idea!