Skip to content

Commit

Permalink
Merge branch 'release/v0.6.0'
Browse files Browse the repository at this point in the history
  • Loading branch information
targodan committed May 14, 2021
2 parents c2703e6 + 20dd9d6 commit 8b0e4ee
Show file tree
Hide file tree
Showing 37 changed files with 3,001 additions and 1,090 deletions.
15 changes: 12 additions & 3 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,16 @@ language: go

go:
- "1.x"
- "1.15"

env:
- OPENSSL_VERSION=OpenSSL_1_1_1-stable YARA_VERSION=v4.0.2
- OPENSSL_VERSION=OpenSSL_1_1_1-stable YARA_VERSION=v4.1.0

services:
- docker

before_install:
- echo $TRAVIS_GO_VERSION
- sudo apt-get update
- sudo apt-get install -y libssl-dev libprotobuf-dev flex bison p7zip-full # libyara-dev package is too old
- mkdir deps
Expand All @@ -26,7 +28,13 @@ install:
- go mod vendor

script:
- go test -race -coverprofile=coverage.txt -covermode=atomic -v ./...
- coverpkg="github.com/fkie-cad/yapscan/..."
- go test -race -coverprofile=coverage.txt -covermode=atomic -coverpkg="$coverpkg" -v $(go list ./... | grep -v /acceptanceTests)
- pushd acceptanceTests
- go test -race -coverprofile=coverage.txt -covermode=atomic -coverpkg="$coverpkg" -v -quickchecks=${QUICKCHECKS:-100} -short .
- popd
- cat acceptanceTests/coverage.txt | tail -n+2 >> coverage.txt
- rm acceptanceTests/coverage.txt
- mkdir -p build/ &>/dev/null
- pushd cmd/yapscan
- go build -trimpath -o ../../cicd/build/yapscan
Expand All @@ -48,7 +56,7 @@ before_deploy:

deploy:
provider: releases
skip_cleanup: 'true'
skip_cleanup: true
api_key:
secure: SEqzkWiVZTZlK3FvJGku7a10rMVqSPTakdAP2M7p6z+S2SE3yRI4R4aiH3t73Yt2nDPGrW1ie43EsJ0WcPIIVaSHUArNBwdyoGRBtdAq461ZSwsskuMLVz28SreyiTwEb4cmRrx5eggGyAoO6kzuGyBVdnBY8Bzxtj0CVV/qeHxvvD6ARYxBQtq6izomd4hoMnCRKNApIUDFbLo2gnF4YOuAMgi45ALe5jchSHPESsa/e8x2e13WEOCIwmiHxRR4EXS/oOxr8kO792HEa324V7uPE1HaIH3z7fBWWcUMXJZSAYynbzsC2WAHzWgIcFHyqVUcs6IKMU8rweJcNNDIVrC/zY7Jm4/aEdyy/Kq3iReuSUkvv9HY0C+CGf8RK/7x/1LolYXDnXGIeEXF/dkqE4p/bOCKrlW8TePoEf8zUlSiZRd0+XdodyxrbjFdx0E5kePxLGeqkopfR/ubKN5DJY93ueFsVlKYGpWLOKxtZTJoxEcwhSm1EF2Lsd77+xySjeJbh00Ozu51C7lu8Obn4wzWCVLDnlt3X5pSqYEHKphIfVYr21LPD9uQc++XR7IfPQG/Gnz/CTBEGvwTYm29jaD0m5e1pshL5nr5lhCOQIH4iZWSmoCwa8GP3Kbix9tYtPAdOkd6crz3vj0EUEWmd9ioEutTOL3wduGzNTcBvXM=
file:
Expand All @@ -59,3 +67,4 @@ deploy:
on:
repo: fkie-cad/yapscan
tags: true
condition: $TRAVIS_GO_VERSION =~ ^1\.x
12 changes: 8 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Other quality-of-life features include
- Listing and dumping memory segments of a specific process
- Compiling yara rules and compressing them into an encrypted zip.
- Provides an "executable DLL" for locked down environments such as VDIs (see section [Executable DLL](#executable-dll))
- Anonymization of reports with either a predefined, or a randomly generated salt

Yapscan comes with support for both Windows and Linux, however Windows is the primary and most thoroughly tested target OS.

Expand Down Expand Up @@ -55,11 +56,14 @@ OPTIONS:
--all-drives, --all-d scan all files in all local drives, implies --recurse (default: false)
--all-shares, --all-s scan all files in all mounted net-shares, implies --recurse (default: false)
--file-extensions value, -e value list of file extensions to scan, use special extension "-" as no extension, use --file-extensions "" to allow any (default: "-", "so", "exe", "dll", "sys")
--threads value, -t value number of threads (goroutines) used for scanning files (default: 12)
--threads value, -t value number of threads (goroutines) used for scanning files (default: 6)
--full-report create a full report (default: false)
--report-dir value the directory to which the report archive will be written (default: current working directory)
--store-dumps store dumps of memory regions that match rules, implies --full-report, the report will be encrypted with --password (default: false)
--keep keep the temporary report directory, by default it will be deleted; ignored without --full-report (default: false)
--password value the password of the encrypted report, ignored unless --store-dumps is set (default: "infected")
--password value setting this will encrypt the report with the given password; ignored without --full-report
--pgpkey value setting this will encrypt the report with the public key in the given file; ignored without --full-report
--anonymize anonymize any output, hashing any usernames, hostnames and IPs with a salt (default: false)
--salt value the salt (hex string) to use for anonymization, ignored unless --anonmyize is provided (default: random salt)
--filter-permissions value, --f-perm value only consider segments with the given permissions or more, examples: "rw" includes segments with rw, rc and rwx
--filter-permissions-exact value, --f-perm-e value comma separated list of permissions to be considered, supported permissions: r, rw, rc, rwx, rcx
--filter-type value, --f-type value comma separated list of considered types, supported types: image, mapped, private
Expand Down Expand Up @@ -191,7 +195,7 @@ Then you open PowerShell and execute `.\cicd\enableMingw.ps1 -MsysPath <msys_pat
Now it's as easy as `go test -tags yara_static ./...`.
The `-tags yara_static` is necessary if you use the build scripts, as they do not install any windows DLLs but only the static libraries.

**NOTE:** You might get inexplicable failures with `-race` on Windows with golagn >=1.14.
**NOTE:** You might get inexplicable failures with `-race` on Windows.
According to the [golang release notes for 1.14](https://golang.org/doc/go1.14#compiler), the new pointer arithmetic checks are somewhat overzealous on Windows.
In Golang v1.15 it seems this may not have been fixed, but the checks are enabled automatically.
You can deactivate them like this `go test -tags yara_static -race -gcflags=all=-d=checkptr=0 ./...`.
Expand Down
241 changes: 241 additions & 0 deletions acceptanceTests/helper_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,241 @@
package acceptanceTests

import (
"bytes"
"context"
"fmt"
"io"
"io/ioutil"
"math/rand"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"

"github.com/fkie-cad/yapscan/testutil"
"golang.org/x/crypto/openpgp"

"github.com/fkie-cad/yapscan/procio"
"github.com/fkie-cad/yapscan/testutil/memory"
. "github.com/smartystreets/goconvey/convey"
)

const testCompilerTimeout = 1 * time.Minute
const testerTimeout = 15 * time.Second
const yapscanTimeout = 10 * time.Second

var memoryTesterCompiler *testutil.Compiler

func initializeMemoryTester() io.Closer {
var err error
memoryTesterCompiler, err = memory.NewTesterCompiler()
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(-1)
}

ctx, cancel := context.WithTimeout(context.Background(), testCompilerTimeout)
memoryTesterCompiler.Compile(ctx)
cancel()

return memoryTesterCompiler
}

func withMemoryTester(t *testing.T, c C, data []byte) (pid int, addressOfData uintptr) {
ctx, cancel := context.WithTimeout(context.Background(), testerTimeout)

tester, err := memory.NewTester(
ctx,
memoryTesterCompiler,
data,
uintptr(procio.PermissionsToNative(procio.Permissions{Read: true})))
if err != nil {
t.Fatal("could not create memory tester process", err)
}

addressOfData, err = tester.WriteDataAndGetAddress()
if err != nil || addressOfData == 0 {
t.Fatal("could not write data to memory tester process", err)
}

if c != nil {
c.Reset(func() {
tester.Close()
cancel()
})
} else {
t.Cleanup(func() {
tester.Close()
cancel()
})
}

return tester.PID(), addressOfData
}

func withYaraRulesFile(t *testing.T, rules []byte) string {
tempDir := t.TempDir()
yaraRulesFile := filepath.Join(tempDir, "rules.yar")
err := ioutil.WriteFile(yaraRulesFile, rules, 0600)
if err != nil {
t.Fatal("could not write temporary rules file", err)
}

return yaraRulesFile
}

func withYaraRulesFileAndMatchingMemoryTester(t *testing.T, c C, data []byte) (yaraRulesPath string, pid int, addressOfData uintptr) {
return withYaraRulesFileAndMemoryTester(t, c, data, data)
}

func withYaraRulesFileAndNotMatchingMemoryTester(t *testing.T, c C, data []byte) (yaraRulesPath string, pid int, addressOfData uintptr) {
memoryData := make([]byte, len(data))
copy(memoryData, data)

replaceCount := rand.Intn(len(data)) + 1
for i := 0; i < replaceCount; i++ {
memoryData[rand.Intn(len(data))] ^= byte(rand.Intn(254) + 1)
}

return withYaraRulesFileAndMemoryTester(t, c, data, memoryData)
}

func withYaraRulesFileAndMemoryTester(t *testing.T, c C, ruleData []byte, memoryData []byte) (yaraRulesPath string, pid int, addressOfData uintptr) {
ruleDataHexString := &strings.Builder{}
for _, b := range ruleData {
ruleDataHexString.WriteString(fmt.Sprintf("%02X ", b))
}

rule := fmt.Sprintf(`
rule rule1 {
meta:
description = "just a dummy rule"
author = "some dude on the internet"
date = "2020-01-01"
strings:
$str1 = { %s}
condition:
$str1
}
`, ruleDataHexString.String())

yaraRulesPath = withYaraRulesFile(t, []byte(rule))
pid, addressOfData = withMemoryTester(t, c, memoryData)
return
}

func withCapturedOutput(t *testing.T) (stdout, stderr *bytes.Buffer, cleanup func()) {
stdoutR, stdoutW, err := os.Pipe()
if err != nil {
t.Fatal("could not create pipe for output capture", err)
}
origOut := os.Stdout
os.Stdout = stdoutW

stderrR, stderrW, err := os.Pipe()
if err != nil {
t.Fatal("could not create pipe for output capture", err)
}
origErr := os.Stderr
os.Stderr = stderrW

stdoutB, stderrB := &bytes.Buffer{}, &bytes.Buffer{}

startWG := &sync.WaitGroup{}
endWG := &sync.WaitGroup{}

copyFunc := func(dst io.Writer, src io.Reader) {
startWG.Done()
defer endWG.Done()
io.Copy(dst, src)
}
startWG.Add(2)
endWG.Add(2)
go copyFunc(stdoutB, stdoutR)
go copyFunc(stderrB, stderrR)

startWG.Wait()

return stdoutB, stderrB, func() {
os.Stdout = origOut
os.Stderr = origErr

stdoutW.Close()
stdoutR.Close()
stderrW.Close()
stderrR.Close()
endWG.Wait()
}
}

func withPGPKey(t *testing.T) (keyringPath string, keyring openpgp.EntityList) {
publicKey := `-----BEGIN PGP PUBLIC KEY BLOCK-----
mI0EYJPogAEEAL4TyjkDBr36Bi3uIogDofX4tDCzLqq7D+eVo422Vb+vKACR7xb9
8b0viKzdxY+riRSYYABpCEdTSSnaxu33jdT8lLJng3nd1LzMOp/FfYP4NFs3lJJb
OossHfXRbn4It0iuS/Kh54stuQhXLKam+5wBiyNyQ7/UZY/URT6KoCk3ABEBAAG0
HEJvYiBTZWNyZXQgPGJvYkBzZWNyZXRzLmNvbT6IzAQTAQoANhYhBOK7KfeJOnY0
75mp8HCIYPeYHOjgBQJgk+iAAhsDBAsJCAcEFQoJCAUWAgMBAAIeAQIXgAAKCRBw
iGD3mBzo4P4PBAC5rliAt9pCeLxbISNXZJaCvGLYfsuz4xQah1U0D81lWUZE/oru
zlLd3kiTVL8Aufb2x22sg8kIuML2wiB7Ssa8CWk3K9zgDY+VTQqeJondhjCJlJNY
A3ePYO4Pzzn3PQSv66q5933DZDj4K7StFBqQBsCUc7lbVdH/dyK7rPJ4RLiNBGCT
6IABBAC4UNzRB3B8JptiHJvo6JypAxRuWzSQPceTIG0IAyUsIk4pVh2NvRARFIuD
c0mmgT5/SFaoRYF5HkVFs8unCBTMZ+BYIoXLqOnTeORJDXYcIl868G464D56nRfL
9/aIxJeP/3MWWonxfEXo5zKhqzbilPmJfojvxr3PKe4VCL6gBQARAQABiLYEGAEK
ACAWIQTiuyn3iTp2NO+ZqfBwiGD3mBzo4AUCYJPogAIbDAAKCRBwiGD3mBzo4Efc
BACqdYohgkt6V8RXiT+vuxYbfavY/TUn97j9Pd3Wea/m9RRvMWYuoFJm8v9OGF1r
aMd+ssVwzMgByyK/ppjALtyRQ5jxxerFZ9V/wXtd9i9gEXYja3yFoLfNCg0FAIKT
DseR8h7KG2EqdA3vRDoS6v7PkaTaoFvTizACRy7GJVki2w==
=06+X
-----END PGP PUBLIC KEY BLOCK-----
`
privateKey := `-----BEGIN PGP PRIVATE KEY BLOCK-----
lQHYBGCT6IABBAC+E8o5Awa9+gYt7iKIA6H1+LQwsy6quw/nlaONtlW/rygAke8W
/fG9L4is3cWPq4kUmGAAaQhHU0kp2sbt943U/JSyZ4N53dS8zDqfxX2D+DRbN5SS
WzqLLB310W5+CLdIrkvyoeeLLbkIVyympvucAYsjckO/1GWP1EU+iqApNwARAQAB
AAP+OPcDXwy6I4tf+LnqnWrBSk9L6WB59u7y+EvPZXQkxLrAuVMDHZfjr/gj9PLN
953ICmUUOGtB8OZUAfgwMDdFyO/i+tZwbHXOAl1ZqzWn8tqnsRa/7SS+9Bgg+6ZU
9FUK8dQLUSSc3Ab+9k34TeAmv20GTZKiWCt6pwHjj0d9CJECANoHBfe2UC0qidiq
Ttofa/MAw9tBIta50ACLwyfoRzCmYmctIMBVIw1XXUMzYtJ9fBJqKTkXeRb6MrMb
YsjidbkCAN8ulNb85P+G902nbliJtjtPuMzPDROMnsSkEgIOW7rrIPhKAw8m0Ovh
gF3a8zIviRT++iIGla0GZXotlX13Dm8CANanph0ExqN8d/C9BBVmCF3q5ACikiAM
K8rsZsteCnvO8K2s9RR318BvMZ1UZrZInOC1OQI/pIWGVR+ZIjkQ4/mgHbQcQm9i
IFNlY3JldCA8Ym9iQHNlY3JldHMuY29tPojMBBMBCgA2FiEE4rsp94k6djTvmanw
cIhg95gc6OAFAmCT6IACGwMECwkIBwQVCgkIBRYCAwEAAh4BAheAAAoJEHCIYPeY
HOjg/g8EALmuWIC32kJ4vFshI1dkloK8Yth+y7PjFBqHVTQPzWVZRkT+iu7OUt3e
SJNUvwC59vbHbayDyQi4wvbCIHtKxrwJaTcr3OANj5VNCp4mid2GMImUk1gDd49g
7g/POfc9BK/rqrn3fcNkOPgrtK0UGpAGwJRzuVtV0f93Irus8nhEnQHYBGCT6IAB
BAC4UNzRB3B8JptiHJvo6JypAxRuWzSQPceTIG0IAyUsIk4pVh2NvRARFIuDc0mm
gT5/SFaoRYF5HkVFs8unCBTMZ+BYIoXLqOnTeORJDXYcIl868G464D56nRfL9/aI
xJeP/3MWWonxfEXo5zKhqzbilPmJfojvxr3PKe4VCL6gBQARAQABAAP+JurOASXK
urAVKXVkdxxcbbRMhVOlKNqej+JuGx285NF2gvRfp5yWrqCRp6b5U1qhRVNTFtMc
OCRr2ICS3NkQOM9yioh3W5ZozIAAQfSnMEKwc7BhHvUWdFfrRjU4uaQFTCyn8VxO
9tQBxwXbbZy5zFjHM4Po4ygNsXwzc6pXkgECAMPnjAEteqdEAh4nNR0e//oMO1b8
9HHU2SnYvN02yPB47fLiVLjEcaMfN0jAkarimQDLzlGJYiB520VnLYET/gECAPDb
Qfd0F+agMzbq50paw+7GkifjPSZOL+AsPyUOCP30RfvugpHSr/2fRuvwVxqIHTkv
VwE873vbULk+E5iuqgUCAJTjwNqs774bUch3LQnjI7qyiL1GcnKhO/6hdu6Hra/x
IB4iWRlU1MPf1MBxbXNAF5qfpx7mxwrK2md5s5stliiio4i2BBgBCgAgFiEE4rsp
94k6djTvmanwcIhg95gc6OAFAmCT6IACGwwACgkQcIhg95gc6OBH3AQAqnWKIYJL
elfEV4k/r7sWG32r2P01J/e4/T3d1nmv5vUUbzFmLqBSZvL/Thhda2jHfrLFcMzI
Acsiv6aYwC7ckUOY8cXqxWfVf8F7XfYvYBF2I2t8haC3zQoNBQCCkw7HkfIeyhth
KnQN70Q6Eur+z5Gk2qBb04swAkcuxiVZIts=
=2Tor
-----END PGP PRIVATE KEY BLOCK-----
`
keyringPath = filepath.Join(t.TempDir(), "pubkey.pgp")
err := ioutil.WriteFile(keyringPath, []byte(publicKey), 0600)
if err != nil {
t.Fatal("could not create test keypair", err)
}

keyring, err = openpgp.ReadArmoredKeyRing(bytes.NewReader([]byte(privateKey)))
if err != nil {
t.Fatal("could not create test pgp keys", err)
}
return
}
Loading

0 comments on commit 8b0e4ee

Please sign in to comment.