Skip to content

Commit

Permalink
Merge branch 'release/v0.5.0'
Browse files Browse the repository at this point in the history
  • Loading branch information
targodan committed Mar 12, 2021
2 parents de2b7fa + e6db10b commit 68686cf
Show file tree
Hide file tree
Showing 105 changed files with 2,697 additions and 661 deletions.
5 changes: 1 addition & 4 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@

# Compiled command
/cmd/yapscan/yapscan
/build
/cicd/build/

# Dependency directories (remove the comment below to include it)
vendor/
Expand All @@ -25,6 +25,3 @@ vendor/

# Logfiles
*.log

# Generated Mocks (at least, while I'm working on the tests)
mock_*_test.go
16 changes: 9 additions & 7 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,21 +23,23 @@ before_install:
- pkg-config --cflags --libs yara

install:
- ./prepare.sh
- go mod vendor

script:
- go test -race -coverprofile=coverage.txt -covermode=atomic -v ./...
- mkdir -p build/ &>/dev/null
- pushd cmd/yapscan
- go build -trimpath -o ../../build/yapscan
- popd
- ./buildForWindows.sh
- pushd cicd/
- ./crossBuildForWindows.sh
- popd

after_success:
- bash <(curl -s https://codecov.io/bash)

before_deploy:
- pushd build
- pushd cicd/build
- 7z a yapscan_windows_amd64.zip yapscan.exe yapscan.dll
- 7z a yapscan_linux_amd64.zip yapscan
- tar -cvzf yapscan_windows_amd64.tar.gz yapscan.exe yapscan.dll
Expand All @@ -50,10 +52,10 @@ deploy:
api_key:
secure: SEqzkWiVZTZlK3FvJGku7a10rMVqSPTakdAP2M7p6z+S2SE3yRI4R4aiH3t73Yt2nDPGrW1ie43EsJ0WcPIIVaSHUArNBwdyoGRBtdAq461ZSwsskuMLVz28SreyiTwEb4cmRrx5eggGyAoO6kzuGyBVdnBY8Bzxtj0CVV/qeHxvvD6ARYxBQtq6izomd4hoMnCRKNApIUDFbLo2gnF4YOuAMgi45ALe5jchSHPESsa/e8x2e13WEOCIwmiHxRR4EXS/oOxr8kO792HEa324V7uPE1HaIH3z7fBWWcUMXJZSAYynbzsC2WAHzWgIcFHyqVUcs6IKMU8rweJcNNDIVrC/zY7Jm4/aEdyy/Kq3iReuSUkvv9HY0C+CGf8RK/7x/1LolYXDnXGIeEXF/dkqE4p/bOCKrlW8TePoEf8zUlSiZRd0+XdodyxrbjFdx0E5kePxLGeqkopfR/ubKN5DJY93ueFsVlKYGpWLOKxtZTJoxEcwhSm1EF2Lsd77+xySjeJbh00Ozu51C7lu8Obn4wzWCVLDnlt3X5pSqYEHKphIfVYr21LPD9uQc++XR7IfPQG/Gnz/CTBEGvwTYm29jaD0m5e1pshL5nr5lhCOQIH4iZWSmoCwa8GP3Kbix9tYtPAdOkd6crz3vj0EUEWmd9ioEutTOL3wduGzNTcBvXM=
file:
- build/yapscan_windows_amd64.zip
- build/yapscan_windows_amd64.tar.gz
- build/yapscan_linux_amd64.zip
- build/yapscan_linux_amd64.tar.gz
- cicd/build/yapscan_windows_amd64.zip
- cicd/build/yapscan_windows_amd64.tar.gz
- cicd/build/yapscan_linux_amd64.zip
- cicd/build/yapscan_linux_amd64.tar.gz
on:
repo: fkie-cad/yapscan
tags: true
59 changes: 45 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# yapscan [![Build Status](https://travis-ci.org/fkie-cad/yapscan.svg?branch=master)](https://travis-ci.org/fkie-cad/yapscan) [![codecov](https://codecov.io/gh/fkie-cad/yapscan/branch/master/graph/badge.svg?token=Y2ANV37QH6)](https://codecov.io/gh/fkie-cad/yapscan)
# yapscan [![Build Status](https://travis-ci.org/fkie-cad/yapscan.svg?branch=master)](https://travis-ci.org/fkie-cad/yapscan) [![codecov](https://codecov.io/gh/fkie-cad/yapscan/branch/master/graph/badge.svg?token=Y2ANV37QH6)](https://codecov.io/gh/fkie-cad/yapscan) [![Go Report Card](https://goreportcard.com/badge/github.com/fkie-cad/yapscan)](https://goreportcard.com/report/github.com/fkie-cad/yapscan)

Yapscan is a **YA**ra based **P**rocess **SCAN**ner, aimed at giving more control about what to scan and giving detailed reports on matches.

Expand Down Expand Up @@ -155,16 +155,13 @@ Yapscan copies one memory segment at a time into a buffer in its own memory and
## Building Yapscan
This project can only be built on Linux at this time.
To build natively on Linux, for Linux you need install Go and the yara library.
To build **natively on Linux**, for Linux you need install Go and the yara library.
Once you have installed the dependencies it's as easy as:
```bash
# Install Golang and libyara
git clone https://github.com/fkie-cad/yapscan
cd yapscan
./prepare.sh
cd cmd/yapscan
cd yapscan/cmd/yapscan
go build
```

Expand All @@ -173,13 +170,47 @@ If you want to build on Linux for Windows, all you need installed is docker.
```bash
# Install docker
git clone https://github.com/fkie-cad/yapscan
cd yapscan
./buildForWindows.sh
cd yapscan/cicd/
./crossBuildForWindows.sh
```

### Why can I not build this project on Windows?

I have been unable, so far, to build libyara on Windows and marry the result to the go toolchain to create a static build.

If you have experience with cgo on Windows, it would be great if you could help out.
This is relatively important for automated testing.
The resulting binaries will be placed in `cicd/build/`.

Building **natively on Windows**, using MSYS2 follow these instructions

1. Install Go
2. Install MSYS2 and follow the first steps on [the MSYS2 Website] of updating via pacman.
3. Install build dependencies `pacman --needed -S base-devel git autoconf automake libtool mingw-w64-{x86_64,i686}-{gcc,make,pkgconf}`
4. Open PowerShell in the `cicd/` directory and execute `.\buildOnWindows.ps1 -MsysPath <msys_path> -BuildDeps`
where `<msys_path>` is the install directory for MSYS2, default is `C:\msys64`.
**NOTE:** You'll have to press `Enter` on the MSYS window, once the dependencies are finished.
5. Enjoy the built files in `cicd/build/`

If you want to run tests on Windows, you have to run `.\cicd\buildOnWindows.ps1 -BuildDeps` only once.
Then you open PowerShell and execute `.\cicd\enableMingw.ps1 -MsysPath <msys_path>` to set the appropriate environment variables.
Now it's as easy as `go test -tags yara_static ./...`.
The `-tags yara_static` is necessary if you use the build scripts, as they do not install any windows DLLs but only the static libraries.

**NOTE:** You might get inexplicable failures with `-race` on Windows with golagn >=1.14.
According to the [golang release notes for 1.14](https://golang.org/doc/go1.14#compiler), the new pointer arithmetic checks are somewhat overzealous on Windows.
In Golang v1.15 it seems this may not have been fixed, but the checks are enabled automatically.
You can deactivate them like this `go test -tags yara_static -race -gcflags=all=-d=checkptr=0 ./...`.

You don't have to rely on the powershell/bash scripts, but they are intended to make things as easy as possible at the cost of control over the compilation.
If you want more control, take a look at the scripts use and modify them or execute the commands individually.
The scripts perform the following tasks.

1. Start "MSYS2 MinGW 64-bit"
1. Download OpenSSL from github
2. Static-Build OpenSSL and install the development files
3. Download libyara from github
4. Static-Build libyara and install it
2. Set some environment variables in powershell, to allow the use of the mingw toolchain
3. Call the `go build` command with the appropriate build tag for static builds

Thanks to [@hillu] (author of [go-yara]), for pointing me in the right direction for building natively on windows.
See #7 and the links therein if you want some more details.

[the MSYS2 Website]: https://www.msys2.org/
[@hillu]: https://github.com/hillu/
[go-yara]: https://github.com/hillu/go-yara/
2 changes: 1 addition & 1 deletion WinApi.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

In this document, an overview of the win32 APIs is given regarding process and memory scanning. If the scanner is a 64-bit application, these calls result in an executable which can scan memory of both 64- and 32-bit processes.

The shown code samples are not valid C code. They are supposed to be more akin to pseudo-code to give you a starting point for researching how to access another processes memory. You can find the shown API calls in the context of Go in these files: [procIO/process_windows.go](procIO/process_windows.go), [procIO/memory_windows.go](procIO/memory_windows.go) and [procIO/reader_windows.go](procIO/reader_windows.go).
The shown code samples are not valid C code. They are supposed to be more akin to pseudo-code to give you a starting point for researching how to access another processes memory. You can find the shown API calls in the context of Go in these files: [procio/process_windows.go](procio/process_windows.go), [procio/memory_windows.go](procio/memory_windows.go) and [procio/reader_windows.go](procio/reader_windows.go).

## Listing Processes

Expand Down
48 changes: 39 additions & 9 deletions app/app.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ import (
"github.com/fkie-cad/yapscan/output"

"github.com/sirupsen/logrus"
"github.com/targodan/go-errors"
"github.com/urfave/cli/v2"
)

Expand Down Expand Up @@ -41,7 +40,7 @@ func initAppAction(c *cli.Context) error {
default:
logfile, err := os.OpenFile(c.String("log-path"), os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
return errors.Errorf("could not open logfile for writing, reason: %w", err)
return fmt.Errorf("could not open logfile for writing, reason: %w", err)
}
logrus.SetOutput(logfile)
logrus.RegisterExitHandler(func() {
Expand All @@ -60,32 +59,32 @@ func filterFromArgs(c *cli.Context) (yapscan.MemorySegmentFilter, error) {

filters[i], err = BuildFilterPermissions(c.String("filter-permissions"))
if err != nil {
return nil, errors.Errorf("invalid flag \"--filter-permissions\", reason: %w", err)
return nil, fmt.Errorf("invalid flag \"--filter-permissions\", reason: %w", err)
}
i += 1
filters[i], err = BuildFilterPermissionsExact(c.StringSlice("filter-permissions-exact"))
if err != nil {
return nil, errors.Errorf("invalid flag \"--filter-permissions-exact\", reason: %w", err)
return nil, fmt.Errorf("invalid flag \"--filter-permissions-exact\", reason: %w", err)
}
i += 1
filters[i], err = BuildFilterType(c.StringSlice("filter-type"))
if err != nil {
return nil, errors.Errorf("invalid flag \"--filter-type\", reason: %w", err)
return nil, fmt.Errorf("invalid flag \"--filter-type\", reason: %w", err)
}
i += 1
filters[i], err = BuildFilterState(c.StringSlice("filter-state"))
if err != nil {
return nil, errors.Errorf("invalid flag \"--filter-state\", reason: %w", err)
return nil, fmt.Errorf("invalid flag \"--filter-state\", reason: %w", err)
}
i += 1
filters[i], err = BuildFilterSizeMax(c.String("filter-size-max"))
if err != nil {
return nil, errors.Errorf("invalid flag \"--filter-size-max\", reason: %w", err)
return nil, fmt.Errorf("invalid flag \"--filter-size-max\", reason: %w", err)
}
i += 1
filters[i], err = BuildFilterSizeMin(c.String("filter-size-min"))
if err != nil {
return nil, errors.Errorf("invalid flag \"--filter-size-min\", reason: %w", err)
return nil, fmt.Errorf("invalid flag \"--filter-size-min\", reason: %w", err)
}
i += 1

Expand Down Expand Up @@ -204,7 +203,7 @@ func RunApp(args []string) {
Name: "yapscan",
HelpName: "yapscan",
Description: "A yara based scanner for files and process memory with some extras.",
Version: "0.4.0",
Version: "0.5.0",
Writer: os.Stdout,
ErrWriter: os.Stderr,
Authors: []*cli.Author{
Expand Down Expand Up @@ -410,9 +409,40 @@ func RunApp(args []string) {
},
},
},
&cli.Command{
Name: "crash-processe",
Aliases: []string{"crash"},
Usage: "crash a processe",
Action: crashProcess,
Flags: []cli.Flag{
&cli.StringFlag{
Name: "method",
Aliases: []string{"m"},
Usage: "output errors if any are encountered",
Value: "CreateThreadOnNull",
},
},
},
},
}

if runtime.GOOS == "windows" {
app.Commands = append(app.Commands,
&cli.Command{
Name: "as-service",
Usage: "executes yapscan as a windows service",
Action: func(c *cli.Context) error {
// This is a dummy
return cli.Exit("\"as-service\" must be the first argument", 1)
},
})

if len(args) >= 2 && args[1] == "as-service" {
asService(args)
return
}
}

err := app.Run(args)
if err != nil {
fmt.Println(err)
Expand Down
55 changes: 55 additions & 0 deletions app/asService.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
package app

import (
"fmt"
"os"
"os/exec"
"path/filepath"
)

func run(cmdName string, cmdArgs ...string) error {
cmd := exec.Command(cmdName, cmdArgs...)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
return cmd.Run()
}

func asService(args []string) {
serviceName := "yapscan"

binpath, err := filepath.Abs(args[0])
if err != nil {
fmt.Printf("ERROR: Could not determine absolute path of yapscan.exe, %v\n", err)
os.Exit(1)
}

args = args[2:]
scStartArguments := []string{"start", serviceName}
scStartArguments = append(scStartArguments, args...)

fmt.Println("WARNING: This feature is experimental!")
fmt.Println("WARNING: You will not see any output of the executed service. Using --log-path is strongly advised.")

fmt.Println("Removing service in case it exists already...")
run("sc.exe", "delete", "yapscan")
fmt.Println("Done removing.")

fmt.Println("Installing service...")
err = run("sc.exe", "create", serviceName, "type=", "own", "start=", "demand", "binpath=", binpath)
if err != nil {
fmt.Println("FAILURE")
fmt.Print(err)
os.Exit(10)
}
fmt.Println("Done installing service.")

fmt.Println("Starting service with arguments...")
err = run("sc.exe", scStartArguments...)
if err != nil {
fmt.Println("FAILURE")
fmt.Print(err)
os.Exit(11)
}
fmt.Println("Done starting service, yapscan should now be running as a service with the following arguments")
fmt.Println(args)
}
38 changes: 38 additions & 0 deletions app/crash.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
package app

import (
"fmt"
"strconv"

"github.com/fkie-cad/yapscan/procio"
"github.com/targodan/go-errors"
"github.com/urfave/cli/v2"
)

func crashProcess(c *cli.Context) error {
err := initAppAction(c)
if err != nil {
return err
}

if c.NArg() != 1 {
return errors.Newf("expected exactly one arguments, got %d", c.NArg())
}
pid_, err := strconv.ParseUint(c.Args().Get(0), 10, 64)
if err != nil {
return errors.Newf("\"%s\" is not a pid", c.Args().Get(0))
}
pid := int(pid_)

crashMethod, err := procio.ParseCrashMethod(c.String("method"))
if err != nil {
return fmt.Errorf("invalid parameter, %w", err)
}

proc, err := procio.OpenProcess(pid)
if err != nil {
return errors.Newf("could not open process %d, reason: %w", pid, err)
}

return proc.Crash(crashMethod)
}
9 changes: 4 additions & 5 deletions app/dump.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import (
"path"
"strconv"

"github.com/fkie-cad/yapscan/procIO"

"github.com/fkie-cad/yapscan/procio"
"github.com/targodan/go-errors"
"github.com/urfave/cli/v2"
)
Expand Down Expand Up @@ -51,7 +50,7 @@ func dumpMemory(c *cli.Context) error {
}
}

proc, err := procIO.OpenProcess(pid)
proc, err := procio.OpenProcess(pid)
if err != nil {
return errors.Newf("could not open process %d, reason: %w", pid, err)
}
Expand All @@ -61,7 +60,7 @@ func dumpMemory(c *cli.Context) error {
return errors.Newf("could not retrieve memory segments of process %d, reason: %w", pid, err)
}
// Unpack segments
segments := make([]*procIO.MemorySegmentInfo, 0, len(baseSegments))
segments := make([]*procio.MemorySegmentInfo, 0, len(baseSegments))
for _, seg := range baseSegments {
if seg.SubSegments == nil || len(seg.SubSegments) == 0 {
segments = append(segments, seg)
Expand All @@ -83,7 +82,7 @@ func dumpMemory(c *cli.Context) error {
continue
}
if found {
rdr, err := procIO.NewMemoryReader(proc, seg)
rdr, err := procio.NewMemoryReader(proc, seg)
if err != nil {
fmt.Println(errors.Newf("could not read memory of process %d at address 0x%016X, reason %w", pid, seg.BaseAddress, err))
continue
Expand Down
Loading

0 comments on commit 68686cf

Please sign in to comment.