[QUESTION]How to use granary appropriately?

[renzhengeek]

I want to see granary's functionality,so I have tried the following:
1.installed granary sucessfully,with config GR_CLIENT=cfg,or null_plus,or bounds_checker, or everything_watched etc;

2.test with a brief kernel module,in which I kmalloc an array memory to a poiter,then
write out of its bound, and without free this memory even after delmod this module.

3.use dmesg, It always only says granary initialised, and test module COMING and GOING so on.It seems granary does not work!

what is wrong I probablly make?
could you provide a brief user manual. thanks a lot! :)

[pgoodman]

These are good questions and I never really documented the process very well. First, any client with null in the name is basically a useless client that does nothing. The null client doesn't add any instrumentation. The null_plus client is similar to the null client, but where if module-only instrumentation is used (the default), then null_plus will instrument module code, and kernel code invoked by module code. The watchpoint_null client again doesn't really do anything but does introduce instrumentation that checks for watched addresses (despite the watchpoint_null client never adding watchpoints to addresses).

Okay, so, the CFG tool. Most of the time, the only time something gets logged to the kernel debug message log is when modules are loaded/unloaded, or when Granary is initializing itself. Beyond that, Granary doesn't support a useful on-demand logging mechanism for transparency reasons (as well as laziness on my part). But, Granary can do "batch" logging that is accessible from user space, and this is how control-flow graph info is extracted from the CFG tool.

So, first, build Granary with the CFG tool, using approximately the following commands:
make clean all KERNEL=1 KERNEL_DIR=... GR_CLIENT=cfg

Next, launch your VM using qemu.sh in the Granary source code directory. You will most likely need to slightly change the paths in qemu.sh to match your local configuration (it's hard-coded to paths on my machine!).

Next, load Granary into the VM. If your VM host name (as configured with SSH) is slothvm then run the following command within the Granary source code directory.
python scripts/fastload.py --remote slothvm

Now Granary has been loaded, but not started. The distinction between loading and starting Granary is especially important for debugging purposes. For example, if you are debugging a new or existing client that is crashing, then you want to be able to catch the crash in the act, especially if it happens early during Granary's initialization or instrumentation of code.

A sane way to debug Granary is to modify the hard-coded paths in the Granary-provided .gdbinit to refer to files on your computer. When the paths have been fixed, run the gdb command (no command-line arguments are necessary), which should automatically attach gdb to your VM and load the kernel and Granary symbols (this is automatic so long as gdb picks up on and executes the .gdbinit file). If you don't care to debug then skip this step for now.

Now, go into your VM, either graphically or through SSH, and run the following command to initialize Granary: sudo touch /dev/granary. This turns Granary on. After turning on, any modules that are loaded into the kernel will be instrumented by Granary.

Now, load your module. If you dump the tail of the kernel debug message log via dmesg | tail (within the VM) then you should see that Granary recognizes the module.

This example started with the cfg tool, so now that our module is instrumented and running, lets pull out the control-flow graph. Run the following command again, which triggers the "batch reporting" feature: sudo touch /dev/granary (yes this is the same command for initializing Granary).

The logs are now temporarily stored in a relay file system, and can stored by running this command:
sudo cat /sys/kernel/debug/granary0 > /tmp/module.dump.

The file /tmp/module.dump now contains the raw data outputted by the CFG tool.

Using the bounds_checker tool is similar, but the batch dump/recording feature is not used. Instead, you need to set up debugging and adding a breakpoint to client::wp::visit_overflow (or client::visit_overflow if I got the symbol name wrong).

[renzhengeek]

thanks!
things are still not understood:
1.so far, I just do test locally without VM. And I wonder why running granary on VM.VM is slower.So, what's difference bettween local host and VM?
2. When using the bounds_checker tool,how to set up debugging and adding a breakpoint?
use kgdb to debug target module?
3.When using watchpoint, how to set watchpoint?

Sorry for boring you frequently.But I really want your help. :)

[pgoodman]

1. Easier to debug. When things work 100%, then go native. Until then, use
   the VM. Are you using kvm? You should be able to load a kernel module on
   your host/native machine that will speed up QEMU+KVM. Also, if you are
   using the provided qemu.sh, then it sets -cpu=host (or something like
   that), which also improves the VM's speed.

2. Set them with gdb; I don't use kgdb (or at least, not directly). When I
   use gdb, I remotely debug the kernel running within the VM. The .gdbinit
   file (with a few changes to fix some hard-coded paths) will automatically
   connect to a running VM and set a bunch of default breakpoints.

3. Watchpoints are set by wrapper code / the instrumentation itself. Here
   is a minimal example:
   https://github.com/Granary/granary/blob/master/clients/lifetime/kernel_wrappers.h

Here is how the bounds_checker does it:
https://github.com/Granary/granary/blob/master/clients/watchpoints/clients/bounds_checker/kernel/linux/wrappers.h

I suggest reading the watchpoints paper. They are not watchpoints in the
usual sense (as provided by debuggers).

Best Regards,

Peter Goodman,
http://www.petergoodman.me
65 High Park Ave.,
Toronto, Ontario
M6P 2R7

On 28 February 2014 20:56, Ren Zhen notifications@github.com wrote:

thanks!
things are still not understood:
1.so far, I just do test locally without VM. And I wonder why running
granary on VM.VM is slower.So, what's difference bettween local host and VM?
2. When using the bounds_checker tool,how to set up debugging and adding a
breakpoint?
use kgdb to debug target module?
3.When using watchpoint, how to set watchpoint?

Sorry for boring you frequently.But I really want your help. :)

Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-36412471
.

[renzhengeek]

Hey,thanks a lot!
1)I will try kvm.
2)ok!
3)I have read the watchpoints paper.

[renzhengeek]

Hi Peter,
In order to test bound_checker, according your guide:
1)adding a breakpoint to client::wp::visit_overflow(this is right) in .gdbinit at
'Granary breakpoints' section.
2)use gdb for remote debugging;
3)after initializing granary, insmod a test module in which there is a out of bound bug.

after all above,
a)when excuting run in gdb,it says :
-> The "remote" target does not support "run". Try "help target" or "continue".
b)excuting next in gdb, it says:
-> Cannot find bounds of current function
c)when excuting quit, vm is shutdown, everything needs to do again, waste lot of time;

what's the right way to debug? thanks!

[pgoodman]

1. are you using qemu.sh to launch qemu? If not, edit that file (it likely has a path to ~/Code/VM/vm.img, you need to replace this path.2) did you edit the. Gdbinit file? Here, look for a path to the vmlinux file (search for vmlinux), and fix it to point to the vmlinux file that was generated when you built your kernel for Granary. Sent from my mobile device. From: Ren ZhenSent: Sunday, March 2, 2014 4:22 AMTo: Granary/granaryReply To: Granary/granaryCc: Peter GoodmanSubject: Re: [granary] [QUESTION]How to use granary appropriately? ([QUESTION]How to use granary appropriately? #17)Hi Peter,
   In order to test bound_checker, according your guide:
   1)adding a breakpoint to client::wp::visit_overflow(this is right) in .gdbinit at

Granary breakpoints section.

2)use gdb for remote debugging;
3)after initializing granary, insmod a test module in which there is a out of bound bug.

after all above, when excuting run in gdb,it says :
-> The "remote" target does not support "run". Try "help target" or "continue".

what's the right way to debug? thanks!

—Reply to this email directly or view it on GitHub.

[renzhengeek]

1). yes,I did.

[pgoodman]

Is that a "yes" on step 2 as well? If so, can you copy and paste what shows
up in your terminal? Thanks.

Best Regards,

Peter Goodman,
http://www.petergoodman.me
65 High Park Ave.,
Toronto, Ontario
M6P 2R7

On 2 March 2014 20:00, Ren Zhen notifications@github.com wrote:

1). yes,I did.

Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-36474303
.

[renzhengeek]

yes,both 1) and 2).
1)$make clean all KERNEL=1 KERNEL_DIR=../linux-3.8 GR_CLIENT=bounds_checker
2)edit .gdbinit:
<...

Kernel setup

30 if !$in_user_space
31 file ~/vmlinuz-3.8.0granary
32 target remote : 9999
33 source ~/granary/granary.syms
34 end
...

Granary breakpoints

38 catch throw
39 b granary_fault
40 b granary_break_on_fault
41 b granary_break_on_translate
42 b granary_break_on_curiosity
43 b client::wp::visit_overflow
...>
3)edit qemu.sh:
<
#!/bin/bash
SMP=2
MEM=2048
NETWORK=e1000
#NETWORK=virtio
GRAPHIC= #-nographic
GRAPHIC_FLAG=0
QEMU=$(which qemu-system-x86_64)
$QEMU -cpu host,level=9 -enable-kvm -d int,cpu_reset -smp $SMP -m $MEM $GRAPHIC -hda /home/renzhen/vm.img -net nic,model=$NETWORK -net "user,hostfwd=tcp::5556-:22" -gdb tcp::9999 &

4)$bash qemu.sh
5)$python scripts/fastload.py --remote renzhen-vm
6)$ssh renzhen-vm; and excute $sudo touch /dev/granary;$dmesg | tail:
<...
renzhen@renzhen-vm:$ dmesg | tail
[ 157.727795] [granary] Registering module notifier...
[ 157.727796] [granary] Registering 'granary' device...
[ 157.727884] [granary] Registered 'granary' device.
[ 157.730475] [granary] Relay channel initialised.
[ 157.730478] [granary] Done; waiting for command to initialise Granary.
[ 157.730483] [granary] Notified of module 0xffffffffa077e240 [.text = ffffffffa0156000]
[ 157.730484] [granary] Module's name is: granary.
[ 157.730485] [granary] Ignoring module state change.
[ 204.257260] [granary] Initialising Granary...
[ 204.303904] [granary] Initialised.
...>
7)insmod my test kernel module:
<
renzhen@renzhen-vm:/hello$ sudo insmod hello.ko
renzhen@renzhen-vm:~/hello$ dmesg | tail
[ 786.557128] [granary] Notified about module (hello) state change: COMING.
[ 786.557254] [granary] Notified Granary of the module.
[ 786.558077] hello,a[0]=0
[ 786.558082] a[3]=1
[ 786.558086] [granary] Notified of module 0xffffffffa0126000 [.text = ffffffffa0124000]
[ 786.558088] [granary] Module's name is: hello.
[ 786.558089] [granary] Got internal representation for module.
[ 786.558091] [granary] Notifying Granary of the module...
[ 786.558092] [granary] Notified about module (hello) state change: LIVE.
[ 786.558093] [granary] Notified Granary of the module.

hello.ko kmalloc an dynamic array,then access out of bound,hello.c:
<
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/slab.h>
MODULE_LICENSE("Dual BSD/GPL");
static int _a;
static int hello_init(void)
{
//kmalloc two size of array
a = (int *)kmalloc(2_sizeof(int),GFP_KERNEL);
a[0] = 0; //ok
printk(KERN_ALERT "hello,a[0]=%i\n", a[0]);
a[3] = 1; //out of bound
printk(KERN_ALERT "a[3]=%i\n", a[3]);
return 0;
}
static void hello_exit(void)
{
printk(KERN_ALERT "goodbye,kernel\n");
}
module_init(hello_init);
module_exit(hello_exit);
MODULE_AUTHOR("renzhen");
MODULE_DESCRIPTION("This is a simple example!\n");
MODULE_ALIAS("A simplest example");

8)$gdb; and screen display:
<
after printf granary.sys
warning: section .strtab not found in /home/renzhen/granary/bin/granary.ko
warning: section .symtab not found in /home/renzhen/granary/bin/granary.ko
Function "__cxa_throw" not defined.
Catchpoint 1 (throw)
Breakpoint 2 at 0xffffffffa014d084: file /home/renzhen/granary/bin/./module.c, line 145.
Breakpoint 3 at 0xffffffffa014d064: file /home/renzhen/granary/bin/./module.c, line 134.
Breakpoint 4 at 0xffffffffa017c3f0: file /home/renzhen/granary/granary/code_cache.cc, line 32.
Breakpoint 5 at 0xffffffffa014d074: file /home/renzhen/granary/bin/./module.c, line 140.
Breakpoint 6 at 0xffffffffa01fd910: file /home/renzhen/granary/clients/watchpoints/instrument.h, line 354.
Breakpoint 7 at 0xffffffffa0203230: file /home/renzhen/granary/granary/kernel/interrupt.cc, line 370.
Breakpoint 8 at 0xffffffffa0200520: file clients/watchpoints/kernel/interrupt.cc, line 24.
Function "granary_break_on_gs_zero" not defined.
Breakpoint 9 (granary_break_on_gs_zero) pending.
Function "panic" not defined.
Breakpoint 10 (panic) pending.
Function "show_fault_oops" not defined.
Breakpoint 11 (show_fault_oops) pending.
Function "invalid_op" not defined.
Breakpoint 12 (invalid_op) pending.
Function "do_invalid_op" not defined.
Breakpoint 13 (do_invalid_op) pending.
Function "do_general_protection" not defined.
Breakpoint 14 (do_general_protection) pending.
Function "__schedule_bug" not defined.
Breakpoint 15 (__schedule_bug) pending.
Function "__stack_chk_fail" not defined.
Breakpoint 16 (__stack_chk_fail) pending.
Function "__die" not defined.
Breakpoint 17 (__die) pending.
Function "do_spurious_interrupt_bug" not defined.
Breakpoint 18 (do_spurious_interrupt_bug) pending.
Function "report_bug" not defined.
Breakpoint 19 (report_bug) pending.
Function "dump_stack" not defined.
Breakpoint 20 (dump_stack) pending.
Function "show_stack" not defined.
Breakpoint 21 (show_stack) pending.
Function "show_trace" not defined.
Breakpoint 22 (show_trace) pending.
Function "show_trace_log_lvl" not defined.
Breakpoint 23 (show_trace_log_lvl) pending.

strangely, for breakpoint 6(visit_overflow),func visit_overflow does not exists in /home/renzhen/granary/clients/watchpoints/instrument.h,instend of existing in /home/renzhen/granary/clients/watchpoints/clients/bounds_checker/instrument.cc at 225 line.
and I have try "break /home/renzhen/granary/clients/watchpoints/clients/bounds_checker/instrument.cc:client::wp::visit_overflow(or :visit_overflow)", gdb says func visit_overflow(or :visit_overflow) not exists in /home/renzhen/granary/clients/watchpoints/clients/bounds_checker/instrument.cc.Actually,the func truely exists in the file.
8)$(gdb) n; screen display:

9)after that VM always died!

[pgoodman]

I think one issue is that you are referencing vmlinuz instead of vmlinux.
The vmlinuz file is a compressed version of vmlinux. vmlinux is an ELF
file. The VM dying is not too surprising because gdb is connecting to it,
but because it isn't backing the connection with correct symbol information
(present in vmlinux), stuff is screwing up.

Try running locate vmlinux to see if it's anywhere on your computer. If
you can't find it, then look up how to use the "extract-vmlinux" script
that is found in linux/scripts/extract-vmlinux. Here's the most recent
version of the script:
https://github.com/torvalds/linux/blob/master/scripts/extract-vmlinux

If gdb doesn't find the client::wp::visit_overflow function, then try
running:
readelf --syms --wide bin/granary.ko | grep visit_overflow

You should be able to get the mangled name from there, and try adding a
breakpoint with that mangled name.

Best Regards,

Peter Goodman,
http://www.petergoodman.me
65 High Park Ave.,
Toronto, Ontario
M6P 2R7

On 2 March 2014 21:33, Ren Zhen notifications@github.com wrote:

yes,both 1) and 2).
1)$make clean all KERNEL=1 KERNEL_DIR=../linux-3.8 GR_CLIENT=bounds_checker
2)edit .gdbinit:
<...

Kernel setup

30 if !$in_user_space
31 file ~/vmlinuz-3.8.0granary
32 target remote : 9999
33 source ~/granary/granary.syms
34 end
...
Granary breakpoints

38 catch throw
39 b granary_fault

40 b granary_break_on_fault

41 b granary_break_on_translate
42 b granary_break_on_curiosity
43 b client::wp::visit_overflow
...>
3)edit qemu.sh:
<
#!/bin/bash
SMP=2
MEM=2048
NETWORK=e1000
#NETWORK=virtio
GRAPHIC= #-nographic
GRAPHIC_FLAG=0
QEMU=$(which qemu-system-x86_64)
$QEMU -cpu host,level=9 -enable-kvm -d int,cpu_reset -smp $SMP -m $MEM
$GRAPHIC -hda /home/renzhen/vm.img -net nic,model=$NETWORK -net
"user,hostfwd=tcp::5556-:22" -gdb tcp::9999 &
...>
4)$bash qemu.sh
5)$python scripts/fastload.py --remote renzhen-vm
6)$ssh renzhen-vm; and excute $sudo touch /dev/granary;$dmesg | tail:
<...
renzhen@renzhen-vm:$ dmesg | tail
[ 157.727795] [granary] Registering module notifier...
[ 157.727796] [granary] Registering 'granary' device...
[ 157.727884] [granary] Registered 'granary' device.
[ 157.730475] [granary] Relay channel initialised.
[ 157.730478] [granary] Done; waiting for command to initialise Granary.
[ 157.730483] [granary] Notified of module 0xffffffffa077e240 [.text =
ffffffffa0156000]
[ 157.730484] [granary] Module's name is: granary.
[ 157.730485] [granary] Ignoring module state change.
[ 204.257260] [granary] Initialising Granary...
[ 204.303904] [granary] Initialised.
...>
7)insmod my test kernel module:
<...
renzhen@renzhen-vm:/hello$ sudo insmod hello.ko
renzhen@renzhen-vm:~/hello$ dmesg | tail
[ 786.557128] [granary] Notified about module (hello) state change: COMING.
[ 786.557254] [granary] Notified Granary of the module.
[ 786.558077] hello,a[0]=0
[ 786.558082] a[3]=1
[ 786.558086] [granary] Notified of module 0xffffffffa0126000 [.text =
ffffffffa0124000]
[ 786.558088] [granary] Module's name is: hello.
[ 786.558089] [granary] Got internal representation for module.
[ 786.558091] [granary] Notifying Granary of the module...
[ 786.558092] [granary] Notified about module (hello) state change: LIVE.
[ 786.558093] [granary] Notified Granary of the module.

...>
hello.ko kmalloc an dynamic array,then access out of bound,hello.c:
<...
#include
#include
#include
#include

MODULE_LICENSE("Dual BSD/GPL");
static int

_a; static int hello_init(void) { //kmalloc two size of array a = (int
*)kmalloc(2_sizeof(int),GFP_KERNEL);
a[0] = 0; //ok
printk(KERN_ALERT "hello,a[0]=%i\n", a[0]);
a[3] = 1; //out of bound
printk(KERN_ALERT "a[3]=%i\n", a[3]);
return 0;
}
static void hello_exit(void)
{
printk(KERN_ALERT "goodbye,kernel\n");
}
module_init(hello_init);
module_exit(hello_exit);
MODULE_AUTHOR("renzhen");
MODULE_DESCRIPTION("This is a simple example!\n");
MODULE_ALIAS("A simplest example");
...>
8)$gdb; and screen display:
<...
after printf granary.sys
warning: section .strtab not found in /home/renzhen/granary/bin/granary.ko
warning: section .symtab not found in /home/renzhen/granary/bin/granary.ko
Function "

_cxa_throw" not defined. Catchpoint 1 (throw) Breakpoint 2 at
0xffffffffa014d084: file /home/renzhen/granary/bin/./module.c, line 145.
Breakpoint 3 at 0xffffffffa014d064: file
/home/renzhen/granary/bin/./module.c, line 134. Breakpoint 4 at
0xffffffffa017c3f0: file /home/renzhen/granary/granary/code_cache.cc, line
32. Breakpoint 5 at 0xffffffffa014d074: file
/home/renzhen/granary/bin/./module.c, line 140. Breakpoint 6 at
0xffffffffa01fd910: file
/home/renzhen/granary/clients/watchpoints/instrument.h, line 354.
Breakpoint 7 at 0xffffffffa0203230: file
/home/renzhen/granary/granary/kernel/interrupt.cc, line 370. Breakpoint 8
at 0xffffffffa0200520: file clients/watchpoints/kernel/interrupt.cc, line
24. Function "granary_break_on_gs_zero" not defined. Breakpoint 9
(granary_break_on_gs_zero) pending. Function "panic" not defined.
Breakpoint 10 (panic) pending. Function "show_fault_oops" not defined.
Breakpoint 11 (show_fault_oops) pending. Function "invalid_op" not defined.
Breakpoint 12 (invalid_op) pending. Function "do_invalid_op" not defined.
Breakpoint 13 (do_invalid_op) pending. Function "do_general_protection" not
defined. Breakpoint 14 (do_general_protection) pending. Function "_schedule_bug"
not defined.
Breakpoint 15 (
_schedule_bug) pending. Function "_stack_chk_fail" not defined.
Breakpoint 16 (
_stack_chk_fail) pending. Function "_die" not defined.
Breakpoint 17 (__die) pending.
Function "do_spurious_interrupt_bug" not defined.
Breakpoint 18 (do_spurious_interrupt_bug) pending.
Function "report_bug" not defined.
Breakpoint 19 (report_bug) pending.
Function "dump_stack" not defined.
Breakpoint 20 (dump_stack) pending.
Function "show_stack" not defined.
Breakpoint 21 (show_stack) pending.
Function "show_trace" not defined.
Breakpoint 22 (show_trace) pending.
Function "show_trace_log_lvl" not defined.
Breakpoint 23 (show_trace_log_lvl) pending.
....>
strangely, for breakpoint 6(visit_overflow),func visit_overflow does not
exists in /home/renzhen/granary/clients/watchpoints/instrument.h,instend of
existing in
/home/renzhen/granary/clients/watchpoints/clients/bounds_checker/instrument.cc
at 225 line.
8)$(gdb) n; screen display:

9)after that VM always died!

Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-36477271
.

[renzhengeek]

1)I found vmlinux in /linux-3.8 dir,then fix it;
2)excute readelf --syms --wide bin/granary.ko | grep visit_overflow:
<
renzhen@renzhen:/granary$ readelf --syms --wide bin/granary.ko | grep visit_overflow
25781: 0000000000000000 77 FUNC GLOBAL DEFAULT 22392 _ZN6client2wp14visit_overflowEmjPPh>
3)then add a breakpoint in .gdbinit file:
<43 b _ZN6client2wp14visit_overflowEmjPPh >
4)$gdb

But VM also died!

[pgoodman]

This might be a dumb question, but did you run the continue (c) command from gdb?

Okay, lets try running these steps:

1. Launch VM with qemu.sh.
2. Load Granary into VM with fastload.py. DO NOT initialize Granary!
3. Attach gdb to the VM by running gdb.
4. Run the bt command. You should see a stack trace, where the most recently called kernel function is native_safe_halt.
5. Try telling the VM to continue, using the c command. If the VM completely restarts then this is a CPU reset and I need to know. If the VM just hangs, then this is also something I need to know.
6. If the VM continues to work, then initialize Granary. Let me know if things get this far.

If you hit any issues then please copy & paste the whole gdb session.

Thanks for your patience!

[renzhengeek]

Do as your steps:
(gdb) bt
#0 default_idle () at arch/x86/kernel/process.c:391
#1 0xffffffff8100a759 in cpu_idle () at arch/x86/kernel/process.c:354
#2 0xffffffff8155653d in rest_init () at init/main.c:385
#3 0xffffffff81ac1bc4 in start_kernel () at init/main.c:643
#4 0xffffffff81ac132d in x86_64_start_reservations (real_mode_data=) at arch/x86/kernel/head64.c:122
#5 0xffffffff81ac1421 in x86_64_start_kernel (real_mode_data=0x8c800 <Address 0x8c800 out of bounds>) at arch/x86/kernel/head64.c:93
#6 0x0000000000000000 in ?? ()
(gdb) c
Continuing.

4)no native_safe_halt.
5)gdb seems hangs, but the VM is ok! neither restart nor hangs.

[renzhengeek]

what is more:
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
default_idle () at arch/x86/kernel/process.c:391
391 current_thread_info()->status |= TS_POLLING;
(gdb) n
394 }
(gdb) n
cpu_idle () at arch/x86/kernel/process.c:356
356 rcu_idle_exit();
(gdb) n
362 __exit_idle();
(gdb) n
331 while (!need_resched()) {
(gdb) n
365 tick_nohz_idle_exit();
(gdb) n
367 schedule();

[renzhengeek]

after a while,VM hangs(stoped),but gdb can excute c(continue).

[pgoodman]

single-stepping and leaving the VM interrupted (i.e. in the GDB command interpreter) for a while will usually kill the VM. This typically happens because of kernel watchdogs that look for soft/hard lockups and unresponsiveness, and then start doing bad things.

I usually see native_safe_halt but not seeing it is fine.

Have you tried loading your special module (after initializing Granary) to see if the breakpoint on visit_overflow gets hit?

[renzhengeek]

1)initializing Granary;
2)loading special module;
3)start gdb;terminal says:
{
warning: section .strtab not found in /home/renzhen/granary/bin/granary.ko
warning: section .symtab not found in /home/renzhen/granary/bin/granary.ko
Function "__cxa_throw" not defined.
Catchpoint 1 (throw)
Breakpoint 2 at 0xffffffffa0157084: file /home/renzhen/granary/bin/./module.c, line 145.
Breakpoint 3 at 0xffffffffa0157064: file /home/renzhen/granary/bin/./module.c, line 134.
Breakpoint 4 at 0xffffffffa01863f0: file /home/renzhen/granary/granary/code_cache.cc, line 32.
Breakpoint 5 at 0xffffffffa0157074: file /home/renzhen/granary/bin/./module.c, line 140.
Breakpoint 6 at 0xffffffffa0207910: file /home/renzhen/granary/clients/watchpoints/instrument.h, line 354.
Breakpoint 7 at 0xffffffffa020d230: file /home/renzhen/granary/granary/kernel/interrupt.cc, line 370.
Breakpoint 8 at 0xffffffffa020a520: file clients/watchpoints/kernel/interrupt.cc, line 24.
Function "granary_break_on_gs_zero" not defined.
Breakpoint 9 (granary_break_on_gs_zero) pending.
Breakpoint 10 at 0xffffffff8156a101: file kernel/panic.c, line 70.
Function "show_fault_oops" not defined.
Breakpoint 11 (show_fault_oops) pending.
Breakpoint 12 at 0xffffffff8157aa43: file arch/x86/kernel/entry_64.S, line 1301.
Breakpoint 13 at 0xffffffff81002d90: file arch/x86/kernel/traps.c, line 213.
Breakpoint 14 at 0xffffffff81573cf0: file arch/x86/kernel/traps.c, line 262.
Breakpoint 15 at 0xffffffff8156a60d: file kernel/sched/core.c, line 2778.
Breakpoint 16 at 0xffffffff81039400: file kernel/panic.c, line 467.
Breakpoint 17 at 0xffffffff81574360: file arch/x86/kernel/dumpstack.c, line 253.
Breakpoint 18 at 0xffffffff81003560: file arch/x86/kernel/traps.c, line 579.
Breakpoint 19 at 0xffffffff81237450: file lib/bug.c, line 128.
Breakpoint 20 at 0xffffffff81568e6c: file arch/x86/kernel/dumpstack.c, line 186.
Breakpoint 21 at 0xffffffff810056c0: file arch/x86/kernel/dumpstack.c, line 178.
Breakpoint 22 at 0xffffffff810056a0: file arch/x86/kernel/dumpstack.c, line 173.
Breakpoint 23 at 0xffffffff81005630: file arch/x86/kernel/dumpstack.c, line 166.
Warning: the current language does not match this frame.
(gdb) n
394 }
(gdb) n
cpu_idle () at arch/x86/kernel/process.c:356
356 rcu_idle_exit();
(gdb) n
362 __exit_idle();
(gdb) r
The "remote" target does not support "run". Try "help target" or "continue".
(gdb) n
331 while (!need_resched()) {
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
default_idle () at arch/x86/kernel/process.c:391
391 current_thread_info()->status |= TS_POLLING;
(gdb) n
394 }
(gdb)

}

breakpoint 6 still be set at wrong file /home/renzhen/granary/clients/watchpoints/instrument.h,
right file should be /home/renzhen/granary/clients/watchpoints/clients/bounds_checker/instrument.cc.
.gdbinit set breakpoint like this as you told me:
b _ZN6client2wp14visit_overflowEmjPPh

[pgoodman]

I agree that is unusual. Can you try doing disass _ZN6client... to list
out the instructions of the function? (or look at the breakpoint address,
and then disassemble it using disass 0x.... I suggest against
single-stepping. After the breakpoints are added and the instructions
disassembled, just run c.

Best Regards,

Peter Goodman,
http://www.petergoodman.me
65 High Park Ave.,
Toronto, Ontario
M6P 2R7

On 3 March 2014 00:27, Ren Zhen notifications@github.com wrote:

1)initializing Granary;
2)loading special module;

3)start gdb;terminal says:
{
warning: section .strtab not found in /home/renzhen/granary/bin/granary.ko
warning: section .symtab not found in /home/renzhen/granary/bin/granary.ko
Function "__cxa_throw" not defined.
Catchpoint 1 (throw)
Breakpoint 2 at 0xffffffffa0157084: file
/home/renzhen/granary/bin/./module.c, line 145.
Breakpoint 3 at 0xffffffffa0157064: file
/home/renzhen/granary/bin/./module.c, line 134.
Breakpoint 4 at 0xffffffffa01863f0: file
/home/renzhen/granary/granary/code_cache.cc, line 32.
Breakpoint 5 at 0xffffffffa0157074: file
/home/renzhen/granary/bin/./module.c, line 140.
Breakpoint 6 at 0xffffffffa0207910: file
/home/renzhen/granary/clients/watchpoints/instrument.h, line 354.
Breakpoint 7 at 0xffffffffa020d230: file
/home/renzhen/granary/granary/kernel/interrupt.cc, line 370.
Breakpoint 8 at 0xffffffffa020a520: file
clients/watchpoints/kernel/interrupt.cc, line 24.

Function "granary_break_on_gs_zero" not defined.
Breakpoint 9 (granary_break_on_gs_zero) pending.
Breakpoint 10 at 0xffffffff8156a101: file kernel/panic.c, line 70.

Function "show_fault_oops" not defined.
Breakpoint 11 (show_fault_oops) pending.
Breakpoint 12 at 0xffffffff8157aa43: file arch/x86/kernel/entry_64.S, line
1301.
Breakpoint 13 at 0xffffffff81002d90: file arch/x86/kernel/traps.c, line
213.
Breakpoint 14 at 0xffffffff81573cf0: file arch/x86/kernel/traps.c, line
262.
Breakpoint 15 at 0xffffffff8156a60d: file kernel/sched/core.c, line 2778.
Breakpoint 16 at 0xffffffff81039400: file kernel/panic.c, line 467.
Breakpoint 17 at 0xffffffff81574360: file arch/x86/kernel/dumpstack.c,
line 253.
Breakpoint 18 at 0xffffffff81003560: file arch/x86/kernel/traps.c, line
579.
Breakpoint 19 at 0xffffffff81237450: file lib/bug.c, line 128.
Breakpoint 20 at 0xffffffff81568e6c: file arch/x86/kernel/dumpstack.c,
line 186.
Breakpoint 21 at 0xffffffff810056c0: file arch/x86/kernel/dumpstack.c,
line 178.
Breakpoint 22 at 0xffffffff810056a0: file arch/x86/kernel/dumpstack.c,
line 173.
Breakpoint 23 at 0xffffffff81005630: file arch/x86/kernel/dumpstack.c,
line 166.
Warning: the current language does not match this frame.
(gdb) n
394 }
(gdb) n
cpu_idle () at arch/x86/kernel/process.c:356
356 rcu_idle_exit();
(gdb) n
362 __exit_idle();
(gdb) r

The "remote" target does not support "run". Try "help target" or
"continue".
(gdb) n
331 while (!need_resched()) {
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.

default_idle () at arch/x86/kernel/process.c:391
391 current_thread_info()->status |= TS_POLLING;
(gdb) n
394 }
(gdb)

}

breakpoint 6 still be set at wrong file
/home/renzhen/granary/clients/watchpoints/instrument.h,
right file should be
/home/renzhen/granary/clients/watchpoints/clients/bounds_checker/instrument.cc.
.gdbinit set breakpoint like this as you told me:
b _ZN6client2wp14visit_overflowEmjPPh

Reply to this email directly or view it on GitHubhttps://github.com//issues/17#issuecomment-36482680
.

[renzhengeek]

1. disass _ZN6client...
   (gdb) disass _ZN6client2wp14visit_overflowEmjPPh
   Dump of assembler code for function _ZN6client2wp14visit_overflowEmjPPh:
   0xffffffffa0207910 <+0>: mov %rdi,%rax
   0xffffffffa0207913 <+3>: shr $0x31,%rax
   0xffffffffa0207917 <+7>: shl $0x4,%rax
   0xffffffffa020791b <+11>: cmp -0x5da4a340(%rax),%edi
   0xffffffffa0207921 <+17>: jb 0xffffffffa020792d <client::wp::visit_overflow(unsigned long, unsigned int, unsigned char**)+29>
   0xffffffffa0207923 <+19>: add %esi,%edi
   0xffffffffa0207925 <+21>: cmp -0x5da4a33c(%rax),%edi
   0xffffffffa020792b <+27>: jbe 0xffffffffa0207930 <client::wp::visit_overflow(unsigned long, unsigned int, unsigned char**)+32>
   0xffffffffa020792d <+29>: repz retq
   0xffffffffa020792f <+31>: nop
   0xffffffffa0207930 <+32>: sub $0x8,%rsp
   0xffffffffa0207934 <+36>: mov $0xf4,%edx
   0xffffffffa0207939 <+41>: mov $0xffffffffa02520e0,%rsi
   0xffffffffa0207940 <+48>: mov $0xffffffffa022db06,%rdi
   0xffffffffa0207947 <+55>: xor %eax,%eax
   0xffffffffa0207949 <+57>: callq _0x5768d1(%rip) # 0xffffffffa077e220 granary::printf
   0xffffffffa020794f <+63>: callq 0xffffffffa0157060 <granary_break_on_fault>
   0xffffffffa0207954 <+68>: add $0x8,%rsp
   0xffffffffa0207958 <+72>: jmpq 0xffffffffa0157080 <granary_fault>
   End of assembler dump.
2. disass 0x...
   (gdb) disass 0xffffffffa0207910
   Dump of assembler code for function ZN6client2wp14visit_overflowEmjPPh:
   0xffffffffa0207910 <+0>: mov %rdi,%rax
   0xffffffffa0207913 <+3>: shr $0x31,%rax
   0xffffffffa0207917 <+7>: shl $0x4,%rax
   0xffffffffa020791b <+11>: cmp -0x5da4a340(%rax),%edi
   0xffffffffa0207921 <+17>: jb 0xffffffffa020792d <client::wp::visit_overflow(unsigned long, unsigned int, unsigned char_)+29>
   0xffffffffa0207923 <+19>: add %esi,%edi
   0xffffffffa0207925 <+21>: cmp -0x5da4a33c(%rax),%edi
   0xffffffffa020792b <+27>: jbe 0xffffffffa0207930 <client::wp::visit_overflow(unsigned long, unsigned int, unsigned char_*)+32>
   0xffffffffa020792d <+29>: repz retq
   0xffffffffa020792f <+31>: nop
   0xffffffffa0207930 <+32>: sub $0x8,%rsp
   0xffffffffa0207934 <+36>: mov $0xf4,%edx
   0xffffffffa0207939 <+41>: mov $0xffffffffa02520e0,%rsi
   0xffffffffa0207940 <+48>: mov $0xffffffffa022db06,%rdi
   0xffffffffa0207947 <+55>: xor %eax,%eax
   0xffffffffa0207949 <+57>: callq *0x5768d1(%rip) # 0xffffffffa077e220 granary::printf
   0xffffffffa020794f <+63>: callq 0xffffffffa0157060 <granary_break_on_fault>
   0xffffffffa0207954 <+68>: add $0x8,%rsp
   0xffffffffa0207958 <+72>: jmpq 0xffffffffa0157080 <granary_fault>
   End of assembler dump.

0xffffffffa0207910 is from Breakpoint 6 at 0xffffffffa0207910: file /home/renzhen/granary/clients/watchpoints/instrument.h, line 354.

3)add breakpoint,it seems failure:
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
default_idle () at arch/x86/kernel/process.c:391
391 current_thread_info()->status |= TS_POLLING;
(gdb) b 0xffffffffa020792d
Function "0xffffffffa020792d" not defined.
Breakpoint 24 (0xffffffffa020792d) pending.
(gdb) b 0xffffffffa0207930
Function "0xffffffffa0207930" not defined.
Breakpoint 25 (0xffffffffa0207930) pending.
(gdb) c
Continuing.

0xffffffffa020792d and 0xffffffffa0207930 are from 2)
0xffffffffa0207921 <+17>: jb 0xffffffffa020792d <client::wp::visit_overflow(unsigned long, unsigned int, unsigned char**)+29>
0xffffffffa0207923 <+19>: add %esi,%edi
0xffffffffa0207925 <+21>: cmp -0x5da4a33c(%rax),%edi
0xffffffffa020792b <+27>: jbe 0xffffffffa0207930 <client::wp::visit_overflow(unsigned long, unsigned int, unsigned char**)+32>

[pgoodman]

 To add a breakpoint to an address, do b *0x.... that looks like it's finding the right function, despite the debug info saying the wrong file/line. Sent from my mobile device. From: Ren ZhenSent: Monday, March 3, 2014 1:16 AMTo: Granary/granaryReply To: Granary/granaryCc: Peter GoodmanSubject: Re: [granary] [QUESTION]How to use granary appropriately? (#17)1) disass _ZN6client...
(gdb) disass ZN6client2wp14visitoverflowEmjPPh
Dump of assembler code for function ZN6client2wp14visitoverflowEmjPPh:
0xffffffffa0207910 <+0>: mov %rdi,%rax
0xffffffffa0207913 <+3>: shr $0x31,%rax
0xffffffffa0207917 <+7>: shl $0x4,%rax
0xffffffffa020791b <+11>: cmp -0x5da4a340(%rax),%edi
0xffffffffa0207921 <+17>: jb 0xffffffffa020792d
0xffffffffa0207923 <+19>: add %esi,%edi
0xffffffffa0207925 <+21>: cmp -0x5da4a33c(%rax),%edi
0xffffffffa020792b <+27>: jbe 0xffffffffa0207930
0xffffffffa020792d <+29>: repz retq
0xffffffffa020792f <+31>: nop
0xffffffffa0207930 <+32>: sub $0x8,%rsp
0xffffffffa0207934 <+36>: mov $0xf4,%edx
0xffffffffa0207939 <+41>: mov $0xffffffffa02520e0,%rsi
0xffffffffa0207940 <+48>: mov $0xffffffffa022db06,%rdi
0xffffffffa0207947 <+55>: xor %eax,%eax
0xffffffffa0207949 <+57>: callq 0x5768d1(%rip) # 0xffffffffa077e220 granary::printf
0xffffffffa020794f <+63>: callq 0xffffffffa0157060
0xffffffffa0207954 <+68>: add $0x8,%rsp
0xffffffffa0207958 <+72>: jmpq 0xffffffffa0157080
End of assembler dump.
2) disass 0x...
(gdb) disass 0xffffffffa0207910
Dump of assembler code for function ZN6client2wp14visitoverflowEmjPPh:
0xffffffffa0207910 <+0>: mov %rdi,%rax
0xffffffffa0207913 <+3>: shr $0x31,%rax
0xffffffffa0207917 <+7>: shl $0x4,%rax
0xffffffffa020791b <+11>: cmp -0x5da4a340(%rax),%edi
0xffffffffa0207921 <+17>: jb 0xffffffffa020792d <client::wp::visit_overflow(unsigned long, unsigned int, unsigned char*)+29>
0xffffffffa0207923 <+19>: add %esi,%edi
0xffffffffa0207925 <+21>: cmp -0x5da4a33c(%rax),%edi
0xffffffffa020792b <+27>: jbe 0xffffffffa0207930
0xffffffffa020792d <+29>: repz retq
0xffffffffa020792f <+31>: nop
0xffffffffa0207930 <+32>: sub $0x8,%rsp
0xffffffffa0207934 <+36>: mov $0xf4,%edx
0xffffffffa0207939 <+41>: mov $0xffffffffa02520e0,%rsi
0xffffffffa0207940 <+48>: mov $0xffffffffa022db06,%rdi
0xffffffffa0207947 <+55>: xor %eax,%eax
0xffffffffa0207949 <+57>: callq *0x5768d1(%rip) # 0xffffffffa077e220 granary::printf
0xffffffffa020794f <+63>: callq 0xffffffffa0157060
0xffffffffa0207954 <+68>: add $0x8,%rsp
0xffffffffa0207958 <+72>: jmpq 0xffffffffa0157080
End of assembler dump.

0xffffffffa0207910 is from Breakpoint 6 at 0xffffffffa0207910: file /home/renzhen/granary/clients/watchpoints/instrument.h, line 354.

3)add breakpoint,it seems failure:
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
default_idle () at arch/x86/kernel/process.c:391
391 current_thread_info()->status |= TS_POLLING;
(gdb) b 0xffffffffa020792d
Function "0xffffffffa020792d" not defined.
Breakpoint 24 (0xffffffffa020792d) pending.
(gdb) b 0xffffffffa0207930
Function "0xffffffffa0207930" not defined.
Breakpoint 25 (0xffffffffa0207930) pending.
(gdb) c
Continuing.

0xffffffffa020792d and 0xffffffffa0207930 are from 2)
0xffffffffa0207921 <+17>: jb 0xffffffffa020792d
0xffffffffa0207923 <+19>: add %esi,%edi
0xffffffffa0207925 <+21>: cmp -0x5da4a33c(%rax),%edi
0xffffffffa020792b <+27>: jbe 0xffffffffa0207930

—Reply to this email directly or view it on GitHub.

[renzhengeek]

Hi peter,
It still acts like befor.
I think I should be firstly familar with the source code to know how it works at earth,
then continue to solve this problem.

1. When quit from gdb, then VM is also killed.
   is this formal? how to fix it.

2)the compiler may optimized the kernel code.
So visit_overflow breakpoint always occurs in the wrong file.Does it possible?

3)Have you tried set breakpoint visi_overflow in your computer?
If your side is ok, then I will try to git the latest source, do everything from begin.