CONTENTS

Home
Updates
Software
Electronics
Android / iOS
Videos
Music
Resume
Contact

HTTPS VERSION


Core File Analysis

(When Missing Libraries)

Posted: May 12, 2019

Introduction

There are times when I have to deal with core files that crashed in a library I worked on where I don't get any of the other libraries or original executable that called the library that crashed. This can sometimes make it difficult to examine the core file. The best example may be when the crash comes in a Java program calling a C library with JNI. Java will catch the segfault and dump a hs_err<pid>.log file with a ton of info, but loading the core in gdb can be problematic if I don't have the exact original java executable.

To make this easier to deal with, I added code to magic_elf so that it can modify registers in a core a file so the RSP and RIP registers can point to the actual point of the crash and not to the missing java binary.

The same thing can be done with the output of libraries such as libSegFault.so.

Analyzing a Java Core

First step is to download magic_elf and build it:

git clone https://github.com/mikeakohn/magic_elf.git cd magic_elf make

In the sample directory is a Java program called Test.java that makes a call to a static method forceSegfault() defined in Crash.java. The method forceSegfault() is a native method that makes a call into libcrash.so. This function will dereference a NULL pointer forcing a crash.

In the samples directory, after typing the following:

ulimit -c unlimited make java export LD_LIBRARY_PATH=. java Test

Two things should now be created: core and an hs_err_<pid>.log file. In my example I can now type:

gdb -c core bt

Which gives a backtrace that looks like this:

(gdb) bt #0 0x00007fc5c216ee97 in ?? () #1 0xfffffffe7ffbfa07 in ?? () #2 0x00007fc5c2964d90 in ?? () #3 0x00007fc5c1adbb80 in ?? () #4 0x00007fc5c129f886 in ?? () #5 0x0000000000000000 in ?? ()

Not very useful. Since I compiled this on my system I could type:

set solib-search-path .

This would cause my library to get loaded along with java and every other supporting library. But to simulate not having access to the original Java executable, I will look into the hs_err file to find the shared library load address of libcrash.so and load it manually:

Dynamic libraries: 7fc58cbc8000-7fc58cbc9000 r-xp 00000000 08:11 6554941 libcrash.so

Since libcrash.so was compiled with the -g option and not stripped, gdb can load the library into memory so bt can show variable names, function names, file names, etc by doing:

(gdb) add-symbol-file libcrash.so 0x7fc58cbc8000

Unfortunately, because the "java" executable has the top entries on the stack and since that executable hasn't been loaded into gdb, gdb still can't figure out how to give a readable stack trace into libcrash.so. At this point, magic_elf can be used to modify the RSP stack register and RIP instruction pointer register in the core file to point to the top of the stack where libcrash.so is being called. Before this can be done, two pieces of information are needed: thread id of the crash and RSP and RIP values. The thread id can be retreived from gdb and the RSP and RIP are in the hs_err file:

(gdb) info thread Id Target Id Frame * 1 LWP 30755 0x00007fc5c216ee97 in ?? () 2 LWP 30757 0x00007fc5c1af59f3 in ?? () 3 LWP 30768 0x00007fc5c1af59f3 in ?? () 4 LWP 30760 0x00007fc5c1af5ed9 in ?? () 5 LWP 30758 0x00007fc5c1af86d6 in ?? () 6 LWP 30754 0x00007fc5c1af0d2d in ?? () ...

So the thread id is 30755. Actually, the thread id is also reported in the hs_err file. The hs_err files also shows:

Registers: RAX=0x0000000000000000, RBX=0x00007fc58dc005b8, RCX=0x0000000000000b40, RDX=0x0000000000000000 RSP=0x00007fc5c2965900, RBP=0x00007fc5c2965920, RSI=0x00007fc5c251d8c0, RDI=0x00007fc5c251c760 R8 =0x00007fc5c251d8c0, R9 =0x00007fc5c2966700, R10=0x000000000000000b, R11=0x00007fc5c21ae7e0 R12=0x0000000000000000, R13=0x00007fc58dc005b8, R14=0x00007fc5c2965998, R15=0x00007fc5b8012000 RIP=0x00007fc58cbc86d4, EFLAGS=0x0000000000010206, CSGSFS=0x002b000000000033, ERR=0x0000000000000006 TRAPNO=0x000000000000000e

Just a word of warning, magic_elf will overwrite information in the core file so it would be a good idea to only do this to a backup of the core. So I will type:

cp core core.modified ../magic_elf -modify_core 30755 rsp 0x7fc5c2965900 core.modified ../magic_elf -modify_core 30755 rip 0x7fc58cbc86d4 core.modified

Now the new core file should be able to be loaded with:

gdb -c core.modified add-symbol-file libcrash.so 0x7fc58cbc8000

Oddly, this wasn't working too. Even weirder, typing "info shared" into gdb was showing the address differently:

0x00007fc58cbc85c0 0x00007fc58cbc86da No libcrash.so

Using magic_elf on libcrash.so, the address of the .text section + the address of the library load address listed in the hs_err file gives:

Section Header 12 (offset=0x64f0) --------------------------------------------- sh_name: 135 (.text) sh_type: 1 (SHT_PROGBITS) sh_flags: 0x6 (SHF_ALLOC SHF_EXECINSTR ) sh_addr: 0x5c0 sh_offset: 0x5c0 sh_size: 282 sh_link: 0 sh_info: 0 sh_addralign: 16 sh_entsize: 0 0x7fc58cbc8000 + 0x5c0 = 0x00007fc58cbc85c0

So instead, this should be typed into gdb:

add-symbol-file libcrash.so 0x7fc58cbc85c0

Kind of seems like a bug in gdb since info shared shows the same load address as it did before. Either way, this seems to solve it:

(gdb) bt (gdb) bt #0 0x00007fc58cbc86d4 in Java_Crash_forceSegfault (env=0x7fc5c128d249, obj=0x7fc5c2964fb0) at crash.c:15 #1 0x00007fc5c14b4cca in ?? () #2 0x00000008008e6c36 in ?? () #3 0x00007fc5c1570ea3 in ?? () #4 0x00007fc5b8012000 in ?? () #5 0x00007fc5c2965130 in ?? ()

This core file can now be analyzed with gdb. It would probably be a good idea to modify all the other registers in the core file too in gdb looks at a variable's value from registers instead of memory. In the last core file I analyzed I did this using a pretty simple bash script that invoked magic_elf for every register listed in the log file.

Copyright 1997-2019 - Michael Kohn