Core File Analysis
(When Missing Libraries)
Posted: May 12, 2019
There are times when I have to deal with core files that crashed in a library I worked on where I don't get any of the other libraries or original executable that called the library that crashed. This can sometimes make it difficult to examine the core file. The best example may be when the crash comes in a Java program calling a C library with JNI. Java will catch the segfault and dump a hs_err<pid>.log file with a ton of info, but loading the core in gdb can be problematic if I don't have the exact original java executable.
To make this easier to deal with, I added code to magic_elf so that it can modify registers in a core a file so the RSP and RIP registers can point to the actual point of the crash and not to the missing java binary.
The same thing can be done with the output of libraries such as libSegFault.so.
Analyzing a Java Core
First step is to download magic_elf and build it:
In the sample directory is a Java program called Test.java that makes a call to a static method forceSegfault() defined in Crash.java. The method forceSegfault() is a native method that makes a call into libcrash.so. This function will dereference a NULL pointer forcing a crash.
In the samples directory, after typing the following:
Two things should now be created: core and an hs_err_<pid>.log file. In my example I can now type:
Which gives a backtrace that looks like this:
Not very useful. Since I compiled this on my system I could type:
This would cause my library to get loaded along with java and every other supporting library. But to simulate not having access to the original Java executable, I will look into the hs_err file to find the shared library load address of libcrash.so and load it manually:
Since libcrash.so was compiled with the -g option and not stripped, gdb can load the library into memory so bt can show variable names, function names, file names, etc by doing:
Unfortunately, because the "java" executable has the top entries on the stack and since that executable hasn't been loaded into gdb, gdb still can't figure out how to give a readable stack trace into libcrash.so. At this point, magic_elf can be used to modify the RSP stack register and RIP instruction pointer register in the core file to point to the top of the stack where libcrash.so is being called. Before this can be done, two pieces of information are needed: thread id of the crash and RSP and RIP values. The thread id can be retreived from gdb and the RSP and RIP are in the hs_err file:
So the thread id is 30755. Actually, the thread id is also reported in the hs_err file. The hs_err files also shows:
Just a word of warning, magic_elf will overwrite information in the core file so it would be a good idea to only do this to a backup of the core. So I will type:
Now the new core file should be able to be loaded with:
Oddly, this wasn't working too. Even weirder, typing "info shared" into gdb was showing the address differently:
Using magic_elf on libcrash.so, the address of the .text section + the address of the library load address listed in the hs_err file gives:
So instead, this should be typed into gdb:
Kind of seems like a bug in gdb since info shared shows the same load address as it did before. Either way, this seems to solve it:
This core file can now be analyzed with gdb. It would probably be a good idea to modify all the other registers in the core file too in gdb looks at a variable's value from registers instead of memory. In the last core file I analyzed I did this using a pretty simple bash script that invoked magic_elf for every register listed in the log file.
Copyright 1997-2019 - Michael Kohn