Chapter 5. Sample lcrash Sessions

5.1. Analyze Kernel Modules

This session should describe how to use lcrash in analyzing kernel modules. First of all we make use of lcrash commands namelist and symtab.

We have a kernel module my_dummy.o containing a locale variable DUMMY of type dummy_t. The corresponding code fragment is as follows:


typedef struct dummy_s{
        int member1;
        char *member2;
        struct dummy_s *member3;
} dummy_t;

static dummy_t DUMMY={0, "just a demonstration", &DUMMY};
	
      

Our intention will be to examine this local data with lcrash. To make it little more tricky we analyze a live dump and the module will be loaded while lcrash is running.

Our module was compiled using gcc option -gstabs to create type information. The symbol table of the module was generated using a command line like nm my_dummy.o > /tmp/my_dummy.map.

The file my_dummy.o was also copied to /tmp.

  1. Start lcrash.

    bash# lcrash /boot/System.map-2.2.18 /dev/mem /boot/Kerntypes 
    map = /boot/System.map-2.2.18, vmdump = /dev/mem, outfile = stdout, kerntypes =
     /boot/Kerntypes
    
    Please wait...
            Loading system map ........................... Done.
            Loading type info (Kerntypes) ... Done.
            Loading ksyms from dump ........ Done.
    >> 
    	
  2. Look what modules are loaded.

    >> module
        ADDR      SIZE USED NAME                               REFS 
    ===========================================================================
    d0103000     17928    1 ibmtr_cs                      []
    d00fe000      6608    2 ds                            [ibmtr_cs]
    d00f3000     23408    2 i82365                        []
    d00e6000     46848    0 pcmcia_core                   [ibmtr_cs
                                                           ds
                                                           i82365]
    c02ad0e0         0    1 kernel_module                 []
    ===========================================================================
    	
  3. From another shell, load module my_dummy.

    bash# insmod my_dummy.o
    bash#
    	
  4. Verify the former action with lcrash.

    >> module
        ADDR      SIZE USED NAME                               REFS 
    ===========================================================================
    d0000000      1120    0 my_dummy                      []
    d0103000     17928    1 ibmtr_cs                      []
    d00fe000      6608    2 ds                            [ibmtr_cs]
    d00f3000     23408    2 i82365                        []
    d00e6000     46848    0 pcmcia_core                   [ibmtr_cs
                                                           ds
                                                           i82365]
    c02ad0e0         0    1 kernel_module                 []
    ===========================================================================
    	
  5. Look which symbols of the new module are exported.

    >> module -f my_dummy
    EXPORTED MODULE SYMBOLS: 
    ===========================================================================
            Module: my_dummy
            Number of exported symbols: 6
    
                ADDR NAME [MODULE]
    
            d0000000 __insmod_my_dummy_O/home/aherrman/CPP/crash_ex/my_dummy.o_
    M3B1CDF3B_V131602 [my_dummy]
            d0000060 dummy_init                           [my_dummy]
            d0000060 __insmod_my_dummy_S.text_L447        [my_dummy]
            d000021f __insmod_my_dummy_S.rodata_L29       [my_dummy]
            d000041c __insmod_my_dummy_S.bss_L16          [my_dummy]
            d0000240 __insmod_my_dummy_S.data_L260        [my_dummy]
    
    ===========================================================================
    	
  6. Load type information of the module.

    >> namelist -a /tmp/my_dummy.o
    .The current namelist is /tmp/my_dummy.o (1)
    >> namelist
    INDEX  NAMELIST
    =================================================
        0  /boot/Kerntypes
        1  /tmp/my_dummy.o
    =================================================
    
    The current namelist is /tmp/my_dummy.o (1)
    	
  7. Load symbol table of the module.

    >> symtab -a /tmp/my_dummy.map my_dummy
    Adding symbol table.
               filename: /tmp/my_dummy.map
            text_offset:        0
            data_offset:        0
          rodata_offset:        0
             bss_offset:        0
            module size:     1120
    ..Done.
    	

    Something went wrong, offsets of text and data sections of the module should not be zero. This is caused by the fact, that we added our module after lcrash was started. We have to remove the loaded symbol table and we have to recreate the table __ksymtab__.

  8. Remove our new symbol table and __ksymtab__.

    >> symtab -l
    Loaded symbol tables:
    ===========================================================================
    #SYMS:     7803 /boot/System.map-2.2.18
     TEXT:        0 DATA:        0 RODATA:        0 BSS:        0
    #SYMS:     1163 __ksymtab__
     TEXT:        0 DATA:        0 RODATA:        0 BSS:        0
    #SYMS:       14 /tmp/my_dummy.map [my_dummy]
     TEXT:        0 DATA:        0 RODATA:        0 BSS:        0
    ===========================================================================
    >> symtab -r /tmp/my_dummy.map
    Removing symbol table.
    Done.
    >> symtab -r __ksymtab__
    Removing symbol table.
    Done.
    	
  9. Recreate symbol table __ksymtab__.

     >> symtab -a __ksymtab__
    Adding symbol table.
    
            Loading ksyms from dump ........
    Done.
    	
  10. Load our new symbol table again.

    >> symtab -a /tmp/my_dummy.map my_dummy
    Adding symbol table.
               filename: /tmp/my_dummy.map
            text_offset: d0000060
            data_offset: d0000240
          rodata_offset: d000021f
             bss_offset: d000041c
            module size:     1120
    ..Done.
    >> symtab -l 
    Loaded symbol tables:
    ===========================================================================
    #SYMS:     7803 /boot/System.map-2.2.18
     TEXT:        0 DATA:        0 RODATA:        0 BSS:        0
    #SYMS:     1169 __ksymtab__
     TEXT:        0 DATA:        0 RODATA:        0 BSS:        0
    #SYMS:       14 /tmp/my_dummy.map [my_dummy]
     TEXT: d0000060 DATA: d0000240 RODATA: d000021f BSS: d000041c
    ===========================================================================
    	
  11. Look which symbols are available in module my_dummy.

    >> symtab -l -f /tmp/my_dummy.map
    ===========================================================================
    #SYMS:       14 /tmp/my_dummy.map [my_dummy]
     TEXT: d0000060 DATA: d0000240 RODATA: d000021f BSS: d000041c
    
        ADDR  OFFSET  TYPE         NAME
    ------------------------------------------------------------
    d0000060       0  GLOBAL_TEXT  dummy_init
    d00000f0       0  LOCAL_TEXT   dummy_xmit
    d0000130       0  LOCAL_TEXT   dummy_get_stats
    d0000140       0  LOCAL_TEXT   dummy_open
    d0000160       0  LOCAL_TEXT   dummy_close
    d0000180       0  LOCAL_TEXT   set_multicast_list
    d0000190       0  LOCAL_TEXT   dummy_probe
    d00001b0       0  GLOBAL_TEXT  init_module
    d00001f0       0  GLOBAL_TEXT  cleanup_module
    d000021f       0  LOCAL_TEXT   Letext
    d0000240       0  LOCAL_DATA   DUMMY
    d0000260       0  LOCAL_DATA   dev_dummy
    d000041c       0  LOCAL_DATA   dummy_name
    d00004c0       0  ABS          /tmp/my_dummy.map_END
    ------------------------------------------------------------
    ===========================================================================
    	
  12. Try to examine the local variable DUMMY of our module.

    >> whatis DUMMY
        ADDR  OFFSET  TYPE         NAME
    ============================================================
    d0000240       0  LOCAL_DATA   DUMMY
    >> whatis dummy_t
    struct dummy_s 
    struct dummy_s {
            int member1;
            char *member2;
            struct dummy_s *member3;
    };
    >> print *(dummy_t*) d0000240
    struct dummy_s {
            member1 = 0
            member2 = 0xd000021f
            member3 = 0xd0000240
    }
    
    >> whatis dummy_s.member2
    char *
    >> print (char*) 0xd000021f
    0xd000021f "just a demonstration"
    	

Furthermore an additional symbol table of a kernel module provides you function names when setting up stack back-traces with trace or strace and when using disassembling routine dis.