ubuntu-gdb调试相关

分类于

Linux 环境下 使用 gdb 调试 程序，主要是 skynet 时不时就崩一下

docker 内使用 gdb 调试 core dumped 的正确姿势

因为 docker 默认会挂载 宿主 的 /proc 目录，在 docker 无法修改此文件，所以在 宿主机上修改即可

前置条件：程序编译时有 -g 标记

  1. 在宿主机上修改文件 

    1
    $ echo '/my_code_dump/core.%t.%e.%p' | sudo tee /proc/sys/kernel/core_pattern

    • 可以进 docker 里面看看是否成功

      1
      2
      root@b4d07a253268:/# cat /proc/sys/kernel/core_pattern
      /my_code_dump/core.%t.%e.%p # 修改ok

  2. 启动 run 镜像是 加上参数

    1
    --ulimit core=-1 --security-opt seccomp=unconfined

    • 举个栗子

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      $ docker run \
      -it \
      --name mmo_skynet \
      -v /home/wilker/Desktop/a_mmo_skynet:/skyet_app \
      -v $PWD/.ssh:/root/.ssh \
      --link redis_server:redis_db \
      -p 9777:9777 \
      -p 9555:9555 \
      --ulimit core=-1 \
      --security-opt seccomp=unconfined \
      my_skynet_gdb:latest \
      bash

      # 然后新建存放 core dumped 的目录
      # mkdir /my_code_dump

      • 进去容器之后查看一下是否打开: ulimit -a ， 以下内容表示是打开的

        1
        2
        3
        4
        5
        6
        7
        8
        9
        10
        11
        12
        13
        14
        15
        16
        17
        root@021a1e631a7c:/my_code_dump# ulimit -a
        core file size          (blocks, -c) unlimited
        data seg size           (kbytes, -d) unlimited
        scheduling priority             (-e) 0
        file size               (blocks, -f) unlimited
        pending signals                 (-i) 7744
        max locked memory       (kbytes, -l) 64
        max memory size         (kbytes, -m) unlimited
        open files                      (-n) 1048576
        pipe size            (512 bytes, -p) 8
        POSIX message queues     (bytes, -q) 819200
        real-time priority              (-r) 0
        stack size              (kbytes, -s) 8192
        cpu time               (seconds, -t) unlimited
        max user processes              (-u) unlimited
        virtual memory          (kbytes, -v) unlimited
        file locks                      (-x) unlimited

  3. 然后让程序跑崩溃，就会在 core dumped 文件了。比如

    1
    2
    3
    4
    5
    6
    7
    8
    # /skyet_app/3rd/skynet/skynet config
    ...
    [:0000000d] 【D|   login_server】--- login_server, socket 已断开，MSG.close
    [:0000000d] 【D|   login_server】--- login_server, socket 已断开，MSG.close
    ./run: line 6:    33 Segmentation fault      (core dumped) ../3rd/skynet/skynet config # 崩了

    root@021a1e631a7c:/my_code_dump# ls # 查看是否产生了
    core.1501679371.skynet.192

  4. 使用 gdb 看看文件

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    root@021a1e631a7c:/my_code_dump# gdb /skyet_app/3rd/skynet/skynet core.1501679371.skynet.192 # 进入查看
    GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
    Copyright (C) 2016 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "x86_64-linux-gnu".
    Type "show configuration" for configuration details.
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/>.
    Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.
    For help, type "help".
    Type "apropos word" to search for commands related to "word"...
    Reading symbols from /skyet_app/3rd/skynet/skynet...done.
    [New LWP 197]
    [New LWP 192]
    [New LWP 196]
    [New LWP 199]
    [New LWP 200]
    [New LWP 202]
    [New LWP 203]
    [New LWP 193]
    [New LWP 198]
    [New LWP 194]
    [New LWP 201]
    [New LWP 195]

    warning: Could not load shared library symbols for 8 libraries, e.g. ../3rd/skynet/cservice/logger.so.
    Use the "info sharedlibrary" command to see the complete listing.
    Do you need "set solib-search-path" or "set sysroot"?
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
    Core was generated by `../3rd/skynet/skynet config'.
    Program terminated with signal SIGSEGV, Segmentation fault.
    #0  luaH_getshortstr (key=0x7f1425224ef0, t=0x7f14252ba000) at ltable.c:525 # 最后挂载这一行
    525         if (ttisshrstring(k) && eqshrstr(tsvalue(k), key))
    [Current thread is 1 (Thread 0x7f1426e20700 (LWP 197))]
    (gdb) bt # 查看调用栈
    #0  luaH_getshortstr (key=0x7f1425224ef0, t=0x7f14252ba000) at ltable.c:525
    #1  luaH_get (t=0x7f14252ba000, key=key@entry=0x7f142526e870) at ltable.c:572
    #2  0x000000000041ec7d in luaV_finishget (L=L@entry=0x7f14252280a8, t=0x7f142528e9e0, t@entry=0x7f14209a21b0, 
        key=0x7f142526e870, val=val@entry=0x7f14209a21d0, slot=<optimized out>) at lvm.c:196
    #3  0x00000000004206cd in luaV_execute (L=L@entry=0x7f14252280a8) at lvm.c:896
    #4  0x00000000004143af in luaD_call (L=L@entry=0x7f14252280a8, func=<optimized out>, nResults=nResults@entry=-1)
        at ldo.c:499
    #5  0x0000000000411ad2 in lua_pcallk (L=L@entry=0x7f14252280a8, nargs=nargs@entry=1, nresults=nresults@entry=-1, 
        errfunc=errfunc@entry=2, ctx=ctx@entry=2, k=k@entry=0x427600 <finishpcall>) at lapi.c:981
    #6  0x00000000004276bf in luaB_xpcall (L=0x7f14252280a8) at lbaselib.c:441
    #7  0x00000000004140e4 in luaD_precall (L=L@entry=0x7f14252280a8, func=func@entry=0x7f14209a2140, 
        nresults=nresults@entry=-1) at ldo.c:434
    #8  0x00000000004200be in luaV_execute (L=L@entry=0x7f14252280a8) at lvm.c:1146
    #9  0x0000000000413e60 in unroll (L=0x7f14252280a8, ud=<optimized out>) at ldo.c:556
    #10 0x000000000041381f in luaD_rawrunprotected (L=L@entry=0x7f14252280a8, f=f@entry=0x4142b0 <resume>, 
        ud=ud@entry=0x7f1426e1e00c) at ldo.c:142
    #11 0x00000000004144cf in lua_resume (L=L@entry=0x7f14252280a8, from=from@entry=0x7f142523c008, nargs=nargs@entry=5)
        at ldo.c:664
    #12 0x00000000004285c4 in auxresume (L=L@entry=0x7f142523c008, co=co@entry=0x7f14252280a8, narg=5) at lcorolib.c:39
    #13 0x00000000004288d1 in luaB_coresume (L=0x7f142523c008) at lcorolib.c:60
    #14 0x00000000004140e4 in luaD_precall (L=L@entry=0x7f142523c008, func=func@entry=0x7f1425277170, 
        nresults=nresults@entry=-1) at ldo.c:434
    #15 0x00000000004200be in luaV_execute (L=L@entry=0x7f142523c008) at lvm.c:1146
    #16 0x00000000004143af in luaD_call (L=L@entry=0x7f142523c008, func=<optimized out>, nResults=<optimized out>) at ldo.c:499
    #17 0x0000000000414401 in luaD_callnoyield (L=0x7f142523c008, func=<optimized out>, nResults=<optimized out>) at ldo.c:509
    #18 0x000000000041381f in luaD_rawrunprotected (L=L@entry=0x7f142523c008, f=f@entry=0x4102e0 <f_call>, 
        ud=ud@entry=0x7f1426e1e300) at ldo.c:142
    ---Type <return> to continue, or q <return> to quit---return # 输入 return 继续展开 调用栈
    #19 0x000000000041471d in luaD_pcall (L=L@entry=0x7f142523c008, func=func@entry=0x4102e0 <f_call>, 
        u=u@entry=0x7f1426e1e300, old_top=176, ef=<optimized out>) at ldo.c:729
    #20 0x0000000000411a3f in lua_pcallk (L=L@entry=0x7f142523c008, nargs=5, nresults=nresults@entry=-1, 
        errfunc=errfunc@entry=0, ctx=ctx@entry=0, k=k@entry=0x427600 <finishpcall>) at lapi.c:969
    #21 0x0000000000427790 in luaB_pcall (L=0x7f142523c008) at lbaselib.c:424
    #22 0x00000000004140e4 in luaD_precall (L=L@entry=0x7f142523c008, func=func@entry=0x7f1425277090, 
        nresults=nresults@entry=2) at ldo.c:434
    #23 0x00000000004200be in luaV_execute (L=L@entry=0x7f142523c008) at lvm.c:1146
    #24 0x00000000004143af in luaD_call (L=L@entry=0x7f142523c008, func=<optimized out>, nResults=<optimized out>) at ldo.c:499
    #25 0x0000000000414401 in luaD_callnoyield (L=0x7f142523c008, func=<optimized out>, nResults=<optimized out>) at ldo.c:509
    #26 0x000000000041381f in luaD_rawrunprotected (L=L@entry=0x7f142523c008, f=f@entry=0x4102e0 <f_call>, 
        ud=ud@entry=0x7f1426e1e5d0) at ldo.c:142
    #27 0x000000000041471d in luaD_pcall (L=L@entry=0x7f142523c008, func=func@entry=0x4102e0 <f_call>, 
        u=u@entry=0x7f1426e1e5d0, old_top=48, ef=<optimized out>) at ldo.c:729
    #28 0x0000000000411a3f in lua_pcallk (L=0x7f142523c008, nargs=5, nresults=0, errfunc=<optimized out>, ctx=0, k=0x0)
        at lapi.c:969
    #29 0x00007f1422c03388 in ?? ()
    #30 0x00007f1426e1e650 in ?? ()
    #31 0x00007f142a53f836 in __GI___clock_gettime (clock_id=623099912, tp=0xa) at ../sysdeps/unix/clock_gettime.c:115
    #32 0x000000000000002e in ?? ()
    #33 0x000000000000000a in ?? ()
    #34 0x0000000000000001 in ?? ()
    #35 0x000000000000000c in ?? ()
    #36 0x00000000004091c7 in dispatch_message (ctx=0x7f142523b080, msg=0x7f1426e1e6c0) at skynet-src/skynet_server.c:274
    #37 0x0000000000409e94 in skynet_context_message_dispatch (sm=sm@entry=0x7f142a0347e0, q=q@entry=0x7f1425226140, 
        weight=weight@entry=-1) at skynet-src/skynet_server.c:334
    #38 0x000000000040a5dd in thread_worker (p=<optimized out>) at skynet-src/skynet_start.c:162
    ---Type <return> to continue, or q <return> to quit---return # 输入 return 继续展开 调用栈
    #39 0x00007f142af106ba in start_thread (arg=0x7f1426e20700) at pthread_create.c:333
    #40 0x00007f142a5313dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
    (gdb)

  5. 参考资料：

参考资料：