Server - Debugging with GDB
Description: Debugging with GDB

Next: Remote Configuration, Previous: File Transfer, Up: Remote Debugging

### 20.3 Using the `gdbserver` Program

`gdbserver` is a control program for Unix-like systems, which allows you to connect your program with a remote gdb via `target remote`—but without linking in the usual debugging stub.

`gdbserver` is not a complete replacement for the debugging stubs, because it requires essentially the same operating-system facilities that gdb itself does. In fact, a system that can run `gdbserver` to connect to a remote gdb could also run gdb locally! `gdbserver` is sometimes useful nevertheless, because it is a much smaller program than gdb itself. It is also easier to port than all of gdb, so you may be able to get started more quickly on a new system by using `gdbserver`. Finally, if you develop code for real-time systems, you may find that the tradeoffs involved in real-time operation make it more convenient to do as much development work as possible on another system, for example by cross-compiling. You can use `gdbserver` to make a similar choice for debugging.

gdb and `gdbserver` communicate via either a serial line or a TCP connection, using the standard gdb remote serial protocol.

> Warning: `gdbserver` does not have any built-in security. Do not run `gdbserver` connected to any public network; a gdb connection to `gdbserver` provides access to the target system with the same privileges as the user running `gdbserver`.

#### 20.3.1 Running `gdbserver`

Run `gdbserver` on the target system. You need a copy of the program you want to debug, including any libraries it requires. `gdbserver` does not need your program's symbol table, so you can strip the program if necessary to save space. gdb on the host system does all the symbol handling.

To use the server, you must tell it how to communicate with gdb; the name of your program; and the arguments for your program. The usual syntax is:

         target> gdbserver comm program [ args ... ]

comm is either a device name (to use a serial line), or a TCP hostname and portnumber, or `-` or `stdio` to use stdin/stdout of `gdbserver`. For example, to debug Emacs with the argument ‘`foo.txt`’ and communicate with gdb over the serial port `/dev/com1`:

         target> gdbserver /dev/com1 emacs foo.txt

`gdbserver` waits passively for the host gdb to communicate with it.

To use a TCP connection instead of a serial line:

         target> gdbserver host:2345 emacs foo.txt

The only difference from the previous example is the first argument, specifying that you are communicating with the host gdb via TCP. The ‘`host:2345`’ argument means that `gdbserver` is to expect a TCP connection from machine ‘`host`’ to local TCP port 2345. (Currently, the ‘`host`’ part is ignored.) You can choose any number you want for the port number as long as it does not conflict with any TCP ports already in use on the target system (for example, `23` is reserved for `telnet`).1 You must use the same port number with the host gdb `target remote` command.

The `stdio` connection is useful when starting `gdbserver` with ssh:

         (gdb) target remote | ssh -T hostname gdbserver - hello

The ‘`-T`’ option to ssh is provided because we don't need a remote pty, and we don't want escape-character handling. Ssh does this by default when a command is provided, the flag is provided to make it explicit. You could elide it if you want to.

Programs started with stdio-connected gdbserver have `/dev/null` for `stdin`, and `stdout`,`stderr` are sent back to gdb for display through a pipe connected to gdbserver. Both `stdout` and `stderr` use the same pipe.

##### 20.3.1.1 Attaching to a Running Program

On some targets, `gdbserver` can also attach to running programs. This is accomplished via the `--attach` argument. The syntax is:

         target> gdbserver --attach comm pid

pid is the process ID of a currently running process. It isn't necessary to point `gdbserver` at a binary for the running process.

You can debug processes by name instead of process ID if your target has the `pidof` utility:

         target> gdbserver --attach comm `pidof program`

In case more than one copy of program is running, or program has multiple threads, most versions of `pidof` support the `-s` option to only return the first process ID.

##### 20.3.1.2 Multi-Process Mode for `gdbserver`

When you connect to `gdbserver` using `target remote`, `gdbserver` debugs the specified program only once. When the program exits, or you detach from it, gdb closes the connection and `gdbserver` exits.

If you connect using `target extended-remote`, `gdbserver` enters multi-process mode. When the debugged program exits, or you detach from it, gdb stays connected to `gdbserver` even though no program is running. The `run` and `attach` commands instruct `gdbserver` to run or attach to a new program. The `run` command uses `set remote exec-file` (see set remote exec-file) to select the program to run. Command line arguments are supported, except for wildcard expansion and I/O redirection (see Arguments).

To start `gdbserver` without supplying an initial command to run or process ID to attach, use the `--multi` command line option. Then you can connect using `target extended-remote` and start the program you want to debug.

In multi-process mode `gdbserver` does not automatically exit unless you use the option `--once`. You can terminate it by using `monitor exit` (see Monitor Commands for gdbserver). Note that the conditions under which `gdbserver` terminates depend on how gdb connects to it (`target remote` or `target extended-remote`). The `--multi` option to `gdbserver` has no influence on that.

##### 20.3.1.3 TCP port allocation lifecycle of `gdbserver`

This section applies only when `gdbserver` is run to listen on a TCP port.

`gdbserver` normally terminates after all of its debugged processes have terminated in `target remote` mode. On the other hand, for `target extended-remote`, `gdbserver` stays running even with no processes left. gdb normally terminates the spawned debugged process on its exit, which normally also terminates `gdbserver` in the `target remote` mode. Therefore, when the connection drops unexpectedly, and gdb cannot ask `gdbserver` to kill its debugged processes, `gdbserver` stays running even in the `target remote` mode.

When `gdbserver` stays running, gdb can connect to it again later. Such reconnecting is useful for features like disconnected tracing. For completeness, at most one gdb can be connected at a time.

By default, `gdbserver` keeps the listening TCP port open, so that additional connections are possible. However, if you start `gdbserver` with the `--once` option, it will stop listening for any further connection attempts after connecting to the first gdb session. This means no further connections to `gdbserver` will be possible after the first one. It also means `gdbserver` will terminate after the first connection with remote gdb has closed, even for unexpectedly closed connections and even in the `target extended-remote` mode. The `--once` option allows reusing the same port number for connecting to multiple instances of `gdbserver` running on the same host, since each instance closes its port after the first connection.

##### 20.3.1.4 Other Command-Line Arguments for `gdbserver`

The `--debug` option tells `gdbserver` to display extra status information about the debugging process. The `--remote-debug` option tells `gdbserver` to display remote protocol debug output. These options are intended for `gdbserver` development and for bug reports to the developers.

The `--wrapper` option specifies a wrapper to launch programs for debugging. The option should be followed by the name of the wrapper, then any command-line arguments to pass to the wrapper, then `--` indicating the end of the wrapper arguments.

`gdbserver` runs the specified wrapper program with a combined command line including the wrapper arguments, then the name of the program to debug, then any arguments to the program. The wrapper runs until it executes your program, and then gdb gains control.

You can use any program that eventually calls `execve` with its arguments as a wrapper. Several standard Unix utilities do this, e.g. `env` and `nohup`. Any Unix shell script ending with `exec "$@"` will also work.

For example, you can use `env` to pass an environment variable to the debugged program, without setting the variable in `gdbserver`'s environment:

         $ gdbserver --wrapper env LD_PRELOAD=libtest.so -- :2222 ./testprog

#### 20.3.2 Connecting to `gdbserver`

Run gdb on the host system.

First make sure you have the necessary symbol files. Load symbols for your application using the `file` command before you connect. Use `set sysroot` to locate target libraries (unless your gdb was compiled with the correct sysroot using `--with-sysroot`).

The symbol file and target libraries must exactly match the executable and libraries on the target, with one exception: the files on the host system should not be stripped, even if the files on the target system are. Mismatched or missing files will lead to confusing results during debugging. On gnu/Linux targets, mismatched or missing files may also prevent `gdbserver` from debugging multi-threaded programs.

Connect to your target (see Connecting to a Remote Target). For TCP connections, you must start up `gdbserver` prior to using the `target remote` command. Otherwise you may get an error whose text depends on the host system, but which usually looks something like ‘`Connection refused`’. Don't use the `load` command in gdb when using `gdbserver`, since the program is already on the target.

#### 20.3.3 Monitor Commands for `gdbserver`

During a gdb session using `gdbserver`, you can use the `monitor` command to send special requests to `gdbserver`. Here are the available commands.

`monitor help`
  List the available monitor commands.
`monitor set debug 0`
`monitor set debug 1`
  Disable or enable general debugging messages.
`monitor set remote-debug 0`
`monitor set remote-debug 1`
  Disable or enable specific debugging messages associated with the remote protocol (see Remote Protocol).
`monitor set libthread-db-search-path [PATH]`
  When this command is issued, path is a colon-separated list of directories to search for `libthread_db` (see set libthread-db-search-path). If you omit path, ‘`libthread-db-search-path`’ will be reset to its default value.

  The special entry ‘`$pdir`’ for ‘`libthread-db-search-path`’ is not supported in `gdbserver`.

`monitor exit`
  Tell gdbserver to exit immediately. This command should be followed by `disconnect` to close the debugging session. `gdbserver` will detach from any attached processes and kill any processes it created. Use `monitor exit` to terminate `gdbserver` at the end of a multi-process mode debug session.

#### 20.3.4 Tracepoints support in `gdbserver`

On some targets, `gdbserver` supports tracepoints, fast tracepoints and static tracepoints.

For fast or static tracepoints to work, a special library called the in-process agent (IPA), must be loaded in the inferior process. This library is built and distributed as an integral part of `gdbserver`. In addition, support for static tracepoints requires building the in-process agent library with static tracepoints support. At present, the UST (LTTng Userspace Tracer, http://lttng.org/ust) tracing engine is supported. This support is automatically available if UST development headers are found in the standard include path when `gdbserver` is built, or if `gdbserver` was explicitly configured using `--with-ust` to point at such headers. You can explicitly disable the support using `--with-ust=no`.

There are several ways to load the in-process agent in your program:

`Specifying it as dependency at link time`
  You can link your program dynamically with the in-process agent library. On most systems, this is accomplished by adding `-linproctrace` to the link command.
`Using the system's preloading mechanisms`
  You can force loading the in-process agent at startup time by using your system's support for preloading shared libraries. Many Unixes support the concept of preloading user defined libraries. In most cases, you do that by specifying `LD_PRELOAD=libinproctrace.so` in the environment. See also the description of `gdbserver`'s `--wrapper` command line option.
`Using` gdb`to force loading the agent at run time`
  On some systems, you can force the inferior to load a shared library, by calling a dynamic loader function in the inferior that takes care of dynamically looking up and loading a shared library. On most Unix systems, the function is `dlopen`. You'll use the `call` command for that. For example:

                (gdb) call dlopen ("libinproctrace.so", ...)

  Note that on most Unix systems, for the `dlopen` function to be available, the program needs to be linked with `-ldl`.

On systems that have a userspace dynamic loader, like most Unix systems, when you connect to `gdbserver` using `target remote`, you'll find that the program is stopped at the dynamic loader's entry point, and no shared library has been loaded in the program's address space yet, including the in-process agent. In that case, before being able to use any of the fast or static tracepoints features, you need to let the loader run and load the shared libraries. The simplest way to do that is to run the program to the main procedure. E.g., if debugging a C or C`++` program, start `gdbserver` like so:

         $ gdbserver :9999 myprogram

Start GDB and connect to `gdbserver` like so, and run to main:

         $ gdb myprogram
         (gdb) target remote myhost:9999
         0x00007f215893ba60 in ?? () from /lib64/ld-linux-x86-64.so.2
         (gdb) b main
         (gdb) continue

The in-process tracing agent library should now be loaded into the process; you can confirm it with the `info sharedlibrary` command, which will list `libinproctrace.so` as loaded in the process. You are now ready to install fast tracepoints, list static tracepoint markers, probe static tracepoints markers, and start tracing.

#### Footnotes

[1] If you choose a port number that conflicts with another service, `gdbserver` prints an error message and exits.