lttng(1) - Linux man page

Name

lttng -- LTTng 2.0 tracer control command line tool

Synopsis

lttng [OPTIONS] <COMMAND>

Description

The LTTng project aims at providing highly efficient tracing tools for Linux. It's tracers help tracking down performance issues and debugging problems involving multiple concurrent processes and threads. Tracing across multiple systems is also possible.

The lttng command line tool from the lttng-tools package is used to control both kernel and user-space tracing. Every interactions with the tracer should be done by this tool or by the liblttng-ctl provided with the lttng-tools package.

LTTng uses a session daemon (lttng-sessiond(8)), acting as a tracing registry, which permits you to interact with multiple tracers (kernel and user-space) inside the same container, a tracing session. Traces can be gathered from the kernel and/or instrumented applications (lttng-ust(3)). Aggregating and reading those traces is done using the babeltrace(1) text viewer.

In order to trace the kernel, the session daemon needs to be running as root. LTTng provides the use of a tracing group (default: tracing). Whomever is in that group can interact with the root session daemon and thus trace the kernel. Session daemons can co-exist meaning that you can have a session daemon running as Alice that can be used to trace her applications along side with a root daemon or even a Bob daemon. We highly recommend to start the session daemon at boot time for stable and long term tracing.

Every user-space applications instrumented with lttng-ust(3), will automatically register to the session daemon. This feature gives you the ability to list available traceable applications and tracepoints on a per user basis. (See list command).

Options

This program follow the usual GNU command line syntax with long options starting with two dashes. Below is a summary of the available options.

-h, --help: Show summary of possible options and commands.
-v, --verbose: Increase verbosity. Three levels of verbosity are available which are triggered by putting additional v to the option (-vv or -vvv)
-q, --quiet: Suppress all messages (even errors).
-g, --group NAME: Set unix tracing group name. (default: tracing)
-n, --no-sessiond: Don't automatically spawn a session daemon.
--sessiond-path: Set session daemon full binary path.
--list-options: Simple listing of lttng options.
--list-commands: Simple listing of lttng commands.

Commands

add-context

Add context to event(s) and/or channel(s).
A context is basically extra information appended to a channel or event.
For
instance, you could ask the tracer to add the PID information within the
"sched_switch" kernel event. You can also add performance monitoring unit
counters (perf PMU) using the perf kernel API).
For example, this command will add the context information 'prio' and two
perf
counters (hardware branch misses and cache misses), to all events in the
trace
data output:
# lttng add-context -k -t prio -t perf:branch-misses -t perf:cache-misses
Please take a look at the help (-h/--help) for a detailed list of available
contexts.
If no channel and no event is given (-c/-e), the context is added to all
channels (which applies automatically to all events in that channel). Otherwise
the context will be added only to the channel (-c) and/or event (-e) indicated.
If -s, --session is omitted, the session name is taken from the .lttngrc
file.

OPTIONS:

-h, --help
        Show summary of possible options and commands.
-s, --session NAME
        Apply on session name.
-c, --channel NAME
        Apply on channel name.
-e, --event NAME
        Apply on event name.
-k, --kernel
        Apply for the kernel tracer
-u, --userspace
        Apply for the user-space tracer
-t, --type TYPE
        Context type. You can repeat this option on the command line. Please
        use "lttng add-context -h" to list all available types.

calibrate

Quantify LTTng overhead
The LTTng calibrate command can be used to find out the combined average
overhead of the LTTng tracer and the instrumentation mechanisms used. This
overhead can be calibrated in terms of time or using any of the PMU performance
counter available on the system.
For now, the only calibration implemented is that of the kernel function
instrumentation (kretprobes).
* Calibrate kernel function instrumentation
Let's use an example to show this calibration. We use an i7 processor with
4
general-purpose PMU registers. This information is available by issuing dmesg,
looking for "generic registers".
This sequence of commands will gather a trace executing a kretprobe hooked
on
an empty function, gathering PMU counters LLC (Last Level Cache) misses
information (see lttng add-context --help to see the list of available PMU
counters).
# lttng create calibrate-function
# lttng enable-event calibrate --kernel --function lttng_calibrate_kretprobe
# lttng add-context --kernel -t perf:LLC-load-misses -t perf:LLC-store-misses \
                  -t perf:LLC-prefetch-misses
# lttng start
# for a in $(seq 1 10); do \
        lttng calibrate --kernel --function;
  done
# lttng destroy
# babeltrace $(ls -1drt ~/lttng-traces/calibrate-function-* | tail -n 1)
The output from babeltrace can be saved to a text file and opened in a
spreadsheet (e.g. oocalc) to focus on the per-PMU counter delta between
consecutive "calibrate_entry" and "calibrate_return" events. Note that these
counters are per-CPU, so scheduling events would need to be present to account
for migration between CPU. Therefore, for calibration purposes, only events
staying on the same CPU must be considered.
The average result, for the i7, on 10 samples:
                          Average     Std.Dev.
perf_LLC_load_misses:       5.0       0.577
perf_LLC_store_misses:      1.6       0.516
perf_LLC_prefetch_misses:   9.0      14.742
As we can notice, the load and store misses are relatively stable across
runs
(their standard deviation is relatively low) compared to the prefetch misses.
We can conclude from this information that LLC load and store misses can
be
accounted for quite precisely, but prefetches within a function seems to
behave
too erratically (not much causality link between the code executed and
the CPU
prefetch activity) to be accounted for.

OPTIONS:

-h, --help
        Show summary of possible options and commands.
-k, --kernel
        Apply for the kernel tracer
-u, --userspace
        Apply for the user-space tracer
--function
        Dynamic function entry/return probe (default)

create [OPTIONS] [NAME]

Create tracing session.
A tracing session contains channel(s) which contains event(s). It is domain
agnostic meaning that you can enable channels and events for either the
user-space tracer and/or the kernel tracer. It acts like a container
aggregating multiple tracing sources.
On creation, a .lttngrc file is created in your $HOME directory
containing the current session name. If NAME is omitted, a session name
is
automatically created having this form: 'auto-yyyymmdd-hhmmss'.
If no -o, --output is specified, the traces will be written in
$HOME/lttng-traces.

OPTIONS:

-h, --help
        Show summary of possible options and commands.
--list-options
        Simple listing of options
-o, --output PATH
        Specify output path for traces

destroy [OPTIONS] [NAME]

Teardown tracing session
Free memory on the session daemon and tracer side. It's gone!
If NAME is omitted, the session name is taken from the .lttngrc file.

OPTIONS:

-h, --help
        Show summary of possible options and commands.
--list-options
        Simple listing of options

enable-channel NAME[,NAME2,...] [-k|-u] [OPTIONS]

Enable tracing channel
To enable event, you must first enable a channel which contains event(s).
If -s, --session is omitted, the session name is taken from the .lttngrc
file.

OPTIONS:

-h, --help
        Show this help
--list-options
        Simple listing of options
-s, --session
        Apply on session name
-k, --kernel
        Apply to the kernel tracer
-u, --userspace
        Apply to the user-space tracer
--discard
        Discard event when subbuffers are full (default)
--overwrite
        Flight recorder mode : overwrites events when subbuffers are full
--subbuf-size
        Subbuffer size in bytes (default: 4096, kernel default: 262144)
--num-subbuf
        Number of subbuffers (default: 4)
        Needs to be a power of 2 for kernel and ust tracers
--switch-timer
        Switch subbuffer timer interval in usec (default: 0)
        Needs to be a power of 2 for kernel and ust tracers
--read-timer
        Read timer interval in usec (default: 200)

enable-event NAME[,NAME2,...] [-k|-u] [OPTIONS]

Enable tracing event
A tracing event is always assigned to a channel. If -c, --channel is
omitted, a default channel named 'channel0' is created and the event is
added to it. For the user-space tracer, using -a, --all is the same as
using the wildcard "*".
If -s, --session is omitted, the session name is taken from the .lttngrc
file.

OPTIONS:

-h, --help
        Show summary of possible options and commands.
--list-options
        Simple listing of options
-s, --session
        Apply on session name
-c, --channel
        Apply on channel name
-a, --all
        Enable all tracepoints and syscalls
-k, --kernel
        Apply for the kernel tracer
-u, --userspace
        Apply for the user-space tracer
--tracepoint
        Tracepoint event (default)
        - userspace tracer supports wildcards at end of string. Don't forget
to
        quote to deal with bash expansion.
        e.g.:
        "*"
        "app_component:na*"
--loglevel
        Tracepoint loglevel
--probe [addr | symbol | symbol+offset]
        Dynamic probe. Addr and offset can be octal (0NNN...), decimal (NNN...)
        or hexadecimal (0xNNN...)
--function [addr | symbol | symbol+offset]
        Dynamic function entry/return probe. Addr and offset can be octal
        (0NNN...), decimal (NNN...) or hexadecimal (0xNNN...)
--syscall
        System call event
        Enabling syscalls tracing (kernel tracer), you will not be able
to disable them
        with disable-event. This is a known limitation. You can disable the
entire
        channel to do the trick.

disable-channel NAME[,NAME2,...] [-k|-u] [OPTIONS]

Disable tracing channel
Disabling a channel makes all event(s) in that channel to stop tracing.
You can
enable it back by calling lttng enable-channel NAME again.
If -s, --session is omitted, the session name is taken from the .lttngrc
file.

OPTIONS:

-h, --help
        Show summary of possible options and commands.
--list-options
        Simple listing of options
-s, --session
        Apply on session name
-k, --kernel
        Apply for the kernel tracer
-u, --userspace
        Apply for the user-space tracer

disable-event NAME[,NAME2,...] [-k|-u] [OPTIONS]

Disable tracing event
The event, once disabled, can be re-enabled by calling lttng enable-event
NAME again.
If -s, --session is omitted, the session name is taken from the .lttngrc
file.

OPTIONS:

-h, --help
        Show summary of possible options and commands.
--list-options
        Simple listing of options
-s, --session
        Apply on session name
-k, --kernel
        Apply for the kernel tracer
-u, --userspace
        Apply for the user-space tracer

list [-k|-u] [SESSION [SESSION_OPTIONS]]

List tracing session information.
With no arguments, it will list available tracing session(s).
With the session name, it will display the details of the session including
the trace file path, the associated channels and their state (activated
and deactivated), the activated events and more.
With -k alone, it will list all available kernel events (except the system
calls events).
With -u alone, it will list all available user-space events from registered
applications. Here is an example of 'lttng list -u':
PID: 7448 - Name: /tmp/lttng-ust/tests/hello/.libs/lt-hello
      ust_tests_hello:tptest_sighandler (type: tracepoint)
      ust_tests_hello:tptest (type: tracepoint)
You can now enable any event listed by using the name :
ust_tests_hello:tptest.

OPTIONS:

-h, --help
        Show summary of possible options and commands.
--list-options
        Simple listing of options
-k, --kernel
        Select kernel domain
-u, --userspace
        Select user-space domain.
Session options:
-c, --channel NAME
        List details of a channel
-d, --domain
        List available domain(s)

set-session NAME

Set current session name
Will change the session name in the .lttngrc file.

OPTIONS:

-h, --help
        Show summary of possible options and commands.
--list-options
        Simple listing of options

start [OPTIONS] [NAME]

Start tracing
It will start tracing for all tracers for a specific tracing session.
If NAME is omitted, the session name is taken from the .lttngrc file.

OPTIONS:

-h, --help
        Show summary of possible options and commands.
--list-options
        Simple listing of options

stop [OPTIONS] [NAME]

Stop tracing
It will stop tracing for all tracers for a specific tracing session.
If NAME is omitted, the session name is taken from the .lttngrc file.

OPTIONS:

-h, --help
        Show summary of possible options and commands.
--list-options
        Simple listing of options

version

Show version information

OPTIONS:

-h, --help
        Show summary of possible options and commands.
--list-options
        Simple listing of options

view [SESSION_NAME] [OPTIONS]

View traces of a tracing session
By default, the babeltrace viewer will be used for text viewing.
If SESSION_NAME is omitted, the session name is taken from the .lttngrc
file.

OPTIONS:

-h, --help
        Show this help
--list-options
        Simple listing of options
-t, --trace-path PATH
        Trace directory path for the viewer
-e, --viewer CMD
        Specify viewer and/or options to use
        This will completely override the default viewers so
        please make sure to specify the full command. The trace
        directory path of the session will be appended at the end
        to the arguments

Exit Values

0: Success
1: Command error
2: Undefined command
3: Fatal error
4: Command warning
16: No session found by the name given
18: Error in session
creation
21: Error in application(s) listing
28: Session name already exists
33: Kernel tracer unavailable
35: Kernel event exists
37: Kernel channel exists
38: Kernel channel creation failed
39: Kernel channel not found
40: Kernel
channel disable failed
41: Kernel channel enable failed
42: Kernel context
failed
43: Kernel enable event failed
44: Kernel disable event failed
53: Kernel
listing events failed
60: UST channel disable failed
61: UST channel enable
failed
62: UST adding context failed
63: UST event enable failed
64: UST event
disable failed
66: UST start failed
67: UST stop failed
75: UST event exists
76: UST event not found
77: UST context exists
78: UST invalid context
79: Tracing the kernel requires a root lttng-sessiond daemon and "tracing" group user membership.
80: Tracing already started
81: Tracing already stopped
98: No
UST consumer detected
99: No Kernel consumer detected
100: Event already enabled with different loglevel

Environment Variables

Note that all command line options override environment variables.

LTTNG_SESSIOND_PATH: Allows one to specify the full session daemon binary path to lttng command line tool. You can also use --sessiond-path option having the same effect.

Bugs

No show stopper bugs are known yet in this version.

If you encounter any issues or usability problem, please report it on our mailing list <lttng-dev@lists.lttng.org> to help improve this project.

Credits

lttng is distributed under the GNU General Public License version 2. See the file COPYING for details.

A Web site is available at http://lttng.org for more information on the LTTng project.

You can also find our git tree at http://git.lttng.org.

Mailing lists for support and development: <lttng-dev@lists.lttng.org>.

You can find us on IRC server irc.oftc.net (OFTC) in #lttng.

Thanks

Thanks to Yannick Brosseau without whom this project would never have been so lean and mean! Also thanks to the Ericsson teams working on tracing which helped us greatly with detailed bug reports and unusual test cases.

Thanks to our beloved packager Alexandre Montplaisir-Goncalves (Ubuntu and PPA maintainer) and Jon Bernard for our Debian packages.

Special thanks to Michel Dagenais and the DORSAL laboratory at Polytechnique de Montreal for the LTTng journey.

Authors

lttng-tools was originally written by Mathieu Desnoyers, Julien Desfossez and David Goulet. More people have since contributed to it. It is currently maintained by David Goulet <dgoulet@efficios.com>.

Referenced By

lttng-gen-tp(1)

lttng(1) - Linux man page

Name

Synopsis

Description

Options

Commands

Exit Values

Environment Variables

See Also

Bugs

Credits

Thanks

Authors

Referenced By