summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2013-08-15Merge tag 'perf-core-for-mingo' of ↵Ingo Molnar
git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux into perf/core Pull perf/core improvements and fixes from Arnaldo Carvalho de Melo: * Allow specifying syscalls in 'perf trace', a la strace. * Simplify symbol filtering by doing it at machine class level, from Adrian Hunter. * Add option to 'perf kvm' to print only events that exceed a specified time duration, from David Ahern. * 'perf sched' improvements, including removing some tracepoints that provide the same information as the PERF_RECORD_{FORK,EXIT} events. * Improve stack trace printing, from David Ahern. * Update documentation with live command, from David Ahern * Fix 'perf test' compile failure on do_sort_something, from David Ahern. * Improve robustness of topology parsing code, from Stephane Eranian. Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> Signed-off-by: Ingo Molnar <mingo@kernel.org>
2013-08-14perf trace: Allow specifying which syscalls to traceArnaldo Carvalho de Melo
Similar to -e in strace, i.e. a comma separated list of syscall names to trace. Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/n/tip-5zku7q5wug3103k1dzn3yy63@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-14perf tools: Improve robustness of topology parsing codeStephane Eranian
This patch improves the robustness of the build_cpu_topo() routine by allowing either the CPU parsing or the thread parsing to fail and yet get perf to produce some topology data which could be useful for the analysis. Without this patch, if the cpu parsing fails, the thread parsing is not attempted vice-versa. Signed-off-by: Stephane Eranian <eranian@google.com> Cc: David Ahern <dsahern@gmail.com> Cc: Ingo Molnar <mingo@elte.hu> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung.kim@lge.com> Cc: Peter Zijlstra <peterz@infradead.org> Link: http://lkml.kernel.org/r/20130814100426.GA3444@quad Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-14perf tests: Fix compile failure on do_sort_somethingDavid Ahern
Commit b55ae0a9 added code-reading.c which fails to compile on Fedora 16 with compiler version: $ gcc --version gcc (GCC) 4.6.3 20120306 (Red Hat 4.6.3-2) Failure message is: tests/code-reading.c: In function ‘do_sort_something’: tests/code-reading.c:305:13: error: stack protector not protecting local variables: variable length buffer [-Werror=stack-protector] cc1: all warnings being treated as errors make: *** [/tmp/junk/tests/code-reading.o] Error 1 make: *** Waiting for unfinished jobs.... v2: as Adrian noticed changed sizeof to ARRAY_SIZE Signed-off-by: David Ahern <dsahern@gmail.com> Acked-by: Adrian Hunter <adrian.hunter@intel.com> Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: Jiri Olsa <jolsa@redhat.com> Link: http://lkml.kernel.org/r/1376454732-83728-1-git-send-email-dsahern@gmail.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12perf tools: Remove filter parameter of thread__find_addr_map()Adrian Hunter
Now that the symbol filter is recorded on the machine there is no need to pass it to thread__find_addr_map(). So remove it. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375961547-30267-9-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12perf tools: Remove filter parameter of thread__find_addr_location()Adrian Hunter
Now that the symbol filter is recorded on the machine there is no need to pass it to thread__find_addr_location(). So remove it. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375961547-30267-8-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12perf tools: Remove filter parameter of perf_event__preprocess_sample()Adrian Hunter
Now that the symbol filter is recorded on the machine there is no need to pass it to perf_event__preprocess_sample(). So remove it. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375961547-30267-7-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12perf annotate: Set the machines symbol filterAdrian Hunter
Take into use the machines symbol filter member. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375961547-30267-6-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12perf mem: Remove unused symbol filter memberAdrian Hunter
Member 'annotate_init' of struct perf_mem is unused. Remove it. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375961547-30267-5-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12perf report: Set the machines symbol filterAdrian Hunter
Take into use the machines' symbol filter member. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375961547-30267-4-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12perf top: Set the machines symbol filterAdrian Hunter
Take into use the machines symbol filter member. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375961547-30267-3-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12perf machine: Add symbol filter to struct machineAdrian Hunter
The symbol filter needs to be applied machine-wide, so add it to struct machine. Currently tools pass the symbol filter as a parameter to various map-related functions. However a need to load a map can occur anywhere in the code, at which point the filter is needed. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375961547-30267-2-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12perf session: Change perf_session__has_traces to actually check for tracepointsDavid Ahern
Any event can have RAW data attribute set. The intent of the function is to determine if the session has tracepoints, so check for the type of each event explicitly. Signed-off-by: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375930261-77273-17-git-send-email-dsahern@gmail.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12perf evsel: Add option to limit stack depth in callchain dumpsDavid Ahern
Option is used by upcoming timehist command. Signed-off-by: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375930261-77273-12-git-send-email-dsahern@gmail.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12perf evsel: Add option to print stack trace on single lineDavid Ahern
Option is used by upcoming timehist command. Signed-off-by: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375930261-77273-11-git-send-email-dsahern@gmail.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12perf tool: Simplify options to perf_evsel__print_ipDavid Ahern
Make print options based on flags. Simplifies addition of more print options which is the subject of upcoming patches. Signed-off-by: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375930261-77273-10-git-send-email-dsahern@gmail.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12perf sched: Remove sched_process_fork tracepointDavid Ahern
The PERF_RECORD_FORK event is already collected as part of the use of cmd_record and those events are analyzed as part of the libperf machinery. Using the fork tracepoint as well just duplicates the event load. Signed-off-by: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375930261-77273-6-git-send-email-dsahern@gmail.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12perf sched: Remove sched_process_exit tracepointDavid Ahern
Event is not needed nor analyzed. Since perf-sched leverages perf-record to capture the sched data, we already capture task events like EXIT. Signed-off-by: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375930261-77273-5-git-send-email-dsahern@gmail.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12perf sched: Remove thread lookup in sample handlerDavid Ahern
Not used in the function, so no sense in doing the lookup here. Thread look up will be done in the timehist command, and no sense in doing it twice. Signed-off-by: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375930261-77273-4-git-send-email-dsahern@gmail.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12perf sched: Simplify arguments to read_eventsDavid Ahern
Destroy argument is not necessary. If session is not returned to caller, then clean it up. Signed-off-by: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375930261-77273-3-git-send-email-dsahern@gmail.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12perf kvm: Update documentation with live commandDavid Ahern
Update perf-kvm documentation with new live subcommand. Add -p/--pid option for perf-kvm-stat-report as well. Signed-off-by: David Ahern <dsahern@gmail.com> Requested-by: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Runzhen Wang <runzhen@linux.vnet.ibm.com> Cc: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com> Link: http://lkml.kernel.org/r/1375926999-75129-2-git-send-email-dsahern@gmail.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12perf kvm: Option to print events that exceed a durationDavid Ahern
This is useful to spot high latency blips. It is normal for HLT reasons to have long exit times, so strip those from the duration check. v2: changed threshold to duration per acme's request Signed-off-by: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Runzhen Wang <runzhen@linux.vnet.ibm.com> Cc: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com> Link: http://lkml.kernel.org/r/1375926999-75129-1-git-send-email-dsahern@gmail.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-12Merge tag 'perf-core-for-mingo' of ↵Ingo Molnar
git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux into perf/core Pull perf/core improvements and fixes from Arnaldo Carvalho de Melo: * Do annotation using /proc/kcore and /proc/kallsyms, removing the need for a vmlinux file kernel assembly annotation. This also improves this use case because vmlinux has just the initial kernel image, not what is actually in use after various code patchings by things like alternatives, etc. From Adrian Hunter. * Add various improvements and fixes to the "vmlinux matches kallsyms" 'perf test' entry, related to the /proc/kcore annotation feature. * Add --initial-delay option to 'perf stat' to skip measuring for the startup phase, from Andi Kleen. * Add perf kvm stat live mode that combines aspects of 'perf kvm stat' record and report, from David Ahern. * Add option to analyze specific VM in perf kvm stat report, from David Ahern. * Do not require /lib/modules/* on a guest, fix from Jason Wessel. * Group leader sampling, that allows just one event in a group to sample while the other events have just its values read, from Jiri Olsa. * Add support for a new modifier "D", which requests that the event, or group of events, be pinned to the PMU, from Michael Ellerman. * Fix segmentation fault on the gtk browser, from Namhyung Kim. Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> Signed-off-by: Ingo Molnar <mingo@kernel.org>
2013-08-07Revert "tools lib lk: Fix for cross build"Joonsoo Kim
This reverts commit 079787f209416416383c74ea5d5044be2d586f5e. Below commit already resolve a cross build problem. I have been noticed this too lately. commit 3c4797d46c14fa0c7cf733a77bd4b28875078b53 Author: Rabin Vincent <rabin@rab.in> Date: Fri May 17 22:27:44 2013 +0200 tools lib lk: Respect CROSS_COMPILE Make lk use CROSS_COMPILE, in order to be able to cross compile perf again. Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Link: http://lkml.kernel.org/r/1373936614-22224-1-git-send-email-iamjoonsoo.kim@lge.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf machine: Do not require /lib/modules/* on a guestJason Wessel
For some types of work loads and special guest environments, you might have a kernel that has no kernel modules. The perf kvm record tool fails instantiate vmlinux maps when the kernel modules directory cannot be opened, even though the kallsyms has been properly processed. This leads to a perf kvm report that has no guest symbols resolved. This patch changes the failure to locate kernel modules to be non-fatal. Signed-off-by: Jason Wessel <jason.wessel@windriver.com> Acked-by: David Ahern <dsahern@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Link: http://lkml.kernel.org/r/1373920073-4874-1-git-send-email-jason.wessel@windriver.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf tests: Add tests of new pinned modifierMichael Ellerman
Add a negative test to test__checkevent_pmu_events() to get lots of coverage of the negative case, ie. when the modifier is not specified. Add a test of a single event, and of the group case. Signed-off-by: Michael Ellerman <michael@ellerman.id.au> Acked-by: Jiri Olsa <jolsa@redhat.com> Acked-by: Namhyung Kim <namhyung@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Link: http://lkml.kernel.org/r/1375795686-4226-2-git-send-email-michael@ellerman.id.au Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf tools: Add support for pinned modifierMichael Ellerman
This commit adds support for a new modifier "D", which requests that the event, or group of events, be pinned to the PMU. The "p" modifier is already taken for precise, and "P" may be used in future to mean "fully precise". So we use "D", which stands for pinneD - and looks like a padlock, or if you're using the ":D" syntax perf smiles at you. This is an oft-requested feature from our HW folks, who want to be able to run a large number of events, but also want 100% accurate results for instructions per cycle. Comparison of results with and without pinning: $ perf stat -e '{cycles,instructions}:D' -e cycles,instructions,... 79,590,480,683 cycles # 0.000 GHz 166,123,716,524 instructions # 2.09 insns per cycle # 0.11 stalled cycles per insn 79,352,134,463 cycles # 0.000 GHz [11.11%] 165,178,301,818 instructions # 2.08 insns per cycle # 0.11 stalled cycles per insn [11.13%] As you can see although perf does a very good job of scaling the values in the non-pinned case, there is some small discrepancy. The patch is fairly straight forward, the one detail is that we need to make sure we only request pinning for the group leader when we have a group. Signed-off-by: Michael Ellerman <michael@ellerman.id.au> Acked-by: Namhyung Kim <namhyung@kernel.org> Acked-by: Jiri Olsa <jolsa@redhat.com> Tested-by: Jiri Olsa <jolsa@redhat.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Link: http://lkml.kernel.org/r/1375795686-4226-1-git-send-email-michael@ellerman.id.au [ Use perf_evsel__is_group_leader instead of open coded equivalent, as suggested by Jiri Olsa ] Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf ui/gtk: Fix segmentation fault on perf_hpp__for_each_format loopNamhyung Kim
The commit 2b8bfa6bb8a7 ("perf tools: Centralize default columns init in perf_hpp__init") moves initialization of common overhead column to perf_hpp__init() but forgot about the gtk code. So the gtk code added the same column to the list twice causing infinite loop when iterating it by perf_hpp__for_each_format loop. When I run perf report --gtk, I can see following messages indefinitely. (perf:11687): Gtk-CRITICAL **: IA__gtk_main_quit: assertion 'main_loops != NULL' failed perf: Segmentation fault Signed-off-by: Namhyung Kim <namhyung@kernel.org> Reviewed-by: Pekka Enberg <penberg@kernel.org> Cc: Andi Kleen <andi@firstfloor.org> Cc: Christoph Hellwig <hch@infradead.org> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Pekka Enberg <penberg@kernel.org> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Link: http://lkml.kernel.org/r/1375766056-19377-2-git-send-email-namhyung@kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf kvm stat report: Add option to analyze specific VMDavid Ahern
Add an option to analyze a specific VM within a data file. This allows the collection of kvm events for all VMs and then analyze data for each VM (or set of VMs) individually. Signed-off-by: David Ahern <dsahern@gmail.com> Reviewed-by: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Runzhen Wang <runzhen@linux.vnet.ibm.com> Cc: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com> Link: http://lkml.kernel.org/r/1375753297-69645-6-git-send-email-dsahern@gmail.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf kvm: Add min and max stats to displayDavid Ahern
Add max and min times for exit events. v2: address Xiao's comment to use get_event function for pulling max and min from stats struct similar to mean and count Signed-off-by: David Ahern <dsahern@gmail.com> Reviewed-by: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Runzhen Wang <runzhen@linux.vnet.ibm.com> Cc: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com> Link: http://lkml.kernel.org/r/1375753297-69645-4-git-send-email-dsahern@gmail.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf kvm: Add live modeDavid Ahern
perf kvm stat currently requires back to back record and report commands to see stats. e.g,. perf kvm stat record -p $pid -- sleep 1 perf kvm stat report This is inconvenvient for on box monitoring of a VM. This patch introduces a 'live' mode that in effect combines the record plus report into one command. e.g., to monitor a single VM: perf kvm stat live -p $pid or all VMs: perf kvm stat live Same stats options for the record+report path work with the live mode. Display rate defaults to 1 second and can be changed using the -d option. v4: - address comments from Xiao -- verify_vcpu check should not look at processors on line for the host, prune configurable options. - set attr->{mmap,comm,task} to 0 - don't need task events so trim events we have to deal with - better control of time for queue event flushing to reduce frequency of "Timestamp below last timeslice flush" failures. v3: updated to use existing tracepoint parsing code v2: removed ABSTIME arg from timerfd_settime as mentioned by Namhyung only call perf_kvm__handle_stdin when poll returns activity. Signed-off-by: David Ahern <dsahern@gmail.com> Reviewed-by: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Runzhen Wang <runzhen@linux.vnet.ibm.com> Cc: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com> Link: http://lkml.kernel.org/r/1375753297-69645-3-git-send-email-dsahern@gmail.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf session: Export queue_event functionDavid Ahern
Taking a lesson from perf-trace and bringing in control of event processing to perf-kvm-stat-live: parse the sample to get access the time leaving just the need to queue it to the ordered samples list. For that the queue_event function needs to be exported. Unexport perf_session__process_event. Signed-off-by: David Ahern <dsahern@gmail.com> Reviewed-by: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Runzhen Wang <runzhen@linux.vnet.ibm.com> Cc: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com> Link: http://lkml.kernel.org/r/1375753297-69645-2-git-send-email-dsahern@gmail.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf annotate browser: Fix typoIngo Molnar
Signed-off-by: Ingo Molnar <mingo@kernel.org> Link: http://lkml.kernel.org/r/20130802111050.GA29126@gmail.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf annotate browser: Improve description of '?' hotkeyArnaldo Carvalho de Melo
The previous description: "Search previous string" is usually associated with the 'N' following a '/string', the opposite of 'n', which is 'Search next string' in the direction established with '/' or '?'. So change it to 'Search string backwards', to clarify that. The 'N' hotkey remains to be implemented with the semantic described above. Cc: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/n/tip-5lw5y15d7vv308xbpm8pqe4g@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf annotate: Add call target name if it is missingAdrian Hunter
The /proc/kcore file has no symbols, so the call target name does not display. Fix by looking up the symbol name if it is on the same map. Reported-by: Arnaldo Carvalho de Melo <acme@redhat.com> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375875537-4509-14-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf annotate: Remove nop at end of annotationAdrian Hunter
When kcore is used for annotation, symbols do not have correct sizes because they come from kallsyms, that has only its start address, with the end address being the next symbol's minus one. That sometimes results in an extra nop being seen after the end of a function. Remove it. Suggested-by: Arnaldo Carvalho de Melo <acme@redhat.com> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375875537-4509-13-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf annotate: Put dso name in symbol annotation titleAdrian Hunter
Currently the symbol name is displayed at the top when displaying symbol annotation. Add to this the dso long name. Suggested-by: Arnaldo Carvalho de Melo <acme@redhat.com> Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375875537-4509-12-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf annotate: Allow disassembly using /proc/kcoreAdrian Hunter
Annotation with /proc/kcore is possible so the logic is adjusted to allow it. The main difference is that /proc/kcore had no symbols so the parsing logic needed a tweak to read jump offsets. The other difference is that objdump cannot always read from kcore. That seems to be a bug with objdump. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375875537-4509-11-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf tests: Add kcore to the object code reading testAdrian Hunter
Make the "object code reading" test attempt to read from kcore. The test uses objdump which struggles with kcore. i.e. doesn't always work, sometimes takes a long time. The test has been made to work around those issues. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375875537-4509-10-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf tests: Adjust the vmlinux symtab matches kallsyms test againAdrian Hunter
The kallsyms maps now may map to kcore and the symbol values now may be file offsets. For comparison with vmlinux the virtual memory address is needed which is obtained by unmapping the symbol value. The "vmlinux symtab matches kallsyms" is adjusted accordingly. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375875537-4509-9-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf symbols: Add support for reading from /proc/kcoreAdrian Hunter
In the absence of vmlinux, perf tools uses kallsyms for symbols. If the user has access, now also map to /proc/kcore. The dso data_type is now set to either DSO_BINARY_TYPE__KCORE or DSO_BINARY_TYPE__GUEST_KCORE as approprite. This patch breaks the "vmlinux symtab matches kallsyms" test. That is fixed in a following patch. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375875537-4509-8-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf tools: Make it possible to read object code from kernel modulesAdrian Hunter
The new "object code reading" test shows that it is not possible to read object code from kernel modules. That is because the mappings do not map to the dsos. This patch fixes that. This involves identifying and flagging relocatable (ELF type ET_REL) files (e.g. kernel modules) for symbol adjustment and updating map__rip_2objdump() accordingly. The kmodule parameter of dso__load_sym() is taken into use and the module map altered to map to the dso. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375875537-4509-7-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf tests: Adjust the vmlinux symtab matches kallsyms testAdrian Hunter
The vmlinux maps now map to the dso and the symbol values are now file offsets. For comparison with kallsyms the virtual memory address is needed which is obtained by unmapping the symbol value. The "vmlinux symtab matches kallsyms" is adjusted accordingly. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375875537-4509-5-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf tools: Make it possible to read object code from vmlinuxAdrian Hunter
The new "object code reading" test shows that it is not possible to read object code from vmlinux. That is because the mappings do not map to the dso. This patch fixes that. A side-effect of changing the kernel map is that the "reloc" offset must be taken into account. As a result of that separate map functions for relocation are no longer needed. Also fixing up the maps to match the symbols no longer makes sense and so is not done. The vmlinux dso data_type is now set to either DSO_BINARY_TYPE__VMLINUX or DSO_BINARY_TYPE__GUEST_VMLINUX as approprite, which enables the correct file name to be determined by dso__binary_type_file(). This patch breaks the "vmlinux symtab matches kallsyms" test. That is fixed in a following patch. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375875537-4509-4-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf symbols: Load kernel maps before usingAdrian Hunter
In order to use kernel maps to read object code, those maps must be adjusted to map to the dso file offset. Because lazy-initialization is used, that is not done until symbols are loaded. However the maps are first used by thread__find_addr_map() before symbols are loaded. So this patch changes thread__find_addr() to "load" kernel maps before using them. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375875537-4509-3-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf tests: Add test for reading object codeAdrian Hunter
Using the information in mmap events, perf tools can read object code associated with sampled addresses. A test is added that compares bytes read by perf with the same bytes read using objdump. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375875537-4509-2-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf symbols: avoid SyS kernel syscall aliasesAdrian Hunter
When removing duplicate symbols, prefer to remove syscall aliases starting with SyS or compat_SyS. A side-effect of that is that it results in slightly improved results for the "vmlinux symtab matches kallsyms" test. Signed-off-by: Adrian Hunter <adrian.hunter@intel.com> Cc: David Ahern <dsahern@gmail.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Namhyung Kim <namhyung@gmail.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375875537-4509-6-git-send-email-adrian.hunter@intel.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf stat: Flush output after each line in interval modeAndi Kleen
When interval mode is outputting to a pipe, each measurement should be flushed individually, so that the reader sees it timely. With a terminal each line is automatically flushed by stdio, but that is disabled with non terminal output. Simply fflush output after each time interval Signed-off-by: Andi Kleen <ak@linux.intel.com> Reviewed-by: Jiri Olsa <jolsa@redhat.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375490473-1503-5-git-send-email-andi@firstfloor.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf stat: Add support for --initial-delay optionAndi Kleen
When measuring workloads the startup phase -- doing page faults, dynamic linking, opening files -- is often very different from the rest of the workload. Especially with smaller kernels and using counter multiplexing this can give significant measurement errors. Multiplexing assumes that the workload is mostly the same over longer periods. But at startup there is typically some spike of activity which is relatively short. If many groups are multiplexing the one group seeing the spike, and which is then scaled up over the time to run all groups, may see a significant error. Also in general it's often not useful to measure the startup, because it is so different from the rest. One way around this is to use interval mode and discard the first sample, but this can be awkward because interval mode doesn't support intervals of less than 100ms, and also a useful interval is not necessarily the same as a useful startup delay. This patch adds a new --initial-delay / -D option to skip measuring for the startup phase. The time can be specified in ms Here's a simple example: perf stat -e page-faults bash -c 'for i in $(seq 100000) ; do true ; done' ... 3,721 page-faults ... If we just wait 20 ms the number of page faults is 1/3 less: perf stat -D 20 -e page-faults bash -c 'for i in $(seq 100000) ; do true ; done' ... 2,823 page-faults ... So we filtered out most of the startup noise from bash. Signed-off-by: Andi Kleen <ak@linux.intel.com> Reviewed-by: Jiri Olsa <jolsa@redhat.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375490473-1503-4-git-send-email-andi@firstfloor.org Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2013-08-07perf evsel: Add support for enabling countersAndi Kleen
Add support for enabling already set up counters by using an ioctl. I share some code with the filter setup. Signed-off-by: Andi Kleen <ak@linux.intel.com> Reviewed-by: Jiri Olsa <jolsa@redhat.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Stephane Eranian <eranian@google.com> Link: http://lkml.kernel.org/r/1375490473-1503-3-git-send-email-andi@firstfloor.org [ Fixed up 'err' variable indentation ] Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>