summaryrefslogtreecommitdiff
path: root/arch/powerpc/kernel/vdso64
AgeCommit message (Collapse)Author
2013-04-23powerpc: Add VDSO version of timeAdhemerval Zanella
On 04/18/2013 07:38 PM, Anton Blanchard wrote: > Since you are only reading one long you shouldn't need to check the > update count and loop, you will always see a consistent value. The > system call version of time() just does an unprotected load for example. Fixed. > With the above change and with Michael's comments covered (decent > changelog entry and Signed-off-by): > > Acked-by: Anton Blanchard <anton@samba.org> Thanks for the review, below the updated patch: From: Adhemerval Zanella <azanella@linux.vnet.ibm.com> This patch implement the time syscall as vDSO. The performance speedups are: Baseline PPC32: 380 nsec Baseline PPC64: 350 nsec vdso PPC32: 20 nsec vsdo PPC64: 20 nsec Tested on 64 bit build with both 32 bit and 64 bit userland. Acked-by: Anton Blanchard <anton@samba.org> Signed-off-by: Adhemerval Zanella <azanella@linux.vnet.ibm.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2012-07-11powerpc: Add VDSO version of getcpuAnton Blanchard
We have a request for a fast method of getting CPU and NUMA node IDs from userspace. This patch implements a getcpu VDSO function, similar to x86. Ben suggested we use SPRG3 which is userspace readable. SPRG3 can be modified by a KVM guest, so we save the SPRG3 value in the paca and restore it when transitioning from the guest to the host. I have a glibc patch that implements sched_getcpu on top of this. Testing on a POWER7: baseline: 538 cycles vdso: 30 cycles Signed-off-by: Anton Blanchard <anton@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2011-03-31Fix common misspellingsLucas De Marchi
Fixes generated by 'codespell' and manually reviewed. Signed-off-by: Lucas De Marchi <lucas.demarchi@profusion.mobi>
2010-10-13powerpc/Makefiles: Change to new flag variablesmatt mooney
Replace EXTRA_CFLAGS with ccflags-y and EXTRA_AFLAGS with asflags-y. Signed-off-by: matt mooney <mfm@muteddisk.com> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2010-07-09powerpc: Rework VDSO gettimeofday to prevent time going backwardsPaul Mackerras
Currently it is possible for userspace to see the result of gettimeofday() going backwards by 1 microsecond, assuming that userspace is using the gettimeofday() in the VDSO. The VDSO gettimeofday() algorithm computes the time in "xsecs", which are units of 2^-20 seconds, or approximately 0.954 microseconds, using the algorithm now = (timebase - tb_orig_stamp) * tb_to_xs + stamp_xsec and then converts the time in xsecs to seconds and microseconds. The kernel updates the tb_orig_stamp and stamp_xsec values every tick in update_vsyscall(). If the length of the tick is not an integer number of xsecs, then some precision is lost in converting the current time to xsecs. For example, with CONFIG_HZ=1000, the tick is 1ms long, which is 1048.576 xsecs. That means that stamp_xsec will advance by either 1048 or 1049 on each tick. With the right conditions, it is possible for userspace to get (timebase - tb_orig_stamp) * tb_to_xs being 1049 if the kernel is slightly late in updating the vdso_datapage, and then for stamp_xsec to advance by 1048 when the kernel does update it, and for userspace to then see (timebase - tb_orig_stamp) * tb_to_xs being zero due to integer truncation. The result is that time appears to go backwards by 1 microsecond. To fix this we change the VDSO gettimeofday to use a new field in the VDSO datapage which stores the nanoseconds part of the time as a fractional number of seconds in a 0.32 binary fraction format. (Or put another way, as a 32-bit number in units of 0.23283 ns.) This is convenient because we can use the mulhwu instruction to convert it to either microseconds or nanoseconds. Since it turns out that computing the time of day using this new field is simpler than either using stamp_xsec (as gettimeofday does) or stamp_xtime.tv_nsec (as clock_gettime does), this converts both gettimeofday and clock_gettime to use the new field. The existing __do_get_tspec function is converted to use the new field and take a parameter in r7 that indicates the desired resolution, 1,000,000 for microseconds or 1,000,000,000 for nanoseconds. The __do_get_xsec function is then unused and is deleted. The new algorithm is now = ((timebase - tb_orig_stamp) << 12) * tb_to_xs + (stamp_xtime_seconds << 32) + stamp_sec_fraction with 'now' in units of 2^-32 seconds. That is then converted to seconds and either microseconds or nanoseconds with seconds = now >> 32 partseconds = ((now & 0xffffffff) * resolution) >> 32 The 32-bit VDSO code also makes a further simplification: it ignores the bottom 32 bits of the tb_to_xs value, which is a 0.64 format binary fraction. Doing so gets rid of 4 multiply instructions. Assuming a timebase frequency of 1GHz or less and an update interval of no more than 10ms, the upper 32 bits of tb_to_xs will be at least 4503599, so the error from ignoring the low 32 bits will be at most 2.2ns, which is more than an order of magnitude less than the time taken to do gettimeofday or clock_gettime on our fastest processors, so there is no possibility of seeing inconsistent values due to this. This also moves update_gtod() down next to its only caller, and makes update_vsyscall use the time passed in via the wall_time argument rather than accessing xtime directly. At present, wall_time always points to xtime, but that could change in future. Signed-off-by: Paul Mackerras <paulus@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2009-09-21Use macros for .data.page_aligned section.Tim Abbott
This patch changes the remaining direct references to .data.page_aligned in C and assembly code to use the macros in include/linux/linkage.h. Signed-off-by: Tim Abbott <tabbott@ksplice.com> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Ingo Molnar <mingo@redhat.com> Cc: H. Peter Anvin <hpa@zytor.com> Cc: Haavard Skinnemoen <hskinnemoen@atmel.com> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Paul Mackerras <paulus@samba.org> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
2009-09-20kbuild: rename ld-option to cc-ldoptionSam Ravnborg
ld-option is misnamed as it test options to gcc, not to ld. Renamed it to reflect this. Cc: Andi Kleen <andi@firstfloor.org> Cc: Roland McGrath <roland@redhat.com> Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
2009-08-20powerpc: Enable GCOVMichael Ellerman
Make it possible to enable GCOV code coverage measurement on powerpc. Lightly tested on 64-bit, seems to work as expected. Signed-off-by: Michael Ellerman <michael@ellerman.id.au> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2008-12-21powerpc/mm: Introduce MMU featuresBenjamin Herrenschmidt
We're soon running out of CPU features and I need to add some new ones for various MMU related bits, so this patch separates the MMU features from the CPU features. I moved over the 32-bit MMU related ones, added base features for MMU type families, but didn't move over any 64-bit only feature yet. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Acked-by: Kumar Gala <galak@kernel.crashing.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2008-11-05powerpc: Improve resolution of VDSO clock_gettimePaul Mackerras
Currently the clock_gettime implementation in the VDSO produces a result with microsecond resolution for the cases that are handled without a system call, i.e. CLOCK_REALTIME and CLOCK_MONOTONIC. The nanoseconds field of the result is obtained by computing a microseconds value and multiplying by 1000. This changes the code in the VDSO to do the computation for clock_gettime with nanosecond resolution. That means that the resolution of the result will ultimately depend on the timebase frequency. Because the timestamp in the VDSO datapage (stamp_xsec, the real time corresponding to the timebase count in tb_orig_stamp) is in units of 2^-20 seconds, it doesn't have sufficient resolution for computing a result with nanosecond resolution. Therefore this adds a copy of xtime to the VDSO datapage and updates it in update_gtod() along with the other time-related fields. Signed-off-by: Paul Mackerras <paulus@samba.org>
2008-07-03powerpc: Fixup lwsync at runtimeKumar Gala
To allow for a single kernel image on e500 v1/v2/mc we need to fixup lwsync at runtime. On e500v1/v2 lwsync causes an illop so we need to patch up the code. We default to 'sync' since that is always safe and if the cpu is capable we will replace 'sync' with 'lwsync'. We introduce CPU_FTR_LWSYNC as a way to determine at runtime if this is needed. This flag could be moved elsewhere since we dont really use it for the normal CPU_FTR purpose. Finally we only store the relative offset in the fixup section to keep it as small as possible rather than using a full fixup_entry. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2008-07-01powerpc: Introduce infrastructure for feature sections with alternativesMichael Ellerman
The current feature section logic only supports nop'ing out code, this means if you want to choose at runtime between instruction sequences, one or both cases will have to execute the nop'ed out contents of the other section, eg: BEGIN_FTR_SECTION or 1,1,1 END_FTR_SECTION_IFSET(FOO) BEGIN_FTR_SECTION or 2,2,2 END_FTR_SECTION_IFCLR(FOO) and the resulting code will be either, or 1,1,1 nop or, nop or 2,2,2 For small code segments this is fine, but for larger code blocks and in performance criticial code segments, it would be nice to avoid the nops. This commit starts to implement logic to allow the following: BEGIN_FTR_SECTION or 1,1,1 FTR_SECTION_ELSE or 2,2,2 ALT_FTR_SECTION_END_IFSET(FOO) and the resulting code will be: or 1,1,1 or, or 2,2,2 We achieve this by extending the existing FTR macros. The current feature section semantic just becomes a special case, ie. if the else case is empty we nop out the default case. The key limitation is that the size of the else case must be less than or equal to the size of the default case. If the else case is smaller the remainder of the section is nop'ed. We let the linker put the else case code in with the rest of the text, so that relative branches from the else case are more likley to link, this has the disadvantage that we can't free the unused else cases. This commit introduces the required macro and linker script changes, but does not enable the patching of the alternative sections. We also need to update two hand-made section entries in reg.h and timex.h Signed-off-by: Michael Ellerman <michael@ellerman.id.au> Signed-off-by: Paul Mackerras <paulus@samba.org>
2008-05-23[POWERPC] Tweak VDSO linker script to avoid upsetting old binutilsRoland McGrath
This works around bugs in older binutils' objcopy. The placement of these sections does not really matter, but it confused the buggy old BFD libraries. Signed-off-by: Roland McGrath <roland@redhat.com> Signed-off-by: Paul Mackerras <paulus@samba.org>
2007-11-20[POWERPC] vdso: Fixes for cache block sizesOlof Johansson
The current VDSO implementation is hardcoded to 128 byte cache blocks, which are only used on IBM's 64-bit processors. Convert it to get the cache block sizes out of vdso_data instead, similar to how the ppc64 in-kernel cache flush does it. Signed-off-by: Olof Johansson <olof@lixom.net> Signed-off-by: Paul Mackerras <paulus@samba.org>
2007-10-17[POWERPC] Fix 64 bits vDSO DWARF info for CR registerBenjamin Herrenschmidt
The current DWARF info for CR are incorrect, causing the gcc unwinder to go to lunch if we take a segfault in the vdso. This fixes it. Problem identified by Andrew Haley, and fix provided by Jakub Jelinek (thanks !). Unfortunately, a bug in gcc cause it to not quite work either, but that is being fixed separately with something around the lines of: linux-unwind.h: fs->regs.reg[R_CR2].loc.offset = (long) &regs->ccr - new_cfa; + /* CR? regs are just 32-bit and PPC is big-endian. */ + fs->regs.reg[R_CR2].loc.offset += sizeof (long) - 4; (According to Jakub) Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2007-10-16powerpc64 vDSO: linker script indentationRoland McGrath
This cleans up the formatting in the vDSO linker script, mostly just the use of whitespace. It's intended to approximate the kernel standard conventions for indenting C, treating elements of the linker script about like initialized variable definitions. Signed-off-by: Roland McGrath <roland@redhat.com> Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Sam Ravnborg <sam@ravnborg.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-08[POWERPC] Update .gitignore for new vdso generated filesKumar Gala
We now generate vdso[32,64].so.dbg as part of the build so add them to .gitignore Signed-off-by: Kumar Gala <galak@kernel.crashing.org>
2007-10-03[POWERPC] powerpc vDSO: install unstripped copies on diskRoland McGrath
This keeps an unstripped copy of the vDSO images built before they are stripped and embedded in the kernel. The unstripped copies get installed in $(MODLIB)/vdso/ by "make install". These files can be useful when they contain source-level debugging information. Signed-off-by: Roland McGrath <roland@redhat.com> Cc: Sam Ravnborg <sam@ravnborg.org> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2007-06-30Fix VDSO gettimeofday() when called with NULL struct timeval.Will Schmidt
The vdso64 portion of patch 74609f4536f2b8fd6a48381bbbe3cd37da20a527 for fixing problems with NULL gettimeofday input mistakenly checks for a null tz field twice, when it should be checking for null tz once, and null tv once; by way of a r10/r11 typo. Any application calling gettimeofday(&tv,NULL) will "fail". This corrects that typo, and makes my G5 happy. Tested on G5. Signed-off-by: Will Schmidt <will_schmidt@vnet.ibm.com> Cc: Tony Breeds <tony@bakeyournoodle.com> Forwarded-by: Ben Herrenschmidt <benh@kernel.crashing.org> [ Ben says: "I checked the 32 bits part of the change is correct. You can probably blame me for originally writing the 2 versions with inversed usage of r10 and r11, thus confusing Tony :-)" Ben duly blamed. - Linus ] Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-06-26[POWERPC] Fix VDSO gettimeofday() when called with NULL struct timevalTony Breeds
Consider the prototype for gettimeofday(): int gettimofday(struct timeval *tv, struct timezone *tz); Although it is valid to call with /either/ tv or tz being NULL, and the C version of sys_gettimeofday() supports this, the current version of gettimeofday() in the VDSO will SEGV if called with a NULL tv. This adds a check for tv being NULL so that it doesn't SEGV. Signed-off-by: Tony Breeds <tony@bakeyournoodle.com> Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-10-25[POWERPC] Cell timebase bug workaroundBenjamin Herrenschmidt
The Cell CPU timebase has an erratum. When reading the entire 64 bits of the timebase with one mftb instruction, there is a handful of cycles window during which one might read a value with the low order 32 bits already reset to 0x00000000 but the high order bits not yet incremeted by one. This fixes it by reading the timebase again until the low order 32 bits is no longer 0. That might introduce occasional latencies if hitting mftb just at the wrong time, but no more than 70ns on a cell blade, and that was considered acceptable. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Acked-by: Olof Johansson <olof@lixom.net> Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-10-25[POWERPC] Support feature fixups in vdso'sBenjamin Herrenschmidt
This patch reworks the feature fixup mecanism so vdso's can be fixed up. The main issue was that the construct: .long label (or .llong on 64 bits) will not work in the case of a shared library like the vdso. It will generate an empty placeholder in the fixup table along with a reloc, which is not something we can deal with in the vdso. The idea here (thanks Alan Modra !) is to instead use something like: 1: .long label - 1b That is, the feature fixup tables no longer contain addresses of bits of code to patch, but offsets of such code from the fixup table entry itself. That is properly resolved by ld when building the .so's. I've modified the fixup mecanism generically to use that method for the rest of the kernel as well. Another trick is that the 32 bits vDSO included in the 64 bits kernel need to have a table in the 64 bits format. However, gas does not support 32 bits code with a statement of the form: .llong label - 1b (Or even just .llong label) That is, it cannot emit the right fixup/relocation for the linker to use to assign a 32 bits address to an .llong field. Thus, in the specific case of the 32 bits vdso built as part of the 64 bits kernel, we are using a modified macro that generates: .long 0xffffffff .llong label - 1b Note that is assumes that the value is negative which is enforced by the .lds (those offsets are always negative as the .text is always before the fixup table and gas doesn't support emiting the reloc the other way around). Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-07-31[PATCH] vDSO hash-style fixRoland McGrath
The latest toolchains can produce a new ELF section in DSOs and dynamically-linked executables. The new section ".gnu.hash" replaces ".hash", and allows for more efficient runtime symbol lookups by the dynamic linker. The new ld option --hash-style={sysv|gnu|both} controls whether to produce the old ".hash", the new ".gnu.hash", or both. In some new systems such as Fedora Core 6, gcc by default passes --hash-style=gnu to the linker, so that a standard invocation of "gcc -shared" results in producing a DSO with only ".gnu.hash". The new ".gnu.hash" sections need to be dealt with the same way as ".hash" sections in all respects; only the dynamic linker cares about their contents. To work with older dynamic linkers (i.e. preexisting releases of glibc), a binary must have the old ".hash" section. The --hash-style=both option produces binaries that a new dynamic linker can use more efficiently, but an old dynamic linker can still handle. The new section runs afoul of the custom linker scripts used to build vDSO images for the kernel. On ia64, the failure mode for this is a boot-time panic because the vDSO's PT_IA_64_UNWIND segment winds up ill-formed. This patch addresses the problem in two ways. First, it mentions ".gnu.hash" in all the linker scripts alongside ".hash". This produces correct vDSO images with --hash-style=sysv (or old tools), with --hash-style=gnu, or with --hash-style=both. Second, it passes the --hash-style=sysv option when building the vDSO images, so that ".gnu.hash" is not actually produced. This is the most conservative choice for compatibility with any old userland. There is some concern that some ancient glibc builds (though not any known old production system) might choke on --hash-style=both binaries. The optimizations provided by the new style of hash section do not really matter for a DSO with a tiny number of symbols, as the vDSO has. If someone wants to use =gnu or =both for their vDSO builds and worry less about that compatibility, just change the option and the linker script changes will make any choice work fine. Signed-off-by: Roland McGrath <roland@redhat.com> Cc: "Luck, Tony" <tony.luck@intel.com> Cc: Kyle McMartin <kyle@mcmartin.ca> Cc: Paul Mackerras <paulus@samba.org> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Jeff Dike <jdike@addtoit.com> Cc: Andi Kleen <ak@muc.de> Cc: Sam Ravnborg <sam@ravnborg.org> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-30Remove obsolete #include <linux/config.h>Jörn Engel
Signed-off-by: Jörn Engel <joern@wohnheim.fh-wedel.de> Signed-off-by: Adrian Bunk <bunk@stusta.de>
2006-03-31[PATCH] Mark unwind info for signal trampolines in vDSOsJakub Jelinek
Mark unwind info for signal trampolines using the new S augmentation flag introduced in: http://gcc.gnu.org/PR26208. GCC 4.2 (or patched earlier GCC) will be able to special case unwinding through frames right above signal trampolines. As the augmentations start with z flag and S is at the very end of the augmentation string, older GCCs will just skip the S flag as unknown (that's why an augmentation flag was chosen over say a new CFA opcode). Signed-off-by: Jakub Jelinek <jakub@redhat.com> Cc: Andi Kleen <ak@muc.de> Cc: Paul Mackerras <paulus@samba.org> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-02-28[PATCH] powerpc: vdso 64bits gettimeofday bugBenjamin Herrenschmidt
A bug in the assembly code of the vdso can cause gettimeofday() to hang or to return incorrect results. The wrong register was used to test for pending updates of the calibration variables and to create a dependency for subsequent loads. This fixes it. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-01-11[PATCH] powerpc: Add some missing .gitignore'sKumar Gala
ignore generated files under arch/powerpc Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-11-16[PATCH] powerpc: Make the vDSO functions set error code (#2)Benjamin Herrenschmidt
The vDSO functions should have the same calling convention as a syscall. Unfortunately, they currently don't set the cr0.so bit which is used to indicate an error. This patch makes them clear this bit unconditionally since all functions currently succeed. The syscall fallback done by some of them will eventually override this if the syscall fails. This also changes the symbol version of all vdso exports to make sure glibc can differenciate between old and fixed calls for existing ones like __kernel_gettimeofday. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-11-14[PATCH] powerpc: vdso fixes (take #2)Benjamin Herrenschmidt
This fixes various errors in the new functions added in the vDSO's, I've now verified all functions on both 32 and 64 bits vDSOs. It also fix a sign extension bug getting the initial time of day at boot that could cause the monotonic clock value to be completely on bogus for 64 bits applications (with either the vDSO or the syscall) on powermacs. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-11-11[PATCH] powerpc: Merge vdso's and add vdso support to 32 bits kernelBenjamin Herrenschmidt
This patch moves the vdso's to arch/powerpc, adds support for the 32 bits vdso to the 32 bits kernel, rename systemcfg (finally !), and adds some new (still untested) routines to both vdso's: clock_gettime() with support for CLOCK_REALTIME and CLOCK_MONOTONIC, clock_getres() (same clocks) and get_tbfreq() for glibc to retreive the timebase frequency. Tom,Steve: The implementation of get_tbfreq() I've done for 32 bits returns a long long (r3, r4) not a long. This is such that if we ever add support for >4Ghz timebases on ppc32, the userland interface won't have to change. I have tested gettimeofday() using some glibc patches in both ppc32 and ppc64 kernels using 32 bits userland (I haven't had a chance to test a 64 bits userland yet, but the implementation didn't change and was tested earlier). I haven't tested yet the new functions. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Paul Mackerras <paulus@samba.org>