- 12 Jul, 2016 3 commits
-
-
Steve Capper authored
It can be useful for JIT software to be aware of MIDR_EL1 and REVIDR_EL1 to ascertain the presence of any core errata that could affect code generation. This patch exposes these registers through sysfs: /sys/devices/system/cpu/cpu$ID/regs/identification/midr_el1 /sys/devices/system/cpu/cpu$ID/regs/identification/revidr_el1 where $ID is the cpu number. For big.LITTLE systems, one can have a mixture of cores (e.g. Cortex A53 and Cortex A57), thus all CPUs need to be enumerated. If the kernel does not have valid information to populate these entries with, an empty string is returned to userspace. Cc: Mark Rutland <mark.rutland@arm.com> Reviewed-by:
Will Deacon <will.deacon@arm.com> Signed-off-by:
Steve Capper <steve.capper@linaro.org> [suzuki.poulose@arm.com: ABI documentation updates, hotplug notifiers, kobject changes] Signed-off-by:
Suzuki K Poulose <suzuki.poulose@arm.com> Signed-off-by:
Catalin Marinas <catalin.marinas@arm.com>
-
Kevin Brodsky authored
So far the arm64 clock_gettime() vDSO implementation only supported the following clocks, falling back to the syscall for the others: - CLOCK_REALTIME{,_COARSE} - CLOCK_MONOTONIC{,_COARSE} This patch adds support for the CLOCK_MONOTONIC_RAW clock, taking advantage of the recent refactoring of the vDSO time functions. Like the non-_COARSE clocks, this only works when the "arch_sys_counter" clocksource is in use (allowing us to read the current time from the virtual counter register), otherwise we also have to fall back to the syscall. Most of the data is shared with CLOCK_MONOTONIC, and the algorithm is similar. The reference implementation in kernel/time/timekeeping.c shows that: - CLOCK_MONOTONIC = tk->wall_to_monotonic + tk->xtime_sec + timekeeping_get_ns(&tk->tkr_mono) - CLOCK_MONOTONIC_RAW = tk->raw_time + timekeeping_get_ns(&tk->tkr_raw) - tkr_mono and tkr_raw are identical (in particular, same clocksource), except these members: * mult (only mono's multiplier is NTP-adjusted) * xtime_nsec (always 0 for raw) Therefore, tk->raw_time and tkr_raw->mult are now also stored in the vDSO data page. Cc: Ali Saidi <ali.saidi@arm.com> Signed-off-by:Kevin Brodsky <kevin.brodsky@arm.com> Reviewed-by:
Dave Martin <dave.martin@arm.com> Acked-by:
-
Kevin Brodsky authored
Time functions are directly implemented in assembly in arm64, and it is desirable to keep it this way for performance reasons (everything fits in registers, so that the stack is not used at all). However, the current implementation is quite difficult to read and understand (even considering it's assembly). Additionally, due to the structure of __kernel_clock_gettime, which heavily uses conditional branches to share code between the different clocks, it is difficult to support a new clock without making the branches even harder to follow. This commit completely refactors the structure of clock_gettime (and gettimeofday along the way) while keeping exactly the same algorithms. We no longer try to share code; instead, macros provide common operations. This new approach comes with a number of advantages: - In clock_gettime, clock implementations are no longer interspersed, making them much more readable. Additionally, macros only use registers passed as arguments or reserved with .req, this way it is easy to make sure that registers are properly allocated. To avoid a large number of branches in a given execution path, a jump table is used; a normal execution uses 3 unconditional branches. - __do_get_tspec has been replaced with 2 macros (get_ts_clock_mono, get_clock_shifted_nsec) and explicit loading of data from the vDSO page. Consequently, clock_gettime and gettimeofday are now leaf functions, and saving x30 (lr) is no longer necessary. - Variables protected by tb_seq_count are now loaded all at once, allowing to merge the seqcnt_read macro into seqcnt_check. - For CLOCK_REALTIME_COARSE, removed an unused load of the wall to monotonic timespec. - For CLOCK_MONOTONIC_COARSE, removed a few shift instructions. Obviously, the downside of sharing less code is an increase in code size. However since the vDSO has its own code page, this does not really matter, as long as the size of the DSO remains below 4 kB. For now this should be all right: Before After vdso.so size (B) 2776 3000 Signed-off-by:
-
- 11 Jul, 2016 2 commits
-
-
Kevin Brodsky authored
arm64/kernel/{vdso,signal}.c include vdso-offsets.h, as well as any file that includes asm/vdso.h. Therefore, vdso-offsets.h must be generated before these files are compiled. The current rules in arm64/kernel/Makefile do not actually enforce this, because even though $(obj)/vdso is listed as a prerequisite for vdso-offsets.h, this does not result in the intended effect of building the vdso subdirectory (before all the other objects). As a consequence, depending on the order in which the rules are followed, vdso-offsets.h is updated or not before arm64/kernel/{vdso,signal}.o are built. The current rules also impose an unnecessary dependency on vdso-offsets.h for all arm64/kernel/*.o, resulting in unnecessary rebuilds. This is made obvious when using make -j: touch arch/arm64/kernel/vdso/gettimeofday.S && make -j$NCPUS arch/arm64/kernel will sometimes result in none of arm64/kernel/*.o being rebuilt, sometimes all of them, or even just some of them. It is quite difficult to ensure that a header is generated before it is used with recursive Makefiles by using normal rules. Instead, arch-specific generated headers are normally built in the archprepare recipe in the arch Makefile (see for instance arch/ia64/Makefile). Unfortunately, asm-offsets.h is included in gettimeofday.S, and must therefore be generated before vdso-offsets.h, which is not the case if archprepare is used. For this reason, a rule run after archprepare has to be used. This commit adds rules in arm64/Makefile to build vdso-offsets.h during the prepare step, ensuring that vdso-offsets.h is generated before building anything. It also removes the now-unnecessary dependencies on vdso-offsets.h in arm64/kernel/Makefile. Finally, it removes the duplication of asm-offsets.h between arm64/kernel/vdso/ and include/generated/ and makes include/generated/vdso-offsets.h a target in arm64/kernel/vdso/Makefile. Cc: Will Deacon <will.deacon@arm.com> Cc: Michal Marek <mmarek@suse.com> Signed-off-by: -
Catalin Marinas authored
This reverts commit 90f777be. While this commit was aimed at fixing the dependencies, with a large make -j the vdso-offsets.h file is not generated, leading to build failures. Signed-off-by:
-
- 08 Jul, 2016 3 commits
-
-
Lorenzo Pieralisi authored
Current bus notifier in ARM64 (__iommu_attach_notifier) attempts to attach dma_ops to a device on BUS_NOTIFY_ADD_DEVICE action notification. This will cause issues on ACPI based systems, where PCI devices can be added before the IOMMUs the devices are attached to had a chance to be probed, causing failures on attempts to attach dma_ops in that the domain for the respective IOMMU may not be set-up yet by the time the bus notifier is run. Devices dma_ops do not require to be set-up till the matching device drivers are probed. This means that instead of running the notifier attaching dma_ops to devices (__iommu_attach_notifier) on BUS_NOTIFY_ADD_DEVICE action, it can be run just before the device driver is bound to the device in question (on action BUS_NOTIFY_BIND_DRIVER) so that it is certain that its IOMMU group and domain are set-up accordingly at the time the notifier is triggered. This patch changes the notifier action upon which dma_ops are attached to devices and defer it to driver binding time, so that IOMMU devices have a chance to be probed and to register their bus notifiers before the dma_ops attach sequence for a device is actually carried out. As a result we also no longer need worry about racing with iommu_bus_notifier(), or about retrying the queue in case devices were added too early on DT-based systems, so clean up the notifier itself plus the additional workaround from 722ec35f ("arm64: dma-mapping: fix handling of devices registered before arch_initcall") Acked-by:
Lorenzo Pieralisi <lorenzo.pieralisi@arm.com> [rm: get rid of other now-redundant bits] Signed-off-by:
Robin Murphy <robin.murphy@arm.com> Signed-off-by:
-
Marc Zyngier authored
On a big-little system, PMUs can be wired to CPUs using per CPU interrups (PPI). In this case, it is important to make sure that the enable/disable do happen on the right set of CPUs. So instead of relying on the interrupt-affinity property, we can use the actual percpu affinity that DT exposes as part of the interrupt specifier. The DT binding is also updated to reflect the fact that the interrupt-affinity property shouldn't be used in that case. Acked-by:
Rob Herring <robh@kernel.org> Tested-by:
Caesar Wang <wxt@rock-chips.com> Signed-off-by:
Marc Zyngier <marc.zyngier@arm.com> Signed-off-by:
-
Catalin Marinas authored
arch/arm64/kernel/{vdso,signal}.c include generated/vdso-offsets.h, and therefore the symbol offsets must be generated before these files are compiled. The current rules in arm64/kernel/Makefile do not actually enforce this, because even though $(obj)/vdso is listed as a prerequisite for vdso-offsets.h, this does not result in the intended effect of building the vdso subdirectory (before all the other objects). As a consequence, depending on the order in which the rules are followed, vdso-offsets.h is updated or not before arm64/kernel/{vdso,signal}.o are built. The current rules also impose an unnecessary dependency on vdso-offsets.h for all arm64/kernel/*.o, resulting in unnecessary rebuilds. This patch removes the arch/arm64/kernel/vdso/vdso-offsets.h file generation, leaving only the include/generated/vdso-offsets.h one. It adds a forced dependency check of the vdso-offsets.h file in arch/arm64/kernel/Makefile which, if not up to date according to the arch/arm64/kernel/vdso/Makefile rules (depending on vdso.so.dbg), will trigger the vdso/ subdirectory build and vdso-offsets.h re-generation. Automatic kbuild dependency rules between kernel/{vdso,signal}.c rules and vdso-offsets.h will guarantee that the vDSO object is built first, followed by the generated symbol offsets header file. Reported-by:
-
- 01 Jul, 2016 11 commits
-
-
Ard Biesheuvel authored
The routine __create_pgd_mapping() does nothing except calling init_pgd(), which has no other callers. So fold the latter into the former. Also, drop a comment that has gone stale. Signed-off-by:
Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by:
-
Catalin Marinas authored
Since the efi_create_mapping() no longer generates block mappings and being the last user of the split_p*d code, remove these functions and the corresponding TLBI. Signed-off-by:
-
Ard Biesheuvel authored
When running the OS with a page size > 4 KB, we need to round up mappings for regions that are not aligned to the OS's page size. We already avoid block mappings for EfiRuntimeServicesCode/Data regions for other reasons, but in the unlikely event that other unaliged regions exists that have the EFI_MEMORY_RUNTIME attribute set, ensure that unaligned regions are always mapped down to pages. This way, the overlapping page is guaranteed not to be covered by a block mapping that needs to be split. Signed-off-by:
-
Ard Biesheuvel authored
To avoid triggering diagnostics in the MMU code that are finicky about splitting block mappings into more granular mappings, ensure that regions that are likely to appear in the Memory Attributes table as well as the UEFI memory map are always mapped down to pages. This way, we can use apply_to_page_range() instead of create_pgd_mapping() for the second pass, which cannot split or merge block entries, and operates strictly on PTEs. Note that this aligns the arm64 Memory Attributes table handling code with the ARM code, which already uses apply_to_page_range() to set the strict permissions. Signed-off-by:
-
Ard Biesheuvel authored
Add a bool parameter 'allow_block_mappings' to create_pgd_mapping() and the various helper functions that it descends into, to give the caller control over whether block entries may be used to create the mapping. The UEFI runtime mapping routines will use this to avoid creating block entries that would need to split up into page entries when applying the permissions listed in the Memory Attributes firmware table. This also replaces the block_mappings_allowed() helper function that was added for DEBUG_PAGEALLOC functionality, but the resulting code is functionally equivalent (given that debug_page_alloc does not operate on EFI page table entries anyway) Signed-off-by:
-
Andre Przywara authored
The ARM errata 819472, 826319, 827319 and 824069 for affected Cortex-A53 cores demand to promote "dc cvau" instructions to "dc civac". Since we allow userspace to also emit those instructions, we should make sure that "dc cvau" gets promoted there too. So lets grasp the nettle here and actually trap every userland cache maintenance instruction once we detect at least one affected core in the system. We then emulate the instruction by executing it on behalf of userland, promoting "dc cvau" to "dc civac" on the way and injecting access fault back into userspace. Signed-off-by:
Andre Przywara <andre.przywara@arm.com> [catalin.marinas@arm.com: s/set_segfault/arm64_notify_segfault/] Signed-off-by:
-
Andre Przywara authored
The code for injecting a signal into userland if a trapped instruction fails emulation due to a _userland_ error (like an illegal address) will be used more often with the next patch. Factor out the core functionality into a separate function and use that both for the existing trap handler and for the deprecated instructions emulation. Signed-off-by:
-
Andre Przywara authored
Currently we call the (optional) enable function for CPU _features_ only. As CPU _errata_ descriptions share the same data structure and having an enable function is useful for errata as well (for instance to set bits in SCTLR), lets call it when enumerating erratas too. Signed-off-by:
-
Andre Przywara authored
The newly introduced dcache_by_line_op macro is used at least in one occassion at the moment to issue a "dc cvau" instruction, which is affected by ARM errata 819472, 826319, 827319 and 824069. Change the macro to allow for alternative patching in there to protect affected Cortex-A53 cores. Signed-off-by:
-
Andre Przywara authored
The ARM errata 819472, 826319, 827319 and 824069 for affected Cortex-A53 cores demand to promote "dc cvau" instructions to "dc civac" as well. Attribute the usage of the instruction in __flush_cache_user_range to also be covered by our alternative patching efforts. For that we introduce an assembly macro which both deals with alternatives while still tagging the instructions as USER. Signed-off-by:
-
Andre Przywara authored
Commit 77ee306c ("arm64: alternatives: add enable parameter to conditional asm macros") extended the alternative assembly macros. Unfortunately this does not really work as one would expect, as the enable parameter in fact correctly protects the alternative section magic, but not the actual code sequences. This results in having both the original instruction(s) _and_ the alternative ones, if enable if false. Since there is no user of this macros anyway, just revert it. This reverts commit 77ee306c. Signed-off-by:
-
- 30 Jun, 2016 1 commit
-
-
Kefeng Wang authored
The memblock_alloc() and memblock_alloc_base() will panic on their own if no free memory, remove pointless BUG_ON. Signed-off-by:
Kefeng Wang <wangkefeng.wang@huawei.com> Acked-by:
Mark Rutland <mark.rutland@arm.com> Signed-off-by:
-
- 27 Jun, 2016 7 commits
-
-
Ard Biesheuvel authored
As Kees Cook notes in the ARM counterpart of this patch [0]: The _etext position is defined to be the end of the kernel text code, and should not include any part of the data segments. This interferes with things that might check memory ranges and expect executable code up to _etext. In particular, Kees is referring to the HARDENED_USERCOPY patch set [1], which rejects attempts to call copy_to_user() on kernel ranges containing executable code, but does allow access to the .rodata segment. Regardless of whether one may or may not agree with the distinction, it makes sense for _etext to have the same meaning across architectures. So let's put _etext where it belongs, between .text and .rodata, and fix up existing references to use __init_begin instead, which unlike _end_rodata includes the exception and notes sections as well. The _etext references in kaslr.c are left untouched, since its references to [_stext, _etext) are meant to capture potential jump instruction targets, and so disregarding .rodata is actually an improvement here. [0] http://article.gmane.org/gmane.linux.kernel/2245084 [1] http://thread.gmane.org/gmane.linux.kernel.hardened.devel/2502Reported-by:
Kees Cook <keescook@chromium.org> Reviewed-by:
-
Mark Rutland authored
We currently open-code extracting the NUMA node of a memblock region, which requires an ifdef to cater for !CONFIG_NUMA builds where the memblock_region::nid field does not exist. The generic memblock_get_region_node helper is intended to cater for this. For CONFIG_HAVE_MEMBLOCK_NODE_MAP, builds this returns reg->nid, and for for !CONFIG_HAVE_MEMBLOCK_NODE_MAP builds this is a static inline that returns 0. Note that for arm64, CONFIG_HAVE_MEMBLOCK_NODE_MAP is selected iff CONFIG_NUMA is. This patch makes use of memblock_get_region_node to simplify the arm64 code. At the same time, we can move the nid variable definition into the loop, as this is the only place it is used. Signed-off-by:
Steve Capper <steve.capper@arm.com> Acked-by:
-
Geoff Levand authored
To aid in debugging kexec problems or when adding new functionality to kexec add a new routine kexec_image_info() and several inline pr_debug statements. Signed-off-by:
Geoff Levand <geoff@infradead.org> Reviewed-by:
James Morse <james.morse@arm.com> Signed-off-by:
-
Geoff Levand authored
Signed-off-by:
-
Geoff Levand authored
Add three new files, kexec.h, machine_kexec.c and relocate_kernel.S to the arm64 architecture that add support for the kexec re-boot mechanism (CONFIG_KEXEC) on arm64 platforms. Signed-off-by:
-
Geoff Levand authored
Commit 68234df4 ("arm64: kill flush_cache_all()") removed the global arm64 routines cpu_reset() and cpu_soft_restart() needed by the arm64 kexec and kdump support. Add back a simplified version of cpu_soft_restart() with some changes needed for kexec in the new files cpu_reset.S, and cpu_reset.h. When a CPU is reset it needs to be put into the exception level it had when it entered the kernel. Update cpu_soft_restart() to accept an argument which signals if the reset address should be entered at EL1 or EL2, and add a new hypercall HVC_SOFT_RESTART which is used for the EL2 switch. Signed-off-by:
-
James Morse authored
kernel/smp.c has a fancy counter that keeps track of the number of CPUs it marked as not-present and left in cpu_park_loop(). If there are any CPUs spinning in here, features like kexec or hibernate may release them by overwriting this memory. This problem also occurs on machines using spin-tables to release secondary cores. After commit 44dbcc93 ("arm64: Fix behavior of maxcpus=N") we bring all known cpus into the secondary holding pen, meaning this memory can't be re-used by kexec or hibernate. Add a function cpus_are_stuck_in_kernel() to determine if either of these cases have occurred. Signed-off-by:
-
- 22 Jun, 2016 1 commit
-
-
Catalin Marinas authored
This reverts commit 6b14c517. The original patch and UBSAN+KASAN enabled causes Linux to fail to link with: lib/built-in.o: In function `get_signed_val': lib/ubsan.c:93: undefined reference to `__ashlti3' lib/ubsan.c:93: undefined reference to `__ashrti3' Signed-off-by:
-
- 21 Jun, 2016 11 commits
-
-
Mark Rutland authored
Currently we treat ESR_EL1 bit 24 as software-defined for distinguishing instruction aborts from data aborts, but this bit is architecturally RES0 for instruction aborts, and could be allocated for an arbitrary purpose in future. Additionally, we hard-code the value in entry.S without the mnemonic, making the code difficult to understand. Instead, remove ESR_LNX_EXEC, and distinguish aborts based on the esr, which we already pass to the sole use of ESR_LNX_EXEC. A new helper, is_el0_instruction_abort() is added to make the logic clear. Any instruction aborts taken from EL1 will already have been handled by bad_mode, so we need not handle that case in the helper. For consistency, the existing permission_fault helper is renamed to is_permission_fault, and the return type is changed to bool. There should be no functional changes as the return value was a boolean expression, and the result is only used in another boolean expression. Signed-off-by:
-
Mark Rutland authored
Now that we have a helper to extract the EC from an ESR_ELx value, make use of this in the arm64 KVM code for simplicity and consistency. There should be no functional changes as a result of this patch. Signed-off-by:
Christoffer Dall <christoffer.dall@linaro.org> Signed-off-by:
-
Mark Rutland authored
Several places open-code extraction of the EC field from an ESR_ELx value, in subtly different ways. This is unfortunate duplication and variation, and the precise logic used to extract the field is a distraction. This patch adds a new macro, ESR_ELx_EC(), to extract the EC field from an ESR_ELx value in a consistent fashion. Existing open-coded extractions in core arm64 code are moved over to the new helper. KVM code is left as-is for the moment. Signed-off-by:
Huang Shijie <shijie.huang@arm.com> Cc: Dave P Martin <dave.martin@arm.com> Cc: James Morse <james.morse@arm.com> Cc: Marc Zyngier <marc.zyngier@arm.com> Cc: Will Deacon <will.deacon@arm.com> Signed-off-by:
-
Jisheng Zhang authored
we only initialize swiotlb when swiotlb_force is true or not all system memory is DMA-able, this trivial optimization saves us 64MB when swiotlb is not necessary. Signed-off-by:
Jisheng Zhang <jszhang@marvell.com> Signed-off-by:
-
Al Stone authored
The ACPI 6.1 specification was recently released at the end of January 2016, but the arm64 kernel documentation for the use of ACPI was written for the 5.1 version of the spec. There were significant additions to the spec that had not yet been mentioned -- for example, the 6.0 mechanisms added to make it easier to define processors and low power idle states, as well as the 6.1 addition allowing regular interrupts (not just from GPIO) be used to signal ACPI general purpose events. This patch reflects going back through and examining the specs in detail and updating content appropriately. Whilst there, a few odds and ends of typos were caught as well. This brings the documentation up to date with ACPI 6.1 for arm64. Signed-off-by:
Al Stone <al.stone@linaro.org> Acked-by:
Hanjun Guo <hanjun.guo@linaro.org> Reviewed-by:
Roy Franz <roy.franz@hpe.com> Signed-off-by:
-
Alexander Potapenko authored
Add ARCH_HAS_KCOV to ARM64 config. To avoid potential crashes, disable instrumentation of the files in arch/arm64/kvm/hyp/*. Signed-off-by:
Alexander Potapenko <glider@google.com> Acked-by:
-
Kefeng Wang authored
The gcc support __SIZEOF_INT128__ and __int128 in arm64, thus, enable ARCH_SUPPORTS_INT128 to make mul_u64_u32_shr() a bit more efficient in scheduler. Signed-off-by:
-
Mark Rutland authored
Currently dump_mem attempts to dump memory in 64-bit chunks when reporting a failure in 64-bit code, or 32-bit chunks when reporting a failure in 32-bit code. We added code to handle these two cases separately in commit e147ae6d ("arm64: modify the dump mem for 64 bit addresses"). However, in all cases dump_mem is called, the failing context is a kernel rather than user context. Additionally dump_mem is assumed to only be used for kernel contexts, as internally it switches to KERNEL_DS, and its callers pass kernel stack bounds. This patch removes the redundant 32-bit chunk logic and associated compat parameter, largely reverting the aforementioned commit. For the call in __die(), the check of in_interrupt() is removed also, as __die() is only called in response to faults from the kernel's exception level, and thus the !user_mode(regs) check is sufficient. Were this not the case, the used of task_stack_page(tsk) to generate the stack bounds would be erroneous. Signed-off-by:
-
Yang Shi authored
The upstream commit 1771c6e1 ("x86/kasan: instrument user memory access API") added KASAN instrument to x86 user memory access API, so added such instrument to ARM64 too. Define __copy_to/from_user in C in order to add kasan_check_read/write call, rename assembly implementation to __arch_copy_to/from_user. Tested by test_kasan module. Acked-by:
Andrey Ryabinin <aryabinin@virtuozzo.com> Reviewed-by:
Yang Shi <yang.shi@linaro.org> Signed-off-by:
-
Mark Rutland authored
For debugging purposes, it would be nice if we could export page tables other than the swapper_pg_dir to userspace. To enable this, this patch refactors the arm64 page table dumping code such that multiple tables may be registered with the framework, and exported under debugfs. Signed-off-by:
-
Robin Murphy authored
AArch64 is capable of 128-bit memory accesses without alignment restrictions, which makes it both possible and highly practical to slurp up a typical 20-byte IP header in just 2 loads. Implement our own version of ip_fast_checksum() to take advantage of that, resulting in considerably fewer instructions and memory accesses than the generic version. We can also get more optimal code generation for csum_fold() by defining it a slightly different way round from the generic version, so throw that into the mix too. Suggested-by:
Luke Starrett <luke.starrett@broadcom.com> Acked-by:
-
- 20 Jun, 2016 1 commit
-
-
Linus Torvalds authored
-
