Skip to content

L
linux
Commit 36c344f3 authored by Paolo Bonzini's avatar Paolo Bonzini
Browse files
Options

Merge tag 'kvm-arm-for-v4.12-round2' of... 

Merge tag 'kvm-arm-for-v4.12-round2' of git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD

Second round of KVM/ARM Changes for v4.12.

Changes include:
 - A fix related to the 32-bit idmap stub
 - A fix to the bitmask used to deode the operands of an AArch32 CP
   instruction
 - We have moved the files shared between arch/arm/kvm and
   arch/arm64/kvm to virt/kvm/arm
 - We add support for saving/restoring the virtual ITS state to
   userspace
parents 03efce6f a2b19e6e
Hide whitespace changes
Inline Side-by-side
Showing with 1837 additions and 537 deletions
Documentation/virtual/kvm/devices/arm-vgic-its.txt
View file @ 36c344f3
......@@ -32,7 +32,128 @@ Groups:
KVM_DEV_ARM_VGIC_CTRL_INIT
request the initialization of the ITS, no additional parameter in
kvm_device_attr.addr.
KVM_DEV_ARM_ITS_SAVE_TABLES
save the ITS table data into guest RAM, at the location provisioned
by the guest in corresponding registers/table entries.
The layout of the tables in guest memory defines an ABI. The entries
are laid out in little endian format as described in the last paragraph.
KVM_DEV_ARM_ITS_RESTORE_TABLES
restore the ITS tables from guest RAM to ITS internal structures.
The GICV3 must be restored before the ITS and all ITS registers but
the GITS_CTLR must be restored before restoring the ITS tables.
The GITS_IIDR read-only register must also be restored before
calling KVM_DEV_ARM_ITS_RESTORE_TABLES as the IIDR revision field
encodes the ABI revision.
The expected ordering when restoring the GICv3/ITS is described in section
"ITS Restore Sequence".
Errors:
-ENXIO: ITS not properly configured as required prior to setting
this attribute
-ENOMEM: Memory shortage when allocating ITS internal data
-EINVAL: Inconsistent restored data
-EFAULT: Invalid guest ram access
-EBUSY: One or more VCPUS are running
KVM_DEV_ARM_VGIC_GRP_ITS_REGS
Attributes:
The attr field of kvm_device_attr encodes the offset of the
ITS register, relative to the ITS control frame base address
(ITS_base).
kvm_device_attr.addr points to a __u64 value whatever the width
of the addressed register (32/64 bits). 64 bit registers can only
be accessed with full length.
Writes to read-only registers are ignored by the kernel except for:
- GITS_CREADR. It must be restored otherwise commands in the queue
will be re-executed after restoring CWRITER. GITS_CREADR must be
restored before restoring the GITS_CTLR which is likely to enable the
ITS. Also it must be restored after GITS_CBASER since a write to
GITS_CBASER resets GITS_CREADR.
- GITS_IIDR. The Revision field encodes the table layout ABI revision.
In the future we might implement direct injection of virtual LPIs.
This will require an upgrade of the table layout and an evolution of
the ABI. GITS_IIDR must be restored before calling
KVM_DEV_ARM_ITS_RESTORE_TABLES.
For other registers, getting or setting a register has the same
effect as reading/writing the register on real hardware.
Errors:
-ENXIO: Offset does not correspond to any supported register
-EFAULT: Invalid user pointer for attr->addr
-EINVAL: Offset is not 64-bit aligned
-EBUSY: one or more VCPUS are running
ITS Restore Sequence:
-------------------------
The following ordering must be followed when restoring the GIC and the ITS:
a) restore all guest memory and create vcpus
b) restore all redistributors
c) provide the its base address
(KVM_DEV_ARM_VGIC_GRP_ADDR)
d) restore the ITS in the following order:
1. Restore GITS_CBASER
2. Restore all other GITS_ registers, except GITS_CTLR!
3. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
4. Restore GITS_CTLR
Then vcpus can be started.
ITS Table ABI REV0:
-------------------
Revision 0 of the ABI only supports the features of a virtual GICv3, and does
not support a virtual GICv4 with support for direct injection of virtual
interrupts for nested hypervisors.
The device table and ITT are indexed by the DeviceID and EventID,
respectively. The collection table is not indexed by CollectionID, and the
entries in the collection are listed in no particular order.
All entries are 8 bytes.
Device Table Entry (DTE):
bits: | 63| 62 ... 49 | 48 ... 5 | 4 ... 0 |
values: | V | next | ITT_addr | Size |
where;
- V indicates whether the entry is valid. If not, other fields
are not meaningful.
- next: equals to 0 if this entry is the last one; otherwise it
corresponds to the DeviceID offset to the next DTE, capped by
2^14 -1.
- ITT_addr matches bits [51:8] of the ITT address (256 Byte aligned).
- Size specifies the supported number of bits for the EventID,
minus one
Collection Table Entry (CTE):
bits: | 63| 62 .. 52 | 51 ... 16 | 15 ... 0 |
values: | V | RES0 | RDBase | ICID |
where:
- V indicates whether the entry is valid. If not, other fields are
not meaningful.
- RES0: reserved field with Should-Be-Zero-or-Preserved behavior.
- RDBase is the PE number (GICR_TYPER.Processor_Number semantic),
- ICID is the collection ID
Interrupt Translation Entry (ITE):
bits: | 63 ... 48 | 47 ... 16 | 15 ... 0 |
values: | next | pINTID | ICID |
where:
- next: equals to 0 if this entry is the last one; otherwise it corresponds
to the EventID offset to the next ITE capped by 2^16 -1.
- pINTID is the physical LPI ID; if zero, it means the entry is not valid
and other fields are not meaningful.
- ICID is the collection ID
Documentation/virtual/kvm/devices/arm-vgic-v3.txt
View file @ 36c344f3
......@@ -167,11 +167,17 @@ Groups:
KVM_DEV_ARM_VGIC_CTRL_INIT
request the initialization of the VGIC, no additional parameter in
kvm_device_attr.addr.
KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES
save all LPI pending bits into guest RAM pending tables.
The first kB of the pending table is not altered by this operation.
Errors:
-ENXIO: VGIC not properly configured as required prior to calling
this attribute
-ENODEV: no online VCPU
-ENOMEM: memory shortage when allocating vgic internal data
-EFAULT: Invalid guest ram access
-EBUSY: One or more VCPUS are running
KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO
......
arch/arm/include/uapi/asm/kvm.h
View file @ 36c344f3
......@@ -196,13 +196,17 @@ struct kvm_arch_memory_slot {
#define KVM_DEV_ARM_VGIC_GRP_REDIST_REGS 5
#define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6
#define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO 7
#define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8
#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT 10
#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \
(0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT)
#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INTID_MASK 0x3ff
#define VGIC_LEVEL_INFO_LINE_LEVEL 0
#define KVM_DEV_ARM_VGIC_CTRL_INIT 0
#define KVM_DEV_ARM_VGIC_CTRL_INIT 0
#define KVM_DEV_ARM_ITS_SAVE_TABLES 1
#define KVM_DEV_ARM_ITS_RESTORE_TABLES 2
#define KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES 3
/* KVM_IRQ_LINE irq field index values */
#define KVM_ARM_IRQ_TYPE_SHIFT 24
......
arch/arm/kvm/Makefile
View file @ 36c344f3
......@@ -18,9 +18,12 @@ KVM := ../../../virt/kvm
kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o $(KVM)/vfio.o
obj-$(CONFIG_KVM_ARM_HOST) += hyp/
obj-y += kvm-arm.o init.o interrupts.o
obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o
obj-y += coproc.o coproc_a15.o coproc_a7.o mmio.o psci.o perf.o vgic-v3-coproc.o
obj-y += handle_exit.o guest.o emulate.o reset.o
obj-y += coproc.o coproc_a15.o coproc_a7.o vgic-v3-coproc.o
obj-y += $(KVM)/arm/arm.o $(KVM)/arm/mmu.o $(KVM)/arm/mmio.o
obj-y += $(KVM)/arm/psci.o $(KVM)/arm/perf.o
obj-y += $(KVM)/arm/aarch32.o
obj-y += $(KVM)/arm/vgic/vgic.o
......
arch/arm/kvm/trace.h
View file @ 36c344f3
......@@ -6,133 +6,6 @@
#undef TRACE_SYSTEM
#define TRACE_SYSTEM kvm
/*
* Tracepoints for entry/exit to guest
*/
TRACE_EVENT(kvm_entry,
TP_PROTO(unsigned long vcpu_pc),
TP_ARGS(vcpu_pc),
TP_STRUCT__entry(
__field( unsigned long, vcpu_pc )
),
TP_fast_assign(
__entry->vcpu_pc = vcpu_pc;
),
TP_printk("PC: 0x%08lx", __entry->vcpu_pc)
);
TRACE_EVENT(kvm_exit,
TP_PROTO(int idx, unsigned int exit_reason, unsigned long vcpu_pc),
TP_ARGS(idx, exit_reason, vcpu_pc),
TP_STRUCT__entry(
__field( int, idx )
__field( unsigned int, exit_reason )
__field( unsigned long, vcpu_pc )
),
TP_fast_assign(
__entry->idx = idx;
__entry->exit_reason = exit_reason;
__entry->vcpu_pc = vcpu_pc;
),
TP_printk("%s: HSR_EC: 0x%04x (%s), PC: 0x%08lx",
__print_symbolic(__entry->idx, kvm_arm_exception_type),
__entry->exit_reason,
__print_symbolic(__entry->exit_reason, kvm_arm_exception_class),
__entry->vcpu_pc)
);
TRACE_EVENT(kvm_guest_fault,
TP_PROTO(unsigned long vcpu_pc, unsigned long hsr,
unsigned long hxfar,
unsigned long long ipa),
TP_ARGS(vcpu_pc, hsr, hxfar, ipa),
TP_STRUCT__entry(
__field( unsigned long, vcpu_pc )
__field( unsigned long, hsr )
__field( unsigned long, hxfar )
__field( unsigned long long, ipa )
),
TP_fast_assign(
__entry->vcpu_pc = vcpu_pc;
__entry->hsr = hsr;
__entry->hxfar = hxfar;
__entry->ipa = ipa;
),
TP_printk("ipa %#llx, hsr %#08lx, hxfar %#08lx, pc %#08lx",
__entry->ipa, __entry->hsr,
__entry->hxfar, __entry->vcpu_pc)
);
TRACE_EVENT(kvm_access_fault,
TP_PROTO(unsigned long ipa),
TP_ARGS(ipa),
TP_STRUCT__entry(
__field( unsigned long, ipa )
),
TP_fast_assign(
__entry->ipa = ipa;
),
TP_printk("IPA: %lx", __entry->ipa)
);
TRACE_EVENT(kvm_irq_line,
TP_PROTO(unsigned int type, int vcpu_idx, int irq_num, int level),
TP_ARGS(type, vcpu_idx, irq_num, level),
TP_STRUCT__entry(
__field( unsigned int, type )
__field( int, vcpu_idx )
__field( int, irq_num )
__field( int, level )
),
TP_fast_assign(
__entry->type = type;
__entry->vcpu_idx = vcpu_idx;
__entry->irq_num = irq_num;
__entry->level = level;
),
TP_printk("Inject %s interrupt (%d), vcpu->idx: %d, num: %d, level: %d",
(__entry->type == KVM_ARM_IRQ_TYPE_CPU) ? "CPU" :
(__entry->type == KVM_ARM_IRQ_TYPE_PPI) ? "VGIC PPI" :
(__entry->type == KVM_ARM_IRQ_TYPE_SPI) ? "VGIC SPI" : "UNKNOWN",
__entry->type, __entry->vcpu_idx, __entry->irq_num, __entry->level)
);
TRACE_EVENT(kvm_mmio_emulate,
TP_PROTO(unsigned long vcpu_pc, unsigned long instr,
unsigned long cpsr),
TP_ARGS(vcpu_pc, instr, cpsr),
TP_STRUCT__entry(
__field( unsigned long, vcpu_pc )
__field( unsigned long, instr )
__field( unsigned long, cpsr )
),
TP_fast_assign(
__entry->vcpu_pc = vcpu_pc;
__entry->instr = instr;
__entry->cpsr = cpsr;
),
TP_printk("Emulate MMIO at: 0x%08lx (instr: %08lx, cpsr: %08lx)",
__entry->vcpu_pc, __entry->instr, __entry->cpsr)
);
/* Architecturally implementation defined CP15 register access */
TRACE_EVENT(kvm_emulate_cp15_imp,
TP_PROTO(unsigned long Op1, unsigned long Rt1, unsigned long CRn,
......@@ -181,87 +54,6 @@ TRACE_EVENT(kvm_wfx,
__entry->is_wfe ? 'e' : 'i', __entry->vcpu_pc)
);
TRACE_EVENT(kvm_unmap_hva,
TP_PROTO(unsigned long hva),
TP_ARGS(hva),
TP_STRUCT__entry(
__field( unsigned long, hva )
),
TP_fast_assign(
__entry->hva = hva;
),
TP_printk("mmu notifier unmap hva: %#08lx", __entry->hva)
);
TRACE_EVENT(kvm_unmap_hva_range,
TP_PROTO(unsigned long start, unsigned long end),
TP_ARGS(start, end),
TP_STRUCT__entry(
__field( unsigned long, start )
__field( unsigned long, end )
),
TP_fast_assign(
__entry->start = start;
__entry->end = end;
),
TP_printk("mmu notifier unmap range: %#08lx -- %#08lx",
__entry->start, __entry->end)
);
TRACE_EVENT(kvm_set_spte_hva,
TP_PROTO(unsigned long hva),
TP_ARGS(hva),
TP_STRUCT__entry(
__field( unsigned long, hva )
),
TP_fast_assign(
__entry->hva = hva;
),
TP_printk("mmu notifier set pte hva: %#08lx", __entry->hva)
);
TRACE_EVENT(kvm_age_hva,
TP_PROTO(unsigned long start, unsigned long end),
TP_ARGS(start, end),
TP_STRUCT__entry(
__field( unsigned long, start )
__field( unsigned long, end )
),
TP_fast_assign(
__entry->start = start;
__entry->end = end;
),
TP_printk("mmu notifier age hva: %#08lx -- %#08lx",
__entry->start, __entry->end)
);
TRACE_EVENT(kvm_test_age_hva,
TP_PROTO(unsigned long hva),
TP_ARGS(hva),
TP_STRUCT__entry(
__field( unsigned long, hva )
),
TP_fast_assign(
__entry->hva = hva;
),
TP_printk("mmu notifier test age hva: %#08lx", __entry->hva)
);
TRACE_EVENT(kvm_hvc,
TP_PROTO(unsigned long vcpu_pc, unsigned long r0, unsigned long imm),
TP_ARGS(vcpu_pc, r0, imm),
......@@ -282,45 +74,6 @@ TRACE_EVENT(kvm_hvc,
__entry->vcpu_pc, __entry->r0, __entry->imm)
);
TRACE_EVENT(kvm_set_way_flush,
TP_PROTO(unsigned long vcpu_pc, bool cache),
TP_ARGS(vcpu_pc, cache),
TP_STRUCT__entry(
__field( unsigned long, vcpu_pc )
__field( bool, cache )
),
TP_fast_assign(
__entry->vcpu_pc = vcpu_pc;
__entry->cache = cache;
),
TP_printk("S/W flush at 0x%016lx (cache %s)",
__entry->vcpu_pc, __entry->cache ? "on" : "off")
);
TRACE_EVENT(kvm_toggle_cache,
TP_PROTO(unsigned long vcpu_pc, bool was, bool now),
TP_ARGS(vcpu_pc, was, now),
TP_STRUCT__entry(
__field( unsigned long, vcpu_pc )
__field( bool, was )
__field( bool, now )
),
TP_fast_assign(
__entry->vcpu_pc = vcpu_pc;
__entry->was = was;
__entry->now = now;
),
TP_printk("VM op at 0x%016lx (cache was %s, now %s)",
__entry->vcpu_pc, __entry->was ? "on" : "off",
__entry->now ? "on" : "off")
);
#endif /* _TRACE_KVM_H */
#undef TRACE_INCLUDE_PATH
......
arch/arm64/include/asm/kvm_emulate.h
View file @ 36c344f3
......@@ -240,6 +240,12 @@ static inline u8 kvm_vcpu_trap_get_fault_type(const struct kvm_vcpu *vcpu)
return kvm_vcpu_get_hsr(vcpu) & ESR_ELx_FSC_TYPE;
}
static inline int kvm_vcpu_sys_get_rt(struct kvm_vcpu *vcpu)
{
u32 esr = kvm_vcpu_get_hsr(vcpu);
return (esr & ESR_ELx_SYS64_ISS_RT_MASK) >> ESR_ELx_SYS64_ISS_RT_SHIFT;
}
static inline unsigned long kvm_vcpu_get_mpidr_aff(struct kvm_vcpu *vcpu)
{
return vcpu_sys_reg(vcpu, MPIDR_EL1) & MPIDR_HWID_BITMASK;
......
arch/arm64/include/uapi/asm/kvm.h
View file @ 36c344f3
......@@ -216,13 +216,17 @@ struct kvm_arch_memory_slot {
#define KVM_DEV_ARM_VGIC_GRP_REDIST_REGS 5
#define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6
#define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO 7
#define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8
#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT 10
#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \
(0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT)
#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INTID_MASK 0x3ff
#define VGIC_LEVEL_INFO_LINE_LEVEL 0
#define KVM_DEV_ARM_VGIC_CTRL_INIT 0
#define KVM_DEV_ARM_VGIC_CTRL_INIT 0
#define KVM_DEV_ARM_ITS_SAVE_TABLES 1
#define KVM_DEV_ARM_ITS_RESTORE_TABLES 2
#define KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES 3
/* Device Control API on vcpu fd */
#define KVM_ARM_VCPU_PMU_V3_CTRL 0
......
arch/arm64/kvm/Makefile
View file @ 36c344f3
......@@ -7,14 +7,13 @@ CFLAGS_arm.o := -I.
CFLAGS_mmu.o := -I.
KVM=../../../virt/kvm
ARM=../../../arch/arm/kvm
obj-$(CONFIG_KVM_ARM_HOST) += kvm.o
obj-$(CONFIG_KVM_ARM_HOST) += hyp/
kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o $(KVM)/vfio.o
kvm-$(CONFIG_KVM_ARM_HOST) += $(ARM)/arm.o $(ARM)/mmu.o $(ARM)/mmio.o
kvm-$(CONFIG_KVM_ARM_HOST) += $(ARM)/psci.o $(ARM)/perf.o
kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arm.o $(KVM)/arm/mmu.o $(KVM)/arm/mmio.o
kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/psci.o $(KVM)/arm/perf.o
kvm-$(CONFIG_KVM_ARM_HOST) += inject_fault.o regmap.o
kvm-$(CONFIG_KVM_ARM_HOST) += hyp.o hyp-init.o handle_exit.o
......
arch/arm64/kvm/sys_regs.c
View file @ 36c344f3
......@@ -1529,8 +1529,8 @@ static int kvm_handle_cp_64(struct kvm_vcpu *vcpu,
{
struct sys_reg_params params;
u32 hsr = kvm_vcpu_get_hsr(vcpu);
int Rt = (hsr >> 5) & 0xf;
int Rt2 = (hsr >> 10) & 0xf;
int Rt = kvm_vcpu_sys_get_rt(vcpu);
int Rt2 = (hsr >> 10) & 0x1f;
params.is_aarch32 = true;
params.is_32bit = false;
......@@ -1586,7 +1586,7 @@ static int kvm_handle_cp_32(struct kvm_vcpu *vcpu,
{
struct sys_reg_params params;
u32 hsr = kvm_vcpu_get_hsr(vcpu);
int Rt = (hsr >> 5) & 0xf;
int Rt = kvm_vcpu_sys_get_rt(vcpu);
params.is_aarch32 = true;
params.is_32bit = true;
......@@ -1688,7 +1688,7 @@ int kvm_handle_sys_reg(struct kvm_vcpu *vcpu, struct kvm_run *run)
{
struct sys_reg_params params;
unsigned long esr = kvm_vcpu_get_hsr(vcpu);
int Rt = (esr >> 5) & 0x1f;
int Rt = kvm_vcpu_sys_get_rt(vcpu);
int ret;
trace_kvm_handle_sys_reg(esr);
......
include/kvm/arm_vgic.h
View file @ 36c344f3
......@@ -148,7 +148,6 @@ struct vgic_its {
gpa_t vgic_its_base;
bool enabled;
bool initialized;
struct vgic_io_device iodev;
struct kvm_device *dev;
......@@ -162,6 +161,9 @@ struct vgic_its {
u32 creadr;
u32 cwriter;
/* migration ABI revision in use */
u32 abi_rev;
/* Protects the device and collection lists */
struct mutex its_lock;
struct list_head device_list;
......@@ -283,6 +285,7 @@ extern struct static_key_false vgic_v2_cpuif_trap;
int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write);
void kvm_vgic_early_init(struct kvm *kvm);
int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu);
int kvm_vgic_create(struct kvm *kvm, u32 type);
void kvm_vgic_destroy(struct kvm *kvm);
void kvm_vgic_vcpu_early_init(struct kvm_vcpu *vcpu);
......
include/linux/irqchip/arm-gic-v3.h
View file @ 36c344f3
......@@ -132,6 +132,9 @@
#define GIC_BASER_SHAREABILITY(reg, type) \
(GIC_BASER_##type << reg##_SHAREABILITY_SHIFT)
/* encode a size field of width @w containing @n - 1 units */
#define GIC_ENCODE_SZ(n, w) (((unsigned long)(n) - 1) & GENMASK_ULL(((w) - 1), 0))
#define GICR_PROPBASER_SHAREABILITY_SHIFT (10)
#define GICR_PROPBASER_INNER_CACHEABILITY_SHIFT (7)
#define GICR_PROPBASER_OUTER_CACHEABILITY_SHIFT (56)
......@@ -156,6 +159,8 @@
#define GICR_PROPBASER_RaWaWb GIC_BASER_CACHEABILITY(GICR_PROPBASER, INNER, RaWaWb)
#define GICR_PROPBASER_IDBITS_MASK (0x1f)
#define GICR_PROPBASER_ADDRESS(x) ((x) & GENMASK_ULL(51, 12))
#define GICR_PENDBASER_ADDRESS(x) ((x) & GENMASK_ULL(51, 16))
#define GICR_PENDBASER_SHAREABILITY_SHIFT (10)
#define GICR_PENDBASER_INNER_CACHEABILITY_SHIFT (7)
......@@ -232,12 +237,18 @@
#define GITS_CTLR_QUIESCENT (1U << 31)
#define GITS_TYPER_PLPIS (1UL << 0)
#define GITS_TYPER_ITT_ENTRY_SIZE_SHIFT 4
#define GITS_TYPER_IDBITS_SHIFT 8
#define GITS_TYPER_DEVBITS_SHIFT 13
#define GITS_TYPER_DEVBITS(r) ((((r) >> GITS_TYPER_DEVBITS_SHIFT) & 0x1f) + 1)
#define GITS_TYPER_PTA (1UL << 19)
#define GITS_TYPER_HWCOLLCNT_SHIFT 24
#define GITS_IIDR_REV_SHIFT 12
#define GITS_IIDR_REV_MASK (0xf << GITS_IIDR_REV_SHIFT)
#define GITS_IIDR_REV(r) (((r) >> GITS_IIDR_REV_SHIFT) & 0xf)
#define GITS_IIDR_PRODUCTID_SHIFT 24
#define GITS_CBASER_VALID (1ULL << 63)
#define GITS_CBASER_SHAREABILITY_SHIFT (10)
#define GITS_CBASER_INNER_CACHEABILITY_SHIFT (59)
......@@ -290,6 +301,7 @@
#define GITS_BASER_TYPE(r) (((r) >> GITS_BASER_TYPE_SHIFT) & 7)
#define GITS_BASER_ENTRY_SIZE_SHIFT (48)
#define GITS_BASER_ENTRY_SIZE(r) ((((r) >> GITS_BASER_ENTRY_SIZE_SHIFT) & 0x1f) + 1)
#define GITS_BASER_ENTRY_SIZE_MASK GENMASK_ULL(52, 48)
#define GITS_BASER_SHAREABILITY_SHIFT (10)
#define GITS_BASER_InnerShareable \
GIC_BASER_SHAREABILITY(GITS_BASER, InnerShareable)
......@@ -337,9 +349,11 @@
#define E_ITS_INT_UNMAPPED_INTERRUPT 0x010307
#define E_ITS_CLEAR_UNMAPPED_INTERRUPT 0x010507
#define E_ITS_MAPD_DEVICE_OOR 0x010801
#define E_ITS_MAPD_ITTSIZE_OOR 0x010802
#define E_ITS_MAPC_PROCNUM_OOR 0x010902
#define E_ITS_MAPC_COLLECTION_OOR 0x010903
#define E_ITS_MAPTI_UNMAPPED_DEVICE 0x010a04
#define E_ITS_MAPTI_ID_OOR 0x010a05
#define E_ITS_MAPTI_PHYSICALID_OOR 0x010a06
#define E_ITS_INV_UNMAPPED_INTERRUPT 0x010c07
#define E_ITS_INVALL_UNMAPPED_COLLECTION 0x010d09
......
include/linux/kvm_host.h
View file @ 36c344f3
......@@ -499,6 +499,17 @@ static inline struct kvm_vcpu *kvm_get_vcpu_by_id(struct kvm *kvm, int id)
return NULL;
}
static inline int kvm_vcpu_get_idx(struct kvm_vcpu *vcpu)
{
struct kvm_vcpu *tmp;
int idx;
kvm_for_each_vcpu(idx, tmp, vcpu->kvm)
if (tmp == vcpu)
return idx;
BUG();
}
#define kvm_for_each_memslot(memslot, slots) \
for (memslot = &slots->memslots[0]; \
memslot < slots->memslots + KVM_MEM_SLOTS_NUM && memslot->npages;\
......
arch/arm/kvm/arm.c → virt/kvm/arm/arm.c
View file @ 36c344f3
......@@ -332,7 +332,7 @@ int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu)
kvm_arm_reset_debug_ptr(vcpu);
return 0;
return kvm_vgic_vcpu_init(vcpu);
}
void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
......
virt/kvm/arm/trace.h
View file @ 36c344f3
......@@ -7,26 +7,250 @@
#define TRACE_SYSTEM kvm
/*
* Tracepoints for vgic
* Tracepoints for entry/exit to guest
*/
TRACE_EVENT(vgic_update_irq_pending,
TP_PROTO(unsigned long vcpu_id, __u32 irq, bool level),
TP_ARGS(vcpu_id, irq, level),
TRACE_EVENT(kvm_entry,
TP_PROTO(unsigned long vcpu_pc),
TP_ARGS(vcpu_pc),
TP_STRUCT__entry(
__field( unsigned long, vcpu_id )
__field( __u32, irq )
__field( bool, level )
__field( unsigned long, vcpu_pc )
),
TP_fast_assign(
__entry->vcpu_id = vcpu_id;
__entry->irq = irq;
__entry->vcpu_pc = vcpu_pc;
),
TP_printk("PC: 0x%08lx", __entry->vcpu_pc)
);
TRACE_EVENT(kvm_exit,
TP_PROTO(int idx, unsigned int exit_reason, unsigned long vcpu_pc),
TP_ARGS(idx, exit_reason, vcpu_pc),
TP_STRUCT__entry(
__field( int, idx )
__field( unsigned int, exit_reason )
__field( unsigned long, vcpu_pc )
),
TP_fast_assign(
__entry->idx = idx;
__entry->exit_reason = exit_reason;
__entry->vcpu_pc = vcpu_pc;
),
TP_printk("%s: HSR_EC: 0x%04x (%s), PC: 0x%08lx",
__print_symbolic(__entry->idx, kvm_arm_exception_type),
__entry->exit_reason,
__print_symbolic(__entry->exit_reason, kvm_arm_exception_class),
__entry->vcpu_pc)
);
TRACE_EVENT(kvm_guest_fault,
TP_PROTO(unsigned long vcpu_pc, unsigned long hsr,
unsigned long hxfar,
unsigned long long ipa),
TP_ARGS(vcpu_pc, hsr, hxfar, ipa),
TP_STRUCT__entry(
__field( unsigned long, vcpu_pc )
__field( unsigned long, hsr )
__field( unsigned long, hxfar )
__field( unsigned long long, ipa )
),
TP_fast_assign(
__entry->vcpu_pc = vcpu_pc;
__entry->hsr = hsr;
__entry->hxfar = hxfar;
__entry->ipa = ipa;
),
TP_printk("ipa %#llx, hsr %#08lx, hxfar %#08lx, pc %#08lx",
__entry->ipa, __entry->hsr,
__entry->hxfar, __entry->vcpu_pc)
);
TRACE_EVENT(kvm_access_fault,
TP_PROTO(unsigned long ipa),
TP_ARGS(ipa),
TP_STRUCT__entry(
__field( unsigned long, ipa )
),
TP_fast_assign(
__entry->ipa = ipa;
),
TP_printk("IPA: %lx", __entry->ipa)
);
TRACE_EVENT(kvm_irq_line,
TP_PROTO(unsigned int type, int vcpu_idx, int irq_num, int level),
TP_ARGS(type, vcpu_idx, irq_num, level),
TP_STRUCT__entry(
__field( unsigned int, type )
__field( int, vcpu_idx )
__field( int, irq_num )
__field( int, level )
),
TP_fast_assign(
__entry->type = type;
__entry->vcpu_idx = vcpu_idx;
__entry->irq_num = irq_num;
__entry->level = level;
),
TP_printk("VCPU: %ld, IRQ %d, level: %d",
__entry->vcpu_id, __entry->irq, __entry->level)
TP_printk("Inject %s interrupt (%d), vcpu->idx: %d, num: %d, level: %d",
(__entry->type == KVM_ARM_IRQ_TYPE_CPU) ? "CPU" :
(__entry->type == KVM_ARM_IRQ_TYPE_PPI) ? "VGIC PPI" :
(__entry->type == KVM_ARM_IRQ_TYPE_SPI) ? "VGIC SPI" : "UNKNOWN",
__entry->type, __entry->vcpu_idx, __entry->irq_num, __entry->level)
);
TRACE_EVENT(kvm_mmio_emulate,
TP_PROTO(unsigned long vcpu_pc, unsigned long instr,
unsigned long cpsr),
TP_ARGS(vcpu_pc, instr, cpsr),
TP_STRUCT__entry(
__field( unsigned long, vcpu_pc )
__field( unsigned long, instr )
__field( unsigned long, cpsr )
),
TP_fast_assign(
__entry->vcpu_pc = vcpu_pc;
__entry->instr = instr;
__entry->cpsr = cpsr;
),
TP_printk("Emulate MMIO at: 0x%08lx (instr: %08lx, cpsr: %08lx)",
__entry->vcpu_pc, __entry->instr, __entry->cpsr)
);
TRACE_EVENT(kvm_unmap_hva,
TP_PROTO(unsigned long hva),
TP_ARGS(hva),
TP_STRUCT__entry(
__field( unsigned long, hva )
),
TP_fast_assign(
__entry->hva = hva;
),
TP_printk("mmu notifier unmap hva: %#08lx", __entry->hva)
);
TRACE_EVENT(kvm_unmap_hva_range,
TP_PROTO(unsigned long start, unsigned long end),
TP_ARGS(start, end),
TP_STRUCT__entry(
__field( unsigned long, start )
__field( unsigned long, end )
),
TP_fast_assign(
__entry->start = start;
__entry->end = end;
),
TP_printk("mmu notifier unmap range: %#08lx -- %#08lx",
__entry->start, __entry->end)
);
TRACE_EVENT(kvm_set_spte_hva,
TP_PROTO(unsigned long hva),
TP_ARGS(hva),
TP_STRUCT__entry(
__field( unsigned long, hva )
),
TP_fast_assign(
__entry->hva = hva;
),
TP_printk("mmu notifier set pte hva: %#08lx", __entry->hva)
);
TRACE_EVENT(kvm_age_hva,
TP_PROTO(unsigned long start, unsigned long end),
TP_ARGS(start, end),
TP_STRUCT__entry(
__field( unsigned long, start )
__field( unsigned long, end )
),
TP_fast_assign(
__entry->start = start;
__entry->end = end;
),
TP_printk("mmu notifier age hva: %#08lx -- %#08lx",
__entry->start, __entry->end)
);
TRACE_EVENT(kvm_test_age_hva,
TP_PROTO(unsigned long hva),
TP_ARGS(hva),
TP_STRUCT__entry(
__field( unsigned long, hva )
),
TP_fast_assign(
__entry->hva = hva;
),
TP_printk("mmu notifier test age hva: %#08lx", __entry->hva)
);
TRACE_EVENT(kvm_set_way_flush,
TP_PROTO(unsigned long vcpu_pc, bool cache),
TP_ARGS(vcpu_pc, cache),
TP_STRUCT__entry(
__field( unsigned long, vcpu_pc )
__field( bool, cache )
),
TP_fast_assign(
__entry->vcpu_pc = vcpu_pc;
__entry->cache = cache;
),
TP_printk("S/W flush at 0x%016lx (cache %s)",
__entry->vcpu_pc, __entry->cache ? "on" : "off")
);
TRACE_EVENT(kvm_toggle_cache,
TP_PROTO(unsigned long vcpu_pc, bool was, bool now),
TP_ARGS(vcpu_pc, was, now),
TP_STRUCT__entry(
__field( unsigned long, vcpu_pc )
__field( bool, was )
__field( bool, now )
),
TP_fast_assign(
__entry->vcpu_pc = vcpu_pc;
__entry->was = was;
__entry->now = now;
),
TP_printk("VM op at 0x%016lx (cache was %s, now %s)",
__entry->vcpu_pc, __entry->was ? "on" : "off",
__entry->now ? "on" : "off")
);
/*
......
virt/kvm/arm/vgic/trace.h 0 → 100644
View file @ 36c344f3
#if !defined(_TRACE_VGIC_H) || defined(TRACE_HEADER_MULTI_READ)
#define _TRACE_VGIC_H
#include <linux/tracepoint.h>
#undef TRACE_SYSTEM
#define TRACE_SYSTEM kvm
TRACE_EVENT(vgic_update_irq_pending,
TP_PROTO(unsigned long vcpu_id, __u32 irq, bool level),
TP_ARGS(vcpu_id, irq, level),
TP_STRUCT__entry(
__field( unsigned long, vcpu_id )
__field( __u32, irq )
__field( bool, level )
),
TP_fast_assign(
__entry->vcpu_id = vcpu_id;
__entry->irq = irq;
__entry->level = level;
),
TP_printk("VCPU: %ld, IRQ %d, level: %d",
__entry->vcpu_id, __entry->irq, __entry->level)
);
#endif /* _TRACE_VGIC_H */
#undef TRACE_INCLUDE_PATH
#define TRACE_INCLUDE_PATH ../../../virt/kvm/arm/vgic
#undef TRACE_INCLUDE_FILE
#define TRACE_INCLUDE_FILE trace
/* This part must be outside protection */
#include <trace/define_trace.h>
virt/kvm/arm/vgic/vgic-init.c
View file @ 36c344f3
......@@ -227,10 +227,27 @@ static int kvm_vgic_dist_init(struct kvm *kvm, unsigned int nr_spis)
}
/**
* kvm_vgic_vcpu_init() - Enable the VCPU interface
* @vcpu: the VCPU which's VGIC should be enabled
* kvm_vgic_vcpu_init() - Register VCPU-specific KVM iodevs
* @vcpu: pointer to the VCPU being created and initialized
*/
static void kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu)
int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu)
{
int ret = 0;
struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
if (!irqchip_in_kernel(vcpu->kvm))
return 0;
/*
* If we are creating a VCPU with a GICv3 we must also register the
* KVM io device for the redistributor that belongs to this VCPU.
*/
if (dist->vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3)
ret = vgic_register_redist_iodev(vcpu);
return ret;
}
static void kvm_vgic_vcpu_enable(struct kvm_vcpu *vcpu)
{
if (kvm_vgic_global_state.type == VGIC_V2)
vgic_v2_enable(vcpu);
......@@ -269,7 +286,7 @@ int vgic_init(struct kvm *kvm)
dist->msis_require_devid = true;
kvm_for_each_vcpu(i, vcpu, kvm)
kvm_vgic_vcpu_init(vcpu);
kvm_vgic_vcpu_enable(vcpu);
ret = kvm_vgic_setup_default_irq_routing(kvm);
if (ret)
......
virt/kvm/arm/vgic/vgic-its.c
View file @ 36c344f3
......@@ -23,6 +23,7 @@
#include <linux/interrupt.h>
#include <linux/list.h>
#include <linux/uaccess.h>
#include <linux/list_sort.h>
#include <linux/irqchip/arm-gic-v3.h>
......@@ -33,6 +34,12 @@
#include "vgic.h"
#include "vgic-mmio.h"
static int vgic_its_save_tables_v0(struct vgic_its *its);
static int vgic_its_restore_tables_v0(struct vgic_its *its);
static int vgic_its_commit_v0(struct vgic_its *its);
static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
struct kvm_vcpu *filter_vcpu);
/*
* Creates a new (reference to a) struct vgic_irq for a given LPI.
* If this LPI is already mapped on another ITS, we increase its refcount
......@@ -40,10 +47,12 @@
* If this is a "new" LPI, we allocate and initialize a new struct vgic_irq.
* This function returns a pointer to the _unlocked_ structure.
*/
static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid)
static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid,
struct kvm_vcpu *vcpu)
{
struct vgic_dist *dist = &kvm->arch.vgic;
struct vgic_irq *irq = vgic_get_irq(kvm, NULL, intid), *oldirq;
int ret;
/* In this case there is no put, since we keep the reference. */
if (irq)
......@@ -60,6 +69,7 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid)
irq->config = VGIC_CONFIG_EDGE;
kref_init(&irq->refcount);
irq->intid = intid;
irq->target_vcpu = vcpu;
spin_lock(&dist->lpi_list_lock);
......@@ -91,6 +101,19 @@ static struct vgic_irq *vgic_add_lpi(struct kvm *kvm, u32 intid)
out_unlock:
spin_unlock(&dist->lpi_list_lock);
/*
* We "cache" the configuration table entries in our struct vgic_irq's.
* However we only have those structs for mapped IRQs, so we read in
* the respective config data from memory here upon mapping the LPI.
*/
ret = update_lpi_config(kvm, irq, NULL);
if (ret)
return ERR_PTR(ret);
ret = vgic_v3_lpi_sync_pending_status(kvm, irq);
if (ret)
return ERR_PTR(ret);
return irq;
}
......@@ -99,6 +122,8 @@ struct its_device {
/* the head for the list of ITTEs */
struct list_head itt_head;
u32 num_eventid_bits;
gpa_t itt_addr;
u32 device_id;
};
......@@ -114,8 +139,8 @@ struct its_collection {
#define its_is_collection_mapped(coll) ((coll) && \
((coll)->target_addr != COLLECTION_NOT_MAPPED))
struct its_itte {
struct list_head itte_list;
struct its_ite {
struct list_head ite_list;
struct vgic_irq *irq;
struct its_collection *collection;
......@@ -123,6 +148,50 @@ struct its_itte {
u32 event_id;
};
/**
* struct vgic_its_abi - ITS abi ops and settings
* @cte_esz: collection table entry size
* @dte_esz: device table entry size
* @ite_esz: interrupt translation table entry size
* @save tables: save the ITS tables into guest RAM
* @restore_tables: restore the ITS internal structs from tables
* stored in guest RAM
* @commit: initialize the registers which expose the ABI settings,
* especially the entry sizes
*/
struct vgic_its_abi {
int cte_esz;
int dte_esz;
int ite_esz;
int (*save_tables)(struct vgic_its *its);
int (*restore_tables)(struct vgic_its *its);
int (*commit)(struct vgic_its *its);
};
static const struct vgic_its_abi its_table_abi_versions[] = {
[0] = {.cte_esz = 8, .dte_esz = 8, .ite_esz = 8,
.save_tables = vgic_its_save_tables_v0,
.restore_tables = vgic_its_restore_tables_v0,
.commit = vgic_its_commit_v0,
},
};
#define NR_ITS_ABIS ARRAY_SIZE(its_table_abi_versions)
inline const struct vgic_its_abi *vgic_its_get_abi(struct vgic_its *its)
{
return &its_table_abi_versions[its->abi_rev];
}
int vgic_its_set_abi(struct vgic_its *its, int rev)
{
const struct vgic_its_abi *abi;
its->abi_rev = rev;
abi = vgic_its_get_abi(its);
return abi->commit(its);
}
/*
* Find and returns a device in the device table for an ITS.
* Must be called with the its_lock mutex held.
......@@ -143,27 +212,27 @@ static struct its_device *find_its_device(struct vgic_its *its, u32 device_id)
* Device ID/Event ID pair on an ITS.
* Must be called with the its_lock mutex held.
*/
static struct its_itte *find_itte(struct vgic_its *its, u32 device_id,
static struct its_ite *find_ite(struct vgic_its *its, u32 device_id,
u32 event_id)
{
struct its_device *device;
struct its_itte *itte;
struct its_ite *ite;
device = find_its_device(its, device_id);
if (device == NULL)
return NULL;
list_for_each_entry(itte, &device->itt_head, itte_list)
if (itte->event_id == event_id)
return itte;
list_for_each_entry(ite, &device->itt_head, ite_list)
if (ite->event_id == event_id)
return ite;
return NULL;
}
/* To be used as an iterator this macro misses the enclosing parentheses */
#define for_each_lpi_its(dev, itte, its) \
#define for_each_lpi_its(dev, ite, its) \
list_for_each_entry(dev, &(its)->device_list, dev_list) \
list_for_each_entry(itte, &(dev)->itt_head, itte_list)
list_for_each_entry(ite, &(dev)->itt_head, ite_list)
/*
* We only implement 48 bits of PA at the moment, although the ITS
......@@ -171,11 +240,14 @@ static struct its_itte *find_itte(struct vgic_its *its, u32 device_id,
*/
#define BASER_ADDRESS(x) ((x) & GENMASK_ULL(47, 16))
#define CBASER_ADDRESS(x) ((x) & GENMASK_ULL(47, 12))
#define PENDBASER_ADDRESS(x) ((x) & GENMASK_ULL(47, 16))
#define PROPBASER_ADDRESS(x) ((x) & GENMASK_ULL(47, 12))
#define GIC_LPI_OFFSET 8192
#define VITS_TYPER_IDBITS 16
#define VITS_TYPER_DEVBITS 16
#define VITS_DTE_MAX_DEVID_OFFSET (BIT(14) - 1)
#define VITS_ITE_MAX_EVENTID_OFFSET (BIT(16) - 1)
/*
* Finds and returns a collection in the ITS collection table.
* Must be called with the its_lock mutex held.
......@@ -204,7 +276,7 @@ static struct its_collection *find_collection(struct vgic_its *its, int coll_id)
static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
struct kvm_vcpu *filter_vcpu)
{
u64 propbase = PROPBASER_ADDRESS(kvm->arch.vgic.propbaser);
u64 propbase = GICR_PROPBASER_ADDRESS(kvm->arch.vgic.propbaser);
u8 prop;
int ret;
......@@ -229,13 +301,13 @@ static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
}
/*
* Create a snapshot of the current LPI list, so that we can enumerate all
* LPIs without holding any lock.
* Returns the array length and puts the kmalloc'ed array into intid_ptr.
* Create a snapshot of the current LPIs targeting @vcpu, so that we can
* enumerate those LPIs without holding any lock.
* Returns their number and puts the kmalloc'ed array into intid_ptr.
*/
static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
static int vgic_copy_lpi_list(struct kvm_vcpu *vcpu, u32 **intid_ptr)
{
struct vgic_dist *dist = &kvm->arch.vgic;
struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
struct vgic_irq *irq;
u32 *intids;
int irq_count = dist->lpi_list_count, i = 0;
......@@ -254,14 +326,14 @@ static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
spin_lock(&dist->lpi_list_lock);
list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
/* We don't need to "get" the IRQ, as we hold the list lock. */
intids[i] = irq->intid;
if (++i == irq_count)
break;
if (irq->target_vcpu != vcpu)
continue;
intids[i++] = irq->intid;
}
spin_unlock(&dist->lpi_list_lock);
*intid_ptr = intids;
return irq_count;
return i;
}
/*
......@@ -270,18 +342,18 @@ static int vgic_copy_lpi_list(struct kvm *kvm, u32 **intid_ptr)
* Needs to be called whenever either the collection for a LPIs has
* changed or the collection itself got retargeted.
*/
static void update_affinity_itte(struct kvm *kvm, struct its_itte *itte)
static void update_affinity_ite(struct kvm *kvm, struct its_ite *ite)
{
struct kvm_vcpu *vcpu;
if (!its_is_collection_mapped(itte->collection))
if (!its_is_collection_mapped(ite->collection))
return;
vcpu = kvm_get_vcpu(kvm, itte->collection->target_addr);
vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
spin_lock(&itte->irq->irq_lock);
itte->irq->target_vcpu = vcpu;
spin_unlock(&itte->irq->irq_lock);
spin_lock(&ite->irq->irq_lock);
ite->irq->target_vcpu = vcpu;
spin_unlock(&ite->irq->irq_lock);
}
/*
......@@ -292,13 +364,13 @@ static void update_affinity_collection(struct kvm *kvm, struct vgic_its *its,
struct its_collection *coll)
{
struct its_device *device;
struct its_itte *itte;
struct its_ite *ite;
for_each_lpi_its(device, itte, its) {
if (!itte->collection || coll != itte->collection)
for_each_lpi_its(device, ite, its) {
if (!ite->collection || coll != ite->collection)
continue;
update_affinity_itte(kvm, itte);
update_affinity_ite(kvm, ite);
}
}
......@@ -310,20 +382,20 @@ static u32 max_lpis_propbaser(u64 propbaser)
}
/*
* Scan the whole LPI pending table and sync the pending bit in there
* Sync the pending table pending bit of LPIs targeting @vcpu
* with our own data structures. This relies on the LPI being
* mapped before.
*/
static int its_sync_lpi_pending_table(struct kvm_vcpu *vcpu)
{
gpa_t pendbase = PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
gpa_t pendbase = GICR_PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
struct vgic_irq *irq;
int last_byte_offset = -1;
int ret = 0;
u32 *intids;
int nr_irqs, i;
nr_irqs = vgic_copy_lpi_list(vcpu->kvm, &intids);
nr_irqs = vgic_copy_lpi_list(vcpu, &intids);
if (nr_irqs < 0)
return nr_irqs;
......@@ -364,6 +436,7 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
struct vgic_its *its,
gpa_t addr, unsigned int len)
{
const struct vgic_its_abi *abi = vgic_its_get_abi(its);
u64 reg = GITS_TYPER_PLPIS;
/*
......@@ -374,8 +447,9 @@ static unsigned long vgic_mmio_read_its_typer(struct kvm *kvm,
* To avoid memory waste in the guest, we keep the number of IDBits and
* DevBits low - as least for the time being.
*/
reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
reg |= GIC_ENCODE_SZ(VITS_TYPER_DEVBITS, 5) << GITS_TYPER_DEVBITS_SHIFT;
reg |= GIC_ENCODE_SZ(VITS_TYPER_IDBITS, 5) << GITS_TYPER_IDBITS_SHIFT;
reg |= GIC_ENCODE_SZ(abi->ite_esz, 4) << GITS_TYPER_ITT_ENTRY_SIZE_SHIFT;
return extract_bytes(reg, addr & 7, len);
}
......@@ -384,7 +458,23 @@ static unsigned long vgic_mmio_read_its_iidr(struct kvm *kvm,
struct vgic_its *its,
gpa_t addr, unsigned int len)
{
return (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
u32 val;
val = (its->abi_rev << GITS_IIDR_REV_SHIFT) & GITS_IIDR_REV_MASK;
val |= (PRODUCT_ID_KVM << GITS_IIDR_PRODUCTID_SHIFT) | IMPLEMENTER_ARM;
return val;
}
static int vgic_mmio_uaccess_write_its_iidr(struct kvm *kvm,
struct vgic_its *its,
gpa_t addr, unsigned int len,
unsigned long val)
{
u32 rev = GITS_IIDR_REV(val);
if (rev >= NR_ITS_ABIS)
return -EINVAL;
return vgic_its_set_abi(its, rev);
}
static unsigned long vgic_mmio_read_its_idregs(struct kvm *kvm,
......@@ -425,25 +515,25 @@ static int vgic_its_trigger_msi(struct kvm *kvm, struct vgic_its *its,
u32 devid, u32 eventid)
{
struct kvm_vcpu *vcpu;
struct its_itte *itte;
struct its_ite *ite;
if (!its->enabled)
return -EBUSY;
itte = find_itte(its, devid, eventid);
if (!itte || !its_is_collection_mapped(itte->collection))
ite = find_ite(its, devid, eventid);
if (!ite || !its_is_collection_mapped(ite->collection))
return E_ITS_INT_UNMAPPED_INTERRUPT;
vcpu = kvm_get_vcpu(kvm, itte->collection->target_addr);
vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
if (!vcpu)
return E_ITS_INT_UNMAPPED_INTERRUPT;
if (!vcpu->arch.vgic_cpu.lpis_enabled)
return -EBUSY;
spin_lock(&itte->irq->irq_lock);
itte->irq->pending_latch = true;
vgic_queue_irq_unlock(kvm, itte->irq);
spin_lock(&ite->irq->irq_lock);
ite->irq->pending_latch = true;
vgic_queue_irq_unlock(kvm, ite->irq);
return 0;
}
......@@ -511,15 +601,15 @@ int vgic_its_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
}
/* Requires the its_lock to be held. */
static void its_free_itte(struct kvm *kvm, struct its_itte *itte)
static void its_free_ite(struct kvm *kvm, struct its_ite *ite)
{
list_del(&itte->itte_list);
list_del(&ite->ite_list);
/* This put matches the get in vgic_add_lpi. */
if (itte->irq)
vgic_put_irq(kvm, itte->irq);
if (ite->irq)
vgic_put_irq(kvm, ite->irq);
kfree(itte);
kfree(ite);
}
static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
......@@ -529,9 +619,11 @@ static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
#define its_cmd_get_command(cmd) its_cmd_mask_field(cmd, 0, 0, 8)
#define its_cmd_get_deviceid(cmd) its_cmd_mask_field(cmd, 0, 32, 32)
#define its_cmd_get_size(cmd) (its_cmd_mask_field(cmd, 1, 0, 5) + 1)
#define its_cmd_get_id(cmd) its_cmd_mask_field(cmd, 1, 0, 32)
#define its_cmd_get_physical_id(cmd) its_cmd_mask_field(cmd, 1, 32, 32)
#define its_cmd_get_collection(cmd) its_cmd_mask_field(cmd, 2, 0, 16)
#define its_cmd_get_ittaddr(cmd) (its_cmd_mask_field(cmd, 2, 8, 44) << 8)
#define its_cmd_get_target_addr(cmd) its_cmd_mask_field(cmd, 2, 16, 32)
#define its_cmd_get_validbit(cmd) its_cmd_mask_field(cmd, 2, 63, 1)
......@@ -544,17 +636,17 @@ static int vgic_its_cmd_handle_discard(struct kvm *kvm, struct vgic_its *its,
{
u32 device_id = its_cmd_get_deviceid(its_cmd);
u32 event_id = its_cmd_get_id(its_cmd);
struct its_itte *itte;
struct its_ite *ite;
itte = find_itte(its, device_id, event_id);
if (itte && itte->collection) {
ite = find_ite(its, device_id, event_id);
if (ite && ite->collection) {
/*
* Though the spec talks about removing the pending state, we
* don't bother here since we clear the ITTE anyway and the
* pending state is a property of the ITTE struct.
*/
its_free_itte(kvm, itte);
its_free_ite(kvm, ite);
return 0;
}
......@@ -572,26 +664,26 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
u32 event_id = its_cmd_get_id(its_cmd);
u32 coll_id = its_cmd_get_collection(its_cmd);
struct kvm_vcpu *vcpu;
struct its_itte *itte;
struct its_ite *ite;
struct its_collection *collection;
itte = find_itte(its, device_id, event_id);
if (!itte)
ite = find_ite(its, device_id, event_id);
if (!ite)
return E_ITS_MOVI_UNMAPPED_INTERRUPT;
if (!its_is_collection_mapped(itte->collection))
if (!its_is_collection_mapped(ite->collection))
return E_ITS_MOVI_UNMAPPED_COLLECTION;
collection = find_collection(its, coll_id);
if (!its_is_collection_mapped(collection))
return E_ITS_MOVI_UNMAPPED_COLLECTION;
itte->collection = collection;
ite->collection = collection;
vcpu = kvm_get_vcpu(kvm, collection->target_addr);
spin_lock(&itte->irq->irq_lock);
itte->irq->target_vcpu = vcpu;
spin_unlock(&itte->irq->irq_lock);
spin_lock(&ite->irq->irq_lock);
ite->irq->target_vcpu = vcpu;
spin_unlock(&ite->irq->irq_lock);
return 0;
}
......@@ -600,16 +692,31 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
* Check whether an ID can be stored into the corresponding guest table.
* For a direct table this is pretty easy, but gets a bit nasty for
* indirect tables. We check whether the resulting guest physical address
* is actually valid (covered by a memslot and guest accessbible).
* is actually valid (covered by a memslot and guest accessible).
* For this we have to read the respective first level entry.
*/
static bool vgic_its_check_id(struct vgic_its *its, u64 baser, int id)
static bool vgic_its_check_id(struct vgic_its *its, u64 baser, u32 id,
gpa_t *eaddr)
{
int l1_tbl_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
u64 indirect_ptr, type = GITS_BASER_TYPE(baser);
int esz = GITS_BASER_ENTRY_SIZE(baser);
int index;
u64 indirect_ptr;
gfn_t gfn;
int esz = GITS_BASER_ENTRY_SIZE(baser);
switch (type) {
case GITS_BASER_TYPE_DEVICE:
if (id >= BIT_ULL(VITS_TYPER_DEVBITS))
return false;
break;
case GITS_BASER_TYPE_COLLECTION:
/* as GITS_TYPER.CIL == 0, ITS supports 16-bit collection ID */
if (id >= BIT_ULL(16))
return false;
break;
default:
return false;
}
if (!(baser & GITS_BASER_INDIRECT)) {
phys_addr_t addr;
......@@ -620,6 +727,8 @@ static bool vgic_its_check_id(struct vgic_its *its, u64 baser, int id)
addr = BASER_ADDRESS(baser) + id * esz;
gfn = addr >> PAGE_SHIFT;
if (eaddr)
*eaddr = addr;
return kvm_is_visible_gfn(its->dev->kvm, gfn);
}
......@@ -652,6 +761,8 @@ static bool vgic_its_check_id(struct vgic_its *its, u64 baser, int id)
indirect_ptr += index * esz;
gfn = indirect_ptr >> PAGE_SHIFT;
if (eaddr)
*eaddr = indirect_ptr;
return kvm_is_visible_gfn(its->dev->kvm, gfn);
}
......@@ -661,7 +772,7 @@ static int vgic_its_alloc_collection(struct vgic_its *its,
{
struct its_collection *collection;
if (!vgic_its_check_id(its, its->baser_coll_table, coll_id))
if (!vgic_its_check_id(its, its->baser_coll_table, coll_id, NULL))
return E_ITS_MAPC_COLLECTION_OOR;
collection = kzalloc(sizeof(*collection), GFP_KERNEL);
......@@ -679,7 +790,7 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
{
struct its_collection *collection;
struct its_device *device;
struct its_itte *itte;
struct its_ite *ite;
/*
* Clearing the mapping for that collection ID removes the
......@@ -690,15 +801,34 @@ static void vgic_its_free_collection(struct vgic_its *its, u32 coll_id)
if (!collection)
return;
for_each_lpi_its(device, itte, its)
if (itte->collection &&
itte->collection->collection_id == coll_id)
itte->collection = NULL;
for_each_lpi_its(device, ite, its)
if (ite->collection &&
ite->collection->collection_id == coll_id)
ite->collection = NULL;
list_del(&collection->coll_list);
kfree(collection);
}
/* Must be called with its_lock mutex held */
static struct its_ite *vgic_its_alloc_ite(struct its_device *device,
struct its_collection *collection,
u32 lpi_id, u32 event_id)
{
struct its_ite *ite;
ite = kzalloc(sizeof(*ite), GFP_KERNEL);
if (!ite)
return ERR_PTR(-ENOMEM);
ite->event_id = event_id;
ite->collection = collection;
ite->lpi = lpi_id;
list_add_tail(&ite->ite_list, &device->itt_head);
return ite;
}
/*
* The MAPTI and MAPI commands map LPIs to ITTEs.
* Must be called with its_lock mutex held.
......@@ -709,16 +839,20 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
u32 device_id = its_cmd_get_deviceid(its_cmd);
u32 event_id = its_cmd_get_id(its_cmd);
u32 coll_id = its_cmd_get_collection(its_cmd);
struct its_itte *itte;
struct its_ite *ite;
struct kvm_vcpu *vcpu = NULL;
struct its_device *device;
struct its_collection *collection, *new_coll = NULL;
int lpi_nr;
struct vgic_irq *irq;
int lpi_nr;
device = find_its_device(its, device_id);
if (!device)
return E_ITS_MAPTI_UNMAPPED_DEVICE;
if (event_id >= BIT_ULL(device->num_eventid_bits))
return E_ITS_MAPTI_ID_OOR;
if (its_cmd_get_command(its_cmd) == GITS_CMD_MAPTI)
lpi_nr = its_cmd_get_physical_id(its_cmd);
else
......@@ -728,7 +862,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
return E_ITS_MAPTI_PHYSICALID_OOR;
/* If there is an existing mapping, behavior is UNPREDICTABLE. */
if (find_itte(its, device_id, event_id))
if (find_ite(its, device_id, event_id))
return 0;
collection = find_collection(its, coll_id);
......@@ -739,36 +873,24 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
new_coll = collection;
}
itte = kzalloc(sizeof(struct its_itte), GFP_KERNEL);
if (!itte) {
ite = vgic_its_alloc_ite(device, collection, lpi_nr, event_id);
if (IS_ERR(ite)) {
if (new_coll)
vgic_its_free_collection(its, coll_id);
return -ENOMEM;
return PTR_ERR(ite);
}
itte->event_id = event_id;
list_add_tail(&itte->itte_list, &device->itt_head);
itte->collection = collection;
itte->lpi = lpi_nr;
if (its_is_collection_mapped(collection))
vcpu = kvm_get_vcpu(kvm, collection->target_addr);
irq = vgic_add_lpi(kvm, lpi_nr);
irq = vgic_add_lpi(kvm, lpi_nr, vcpu);
if (IS_ERR(irq)) {
if (new_coll)
vgic_its_free_collection(its, coll_id);
its_free_itte(kvm, itte);
its_free_ite(kvm, ite);
return PTR_ERR(irq);
}
itte->irq = irq;
update_affinity_itte(kvm, itte);
/*
* We "cache" the configuration table entries in out struct vgic_irq's.
* However we only have those structs for mapped IRQs, so we read in
* the respective config data from memory here upon mapping the LPI.
*/
update_lpi_config(kvm, itte->irq, NULL);
ite->irq = irq;
return 0;
}
......@@ -776,20 +898,40 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
/* Requires the its_lock to be held. */
static void vgic_its_unmap_device(struct kvm *kvm, struct its_device *device)
{
struct its_itte *itte, *temp;
struct its_ite *ite, *temp;
/*
* The spec says that unmapping a device with still valid
* ITTEs associated is UNPREDICTABLE. We remove all ITTEs,
* since we cannot leave the memory unreferenced.
*/
list_for_each_entry_safe(itte, temp, &device->itt_head, itte_list)
its_free_itte(kvm, itte);
list_for_each_entry_safe(ite, temp, &device->itt_head, ite_list)
its_free_ite(kvm, ite);
list_del(&device->dev_list);
kfree(device);
}
/* Must be called with its_lock mutex held */
static struct its_device *vgic_its_alloc_device(struct vgic_its *its,
u32 device_id, gpa_t itt_addr,
u8 num_eventid_bits)
{
struct its_device *device;
device = kzalloc(sizeof(*device), GFP_KERNEL);
if (!device)
return ERR_PTR(-ENOMEM);
device->device_id = device_id;
device->itt_addr = itt_addr;
device->num_eventid_bits = num_eventid_bits;
INIT_LIST_HEAD(&device->itt_head);
list_add_tail(&device->dev_list, &its->device_list);
return device;
}
/*
* MAPD maps or unmaps a device ID to Interrupt Translation Tables (ITTs).
* Must be called with the its_lock mutex held.
......@@ -799,11 +941,16 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
{
u32 device_id = its_cmd_get_deviceid(its_cmd);
bool valid = its_cmd_get_validbit(its_cmd);
u8 num_eventid_bits = its_cmd_get_size(its_cmd);
gpa_t itt_addr = its_cmd_get_ittaddr(its_cmd);
struct its_device *device;
if (!vgic_its_check_id(its, its->baser_device_table, device_id))
if (!vgic_its_check_id(its, its->baser_device_table, device_id, NULL))
return E_ITS_MAPD_DEVICE_OOR;
if (valid && num_eventid_bits > VITS_TYPER_IDBITS)
return E_ITS_MAPD_ITTSIZE_OOR;
device = find_its_device(its, device_id);
/*
......@@ -821,14 +968,10 @@ static int vgic_its_cmd_handle_mapd(struct kvm *kvm, struct vgic_its *its,
if (!valid)
return 0;
device = kzalloc(sizeof(struct its_device), GFP_KERNEL);
if (!device)
return -ENOMEM;
device->device_id = device_id;
INIT_LIST_HEAD(&device->itt_head);
list_add_tail(&device->dev_list, &its->device_list);
device = vgic_its_alloc_device(its, device_id, itt_addr,
num_eventid_bits);
if (IS_ERR(device))
return PTR_ERR(device);
return 0;
}
......@@ -883,14 +1026,14 @@ static int vgic_its_cmd_handle_clear(struct kvm *kvm, struct vgic_its *its,
{
u32 device_id = its_cmd_get_deviceid(its_cmd);
u32 event_id = its_cmd_get_id(its_cmd);
struct its_itte *itte;
struct its_ite *ite;
itte = find_itte(its, device_id, event_id);
if (!itte)
ite = find_ite(its, device_id, event_id);
if (!ite)
return E_ITS_CLEAR_UNMAPPED_INTERRUPT;
itte->irq->pending_latch = false;
ite->irq->pending_latch = false;
return 0;
}
......@@ -904,14 +1047,14 @@ static int vgic_its_cmd_handle_inv(struct kvm *kvm, struct vgic_its *its,
{
u32 device_id = its_cmd_get_deviceid(its_cmd);
u32 event_id = its_cmd_get_id(its_cmd);
struct its_itte *itte;
struct its_ite *ite;
itte = find_itte(its, device_id, event_id);
if (!itte)
ite = find_ite(its, device_id, event_id);
if (!ite)
return E_ITS_INV_UNMAPPED_INTERRUPT;
return update_lpi_config(kvm, itte->irq, NULL);
return update_lpi_config(kvm, ite->irq, NULL);
}
/*
......@@ -938,7 +1081,7 @@ static int vgic_its_cmd_handle_invall(struct kvm *kvm, struct vgic_its *its,
vcpu = kvm_get_vcpu(kvm, collection->target_addr);
irq_count = vgic_copy_lpi_list(kvm, &intids);
irq_count = vgic_copy_lpi_list(vcpu, &intids);
if (irq_count < 0)
return irq_count;
......@@ -1213,6 +1356,33 @@ static unsigned long vgic_mmio_read_its_creadr(struct kvm *kvm,
return extract_bytes(its->creadr, addr & 0x7, len);
}
static int vgic_mmio_uaccess_write_its_creadr(struct kvm *kvm,
struct vgic_its *its,
gpa_t addr, unsigned int len,
unsigned long val)
{
u32 cmd_offset;
int ret = 0;
mutex_lock(&its->cmd_lock);
if (its->enabled) {
ret = -EBUSY;
goto out;
}
cmd_offset = ITS_CMD_OFFSET(val);
if (cmd_offset >= ITS_CMD_BUFFER_SIZE(its->cbaser)) {
ret = -EINVAL;
goto out;
}
its->creadr = cmd_offset;
out:
mutex_unlock(&its->cmd_lock);
return ret;
}
#define BASER_INDEX(addr) (((addr) / sizeof(u64)) & 0x7)
static unsigned long vgic_mmio_read_its_baser(struct kvm *kvm,
struct vgic_its *its,
......@@ -1241,6 +1411,7 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
gpa_t addr, unsigned int len,
unsigned long val)
{
const struct vgic_its_abi *abi = vgic_its_get_abi(its);
u64 entry_size, device_type;
u64 reg, *regptr, clearbits = 0;
......@@ -1251,12 +1422,12 @@ static void vgic_mmio_write_its_baser(struct kvm *kvm,
switch (BASER_INDEX(addr)) {
case 0:
regptr = &its->baser_device_table;
entry_size = 8;
entry_size = abi->dte_esz;
device_type = GITS_BASER_TYPE_DEVICE;
break;
case 1:
regptr = &its->baser_coll_table;
entry_size = 8;
entry_size = abi->cte_esz;
device_type = GITS_BASER_TYPE_COLLECTION;
clearbits = GITS_BASER_INDIRECT;
break;
......@@ -1317,6 +1488,16 @@ static void vgic_mmio_write_its_ctlr(struct kvm *kvm, struct vgic_its *its,
.its_write = wr, \
}
#define REGISTER_ITS_DESC_UACCESS(off, rd, wr, uwr, length, acc)\
{ \
.reg_offset = off, \
.len = length, \
.access_flags = acc, \
.its_read = rd, \
.its_write = wr, \
.uaccess_its_write = uwr, \
}
static void its_mmio_write_wi(struct kvm *kvm, struct vgic_its *its,
gpa_t addr, unsigned int len, unsigned long val)
{
......@@ -1327,8 +1508,9 @@ static struct vgic_register_region its_registers[] = {
REGISTER_ITS_DESC(GITS_CTLR,
vgic_mmio_read_its_ctlr, vgic_mmio_write_its_ctlr, 4,
VGIC_ACCESS_32bit),
REGISTER_ITS_DESC(GITS_IIDR,
vgic_mmio_read_its_iidr, its_mmio_write_wi, 4,
REGISTER_ITS_DESC_UACCESS(GITS_IIDR,
vgic_mmio_read_its_iidr, its_mmio_write_wi,
vgic_mmio_uaccess_write_its_iidr, 4,
VGIC_ACCESS_32bit),
REGISTER_ITS_DESC(GITS_TYPER,
vgic_mmio_read_its_typer, its_mmio_write_wi, 8,
......@@ -1339,8 +1521,9 @@ static struct vgic_register_region its_registers[] = {
REGISTER_ITS_DESC(GITS_CWRITER,
vgic_mmio_read_its_cwriter, vgic_mmio_write_its_cwriter, 8,
VGIC_ACCESS_64bit | VGIC_ACCESS_32bit),
REGISTER_ITS_DESC(GITS_CREADR,
vgic_mmio_read_its_creadr, its_mmio_write_wi, 8,
REGISTER_ITS_DESC_UACCESS(GITS_CREADR,
vgic_mmio_read_its_creadr, its_mmio_write_wi,
vgic_mmio_uaccess_write_its_creadr, 8,
VGIC_ACCESS_64bit | VGIC_ACCESS_32bit),
REGISTER_ITS_DESC(GITS_BASER,
vgic_mmio_read_its_baser, vgic_mmio_write_its_baser, 0x40,
......@@ -1357,17 +1540,19 @@ void vgic_enable_lpis(struct kvm_vcpu *vcpu)
its_sync_lpi_pending_table(vcpu);
}
static int vgic_register_its_iodev(struct kvm *kvm, struct vgic_its *its)
static int vgic_register_its_iodev(struct kvm *kvm, struct vgic_its *its,
u64 addr)
{
struct vgic_io_device *iodev = &its->iodev;
int ret;
if (!its->initialized)
return -EBUSY;
if (IS_VGIC_ADDR_UNDEF(its->vgic_its_base))
return -ENXIO;
mutex_lock(&kvm->slots_lock);
if (!IS_VGIC_ADDR_UNDEF(its->vgic_its_base)) {
ret = -EBUSY;
goto out;
}
its->vgic_its_base = addr;
iodev->regions = its_registers;
iodev->nr_regions = ARRAY_SIZE(its_registers);
kvm_iodevice_init(&iodev->dev, &kvm_io_gic_ops);
......@@ -1375,9 +1560,9 @@ static int vgic_register_its_iodev(struct kvm *kvm, struct vgic_its *its)
iodev->base_addr = its->vgic_its_base;
iodev->iodev_type = IODEV_ITS;
iodev->its = its;
mutex_lock(&kvm->slots_lock);
ret = kvm_io_bus_register_dev(kvm, KVM_MMIO_BUS, iodev->base_addr,
KVM_VGIC_V3_ITS_SIZE, &iodev->dev);
out:
mutex_unlock(&kvm->slots_lock);
return ret;
......@@ -1387,7 +1572,6 @@ static int vgic_register_its_iodev(struct kvm *kvm, struct vgic_its *its)
(GIC_BASER_CACHEABILITY(GITS_BASER, INNER, RaWb) | \
GIC_BASER_CACHEABILITY(GITS_BASER, OUTER, SameAsInner) | \
GIC_BASER_SHAREABILITY(GITS_BASER, InnerShareable) | \
((8ULL - 1) << GITS_BASER_ENTRY_SIZE_SHIFT) | \
GITS_BASER_PAGE_SIZE_64K)
#define INITIAL_PROPBASER_VALUE \
......@@ -1415,7 +1599,6 @@ static int vgic_its_create(struct kvm_device *dev, u32 type)
INIT_LIST_HEAD(&its->collection_list);
dev->kvm->arch.vgic.has_its = true;
its->initialized = false;
its->enabled = false;
its->dev = dev;
......@@ -1427,16 +1610,23 @@ static int vgic_its_create(struct kvm_device *dev, u32 type)
dev->private = its;
return 0;
return vgic_its_set_abi(its, NR_ITS_ABIS - 1);
}
static void vgic_its_free_device(struct kvm *kvm, struct its_device *dev)
{
struct its_ite *ite, *tmp;
list_for_each_entry_safe(ite, tmp, &dev->itt_head, ite_list)
its_free_ite(kvm, ite);
list_del(&dev->dev_list);
kfree(dev);
}
static void vgic_its_destroy(struct kvm_device *kvm_dev)
{
struct kvm *kvm = kvm_dev->kvm;
struct vgic_its *its = kvm_dev->private;
struct its_device *dev;
struct its_itte *itte;
struct list_head *dev_cur, *dev_temp;
struct list_head *cur, *temp;
/*
......@@ -1447,25 +1637,710 @@ static void vgic_its_destroy(struct kvm_device *kvm_dev)
return;
mutex_lock(&its->its_lock);
list_for_each_safe(dev_cur, dev_temp, &its->device_list) {
dev = container_of(dev_cur, struct its_device, dev_list);
list_for_each_safe(cur, temp, &dev->itt_head) {
itte = (container_of(cur, struct its_itte, itte_list));
its_free_itte(kvm, itte);
}
list_del(dev_cur);
kfree(dev);
list_for_each_safe(cur, temp, &its->device_list) {
struct its_device *dev;
dev = list_entry(cur, struct its_device, dev_list);
vgic_its_free_device(kvm, dev);
}
list_for_each_safe(cur, temp, &its->collection_list) {
struct its_collection *coll;
coll = list_entry(cur, struct its_collection, coll_list);
list_del(cur);
kfree(container_of(cur, struct its_collection, coll_list));
kfree(coll);
}
mutex_unlock(&its->its_lock);
kfree(its);
}
int vgic_its_has_attr_regs(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
const struct vgic_register_region *region;
gpa_t offset = attr->attr;
int align;
align = (offset < GITS_TYPER) || (offset >= GITS_PIDR4) ? 0x3 : 0x7;
if (offset & align)
return -EINVAL;
region = vgic_find_mmio_region(its_registers,
ARRAY_SIZE(its_registers),
offset);
if (!region)
return -ENXIO;
return 0;
}
int vgic_its_attr_regs_access(struct kvm_device *dev,
struct kvm_device_attr *attr,
u64 *reg, bool is_write)
{
const struct vgic_register_region *region;
struct vgic_its *its;
gpa_t addr, offset;
unsigned int len;
int align, ret = 0;
its = dev->private;
offset = attr->attr;
/*
* Although the spec supports upper/lower 32-bit accesses to
* 64-bit ITS registers, the userspace ABI requires 64-bit
* accesses to all 64-bit wide registers. We therefore only
* support 32-bit accesses to GITS_CTLR, GITS_IIDR and GITS ID
* registers
*/
if ((offset < GITS_TYPER) || (offset >= GITS_PIDR4))
align = 0x3;
else
align = 0x7;
if (offset & align)
return -EINVAL;
mutex_lock(&dev->kvm->lock);
if (IS_VGIC_ADDR_UNDEF(its->vgic_its_base)) {
ret = -ENXIO;
goto out;
}
region = vgic_find_mmio_region(its_registers,
ARRAY_SIZE(its_registers),
offset);
if (!region) {
ret = -ENXIO;
goto out;
}
if (!lock_all_vcpus(dev->kvm)) {
ret = -EBUSY;
goto out;
}
addr = its->vgic_its_base + offset;
len = region->access_flags & VGIC_ACCESS_64bit ? 8 : 4;
if (is_write) {
if (region->uaccess_its_write)
ret = region->uaccess_its_write(dev->kvm, its, addr,
len, *reg);
else
region->its_write(dev->kvm, its, addr, len, *reg);
} else {
*reg = region->its_read(dev->kvm, its, addr, len);
}
unlock_all_vcpus(dev->kvm);
out:
mutex_unlock(&dev->kvm->lock);
return ret;
}
static u32 compute_next_devid_offset(struct list_head *h,
struct its_device *dev)
{
struct its_device *next;
u32 next_offset;
if (list_is_last(&dev->dev_list, h))
return 0;
next = list_next_entry(dev, dev_list);
next_offset = next->device_id - dev->device_id;
return min_t(u32, next_offset, VITS_DTE_MAX_DEVID_OFFSET);
}
static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite)
{
struct its_ite *next;
u32 next_offset;
if (list_is_last(&ite->ite_list, h))
return 0;
next = list_next_entry(ite, ite_list);
next_offset = next->event_id - ite->event_id;
return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET);
}
/**
* entry_fn_t - Callback called on a table entry restore path
* @its: its handle
* @id: id of the entry
* @entry: pointer to the entry
* @opaque: pointer to an opaque data
*
* Return: < 0 on error, 0 if last element was identified, id offset to next
* element otherwise
*/
typedef int (*entry_fn_t)(struct vgic_its *its, u32 id, void *entry,
void *opaque);
/**
* scan_its_table - Scan a contiguous table in guest RAM and applies a function
* to each entry
*
* @its: its handle
* @base: base gpa of the table
* @size: size of the table in bytes
* @esz: entry size in bytes
* @start_id: the ID of the first entry in the table
* (non zero for 2d level tables)
* @fn: function to apply on each entry
*
* Return: < 0 on error, 0 if last element was identified, 1 otherwise
* (the last element may not be found on second level tables)
*/
static int scan_its_table(struct vgic_its *its, gpa_t base, int size, int esz,
int start_id, entry_fn_t fn, void *opaque)
{
void *entry = kzalloc(esz, GFP_KERNEL);
struct kvm *kvm = its->dev->kvm;
unsigned long len = size;
int id = start_id;
gpa_t gpa = base;
int ret;
while (len > 0) {
int next_offset;
size_t byte_offset;
ret = kvm_read_guest(kvm, gpa, entry, esz);
if (ret)
goto out;
next_offset = fn(its, id, entry, opaque);
if (next_offset <= 0) {
ret = next_offset;
goto out;
}
byte_offset = next_offset * esz;
id += next_offset;
gpa += byte_offset;
len -= byte_offset;
}
ret = 1;
out:
kfree(entry);
return ret;
}
/**
* vgic_its_save_ite - Save an interrupt translation entry at @gpa
*/
static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
struct its_ite *ite, gpa_t gpa, int ite_esz)
{
struct kvm *kvm = its->dev->kvm;
u32 next_offset;
u64 val;
next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
((u64)ite->lpi << KVM_ITS_ITE_PINTID_SHIFT) |
ite->collection->collection_id;
val = cpu_to_le64(val);
return kvm_write_guest(kvm, gpa, &val, ite_esz);
}
/**
* vgic_its_restore_ite - restore an interrupt translation entry
* @event_id: id used for indexing
* @ptr: pointer to the ITE entry
* @opaque: pointer to the its_device
*/
static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
void *ptr, void *opaque)
{
struct its_device *dev = (struct its_device *)opaque;
struct its_collection *collection;
struct kvm *kvm = its->dev->kvm;
struct kvm_vcpu *vcpu = NULL;
u64 val;
u64 *p = (u64 *)ptr;
struct vgic_irq *irq;
u32 coll_id, lpi_id;
struct its_ite *ite;
u32 offset;
val = *p;
val = le64_to_cpu(val);
coll_id = val & KVM_ITS_ITE_ICID_MASK;
lpi_id = (val & KVM_ITS_ITE_PINTID_MASK) >> KVM_ITS_ITE_PINTID_SHIFT;
if (!lpi_id)
return 1; /* invalid entry, no choice but to scan next entry */
if (lpi_id < VGIC_MIN_LPI)
return -EINVAL;
offset = val >> KVM_ITS_ITE_NEXT_SHIFT;
if (event_id + offset >= BIT_ULL(dev->num_eventid_bits))
return -EINVAL;
collection = find_collection(its, coll_id);
if (!collection)
return -EINVAL;
ite = vgic_its_alloc_ite(dev, collection, lpi_id, event_id);
if (IS_ERR(ite))
return PTR_ERR(ite);
if (its_is_collection_mapped(collection))
vcpu = kvm_get_vcpu(kvm, collection->target_addr);
irq = vgic_add_lpi(kvm, lpi_id, vcpu);
if (IS_ERR(irq))
return PTR_ERR(irq);
ite->irq = irq;
return offset;
}
static int vgic_its_ite_cmp(void *priv, struct list_head *a,
struct list_head *b)
{
struct its_ite *itea = container_of(a, struct its_ite, ite_list);
struct its_ite *iteb = container_of(b, struct its_ite, ite_list);
if (itea->event_id < iteb->event_id)
return -1;
else
return 1;
}
static int vgic_its_save_itt(struct vgic_its *its, struct its_device *device)
{
const struct vgic_its_abi *abi = vgic_its_get_abi(its);
gpa_t base = device->itt_addr;
struct its_ite *ite;
int ret;
int ite_esz = abi->ite_esz;
list_sort(NULL, &device->itt_head, vgic_its_ite_cmp);
list_for_each_entry(ite, &device->itt_head, ite_list) {
gpa_t gpa = base + ite->event_id * ite_esz;
ret = vgic_its_save_ite(its, device, ite, gpa, ite_esz);
if (ret)
return ret;
}
return 0;
}
static int vgic_its_restore_itt(struct vgic_its *its, struct its_device *dev)
{
const struct vgic_its_abi *abi = vgic_its_get_abi(its);
gpa_t base = dev->itt_addr;
int ret;
int ite_esz = abi->ite_esz;
size_t max_size = BIT_ULL(dev->num_eventid_bits) * ite_esz;
ret = scan_its_table(its, base, max_size, ite_esz, 0,
vgic_its_restore_ite, dev);
return ret;
}
/**
* vgic_its_save_dte - Save a device table entry at a given GPA
*
* @its: ITS handle
* @dev: ITS device
* @ptr: GPA
*/
static int vgic_its_save_dte(struct vgic_its *its, struct its_device *dev,
gpa_t ptr, int dte_esz)
{
struct kvm *kvm = its->dev->kvm;
u64 val, itt_addr_field;
u32 next_offset;
itt_addr_field = dev->itt_addr >> 8;
next_offset = compute_next_devid_offset(&its->device_list, dev);
val = (1ULL << KVM_ITS_DTE_VALID_SHIFT |
((u64)next_offset << KVM_ITS_DTE_NEXT_SHIFT) |
(itt_addr_field << KVM_ITS_DTE_ITTADDR_SHIFT) |
(dev->num_eventid_bits - 1));
val = cpu_to_le64(val);
return kvm_write_guest(kvm, ptr, &val, dte_esz);
}
/**
* vgic_its_restore_dte - restore a device table entry
*
* @its: its handle
* @id: device id the DTE corresponds to
* @ptr: kernel VA where the 8 byte DTE is located
* @opaque: unused
*
* Return: < 0 on error, 0 if the dte is the last one, id offset to the
* next dte otherwise
*/
static int vgic_its_restore_dte(struct vgic_its *its, u32 id,
void *ptr, void *opaque)
{
struct its_device *dev;
gpa_t itt_addr;
u8 num_eventid_bits;
u64 entry = *(u64 *)ptr;
bool valid;
u32 offset;
int ret;
entry = le64_to_cpu(entry);
valid = entry >> KVM_ITS_DTE_VALID_SHIFT;
num_eventid_bits = (entry & KVM_ITS_DTE_SIZE_MASK) + 1;
itt_addr = ((entry & KVM_ITS_DTE_ITTADDR_MASK)
>> KVM_ITS_DTE_ITTADDR_SHIFT) << 8;
if (!valid)
return 1;
/* dte entry is valid */
offset = (entry & KVM_ITS_DTE_NEXT_MASK) >> KVM_ITS_DTE_NEXT_SHIFT;
dev = vgic_its_alloc_device(its, id, itt_addr, num_eventid_bits);
if (IS_ERR(dev))
return PTR_ERR(dev);
ret = vgic_its_restore_itt(its, dev);
if (ret) {
vgic_its_free_device(its->dev->kvm, dev);
return ret;
}
return offset;
}
static int vgic_its_device_cmp(void *priv, struct list_head *a,
struct list_head *b)
{
struct its_device *deva = container_of(a, struct its_device, dev_list);
struct its_device *devb = container_of(b, struct its_device, dev_list);
if (deva->device_id < devb->device_id)
return -1;
else
return 1;
}
/**
* vgic_its_save_device_tables - Save the device table and all ITT
* into guest RAM
*
* L1/L2 handling is hidden by vgic_its_check_id() helper which directly
* returns the GPA of the device entry
*/
static int vgic_its_save_device_tables(struct vgic_its *its)
{
const struct vgic_its_abi *abi = vgic_its_get_abi(its);
struct its_device *dev;
int dte_esz = abi->dte_esz;
u64 baser;
baser = its->baser_device_table;
list_sort(NULL, &its->device_list, vgic_its_device_cmp);
list_for_each_entry(dev, &its->device_list, dev_list) {
int ret;
gpa_t eaddr;
if (!vgic_its_check_id(its, baser,
dev->device_id, &eaddr))
return -EINVAL;
ret = vgic_its_save_itt(its, dev);
if (ret)
return ret;
ret = vgic_its_save_dte(its, dev, eaddr, dte_esz);
if (ret)
return ret;
}
return 0;
}
/**
* handle_l1_dte - callback used for L1 device table entries (2 stage case)
*
* @its: its handle
* @id: index of the entry in the L1 table
* @addr: kernel VA
* @opaque: unused
*
* L1 table entries are scanned by steps of 1 entry
* Return < 0 if error, 0 if last dte was found when scanning the L2
* table, +1 otherwise (meaning next L1 entry must be scanned)
*/
static int handle_l1_dte(struct vgic_its *its, u32 id, void *addr,
void *opaque)
{
const struct vgic_its_abi *abi = vgic_its_get_abi(its);
int l2_start_id = id * (SZ_64K / abi->dte_esz);
u64 entry = *(u64 *)addr;
int dte_esz = abi->dte_esz;
gpa_t gpa;
int ret;
entry = le64_to_cpu(entry);
if (!(entry & KVM_ITS_L1E_VALID_MASK))
return 1;
gpa = entry & KVM_ITS_L1E_ADDR_MASK;
ret = scan_its_table(its, gpa, SZ_64K, dte_esz,
l2_start_id, vgic_its_restore_dte, NULL);
if (ret <= 0)
return ret;
return 1;
}
/**
* vgic_its_restore_device_tables - Restore the device table and all ITT
* from guest RAM to internal data structs
*/
static int vgic_its_restore_device_tables(struct vgic_its *its)
{
const struct vgic_its_abi *abi = vgic_its_get_abi(its);
u64 baser = its->baser_device_table;
int l1_esz, ret;
int l1_tbl_size = GITS_BASER_NR_PAGES(baser) * SZ_64K;
gpa_t l1_gpa;
if (!(baser & GITS_BASER_VALID))
return 0;
l1_gpa = BASER_ADDRESS(baser);
if (baser & GITS_BASER_INDIRECT) {
l1_esz = GITS_LVL1_ENTRY_SIZE;
ret = scan_its_table(its, l1_gpa, l1_tbl_size, l1_esz, 0,
handle_l1_dte, NULL);
} else {
l1_esz = abi->dte_esz;
ret = scan_its_table(its, l1_gpa, l1_tbl_size, l1_esz, 0,
vgic_its_restore_dte, NULL);
}
if (ret > 0)
ret = -EINVAL;
return ret;
}
static int vgic_its_save_cte(struct vgic_its *its,
struct its_collection *collection,
gpa_t gpa, int esz)
{
u64 val;
val = (1ULL << KVM_ITS_CTE_VALID_SHIFT |
((u64)collection->target_addr << KVM_ITS_CTE_RDBASE_SHIFT) |
collection->collection_id);
val = cpu_to_le64(val);
return kvm_write_guest(its->dev->kvm, gpa, &val, esz);
}
static int vgic_its_restore_cte(struct vgic_its *its, gpa_t gpa, int esz)
{
struct its_collection *collection;
struct kvm *kvm = its->dev->kvm;
u32 target_addr, coll_id;
u64 val;
int ret;
BUG_ON(esz > sizeof(val));
ret = kvm_read_guest(kvm, gpa, &val, esz);
if (ret)
return ret;
val = le64_to_cpu(val);
if (!(val & KVM_ITS_CTE_VALID_MASK))
return 0;
target_addr = (u32)(val >> KVM_ITS_CTE_RDBASE_SHIFT);
coll_id = val & KVM_ITS_CTE_ICID_MASK;
if (target_addr >= atomic_read(&kvm->online_vcpus))
return -EINVAL;
collection = find_collection(its, coll_id);
if (collection)
return -EEXIST;
ret = vgic_its_alloc_collection(its, &collection, coll_id);
if (ret)
return ret;
collection->target_addr = target_addr;
return 1;
}
/**
* vgic_its_save_collection_table - Save the collection table into
* guest RAM
*/
static int vgic_its_save_collection_table(struct vgic_its *its)
{
const struct vgic_its_abi *abi = vgic_its_get_abi(its);
struct its_collection *collection;
u64 val;
gpa_t gpa;
size_t max_size, filled = 0;
int ret, cte_esz = abi->cte_esz;
gpa = BASER_ADDRESS(its->baser_coll_table);
if (!gpa)
return 0;
max_size = GITS_BASER_NR_PAGES(its->baser_coll_table) * SZ_64K;
list_for_each_entry(collection, &its->collection_list, coll_list) {
ret = vgic_its_save_cte(its, collection, gpa, cte_esz);
if (ret)
return ret;
gpa += cte_esz;
filled += cte_esz;
}
if (filled == max_size)
return 0;
/*
* table is not fully filled, add a last dummy element
* with valid bit unset
*/
val = 0;
BUG_ON(cte_esz > sizeof(val));
ret = kvm_write_guest(its->dev->kvm, gpa, &val, cte_esz);
return ret;
}
/**
* vgic_its_restore_collection_table - reads the collection table
* in guest memory and restores the ITS internal state. Requires the
* BASER registers to be restored before.
*/
static int vgic_its_restore_collection_table(struct vgic_its *its)
{
const struct vgic_its_abi *abi = vgic_its_get_abi(its);
int cte_esz = abi->cte_esz;
size_t max_size, read = 0;
gpa_t gpa;
int ret;
if (!(its->baser_coll_table & GITS_BASER_VALID))
return 0;
gpa = BASER_ADDRESS(its->baser_coll_table);
max_size = GITS_BASER_NR_PAGES(its->baser_coll_table) * SZ_64K;
while (read < max_size) {
ret = vgic_its_restore_cte(its, gpa, cte_esz);
if (ret <= 0)
break;
gpa += cte_esz;
read += cte_esz;
}
return ret;
}
/**
* vgic_its_save_tables_v0 - Save the ITS tables into guest ARM
* according to v0 ABI
*/
static int vgic_its_save_tables_v0(struct vgic_its *its)
{
struct kvm *kvm = its->dev->kvm;
int ret;
mutex_lock(&kvm->lock);
mutex_lock(&its->its_lock);
if (!lock_all_vcpus(kvm)) {
mutex_unlock(&its->its_lock);
mutex_unlock(&kvm->lock);
return -EBUSY;
}
ret = vgic_its_save_device_tables(its);
if (ret)
goto out;
ret = vgic_its_save_collection_table(its);
out:
unlock_all_vcpus(kvm);
mutex_unlock(&its->its_lock);
mutex_unlock(&kvm->lock);
return ret;
}
/**
* vgic_its_restore_tables_v0 - Restore the ITS tables from guest RAM
* to internal data structs according to V0 ABI
*
*/
static int vgic_its_restore_tables_v0(struct vgic_its *its)
{
struct kvm *kvm = its->dev->kvm;
int ret;
mutex_lock(&kvm->lock);
mutex_lock(&its->its_lock);
if (!lock_all_vcpus(kvm)) {
mutex_unlock(&its->its_lock);
mutex_unlock(&kvm->lock);
return -EBUSY;
}
ret = vgic_its_restore_collection_table(its);
if (ret)
goto out;
ret = vgic_its_restore_device_tables(its);
out:
unlock_all_vcpus(kvm);
mutex_unlock(&its->its_lock);
mutex_unlock(&kvm->lock);
return ret;
}
static int vgic_its_commit_v0(struct vgic_its *its)
{
const struct vgic_its_abi *abi;
abi = vgic_its_get_abi(its);
its->baser_coll_table &= ~GITS_BASER_ENTRY_SIZE_MASK;
its->baser_device_table &= ~GITS_BASER_ENTRY_SIZE_MASK;
its->baser_coll_table |= (GIC_ENCODE_SZ(abi->cte_esz, 5)
<< GITS_BASER_ENTRY_SIZE_SHIFT);
its->baser_device_table |= (GIC_ENCODE_SZ(abi->dte_esz, 5)
<< GITS_BASER_ENTRY_SIZE_SHIFT);
return 0;
}
static int vgic_its_has_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
......@@ -1480,8 +2355,14 @@ static int vgic_its_has_attr(struct kvm_device *dev,
switch (attr->attr) {
case KVM_DEV_ARM_VGIC_CTRL_INIT:
return 0;
case KVM_DEV_ARM_ITS_SAVE_TABLES:
return 0;
case KVM_DEV_ARM_ITS_RESTORE_TABLES:
return 0;
}
break;
case KVM_DEV_ARM_VGIC_GRP_ITS_REGS:
return vgic_its_has_attr_regs(dev, attr);
}
return -ENXIO;
}
......@@ -1509,18 +2390,30 @@ static int vgic_its_set_attr(struct kvm_device *dev,
if (ret)
return ret;
its->vgic_its_base = addr;
return 0;
return vgic_register_its_iodev(dev->kvm, its, addr);
}
case KVM_DEV_ARM_VGIC_GRP_CTRL:
case KVM_DEV_ARM_VGIC_GRP_CTRL: {
const struct vgic_its_abi *abi = vgic_its_get_abi(its);
switch (attr->attr) {
case KVM_DEV_ARM_VGIC_CTRL_INIT:
its->initialized = true;
/* Nothing to do */
return 0;
case KVM_DEV_ARM_ITS_SAVE_TABLES:
return abi->save_tables(its);
case KVM_DEV_ARM_ITS_RESTORE_TABLES:
return abi->restore_tables(its);
}
break;
}
case KVM_DEV_ARM_VGIC_GRP_ITS_REGS: {
u64 __user *uaddr = (u64 __user *)(long)attr->addr;
u64 reg;
if (get_user(reg, uaddr))
return -EFAULT;
return vgic_its_attr_regs_access(dev, attr, &reg, true);
}
}
return -ENXIO;
}
......@@ -1541,10 +2434,20 @@ static int vgic_its_get_attr(struct kvm_device *dev,
if (copy_to_user(uaddr, &addr, sizeof(addr)))
return -EFAULT;
break;
}
case KVM_DEV_ARM_VGIC_GRP_ITS_REGS: {
u64 __user *uaddr = (u64 __user *)(long)attr->addr;
u64 reg;
int ret;
ret = vgic_its_attr_regs_access(dev, attr, &reg, false);
if (ret)
return ret;
return put_user(reg, uaddr);
}
default:
return -ENXIO;
}
}
return 0;
}
......@@ -1563,30 +2466,3 @@ int kvm_vgic_register_its_device(void)
return kvm_register_device_ops(&kvm_arm_vgic_its_ops,
KVM_DEV_TYPE_ARM_VGIC_ITS);
}
/*
* Registers all ITSes with the kvm_io_bus framework.
* To follow the existing VGIC initialization sequence, this has to be
* done as late as possible, just before the first VCPU runs.
*/
int vgic_register_its_iodevs(struct kvm *kvm)
{
struct kvm_device *dev;
int ret = 0;
list_for_each_entry(dev, &kvm->devices, vm_node) {
if (dev->ops != &kvm_arm_vgic_its_ops)
continue;
ret = vgic_register_its_iodev(kvm, dev->private);
if (ret)
return ret;
/*
* We don't need to care about tearing down previously
* registered ITSes, as the kvm_io_bus framework removes
* them for us if the VM gets destroyed.
*/
}
return ret;
}
virt/kvm/arm/vgic/vgic-kvm-device.c
View file @ 36c344f3
......@@ -37,6 +37,14 @@ int vgic_check_ioaddr(struct kvm *kvm, phys_addr_t *ioaddr,
return 0;
}
static int vgic_check_type(struct kvm *kvm, int type_needed)
{
if (kvm->arch.vgic.vgic_model != type_needed)
return -ENODEV;
else
return 0;
}
/**
* kvm_vgic_addr - set or get vgic VM base addresses
* @kvm: pointer to the vm struct
......@@ -57,40 +65,41 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write)
{
int r = 0;
struct vgic_dist *vgic = &kvm->arch.vgic;
int type_needed;
phys_addr_t *addr_ptr, alignment;
mutex_lock(&kvm->lock);
switch (type) {
case KVM_VGIC_V2_ADDR_TYPE_DIST:
type_needed = KVM_DEV_TYPE_ARM_VGIC_V2;
r = vgic_check_type(kvm, KVM_DEV_TYPE_ARM_VGIC_V2);
addr_ptr = &vgic->vgic_dist_base;
alignment = SZ_4K;
break;
case KVM_VGIC_V2_ADDR_TYPE_CPU:
type_needed = KVM_DEV_TYPE_ARM_VGIC_V2;
r = vgic_check_type(kvm, KVM_DEV_TYPE_ARM_VGIC_V2);
addr_ptr = &vgic->vgic_cpu_base;
alignment = SZ_4K;
break;
case KVM_VGIC_V3_ADDR_TYPE_DIST:
type_needed = KVM_DEV_TYPE_ARM_VGIC_V3;
r = vgic_check_type(kvm, KVM_DEV_TYPE_ARM_VGIC_V3);
addr_ptr = &vgic->vgic_dist_base;
alignment = SZ_64K;
break;
case KVM_VGIC_V3_ADDR_TYPE_REDIST:
type_needed = KVM_DEV_TYPE_ARM_VGIC_V3;
r = vgic_check_type(kvm, KVM_DEV_TYPE_ARM_VGIC_V3);
if (r)
break;
if (write) {
r = vgic_v3_set_redist_base(kvm, *addr);
goto out;
}
addr_ptr = &vgic->vgic_redist_base;
alignment = SZ_64K;
break;
default:
r = -ENODEV;
goto out;
}
if (vgic->vgic_model != type_needed) {
r = -ENODEV;
if (r)
goto out;
}
if (write) {
r = vgic_check_ioaddr(kvm, addr_ptr, *addr, alignment);
......@@ -259,13 +268,13 @@ static void unlock_vcpus(struct kvm *kvm, int vcpu_lock_idx)
}
}
static void unlock_all_vcpus(struct kvm *kvm)
void unlock_all_vcpus(struct kvm *kvm)
{
unlock_vcpus(kvm, atomic_read(&kvm->online_vcpus) - 1);
}
/* Returns true if all vcpus were locked, false otherwise */
static bool lock_all_vcpus(struct kvm *kvm)
bool lock_all_vcpus(struct kvm *kvm)
{
struct kvm_vcpu *tmp_vcpu;
int c;
......@@ -580,6 +589,24 @@ static int vgic_v3_set_attr(struct kvm_device *dev,
reg = tmp32;
return vgic_v3_attr_regs_access(dev, attr, &reg, true);
}
case KVM_DEV_ARM_VGIC_GRP_CTRL: {
int ret;
switch (attr->attr) {
case KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES:
mutex_lock(&dev->kvm->lock);
if (!lock_all_vcpus(dev->kvm)) {
mutex_unlock(&dev->kvm->lock);
return -EBUSY;
}
ret = vgic_v3_save_pending_tables(dev->kvm);
unlock_all_vcpus(dev->kvm);
mutex_unlock(&dev->kvm->lock);
return ret;
}
break;
}
}
return -ENXIO;
}
......@@ -658,6 +685,8 @@ static int vgic_v3_has_attr(struct kvm_device *dev,
switch (attr->attr) {
case KVM_DEV_ARM_VGIC_CTRL_INIT:
return 0;
case KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES:
return 0;
}
}
return -ENXIO;
......
virt/kvm/arm/vgic/vgic-mmio-v3.c
View file @ 36c344f3
......@@ -556,67 +556,130 @@ unsigned int vgic_v3_init_dist_iodev(struct vgic_io_device *dev)
return SZ_64K;
}
int vgic_register_redist_iodevs(struct kvm *kvm, gpa_t redist_base_address)
/**
* vgic_register_redist_iodev - register a single redist iodev
* @vcpu: The VCPU to which the redistributor belongs
*
* Register a KVM iodev for this VCPU's redistributor using the address
* provided.
*
* Return 0 on success, -ERRNO otherwise.
*/
int vgic_register_redist_iodev(struct kvm_vcpu *vcpu)
{
struct kvm *kvm = vcpu->kvm;
struct vgic_dist *vgic = &kvm->arch.vgic;
struct vgic_io_device *rd_dev = &vcpu->arch.vgic_cpu.rd_iodev;
struct vgic_io_device *sgi_dev = &vcpu->arch.vgic_cpu.sgi_iodev;
gpa_t rd_base, sgi_base;
int ret;
/*
* We may be creating VCPUs before having set the base address for the
* redistributor region, in which case we will come back to this
* function for all VCPUs when the base address is set. Just return
* without doing any work for now.
*/
if (IS_VGIC_ADDR_UNDEF(vgic->vgic_redist_base))
return 0;
if (!vgic_v3_check_base(kvm))
return -EINVAL;
rd_base = vgic->vgic_redist_base + kvm_vcpu_get_idx(vcpu) * SZ_64K * 2;
sgi_base = rd_base + SZ_64K;
kvm_iodevice_init(&rd_dev->dev, &kvm_io_gic_ops);
rd_dev->base_addr = rd_base;
rd_dev->iodev_type = IODEV_REDIST;
rd_dev->regions = vgic_v3_rdbase_registers;
rd_dev->nr_regions = ARRAY_SIZE(vgic_v3_rdbase_registers);
rd_dev->redist_vcpu = vcpu;
mutex_lock(&kvm->slots_lock);
ret = kvm_io_bus_register_dev(kvm, KVM_MMIO_BUS, rd_base,
SZ_64K, &rd_dev->dev);
mutex_unlock(&kvm->slots_lock);
if (ret)
return ret;
kvm_iodevice_init(&sgi_dev->dev, &kvm_io_gic_ops);
sgi_dev->base_addr = sgi_base;
sgi_dev->iodev_type = IODEV_REDIST;
sgi_dev->regions = vgic_v3_sgibase_registers;
sgi_dev->nr_regions = ARRAY_SIZE(vgic_v3_sgibase_registers);
sgi_dev->redist_vcpu = vcpu;
mutex_lock(&kvm->slots_lock);
ret = kvm_io_bus_register_dev(kvm, KVM_MMIO_BUS, sgi_base,
SZ_64K, &sgi_dev->dev);
mutex_unlock(&kvm->slots_lock);
if (ret)
kvm_io_bus_unregister_dev(kvm, KVM_MMIO_BUS,
&rd_dev->dev);
return ret;
}
static void vgic_unregister_redist_iodev(struct kvm_vcpu *vcpu)
{
struct vgic_io_device *rd_dev = &vcpu->arch.vgic_cpu.rd_iodev;
struct vgic_io_device *sgi_dev = &vcpu->arch.vgic_cpu.sgi_iodev;
kvm_io_bus_unregister_dev(vcpu->kvm, KVM_MMIO_BUS, &rd_dev->dev);
kvm_io_bus_unregister_dev(vcpu->kvm, KVM_MMIO_BUS, &sgi_dev->dev);
}
static int vgic_register_all_redist_iodevs(struct kvm *kvm)
{
struct kvm_vcpu *vcpu;
int c, ret = 0;
kvm_for_each_vcpu(c, vcpu, kvm) {
gpa_t rd_base = redist_base_address + c * SZ_64K * 2;
gpa_t sgi_base = rd_base + SZ_64K;
struct vgic_io_device *rd_dev = &vcpu->arch.vgic_cpu.rd_iodev;
struct vgic_io_device *sgi_dev = &vcpu->arch.vgic_cpu.sgi_iodev;
kvm_iodevice_init(&rd_dev->dev, &kvm_io_gic_ops);
rd_dev->base_addr = rd_base;
rd_dev->iodev_type = IODEV_REDIST;
rd_dev->regions = vgic_v3_rdbase_registers;
rd_dev->nr_regions = ARRAY_SIZE(vgic_v3_rdbase_registers);
rd_dev->redist_vcpu = vcpu;
mutex_lock(&kvm->slots_lock);
ret = kvm_io_bus_register_dev(kvm, KVM_MMIO_BUS, rd_base,
SZ_64K, &rd_dev->dev);
mutex_unlock(&kvm->slots_lock);
ret = vgic_register_redist_iodev(vcpu);
if (ret)
break;
kvm_iodevice_init(&sgi_dev->dev, &kvm_io_gic_ops);
sgi_dev->base_addr = sgi_base;
sgi_dev->iodev_type = IODEV_REDIST;
sgi_dev->regions = vgic_v3_sgibase_registers;
sgi_dev->nr_regions = ARRAY_SIZE(vgic_v3_sgibase_registers);
sgi_dev->redist_vcpu = vcpu;
mutex_lock(&kvm->slots_lock);
ret = kvm_io_bus_register_dev(kvm, KVM_MMIO_BUS, sgi_base,
SZ_64K, &sgi_dev->dev);
mutex_unlock(&kvm->slots_lock);
if (ret) {
kvm_io_bus_unregister_dev(kvm, KVM_MMIO_BUS,
&rd_dev->dev);
break;
}
}
if (ret) {
/* The current c failed, so we start with the previous one. */
for (c--; c >= 0; c--) {
struct vgic_cpu *vgic_cpu;
vcpu = kvm_get_vcpu(kvm, c);
vgic_cpu = &vcpu->arch.vgic_cpu;
kvm_io_bus_unregister_dev(kvm, KVM_MMIO_BUS,
&vgic_cpu->rd_iodev.dev);
kvm_io_bus_unregister_dev(kvm, KVM_MMIO_BUS,
&vgic_cpu->sgi_iodev.dev);
vgic_unregister_redist_iodev(vcpu);
}
}
return ret;
}
int vgic_v3_set_redist_base(struct kvm *kvm, u64 addr)
{
struct vgic_dist *vgic = &kvm->arch.vgic;
int ret;
/* vgic_check_ioaddr makes sure we don't do this twice */
ret = vgic_check_ioaddr(kvm, &vgic->vgic_redist_base, addr, SZ_64K);
if (ret)
return ret;
vgic->vgic_redist_base = addr;
if (!vgic_v3_check_base(kvm)) {
vgic->vgic_redist_base = VGIC_ADDR_UNDEF;
return -EINVAL;
}
/*
* Register iodevs for each existing VCPU. Adding more VCPUs
* afterwards will register the iodevs when needed.
*/
ret = vgic_register_all_redist_iodevs(kvm);
if (ret)
return ret;
return 0;
}
int vgic_v3_has_attr_regs(struct kvm_device *dev, struct kvm_device_attr *attr)
{
const struct vgic_register_region *region;
......
virt/kvm/arm/vgic/vgic-mmio.c
View file @ 36c344f3
......@@ -446,13 +446,12 @@ static int match_region(const void *key, const void *elt)
return 0;
}
/* Find the proper register handler entry given a certain address offset. */
static const struct vgic_register_region *
vgic_find_mmio_region(const struct vgic_register_region *region, int nr_regions,
unsigned int offset)
const struct vgic_register_region *
vgic_find_mmio_region(const struct vgic_register_region *regions,
int nr_regions, unsigned int offset)
{
return bsearch((void *)(uintptr_t)offset, region, nr_regions,
sizeof(region[0]), match_region);
return bsearch((void *)(uintptr_t)offset, regions, nr_regions,
sizeof(regions[0]), match_region);
}
void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr)
......
virt/kvm/arm/vgic/vgic-mmio.h
View file @ 36c344f3
......@@ -36,8 +36,13 @@ struct vgic_register_region {
};
unsigned long (*uaccess_read)(struct kvm_vcpu *vcpu, gpa_t addr,
unsigned int len);
void (*uaccess_write)(struct kvm_vcpu *vcpu, gpa_t addr,
unsigned int len, unsigned long val);
union {
void (*uaccess_write)(struct kvm_vcpu *vcpu, gpa_t addr,
unsigned int len, unsigned long val);
int (*uaccess_its_write)(struct kvm *kvm, struct vgic_its *its,
gpa_t addr, unsigned int len,
unsigned long val);
};
};
extern struct kvm_io_device_ops kvm_io_gic_ops;
......@@ -192,4 +197,9 @@ u64 vgic_sanitise_shareability(u64 reg);
u64 vgic_sanitise_field(u64 reg, u64 field_mask, int field_shift,
u64 (*sanitise_fn)(u64));
/* Find the proper register handler entry given a certain address offset */
const struct vgic_register_region *
vgic_find_mmio_region(const struct vgic_register_region *regions,
int nr_regions, unsigned int offset);
#endif
virt/kvm/arm/vgic/vgic-v3.c
View file @ 36c344f3
......@@ -234,19 +234,125 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu)
vgic_v3->vgic_hcr = ICH_HCR_EN;
}
/* check for overlapping regions and for regions crossing the end of memory */
static bool vgic_v3_check_base(struct kvm *kvm)
int vgic_v3_lpi_sync_pending_status(struct kvm *kvm, struct vgic_irq *irq)
{
struct kvm_vcpu *vcpu;
int byte_offset, bit_nr;
gpa_t pendbase, ptr;
bool status;
u8 val;
int ret;
retry:
vcpu = irq->target_vcpu;
if (!vcpu)
return 0;
pendbase = GICR_PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
byte_offset = irq->intid / BITS_PER_BYTE;
bit_nr = irq->intid % BITS_PER_BYTE;
ptr = pendbase + byte_offset;
ret = kvm_read_guest(kvm, ptr, &val, 1);
if (ret)
return ret;
status = val & (1 << bit_nr);
spin_lock(&irq->irq_lock);
if (irq->target_vcpu != vcpu) {
spin_unlock(&irq->irq_lock);
goto retry;
}
irq->pending_latch = status;
vgic_queue_irq_unlock(vcpu->kvm, irq);
if (status) {
/* clear consumed data */
val &= ~(1 << bit_nr);
ret = kvm_write_guest(kvm, ptr, &val, 1);
if (ret)
return ret;
}
return 0;
}
/**
* vgic_its_save_pending_tables - Save the pending tables into guest RAM
* kvm lock and all vcpu lock must be held
*/
int vgic_v3_save_pending_tables(struct kvm *kvm)
{
struct vgic_dist *dist = &kvm->arch.vgic;
int last_byte_offset = -1;
struct vgic_irq *irq;
int ret;
list_for_each_entry(irq, &dist->lpi_list_head, lpi_list) {
int byte_offset, bit_nr;
struct kvm_vcpu *vcpu;
gpa_t pendbase, ptr;
bool stored;
u8 val;
vcpu = irq->target_vcpu;
if (!vcpu)
continue;
pendbase = GICR_PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser);
byte_offset = irq->intid / BITS_PER_BYTE;
bit_nr = irq->intid % BITS_PER_BYTE;
ptr = pendbase + byte_offset;
if (byte_offset != last_byte_offset) {
ret = kvm_read_guest(kvm, ptr, &val, 1);
if (ret)
return ret;
last_byte_offset = byte_offset;
}
stored = val & (1U << bit_nr);
if (stored == irq->pending_latch)
continue;
if (irq->pending_latch)
val |= 1 << bit_nr;
else
val &= ~(1 << bit_nr);
ret = kvm_write_guest(kvm, ptr, &val, 1);
if (ret)
return ret;
}
return 0;
}
/*
* Check for overlapping regions and for regions crossing the end of memory
* for base addresses which have already been set.
*/
bool vgic_v3_check_base(struct kvm *kvm)
{
struct vgic_dist *d = &kvm->arch.vgic;
gpa_t redist_size = KVM_VGIC_V3_REDIST_SIZE;
redist_size *= atomic_read(&kvm->online_vcpus);
if (d->vgic_dist_base + KVM_VGIC_V3_DIST_SIZE < d->vgic_dist_base)
if (!IS_VGIC_ADDR_UNDEF(d->vgic_dist_base) &&
d->vgic_dist_base + KVM_VGIC_V3_DIST_SIZE < d->vgic_dist_base)
return false;
if (d->vgic_redist_base + redist_size < d->vgic_redist_base)
if (!IS_VGIC_ADDR_UNDEF(d->vgic_redist_base) &&
d->vgic_redist_base + redist_size < d->vgic_redist_base)
return false;
/* Both base addresses must be set to check if they overlap */
if (IS_VGIC_ADDR_UNDEF(d->vgic_dist_base) ||
IS_VGIC_ADDR_UNDEF(d->vgic_redist_base))
return true;
if (d->vgic_dist_base + KVM_VGIC_V3_DIST_SIZE <= d->vgic_redist_base)
return true;
if (d->vgic_redist_base + redist_size <= d->vgic_dist_base)
......@@ -291,20 +397,6 @@ int vgic_v3_map_resources(struct kvm *kvm)
goto out;
}
ret = vgic_register_redist_iodevs(kvm, dist->vgic_redist_base);
if (ret) {
kvm_err("Unable to register VGICv3 redist MMIO regions\n");
goto out;
}
if (vgic_has_its(kvm)) {
ret = vgic_register_its_iodevs(kvm);
if (ret) {
kvm_err("Unable to register VGIC ITS MMIO regions\n");
goto out;
}
}
dist->ready = true;
out:
......
virt/kvm/arm/vgic/vgic.c
View file @ 36c344f3
......@@ -21,7 +21,7 @@
#include "vgic.h"
#define CREATE_TRACE_POINTS
#include "../trace.h"
#include "trace.h"
#ifdef CONFIG_DEBUG_SPINLOCK
#define DEBUG_SPINLOCK_BUG_ON(p) BUG_ON(p)
......
virt/kvm/arm/vgic/vgic.h
View file @ 36c344f3
......@@ -73,6 +73,29 @@
KVM_REG_ARM_VGIC_SYSREG_CRM_MASK | \
KVM_REG_ARM_VGIC_SYSREG_OP2_MASK)
/*
* As per Documentation/virtual/kvm/devices/arm-vgic-its.txt,
* below macros are defined for ITS table entry encoding.
*/
#define KVM_ITS_CTE_VALID_SHIFT 63
#define KVM_ITS_CTE_VALID_MASK BIT_ULL(63)
#define KVM_ITS_CTE_RDBASE_SHIFT 16
#define KVM_ITS_CTE_ICID_MASK GENMASK_ULL(15, 0)
#define KVM_ITS_ITE_NEXT_SHIFT 48
#define KVM_ITS_ITE_PINTID_SHIFT 16
#define KVM_ITS_ITE_PINTID_MASK GENMASK_ULL(47, 16)
#define KVM_ITS_ITE_ICID_MASK GENMASK_ULL(15, 0)
#define KVM_ITS_DTE_VALID_SHIFT 63
#define KVM_ITS_DTE_VALID_MASK BIT_ULL(63)
#define KVM_ITS_DTE_NEXT_SHIFT 49
#define KVM_ITS_DTE_NEXT_MASK GENMASK_ULL(62, 49)
#define KVM_ITS_DTE_ITTADDR_SHIFT 5
#define KVM_ITS_DTE_ITTADDR_MASK GENMASK_ULL(48, 5)
#define KVM_ITS_DTE_SIZE_MASK GENMASK_ULL(4, 0)
#define KVM_ITS_L1E_VALID_MASK BIT_ULL(63)
/* we only support 64 kB translation table page size */
#define KVM_ITS_L1E_ADDR_MASK GENMASK_ULL(51, 16)
static inline bool irq_is_pending(struct vgic_irq *irq)
{
if (irq->config == VGIC_CONFIG_EDGE)
......@@ -157,12 +180,15 @@ void vgic_v3_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
void vgic_v3_enable(struct kvm_vcpu *vcpu);
int vgic_v3_probe(const struct gic_kvm_info *info);
int vgic_v3_map_resources(struct kvm *kvm);
int vgic_register_redist_iodevs(struct kvm *kvm, gpa_t dist_base_address);
int vgic_v3_lpi_sync_pending_status(struct kvm *kvm, struct vgic_irq *irq);
int vgic_v3_save_pending_tables(struct kvm *kvm);
int vgic_v3_set_redist_base(struct kvm *kvm, u64 addr);
int vgic_register_redist_iodev(struct kvm_vcpu *vcpu);
bool vgic_v3_check_base(struct kvm *kvm);
void vgic_v3_load(struct kvm_vcpu *vcpu);
void vgic_v3_put(struct kvm_vcpu *vcpu);
int vgic_register_its_iodevs(struct kvm *kvm);
bool vgic_has_its(struct kvm *kvm);
int kvm_vgic_register_its_device(void);
void vgic_enable_lpis(struct kvm_vcpu *vcpu);
......@@ -187,4 +213,7 @@ int vgic_init(struct kvm *kvm);
int vgic_debug_init(struct kvm *kvm);
int vgic_debug_destroy(struct kvm *kvm);
bool lock_all_vcpus(struct kvm *kvm);
void unlock_all_vcpus(struct kvm *kvm);
#endif
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7