Skip to content

L
linux
Commit 41ac3c2a authored by Jason Gunthorpe's avatar Jason Gunthorpe
Browse files
Options

Merge branch 'v6.6/vfio/cdev' of https://github.com/awilliam/linux-vfio into iommufd for-next 

Shared branch with VFIO containing the enablement for VFIO "cdev"
devices. This is required by following iommufd patches which add new
ioctls to the VFIO cdev.

=======
Existing VFIO provides group-centric user APIs for userspace. Userspace
opens the /dev/vfio/$group_id first before getting device fd and hence
getting access to device. This is not the desired model for iommufd. Per
the conclusion of community discussion[1], iommufd provides device-centric
kAPIs and requires its consumer (like VFIO) to be device-centric user
APIs. Such user APIs are used to associate device with iommufd and also
the I/O address spaces managed by the iommufd.

This series first introduces a per device file structure to be prepared
for further enhancement and refactors the kvm-vfio code to be prepared
for accepting device file from userspace. After this, adds a mechanism for
blocking device access before iommufd bind. Then refactors the vfio to be
able to handle cdev paths (e.g. iommufd binding, no-iommufd, [de]attach ioas).
This refactor includes making the device_open exclusive between the group
and the cdev path, only allow single device open in cdev path; vfio-iommufd
code is also refactored to support cdev. e.g. split the vfio_iommufd_bind()
into two steps. Eventually, adds the cdev support for vfio device and the
new ioctls, then makes group infrastructure optional as it is not needed
when vfio device cdev is compiled.

This series is based on some preparation works done to vfio emulated devices[2]
and vfio pci hot reset enhancements[3]. Per discussion[4], this series does not
support cdev for physical devices that do not have IOMMU. Such devices only
have group-centric user APIs.

This series is a prerequisite for iommu nesting for vfio device[5] [6].

[1] https://lore.kernel.org/kvm/BN9PR11MB5433B1E4AE5B0480369F97178C189@BN9PR11MB5433.namprd11.prod.outlook.com/
[2] https://lore.kernel.org/kvm/20230327093351.44505-1-yi.l.liu@intel.com/ - merged
[3] https://lore.kernel.org/kvm/20230718105542.4138-1-yi.l.liu@intel.com/
[4] https://lore.kernel.org/kvm/20230525095939.37ddb8ce.alex.williamson@redhat.com/
[5] https://lore.kernel.org/linux-iommu/20230511143844.22693-1-yi.l.liu@intel.com/
[6] https://lore.kernel.org/linux-iommu/20230511145110.27707-1-yi.l.liu@intel.com/#t
=======

* 'v6.6/vfio/cdev' of https://github.com/awilliam/linux-vfio: (36 commits)
  docs: vfio: Add vfio device cdev description
  vfio: Compile vfio_group infrastructure optionally
  vfio: Move the IOMMU_CAP_CACHE_COHERENCY check in __vfio_register_dev()
  vfio: Add VFIO_DEVICE_[AT|DE]TACH_IOMMUFD_PT
  vfio: Add VFIO_DEVICE_BIND_IOMMUFD
  vfio: Avoid repeated user pointer cast in vfio_device_fops_unl_ioctl()
  iommufd: Add iommufd_ctx_from_fd()
  vfio: Test kvm pointer in _vfio_device_get_kvm_safe()
  vfio: Add cdev for vfio_device
  vfio: Move device_del() before waiting for the last vfio_device registration refcount
  vfio: Move vfio_device_group_unregister() to be the first operation in unregister
  vfio-iommufd: Add detach_ioas support for emulated VFIO devices
  iommufd/device: Add iommufd_access_detach() API
  vfio-iommufd: Add detach_ioas support for physical VFIO devices
  vfio: Record devid in vfio_device_file
  vfio-iommufd: Split bind/attach into two steps
  vfio-iommufd: Move noiommu compat validation out of vfio_iommufd_bind()
  vfio: Make vfio_df_open() single open for device cdev path
  vfio: Add cdev_device_open_cnt to vfio_group
  vfio: Block device access via device fd until device is opened
  vfio: Pass struct vfio_device_file * to vfio_device_open/close()
  kvm/vfio: Accept vfio device file from userspace
  kvm/vfio: Prepare for accepting vfio device fd
  vfio: Accept vfio device file in the KVM facing kAPI
  vfio: Refine vfio file kAPIs for KVM
  vfio: Allocate per device file structure
  vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET
  vfio/pci: Copy hot-reset device info to userspace in the devices loop
  vfio/pci: Extend VFIO_DEVICE_GET_PCI_HOT_RESET_INFO for vfio device cdev
  vfio: Add helper to search vfio_device in a dev_set
  ...
parents 6eaae198 09467130
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 1669 additions and 342 deletions
Documentation/driver-api/vfio.rst
View file @ 41ac3c2a
......@@ -239,6 +239,137 @@ group and can access them as follows::
/* Gratuitous device reset and go... */
ioctl(device, VFIO_DEVICE_RESET);
IOMMUFD and vfio_iommu_type1
----------------------------
IOMMUFD is the new user API to manage I/O page tables from userspace.
It intends to be the portal of delivering advanced userspace DMA
features (nested translation [5]_, PASID [6]_, etc.) while also providing
a backwards compatibility interface for existing VFIO_TYPE1v2_IOMMU use
cases. Eventually the vfio_iommu_type1 driver, as well as the legacy
vfio container and group model is intended to be deprecated.
The IOMMUFD backwards compatibility interface can be enabled two ways.
In the first method, the kernel can be configured with
CONFIG_IOMMUFD_VFIO_CONTAINER, in which case the IOMMUFD subsystem
transparently provides the entire infrastructure for the VFIO
container and IOMMU backend interfaces. The compatibility mode can
also be accessed if the VFIO container interface, ie. /dev/vfio/vfio is
simply symlink'd to /dev/iommu. Note that at the time of writing, the
compatibility mode is not entirely feature complete relative to
VFIO_TYPE1v2_IOMMU (ex. DMA mapping MMIO) and does not attempt to
provide compatibility to the VFIO_SPAPR_TCE_IOMMU interface. Therefore
it is not generally advisable at this time to switch from native VFIO
implementations to the IOMMUFD compatibility interfaces.
Long term, VFIO users should migrate to device access through the cdev
interface described below, and native access through the IOMMUFD
provided interfaces.
VFIO Device cdev
----------------
Traditionally user acquires a device fd via VFIO_GROUP_GET_DEVICE_FD
in a VFIO group.
With CONFIG_VFIO_DEVICE_CDEV=y the user can now acquire a device fd
by directly opening a character device /dev/vfio/devices/vfioX where
"X" is the number allocated uniquely by VFIO for registered devices.
cdev interface does not support noiommu devices, so user should use
the legacy group interface if noiommu is wanted.
The cdev only works with IOMMUFD. Both VFIO drivers and applications
must adapt to the new cdev security model which requires using
VFIO_DEVICE_BIND_IOMMUFD to claim DMA ownership before starting to
actually use the device. Once BIND succeeds then a VFIO device can
be fully accessed by the user.
VFIO device cdev doesn't rely on VFIO group/container/iommu drivers.
Hence those modules can be fully compiled out in an environment
where no legacy VFIO application exists.
So far SPAPR does not support IOMMUFD yet. So it cannot support device
cdev either.
vfio device cdev access is still bound by IOMMU group semantics, ie. there
can be only one DMA owner for the group. Devices belonging to the same
group can not be bound to multiple iommufd_ctx or shared between native
kernel and vfio bus driver or other driver supporting the driver_managed_dma
flag. A violation of this ownership requirement will fail at the
VFIO_DEVICE_BIND_IOMMUFD ioctl, which gates full device access.
Device cdev Example
-------------------
Assume user wants to access PCI device 0000:6a:01.0::
$ ls /sys/bus/pci/devices/0000:6a:01.0/vfio-dev/
vfio0
This device is therefore represented as vfio0. The user can verify
its existence::
$ ls -l /dev/vfio/devices/vfio0
crw------- 1 root root 511, 0 Feb 16 01:22 /dev/vfio/devices/vfio0
$ cat /sys/bus/pci/devices/0000:6a:01.0/vfio-dev/vfio0/dev
511:0
$ ls -l /dev/char/511\:0
lrwxrwxrwx 1 root root 21 Feb 16 01:22 /dev/char/511:0 -> ../vfio/devices/vfio0
Then provide the user with access to the device if unprivileged
operation is desired::
$ chown user:user /dev/vfio/devices/vfio0
Finally the user could get cdev fd by::
cdev_fd = open("/dev/vfio/devices/vfio0", O_RDWR);
An opened cdev_fd doesn't give the user any permission of accessing
the device except binding the cdev_fd to an iommufd. After that point
then the device is fully accessible including attaching it to an
IOMMUFD IOAS/HWPT to enable userspace DMA::
struct vfio_device_bind_iommufd bind = {
.argsz = sizeof(bind),
.flags = 0,
};
struct iommu_ioas_alloc alloc_data = {
.size = sizeof(alloc_data),
.flags = 0,
};
struct vfio_device_attach_iommufd_pt attach_data = {
.argsz = sizeof(attach_data),
.flags = 0,
};
struct iommu_ioas_map map = {
.size = sizeof(map),
.flags = IOMMU_IOAS_MAP_READABLE |
IOMMU_IOAS_MAP_WRITEABLE |
IOMMU_IOAS_MAP_FIXED_IOVA,
.__reserved = 0,
};
iommufd = open("/dev/iommu", O_RDWR);
bind.iommufd = iommufd;
ioctl(cdev_fd, VFIO_DEVICE_BIND_IOMMUFD, &bind);
ioctl(iommufd, IOMMU_IOAS_ALLOC, &alloc_data);
attach_data.pt_id = alloc_data.out_ioas_id;
ioctl(cdev_fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &attach_data);
/* Allocate some space and setup a DMA mapping */
map.user_va = (int64_t)mmap(0, 1024 * 1024, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, 0, 0);
map.iova = 0; /* 1MB starting at 0x0 from device view */
map.length = 1024 * 1024;
map.ioas_id = alloc_data.out_ioas_id;;
ioctl(iommufd, IOMMU_IOAS_MAP, &map);
/* Other device operations as stated in "VFIO Usage Example" */
VFIO User API
-------------------------------------------------------------------------------
......@@ -279,6 +410,7 @@ similar to a file operations structure::
struct iommufd_ctx *ictx, u32 *out_device_id);
void (*unbind_iommufd)(struct vfio_device *vdev);
int (*attach_ioas)(struct vfio_device *vdev, u32 *pt_id);
void (*detach_ioas)(struct vfio_device *vdev);
int (*open_device)(struct vfio_device *vdev);
void (*close_device)(struct vfio_device *vdev);
ssize_t (*read)(struct vfio_device *vdev, char __user *buf,
......@@ -315,9 +447,10 @@ container_of().
- The [un]bind_iommufd callbacks are issued when the device is bound to
and unbound from iommufd.
- The attach_ioas callback is issued when the device is attached to an
IOAS managed by the bound iommufd. The attached IOAS is automatically
detached when the device is unbound from iommufd.
- The [de]attach_ioas callback is issued when the device is attached to
and detached from an IOAS managed by the bound iommufd. However, the
attached IOAS can also be automatically detached when the device is
unbound from iommufd.
- The read/write/mmap callbacks implement the device region access defined
by the device's own VFIO_DEVICE_GET_REGION_INFO ioctl.
......@@ -564,3 +697,11 @@ This implementation has some specifics:
\-0d.1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
.. [5] Nested translation is an IOMMU feature which supports two stage
address translations. This improves the address translation efficiency
in IOMMU virtualization.
.. [6] PASID stands for Process Address Space ID, introduced by PCI
Express. It is a prerequisite for Shared Virtual Addressing (SVA)
and Scalable I/O Virtualization (Scalable IOV).
Documentation/virt/kvm/devices/vfio.rst
View file @ 41ac3c2a
......@@ -9,22 +9,34 @@ Device types supported:
- KVM_DEV_TYPE_VFIO
Only one VFIO instance may be created per VM. The created device
tracks VFIO groups in use by the VM and features of those groups
important to the correctness and acceleration of the VM. As groups
are enabled and disabled for use by the VM, KVM should be updated
about their presence. When registered with KVM, a reference to the
VFIO-group is held by KVM.
tracks VFIO files (group or device) in use by the VM and features
of those groups/devices important to the correctness and acceleration
of the VM. As groups/devices are enabled and disabled for use by the
VM, KVM should be updated about their presence. When registered with
KVM, a reference to the VFIO file is held by KVM.
Groups:
KVM_DEV_VFIO_GROUP
KVM_DEV_VFIO_GROUP attributes:
KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device tracking
kvm_device_attr.addr points to an int32_t file descriptor
for the VFIO group.
KVM_DEV_VFIO_GROUP_DEL: Remove a VFIO group from VFIO-KVM device tracking
kvm_device_attr.addr points to an int32_t file descriptor
for the VFIO group.
KVM_DEV_VFIO_FILE
alias: KVM_DEV_VFIO_GROUP
KVM_DEV_VFIO_FILE attributes:
KVM_DEV_VFIO_FILE_ADD: Add a VFIO file (group/device) to VFIO-KVM device
tracking
kvm_device_attr.addr points to an int32_t file descriptor for the
VFIO file.
KVM_DEV_VFIO_FILE_DEL: Remove a VFIO file (group/device) from VFIO-KVM
device tracking
kvm_device_attr.addr points to an int32_t file descriptor for the
VFIO file.
KVM_DEV_VFIO_GROUP (legacy kvm device group restricted to the handling of VFIO group fd):
KVM_DEV_VFIO_GROUP_ADD: same as KVM_DEV_VFIO_FILE_ADD for group fd only
KVM_DEV_VFIO_GROUP_DEL: same as KVM_DEV_VFIO_FILE_DEL for group fd only
KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE: attaches a guest visible TCE table
allocated by sPAPR KVM.
kvm_device_attr.addr points to a struct::
......@@ -40,7 +52,10 @@ KVM_DEV_VFIO_GROUP attributes:
- @tablefd is a file descriptor for a TCE table allocated via
KVM_CREATE_SPAPR_TCE.
The GROUP_ADD operation above should be invoked prior to accessing the
The FILE/GROUP_ADD operation above should be invoked prior to accessing the
device file descriptor via VFIO_GROUP_GET_DEVICE_FD in order to support
drivers which require a kvm pointer to be set in their .open_device()
callback.
callback. It is the same for device file descriptor via character device
open which gets device access via VFIO_DEVICE_BIND_IOMMUFD. For such file
descriptors, FILE_ADD should be invoked before VFIO_DEVICE_BIND_IOMMUFD
to support the drivers mentioned in prior sentence as well.
drivers/gpu/drm/i915/gvt/kvmgt.c
View file @ 41ac3c2a
......@@ -1474,6 +1474,7 @@ static const struct vfio_device_ops intel_vgpu_dev_ops = {
.bind_iommufd = vfio_iommufd_emulated_bind,
.unbind_iommufd = vfio_iommufd_emulated_unbind,
.attach_ioas = vfio_iommufd_emulated_attach_ioas,
.detach_ioas = vfio_iommufd_emulated_detach_ioas,
};
static int intel_vgpu_probe(struct mdev_device *mdev)
......
drivers/iommu/iommufd/Kconfig
View file @ 41ac3c2a
......@@ -14,8 +14,8 @@ config IOMMUFD
if IOMMUFD
config IOMMUFD_VFIO_CONTAINER
bool "IOMMUFD provides the VFIO container /dev/vfio/vfio"
depends on VFIO && !VFIO_CONTAINER
default VFIO && !VFIO_CONTAINER
depends on VFIO_GROUP && !VFIO_CONTAINER
default VFIO_GROUP && !VFIO_CONTAINER
help
IOMMUFD will provide /dev/vfio/vfio instead of VFIO. This relies on
IOMMUFD providing compatibility emulation to give the same ioctls.
......
drivers/iommu/iommufd/device.c
View file @ 41ac3c2a
......@@ -98,6 +98,36 @@ struct iommufd_device *iommufd_device_bind(struct iommufd_ctx *ictx,
}
EXPORT_SYMBOL_NS_GPL(iommufd_device_bind, IOMMUFD);
/**
* iommufd_ctx_has_group - True if any device within the group is bound
* to the ictx
* @ictx: iommufd file descriptor
* @group: Pointer to a physical iommu_group struct
*
* True if any device within the group has been bound to this ictx, ex. via
* iommufd_device_bind(), therefore implying ictx ownership of the group.
*/
bool iommufd_ctx_has_group(struct iommufd_ctx *ictx, struct iommu_group *group)
{
struct iommufd_object *obj;
unsigned long index;
if (!ictx || !group)
return false;
xa_lock(&ictx->objects);
xa_for_each(&ictx->objects, index, obj) {
if (obj->type == IOMMUFD_OBJ_DEVICE &&
container_of(obj, struct iommufd_device, obj)->group == group) {
xa_unlock(&ictx->objects);
return true;
}
}
xa_unlock(&ictx->objects);
return false;
}
EXPORT_SYMBOL_NS_GPL(iommufd_ctx_has_group, IOMMUFD);
/**
* iommufd_device_unbind - Undo iommufd_device_bind()
* @idev: Device returned by iommufd_device_bind()
......@@ -116,6 +146,18 @@ void iommufd_device_unbind(struct iommufd_device *idev)
}
EXPORT_SYMBOL_NS_GPL(iommufd_device_unbind, IOMMUFD);
struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev)
{
return idev->ictx;
}
EXPORT_SYMBOL_NS_GPL(iommufd_device_to_ictx, IOMMUFD);
u32 iommufd_device_to_id(struct iommufd_device *idev)
{
return idev->obj.id;
}
EXPORT_SYMBOL_NS_GPL(iommufd_device_to_id, IOMMUFD);
static int iommufd_device_setup_msi(struct iommufd_device *idev,
struct iommufd_hw_pagetable *hwpt,
phys_addr_t sw_msi_start)
......@@ -444,6 +486,7 @@ iommufd_access_create(struct iommufd_ctx *ictx,
iommufd_ctx_get(ictx);
iommufd_object_finalize(ictx, &access->obj);
*id = access->obj.id;
mutex_init(&access->ioas_lock);
return access;
}
EXPORT_SYMBOL_NS_GPL(iommufd_access_create, IOMMUFD);
......@@ -463,26 +506,60 @@ void iommufd_access_destroy(struct iommufd_access *access)
}
EXPORT_SYMBOL_NS_GPL(iommufd_access_destroy, IOMMUFD);
void iommufd_access_detach(struct iommufd_access *access)
{
struct iommufd_ioas *cur_ioas = access->ioas;
mutex_lock(&access->ioas_lock);
if (WARN_ON(!access->ioas))
goto out;
/*
* Set ioas to NULL to block any further iommufd_access_pin_pages().
* iommufd_access_unpin_pages() can continue using access->ioas_unpin.
*/
access->ioas = NULL;
if (access->ops->unmap) {
mutex_unlock(&access->ioas_lock);
access->ops->unmap(access->data, 0, ULONG_MAX);
mutex_lock(&access->ioas_lock);
}
iopt_remove_access(&cur_ioas->iopt, access);
refcount_dec(&cur_ioas->obj.users);
out:
access->ioas_unpin = NULL;
mutex_unlock(&access->ioas_lock);
}
EXPORT_SYMBOL_NS_GPL(iommufd_access_detach, IOMMUFD);
int iommufd_access_attach(struct iommufd_access *access, u32 ioas_id)
{
struct iommufd_ioas *new_ioas;
int rc = 0;
if (access->ioas)
mutex_lock(&access->ioas_lock);
if (WARN_ON(access->ioas || access->ioas_unpin)) {
mutex_unlock(&access->ioas_lock);
return -EINVAL;
}
new_ioas = iommufd_get_ioas(access->ictx, ioas_id);
if (IS_ERR(new_ioas))
if (IS_ERR(new_ioas)) {
mutex_unlock(&access->ioas_lock);
return PTR_ERR(new_ioas);
}
rc = iopt_add_access(&new_ioas->iopt, access);
if (rc) {
mutex_unlock(&access->ioas_lock);
iommufd_put_object(&new_ioas->obj);
return rc;
}
iommufd_ref_to_users(&new_ioas->obj);
access->ioas = new_ioas;
access->ioas_unpin = new_ioas;
mutex_unlock(&access->ioas_lock);
return 0;
}
EXPORT_SYMBOL_NS_GPL(iommufd_access_attach, IOMMUFD);
......@@ -537,8 +614,8 @@ void iommufd_access_notify_unmap(struct io_pagetable *iopt, unsigned long iova,
void iommufd_access_unpin_pages(struct iommufd_access *access,
unsigned long iova, unsigned long length)
{
struct io_pagetable *iopt = &access->ioas->iopt;
struct iopt_area_contig_iter iter;
struct io_pagetable *iopt;
unsigned long last_iova;
struct iopt_area *area;
......@@ -546,6 +623,17 @@ void iommufd_access_unpin_pages(struct iommufd_access *access,
WARN_ON(check_add_overflow(iova, length - 1, &last_iova)))
return;
mutex_lock(&access->ioas_lock);
/*
* The driver must be doing something wrong if it calls this before an
* iommufd_access_attach() or after an iommufd_access_detach().
*/
if (WARN_ON(!access->ioas_unpin)) {
mutex_unlock(&access->ioas_lock);
return;
}
iopt = &access->ioas_unpin->iopt;
down_read(&iopt->iova_rwsem);
iopt_for_each_contig_area(&iter, area, iopt, iova, last_iova)
iopt_area_remove_access(
......@@ -555,6 +643,7 @@ void iommufd_access_unpin_pages(struct iommufd_access *access,
min(last_iova, iopt_area_last_iova(area))));
WARN_ON(!iopt_area_contig_done(&iter));
up_read(&iopt->iova_rwsem);
mutex_unlock(&access->ioas_lock);
}
EXPORT_SYMBOL_NS_GPL(iommufd_access_unpin_pages, IOMMUFD);
......@@ -600,8 +689,8 @@ int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova,
unsigned long length, struct page **out_pages,
unsigned int flags)
{
struct io_pagetable *iopt = &access->ioas->iopt;
struct iopt_area_contig_iter iter;
struct io_pagetable *iopt;
unsigned long last_iova;
struct iopt_area *area;
int rc;
......@@ -616,6 +705,13 @@ int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova,
if (check_add_overflow(iova, length - 1, &last_iova))
return -EOVERFLOW;
mutex_lock(&access->ioas_lock);
if (!access->ioas) {
mutex_unlock(&access->ioas_lock);
return -ENOENT;
}
iopt = &access->ioas->iopt;
down_read(&iopt->iova_rwsem);
iopt_for_each_contig_area(&iter, area, iopt, iova, last_iova) {
unsigned long last = min(last_iova, iopt_area_last_iova(area));
......@@ -646,6 +742,7 @@ int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova,
}
up_read(&iopt->iova_rwsem);
mutex_unlock(&access->ioas_lock);
return 0;
err_remove:
......@@ -660,6 +757,7 @@ int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova,
iopt_area_last_iova(area))));
}
up_read(&iopt->iova_rwsem);
mutex_unlock(&access->ioas_lock);
return rc;
}
EXPORT_SYMBOL_NS_GPL(iommufd_access_pin_pages, IOMMUFD);
......@@ -679,8 +777,8 @@ EXPORT_SYMBOL_NS_GPL(iommufd_access_pin_pages, IOMMUFD);
int iommufd_access_rw(struct iommufd_access *access, unsigned long iova,
void *data, size_t length, unsigned int flags)
{
struct io_pagetable *iopt = &access->ioas->iopt;
struct iopt_area_contig_iter iter;
struct io_pagetable *iopt;
struct iopt_area *area;
unsigned long last_iova;
int rc;
......@@ -690,6 +788,13 @@ int iommufd_access_rw(struct iommufd_access *access, unsigned long iova,
if (check_add_overflow(iova, length - 1, &last_iova))
return -EOVERFLOW;
mutex_lock(&access->ioas_lock);
if (!access->ioas) {
mutex_unlock(&access->ioas_lock);
return -ENOENT;
}
iopt = &access->ioas->iopt;
down_read(&iopt->iova_rwsem);
iopt_for_each_contig_area(&iter, area, iopt, iova, last_iova) {
unsigned long last = min(last_iova, iopt_area_last_iova(area));
......@@ -716,6 +821,7 @@ int iommufd_access_rw(struct iommufd_access *access, unsigned long iova,
rc = -ENOENT;
err_out:
up_read(&iopt->iova_rwsem);
mutex_unlock(&access->ioas_lock);
return rc;
}
EXPORT_SYMBOL_NS_GPL(iommufd_access_rw, IOMMUFD);
drivers/iommu/iommufd/iommufd_private.h
View file @ 41ac3c2a
......@@ -285,6 +285,8 @@ struct iommufd_access {
struct iommufd_object obj;
struct iommufd_ctx *ictx;
struct iommufd_ioas *ioas;
struct iommufd_ioas *ioas_unpin;
struct mutex ioas_lock;
const struct iommufd_access_ops *ops;
void *data;
unsigned long iova_alignment;
......
drivers/iommu/iommufd/main.c
View file @ 41ac3c2a
......@@ -50,7 +50,7 @@ struct iommufd_object *_iommufd_object_alloc(struct iommufd_ctx *ictx,
* before calling iommufd_object_finalize().
*/
rc = xa_alloc(&ictx->objects, &obj->id, XA_ZERO_ENTRY,
xa_limit_32b, GFP_KERNEL_ACCOUNT);
xa_limit_31b, GFP_KERNEL_ACCOUNT);
if (rc)
goto out_free;
return obj;
......@@ -377,6 +377,30 @@ struct iommufd_ctx *iommufd_ctx_from_file(struct file *file)
}
EXPORT_SYMBOL_NS_GPL(iommufd_ctx_from_file, IOMMUFD);
/**
* iommufd_ctx_from_fd - Acquires a reference to the iommufd context
* @fd: File descriptor to obtain the reference from
*
* Returns a pointer to the iommufd_ctx, otherwise ERR_PTR. On success
* the caller is responsible to call iommufd_ctx_put().
*/
struct iommufd_ctx *iommufd_ctx_from_fd(int fd)
{
struct file *file;
file = fget(fd);
if (!file)
return ERR_PTR(-EBADF);
if (file->f_op != &iommufd_fops) {
fput(file);
return ERR_PTR(-EBADFD);
}
/* fget is the same as iommufd_ctx_get() */
return file->private_data;
}
EXPORT_SYMBOL_NS_GPL(iommufd_ctx_from_fd, IOMMUFD);
/**
* iommufd_ctx_put - Put back a reference
* @ictx: Context to put back
......
drivers/s390/cio/vfio_ccw_ops.c
View file @ 41ac3c2a
......@@ -632,6 +632,7 @@ static const struct vfio_device_ops vfio_ccw_dev_ops = {
.bind_iommufd = vfio_iommufd_emulated_bind,
.unbind_iommufd = vfio_iommufd_emulated_unbind,
.attach_ioas = vfio_iommufd_emulated_attach_ioas,
.detach_ioas = vfio_iommufd_emulated_detach_ioas,
};
struct mdev_driver vfio_ccw_mdev_driver = {
......
drivers/s390/crypto/vfio_ap_ops.c
View file @ 41ac3c2a
......@@ -1975,6 +1975,7 @@ static const struct vfio_device_ops vfio_ap_matrix_dev_ops = {
.bind_iommufd = vfio_iommufd_emulated_bind,
.unbind_iommufd = vfio_iommufd_emulated_unbind,
.attach_ioas = vfio_iommufd_emulated_attach_ioas,
.detach_ioas = vfio_iommufd_emulated_detach_ioas,
.request = vfio_ap_mdev_request
};
......
drivers/vfio/Kconfig
View file @ 41ac3c2a
......@@ -4,6 +4,8 @@ menuconfig VFIO
select IOMMU_API
depends on IOMMUFD || !IOMMUFD
select INTERVAL_TREE
select VFIO_GROUP if SPAPR_TCE_IOMMU || IOMMUFD=n
select VFIO_DEVICE_CDEV if !VFIO_GROUP
select VFIO_CONTAINER if IOMMUFD=n
help
VFIO provides a framework for secure userspace device drivers.
......@@ -12,9 +14,33 @@ menuconfig VFIO
If you don't know what to do here, say N.
if VFIO
config VFIO_DEVICE_CDEV
bool "Support for the VFIO cdev /dev/vfio/devices/vfioX"
depends on IOMMUFD && !SPAPR_TCE_IOMMU
default !VFIO_GROUP
help
The VFIO device cdev is another way for userspace to get device
access. Userspace gets device fd by opening device cdev under
/dev/vfio/devices/vfioX, and then bind the device fd with an iommufd
to set up secure DMA context for device access. This interface does
not support noiommu.
If you don't know what to do here, say N.
config VFIO_GROUP
bool "Support for the VFIO group /dev/vfio/$group_id"
default y
help
VFIO group support provides the traditional model for accessing
devices through VFIO and is used by the majority of userspace
applications and drivers making use of VFIO.
If you don't know what to do here, say Y.
config VFIO_CONTAINER
bool "Support for the VFIO container /dev/vfio/vfio"
select VFIO_IOMMU_TYPE1 if MMU && (X86 || S390 || ARM || ARM64)
depends on VFIO_GROUP
default y
help
The VFIO container is the classic interface to VFIO for establishing
......@@ -36,6 +62,7 @@ endif
config VFIO_NOIOMMU
bool "VFIO No-IOMMU support"
depends on VFIO_GROUP
help
VFIO is built on the ability to isolate devices using the IOMMU.
Only with an IOMMU can userspace access to DMA capable devices be
......
drivers/vfio/Makefile
View file @ 41ac3c2a
......@@ -2,8 +2,9 @@
obj-$(CONFIG_VFIO) += vfio.o
vfio-y += vfio_main.o \
group.o \
iova_bitmap.o
vfio-$(CONFIG_VFIO_DEVICE_CDEV) += device_cdev.o
vfio-$(CONFIG_VFIO_GROUP) += group.o
vfio-$(CONFIG_IOMMUFD) += iommufd.o
vfio-$(CONFIG_VFIO_CONTAINER) += container.o
vfio-$(CONFIG_VFIO_VIRQFD) += virqfd.o
......
drivers/vfio/device_cdev.c 0 → 100644
View file @ 41ac3c2a
// SPDX-License-Identifier: GPL-2.0-only
/*
* Copyright (c) 2023 Intel Corporation.
*/
#include <linux/vfio.h>
#include <linux/iommufd.h>
#include "vfio.h"
static dev_t device_devt;
void vfio_init_device_cdev(struct vfio_device *device)
{
device->device.devt = MKDEV(MAJOR(device_devt), device->index);
cdev_init(&device->cdev, &vfio_device_fops);
device->cdev.owner = THIS_MODULE;
}
/*
* device access via the fd opened by this function is blocked until
* .open_device() is called successfully during BIND_IOMMUFD.
*/
int vfio_device_fops_cdev_open(struct inode *inode, struct file *filep)
{
struct vfio_device *device = container_of(inode->i_cdev,
struct vfio_device, cdev);
struct vfio_device_file *df;
int ret;
/* Paired with the put in vfio_device_fops_release() */
if (!vfio_device_try_get_registration(device))
return -ENODEV;
df = vfio_allocate_device_file(device);
if (IS_ERR(df)) {
ret = PTR_ERR(df);
goto err_put_registration;
}
filep->private_data = df;
return 0;
err_put_registration:
vfio_device_put_registration(device);
return ret;
}
static void vfio_df_get_kvm_safe(struct vfio_device_file *df)
{
spin_lock(&df->kvm_ref_lock);
vfio_device_get_kvm_safe(df->device, df->kvm);
spin_unlock(&df->kvm_ref_lock);
}
long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df,
struct vfio_device_bind_iommufd __user *arg)
{
struct vfio_device *device = df->device;
struct vfio_device_bind_iommufd bind;
unsigned long minsz;
int ret;
static_assert(__same_type(arg->out_devid, df->devid));
minsz = offsetofend(struct vfio_device_bind_iommufd, out_devid);
if (copy_from_user(&bind, arg, minsz))
return -EFAULT;
if (bind.argsz < minsz || bind.flags || bind.iommufd < 0)
return -EINVAL;
/* BIND_IOMMUFD only allowed for cdev fds */
if (df->group)
return -EINVAL;
ret = vfio_device_block_group(device);
if (ret)
return ret;
mutex_lock(&device->dev_set->lock);
/* one device cannot be bound twice */
if (df->access_granted) {
ret = -EINVAL;
goto out_unlock;
}
df->iommufd = iommufd_ctx_from_fd(bind.iommufd);
if (IS_ERR(df->iommufd)) {
ret = PTR_ERR(df->iommufd);
df->iommufd = NULL;
goto out_unlock;
}
/*
* Before the device open, get the KVM pointer currently
* associated with the device file (if there is) and obtain
* a reference. This reference is held until device closed.
* Save the pointer in the device for use by drivers.
*/
vfio_df_get_kvm_safe(df);
ret = vfio_df_open(df);
if (ret)
goto out_put_kvm;
ret = copy_to_user(&arg->out_devid, &df->devid,
sizeof(df->devid)) ? -EFAULT : 0;
if (ret)
goto out_close_device;
device->cdev_opened = true;
/*
* Paired with smp_load_acquire() in vfio_device_fops::ioctl/
* read/write/mmap
*/
smp_store_release(&df->access_granted, true);
mutex_unlock(&device->dev_set->lock);
return 0;
out_close_device:
vfio_df_close(df);
out_put_kvm:
vfio_device_put_kvm(device);
iommufd_ctx_put(df->iommufd);
df->iommufd = NULL;
out_unlock:
mutex_unlock(&device->dev_set->lock);
vfio_device_unblock_group(device);
return ret;
}
void vfio_df_unbind_iommufd(struct vfio_device_file *df)
{
struct vfio_device *device = df->device;
/*
* In the time of close, there is no contention with another one
* changing this flag. So read df->access_granted without lock
* and no smp_load_acquire() is ok.
*/
if (!df->access_granted)
return;
mutex_lock(&device->dev_set->lock);
vfio_df_close(df);
vfio_device_put_kvm(device);
iommufd_ctx_put(df->iommufd);
device->cdev_opened = false;
mutex_unlock(&device->dev_set->lock);
vfio_device_unblock_group(device);
}
int vfio_df_ioctl_attach_pt(struct vfio_device_file *df,
struct vfio_device_attach_iommufd_pt __user *arg)
{
struct vfio_device *device = df->device;
struct vfio_device_attach_iommufd_pt attach;
unsigned long minsz;
int ret;
minsz = offsetofend(struct vfio_device_attach_iommufd_pt, pt_id);
if (copy_from_user(&attach, arg, minsz))
return -EFAULT;
if (attach.argsz < minsz || attach.flags)
return -EINVAL;
mutex_lock(&device->dev_set->lock);
ret = device->ops->attach_ioas(device, &attach.pt_id);
if (ret)
goto out_unlock;
if (copy_to_user(&arg->pt_id, &attach.pt_id, sizeof(attach.pt_id))) {
ret = -EFAULT;
goto out_detach;
}
mutex_unlock(&device->dev_set->lock);
return 0;
out_detach:
device->ops->detach_ioas(device);
out_unlock:
mutex_unlock(&device->dev_set->lock);
return ret;
}
int vfio_df_ioctl_detach_pt(struct vfio_device_file *df,
struct vfio_device_detach_iommufd_pt __user *arg)
{
struct vfio_device *device = df->device;
struct vfio_device_detach_iommufd_pt detach;
unsigned long minsz;
minsz = offsetofend(struct vfio_device_detach_iommufd_pt, flags);
if (copy_from_user(&detach, arg, minsz))
return -EFAULT;
if (detach.argsz < minsz || detach.flags)
return -EINVAL;
mutex_lock(&device->dev_set->lock);
device->ops->detach_ioas(device);
mutex_unlock(&device->dev_set->lock);
return 0;
}
static char *vfio_device_devnode(const struct device *dev, umode_t *mode)
{
return kasprintf(GFP_KERNEL, "vfio/devices/%s", dev_name(dev));
}
int vfio_cdev_init(struct class *device_class)
{
device_class->devnode = vfio_device_devnode;
return alloc_chrdev_region(&device_devt, 0,
MINORMASK + 1, "vfio-dev");
}
void vfio_cdev_cleanup(void)
{
unregister_chrdev_region(device_devt, MINORMASK + 1);
}
drivers/vfio/fsl-mc/vfio_fsl_mc.c
View file @ 41ac3c2a
......@@ -593,6 +593,7 @@ static const struct vfio_device_ops vfio_fsl_mc_ops = {
.bind_iommufd = vfio_iommufd_physical_bind,
.unbind_iommufd = vfio_iommufd_physical_unbind,
.attach_ioas = vfio_iommufd_physical_attach_ioas,
.detach_ioas = vfio_iommufd_physical_detach_ioas,
};
static struct fsl_mc_driver vfio_fsl_mc_driver = {
......
drivers/vfio/group.c
View file @ 41ac3c2a
......@@ -160,17 +160,13 @@ static int vfio_group_ioctl_set_container(struct vfio_group *group,
static void vfio_device_group_get_kvm_safe(struct vfio_device *device)
{
spin_lock(&device->group->kvm_ref_lock);
if (!device->group->kvm)
goto unlock;
_vfio_device_get_kvm_safe(device, device->group->kvm);
unlock:
vfio_device_get_kvm_safe(device, device->group->kvm);
spin_unlock(&device->group->kvm_ref_lock);
}
static int vfio_device_group_open(struct vfio_device *device)
static int vfio_df_group_open(struct vfio_device_file *df)
{
struct vfio_device *device = df->device;
int ret;
mutex_lock(&device->group->group_lock);
......@@ -190,24 +186,62 @@ static int vfio_device_group_open(struct vfio_device *device)
if (device->open_count == 0)
vfio_device_group_get_kvm_safe(device);
ret = vfio_device_open(device, device->group->iommufd);
df->iommufd = device->group->iommufd;
if (df->iommufd && vfio_device_is_noiommu(device) && device->open_count == 0) {
/*
* Require no compat ioas to be assigned to proceed. The basic
* statement is that the user cannot have done something that
* implies they expected translation to exist
*/
if (!capable(CAP_SYS_RAWIO) ||
vfio_iommufd_device_has_compat_ioas(device, df->iommufd))
ret = -EPERM;
else
ret = 0;
goto out_put_kvm;
}
if (device->open_count == 0)
vfio_device_put_kvm(device);
ret = vfio_df_open(df);
if (ret)
goto out_put_kvm;
if (df->iommufd && device->open_count == 1) {
ret = vfio_iommufd_compat_attach_ioas(device, df->iommufd);
if (ret)
goto out_close_device;
}
/*
* Paired with smp_load_acquire() in vfio_device_fops::ioctl/
* read/write/mmap and vfio_file_has_device_access()
*/
smp_store_release(&df->access_granted, true);
mutex_unlock(&device->dev_set->lock);
mutex_unlock(&device->group->group_lock);
return 0;
out_close_device:
vfio_df_close(df);
out_put_kvm:
df->iommufd = NULL;
if (device->open_count == 0)
vfio_device_put_kvm(device);
mutex_unlock(&device->dev_set->lock);
out_unlock:
mutex_unlock(&device->group->group_lock);
return ret;
}
void vfio_device_group_close(struct vfio_device *device)
void vfio_df_group_close(struct vfio_device_file *df)
{
struct vfio_device *device = df->device;
mutex_lock(&device->group->group_lock);
mutex_lock(&device->dev_set->lock);
vfio_device_close(device, device->group->iommufd);
vfio_df_close(df);
df->iommufd = NULL;
if (device->open_count == 0)
vfio_device_put_kvm(device);
......@@ -218,19 +252,28 @@ void vfio_device_group_close(struct vfio_device *device)
static struct file *vfio_device_open_file(struct vfio_device *device)
{
struct vfio_device_file *df;
struct file *filep;
int ret;
ret = vfio_device_group_open(device);
if (ret)
df = vfio_allocate_device_file(device);
if (IS_ERR(df)) {
ret = PTR_ERR(df);
goto err_out;
}
df->group = device->group;
ret = vfio_df_group_open(df);
if (ret)
goto err_free;
/*
* We can't use anon_inode_getfd() because we need to modify
* the f_mode flags directly to allow more than just ioctls
*/
filep = anon_inode_getfile("[vfio-device]", &vfio_device_fops,
device, O_RDWR);
df, O_RDWR);
if (IS_ERR(filep)) {
ret = PTR_ERR(filep);
goto err_close_device;
......@@ -253,7 +296,9 @@ static struct file *vfio_device_open_file(struct vfio_device *device)
return filep;
err_close_device:
vfio_device_group_close(device);
vfio_df_group_close(df);
err_free:
kfree(df);
err_out:
return ERR_PTR(ret);
}
......@@ -357,6 +402,33 @@ static long vfio_group_fops_unl_ioctl(struct file *filep,
}
}
int vfio_device_block_group(struct vfio_device *device)
{
struct vfio_group *group = device->group;
int ret = 0;
mutex_lock(&group->group_lock);
if (group->opened_file) {
ret = -EBUSY;
goto out_unlock;
}
group->cdev_device_open_cnt++;
out_unlock:
mutex_unlock(&group->group_lock);
return ret;
}
void vfio_device_unblock_group(struct vfio_device *device)
{
struct vfio_group *group = device->group;
mutex_lock(&group->group_lock);
group->cdev_device_open_cnt--;
mutex_unlock(&group->group_lock);
}
static int vfio_group_fops_open(struct inode *inode, struct file *filep)
{
struct vfio_group *group =
......@@ -379,6 +451,11 @@ static int vfio_group_fops_open(struct inode *inode, struct file *filep)
goto out_unlock;
}
if (group->cdev_device_open_cnt) {
ret = -EBUSY;
goto out_unlock;
}
/*
* Do we need multiple instances of the group open? Seems not.
*/
......@@ -453,6 +530,7 @@ static void vfio_group_release(struct device *dev)
mutex_destroy(&group->device_lock);
mutex_destroy(&group->group_lock);
WARN_ON(group->iommu_group);
WARN_ON(group->cdev_device_open_cnt);
ida_free(&vfio.group_ida, MINOR(group->dev.devt));
kfree(group);
}
......@@ -604,16 +682,6 @@ static struct vfio_group *vfio_group_find_or_alloc(struct device *dev)
if (!iommu_group)
return ERR_PTR(-EINVAL);
/*
* VFIO always sets IOMMU_CACHE because we offer no way for userspace to
* restore cache coherency. It has to be checked here because it is only
* valid for cases where we are using iommu groups.
*/
if (!device_iommu_capable(dev, IOMMU_CAP_CACHE_COHERENCY)) {
iommu_group_put(iommu_group);
return ERR_PTR(-EINVAL);
}
mutex_lock(&vfio.group_lock);
group = vfio_group_find_from_iommu(iommu_group);
if (group) {
......@@ -745,6 +813,15 @@ bool vfio_device_has_container(struct vfio_device *device)
return device->group->container;
}
struct vfio_group *vfio_group_from_file(struct file *file)
{
struct vfio_group *group = file->private_data;
if (file->f_op != &vfio_group_fops)
return NULL;
return group;
}
/**
* vfio_file_iommu_group - Return the struct iommu_group for the vfio group file
* @file: VFIO group file
......@@ -755,13 +832,13 @@ bool vfio_device_has_container(struct vfio_device *device)
*/
struct iommu_group *vfio_file_iommu_group(struct file *file)
{
struct vfio_group *group = file->private_data;
struct vfio_group *group = vfio_group_from_file(file);
struct iommu_group *iommu_group = NULL;
if (!IS_ENABLED(CONFIG_SPAPR_TCE_IOMMU))
return NULL;
if (!vfio_file_is_group(file))
if (!group)
return NULL;
mutex_lock(&group->group_lock);
......@@ -775,33 +852,20 @@ struct iommu_group *vfio_file_iommu_group(struct file *file)
EXPORT_SYMBOL_GPL(vfio_file_iommu_group);
/**
* vfio_file_is_group - True if the file is usable with VFIO aPIS
* vfio_file_is_group - True if the file is a vfio group file
* @file: VFIO group file
*/
bool vfio_file_is_group(struct file *file)
{
return file->f_op == &vfio_group_fops;
return vfio_group_from_file(file);
}
EXPORT_SYMBOL_GPL(vfio_file_is_group);
/**
* vfio_file_enforced_coherent - True if the DMA associated with the VFIO file
* is always CPU cache coherent
* @file: VFIO group file
*
* Enforced coherency means that the IOMMU ignores things like the PCIe no-snoop
* bit in DMA transactions. A return of false indicates that the user has
* rights to access additional instructions such as wbinvd on x86.
*/
bool vfio_file_enforced_coherent(struct file *file)
bool vfio_group_enforced_coherent(struct vfio_group *group)
{
struct vfio_group *group = file->private_data;
struct vfio_device *device;
bool ret = true;
if (!vfio_file_is_group(file))
return true;
/*
* If the device does not have IOMMU_CAP_ENFORCE_CACHE_COHERENCY then
* any domain later attached to it will also not support it. If the cap
......@@ -819,28 +883,13 @@ bool vfio_file_enforced_coherent(struct file *file)
mutex_unlock(&group->device_lock);
return ret;
}
EXPORT_SYMBOL_GPL(vfio_file_enforced_coherent);
/**
* vfio_file_set_kvm - Link a kvm with VFIO drivers
* @file: VFIO group file
* @kvm: KVM to link
*
* When a VFIO device is first opened the KVM will be available in
* device->kvm if one was associated with the group.
*/
void vfio_file_set_kvm(struct file *file, struct kvm *kvm)
void vfio_group_set_kvm(struct vfio_group *group, struct kvm *kvm)
{
struct vfio_group *group = file->private_data;
if (!vfio_file_is_group(file))
return;
spin_lock(&group->kvm_ref_lock);
group->kvm = kvm;
spin_unlock(&group->kvm_ref_lock);
}
EXPORT_SYMBOL_GPL(vfio_file_set_kvm);
/**
* vfio_file_has_dev - True if the VFIO file is a handle for device
......@@ -851,9 +900,9 @@ EXPORT_SYMBOL_GPL(vfio_file_set_kvm);
*/
bool vfio_file_has_dev(struct file *file, struct vfio_device *device)
{
struct vfio_group *group = file->private_data;
struct vfio_group *group = vfio_group_from_file(file);
if (!vfio_file_is_group(file))
if (!group)
return false;
return group == device->group;
......
drivers/vfio/iommufd.c
View file @ 41ac3c2a
......@@ -10,53 +10,48 @@
MODULE_IMPORT_NS(IOMMUFD);
MODULE_IMPORT_NS(IOMMUFD_VFIO);
int vfio_iommufd_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx)
bool vfio_iommufd_device_has_compat_ioas(struct vfio_device *vdev,
struct iommufd_ctx *ictx)
{
u32 ioas_id;
return !iommufd_vfio_compat_ioas_get_id(ictx, &ioas_id);
}
int vfio_df_iommufd_bind(struct vfio_device_file *df)
{
struct vfio_device *vdev = df->device;
struct iommufd_ctx *ictx = df->iommufd;
lockdep_assert_held(&vdev->dev_set->lock);
return vdev->ops->bind_iommufd(vdev, ictx, &df->devid);
}
int vfio_iommufd_compat_attach_ioas(struct vfio_device *vdev,
struct iommufd_ctx *ictx)
{
u32 ioas_id;
u32 device_id;
int ret;
lockdep_assert_held(&vdev->dev_set->lock);
if (vfio_device_is_noiommu(vdev)) {
if (!capable(CAP_SYS_RAWIO))
return -EPERM;
/*
* Require no compat ioas to be assigned to proceed. The basic
* statement is that the user cannot have done something that
* implies they expected translation to exist
*/
if (!iommufd_vfio_compat_ioas_get_id(ictx, &ioas_id))
return -EPERM;
/* compat noiommu does not need to do ioas attach */
if (vfio_device_is_noiommu(vdev))
return 0;
}
ret = vdev->ops->bind_iommufd(vdev, ictx, &device_id);
if (ret)
return ret;
ret = iommufd_vfio_compat_ioas_get_id(ictx, &ioas_id);
if (ret)
goto err_unbind;
ret = vdev->ops->attach_ioas(vdev, &ioas_id);
if (ret)
goto err_unbind;
/*
* The legacy path has no way to return the device id or the selected
* pt_id
*/
return 0;
return ret;
err_unbind:
if (vdev->ops->unbind_iommufd)
vdev->ops->unbind_iommufd(vdev);
return ret;
/* The legacy path has no way to return the selected pt_id */
return vdev->ops->attach_ioas(vdev, &ioas_id);
}
void vfio_iommufd_unbind(struct vfio_device *vdev)
void vfio_df_iommufd_unbind(struct vfio_device_file *df)
{
struct vfio_device *vdev = df->device;
lockdep_assert_held(&vdev->dev_set->lock);
if (vfio_device_is_noiommu(vdev))
......@@ -66,6 +61,50 @@ void vfio_iommufd_unbind(struct vfio_device *vdev)
vdev->ops->unbind_iommufd(vdev);
}
struct iommufd_ctx *vfio_iommufd_device_ictx(struct vfio_device *vdev)
{
if (vdev->iommufd_device)
return iommufd_device_to_ictx(vdev->iommufd_device);
return NULL;
}
EXPORT_SYMBOL_GPL(vfio_iommufd_device_ictx);
static int vfio_iommufd_device_id(struct vfio_device *vdev)
{
if (vdev->iommufd_device)
return iommufd_device_to_id(vdev->iommufd_device);
return -EINVAL;
}
/*
* Return devid for a device.
* valid ID for the device that is owned by the ictx
* -ENOENT = device is owned but there is no ID
* -ENODEV or other error = device is not owned
*/
int vfio_iommufd_get_dev_id(struct vfio_device *vdev, struct iommufd_ctx *ictx)
{
struct iommu_group *group;
int devid;
if (vfio_iommufd_device_ictx(vdev) == ictx)
return vfio_iommufd_device_id(vdev);
group = iommu_group_get(vdev->dev);
if (!group)
return -ENODEV;
if (iommufd_ctx_has_group(ictx, group))
devid = -ENOENT;
else
devid = -ENODEV;
iommu_group_put(group);
return devid;
}
EXPORT_SYMBOL_GPL(vfio_iommufd_get_dev_id);
/*
* The physical standard ops mean that the iommufd_device is bound to the
* physical device vdev->dev that was provided to vfio_init_group_dev(). Drivers
......@@ -101,6 +140,14 @@ int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id)
{
int rc;
lockdep_assert_held(&vdev->dev_set->lock);
if (WARN_ON(!vdev->iommufd_device))
return -EINVAL;
if (vdev->iommufd_attached)
return -EBUSY;
rc = iommufd_device_attach(vdev->iommufd_device, pt_id);
if (rc)
return rc;
......@@ -109,6 +156,18 @@ int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id)
}
EXPORT_SYMBOL_GPL(vfio_iommufd_physical_attach_ioas);
void vfio_iommufd_physical_detach_ioas(struct vfio_device *vdev)
{
lockdep_assert_held(&vdev->dev_set->lock);
if (WARN_ON(!vdev->iommufd_device) || !vdev->iommufd_attached)
return;
iommufd_device_detach(vdev->iommufd_device);
vdev->iommufd_attached = false;
}
EXPORT_SYMBOL_GPL(vfio_iommufd_physical_detach_ioas);
/*
* The emulated standard ops mean that vfio_device is going to use the
* "mdev path" and will call vfio_pin_pages()/vfio_dma_rw(). Drivers using this
......@@ -172,3 +231,16 @@ int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id)
return 0;
}
EXPORT_SYMBOL_GPL(vfio_iommufd_emulated_attach_ioas);
void vfio_iommufd_emulated_detach_ioas(struct vfio_device *vdev)
{
lockdep_assert_held(&vdev->dev_set->lock);
if (WARN_ON(!vdev->iommufd_access) ||
!vdev->iommufd_attached)
return;
iommufd_access_detach(vdev->iommufd_access);
vdev->iommufd_attached = false;
}
EXPORT_SYMBOL_GPL(vfio_iommufd_emulated_detach_ioas);
drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
View file @ 41ac3c2a
......@@ -1373,6 +1373,7 @@ static const struct vfio_device_ops hisi_acc_vfio_pci_migrn_ops = {
.bind_iommufd = vfio_iommufd_physical_bind,
.unbind_iommufd = vfio_iommufd_physical_unbind,
.attach_ioas = vfio_iommufd_physical_attach_ioas,
.detach_ioas = vfio_iommufd_physical_detach_ioas,
};
static const struct vfio_device_ops hisi_acc_vfio_pci_ops = {
......@@ -1391,6 +1392,7 @@ static const struct vfio_device_ops hisi_acc_vfio_pci_ops = {
.bind_iommufd = vfio_iommufd_physical_bind,
.unbind_iommufd = vfio_iommufd_physical_unbind,
.attach_ioas = vfio_iommufd_physical_attach_ioas,
.detach_ioas = vfio_iommufd_physical_detach_ioas,
};
static int hisi_acc_vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
......
drivers/vfio/pci/mlx5/main.c
View file @ 41ac3c2a
......@@ -1320,6 +1320,7 @@ static const struct vfio_device_ops mlx5vf_pci_ops = {
.bind_iommufd = vfio_iommufd_physical_bind,
.unbind_iommufd = vfio_iommufd_physical_unbind,
.attach_ioas = vfio_iommufd_physical_attach_ioas,
.detach_ioas = vfio_iommufd_physical_detach_ioas,
};
static int mlx5vf_pci_probe(struct pci_dev *pdev,
......
drivers/vfio/pci/vfio_pci.c
View file @ 41ac3c2a
......@@ -141,6 +141,7 @@ static const struct vfio_device_ops vfio_pci_ops = {
.bind_iommufd = vfio_iommufd_physical_bind,
.unbind_iommufd = vfio_iommufd_physical_unbind,
.attach_ioas = vfio_iommufd_physical_attach_ioas,
.detach_ioas = vfio_iommufd_physical_detach_ioas,
};
static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
......
drivers/vfio/pci/vfio_pci_core.c
View file @ 41ac3c2a
This diff is collapsed.
drivers/vfio/platform/vfio_amba.c
View file @ 41ac3c2a
......@@ -119,6 +119,7 @@ static const struct vfio_device_ops vfio_amba_ops = {
.bind_iommufd = vfio_iommufd_physical_bind,
.unbind_iommufd = vfio_iommufd_physical_unbind,
.attach_ioas = vfio_iommufd_physical_attach_ioas,
.detach_ioas = vfio_iommufd_physical_detach_ioas,
};
static const struct amba_id pl330_ids[] = {
......
drivers/vfio/platform/vfio_platform.c
View file @ 41ac3c2a
......@@ -108,6 +108,7 @@ static const struct vfio_device_ops vfio_platform_ops = {
.bind_iommufd = vfio_iommufd_physical_bind,
.unbind_iommufd = vfio_iommufd_physical_unbind,
.attach_ioas = vfio_iommufd_physical_attach_ioas,
.detach_ioas = vfio_iommufd_physical_detach_ioas,
};
static struct platform_driver vfio_platform_driver = {
......
drivers/vfio/vfio.h
View file @ 41ac3c2a
......@@ -16,14 +16,32 @@ struct iommufd_ctx;
struct iommu_group;
struct vfio_container;
struct vfio_device_file {
struct vfio_device *device;
struct vfio_group *group;
u8 access_granted;
u32 devid; /* only valid when iommufd is valid */
spinlock_t kvm_ref_lock; /* protect kvm field */
struct kvm *kvm;
struct iommufd_ctx *iommufd; /* protected by struct vfio_device_set::lock */
};
void vfio_device_put_registration(struct vfio_device *device);
bool vfio_device_try_get_registration(struct vfio_device *device);
int vfio_device_open(struct vfio_device *device, struct iommufd_ctx *iommufd);
void vfio_device_close(struct vfio_device *device,
struct iommufd_ctx *iommufd);
int vfio_df_open(struct vfio_device_file *df);
void vfio_df_close(struct vfio_device_file *df);
struct vfio_device_file *
vfio_allocate_device_file(struct vfio_device *device);
extern const struct file_operations vfio_device_fops;
#ifdef CONFIG_VFIO_NOIOMMU
extern bool vfio_noiommu __read_mostly;
#else
enum { vfio_noiommu = false };
#endif
enum vfio_group_type {
/*
* Physical device with IOMMU backing.
......@@ -48,6 +66,7 @@ enum vfio_group_type {
VFIO_NO_IOMMU,
};
#if IS_ENABLED(CONFIG_VFIO_GROUP)
struct vfio_group {
struct device dev;
struct cdev cdev;
......@@ -74,8 +93,11 @@ struct vfio_group {
struct blocking_notifier_head notifier;
struct iommufd_ctx *iommufd;
spinlock_t kvm_ref_lock;
unsigned int cdev_device_open_cnt;
};
int vfio_device_block_group(struct vfio_device *device);
void vfio_device_unblock_group(struct vfio_device *device);
int vfio_device_set_group(struct vfio_device *device,
enum vfio_group_type type);
void vfio_device_remove_group(struct vfio_device *device);
......@@ -83,7 +105,10 @@ void vfio_device_group_register(struct vfio_device *device);
void vfio_device_group_unregister(struct vfio_device *device);
int vfio_device_group_use_iommu(struct vfio_device *device);
void vfio_device_group_unuse_iommu(struct vfio_device *device);
void vfio_device_group_close(struct vfio_device *device);
void vfio_df_group_close(struct vfio_device_file *df);
struct vfio_group *vfio_group_from_file(struct file *file);
bool vfio_group_enforced_coherent(struct vfio_group *group);
void vfio_group_set_kvm(struct vfio_group *group, struct kvm *kvm);
bool vfio_device_has_container(struct vfio_device *device);
int __init vfio_group_init(void);
void vfio_group_cleanup(void);
......@@ -93,6 +118,82 @@ static inline bool vfio_device_is_noiommu(struct vfio_device *vdev)
return IS_ENABLED(CONFIG_VFIO_NOIOMMU) &&
vdev->group->type == VFIO_NO_IOMMU;
}
#else
struct vfio_group;
static inline int vfio_device_block_group(struct vfio_device *device)
{
return 0;
}
static inline void vfio_device_unblock_group(struct vfio_device *device)
{
}
static inline int vfio_device_set_group(struct vfio_device *device,
enum vfio_group_type type)
{
return 0;
}
static inline void vfio_device_remove_group(struct vfio_device *device)
{
}
static inline void vfio_device_group_register(struct vfio_device *device)
{
}
static inline void vfio_device_group_unregister(struct vfio_device *device)
{
}
static inline int vfio_device_group_use_iommu(struct vfio_device *device)
{
return -EOPNOTSUPP;
}
static inline void vfio_device_group_unuse_iommu(struct vfio_device *device)
{
}
static inline void vfio_df_group_close(struct vfio_device_file *df)
{
}
static inline struct vfio_group *vfio_group_from_file(struct file *file)
{
return NULL;
}
static inline bool vfio_group_enforced_coherent(struct vfio_group *group)
{
return true;
}
static inline void vfio_group_set_kvm(struct vfio_group *group, struct kvm *kvm)
{
}
static inline bool vfio_device_has_container(struct vfio_device *device)
{
return false;
}
static inline int __init vfio_group_init(void)
{
return 0;
}
static inline void vfio_group_cleanup(void)
{
}
static inline bool vfio_device_is_noiommu(struct vfio_device *vdev)
{
return false;
}
#endif /* CONFIG_VFIO_GROUP */
#if IS_ENABLED(CONFIG_VFIO_CONTAINER)
/**
......@@ -217,20 +318,109 @@ static inline void vfio_container_cleanup(void)
#endif
#if IS_ENABLED(CONFIG_IOMMUFD)
int vfio_iommufd_bind(struct vfio_device *device, struct iommufd_ctx *ictx);
void vfio_iommufd_unbind(struct vfio_device *device);
bool vfio_iommufd_device_has_compat_ioas(struct vfio_device *vdev,
struct iommufd_ctx *ictx);
int vfio_df_iommufd_bind(struct vfio_device_file *df);
void vfio_df_iommufd_unbind(struct vfio_device_file *df);
int vfio_iommufd_compat_attach_ioas(struct vfio_device *device,
struct iommufd_ctx *ictx);
#else
static inline int vfio_iommufd_bind(struct vfio_device *device,
static inline bool
vfio_iommufd_device_has_compat_ioas(struct vfio_device *vdev,
struct iommufd_ctx *ictx)
{
return false;
}
static inline int vfio_df_iommufd_bind(struct vfio_device_file *fd)
{
return -EOPNOTSUPP;
}
static inline void vfio_iommufd_unbind(struct vfio_device *device)
static inline void vfio_df_iommufd_unbind(struct vfio_device_file *df)
{
}
static inline int
vfio_iommufd_compat_attach_ioas(struct vfio_device *device,
struct iommufd_ctx *ictx)
{
return -EOPNOTSUPP;
}
#endif
int vfio_df_ioctl_attach_pt(struct vfio_device_file *df,
struct vfio_device_attach_iommufd_pt __user *arg);
int vfio_df_ioctl_detach_pt(struct vfio_device_file *df,
struct vfio_device_detach_iommufd_pt __user *arg);
#if IS_ENABLED(CONFIG_VFIO_DEVICE_CDEV)
void vfio_init_device_cdev(struct vfio_device *device);
static inline int vfio_device_add(struct vfio_device *device)
{
/* cdev does not support noiommu device */
if (vfio_device_is_noiommu(device))
return device_add(&device->device);
vfio_init_device_cdev(device);
return cdev_device_add(&device->cdev, &device->device);
}
static inline void vfio_device_del(struct vfio_device *device)
{
if (vfio_device_is_noiommu(device))
device_del(&device->device);
else
cdev_device_del(&device->cdev, &device->device);
}
int vfio_device_fops_cdev_open(struct inode *inode, struct file *filep);
long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df,
struct vfio_device_bind_iommufd __user *arg);
void vfio_df_unbind_iommufd(struct vfio_device_file *df);
int vfio_cdev_init(struct class *device_class);
void vfio_cdev_cleanup(void);
#else
static inline void vfio_init_device_cdev(struct vfio_device *device)
{
}
static inline int vfio_device_add(struct vfio_device *device)
{
return device_add(&device->device);
}
static inline void vfio_device_del(struct vfio_device *device)
{
device_del(&device->device);
}
static inline int vfio_device_fops_cdev_open(struct inode *inode,
struct file *filep)
{
return 0;
}
static inline long vfio_df_ioctl_bind_iommufd(struct vfio_device_file *df,
struct vfio_device_bind_iommufd __user *arg)
{
return -ENOTTY;
}
static inline void vfio_df_unbind_iommufd(struct vfio_device_file *df)
{
}
static inline int vfio_cdev_init(struct class *device_class)
{
return 0;
}
static inline void vfio_cdev_cleanup(void)
{
}
#endif /* CONFIG_VFIO_DEVICE_CDEV */
#if IS_ENABLED(CONFIG_VFIO_VIRQFD)
int __init vfio_virqfd_init(void);
void vfio_virqfd_exit(void);
......@@ -244,18 +434,12 @@ static inline void vfio_virqfd_exit(void)
}
#endif
#ifdef CONFIG_VFIO_NOIOMMU
extern bool vfio_noiommu __read_mostly;
#else
enum { vfio_noiommu = false };
#endif
#ifdef CONFIG_HAVE_KVM
void _vfio_device_get_kvm_safe(struct vfio_device *device, struct kvm *kvm);
void vfio_device_get_kvm_safe(struct vfio_device *device, struct kvm *kvm);
void vfio_device_put_kvm(struct vfio_device *device);
#else
static inline void _vfio_device_get_kvm_safe(struct vfio_device *device,
struct kvm *kvm)
static inline void vfio_device_get_kvm_safe(struct vfio_device *device,
struct kvm *kvm)
{
}
......
drivers/vfio/vfio_main.c
View file @ 41ac3c2a
This diff is collapsed.
include/linux/iommufd.h
View file @ 41ac3c2a
......@@ -16,6 +16,7 @@ struct page;
struct iommufd_ctx;
struct iommufd_access;
struct file;
struct iommu_group;
struct iommufd_device *iommufd_device_bind(struct iommufd_ctx *ictx,
struct device *dev, u32 *id);
......@@ -24,6 +25,9 @@ void iommufd_device_unbind(struct iommufd_device *idev);
int iommufd_device_attach(struct iommufd_device *idev, u32 *pt_id);
void iommufd_device_detach(struct iommufd_device *idev);
struct iommufd_ctx *iommufd_device_to_ictx(struct iommufd_device *idev);
u32 iommufd_device_to_id(struct iommufd_device *idev);
struct iommufd_access_ops {
u8 needs_pin_pages : 1;
void (*unmap)(void *data, unsigned long iova, unsigned long length);
......@@ -44,12 +48,15 @@ iommufd_access_create(struct iommufd_ctx *ictx,
const struct iommufd_access_ops *ops, void *data, u32 *id);
void iommufd_access_destroy(struct iommufd_access *access);
int iommufd_access_attach(struct iommufd_access *access, u32 ioas_id);
void iommufd_access_detach(struct iommufd_access *access);
void iommufd_ctx_get(struct iommufd_ctx *ictx);
#if IS_ENABLED(CONFIG_IOMMUFD)
struct iommufd_ctx *iommufd_ctx_from_file(struct file *file);
struct iommufd_ctx *iommufd_ctx_from_fd(int fd);
void iommufd_ctx_put(struct iommufd_ctx *ictx);
bool iommufd_ctx_has_group(struct iommufd_ctx *ictx, struct iommu_group *group);
int iommufd_access_pin_pages(struct iommufd_access *access, unsigned long iova,
unsigned long length, struct page **out_pages,
......
include/linux/vfio.h
View file @ 41ac3c2a
......@@ -13,6 +13,7 @@
#include <linux/mm.h>
#include <linux/workqueue.h>
#include <linux/poll.h>
#include <linux/cdev.h>
#include <uapi/linux/vfio.h>
#include <linux/iova_bitmap.h>
......@@ -42,7 +43,11 @@ struct vfio_device {
*/
const struct vfio_migration_ops *mig_ops;
const struct vfio_log_ops *log_ops;
#if IS_ENABLED(CONFIG_VFIO_GROUP)
struct vfio_group *group;
struct list_head group_next;
struct list_head iommu_entry;
#endif
struct vfio_device_set *dev_set;
struct list_head dev_set_list;
unsigned int migration_flags;
......@@ -51,17 +56,19 @@ struct vfio_device {
/* Members below here are private, not for driver use */
unsigned int index;
struct device device; /* device.kref covers object life circle */
#if IS_ENABLED(CONFIG_VFIO_DEVICE_CDEV)
struct cdev cdev;
#endif
refcount_t refcount; /* user count on registered device*/
unsigned int open_count;
struct completion comp;
struct list_head group_next;
struct list_head iommu_entry;
struct iommufd_access *iommufd_access;
void (*put_kvm)(struct kvm *kvm);
#if IS_ENABLED(CONFIG_IOMMUFD)
struct iommufd_device *iommufd_device;
bool iommufd_attached;
u8 iommufd_attached:1;
#endif
u8 cdev_opened:1;
};
/**
......@@ -73,7 +80,9 @@ struct vfio_device {
* @bind_iommufd: Called when binding the device to an iommufd
* @unbind_iommufd: Opposite of bind_iommufd
* @attach_ioas: Called when attaching device to an IOAS/HWPT managed by the
* bound iommufd. Undo in unbind_iommufd.
* bound iommufd. Undo in unbind_iommufd if @detach_ioas is not
* called.
* @detach_ioas: Opposite of attach_ioas
* @open_device: Called when the first file descriptor is opened for this device
* @close_device: Opposite of open_device
* @read: Perform read(2) on device file descriptor
......@@ -97,6 +106,7 @@ struct vfio_device_ops {
struct iommufd_ctx *ictx, u32 *out_device_id);
void (*unbind_iommufd)(struct vfio_device *vdev);
int (*attach_ioas)(struct vfio_device *vdev, u32 *pt_id);
void (*detach_ioas)(struct vfio_device *vdev);
int (*open_device)(struct vfio_device *vdev);
void (*close_device)(struct vfio_device *vdev);
ssize_t (*read)(struct vfio_device *vdev, char __user *buf,
......@@ -114,15 +124,31 @@ struct vfio_device_ops {
};
#if IS_ENABLED(CONFIG_IOMMUFD)
struct iommufd_ctx *vfio_iommufd_device_ictx(struct vfio_device *vdev);
int vfio_iommufd_get_dev_id(struct vfio_device *vdev, struct iommufd_ctx *ictx);
int vfio_iommufd_physical_bind(struct vfio_device *vdev,
struct iommufd_ctx *ictx, u32 *out_device_id);
void vfio_iommufd_physical_unbind(struct vfio_device *vdev);
int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
void vfio_iommufd_physical_detach_ioas(struct vfio_device *vdev);
int vfio_iommufd_emulated_bind(struct vfio_device *vdev,
struct iommufd_ctx *ictx, u32 *out_device_id);
void vfio_iommufd_emulated_unbind(struct vfio_device *vdev);
int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
void vfio_iommufd_emulated_detach_ioas(struct vfio_device *vdev);
#else
static inline struct iommufd_ctx *
vfio_iommufd_device_ictx(struct vfio_device *vdev)
{
return NULL;
}
static inline int
vfio_iommufd_get_dev_id(struct vfio_device *vdev, struct iommufd_ctx *ictx)
{
return VFIO_PCI_DEVID_NOT_OWNED;
}
#define vfio_iommufd_physical_bind \
((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx, \
u32 *out_device_id)) NULL)
......@@ -130,6 +156,8 @@ int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
((void (*)(struct vfio_device *vdev)) NULL)
#define vfio_iommufd_physical_attach_ioas \
((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL)
#define vfio_iommufd_physical_detach_ioas \
((void (*)(struct vfio_device *vdev)) NULL)
#define vfio_iommufd_emulated_bind \
((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx, \
u32 *out_device_id)) NULL)
......@@ -137,8 +165,15 @@ int vfio_iommufd_emulated_attach_ioas(struct vfio_device *vdev, u32 *pt_id);
((void (*)(struct vfio_device *vdev)) NULL)
#define vfio_iommufd_emulated_attach_ioas \
((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL)
#define vfio_iommufd_emulated_detach_ioas \
((void (*)(struct vfio_device *vdev)) NULL)
#endif
static inline bool vfio_device_cdev_opened(struct vfio_device *device)
{
return device->cdev_opened;
}
/**
* struct vfio_migration_ops - VFIO bus device driver migration callbacks
*
......@@ -239,6 +274,9 @@ void vfio_unregister_group_dev(struct vfio_device *device);
int vfio_assign_device_set(struct vfio_device *device, void *set_id);
unsigned int vfio_device_set_open_count(struct vfio_device_set *dev_set);
struct vfio_device *
vfio_find_device_in_devset(struct vfio_device_set *dev_set,
struct device *dev);
int vfio_mig_get_next_state(struct vfio_device *device,
enum vfio_device_mig_state cur_fsm,
......@@ -248,11 +286,29 @@ int vfio_mig_get_next_state(struct vfio_device *device,
/*
* External user API
*/
#if IS_ENABLED(CONFIG_VFIO_GROUP)
struct iommu_group *vfio_file_iommu_group(struct file *file);
bool vfio_file_is_group(struct file *file);
bool vfio_file_has_dev(struct file *file, struct vfio_device *device);
#else
static inline struct iommu_group *vfio_file_iommu_group(struct file *file)
{
return NULL;
}
static inline bool vfio_file_is_group(struct file *file)
{
return false;
}
static inline bool vfio_file_has_dev(struct file *file, struct vfio_device *device)
{
return false;
}
#endif
bool vfio_file_is_valid(struct file *file);
bool vfio_file_enforced_coherent(struct file *file);
void vfio_file_set_kvm(struct file *file, struct kvm *kvm);
bool vfio_file_has_dev(struct file *file, struct vfio_device *device);
#define VFIO_PIN_PAGES_MAX_ENTRIES (PAGE_SIZE/sizeof(unsigned long))
......
include/uapi/linux/kvm.h
View file @ 41ac3c2a
......@@ -1418,9 +1418,16 @@ struct kvm_device_attr {
__u64 addr; /* userspace address of attr data */
};
#define KVM_DEV_VFIO_GROUP 1
#define KVM_DEV_VFIO_GROUP_ADD 1
#define KVM_DEV_VFIO_GROUP_DEL 2
#define KVM_DEV_VFIO_FILE 1
#define KVM_DEV_VFIO_FILE_ADD 1
#define KVM_DEV_VFIO_FILE_DEL 2
/* KVM_DEV_VFIO_GROUP aliases are for compile time uapi compatibility */
#define KVM_DEV_VFIO_GROUP KVM_DEV_VFIO_FILE
#define KVM_DEV_VFIO_GROUP_ADD KVM_DEV_VFIO_FILE_ADD
#define KVM_DEV_VFIO_GROUP_DEL KVM_DEV_VFIO_FILE_DEL
#define KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE 3
enum kvm_device_type {
......
include/uapi/linux/vfio.h
View file @ 41ac3c2a
......@@ -677,11 +677,60 @@ enum {
* VFIO_DEVICE_GET_PCI_HOT_RESET_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 12,
* struct vfio_pci_hot_reset_info)
*
* This command is used to query the affected devices in the hot reset for
* a given device.
*
* This command always reports the segment, bus, and devfn information for
* each affected device, and selectively reports the group_id or devid per
* the way how the calling device is opened.
*
* - If the calling device is opened via the traditional group/container
* API, group_id is reported. User should check if it has owned all
* the affected devices and provides a set of group fds to prove the
* ownership in VFIO_DEVICE_PCI_HOT_RESET ioctl.
*
* - If the calling device is opened as a cdev, devid is reported.
* Flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set to indicate this
* data type. All the affected devices should be represented in
* the dev_set, ex. bound to a vfio driver, and also be owned by
* this interface which is determined by the following conditions:
* 1) Has a valid devid within the iommufd_ctx of the calling device.
* Ownership cannot be determined across separate iommufd_ctx and
* the cdev calling conventions do not support a proof-of-ownership
* model as provided in the legacy group interface. In this case
* valid devid with value greater than zero is provided in the return
* structure.
* 2) Does not have a valid devid within the iommufd_ctx of the calling
* device, but belongs to the same IOMMU group as the calling device
* or another opened device that has a valid devid within the
* iommufd_ctx of the calling device. This provides implicit ownership
* for devices within the same DMA isolation context. In this case
* the devid value of VFIO_PCI_DEVID_OWNED is provided in the return
* structure.
*
* A devid value of VFIO_PCI_DEVID_NOT_OWNED is provided in the return
* structure for affected devices where device is NOT represented in the
* dev_set or ownership is not available. Such devices prevent the use
* of VFIO_DEVICE_PCI_HOT_RESET ioctl outside of the proof-of-ownership
* calling conventions (ie. via legacy group accessed devices). Flag
* VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED would be set when all the
* affected devices are represented in the dev_set and also owned by
* the user. This flag is available only when
* flag VFIO_PCI_HOT_RESET_FLAG_DEV_ID is set, otherwise reserved.
* When set, user could invoke VFIO_DEVICE_PCI_HOT_RESET with a zero
* length fd array on the calling device as the ownership is validated
* by iommufd_ctx.
*
* Return: 0 on success, -errno on failure:
* -enospc = insufficient buffer, -enodev = unsupported for device.
*/
struct vfio_pci_dependent_device {
__u32 group_id;
union {
__u32 group_id;
__u32 devid;
#define VFIO_PCI_DEVID_OWNED 0
#define VFIO_PCI_DEVID_NOT_OWNED -1
};
__u16 segment;
__u8 bus;
__u8 devfn; /* Use PCI_SLOT/PCI_FUNC */
......@@ -690,6 +739,8 @@ struct vfio_pci_dependent_device {
struct vfio_pci_hot_reset_info {
__u32 argsz;
__u32 flags;
#define VFIO_PCI_HOT_RESET_FLAG_DEV_ID (1 << 0)
#define VFIO_PCI_HOT_RESET_FLAG_DEV_ID_OWNED (1 << 1)
__u32 count;
struct vfio_pci_dependent_device devices[];
};
......@@ -700,6 +751,24 @@ struct vfio_pci_hot_reset_info {
* VFIO_DEVICE_PCI_HOT_RESET - _IOW(VFIO_TYPE, VFIO_BASE + 13,
* struct vfio_pci_hot_reset)
*
* A PCI hot reset results in either a bus or slot reset which may affect
* other devices sharing the bus/slot. The calling user must have
* ownership of the full set of affected devices as determined by the
* VFIO_DEVICE_GET_PCI_HOT_RESET_INFO ioctl.
*
* When called on a device file descriptor acquired through the vfio
* group interface, the user is required to provide proof of ownership
* of those affected devices via the group_fds array in struct
* vfio_pci_hot_reset.
*
* When called on a direct cdev opened vfio device, the flags field of
* struct vfio_pci_hot_reset_info reports the ownership status of the
* affected devices and this ioctl must be called with an empty group_fds
* array. See above INFO ioctl definition for ownership requirements.
*
* Mixed usage of legacy groups and cdevs across the set of affected
* devices is not supported.
*
* Return: 0 on success, -errno on failure.
*/
struct vfio_pci_hot_reset {
......@@ -828,6 +897,77 @@ struct vfio_device_feature {
#define VFIO_DEVICE_FEATURE _IO(VFIO_TYPE, VFIO_BASE + 17)
/*
* VFIO_DEVICE_BIND_IOMMUFD - _IOR(VFIO_TYPE, VFIO_BASE + 18,
* struct vfio_device_bind_iommufd)
* @argsz: User filled size of this data.
* @flags: Must be 0.
* @iommufd: iommufd to bind.
* @out_devid: The device id generated by this bind. devid is a handle for
* this device/iommufd bond and can be used in IOMMUFD commands.
*
* Bind a vfio_device to the specified iommufd.
*
* User is restricted from accessing the device before the binding operation
* is completed. Only allowed on cdev fds.
*
* Unbind is automatically conducted when device fd is closed.
*
* Return: 0 on success, -errno on failure.
*/
struct vfio_device_bind_iommufd {
__u32 argsz;
__u32 flags;
__s32 iommufd;
__u32 out_devid;
};
#define VFIO_DEVICE_BIND_IOMMUFD _IO(VFIO_TYPE, VFIO_BASE + 18)
/*
* VFIO_DEVICE_ATTACH_IOMMUFD_PT - _IOW(VFIO_TYPE, VFIO_BASE + 19,
* struct vfio_device_attach_iommufd_pt)
* @argsz: User filled size of this data.
* @flags: Must be 0.
* @pt_id: Input the target id which can represent an ioas or a hwpt
* allocated via iommufd subsystem.
* Output the input ioas id or the attached hwpt id which could
* be the specified hwpt itself or a hwpt automatically created
* for the specified ioas by kernel during the attachment.
*
* Associate the device with an address space within the bound iommufd.
* Undo by VFIO_DEVICE_DETACH_IOMMUFD_PT or device fd close. This is only
* allowed on cdev fds.
*
* Return: 0 on success, -errno on failure.
*/
struct vfio_device_attach_iommufd_pt {
__u32 argsz;
__u32 flags;
__u32 pt_id;
};
#define VFIO_DEVICE_ATTACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 19)
/*
* VFIO_DEVICE_DETACH_IOMMUFD_PT - _IOW(VFIO_TYPE, VFIO_BASE + 20,
* struct vfio_device_detach_iommufd_pt)
* @argsz: User filled size of this data.
* @flags: Must be 0.
*
* Remove the association of the device and its current associated address
* space. After it, the device should be in a blocking DMA state. This is only
* allowed on cdev fds.
*
* Return: 0 on success, -errno on failure.
*/
struct vfio_device_detach_iommufd_pt {
__u32 argsz;
__u32 flags;
};
#define VFIO_DEVICE_DETACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 20)
/*
* Provide support for setting a PCI VF Token, which is used as a shared
* secret between PF and VF drivers. This feature may only be set on a
......
samples/vfio-mdev/mbochs.c
View file @ 41ac3c2a
......@@ -1377,6 +1377,7 @@ static const struct vfio_device_ops mbochs_dev_ops = {
.bind_iommufd = vfio_iommufd_emulated_bind,
.unbind_iommufd = vfio_iommufd_emulated_unbind,
.attach_ioas = vfio_iommufd_emulated_attach_ioas,
.detach_ioas = vfio_iommufd_emulated_detach_ioas,
};
static struct mdev_driver mbochs_driver = {
......
samples/vfio-mdev/mdpy.c
View file @ 41ac3c2a
......@@ -666,6 +666,7 @@ static const struct vfio_device_ops mdpy_dev_ops = {
.bind_iommufd = vfio_iommufd_emulated_bind,
.unbind_iommufd = vfio_iommufd_emulated_unbind,
.attach_ioas = vfio_iommufd_emulated_attach_ioas,
.detach_ioas = vfio_iommufd_emulated_detach_ioas,
};
static struct mdev_driver mdpy_driver = {
......
samples/vfio-mdev/mtty.c
View file @ 41ac3c2a
......@@ -1272,6 +1272,7 @@ static const struct vfio_device_ops mtty_dev_ops = {
.bind_iommufd = vfio_iommufd_emulated_bind,
.unbind_iommufd = vfio_iommufd_emulated_unbind,
.attach_ioas = vfio_iommufd_emulated_attach_ioas,
.detach_ioas = vfio_iommufd_emulated_detach_ioas,
};
static struct mdev_driver mtty_driver = {
......
virt/kvm/vfio.c
View file @ 41ac3c2a
......@@ -21,7 +21,7 @@
#include <asm/kvm_ppc.h>
#endif
struct kvm_vfio_group {
struct kvm_vfio_file {
struct list_head node;
struct file *file;
#ifdef CONFIG_SPAPR_TCE_IOMMU
......@@ -30,7 +30,7 @@ struct kvm_vfio_group {
};
struct kvm_vfio {
struct list_head group_list;
struct list_head file_list;
struct mutex lock;
bool noncoherent;
};
......@@ -64,18 +64,18 @@ static bool kvm_vfio_file_enforced_coherent(struct file *file)
return ret;
}
static bool kvm_vfio_file_is_group(struct file *file)
static bool kvm_vfio_file_is_valid(struct file *file)
{
bool (*fn)(struct file *file);
bool ret;
fn = symbol_get(vfio_file_is_group);
fn = symbol_get(vfio_file_is_valid);
if (!fn)
return false;
ret = fn(file);
symbol_put(vfio_file_is_group);
symbol_put(vfio_file_is_valid);
return ret;
}
......@@ -98,34 +98,35 @@ static struct iommu_group *kvm_vfio_file_iommu_group(struct file *file)
}
static void kvm_spapr_tce_release_vfio_group(struct kvm *kvm,
struct kvm_vfio_group *kvg)
struct kvm_vfio_file *kvf)
{
if (WARN_ON_ONCE(!kvg->iommu_group))
if (WARN_ON_ONCE(!kvf->iommu_group))
return;
kvm_spapr_tce_release_iommu_group(kvm, kvg->iommu_group);
iommu_group_put(kvg->iommu_group);
kvg->iommu_group = NULL;
kvm_spapr_tce_release_iommu_group(kvm, kvf->iommu_group);
iommu_group_put(kvf->iommu_group);
kvf->iommu_group = NULL;
}
#endif
/*
* Groups can use the same or different IOMMU domains. If the same then
* adding a new group may change the coherency of groups we've previously
* been told about. We don't want to care about any of that so we retest
* each group and bail as soon as we find one that's noncoherent. This
* means we only ever [un]register_noncoherent_dma once for the whole device.
* Groups/devices can use the same or different IOMMU domains. If the same
* then adding a new group/device may change the coherency of groups/devices
* we've previously been told about. We don't want to care about any of
* that so we retest each group/device and bail as soon as we find one that's
* noncoherent. This means we only ever [un]register_noncoherent_dma once
* for the whole device.
*/
static void kvm_vfio_update_coherency(struct kvm_device *dev)
{
struct kvm_vfio *kv = dev->private;
bool noncoherent = false;
struct kvm_vfio_group *kvg;
struct kvm_vfio_file *kvf;
mutex_lock(&kv->lock);
list_for_each_entry(kvg, &kv->group_list, node) {
if (!kvm_vfio_file_enforced_coherent(kvg->file)) {
list_for_each_entry(kvf, &kv->file_list, node) {
if (!kvm_vfio_file_enforced_coherent(kvf->file)) {
noncoherent = true;
break;
}
......@@ -143,10 +144,10 @@ static void kvm_vfio_update_coherency(struct kvm_device *dev)
mutex_unlock(&kv->lock);
}
static int kvm_vfio_group_add(struct kvm_device *dev, unsigned int fd)
static int kvm_vfio_file_add(struct kvm_device *dev, unsigned int fd)
{
struct kvm_vfio *kv = dev->private;
struct kvm_vfio_group *kvg;
struct kvm_vfio_file *kvf;
struct file *filp;
int ret;
......@@ -154,35 +155,35 @@ static int kvm_vfio_group_add(struct kvm_device *dev, unsigned int fd)
if (!filp)
return -EBADF;
/* Ensure the FD is a vfio group FD.*/
if (!kvm_vfio_file_is_group(filp)) {
/* Ensure the FD is a vfio FD. */
if (!kvm_vfio_file_is_valid(filp)) {
ret = -EINVAL;
goto err_fput;
}
mutex_lock(&kv->lock);
list_for_each_entry(kvg, &kv->group_list, node) {
if (kvg->file == filp) {
list_for_each_entry(kvf, &kv->file_list, node) {
if (kvf->file == filp) {
ret = -EEXIST;
goto err_unlock;
}
}
kvg = kzalloc(sizeof(*kvg), GFP_KERNEL_ACCOUNT);
if (!kvg) {
kvf = kzalloc(sizeof(*kvf), GFP_KERNEL_ACCOUNT);
if (!kvf) {
ret = -ENOMEM;
goto err_unlock;
}
kvg->file = filp;
list_add_tail(&kvg->node, &kv->group_list);
kvf->file = filp;
list_add_tail(&kvf->node, &kv->file_list);
kvm_arch_start_assignment(dev->kvm);
mutex_unlock(&kv->lock);
kvm_vfio_file_set_kvm(kvg->file, dev->kvm);
kvm_vfio_file_set_kvm(kvf->file, dev->kvm);
kvm_vfio_update_coherency(dev);
return 0;
......@@ -193,10 +194,10 @@ static int kvm_vfio_group_add(struct kvm_device *dev, unsigned int fd)
return ret;
}
static int kvm_vfio_group_del(struct kvm_device *dev, unsigned int fd)
static int kvm_vfio_file_del(struct kvm_device *dev, unsigned int fd)
{
struct kvm_vfio *kv = dev->private;
struct kvm_vfio_group *kvg;
struct kvm_vfio_file *kvf;
struct fd f;
int ret;
......@@ -208,18 +209,18 @@ static int kvm_vfio_group_del(struct kvm_device *dev, unsigned int fd)
mutex_lock(&kv->lock);
list_for_each_entry(kvg, &kv->group_list, node) {
if (kvg->file != f.file)
list_for_each_entry(kvf, &kv->file_list, node) {
if (kvf->file != f.file)
continue;
list_del(&kvg->node);
list_del(&kvf->node);
kvm_arch_end_assignment(dev->kvm);
#ifdef CONFIG_SPAPR_TCE_IOMMU
kvm_spapr_tce_release_vfio_group(dev->kvm, kvg);
kvm_spapr_tce_release_vfio_group(dev->kvm, kvf);
#endif
kvm_vfio_file_set_kvm(kvg->file, NULL);
fput(kvg->file);
kfree(kvg);
kvm_vfio_file_set_kvm(kvf->file, NULL);
fput(kvf->file);
kfree(kvf);
ret = 0;
break;
}
......@@ -234,12 +235,12 @@ static int kvm_vfio_group_del(struct kvm_device *dev, unsigned int fd)
}
#ifdef CONFIG_SPAPR_TCE_IOMMU
static int kvm_vfio_group_set_spapr_tce(struct kvm_device *dev,
void __user *arg)
static int kvm_vfio_file_set_spapr_tce(struct kvm_device *dev,
void __user *arg)
{
struct kvm_vfio_spapr_tce param;
struct kvm_vfio *kv = dev->private;
struct kvm_vfio_group *kvg;
struct kvm_vfio_file *kvf;
struct fd f;
int ret;
......@@ -254,20 +255,20 @@ static int kvm_vfio_group_set_spapr_tce(struct kvm_device *dev,
mutex_lock(&kv->lock);
list_for_each_entry(kvg, &kv->group_list, node) {
if (kvg->file != f.file)
list_for_each_entry(kvf, &kv->file_list, node) {
if (kvf->file != f.file)
continue;
if (!kvg->iommu_group) {
kvg->iommu_group = kvm_vfio_file_iommu_group(kvg->file);
if (WARN_ON_ONCE(!kvg->iommu_group)) {
if (!kvf->iommu_group) {
kvf->iommu_group = kvm_vfio_file_iommu_group(kvf->file);
if (WARN_ON_ONCE(!kvf->iommu_group)) {
ret = -EIO;
goto err_fdput;
}
}
ret = kvm_spapr_tce_attach_iommu_group(dev->kvm, param.tablefd,
kvg->iommu_group);
kvf->iommu_group);
break;
}
......@@ -278,26 +279,26 @@ static int kvm_vfio_group_set_spapr_tce(struct kvm_device *dev,
}
#endif
static int kvm_vfio_set_group(struct kvm_device *dev, long attr,
void __user *arg)
static int kvm_vfio_set_file(struct kvm_device *dev, long attr,
void __user *arg)
{
int32_t __user *argp = arg;
int32_t fd;
switch (attr) {
case KVM_DEV_VFIO_GROUP_ADD:
case KVM_DEV_VFIO_FILE_ADD:
if (get_user(fd, argp))
return -EFAULT;
return kvm_vfio_group_add(dev, fd);
return kvm_vfio_file_add(dev, fd);
case KVM_DEV_VFIO_GROUP_DEL:
case KVM_DEV_VFIO_FILE_DEL:
if (get_user(fd, argp))
return -EFAULT;
return kvm_vfio_group_del(dev, fd);
return kvm_vfio_file_del(dev, fd);
#ifdef CONFIG_SPAPR_TCE_IOMMU
case KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE:
return kvm_vfio_group_set_spapr_tce(dev, arg);
return kvm_vfio_file_set_spapr_tce(dev, arg);
#endif
}
......@@ -308,9 +309,9 @@ static int kvm_vfio_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
switch (attr->group) {
case KVM_DEV_VFIO_GROUP:
return kvm_vfio_set_group(dev, attr->attr,
u64_to_user_ptr(attr->addr));
case KVM_DEV_VFIO_FILE:
return kvm_vfio_set_file(dev, attr->attr,
u64_to_user_ptr(attr->addr));
}
return -ENXIO;
......@@ -320,10 +321,10 @@ static int kvm_vfio_has_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
switch (attr->group) {
case KVM_DEV_VFIO_GROUP:
case KVM_DEV_VFIO_FILE:
switch (attr->attr) {
case KVM_DEV_VFIO_GROUP_ADD:
case KVM_DEV_VFIO_GROUP_DEL:
case KVM_DEV_VFIO_FILE_ADD:
case KVM_DEV_VFIO_FILE_DEL:
#ifdef CONFIG_SPAPR_TCE_IOMMU
case KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE:
#endif
......@@ -339,16 +340,16 @@ static int kvm_vfio_has_attr(struct kvm_device *dev,
static void kvm_vfio_release(struct kvm_device *dev)
{
struct kvm_vfio *kv = dev->private;
struct kvm_vfio_group *kvg, *tmp;
struct kvm_vfio_file *kvf, *tmp;
list_for_each_entry_safe(kvg, tmp, &kv->group_list, node) {
list_for_each_entry_safe(kvf, tmp, &kv->file_list, node) {
#ifdef CONFIG_SPAPR_TCE_IOMMU
kvm_spapr_tce_release_vfio_group(dev->kvm, kvg);
kvm_spapr_tce_release_vfio_group(dev->kvm, kvf);
#endif
kvm_vfio_file_set_kvm(kvg->file, NULL);
fput(kvg->file);
list_del(&kvg->node);
kfree(kvg);
kvm_vfio_file_set_kvm(kvf->file, NULL);
fput(kvf->file);
list_del(&kvf->node);
kfree(kvf);
kvm_arch_end_assignment(dev->kvm);
}
......@@ -382,7 +383,7 @@ static int kvm_vfio_create(struct kvm_device *dev, u32 type)
if (!kv)
return -ENOMEM;
INIT_LIST_HEAD(&kv->group_list);
INIT_LIST_HEAD(&kv->file_list);
mutex_init(&kv->lock);
dev->private = kv;
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7