Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

This add an ioctl to migrate a range of process address space to the
device memory. On platform without cache coherent bus (x86, ARM, ...)
this means that CPU can not access that range directly, instead CPU
will fault which will migrate the memory back to system memory.

This is behind a staging flag so that we can evolve the API.
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>

Device memory can be use in SVM, in which case we do not have any of
the existing buffer object. This commit add infrastructure to allow
use of device memory without nouveau_bo. Again this is a temporary
solution until a rework of GPU memory management.
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>

This uses HMM to mirror a process' CPU page tables into a channel's page
tables, and keep them synchronised so that both the CPU and GPU are able
to access the same memory at the same virtual address.

While this code also supports Volta/Turing, it's only enabled for Pascal
GPUs currently due to channel recovery being unreliable right now on the
later GPUs.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

For a channel to make use of SVM features, it requires a different GPU MMU
configuration than we would normally use, which is not desirable to switch
to unless a client is actively going to use SVM.

In order to supporting SVM without more extensive changes to the userspace
interfaces, the SVM_INIT ioctl needs to replace the previous configuration
safely.

The only way we can currently do this safely, accounting for some unlikely
failure conditions, is to allocate the new VMM without destroying the last
one, and prioritising the SVM-enabled configuration in the code that cares.

This will get cleaned up again further down the track.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

This nvclass exposes the replayable fault buffer, which will be used
by SVM to manage GPU page faults.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

This nvclass exposes the replayable fault buffer, which will be used
by SVM to manage GPU page faults.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Some GPU units are capable of supporting "replayable" page faults, where
the execution unit will wait for SW to fixup GPU page tables rather than
triggering a channel-fatal fault.

This feature isn't useful (it's harmful, even) unless something like HMM
is being used to manage events appearing in the replayable fault buffer,
so, it's disabled by default.

This commit allows a client to request it be enabled.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Host methods exist to do at least some of what we need, but we are not
currently pushing replay/cancels through a channel like UVM does as it's
not clear whether it's necessary in our case (UVM also updates PTEs with
the GPU).

UVM also pushes a software method for fault cancels on Pascal, seemingly
because the host methods don't appear to be sufficient.  If/when we want
to push the replay/cancel on the GPU, we can re-purpose the cancellation
code here to implement that swmthd.

Keep it simple for now, until we figure out exactly what we need here.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

This provides a somewhat more direct method of manipulating the GPU page
tables, which will be required to support SVM.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

This will be used to support a privileged client providing PTEs directly,
without a memory object to use as a reference.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

NVKM is currently responsible for managing the allocation of a client's
GPU address-space, but there's various use-cases (ie. HMM address-space
mirroring) where giving a client more direct control is desirable.

This commit allows for a VMM to be created where the area allocated for
NVKM is limited to a client-specified window, the remainder of address-
space is controlled directly by the client.

Leaving a window is necessary to support various internal requirements,
but also to support existing allocation interfaces as not all of the HW
is capable of working with a HMM allocation.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

MMU will need access to this info.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

MMU will need access to these.

v2. Apply fix from Rhys Kidd to send correct FECS method for STOP_CTXSW.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

There are a few statements that are indented incorrectly. Fix these.
Signed-off-by: Colin Ian King <colin.king@canonical.com>
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

It appears that Pascal and newer need something different.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Will want to reuse this for fault replay/cancellation swmthds.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Future changes will want to add some additional things here, keep them
grouped together.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Makes the code somewhat less magic.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

This is already done during golden context creation.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Makes the code somewhat less magic.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Makes the code somewhat less magic.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Makes the code somewhat less magic.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Makes the code somewhat less magic.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Makes the code somewhat less magic.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

There's no need to avoid using copy engines if gr init fails for some
reason (usually missing FW, or incomplete bring-up).

It's not terribly useful for an end-user, but it'll slightly speed up
suspend/resume when saving fb contents, and allow for host/ce code to
be validated.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

It has no relevance to the atomic path used by newer GPUs.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

It has no relevance to the atomic path used by newer GPUs.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Swapped order of functions in dispnv04 to allow this, but no code changes.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Some of the pre-NV50 depends on SW methods to implement synchronisation
for page flips, and we want to move this setup out of common code, thus
we require the channel to have been allocation before display init.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

We also change the error strings to match NVIDIA's naming.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

As I currently understand it, this is related to features we have no
support for as of yet.

In theory, this change should be a noop, just without the warning.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Required for ACR.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Turing has its SEC2 instance in an alternate location, and this avoids
needing to duplicate the code here for it.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Required to run VPR scrubber binary as part of secboot.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Turing has its NVDEC instances in an alternate location.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

We need this for Turing ACR, but it's present from Volta onwards.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>