We don't support address-only transactions there.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

The code is about to be split up, and this matches dispnv04.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

This function is useful outside of DRM code.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Hasn't been required for a long time.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Fences attached to deferred client work items now originate from channels
belonging to the client, meaning we can be certain they've been signalled
before we destroy a client.

This closes a race that could happen if the dma_fence_wait_timeout() call
didn't succeed.  When the fence was later signalled, a use-after-free was
possible.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

As VMAs are per-client, unlike buffers, this allows us to avoid referencing
foreign fences (those that belong to another client/driver) from the client
deferred work handler, and prevent some not-fun race conditions that can be
triggered when a fence stalls.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

An upcoming patch will use these to fix issues related to the deferred
unmapping of GEM objects.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

We previously only did this for push buffers, but an upcoming patch will
need to attach fences to all VMAs to resolve another issue.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

These were missed the first time around due to the driver version I traced
using the older registers still.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

There are differences on GM200 and newer too, but we can't fix them there
as they come from firmware packages.

A request has been made to NVIDIA to release updated firmware.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Makes it easier to diff against RM traces.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

There's a number of places that require this data, so let's separate out
the calculations to ensure they remain consistent.

This is incorrect for GM200 and newer, but will produce the same results
as we did before.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

There's also a couple of hardcoded tables for a couple of very specific
configurations that NVGPU's algorithm didn't work for.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Required to support Volta.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

RM and NVGPU both have a variant of this, we probably should too.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

We weren't placing higher TPC IDs in the right place on some configurations.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Removed from GK110[B]/GK208 as RM traces show it not being touched.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

The algorithm for GM200 and newer matches RM for all the boards I have, but
I don't have enough data to try and figure something out for earlier boards,
so these will still write zeroes to the table as we did before.

The code in NVGPU isn't helpful here, it appears to handle specific cases.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

I don't think this is done after Fermi, NVGPU used to do it but removed
the code, and I've not seen RM traces touching it either.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

I haven't yet been able to find a fully programatic way of calculating the
same mapping as NVIDIA for GF100-GF119, so the algorithm partially depends
on data tables for specific configurations.

I couldn't find traces for every possibility, so the algorithm will switch
to a mapping similar to what GK104-GM10x use if it encounters one.  We did
the wrong thing before anyway, so shouldn't matter too much.

The algorithm used in the GK104 implementation was ported from NVGPU.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Also fixes some GPUs where we write too many registers.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

GM20B now also shares the same code, as NVGPU shows it doesn't need
special treatment.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>