Finally, this add the first huge gtt support for GVTg - 64K pages. Since
64K page and 4K page cannot be mixed on the same page table, so we always
split a 64K entry into small 4K page. And when unshadow guest 64K entry,
we need ensure all the shadowed entries in shadow page table also get
cleared.

For page table which has 64K gtt entry, only PTE#0, PTE#16, PTE#32, ...
PTE#496 are used. Unused PTEs update should be ignored.
Signed-off-by: Changbin Du <changbin.du@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

64K PTE is special, only PTE#0, PTE#16, PTE#32, ... PTE#496 are used in
the page table.
Signed-off-by: Changbin Du <changbin.du@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

We need a interface to allocate a pure shadow page which doesn't have
a guest page associated with. Such shadow page is used to shadow 2M
huge gtt entry.
Signed-off-by: Changbin Du <changbin.du@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

Add clear_pse operation in case we need to split huge gtt into small pages.

v2: correct description.
Signed-off-by: Changbin Du <changbin.du@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

This add a software PTE flag on the Ignored bit of PTE. It will be used
to identify splited 64K shadow entries.

v2: fix mask definition.
Signed-off-by: Changbin Du <changbin.du@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

This change help us detect the real entry type per PSE and IPS setting.
For 64K entry, we also need to check reg GEN8_GAMW_ECO_DEV_RW_IA.

v2: Extend IPS mmio control to Gen10. (Matthew Auld)
Signed-off-by: Changbin Du <changbin.du@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

The register RENDER_HWS_PGA_GEN7 is renamed to GEN8_GAMW_ECO_DEV_RW_IA
from GEN8 which can control IPS enabling.

v3: MMIO control for IPS is not removed from gen9 but gen10 (Matthew Auld)
v2: IPS of all engines must be enabled together for gen9.
Signed-off-by: Changbin Du <changbin.du@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

Add three IPS operation functions to test/set/clear IPS in PDE.
Signed-off-by: Changbin Du <changbin.du@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

Add a new entry type GTT_TYPE_PPGTT_PTE_64K_ENTRY. 64K entry is very
different from 2M/1G entry. 64K entry is controlled by IPS bit in upper
PDE. To leverage the current logic, I take IPS bit as 'PSE' for PTE
level. Which means, 64K entries can also processed by get_pse_type().

v2: Make it bisectable.
Signed-off-by: Changbin Du <changbin.du@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

Enable KVMGT for BXT.
is_supported_device() acting as the gatekeeper of GVT-g init.
If all supported platforms share the same configurations for some
specific feature, platform check will rely on this check only.
Signed-off-by: Colin Xu <colin.xu@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

Leverage most SKL/KBL mmio init info and add different mmio to
BXT specific function init_bxt_mmio_info().
Signed-off-by: Colin Xu <colin.xu@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

Handle dma_buf on BXT as SKL and KBL.
Signed-off-by: Colin Xu <colin.xu@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

Virtual monitor on BXT start from port B.
Unlike SKL/KBL, digital display port connectivity is detected via
GEN8_DE_PORT_ISR so emulate monitor state change by setting it.
Signed-off-by: Colin Xu <colin.xu@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

BXT forcewake is handled in the same way as SKL/KBL.

v2: Add missing inhibit_context restore for BXT.
Signed-off-by: Colin Xu <colin.xu@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

Handle BXT cmd_parser as SKL/KBL.

v2: All supported platforms share the same routines.
    Remove the platform check by now and let is_supported_device()
    be the gate keeper.
Signed-off-by: Colin Xu <colin.xu@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

Handle pending tlb flush, mocs/mmio switch and context as KBL.
Signed-off-by: Colin Xu <colin.xu@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

Initialize BXT irq handler as SKL/KBL.

v2: All supported platforms share the same irq ops and map.
    Remove the platform check by now and let is_supported_device()
    be the gate keeper.
Signed-off-by: Colin Xu <colin.xu@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

Initialize BXT gtt as SKL/KBL.

v2: All supported platforms share the same gtt ops.
    Remove the platform check by now and let is_supported_device()
    be the gate keeper.
Signed-off-by: Colin Xu <colin.xu@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

Initialize BXT device info as SKL/KBL.

v2: All supported platforms share the same device configuration.
    Remove the platform check by now and let is_supported_device()
    be the gate keeper.
Signed-off-by: Colin Xu <colin.xu@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

As referred in PRM for Broxton Graphics on 01.org
Signed-off-by: Colin Xu <colin.xu@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

Broxton belongs to GEN9 family so add to SKL and GEN9 plus.
Signed-off-by: Colin Xu <colin.xu@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

Array 'pdp_pair' of size 1 may use index value(s) 1..7.
Changed to pdps[8] to avoid confusion.
Signed-off-by: Xinyun Liu <xinyun.liu@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

type is already checked in the function entry. So it is unnecessary
to check it again.
Signed-off-by: Xinyun Liu <xinyun.liu@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

Add sanity check for up_irq_info.
Signed-off-by: Xinyun Liu <xinyun.liu@intel.com>
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

Backmerge for recent request->hw_context change and
new vGPU huge page capability definition.
Signed-off-by: Zhenyu Wang <zhenyuw@linux.intel.com>

Signed-off-by: Rodrigo Vivi <rodrigo.vivi@intel.com>

The macro declared the ppgtt parameter but implicitly used the local vm
instead.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Cc: Matthew Auld <matthew.william.auld@gmail.com>
Reviewed-by: Matthew Auld <matthew.william.auld@gmail.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180606205128.25952-1-chris@chris-wilson.co.uk

When we reach the magic value and do inject a fault into our module load,
mark the module option as being hit. Since we fail from inside pci
probe, the module load isn't actually aborted and the module (and
parameters) are left lingering. igt can then inspect the parameter on its
synchronous completion of modprobe to see if the fault injection was
successful, and will keeping on injecting new faults until the module
succeeds in loading having surpassed the number of fault points.

v2: Reset to 0 after being hit;
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michał Winiarski <michal.winiarski@intel.com>
Cc: Imre Deak <imre.deak@intel.com>
Reviewed-by: Michał Winiarski <michal.winiarski@intel.com>
Reviewed-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180606144153.4244-1-chris@chris-wilson.co.uk

In the near future, I want to subclass gen6_hw_ppgtt as it contains a
few specialised members and I wish to add more. To avoid the ugliness of
using ppgtt->base.base, rename the i915_hw_ppgtt base member
(i915_address_space) as vm, which is our common shorthand for an
i915_address_space local.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Cc: Matthew Auld <matthew.william.auld@gmail.com>
Reviewed-by: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180605153758.18422-1-chris@chris-wilson.co.uk

The inactive counter was over the active list, and vice versa.
Fortuitously this should not cause a problem in practice as they shared
the same array and clamped the number of entries they would write.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Cc: Matthew Auld <matthew.william.auld@gmail.com>
Reviewed-by: Matthew Auld <matthew.william.auld@gmail.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180605160623.30163-1-chris@chris-wilson.co.uk

In case of failure during GuC clients creation, we forget to
cleanup earlier pool allocation. Use proper teardown to fix that.
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Michal Winiarski <michal.winiarski@intel.com>
Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/msgid/20180605120547.16468-1-michal.wajdeczko@intel.com

As Chris has discovered on his Ivybridge, and later automated test runs
have confirmed, on most of our platforms hrtimer faced with heavy GPU load
can occasionally become sufficiently imprecise to affect PMU sampling
calculations.

This means we cannot assume sampling frequency is what we asked for, but
we need to measure the interval ourselves.

This patch is similar to Chris' original proposal for per-engine counters,
but instead of introducing a new set to work around the problem with
frequency sampling, it swaps around the way internal frequency accounting
is done. Instead of accumulating current frequency and dividing by
sampling frequency on readout, it accumulates frequency scaled by each
period.

v2:
 * Typo in commit message, comment on period calculation and USEC_PER_SEC.
   (Chris Wilson)

Testcase: igt/perf_pmu/*busy* # snb, ivb, hsw
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Suggested-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/msgid/20180605140253.3541-1-tvrtko.ursulin@linux.intel.com

Underlaying field is u64 so the tracepoint needs to be as well.

v2:
 * Re-order binary packet for 64-bit alignment. (Chris Wilson)
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Suggested-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/msgid/20180605134124.25672-1-tvrtko.ursulin@linux.intel.com

In the string tracepoint representation we ended up with the engine
sandwiched between context hardware id and context fence id.

Move the two pieces of context data together for redability.

Binary records are left as is, that is both fields remaing under the
existing name and ordering.

v2:
 * Do not consolidate the printk format, just reorder. (Lionel)
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180525082642.18246-2-tvrtko.ursulin@linux.intel.com

Instead of using the engine->id, use uabi_class:instance pairs in trace-
points including engine info.

This will be more readable, more future proof and more stable for
userspace consumption.

v2:
 * Use u16 for class and instance. (Chris Wilson)
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Cc: svetlana.kukanova@intel.com
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180525082642.18246-1-tvrtko.ursulin@linux.intel.com

Since the kernel provides SZ_1M, use it in preference of 1 << 20.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180605135746.8020-1-chris@chris-wilson.co.uk

In function gem_init_hw() we are calling uc_init_hw() but in case
of error later in function, we missed to call matching uc_fini_hw()

v2: pulled out from the series
Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com>
Cc: Sagar Arun Kamble <sagar.a.kamble@intel.com>
Cc: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Sagar Arun Kamble <sagar.a.kamble@intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/msgid/20180605122443.23776-1-michal.wajdeczko@intel.com

To spare ourselves a long line later, refactor the repeated check of
bind_count vs pin_count to a helper.

v2: Fix up the commentary!
Suggested-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Cc: Matthew Auld <matthew.william.auld@gmail.com>
Reviewed-by: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180605094107.31367-1-chris@chris-wilson.co.uk

In preparation, for having non-vma objects stored inside the ggtt, to
handle restoration of the GGTT following resume, we need to walk over
the ggtt address space rebinding vma, as opposed to walking over bound
objects looking for ggtt entries.

v2: Skip objects only bound for the aliasing_ppgtt
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Cc: Matthew Auld <matthew.william.auld@gmail.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> #v1
Link: https://patchwork.freedesktop.org/patch/msgid/20180605082856.19221-1-chris@chris-wilson.co.uk

Currently, we have a special routine for pinning the context state at
the start of activity tracking, but lack the complementary unpin
routine. Create it to to ease later patches that want to do partial
teardown on error, and, not least, to improve the readability of the
code.
Suggested-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Mika Kuoppala <mika.kuoppala@linux.intel.com>
Cc: Matthew Auld <matthew.william.auld@gmail.com>
Reviewed-by: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20180605085348.3018-1-chris@chris-wilson.co.uk