Commit 0f580386 authored by Thomas Hellstrom's avatar Thomas Hellstrom Committed by Sinclair Yeh

drm/vmwgfx: Fix a potential integer overflow

Found by coverity.
Signed-off-by: default avatarThomas Hellstrom <thellstrom@vmware.com>
Reviewed-by: default avatarSinclair Yeh <syeh@vmware.com>
Reviewed-by: default avatarCharmaine Lee <charmainel@vmware.com>
parent 0fd67c1c
...@@ -1671,7 +1671,7 @@ int vmw_kms_update_layout_ioctl(struct drm_device *dev, void *data, ...@@ -1671,7 +1671,7 @@ int vmw_kms_update_layout_ioctl(struct drm_device *dev, void *data,
* 1. Bounding box (assuming 32bpp) must be < prim_bb_mem * 1. Bounding box (assuming 32bpp) must be < prim_bb_mem
* 2. Total pixels (assuming 32bpp) must be < prim_bb_mem * 2. Total pixels (assuming 32bpp) must be < prim_bb_mem
*/ */
u64 bb_mem = bounding_box.w * bounding_box.h * 4; u64 bb_mem = (u64) bounding_box.w * bounding_box.h * 4;
u64 pixel_mem = total_pixels * 4; u64 pixel_mem = total_pixels * 4;
if (bb_mem > dev_priv->prim_bb_mem) { if (bb_mem > dev_priv->prim_bb_mem) {
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment