[PATCH] uml: fail xterm_open when we have no $DISPLAY

If UML wants to open an xterm channel and the xterm does not run properly (eg.
 terminates soon after starting) we will get a hang (a comment added in the
patch explains why).

This avoids the most common cause for this and adds a comment (which long term
will go away with a rewrite of that code); the complete fix would be to catch
the xterm process dying, up(&data->sem), and -EIO all requests from that point
onwards.

That applies for some of the other channels too, so part of the code should
probably be abstracted a little and generalized.
Signed-off-by: Chris Wedgwood <cw@f00f.org>
Signed-off-by: Paolo 'Blaisorblade' Giarrusso <blaisorblade_spam@yahoo.it>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>

arch/um/drivers/xterm.c

@@ -97,6 +97,13 @@ int xterm_open(int input, int output, int primary, void *d, char **dev_out)
 	if(os_access(argv[4], OS_ACC_X_OK) < 0)
 		argv[4] = "port-helper";
 
+	/* Check that DISPLAY is set, this doesn't guarantee the xterm
+	 * will work but w/o it we can be pretty sure it won't. */
+	if (!getenv("DISPLAY")) {
+		printk("xterm_open: $DISPLAY not set.\n");
+		return -ENODEV;
+	}
+
 	fd = mkstemp(file);
 	if(fd < 0){
 		printk("xterm_open : mkstemp failed, errno = %d\n", errno);

arch/um/drivers/xterm_kern.c

@@ -46,6 +46,8 @@ int xterm_fd(int socket, int *pid_out)
 		printk(KERN_ERR "xterm_fd : failed to allocate xterm_wait\n");
 		return(-ENOMEM);
 	}
+
+	/* This is a locked semaphore... */
 	*data = ((struct xterm_wait) 
 		{ .sem  	= __SEMAPHORE_INITIALIZER(data->sem, 0),
 		  .fd 		= socket,
@@ -55,12 +57,17 @@ int xterm_fd(int socket, int *pid_out)
 	err = um_request_irq(XTERM_IRQ, socket, IRQ_READ, xterm_interrupt, 
 			     SA_INTERRUPT | SA_SHIRQ | SA_SAMPLE_RANDOM, 
 			     "xterm", data);
-	if(err){
+	if (err){
 		printk(KERN_ERR "xterm_fd : failed to get IRQ for xterm, "
 		       "err = %d\n",  err);
 		ret = err;
 		goto out;
 	}
+
+	/* ... so here we wait for an xterm interrupt.
+	 *
+	 * XXX Note, if the xterm doesn't work for some reason (eg. DISPLAY
+	 * isn't set) this will hang... */
 	down(&data->sem);
 
 	free_irq_by_irq_and_dev(XTERM_IRQ, data);