Merge ZEO2-branch to trunk. (Files added on branch.)

trunk/src/ZEO/zrpc/NOTES

+The Connection object should be extended to support more flexible
+handling for outstanding calls.  In particular, it should be possible
+to have multiple calls with return values outstanding.
+
+The mechanism described here is based on the promises mechanism in
+Argus, which was influenced by futures in Multilisp.
+
+    Promises: Linguistic Support for Efficient Asynchronous Procedure
+    Calls in Distributed Systems.  Barbara Liskov and Liuba Shrira.
+    Proc. of Conf. on Programming Language Design and Implementation
+    (PLDI), June 1988.
+
+We want to support two different kinds of calls:
+
+  - send : invoke a method that returns no value
+  - call : invoke a method that returns a value
+
+On the client, a call immediately returns a promise.  A promise is an
+object that can be used to claim the return value when it becomes
+available. 
+
+  - ready(): returns true if the return value is ready or an exception
+             occurred
+  - claim(): returns the call's return value or raises an exception,
+             blocking if necessary
+
+The server side of a zrpc connection can be implemented using
+asyncore.  In that case, a method call blocks other RPC activity until
+it returns.  If a call needs to return a value, but can't return
+immediately, it returns a delay object (ZEO.zrpc.server.Delay).  
+
+When the zrpc connection receives a Delay object, it does not
+immediately return to the caller.  Instead, it returns when the
+reply() method is called.  A Delay has two methods:
+
+  - set_sender()
+  - reply(obj): returns obj to the sender
+
+-----------------------------------------
+
+Open issues:
+
+Delayed exception
+
+There is currently no mechanism to raise an exception from a delayed
+pcall. 
+
+Synchronization
+
+The following item is part of Argus, but the motivation isn't entirely
+clear.
+
+    For any two calls, C1 and C2, C1 always starts on the server
+    first.  For the promises, C2 is ready() iff C1 is also ready().
+    The promises can be claimed in any order.
+
+A related notion:
+
+    The connection should also support a synch() method that returns
+    only when all outstanding calls have completed.  If any of these
+    calls raised an exception, the synch() call raises an exception.
+
+XXX synch() sounds potentially useful, but it's not clear if it would
+be useful for ZEO.  In ZEO a single connection object handles multiple
+threads, each thread is going to make independent calls.  When a
+particular tpc_begin() returns and a thread commits its transaction,
+it makes more calls.  These calls will before any of the other
+tpc_begin() calls.
+
+I think the Argus approach would be to use separate handlers for each
+thread (not sure Argus had threads), so that a single thread could
+rely on ordering guarantees.
+
+Multithreaded server
+
+There are lots of issues to work out here.
+
+Delays may not be necessary if the connecftion handler runs in a
+different thread than the object the handles the calls.  
\ No newline at end of file

trunk/src/ZEO/zrpc/__init__.py

+##############################################################################
+#
+# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
+# All Rights Reserved.
+# 
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.0 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE
+# 
+##############################################################################
+# zrpc is a package with the following modules
+# error -- exceptions raised by zrpc
+# marshal -- internal, handles basic protocol issues
+# connection -- object dispatcher
+# client -- manages connection creation to remote server
+# server -- manages incoming connections from remote clients
+# trigger -- medusa's trigger

trunk/src/ZEO/zrpc/error.py

+##############################################################################
+#
+# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
+# All Rights Reserved.
+# 
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.0 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE
+# 
+##############################################################################
+from ZODB import POSException
+from ZEO.Exceptions import Disconnected
+
+class ZRPCError(POSException.StorageError):
+    pass
+
+class DecodingError(ZRPCError):
+    """A ZRPC message could not be decoded."""
+
+class DisconnectedError(ZRPCError, Disconnected):
+    """The database storage is disconnected from the storage server."""

trunk/src/ZEO/zrpc/log.py

+##############################################################################
+#
+# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
+# All Rights Reserved.
+# 
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.0 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE
+# 
+##############################################################################
+import os
+import types
+import zLOG
+
+_label = "zrpc:%s" % os.getpid()
+
+def new_label():
+    global _label
+    _label = "zrpc:%s" % os.getpid()
+
+def log(message, level=zLOG.BLATHER, label=None, error=None):
+    zLOG.LOG(label or _label, level, message, error=error)
+
+REPR_LIMIT = 40
+
+def short_repr(obj):
+    "Return an object repr limited to REPR_LIMIT bytes."
+    # Some of the objects being repr'd are large strings.  It's wastes
+    # a lot of memory to repr them and then truncate, so special case
+    # them in this function.
+    # Also handle short repr of a tuple containing a long string.
+    if isinstance(obj, types.StringType):
+        obj = obj[:REPR_LIMIT]
+    elif isinstance(obj, types.TupleType):
+        elts = []
+        size = 0
+        for elt in obj:
+            r = repr(elt)
+            elts.append(r)
+            size += len(r)
+            if size > REPR_LIMIT:
+                break
+        obj = tuple(elts)
+    return repr(obj)[:REPR_LIMIT]

trunk/src/ZEO/zrpc/marshal.py

+##############################################################################
+#
+# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
+# All Rights Reserved.
+# 
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.0 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE
+# 
+##############################################################################
+import cPickle
+from cStringIO import StringIO
+import struct
+import types
+
+from ZEO.zrpc.error import ZRPCError
+
+class Marshaller:
+    """Marshal requests and replies to second across network"""
+
+    # It's okay to share a single Pickler as long as it's in fast
+    # mode, which means that it doesn't have a memo.
+
+    pickler = cPickle.Pickler()
+    pickler.fast = 1
+    pickle = pickler.dump
+
+    errors = (cPickle.UnpickleableError,
+              cPickle.UnpicklingError,
+              cPickle.PickleError,
+              cPickle.PicklingError)
+
+    VERSION = 1
+
+    def encode(self, msgid, flags, name, args):
+        """Returns an encoded message"""
+        return self.pickle((msgid, flags, name, args), 1)
+
+    def decode(self, msg):
+        """Decodes msg and returns its parts"""
+        unpickler = cPickle.Unpickler(StringIO(msg))
+        unpickler.find_global = find_global
+
+        try:
+            return unpickler.load() # msgid, flags, name, args
+        except (self.errors, IndexError), err_msg:
+            log("can't decode %s" % repr(msg), level=zLOG.ERROR)
+            raise DecodingError(msg)
+        
+_globals = globals()
+_silly = ('__doc__',)
+
+def find_global(module, name):
+    """Helper for message unpickler"""
+    try:
+        m = __import__(module, _globals, _globals, _silly)
+    except ImportError, msg:
+        raise ZRPCError("import error %s: %s" % (module, msg))
+
+    try:
+        r = getattr(m, name)
+    except AttributeError:
+        raise ZRPCError("module %s has no global %s" % (module, name))
+
+    safe = getattr(r, '__no_side_effects__', 0)
+    if safe:
+        return r
+
+    # XXX what's a better way to do this?  esp w/ 2.1 & 2.2
+    if type(r) == types.ClassType and issubclass(r, Exception):
+        return r
+
+    raise ZRPCError("Unsafe global: %s.%s" % (module, name))

trunk/src/ZEO/zrpc/server.py

+##############################################################################
+#
+# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
+# All Rights Reserved.
+# 
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.0 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE
+# 
+##############################################################################
+import asyncore
+import socket
+import types
+
+from ZEO.zrpc.connection import Connection, Delay
+from ZEO.zrpc.log import log
+
+# Export the main asyncore loop
+loop = asyncore.loop
+
+class Dispatcher(asyncore.dispatcher):
+    """A server that accepts incoming RPC connections"""
+    __super_init = asyncore.dispatcher.__init__
+
+    reuse_addr = 1
+
+    def __init__(self, addr, factory=Connection, reuse_addr=None):
+        self.__super_init()
+        self.addr = addr
+        self.factory = factory
+        self.clients = []
+        if reuse_addr is not None:
+            self.reuse_addr = reuse_addr
+        self._open_socket()
+
+    def _open_socket(self):
+        if type(self.addr) == types.TupleType:
+            self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
+        else:
+            self.create_socket(socket.AF_UNIX, socket.SOCK_STREAM)
+        self.set_reuse_addr()
+        log("listening on %s" % str(self.addr))
+        self.bind(self.addr)
+        self.listen(5)
+
+    def writable(self):
+        return 0
+
+    def readable(self):
+        return 1
+
+    def handle_accept(self):
+        try:
+            sock, addr = self.accept()
+        except socket.error, msg:
+            log("accepted failed: %s" % msg)
+            return
+        c = self.factory(sock, addr)
+        log("connect from %s: %s" % (repr(addr), c))
+        self.clients.append(c)

trunk/src/ZEO/zrpc/trigger.py

+##############################################################################
+#
+# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
+# All Rights Reserved.
+# 
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.0 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE
+# 
+##############################################################################
+# This module is a simplified version of the select_trigger module
+# from Sam Rushing's Medusa server.
+
+import asyncore
+
+import os
+import socket
+import thread
+
+if os.name == 'posix':
+
+    class trigger (asyncore.file_dispatcher):
+
+        "Wake up a call to select() running in the main thread"
+
+        # This is useful in a context where you are using Medusa's I/O
+        # subsystem to deliver data, but the data is generated by another
+        # thread.  Normally, if Medusa is in the middle of a call to
+        # select(), new output data generated by another thread will have
+        # to sit until the call to select() either times out or returns.
+        # If the trigger is 'pulled' by another thread, it should immediately
+        # generate a READ event on the trigger object, which will force the
+        # select() invocation to return.
+
+        # A common use for this facility: letting Medusa manage I/O for a
+        # large number of connections; but routing each request through a
+        # thread chosen from a fixed-size thread pool.  When a thread is
+        # acquired, a transaction is performed, but output data is
+        # accumulated into buffers that will be emptied more efficiently
+        # by Medusa. [picture a server that can process database queries
+        # rapidly, but doesn't want to tie up threads waiting to send data
+        # to low-bandwidth connections]
+
+        # The other major feature provided by this class is the ability to
+        # move work back into the main thread: if you call pull_trigger()
+        # with a thunk argument, when select() wakes up and receives the
+        # event it will call your thunk from within that thread.  The main
+        # purpose of this is to remove the need to wrap thread locks around
+        # Medusa's data structures, which normally do not need them.  [To see
+        # why this is true, imagine this scenario: A thread tries to push some
+        # new data onto a channel's outgoing data queue at the same time that
+        # the main thread is trying to remove some]
+
+        def __init__ (self):
+            r, w = os.pipe()
+            self.trigger = w
+            asyncore.file_dispatcher.__init__ (self, r)
+            self.lock = thread.allocate_lock()
+            self.thunks = []
+
+        def close(self):
+            self.del_channel()
+            self.socket.close() # the read side of the pipe
+            os.close(self.trigger) # the write side of the pipe
+
+        def __repr__ (self):
+            return '<select-trigger (pipe) at %x>' % id(self)
+
+        def readable (self):
+            return 1
+
+        def writable (self):
+            return 0
+
+        def handle_connect (self):
+            pass
+
+        def pull_trigger (self, thunk=None):
+            # print 'PULL_TRIGGER: ', len(self.thunks)
+            if thunk:
+                try:
+                    self.lock.acquire()
+                    self.thunks.append (thunk)
+                finally:
+                    self.lock.release()
+            os.write (self.trigger, 'x')
+
+        def handle_read (self):
+            self.recv (8192)
+            try:
+                self.lock.acquire()
+                for thunk in self.thunks:
+                    try:
+                        thunk()
+                    except:
+                        (file, fun, line), t, v, tbinfo = asyncore.compact_traceback()
+                        print 'exception in trigger thunk: (%s:%s %s)' % (t, v, tbinfo)
+                self.thunks = []
+            finally:
+                self.lock.release()
+
+else:
+
+    # win32-safe version
+
+    class trigger (asyncore.dispatcher):
+
+        address = ('127.9.9.9', 19999)
+
+        def __init__ (self):
+            a = socket.socket (socket.AF_INET, socket.SOCK_STREAM)
+            w = socket.socket (socket.AF_INET, socket.SOCK_STREAM)
+
+            # set TCP_NODELAY to true to avoid buffering
+            w.setsockopt(socket.IPPROTO_TCP, 1, 1)
+
+            # tricky: get a pair of connected sockets
+            host='127.0.0.1'
+            port=19999
+            while 1:
+                try:
+                    self.address=(host, port)
+                    a.bind(self.address)
+                    break
+                except:
+                    if port <= 19950:
+                        raise 'Bind Error', 'Cannot bind trigger!'
+                    port=port - 1
+
+            a.listen (1)
+            w.setblocking (0)
+            try:
+                w.connect (self.address)
+            except:
+                pass
+            r, addr = a.accept()
+            a.close()
+            w.setblocking (1)
+            self.trigger = w
+
+            asyncore.dispatcher.__init__ (self, r)
+            self.lock = thread.allocate_lock()
+            self.thunks = []
+            self._trigger_connected = 0
+
+        def __repr__ (self):
+            return '<select-trigger (loopback) at %x>' % id(self)
+
+        def readable (self):
+            return 1
+
+        def writable (self):
+            return 0
+
+        def handle_connect (self):
+            pass
+
+        def pull_trigger (self, thunk=None):
+            if thunk:
+                try:
+                    self.lock.acquire()
+                    self.thunks.append (thunk)
+                finally:
+                    self.lock.release()
+            self.trigger.send ('x')
+
+        def handle_read (self):
+            self.recv (8192)
+            try:
+                self.lock.acquire()
+                for thunk in self.thunks:
+                    try:
+                        thunk()
+                    except:
+                        (file, fun, line), t, v, tbinfo = asyncore.compact_traceback()
+                        print 'exception in trigger thunk: (%s:%s %s)' % (t, v, tbinfo)
+                self.thunks = []
+            finally:
+                self.lock.release()