Skip to content

S
shrapnel
Commit e921e61e authored by Sam Rushing's avatar Sam Rushing
Browse files
Options

Merge branch 'master' into zstack 

Conflicts:
	coro/_coro.pyx
parents 7ba1e191 5187b9ce
Hide whitespace changes
Inline Side-by-side
Showing with 6045 additions and 12203 deletions
.gitignore
View file @ e921e61e
......@@ -5,5 +5,7 @@ build/
coro/_coro.[ch]
coro/oserrors.[ch]
coro/clocks/tsc_time.c
coro/dns/packet.c
coro/event_queue.cpp
*.pyc
*.so
README.rst
View file @ e921e61e
......@@ -8,6 +8,12 @@ around FreeBSD's kqueue() system call. It's designed for
single-process servers that can handle 10,000+ simultaneous network
connections.
API Documentation
=================
See http://ironport.github.com/shrapnel/
Short History Of Python Coroutine Implementations
=================================================
......
coro/__init__.py
View file @ e921e61e
......@@ -20,113 +20,7 @@
# $Header: //prod/main/ap/shrapnel/coro/__init__.py#31 $
"""Coroutine threading library.
Introduction
============
Shrapnel is a cooperative threading library.
Getting Started
===============
When your process starts up, you must spawn a thread to do some work, and then
start the event loop. The event loop runs forever processing events until the
process exits. An example::
import coro
def main():
print 'Hello world!'
# This will cause the process to exit.
coro.set_exit(0)
coro.spawn(main)
coro.event_loop()
Coroutines
==========
Every coroutine thread is created with either the `new` function (which does
NOT automatically start the thread) or the `spawn` function (which DOES
automatically start it).
Every thread has a unique numeric ID. You may also set the name of the thread
when you create it.
Timeouts
========
The shrapnel timeout facility allows you to execute a function which will be
interrupted if it does finish within a specified period of time. The
`coro.TimeoutError` exception will be raised if the timeout expires. See the
`with_timeout` docstring for more detail.
If the event loop is not running (such as in a non-coro process), a custom
version of `with_timeout` is installed that will operate using SIGALRM so that
you may use `with_timeout` in code that needs to run in non-coro processes
(though this is not recommended and should be avoided if possible).
Thread Local Storage
====================
There is a tread-local storage interface available for storing global data this
is thread-specific. You instantiate a `ThreadLocal` instance and you can
assign attributes to it that will be specific to that thread. See the
`ThreadLocal` docs for more detail.
Signal Handlers
===============
By default when you start the event loop, two signal handlers are installed
(for SIGTERM and SIGINT). The default signal handler will exit the event loop.
You can change this behavior by setting `install_signal_handlers` to False
before starting the event loop.
See `coro.signal_handler` for more detail on setting coro signal handlers.
Selfishness
===========
Certain socket operations are allowed to try to execute without blocking if
they are able to (such as send/receiving data on a local socket or on a
high-speed network). However, there is a limit to the number of times a thread
is allowed to do this. The default is 4. The default may be changed
(`set_selfishness`) and the value on a per-thread may be changed
(`coro.coro.set_max_selfish_acts`).
Time
====
Shrapnel uses the `tsc_time` module for handling time. It uses the TSC
value for a stable and high-resolution unit of time. See that module's
documentation for more detail.
A thread is always created when you start the event loop that will
resynchronize the TSC relationship to accomodate any clock drift (see
`tick_updater` and `tsc_time.update_time_relation`).
Exception Notifier
==================
When a thread exits due to an exception, by default a stack trace is printed to
stderr. You may install your own callback to handle this situation. See the
`set_exception_notifier` function for more detail.
Debug Output
============
The shrapnel library provides a mechanism for printing debug information to
stderr. The `print_stderr` function will print a string with a timestamp
and the thread number. The `write_stderr` function writes the string verbatim.
Shrapnel keeps a reference to the "real" stderr (in `saved_stderr`) and the
`print_stderr` and `write_stderr` functions always use the real stderr value. A
particular reason for doing this is the backdoor module replaces sys.stderr and
sys.stdout, but we do not want debug output to go to the interactive session.
Profiling
=========
Shrapnel has its own profiler that is coro-aware. See `coro.profiler` for
details on how to run the profiler.
:Variables:
- `all_threads`: A dictionary of all live coroutine objects. The key is
the coroutine ID, and the value is the coroutine object.
- `saved_stderr`: The actual stderr object for the process. This normally
should not be used. An example of why this exists is because the
backdoor replaces sys.stderr while executing user code.
"""
"""Coroutine threading library."""
from coro._coro import *
from coro._coro import _yield
......@@ -166,13 +60,12 @@ set_exception_notifier (default_exception_notifier)
class InParallelError (Exception):
"""An error occurred in the `in_parallel` function.
"""An error occurred in the :func:`in_parallel` function.
:IVariables:
- `result_list`: A list of ``(status, result)`` tuples. ``status`` is
either `SUCCESS` or `FAILURE`. For success, the result is the return
:ivar result_list: A list of ``(status, result)`` tuples. ``status`` is
either :data:`SUCCESS` or :data:`FAILURE`. For success, the result is the return
value of the function. For failure, it is the output from
`sys.exc_info`.
``sys.exc_info``.
"""
def __init__(self, result_list):
......@@ -195,17 +88,14 @@ def in_parallel (fun_arg_list):
This will block until all functions have returned or raised an exception.
If one or more functions raises an exception, then the `InParallelError`
If one or more functions raises an exception, then the :exc:`InParallelError`
exception will be raised.
:Parameters:
- `fun_arg_list`: A list of ``(fun, args)`` tuples.
:param fun_arg_list: A list of ``(fun, args)`` tuples.
:Return:
Returns a list of return values from the functions.
:returns: A list of return values from the functions.
:Exceptions:
- `InParallelError`: One or more of the functions raised an exception.
:raises InParallelError: One or more of the functions raised an exception.
"""
# InParallelError, [(SUCCESS, result0), (FAILURE, exc_info1), ...]
......@@ -257,14 +147,11 @@ def tick_updater():
def waitpid (pid):
"""Wait for a process to exit.
:Parameters:
- `pid`: The process ID to wait for.
:param pid: The process ID to wait for.
:Return:
Returns a tuple ``(pid, status)`` of the process.
:returns: A tuple ``(pid, status)`` of the process.
:Exceptions:
- `SimultaneousError`: Something is already waiting for this process
:raises SimultaneousError: Something is already waiting for this process
ID.
"""
if UNAME == "Linux":
......@@ -290,14 +177,11 @@ def waitpid (pid):
def get_thread_by_id (thread_id):
"""Get a coro thread by ID.
:Parameters:
- `thread_id`: The thread ID.
:param thread_id: The thread ID.
:Return:
Returns the coroutine object.
:returns: The coroutine object.
:Exceptions:
- `KeyError`: The coroutine does not exist.
:raises KeyError: The coroutine does not exist.
"""
return all_threads[thread_id]
......@@ -305,11 +189,9 @@ def where (co):
"""Return a string indicating where the given coroutine thread is currently
running.
:Parameters:
- `co`: The coroutine object.
:param co: The coroutine object.
:Return:
Returns a string displaying where the coro thread is currently
:returns: A string displaying where the coro thread is currently
executing.
"""
f = co.get_frame()
......@@ -318,8 +200,7 @@ def where (co):
def where_all():
"""Get a dictionary of where all coroutines are currently executing.
:Return:
Returns a dictionary mapping the coroutine ID to a tuple of ``(name,
:returns: A dictionary mapping the coroutine ID to a tuple of ``(name,
coro, where)`` where ``where`` is a string representing where the
coroutine is currently running.
"""
......@@ -339,13 +220,11 @@ def spawn (fun, *args, **kwargs):
Additional arguments and keyword arguments will be passed to the given function.
:Parameters:
- `fun`: The function to call when the coroutine starts.
- `thread_name`: The name of the thread. Defaults to the name of the
:param fun: The function to call when the coroutine starts.
:param thread_name: The name of the thread. Defaults to the name of the
function.
:Return:
Returns the new coroutine object.
:returns: The new coroutine object.
"""
if kwargs.has_key('thread_name'):
thread_name = kwargs['thread_name']
......@@ -364,13 +243,11 @@ def new (fun, *args, **kwargs):
This will not start the coroutine. Call the ``start`` method on the
coroutine to schedule it to run.
:Parameters:
- `fun`: The function to call when the coroutine starts.
- `thread_name`: The name of the thread. Defaults to the name of the
:param fun: The function to call when the coroutine starts.
:param thread_name: The name of the thread. Defaults to the name of the
function.
:Return:
Returns the new coroutine object.
:returns: The new coroutine object.
"""
if kwargs.has_key('thread_name'):
thread_name = kwargs['thread_name']
......@@ -457,8 +334,7 @@ event_loop_is_running = False
def coro_is_running():
"""Determine if the coro event loop is running.
:Return:
Returns True if the event loop is running, otherwise False.
:returns: True if the event loop is running, otherwise False.
"""
return event_loop_is_running
......@@ -468,8 +344,7 @@ def sigterm_handler (*_unused_args):
def event_loop (timeout=30):
"""Start the event loop.
:Parameters:
- `timeout`: The amount of time to wait for kevent to return
:param timeout: The amount of time to wait for kevent to return
events. You should probably *not* set this value.
"""
global event_loop_is_running, with_timeout, sleep_relative
......
coro/_coro.pyx
View file @ e921e61e
# Copyright (c) 2002-2011 IronPort Systems and Cisco Systems
#
# Copyright (c) 2002-2011 IronPort Systems and Cisco Systems
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
......@@ -19,14 +19,13 @@
# SOFTWARE.
# -*- Mode: Pyrex -*-
#cython: embedsignature=True
"""Pyrex module for coroutine implementation.
Module variables defined below are available only from Pyrex. Python-accessible
variables are documented in the top level of the coro package ``__init__.py``.
:Variables:
- `_ticks_per_sec`: Number of CPU ticks per second (uint64_t).
"""
__coro_version__ = "$Id: //prod/main/ap/shrapnel/coro/_coro.pyx#114 $"
......@@ -83,6 +82,7 @@ cdef extern from "Python.h":
# ================================================================================
# global variables
# ================================================================================
# Number of CPU ticks per second (uint64_t).
cdef uint64_t _ticks_per_sec
cdef object ticks_per_sec
_ticks_per_sec = tsc_time_module.ticks_per_sec
......@@ -156,10 +156,6 @@ cdef enum:
import sys
class Exit (Exception):
"exception used to exit the event loop"
pass
class ScheduleError (Exception):
"attempt to schedule an already-scheduled coroutine"
pass
......@@ -207,6 +203,12 @@ cdef int default_selfishness
default_selfishness = 4
def set_selfishness(n):
"""Set the global default selfishness limit.
This sets the default for every new coroutine.
:param n: The new limit.
"""
global default_selfishness
default_selfishness = n
......@@ -215,11 +217,10 @@ live_coros = 0
cdef public class coro [ object _coro_object, type _coro_type ]:
"""XXX
"""The coroutine object.
:IVariables:
- `top`: A `call_stack` object used by the profiler. NULL if the
profiler is not enabled or if this is the first call of the coroutine.
Do not create this object directly. Use either :func:`new` or
:func:`spawn` to create one.
"""
cdef machine_state state
......@@ -233,7 +234,8 @@ cdef public class coro [ object _coro_object, type _coro_type ]:
cdef size_t stack_size
cdef PyFrameObject * frame
cdef void * saved_exception_data[6]
# used only by the profiler
# used only by the profiler, a call_stack object. NULL if the profiler is
# not enabled or if this is the first call of the coroutine.
cdef call_stack * top
cdef int saved_recursion_depth
cdef int selfish_acts, max_selfish_acts
......@@ -490,45 +492,39 @@ cdef public class coro [ object _coro_object, type _coro_type ]:
cdef _schedule (self, value):
"""Schedule this coroutine to run.
:Parameters:
- `value`: The value to resume the coroutine with. Note that
"interrupting" the coroutine resumes it with a special
`exception` value which is checked when this coro is resumed.
:param value: The value to resume the coroutine with. Note that
"interrupting" the coroutine resumes it with a special
``exception`` value which is checked when this coro is resumed.
:Exceptions:
- `DeadCoroutine`: The coroutine is dead (it has already exited).
- `ScheduleError`: The coroutine is already scheduled to run.
- `ScheduleError`: Attempted to schedule the currently running coro.
:raises DeadCoroutine: The coroutine is dead (it has already exited).
:raises ScheduleError: The coroutine is already scheduled to run.
:raises ScheduleError: Attempted to schedule the currently running coro.
"""
the_scheduler._schedule (self, value)
cdef _unschedule (self):
"""Unschedule this coroutine.
:Return:
Returns True if it was successfully unscheduled, False if not.
:returns: True if it was successfully unscheduled, False if not.
"""
return the_scheduler._unschedule (self)
def schedule (self, value=None):
"""Schedule this coroutine to run.
:Parameters:
- `value`: The value to resume the coroutine with. Defaults to
None.
:param value: The value to resume the coroutine with. Defaults to
None.
:Exceptions:
- `DeadCoroutine`: The coroutine is dead (it has already exited).
- `ScheduleError`: The coroutine is already scheduled to run.
- `ScheduleError`: Attempted to schedule the currently running coro.
:raises DeadCoroutine: The coroutine is dead (it has already exited).
:raises ScheduleError: The coroutine is already scheduled to run.
:raises ScheduleError: Attempted to schedule the currently running coro.
"""
return self._schedule (value)
def start (self):
"""Start the coroutine for the first time.
:Exceptions:
- `ScheduleError`: The coro is already started.
:raises ScheduleError: The coro is already started.
"""
if self.started:
raise ScheduleError(self)
......@@ -551,20 +547,18 @@ cdef public class coro [ object _coro_object, type _coro_type ]:
cdef __interrupt (self, the_exception):
"""Schedule the coro to resume with an exception.
:Parameters:
- `the_exception`: The exception to raise (may be class or instance).
:param the_exception: The exception to raise (may be class or instance).
:Exceptions:
- `DeadCoroutine`: The coroutine is dead (it has already exited).
- `ScheduleError`: The coroutine is already scheduled to run.
- `ScheduleError`: Attempted to interrupt the currently running coro.
:raises DeadCoroutine: The coroutine is dead (it has already exited).
:raises ScheduleError: The coroutine is already scheduled to run.
:raises ScheduleError: Attempted to interrupt the currently running coro.
"""
self._schedule (exception (the_exception))
def shutdown (self):
"""Shut down this coroutine.
This will raise the `Shutdown` exception on this thread.
This will raise the :exc:`Shutdown` exception on this thread.
This method will not fail. If the thread is already dead, then it is
ignored. If the thread hasn't started, then it is canceled.
......@@ -575,22 +569,19 @@ cdef public class coro [ object _coro_object, type _coro_type ]:
def raise_exception (self, the_exception, force=True, cancel_start=False):
"""Schedule this coroutine to resume with an exception.
:Parameters:
- `the_exception`: The exception to raise. May be an Exception
class or instance.
- `force`: If True, will force the exception to be raised, even if
:param the_exception: The exception to raise. May be an Exception class or instance.
:param force: If True, will force the exception to be raised, even if
the coroutine is already scheduled. Defaults to True.
- `cancel_start`: If True, will cancel the coroutine if it has not
:param cancel_start: If True, will cancel the coroutine if it has not
started, yet. If False, and the couroutine has not started, then
it will rise `NotStartedError`. Defaults to False.
it will rise :exc:`NotStartedError`. Defaults to False.
:Exceptions:
- `DeadCoroutine`: The coroutine is dead (it has already exited).
- `ScheduleError`: The coroutine is already scheduled to run (and
`force` was set to False).
- `ScheduleError`: Attempted to raise an exception on the currently
:raises DeadCoroutine: The coroutine is dead (it has already exited).
:raises ScheduleError: The coroutine is already scheduled to run (and
``force`` was set to False).
:raises ScheduleError: Attempted to raise an exception on the currently
running coro.
- `NotStartedError`: The coroutine has not started, yet.
:raises NotStartedError: The coroutine has not started, yet.
"""
IF CORO_DEBUG:
# Minstack coro used to take an "exception value" as the second
......@@ -653,8 +644,7 @@ cdef public class coro [ object _coro_object, type _coro_type ]:
def set_name (self, name):
"""Set the name of this coroutine thread.
:Parameters:
- `name`: The name of the thread.
:param name: The name of the thread.
"""
self.name = name
return self
......@@ -668,8 +658,7 @@ cdef public class coro [ object _coro_object, type _coro_type ]:
If no name has been specified, then a name is generated.
:Return:
Returns the coroutine name.
:returns: The coroutine name.
"""
return self.name
......@@ -687,8 +676,7 @@ cdef public class coro [ object _coro_object, type _coro_type ]:
When a coroutine is created, it defaults to 4.
:Parameters:
- `maximum`: The maximum number of selfish acts.
:param maximum: The maximum number of selfish acts.
"""
if maximum > 32768:
raise ValueError('Value too large.')
......@@ -727,8 +715,7 @@ def get_live_coros():
Note that this includes coroutines that have not started or have exited,
but not deallocated, yet.
:Return:
Returns the number of live coroutine objects.
:returns: The number of live coroutine objects.
"""
global live_coros
return live_coros
......@@ -765,11 +752,9 @@ def set_exception_notifier (new_func):
due to an exception. The default exception notifier simply prints the name
of the coroutine and a traceback of where the exception was raised.
:Parameters:
- `new_func`: The exception notifier to call. It takes no arguments.
:param new_func: The exception notifier to call. It takes no arguments.
:Return:
Returns the old exception notifier.
:returns: The old exception notifier.
"""
global exception_notifier
old_func = exception_notifier
......@@ -805,7 +790,7 @@ cdef public void _wrap1 "_wrap1" (coro co):
# event queue
# ================================================================================
include "event_queue.pyx"
from event_queue import event_queue, __event_queue_version__
cdef class event:
cdef public uint64_t t
......@@ -875,12 +860,11 @@ class SimultaneousError (Exception):
"""Two threads attempted a conflicting blocking operation (e.g., read() on
the same descriptor).
:IVariables:
- `co`: The coroutine that is trying to block on an event.
- `other`: The coroutine or function that is already waiting on the
:ivar co: The coroutine that is trying to block on an event.
:ivar other: The coroutine or function that is already waiting on the
event.
- `event`: The event that it is trying to block on. For kqueue, this
is normally a `kevent_key` object.
:ivar event: The event that it is trying to block on. For kqueue, this
is normally a ``kevent_key`` object.
"""
def __init__(self, co, other, object event):
......@@ -923,10 +907,10 @@ cdef public class sched [ object sched_object, type sched_type ]:
cdef int stack_size
cdef public object _current, pending, staging
cdef coro _last
cdef readonly event_queue events
cdef int profiling
cdef uint64_t latency_threshold
cdef zstack squish
cdef object events
def __init__ (self, stack_size=4*1024*1024):
self.stack_size = stack_size
......@@ -1024,11 +1008,9 @@ cdef public class sched [ object sched_object, type sched_type ]:
cdef _unschedule (self, coro co):
"""Unschedule this coroutine.
:Parameters:
- `co`: The coroutine to unschedule.
:param co: The coroutine to unschedule.
:Return:
Returns True if it was successfully unscheduled, False if not.
:returns: True if it was successfully unscheduled, False if not.
"""
cdef int i
for i from 0 <= i < len(self.pending):
......@@ -1079,15 +1061,12 @@ cdef public class sched [ object sched_object, type sched_type ]:
The default latency warning threshold is 0.2 seconds. This will allow
you to change the threshold by multiplying the 0.2 value.
:Parameters:
- `factor`: The latency threshold multiplier. May be a number from
:param factor: The latency threshold multiplier. May be a number from
0 to 300. A value of 0 disables latency warnings.
:Return:
Returns the old multipler factor.
:returns: The old multipler factor.
:Exceptions:
- `ValueError`: The factor is too small or too large.
:raises ValueError: The factor is too small or too large.
"""
if factor < 0 or factor > 300:
raise ValueError('Latency factor must be a number from 0 to 300.')
......@@ -1116,25 +1095,22 @@ cdef public class sched [ object sched_object, type sched_type ]:
Nested timeouts will be handled correctly. If an outer timeout fires
first, then only the outer ``except TimeoutError`` exception handler
will catch it. An exception handlers on the inside will be skipped
becaue the actual exception is the `Interrupted` exception until it
becaue the actual exception is the :exc:`Interrupted` exception until it
gets to the original ``with_timeout`` frame.
Nested timeouts that are set to fire at the exact same time are not
defined which one will fire first.
Care must be taken to *never* catch the `Interrupted` exception within
Care must be taken to *never* catch the :exc:`Interrupted` exception within
code that is wrapped with a timeout.
:Parameters:
- `delta`: The number of seconds to wait before raising a timeout.
:param delta: The number of seconds to wait before raising a timeout.
Should be >= 0. Negative value will be treated as 0.
- `function`: The function to call.
:param function: The function to call.
:Return:
Returns the return value of the function.
:returns: The return value of the function.
:Exceptions:
- `TimeoutError`: The function did not return within the specified
:raises TimeoutError: The function did not return within the specified
timeout.
"""
cdef timebomb tb
......@@ -1149,7 +1125,7 @@ cdef public class sched [ object sched_object, type sched_type ]:
tb = timebomb (self._current, delta)
e = event (tb.when, tb)
self.events.c_insert (e.t, e)
self.events.insert (e.t, e)
try:
try:
return PyObject_Call (function, args, kwargs)
......@@ -1162,25 +1138,24 @@ cdef public class sched [ object sched_object, type sched_type ]:
raise
finally:
if not e.expired:
self.events.c_delete (e.t, e)
self.events.remove (e.t, e)
e.expire()
cdef sleep (self, uint64_t when):
"""Sleep until a specific point in time.
:Parameters:
- `when`: The TSC value when you want the coroutine to wake up.
:param when: The TSC value when you want the coroutine to wake up.
"""
cdef event e
IF CORO_DEBUG:
assert self._current is not None
e = event (when, self._current)
self.events.c_insert (when, e)
self.events.insert (when, e)
try:
(<coro>self._current).__yield ()
finally:
if not e.expired:
self.events.c_delete (e.t, e)
self.events.remove (e.t, e)
e.expire()
def sleep_relative (self, delta):
......@@ -1191,8 +1166,7 @@ cdef public class sched [ object sched_object, type sched_type ]:
Your thread may continue running (with the interrupt rescheduled to try
again later), or it may be interrupted.
:Parameters:
- `delta`: The number of seconds to sleep.
:param delta: The number of seconds to sleep.
"""
cdef uint64_t when
# Two lines to avoid Pyrex Python conversion.
......@@ -1201,7 +1175,10 @@ cdef public class sched [ object sched_object, type sched_type ]:
self.sleep (when)
def sleep_absolute (self, uint64_t when):
"""This is an alias for the `sleep` method."""
"""Sleep until a specific point in time.
:param when: The TSC value when you want the coroutine to wake up.
"""
self.sleep (when)
cdef schedule_ready_events (self, uint64_t now):
......@@ -1211,10 +1188,10 @@ cdef public class sched [ object sched_object, type sched_type ]:
cdef coro c
retry = _fifo()
while self.events.len():
e = self.events.c_top(NULL)
while len(self.events):
e = self.events.top()
if e.t <= now:
self.events.c_pop(NULL)
self.events.pop()
# two kinds of event values:
# 1) a coro (for sleep_{relative,absolute}())
# 2) a timebomb (for with_timeout())
......@@ -1239,19 +1216,19 @@ cdef public class sched [ object sched_object, type sched_type ]:
# retry all the timebombs that failed due to ScheduleError
while retry.size:
e = retry._pop()
self.events.c_insert (e.t, e)
self.events.insert (e.t, e)
cdef get_timeout_to_next_event (self, int default_timeout):
cdef uint64_t delta, now
cdef event e
# default_timeout is in seconds
now = c_rdtsc()
if self.events.len():
if len(self.events):
# 1) get time to next event
while 1:
e = self.events.c_top(NULL)
e = self.events.top()
if e.expired:
self.events.c_pop(NULL)
self.events.pop()
else:
break
if e.t < now:
......@@ -1272,8 +1249,7 @@ cdef public class sched [ object sched_object, type sched_type ]:
def event_loop (self, timeout=30):
"""Start the event loop.
:Parameters:
- `timeout`: The amount of time to wait for kevent to return
:param timeout: The amount of time to wait for kevent to return
events. You should probably *not* set this value. Defaults to 30
seconds.
"""
......@@ -1342,11 +1318,10 @@ def yield_slice():
def schedule (coro co, value=None):
"""Schedule a coroutine to run.
See `coro.schedule` for more detail.
See :meth:`coro.schedule` for more detail.
:Parameters:
- `co`: The coroutine to schedule.
- `value`: The value to resume the coroutine with. Defaults to None.
:param co: The coroutine to schedule.
:param value: The value to resume the coroutine with. Defaults to None.
"""
return co._schedule (value)
......@@ -1396,8 +1371,7 @@ def print_stderr (s):
This will print the thread id, followed by a timestamp, followed by the
string. If the string does not end with a newline, one will be added.
:Parameters:
- `s`: A string to print.
:param s: A string to print.
"""
try:
current_thread = current()
......@@ -1435,11 +1409,9 @@ def spawn (fun, *args, **kwargs):
Additional arguments and keyword arguments will be passed to the given function.
:Parameters:
- `fun`: The function to call when the coroutine starts.
:param fun: The function to call when the coroutine starts.
:Return:
Returns the new coroutine object.
:returns: The new coroutine object.
"""
return _spawn (fun, args, kwargs)
......@@ -1451,11 +1423,9 @@ def new (fun, *args, **kwargs):
This will not start the coroutine. Call the ``start`` method on the
coroutine to schedule it to run.
:Parameters:
- `fun`: The function to call when the coroutine starts.
:param fun: The function to call when the coroutine starts.
:Return:
Returns the new coroutine object.
:returns: The new coroutine object.
"""
id = get_coro_id()
co = coro (fun, args, kwargs, id)
......@@ -1468,8 +1438,7 @@ def set_exit(exit_code=0):
Note that if any other coroutines are scheduled to run, they will be given
a chance to run before exiting.
:Parameters:
- `exit_code`: The exit code of the process. Defaults to 0.
:param exit_code: The exit code of the process. Defaults to 0.
"""
global _exit
global _exit_code
......@@ -1482,8 +1451,7 @@ def set_print_exit_string(val):
By default, the string will be printed.
:Parameters:
- `val`: Whether or not "Exiting" should be printed when the event loop
:param val: Whether or not "Exiting" should be printed when the event loop
exits.
"""
global _print_exit_string
......
coro/dns/__init__.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
coro/dns/packet.pyx 0 → 100644
View file @ e921e61e
# -*- Mode: Cython -*-
# See RFC1035
# History/Sources: Old-School Python DNS Demo, IronPort, Eric Huss's hand-written C update.
import socket
from libc.stdint cimport uint32_t, uint16_t, uint8_t
from libc.string cimport memcpy
from cpython.string cimport PyString_FromStringAndSize
# updated from a document at the IANA
# TYPE values (section 3.2.2)
class TYPE:
A = 1 # a host address [RFC1035]
NS = 2 # an authoritative name server [RFC1035]
MD = 3 # a mail destination (Obsolete - use MX) [RFC1035]
MF = 4 # a mail forwarder (Obsolete - use MX) [RFC1035]
CNAME = 5 # the canonical name for an alias [RFC1035]
SOA = 6 # marks the start of a zone of authority [RFC1035]
MB = 7 # a mailbox domain name (EXPERIMENTAL) [RFC1035]
MG = 8 # a mail group member (EXPERIMENTAL) [RFC1035]
MR = 9 # a mail rename domain name (EXPERIMENTAL) [RFC1035]
_NULL = 10 # a null RR (EXPERIMENTAL) [RFC1035]
WKS = 11 # a well known service description [RFC1035]
PTR = 12 # a domain name pointer [RFC1035]
HINFO = 13 # host information [RFC1035]
MINFO = 14 # mailbox or mail list information [RFC1035]
MX = 15 # mail exchange [RFC1035]
TXT = 16 # text strings [RFC1035]
RP = 17 # for Responsible Person [RFC1183]
AFSDB = 18 # for AFS Data Base location [RFC1183]
X25 = 19 # for X.25 PSDN address [RFC1183]
ISDN = 20 # for ISDN address [RFC1183]
RT = 21 # for Route Through [RFC1183]
NSAP = 22 # for NSAP address, NSAP style A record [RFC1706]
NSAPPTR = 23 #
SIG = 24 # for security signature [RFC2535]
KEY = 25 # for security key [RFC2535]
PX = 26 # X.400 mail mapping information [RFC2163]
GPOS = 27 # Geographical Position [RFC1712]
AAAA = 28 # IP6 Address [Thomson]
LOC = 29 # Location Information [Vixie]
NXT = 30 # Next Domain [RFC2535]
EID = 31 # Endpoint Identifier [Patton]
NIMLOC = 32 # Nimrod Locator [Patton]
SRV = 33 # Server Selection [RFC2782]
ATMA = 34 # ATM Address [Dobrowski]
NAPTR = 35 # Naming Authority Pointer [RFC2168, RFC2915]
KX = 36 # Key Exchanger [RFC2230]
CERT = 37 # CERT [RFC2538]
A6 = 38 # A6 [RFC2874]
DNAME = 39 # DNAME [RFC2672]
SINK = 40 # SINK [Eastlake]
OPT = 41 # OPT [RFC2671]
UINFO = 100 # [IANA-Reserved]
UID = 101 # [IANA-Reserved]
GID = 102 # [IANA-Reserved]
UNSPEC = 103 # [IANA-Reserved]
TKEY = 249 # Transaction Key [RFC2930]
TSIG = 250 # Transaction Signature [RFC2845]
IXFR = 251 # incremental transfer [RFC1995]
AXFR = 252 # transfer of an entire zone [RFC1035]
MAILB = 253 # mailbox-related RRs (MB, MG or MR) [RFC1035]
MAILA = 254 # mail agent RRs (Obsolete - see MX) [RFC1035]
ANY = 255 # A request for all records [RFC1035]
# Additional TYPE values from host.c source
UNAME = 110
MP = 240
TYPE_MAP = {}
for name in dir(TYPE):
if not name.startswith ('__'):
TYPE_MAP [getattr (TYPE, name)] = name
# CLASS values (section 3.2.4)
class CLASS:
IN = 1 # the Internet
CS = 2 # the CSNET class (Obsolete - used only for examples in some obsolete RFCs)
CH = 3 # the CHAOS class
HS = 4 # Hesiod [Dyer 87]
CLASS_MAP = {}
for name in dir(CLASS):
if not name.startswith ('__'):
CLASS_MAP [getattr (CLASS, name)] = name
# QCLASS values (section 3.2.5)
class QCLASS:
ANY = 255 # any class
class OPCODE:
QUERY = 0
IQUERY = 1
STATUS = 2
# Low-level 16 and 32 bit integer packing and unpacking
cpdef bytes pack16bit (uint32_t n):
cdef unsigned char r[2]
r[0] = (n >> 8) & 0xff
r[1] = n & 0xff
return r[:2]
cpdef bytes pack32bit (uint32_t n):
cdef unsigned char r[4]
r[0] = (n >> 24) & 0xff
r[1] = (n >> 16) & 0xff
r[2] = (n >> 8) & 0xff
r[3] = (n >> 0) & 0xff
return r[:4]
cpdef int unpack16bit (unsigned char * s):
return (s[0])<<8 | (s[1])
cpdef int unpack32bit (unsigned char * s):
return ( (s[0]) << 24
| (s[1]) << 16
| (s[2]) << 8
| (s[3]) << 0 )
# XXX Use inet_pton, inet_ntop directly.
def addr2bin (addr):
cdef uint32_t n
if type (addr) == type (0):
return addr
parts = addr.split ('.')
if len (parts) != 4:
raise ValueError ('bad IP address')
n = 0
for byte in parts:
n = n<<8 | int (byte)
return n
def bin2addr (n):
return '%d.%d.%d.%d' % (
(n>>24)&0xFF,
(n>>16)&0xFF,
(n>>8)&0xFF,
n&0xFF
)
cdef class buffer:
cdef uint32_t offset, size
cdef bytes data
def __init__ (self, size=500):
self.data = PyString_FromStringAndSize (NULL, size)
self.size = size
self.offset = 0
cdef ensure (self, int n):
cdef int new_size = self.size + (self.size // 2)
if self.offset + n > self.size:
self.data = PyString_FromStringAndSize (self.data, new_size)
def add (self, bytes d):
cdef char * buf = self.data
cdef char * dp = d
cdef int n = len (d)
self.ensure (n)
memcpy (buf + self.offset, dp, n)
self.offset += n
def add_byte (self, unsigned char d):
cdef char * buf = self.data
self.ensure (1)
buf[self.offset] = d
self.offset += 1
def get (self):
return self.data[:self.offset]
# used to retroactively set the size of an RR
def set16 (self, uint32_t offset, uint16_t val):
cdef char * buf = self.data
if offset > 0 and offset <= self.offset - 2:
buf[offset+0] = (val >> 8) & 0xff
buf[offset+1] = val & 0xff
else:
raise IndexError (offset)
cdef class Header:
cdef public uint16_t id, qdcount, ancount, nscount, arcount
cdef public uint8_t qr, opcode, aa, tc, rd, ra, z, rcode
cdef set (self, id,
qr, opcode, aa, tc, rd, ra, z, rcode,
qdcount, ancount, nscount, arcount
):
self.id = id
self.qr = qr
self.opcode = opcode
self.aa = aa
self.tc = tc
self.rd = rd
self.ra = ra
self.z = z
self.rcode = rcode
self.qdcount = qdcount
self.ancount = ancount
self.nscount = nscount
self.arcount = arcount
return self
class PackError (Exception):
pass
cdef class Packer:
cdef buffer buf
cdef dict index
cdef uint32_t rdstart
def __init__ (self):
self.buf = buffer()
self.index = {}
self.rdstart = 0
cdef add (self, bytes d):
self.buf.add (d)
def getbuf (self):
return self.buf.get()
cpdef addbyte (self, unsigned char c):
self.buf.add_byte (c)
cpdef addbytes (self, bytes b):
self.add (b)
cpdef add16bit (self, uint16_t n):
self.add (pack16bit (n))
cpdef add32bit (self, uint32_t n):
self.add (pack32bit (n))
cpdef addaddr (self, bytes addr):
n = addr2bin (addr)
self.add (pack32bit (n))
cpdef addstring (self, bytes s):
self.addbyte (len(s))
self.addbytes (s)
cpdef compressed_addname (self, bytes name):
# Domain name packing (section 4.1.4)
# Add a domain name to the buffer, possibly using pointers.
# The case of the first occurrence of a name is preserved.
# Redundant dots are ignored.
parts = [p for p in name.split ('.') if p]
for part in parts:
if len (part) > 63:
raise PackError ('label too long')
keys = []
for i in range (len (parts)):
key = ('.'.join (parts[i:])).lower()
keys.append (key)
if self.index.has_key (key):
pointer = self.index[key]
break
else:
i = len (parts)
pointer = None
# Do it into temporaries first so exceptions don't
# mess up self.index and self.buf
buf = buffer()
new_keys = []
for j in range(i):
label = parts[j]
n = len (label)
if self.buf.offset + buf.offset < 0x3FFF:
new_keys.append ((keys[j], self.buf.offset + buf.offset))
buf.add_byte (n)
buf.add (label)
if pointer is not None:
buf.add (pack16bit (pointer | 0xc000))
else:
buf.add (b'\0')
self.buf.add (buf.get())
for key, value in new_keys:
self.index[key] = value
cpdef addname (self, name):
self.compressed_addname (name)
# Question (section 4.1.2)
cpdef addQuestion (self, qname, qtype, qclass):
self.addname (qname)
self.add16bit (qtype)
self.add16bit (qclass)
def addHeader (self, Header h):
self.add16bit (h.id)
self.add16bit (
(h.qr&1)<<15 | (h.opcode&0xF)<<11 | (h.aa&1)<<10
| (h.tc&1)<<9 | (h.rd&1)<<8 | (h.ra&1)<<7
| (h.z&7)<<4 | (h.rcode&0xF)
)
self.add16bit (h.qdcount)
self.add16bit (h.ancount)
self.add16bit (h.nscount)
self.add16bit (h.arcount)
# RR toplevel format (section 3.2.1)
cpdef addRRheader (self, bytes name, int type, int klass, int ttl):
self.addname(name)
self.add16bit(type)
self.add16bit(klass)
self.add32bit(ttl)
self.add16bit(0)
self.rdstart = self.buf.offset
cpdef endRR (self):
cdef int rdlength = self.buf.offset - self.rdstart
self.buf.set16 (self.rdstart-2, rdlength)
# Standard RRs (section 3.3)
cpdef addCNAME (self, name, klass, ttl, cname):
self.addRRheader (name, TYPE.CNAME, klass, ttl)
self.addname (cname)
self.endRR()
cpdef addHINFO (self, name, klass, ttl, cpu, os):
self.addRRheader (name, TYPE.HINFO, klass, ttl)
self.addstring (cpu)
self.addstring (os)
self.endRR()
cpdef addMX (self, name, klass, ttl, preference, exchange):
self.addRRheader (name, TYPE.MX, klass, ttl)
self.add16bit (preference)
self.addname (exchange)
self.endRR()
cpdef addNS (self, name, klass, ttl, nsdname):
self.addRRheader (name, TYPE.NS, klass, ttl)
self.addname (nsdname)
self.endRR()
cpdef addPTR (self, name, klass, ttl, ptrdname):
self.addRRheader (name, TYPE.PTR, klass, ttl)
self.addname (ptrdname)
self.endRR()
cpdef addSOA (self, name, klass, ttl, mname, rname, serial, refresh, retry, expire, minimum):
self.addRRheader (name, TYPE.SOA, klass, ttl)
self.addname (mname)
self.addname (rname)
self.add32bit (serial)
self.add32bit (refresh)
self.add32bit (retry)
self.add32bit (expire)
self.add32bit (minimum)
self.endRR()
cpdef addTXT (self, name, klass, ttl, list):
self.addRRheader (name, TYPE.TXT, klass, ttl)
for txtdata in list:
self.addstring (txtdata)
self.endRR()
# Internet specific RRs (section 3.4) -- class = IN
cpdef addA (self, name, ttl, address):
self.addRRheader (name, TYPE.A, CLASS.IN, ttl)
self.addaddr (address)
self.endRR()
class UnpackError (Exception):
pass
import sys
W = sys.stderr.write
cdef class Unpacker:
cdef bytes buf
cdef uint32_t offset
cdef uint32_t length
cdef uint32_t rdend
def __init__ (self, bytes buf):
self.buf = buf
self.offset = 0
self.length = len (self.buf)
cpdef unsigned char getbyte (self):
cdef unsigned char * buf = self.buf
c = buf[self.offset]
self.offset += 1
return c
cdef unsigned char * get_pointer (self, uint32_t n):
cdef unsigned char * p = self.buf
self.ensure (n)
return p + self.offset
cpdef bytes getbytes (self, uint32_t n):
cdef bytes s
self.ensure (n)
s = self.buf[self.offset : self.offset + n]
self.offset += n
return s
cpdef ensure (self, uint32_t n):
if self.offset + n > self.length:
raise UnpackError ('not enough data left %d+%d > %d' % (n, self.offset, self.length))
cpdef get16bit (self):
r = unpack16bit (self.get_pointer (2))
self.offset += 2
return r
cpdef get32bit (self):
r = unpack32bit (self.get_pointer (4))
self.offset += 4
return r
cpdef getaddr (self):
return bin2addr (self.get32bit())
cpdef getstring (self):
return self.getbytes (self.getbyte())
cpdef getname (self):
# Domain name unpacking (section 4.1.4)
cdef uint32_t i = self.getbyte()
cdef uint32_t j = 0
cdef uint32_t pointer = 0
if i & 0xC0 == 0xC0:
j = self.getbyte()
pointer = ((i<<8) | j) & ~0xC000
save_offset = self.offset
try:
self.offset = pointer
domain = self.getname()
finally:
self.offset = save_offset
return domain
if i == 0:
return ''
else:
domain = self.getbytes(i)
remains = self.getname()
if not remains:
return domain.lower()
else:
return (domain + '.' + remains).lower()
def getHeader (self):
cdef uint16_t flags
h = Header()
h.id = self.get16bit()
flags = self.get16bit()
h.qr = (flags>>15)&1
h.opcode = (flags>>11)&0xF
h.aa = (flags>>10)&1
h.tc = (flags>>9)&1
h.rd = (flags>>8)&1
h.ra = (flags>>7)&1
h.z = (flags>>4)&7
h.rcode = (flags>>0)&0xF
h.qdcount = self.get16bit()
h.ancount = self.get16bit()
h.nscount = self.get16bit()
h.arcount = self.get16bit()
return h
# resource records
cpdef getRRheader(self):
rname = self.getname()
rtype = self.get16bit()
rclass = self.get16bit()
ttl = self.get32bit()
rdlength = self.get16bit()
self.rdend = self.offset + rdlength
return (rname, rtype, rclass, ttl, rdlength)
cpdef getCNAMEdata(self):
return self.getname()
cpdef getHINFOdata(self):
return (
self.getstring(),
self.getstring()
)
cpdef getMXdata(self):
return self.get16bit(), self.getname()
cpdef getNSdata(self):
return self.getname()
cpdef getPTRdata(self):
return self.getname()
cpdef getSOAdata(self):
return (
self.getname(),
self.getname(),
self.get32bit(),
self.get32bit(),
self.get32bit(),
self.get32bit(),
self.get32bit()
)
cpdef getTXTdata(self):
cdef list parts = []
while self.offset != self.rdend:
parts.append (self.getstring())
return parts
# XXX replace this with inet_ntop
cpdef getAdata(self):
return self.getaddr()
cpdef getAAAAdata (self):
return socket.inet_ntop (socket.AF_INET6, self.getbytes (16))
# Question (section 4.1.2)
cpdef getQuestion (self):
return self.getname(), self.get16bit(), self.get16bit()
# ---------------
cpdef getRR (self):
rname, rtype, rclass, ttl, rdlength = self.getRRheader()
if rtype == TYPE.CNAME:
data = self.getCNAMEdata()
elif rtype == TYPE.MX:
data = self.getMXdata()
elif rtype == TYPE.NS:
data = self.getNSdata()
elif rtype == TYPE.PTR:
data = self.getPTRdata()
elif rtype == TYPE.SOA:
data = self.getSOAdata()
elif rtype == TYPE.TXT:
data = self.getTXTdata()
elif rtype == TYPE.A:
data = self.getAdata()
elif rtype == TYPE.AAAA:
data = self.getAAAAdata()
elif rtype == TYPE.HINFO:
data = self.getHINFOdata()
else:
data = self.getbytes (rdlength)
if self.offset != self.rdend:
raise UnpackError ('end of RR not reached')
else:
rtype = TYPE_MAP.get (rtype, rtype)
rclass = CLASS_MAP.get (rclass, rclass)
return rname, rtype, rclass, ttl, data
def unpack (self):
h = self.getHeader()
qdl = []
for i in range (h.qdcount):
qdl.append (self.getQuestion())
anl = []
for i in range (h.ancount):
anl.append (self.getRR())
nsl = []
for i in range (h.nscount):
nsl.append (self.getRR())
arl = []
for i in range (h.arcount):
arl.append (self.getRR())
return h, qdl, anl, nsl, arl
coro/dns/stub_resolver.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
import coro
import coro.dns
import coro.dns.packet as packet
import random
class QueryFailed (Exception):
pass
class stub_resolver:
def __init__ (self, nameservers, inflight=200):
self.nameservers = nameservers
self.inflight = coro.semaphore (inflight)
self.inflight_ids = set()
def lookup (self, qname, qtype, timeout=10, retries=3):
m = packet.Packer()
h = packet.Header()
while 1:
qid = random.randrange (65536)
# avoid collisions
if qid not in self.inflight_ids:
break
h.id = qid
h.opcode = packet.OPCODE.QUERY
h.rd = 1
h.qdcount = 1
m.addHeader (h)
m.addQuestion (qname, qtype, packet.CLASS.IN)
p = m.getbuf()
for addr in self.nameservers:
for i in range (retries):
self.inflight.acquire (1)
self.inflight_ids.add (qid)
try:
s = coro.udp_sock()
s.connect ((addr, 53))
s.send (p)
try:
reply = coro.with_timeout (timeout, s.recv, 1000)
u = packet.Unpacker (reply)
result = u.unpack()
rh = result[0]
if rh.id != qid:
raise QueryFailed ("bad id in reply")
else:
return result
except coro.TimeoutError:
pass
finally:
self.inflight.release (1)
self.inflight_ids.remove (qid)
raise QueryFailed ("no reply from nameservers")
def gethostbyname (self, name, qtype):
header, qdl, anl, nsl, arl = self.lookup (name, qtype)
for answer in anl:
name, rtype, _, ttl, addr = answer
if getattr (packet.TYPE, rtype) == qtype:
return addr
else:
raise QueryFailed ("no answer in nameserver reply")
def resolve_ipv4 (self, name):
return self.gethostbyname (name, packet.TYPE.A)
def resolve_ipv6 (self, name):
return self.gethostbyname (name, packet.TYPE.AAAA)
def install (nameserver_ips):
"install a stub resolver into the coro socket layer"
coro.set_resolver (
stub_resolver (nameserver_ips)
)
coro/dns/test/tpack.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
import coro
import coro.dns.packet as dns
def testpacker():
# See section 4.1.4 of RFC 1035
p = dns.Packer()
p.addbytes('*' * 20)
p.addname('f.ISI.ARPA')
p.addbytes('*' * 8)
p.addname('Foo.F.isi.arpa')
p.addbytes('*' * 18)
p.addname('arpa')
p.addbytes('*' * 26)
p.addname('')
packet = p.getbuf()
assert packet == (
'********************\x01f\x03ISI\x04ARPA\x00'
'********\x03Foo\xc0\x14******************\xc0\x1a'
'**************************\x00'
)
u = dns.Unpacker (packet)
res = (
u.getbytes(20),
u.getname(),
u.getbytes(8),
u.getname(),
u.getbytes(18),
u.getname(),
u.getbytes(26),
u.getname(),
)
assert res == (
'********************',
'f.isi.arpa',
'********',
'foo.f.isi.arpa',
'******************',
'arpa',
'**************************',
''
)
def test_packer_2 ():
p = dns.Packer()
h = dns.Header()
h.id = 3141
h.opcode = dns.OPCODE.QUERY
h.rd = 0
h.ancount = 1
h.arcount = 1
h.qdcount = 1
p.addHeader (h)
p.addQuestion ('glerg.org', dns.TYPE.CNAME, dns.CLASS.IN)
p.addCNAME ('glerg.org', dns.CLASS.IN, 3000, 'blerb.com')
p.addHINFO ('brodig.com', dns.CLASS.IN, 5000, 'vax', 'vms')
data = p.getbuf()
u = dns.Unpacker (data)
h, qdl, anl, nsl, arl = u.unpack()
assert qdl == [('glerg.org', 5, 1)]
assert anl == [('glerg.org', 'CNAME', 'IN', 3000, 'blerb.com')]
assert arl == [('brodig.com', 'HINFO', 'IN', 5000, ('vax', 'vms'))]
assert nsl == []
assert h.id == 3141
assert h.opcode == dns.OPCODE.QUERY
assert h.ancount == 1
assert h.qdcount == 1
def t0 (qname='www.nightmare.com', qtype=dns.TYPE.A):
m = dns.Packer()
h = dns.Header()
h.id = 3141
h.opcode = dns.OPCODE.QUERY
h.rd = 1
h.qdcount = 1
m.addHeader (h)
m.addQuestion (qname, qtype, dns.CLASS.IN)
p = m.getbuf()
return p
def t1 (qname, qtype):
p = t0 (qname, getattr (dns.TYPE, qtype))
s = coro.udp_sock()
s.connect (('192.168.200.1', 53))
s.send (p)
r = s.recv (8192)
coro.write_stderr ('reply=%r\n' % (r,))
u = dns.Unpacker (r)
return u.unpack()
def t2():
import coro.dns.stub_resolver
r = coro.dns.stub_resolver.stub_resolver (['192.168.200.1'])
coro.set_resolver (r)
# XXX make this into a real unit test.
if __name__ == '__main__':
#import coro.backdoor
#coro.spawn (coro.backdoor.serve, unix_path='/tmp/xx.bd')
#coro.event_loop()
test_packer_2()
coro/event_queue.cc deleted 100644 → 0
View file @ 7ba1e191
/*
Copyright (c) 2002-2011 IronPort Systems and Cisco Systems
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#include "Python.h"
#include <map>
typedef std::multimap <unsigned long long, PyObject *> event_queue;
typedef event_queue::iterator event_queue_iter;
/* Needed because Pyrex is compiled as a C program and is expecting C-like
symbols.
*/
extern "C" {
/*
Create a new event queue.
*/
event_queue *
event_queue_new()
{
return new event_queue;
}
/*
Delete the event queue and free all data.
*/
void
event_queue_dealloc(event_queue * q)
{
event_queue_iter iter;
for (iter = q->begin(); iter != q->end(); iter++) {
Py_DECREF(iter->second);
}
delete q;
}
/*
Returns the length of the queue.
*/
int
event_queue_len(event_queue * q)
{
return q->size();
}
/*
Returns a new iterator.
*/
event_queue_iter
event_queue_new_iter(event_queue * q)
{
return q->begin();
}
/*
Return the current value of the iterator and move the position forward.
The timestamp of the first element is stored in time (if not NULL). The value is returned.
Returns NULL if the queue is empty with StopIteration set.
*/
PyObject *
event_queue_iter_next(event_queue * q, event_queue_iter * iter, uint64_t * time)
{
PyObject * value;
if (*iter == q->end()) {
PyErr_SetObject(PyExc_StopIteration, NULL);
return NULL;
} else {
if (time) {
*time = (*iter)->first;
}
value = (*iter)->second;
Py_INCREF(value);
(*iter)++;
return value;
}
}
/*
Peek at the top of the queue.
The timestamp of the first element is stored in time (if not NULL). The value is returned.
Returns NULL if the queue is empty with IndexError set.
*/
PyObject *
event_queue_top(event_queue * q, uint64_t * time)
{
PyObject * value;
if (q->size()) {
event_queue_iter iter = q->begin();
if (time) {
*time = iter->first;
}
value = iter->second;
Py_INCREF(value);
return value;
} else {
PyErr_SetString(PyExc_IndexError, "top of empty queue");
return NULL;
}
}
/*
Pop the first element off the queue.
The timestamp of the first element is stored in time (if not NULL). The value is returned.
Returns NULL if the queue is empty with IndexError set.
*/
PyObject *
event_queue_pop(event_queue * q, uint64_t * time)
{
PyObject * value;
if (q->size()) {
event_queue_iter iter = q->begin();
if (time) {
*time = iter->first;
}
value = iter->second;
q->erase (iter);
return value;
} else {
PyErr_SetString(PyExc_IndexError, "pop from empty queue");
return NULL;
}
}
/*
Insert a new entry into the queue.
Returns 0 on succes, -1 on failure.
(Currently never fails.)
*/
int
event_queue_insert(event_queue * q, uint64_t time, PyObject * value)
{
q->insert (std::pair <uint64_t, PyObject *> (time, value));
Py_INCREF(value);
return 0;
}
/*
Delete an entry from the queue.
Returns 0 on success, -1 on failure with IndexError set.
*/
int
event_queue_delete(event_queue * q, uint64_t time, PyObject * value)
{
event_queue_iter iter = q->find(time);
// Iterate since we support duplicate keys.
while (iter != q->end()) {
if (iter->first != time) {
break;
}
if (iter->second == value) {
Py_DECREF(iter->second);
q->erase(iter);
return 0;
} else {
iter++;
}
}
PyErr_SetString(PyExc_IndexError, "event not found");
return -1;
}
} /* extern "C" */
coro/event_queue.h deleted 100644 → 0
View file @ 7ba1e191
/*
Copyright (c) 2002-2011 IronPort Systems and Cisco Systems
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#ifndef _EVENT_QUEUE_H_
#define _EVENT_QUEUE_H_
#include "Python.h"
/* Types are defined as void * to fake out Pyrex which can't handle C++ types. */
//typedef std::multimap <long long, PyObject *> event_queue;
typedef void * event_queue;
event_queue * event_queue_new(void);
void event_queue_dealloc(event_queue * q);
PyObject * event_queue_top(event_queue * q, uint64_t * time);
PyObject * event_queue_pop(event_queue * q, uint64_t * time);
int event_queue_insert(event_queue * q, uint64_t time, PyObject * value);
int event_queue_delete(event_queue * q, uint64_t time, PyObject * value);
int event_queue_len(event_queue * q);
//typedef event_queue::iterator event_queue_iter;
typedef void * event_queue_iter;
event_queue_iter event_queue_new_iter(event_queue * q);
PyObject * event_queue_iter_next(event_queue * q, event_queue_iter * iter, uint64_t * time);
#endif /* _EVENT_QUEUE_H_ */
coro/event_queue.pyx
View file @ e921e61e
......@@ -22,117 +22,111 @@
__event_queue_version__ = "$Id: event_queue.pyx,v 1.1 2007/01/03 00:19:50 ehuss Exp $"
cdef extern from "event_queue.h":
ctypedef void * cpp_event_queue "event_queue"
cpp_event_queue * event_queue_new()
void event_queue_dealloc(cpp_event_queue * q)
object event_queue_top(cpp_event_queue * q, uint64_t * time)
object event_queue_pop(cpp_event_queue * q, uint64_t * time)
int event_queue_insert(cpp_event_queue * q, uint64_t time, object) except -1
int event_queue_delete(cpp_event_queue * q, uint64_t time, object) except -1
int event_queue_len(cpp_event_queue * q)
ctypedef void * cpp_event_queue_iter "event_queue_iter"
cpp_event_queue_iter event_queue_new_iter(cpp_event_queue * q)
object event_queue_iter_next(cpp_event_queue * q, cpp_event_queue_iter * iter, uint64_t * time)
cdef class event_queue_iter
include "python.pxi"
from cython.operator cimport dereference as deref, preincrement as inc
from libcpp.utility cimport pair
from libc cimport uint64_t
cdef extern from "<map>" namespace "std":
cdef cppclass multimap[T, U]:
cppclass iterator:
pair[T,U]& operator*()
iterator operator++()
iterator operator--()
bint operator==(iterator)
bint operator!=(iterator)
map()
U& operator[](T&)
U& at(T&)
iterator begin()
size_t count(T&)
bint empty()
iterator end()
void erase(iterator)
void erase(iterator, iterator)
size_t erase(T&)
iterator find(T&)
pair[iterator, bint] insert(pair[T,U])
size_t size()
cdef class event_queue:
cdef cpp_event_queue * q
cdef multimap[uint64_t, PyObject*] *q
def __cinit__(self):
self.q = event_queue_new()
self.q = new multimap[uint64_t, PyObject*]()
def __dealloc__(self):
event_queue_dealloc(self.q)
cdef multimap[uint64_t, PyObject*].iterator it = self.q.begin()
while it != self.q.end():
Py_DECREF(<object> deref(it).second)
inc(it)
del self.q
def __len__(self):
return event_queue_len(self.q)
cpdef insert(self, uint64_t time, value):
"""Insert a new value into the queue.
cdef int len(self):
return event_queue_len(self.q)
:Parameters:
- `time`: The uint64 time.
- `value`: The value to insert.
"""
cdef pair[uint64_t, PyObject*] p
p.first, p.second = time, <PyObject *> value
self.q.insert(p)
Py_INCREF(value)
cdef c_top(self, uint64_t * time):
return event_queue_top(self.q, time)
def __len__(self):
return self.q.size()
def top(self):
cpdef top(self):
"""Peek at the top value of the queue.
:Return:
Returns a ``(time, value)`` tuple from the top of the queue.
Returns value from the top of the queue.
:Exceptions:
- `IndexError`: The queue is empty.
"""
cdef uint64_t time
if not self.q.size():
raise IndexError('Top of empty queue')
cdef multimap[uint64_t, PyObject*].iterator it = self.q.begin()
return <object> deref(it).second
value = event_queue_top(self.q, &time)
return (time, value)
cdef c_pop(self, uint64_t * time):
return event_queue_pop(self.q, time)
def pop(self):
cpdef pop(self):
"""Grab the top value of the queue and remove it.
:Return:
Returns a ``(time, value)`` tuple from the top of the queue.
Returns value from the top of the queue.
:Exceptions:
- `IndexError`: The queue is empty.
"""
cdef uint64_t time
value = event_queue_pop(self.q, &time)
return (time, value)
cdef c_insert(self, uint64_t time, value):
event_queue_insert(self.q, time, value)
def insert(self, uint64_t time, value):
"""Insert a new value into the queue.
:Parameters:
- `time`: The uint64 time.
- `value`: The value to insert.
"""
event_queue_insert(self.q, time, value)
cdef c_delete(self, uint64_t time, value):
event_queue_delete(self.q, time, value)
def delete(self, uint64_t time, value):
if not self.q.size():
raise IndexError('Top of empty queue')
cdef multimap[uint64_t, PyObject*].iterator it = self.q.begin()
value = <object> deref(it).second
self.q.erase(it)
Py_DECREF(value)
return value
cpdef remove(self, uint64_t time, value):
"""Delete a value from the queue.
:Parameters:
- `time`: The uint64 time.
- `value`: The value to delete.
"""
event_queue_delete(self.q, time, value)
def __iter__(self):
cdef event_queue_iter i
i = event_queue_iter()
i.q = self.q
i.iter = event_queue_new_iter(self.q)
return i
cdef class event_queue_iter:
cdef cpp_event_queue * q
cdef cpp_event_queue_iter iter
def __iter__(self):
return self
def __next__(self):
cdef uint64_t time
value = event_queue_iter_next(self.q, &self.iter, &time)
return (time, value)
cdef PyObject *val
cdef multimap[uint64_t, PyObject*].iterator it = self.q.find(time)
cdef PyObject *v = <PyObject *> value
while it != self.q.end():
if deref(it).first != time:
break
val = <PyObject *> deref(it).second
if v == val:
self.q.erase(it)
Py_DECREF(<object>val)
return 0
else:
inc(it)
raise IndexError('Event not found')
coro/http/__init__.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
from server import server, tlslite_server
import handlers
import coro
from coro import read_stream
import http_date
import session_handler
coro/http/client.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
import coro
import coro.read_stream
from protocol import http_file, header_set, latch
W = coro.write_stderr
class HTTP_Protocol_Error (Exception):
pass
# viewed at its core, HTTP is a two-way exchange of messages,
# some of which may have content associated with them.
# two different usage patterns for pipelined requests:
# 1) requests are made by different threads
# 2) requests are made by a single thread
#
# we accommodate both patterns here.
# for #2, use the lower-level send_request() method, for #1,
# use GET, PUT, etc...
class request:
def __init__ (self, force=True):
self.latch = latch()
self.force = force
self.content = None
self.response = None
self.rheader = None
self.rfile = None
def wake (self):
if self.rfile and self.force:
self.content = self.rfile.read()
self.latch.wake_all()
if self.rfile and not self.force:
self.rfile.wait()
def wait (self):
return self.latch.wait()
class client:
def __init__ (self, host, port=80, conn=None, inflight=100):
self.host = host
self.inflight = coro.semaphore (inflight)
if conn is None:
self.conn = coro.tcp_sock()
self.conn.connect ((host, port))
else:
self.conn = conn
self.stream = coro.read_stream.sock_stream (self.conn)
self.pending = coro.fifo()
coro.spawn (self.read_thread)
def read_thread (self):
while 1:
req = self.pending.pop()
self._read_message (req)
if not req.response:
break
else:
req.wake()
def _read_message (self, req):
req.response = self.stream.read_line()[:-2]
lines = []
while 1:
line = self.stream.read_line()
if not line:
raise HTTP_Protocol_Error ('unexpected close')
elif line == '\r\n':
break
else:
lines.append (line[:-2])
req.rheader = h = header_set (lines)
if h['content-length'] or h['transfer-encoding']:
req.rfile = http_file (h, self.stream)
def send_request (self, method, uri, headers, content=None, force=False):
try:
self.inflight.acquire (1)
req = request (force)
self._send_request (method, uri, headers, content)
self.pending.push (req)
return req
finally:
self.inflight.release (1)
def _send_request (self, method, uri, headers, content):
if not headers.has_key ('host'):
headers['host'] = self.host
if content:
if type(content) is str:
headers['content-length'] = len(content)
elif not headers.has_key ('content-length'):
headers['transfer-encoding'] = 'chunked'
req = (
'%s %s HTTP/1.1\r\n'
'%s\r\n' % (method, uri, headers)
)
self.conn.send (req)
# XXX 100 continue
if content:
if type(content) is str:
self.conn.send (content)
elif headers.has_key ('content-length'):
clen = int (headers.get_one ('content-length'))
slen = 0
for block in content:
self.conn.send (block)
slen += len(block)
if slen > clen:
raise HTTP_Protocol_Error ("content larger than declared length", clen, slen)
else:
if slen != clen:
raise HTTP_Protocol_Error ("content smaller than declared length", clen, slen)
else:
# chunked encoding
for block in content:
if block:
self.conn.writev (['%x\r\n' % (len (block),), block])
self.conn.send ('0\r\n')
def GET (self, uri, **headers):
headers = header_set().from_keywords (headers)
req = self.send_request ('GET', uri, headers, force=True)
req.wait()
return req
def GET_file (self, uri, **headers):
headers = header_set().from_keywords (headers)
req = self.send_request ('GET', uri, headers, force=False)
req.wait()
return req
def PUT (self, uri, content, **headers):
headers = header_set().from_keywords (headers)
req = self.send_request ('PUT', uri, headers, content, force=True)
req.wait()
return req
def POST (self, uri, content, **headers):
headers = header_set().from_keywords (headers)
req = self.send_request ('POST', uri, headers, content, force=True)
req.wait()
return req
coro/http/demo/grid.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
import coro
import coro.http
import backdoor
# toy: move an X through a grid.
# tests: POST data, compression, persistent connections, shared state
import sys
W = sys.stderr.write
class grid_handler:
def __init__ (self, w, h):
self.w = w
self.h = h
self.grid = [['.' for x in range (w)] for y in range (h)]
self.pos = [w/2, h/2]
self.grid[self.pos[1]][self.pos[0]] = 'X'
def match (self, request):
return request.path.startswith ('/grid')
def handle_request (self, request):
if request.path == '/grid/source':
request['content-type'] = 'text/plain'
request.set_deflate()
request.push (open ('grid.py', 'rb').read())
request.done()
return
request['content-type'] = 'text/html'
request.set_deflate()
if request.file:
data = request.file.read()
pairs = [ x.split('=') for x in data.split ('&') ]
for k, v in pairs:
if k == 'dir':
x0, y0 = self.pos
x1, y1 = self.pos
if v == 'left':
x1 = max (x0-1, 0)
elif v == 'right':
x1 = min (x0+1, self.w-1)
elif v == 'up':
y1 = max (y0-1, 0)
elif v == 'down':
y1 = min (y0+1, self.h-1)
else:
pass
self.grid[y0][x0] = '*'
self.grid[y1][x1] = 'X'
self.pos = [x1, y1]
else:
pass
l = []
for y in self.grid:
l.append (''.join (y))
request.push ('<pre>')
request.push ('\n'.join (l))
request.push ('\n</pre>\n')
request.push (
'<form name="input" action="grid" method="post">'
'<input type="submit" name="dir" value="left" />'
'<input type="submit" name="dir" value="right" />'
'<input type="submit" name="dir" value="up" />'
'<input type="submit" name="dir" value="down" />'
'</form>'
'<a href="/grid/source">source for this handler</a>'
)
request.done()
server = coro.http.server()
server.push_handler (grid_handler (50, 30))
server.push_handler (coro.http.handlers.coro_status_handler())
server.push_handler (coro.http.handlers.favicon_handler())
coro.spawn (server.start, ('0.0.0.0', 9001))
coro.spawn (backdoor.serve, unix_path='/tmp/httpd.bd')
coro.event_loop (30.0)
coro/http/demo/session.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
import coro
import coro.http
import backdoor
# demonstrate the session handler
import sys
W = sys.stderr.write
def session (sid, fifo):
i = 0
while 1:
try:
# wait a half hour for a new hit
request = coro.with_timeout (1800, fifo.pop)
except coro.TimeoutError:
break
else:
request['content-type'] = 'text/html'
if i == 10:
request.push (
'<html><h1>Session Over! Bye!</h1>'
'<a href="session">start over</a>'
'</html>'
)
request.done()
break
else:
request.push (
'<html><h1>Session Demo</h1><br><h2>Hit=%d</h2>'
'<a href="session">hit me!</a>'
'</html>' % (i,)
)
request.done()
i += 1
server = coro.http.server()
server.push_handler (coro.http.handlers.coro_status_handler())
server.push_handler (coro.http.session_handler.session_handler ('session', session))
server.push_handler (coro.http.handlers.favicon_handler())
coro.spawn (server.start, ('0.0.0.0', 9001))
coro.spawn (backdoor.serve, unix_path='/tmp/httpd.bd')
coro.event_loop (30.0)
coro/http/demo/tclient.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
import coro
W = coro.write_stderr
from coro.httpd.client import client as http_client
def t0():
c = http_client ('127.0.0.1', 80)
l = [ c.send_request ('GET', '/postgresql/html/', {}, content=None, force=True) for x in range (10) ]
for req in l:
req.wait()
W ('%s\n' % (req.response,))
def t1():
c = http_client ('127.0.0.1', 80)
rl = coro.in_parallel ([(c.GET, ('/postgresql/html/',))] * 10)
for x in rl:
W ('%s\n' % (x.response,))
return rl
if __name__ == '__main__':
import coro.backdoor
coro.spawn (t0)
coro.spawn (coro.backdoor.serve, unix_path='/tmp/xx.bd')
coro.event_loop()
coro/http/demo/tlslite_server.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
# demo an https server using the TLSLite package.
import coro
import coro.http
import coro.backdoor
# -----------------------------------------------------------------------
# --- change the location of the chain and key files on the next line ---
# -----------------------------------------------------------------------
server = coro.http.tlslite_server (
'cert/server.crt',
'cert/server.key',
)
server.push_handler (coro.http.handlers.coro_status_handler())
server.push_handler (coro.http.handlers.favicon_handler())
coro.spawn (server.start, ('0.0.0.0', 9443))
coro.spawn (coro.backdoor.serve, unix_path='/tmp/httpsd.bd')
coro.event_loop (30.0)
coro/http/handlers.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
import coro
import os
import re
import sys
import time
import zlib
from coro.http.http_date import build_http_date
W = sys.stderr.write
# these two aren't real handlers, they're more like templates
# to give you an idea how to write one.
class post_handler:
def match (self, request):
# override to do a better job of matching
return request._method == 'post'
def handle_request (self, request):
data = request.file.read()
W ('post handler, data=%r\n' % (data,))
request.done()
class put_handler:
def match (self, request):
# override to do a better job of matching
return request.method == 'put'
def handle_request (self, request):
fp = request.file
while 1:
line = fp.readline()
if not line:
W ('line: DONE!\n')
break
else:
W ('line: %r\n' % (line,))
request.done()
class coro_status_handler:
def match (self, request):
return request.path.split ('/')[1] == 'status'
def clean (self, s):
s = s.replace ('<','&lt;')
s = s.replace ('>','&gt;')
return s
def handle_request (self, request):
request['content-type'] = 'text/html; charset=utf-8'
request.set_deflate()
request.push (
'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" '
'"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">'
'<html xmlns="http://www.w3.org/1999/xhtml">\r\n'
)
request.push ('<head><title>status</title></head><body>\r\n')
request.push ('<p>Listening on\r\n')
request.push (repr (request.server.addr))
request.push ('</p>\r\n')
request.push ('<table border="1">\r\n')
all_threads = ( (x, coro.where(x)) for x in coro.all_threads.values() )
for thread, traceback in all_threads:
request.push ('<tr><td>%s\r\n' % self.clean (repr(thread)))
request.push ('<pre>\r\n')
# traceback format seems to have changed
for level in traceback[1:-1].split ('] ['):
[file, fun] = level.split (' ')
fun, line = fun.split ('|')
request.push ('<b>%20s</b>:%3d %s\r\n' % (self.clean (fun), int(line), self.clean (file)))
request.push ('</pre></td></tr>')
request.push ('</table>\r\n')
request.push ('<p><a href="status">Update</a></p>')
request.push ('</body></html>')
request.done()
class file_handler:
block_size = 16000
def __init__ (self, doc_root):
self.doc_root = doc_root
def match (self, request):
path = request.path
filename = os.path.join (self.doc_root, path[1:])
return os.path.exists (filename)
crack_if_modified_since = re.compile ('([^;]+)(; length=([0-9]+))?$', re.IGNORECASE)
def handle_request (self, request):
path = request.path
filename = os.path.join (self.doc_root, path[1:])
if request.method not in ('get', 'head'):
request.error (405)
return
if os.path.isdir (filename):
filename = os.path.join (filename, 'index.html')
if not os.path.isfile (filename):
request.error (404)
else:
stat_info = os.stat (filename)
mtime = stat_info[stat.ST_MTIME]
file_length = stat_info[stat.ST_SIZE]
ims = request['if-modified-since']
if ims:
length_match = 1
m = self.crack_if_modified_since.match (ims)
if m:
length = m.group (3)
if length:
if int(length) != file_length:
length_match = 0
ims_date = http_date.parse_http_date (m.group(1))
if length_match and ims_date:
if mtime <= ims_date:
request.error (304)
return
ftype, fencoding = mimetypes.guess_type (filename)
request['Content-Type'] = ftype or 'text/plain'
request['Last-Modified'] = build_http_date (mtime)
# Note: these are blocking file operations.
if request.method == 'get':
f = open (filename, 'rb')
block = f.read (self.block_size)
if not block:
request.error (204) # no content
else:
while 1:
request.push (block)
block = f.read (self.block_size)
if not block:
break
elif request.method == 'head':
pass
else:
# should be impossible
request.error (405)
sample = (
'AAABAAEAICAQAAEABADoAgAAFgAAACgAAAAgAAAAQAAAAAEABAAAAAAAAAAAAAAAAAAAAAAAAAAA'
'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
'AAAAAAAAAAAAAAD/+K///8AH//+iI///QAH//r4g//x3AH//Z6J//UABP/ovgD/458Ef+u+wv/Tn'
'0R/+79if9OXZH/6gCJ/2BwAf/u/8n/h33R/7Z7kf/ReQH/+qu7//BUW//7vrv//RR3//7r///80d'
'///pq///8EP//+rH///d9///6j///9Af/w=='
).decode ('base64')
zsample = zlib.compress (sample, 9)
last_modified = build_http_date (time.time())
class favicon_handler:
def __init__ (self, data=None):
if data is None:
self.data = zsample
else:
self.data = data
def match (self, request):
return request.path == '/favicon.ico'
def handle_request (self, request):
if request['if-modified-since']:
# if we cared, we could check that timestamp.
request.error (304)
else:
request['content-type'] = 'image/x-icon'
request['last-modified'] = last_modified
# are there browsers that don't accept deflate?
request['content-encoding'] = 'deflate'
request.push (self.data)
request.done()
coro/http/protocol.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
import coro
from coro import read_stream
W = coro.write_stderr
# candidate for sync.pyx?
class latch:
"Like a CV, except without the race - if the event has already fired then wait() will return immediately."
def __init__ (self):
self.cv = coro.condition_variable()
self.done = False
def wake_all (self, args=()):
self.done = True
self.args = args
return self.cv.wake_all (args)
def wait (self):
if not self.done:
return self.cv.wait()
else:
return self.args
class http_file:
"HTTP message content, as a file-like object."
buffer_size = 8000
def __init__ (self, headers, stream):
self.streami = stream
self.done_cv = latch()
if headers.get_one ('transfer-encoding') == 'chunked':
self.streamo = read_stream.buffered_stream (self._gen_read_chunked().next)
else:
content_length = headers.get_one ('content-length')
if content_length:
self.content_length = int (content_length)
self.streamo = read_stream.buffered_stream (self._gen_read_fixed().next)
else:
raise HTTP_Protocol_Error ("no way to determine length of HTTP data")
def _gen_read_chunked (self):
"generator: decodes chunked transfer-encoding."
s = self.streami
while 1:
chunk_size = int (s.read_line()[:-2], 16)
if chunk_size == 0:
self.done_cv.wake_all()
return
else:
remain = chunk_size
while remain:
ask = min (remain, self.buffer_size)
block = s.read_exact (ask)
assert (s.read_exact (2) == '\r\n')
remain -= ask
yield block
def _gen_read_fixed (self):
"generate fixed-size blocks of content."
s = self.streami
remain = self.content_length
while remain:
ask = min (remain, self.buffer_size)
block = s.read_exact (ask)
remain -= ask
yield block
self.done_cv.wake_all()
return
# XXX implement <size> argument
def read (self, join=True):
"read the entire contents. join=False returns a generator, join=True returns a string."
r = (x for x in self.streamo.read_all())
if join:
return ''.join (r)
else:
return r
def readline (self):
"read a newline-delimited line."
if self.done_cv.done:
return ''
else:
return self.streamo.read_until ('\n')
def wait (self):
"wait until all the content has been read."
self.done_cv.wait()
class header_set:
def __init__ (self, headers=()):
self.headers = {}
for h in headers:
self.crack (h)
def from_keywords (self, kwds):
"Populate this header set from a dictionary of keyword arguments (e.g., 'content_length' becomes 'content-length')"
r = []
for k, v in kwds.items():
k = k.replace ('_', '-')
self[k] = v
return self
def crack (self, h):
"Crack one header line."
# deliberately ignoring 822 crap like continuation lines.
try:
i = h.index (': ')
name, value = h[:i], h[i+2:]
self[name] = value
except ValueError:
coro.write_stderr ('dropping bogus header %r\n' % (h,))
pass
def get_one (self, key):
"Get the value of a header expected to have at most one value. If not present, return None. If more than one, raise ValueError."
r = self.headers.get (key, None)
if r is None:
return r
elif isinstance (r, list) and len (r) > 1:
raise ValueError ("expected only one %s header, got %r" % (key, r))
else:
return r[0]
def has_key (self, key):
"Is this header present?"
return self.headers.has_key (key.lower())
def __getitem__ (self, key):
"Returns the list of values for this header, or None."
return self.headers.get (key, None)
def __setitem__ (self, name, value):
"Add a value to the header <name>."
name = name.lower()
probe = self.headers.get (name)
if probe is None:
self.headers[name] = [value]
else:
probe.append (value)
def __str__ (self):
"Render the set of headers."
r = []
for k, vl in self.headers.iteritems():
for v in vl:
r.append ('%s: %s\r\n' % (k, v))
return ''.join (r)
coro/http/server.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
# history: this code traces all the way back to medusa, through egroups, then ironport, and into shrapnel.
# Very Rewritten in Feb 2012.
import coro
import errno
import http_date
import mimetypes
import os
import re
from coro import read_stream
import socket
import stat
import sys
import time
import zlib
from protocol import latch, http_file, header_set
W = sys.stderr.write
__version__ = '0.1'
class request_stream:
def __init__ (self, conn, stream):
self.timeout = conn.server.client_timeout
self.conn = conn
self.stream = stream
def get_request (self):
request_line = self.stream.read_line()
if not request_line:
raise StopIteration
else:
# read header
lines = []
while 1:
line = self.stream.read_line()
# XXX handle continuation lines
if line == '':
raise StopIteration
elif line == '\r\n':
break
else:
lines.append (line[:-2])
return http_request (self.conn, request_line[:-2], header_set (lines))
def gen_requests (self):
# read HTTP requests on this stream
while 1:
try:
request = coro.with_timeout (self.timeout, self.get_request)
except coro.TimeoutError:
return
else:
yield request
# can't read another request until we finish reading this one
# [it might have a body]
request.wait_until_read()
class connection:
def __init__ (self, server):
self.server = server
self.stream = None
def run (self, conn, peer):
self.conn = conn
self.peer = peer
self.stream = read_stream.sock_stream (self.conn)
try:
try:
for request in request_stream (self, self.stream).gen_requests():
if request.bad:
# bad request
request.error (400)
else:
try:
handler = self.pick_handler (request)
if handler:
# XXX with_timeout() ?
handler.handle_request (request)
else:
request.error (404)
request.wait_until_done()
except (coro.TimeoutError, coro.Interrupted):
raise
except:
tb = coro.compact_traceback()
request.error (500, tb)
self.server.log ('error: %r request=%r tb=%r' % (self.peer, request, tb))
except (OSError, coro.TimeoutError, coro.ClosedError):
pass
finally:
self.conn.close()
def pick_handler (self, request):
for handler in self.server.handlers:
if handler.match (request):
return handler
return None
def send (self, data):
return self.conn.send (data)
def close (self):
self.conn.close()
class http_request:
request_count = 0
# <path>;<params>?<query>#<fragment>
path_re = re.compile ('(/[^;?#]*)(;[^?#]*)?(\?[^#]*)?(#.*)?')
# <method> <uri> HTTP/<version>
request_re = re.compile ('([^ ]+) ([^ ]+) *(HTTP/([0-9.]+))?')
# shadowed instance variables
chunking = False
close = False
is_done = False
sent_headers = False
bad = False
body_done = False
file = None
def __init__ (self, client, request, headers):
self.reply_headers = header_set()
self.reply_code = 200
http_request.request_count = http_request.request_count + 1
self.request_number = http_request.request_count
self.request = request
self.request_headers = headers
#W ('headers=%s\n' % (headers,))
self.client = client
self.server = client.server
self.tstart = time.time()
self.peer = client.peer
self.output = buffered_output (self.client.conn)
self.done_cv = latch()
self.deflate = None
m = http_request.request_re.match (request)
if m:
(self.method, self.uri, ver, self.version) = m.groups()
self.method = self.method.lower()
if not self.version:
self.version = "0.9"
m = http_request.path_re.match (self.uri)
if m:
(self.path, self.params, self.query, self.frag) = m.groups()
else:
self.bad = True
else:
self.version = "1.0"
self.bad = True
if self.has_body():
self.file = http_file (headers, client.stream)
def wait_until_read (self):
"wait until this entire request body has been read"
if self.file:
self.file.done_cv.wait()
def wait_until_done (self):
"wait until this request is done (i.e, the response has been sent)"
if not self.is_done:
self.done_cv.wait()
def has_body (self):
if self.request_headers.has_key ('transfer-encoding'):
# 4.4 ignore any content-length
return True
else:
probe = self.request_headers.get_one ('content-length')
if probe:
try:
size = int (probe)
if size == 0:
return False
elif size > 0:
return True
else:
return False
except ValueError:
return False
def can_deflate (self):
acc_enc = self.request_headers.get_one ('accept-encoding')
if acc_enc:
for kind in acc_enc.split (','):
if kind.strip().lower() == 'deflate':
return True
return False
def set_deflate (self):
"set this request for on-the-fly compression (via zlib DEFLATE)"
if self.can_deflate():
self.deflate = zlib.compressobj()
self['content-encoding'] = 'deflate'
# http://zoompf.com/blog/2012/02/lose-the-wait-http-compression
# Note: chrome,firefox,safari,opera all handle the header. Not MSIE, sigh. Discard it.
assert (self.deflate.compress ('') == '\x78\x9c')
return self.deflate
def push (self, data, flush=False):
"push output data for this request. buffered, maybe chunked, maybe compressed"
if not self.sent_headers:
self.sent_headers = 1
self.output.write (self.get_headers())
if self.chunking:
self.output.set_chunk()
if self.deflate:
if data:
data = self.deflate.compress (data)
if flush:
data += self.deflate.flush()
if data:
self.output.write (data)
def done (self):
if self.is_done:
W ('done called twice?\n')
return
if not self.sent_headers:
self.push ('')
if self.deflate:
self.push ('', flush=True)
self.output.flush()
if self.close:
self.client.close()
self.is_done = True
self.client.server.log (self.log_line())
self.done_cv.wake_all()
# note: the difference of meaning between getitem/setitem
def __getitem__ (self, key):
# fetch a request header
# use this only when you expect at most one of this header.
return self.request_headers.get_one (key)
def __setitem__ (self, key, val):
# set a reply header
self.reply_headers[key] = val
def get_headers (self):
chunked = False
# here is were we decide things like keep-alive, 1.0 vs 1.1, chunking, etc.
hi = self.request_headers
ho = self.reply_headers
connection = hi.get_one('connection')
if connection:
connection_tokens = [ x.strip() for x in connection.split(',') ]
else:
connection_tokens = ()
close_it = False
if self.version == '1.1':
if 'close' in connection_tokens:
close_it = True
elif not ho.get_one ('content-length'):
ho['transfer-encoding'] = 'chunked'
chunked = True
elif self.version == '1.0':
if 'keep-alive' in connection_tokens:
if not ho.get_one ('content-length'):
close_it = True
else:
ho['connection'] = 'keep-alive'
else:
close_it = True
elif self.version == '0.9':
close_it = True
if close_it:
ho['connection'] = 'close'
self.chunking = chunked
ho['server'] = 'shrapnel httpd/%s' % __version__
ho['date'] = http_date.build_http_date (coro.now_usec / coro.microseconds)
return self.response (self.reply_code) + '\r\n' + str (self.reply_headers) + '\r\n'
def response (self, code=200):
message = self.responses[code]
self.reply_code = code
return 'HTTP/%s %d %s' % (self.version, code, message)
def error (self, code, reason=None):
self.reply_code = code
message = self.responses[code]
s = self.DEFAULT_ERROR_MESSAGE % {
'code': code, 'message': message, 'reason': reason
}
self['Content-Length'] = len(s)
self['Content-Type'] = 'text/html'
self.push (s)
self.done()
def log_line (self):
now = time.time()
# somewhere between common log format and squid, avoid the
# expense of formatting time
return '%.03f %s "%s" %d %d %0.2f' % (
now,
'%s:%d' % self.peer,
self.request,
self.reply_code,
self.output.sent,
now - self.tstart,
)
responses = {
100: "Continue",
101: "Switching Protocols",
200: "OK",
201: "Created",
202: "Accepted",
203: "Non-Authoritative Information",
204: "No Content",
205: "Reset Content",
206: "Partial Content",
300: "Multiple Choices",
301: "Moved Permanently",
302: "Moved Temporarily",
303: "See Other",
304: "Not Modified",
305: "Use Proxy",
400: "Bad Request",
401: "Unauthorized",
402: "Payment Required",
403: "Forbidden",
404: "Not Found",
405: "Method Not Allowed",
406: "Not Acceptable",
407: "Proxy Authentication Required",
408: "Request Time-out",
409: "Conflict",
410: "Gone",
411: "Length Required",
412: "Precondition Failed",
413: "Request Entity Too Large",
414: "Request-URI Too Large",
415: "Unsupported Media Type",
500: "Internal Server Error",
501: "Not Implemented",
502: "Bad Gateway",
503: "Service Unavailable",
504: "Gateway Time-out",
505: "HTTP Version not supported"
}
# Default error message
DEFAULT_ERROR_MESSAGE = '\r\n'.join ([
'<html>',
'<head>',
'<title>Error response</title>',
'</head>',
'<body>',
'<h1>Error response</h1>',
'<p>Error code %(code)d.',
'<p>Message: %(message)s.',
'<p>Reason: %(reason)s.',
'</body>',
'</html>',
''
])
class HTTP_Protocol_Error (Exception):
pass
# chunking works thus:
# <data>
# becomes:
# <hex-length><CRLF>
# <data><CRLF>
# when done, signal with
# 0<CRLF><CRLF>
class buffered_output:
"Buffer HTTP output data; handle the 'chunked' transfer-encoding"
def __init__ (self, conn, size=8000):
self.conn = conn
self.size = size
self.buffer = []
self.len = 0
self.sent = 0
self.chunk_index = -1
self.chunk_len = 0
# at this point *exactly*, we want to start chunking the output.
# this is called immediately after the headers are pushed.
def set_chunk (self):
"start chunking here, exactly."
self.chunk_index = len (self.buffer)
def get_data (self):
"get data to send. may chunk."
data, self.buffer = self.buffer, []
if self.chunk_index >= 0:
# chunkify (the post-header portion of) our output list
data.insert (self.chunk_index, '%x\r\n' % (self.chunk_len,))
data.append ('\r\n')
self.chunk_len = 0
self.chunk_index = 0
self.len = 0
return data
def write (self, data):
"Push data to the buffer. If the accumulated data goes over the buffer size, send it."
self.buffer.append (data)
self.len += len (data)
if self.chunk_index >= 0:
self.chunk_len += len (data)
if self.len >= self.size:
self.send (self.get_data())
def flush (self):
"Flush the data from this buffer."
data = self.get_data()
if self.chunk_index >= 0:
data.append ('0\r\n\r\n')
self.send (data)
def send (self, data):
try:
self.sent += self.conn.writev (data)
except AttributeError:
# underlying socket may not support writev (e.g., tlslite)
self.sent += self.conn.send (''.join (data))
class server:
client_timeout = 30
def __init__ (self):
self.handlers = []
self.shutdown_flag = 0
self.thread_id = None
self.addr = ()
self.sock = None
def log (self, line):
sys.stderr.write ('http %s:%d: %s\n' % (self.addr[0], self.addr[1], line))
def push_handler (self, handler):
self.handlers.append (handler)
def start (self, addr, retries=5):
"""Start the web server listening on addr in a new coroutine.
Try up to <retries> time to bind to that address.
Raises an exception if the bind fails."""
self.sock = coro.tcp_sock()
self.sock.set_reuse_addr()
done = 0
save_errno = 0
self.addr = addr
while not done:
for x in xrange (retries):
try:
self.sock.bind (addr)
except OSError, why:
if why.errno not in (errno.EADDRNOTAVAIL, errno.EADDRINUSE):
raise
else:
save_errno = 0
if why.errno == errno.EADDRINUSE:
was_eaddrinuse = 1
else:
done = 1
break
coro.sleep_relative (1)
else:
self.log ('cannot bind to %s:%d after 5 attempts, errno = %d' % (addr[0], addr[1], save_errno))
coro.sleep_relative (15)
self.sock.listen (1024)
c = coro.spawn (self.run)
c.set_name ('http_server (%s:%d)' % addr)
def run (self):
self.thread_id = coro.current().thread_id()
while not self.shutdown_flag:
try:
conn, addr = self.accept()
client = connection (self)
c = coro.spawn (client.run, conn, addr)
c.set_name ('http connection on %r' % (addr,))
except coro.Shutdown:
break
except:
self.log ('error: %r\n' % (coro.compact_traceback(),))
coro.sleep_relative (0.25)
continue
self.sock.close()
def accept (self):
return self.sock.accept()
def shutdown (self):
self.shutdown_flag = 1
try:
# XXX SMR is this really necessary?
thread = coro.get_thread_by_id (self.thread_id)
thread.shutdown()
except KeyError:
return # already exited
class tlslite_server (server):
"https server using the tlslite package"
def __init__ (self, cert_path, key_path):
server.__init__ (self)
self.cert_path = cert_path
self.key_path = key_path
self.read_chain()
self.read_private()
def accept (self):
import tlslite
conn0, addr = server.accept (self)
conn = tlslite.TLSConnection (conn0)
conn.handshakeServer (certChain=self.chain, privateKey=self.private)
return conn, addr
def read_chain (self):
"cert chain is all in one file, in LEAF -> ROOT order"
import tlslite
delim = '-----END CERTIFICATE-----\n'
data = open (self.cert_path).read()
certs = data.split (delim)
chain = []
for cert in certs:
if cert:
x = tlslite.X509()
x.parse (cert + delim)
chain.append (x)
self.chain = tlslite.X509CertChain (chain)
def read_private (self):
import tlslite
self.private = tlslite.parsePEMKey (
open (self.key_path).read(),
private=True
)
coro/http/session_handler.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
import coro
import time
import uuid
import sys
W = sys.stderr.write
# See: http://en.wikipedia.org/wiki/HTTP_cookie#Session_cookie
def extract_session (cookie):
parts = cookie.split (';')
for part in parts:
pair = [x.strip() for x in part.split ('=')]
if len (pair) == 2:
if pair[0] == 'session':
return pair[1]
return None
class session_handler:
def __init__ (self, name, function):
self.name = name
self.function = function
self.sessions = {}
def match (self, request):
path = request.path.split ('/')
if (len(path) > 1) and (path[1] == self.name):
return 1
else:
return 0
def find_session (self, request):
# XXX does http allow more than one cookie header?
cookie = request['cookie']
if cookie:
sid = extract_session (cookie)
return sid, self.sessions.get (sid, None)
else:
return None, None
def gen_session_id (self):
return str (uuid.uuid4())
def handle_request (self, request):
sid, fifo = self.find_session (request)
if fifo is None:
# login
fifo = coro.fifo()
fifo.push (request)
sid = self.gen_session_id()
request['set-cookie'] = 'session=%s' % (sid,)
self.sessions[sid] = fifo
coro.spawn (self.wrap, sid, fifo)
else:
fifo.push (request)
def wrap (self, sid, fifo):
try:
self.function (sid, fifo)
finally:
del self.sessions[sid]
coro/linux_poller.pyx
View file @ e921e61e
......@@ -67,7 +67,7 @@ cdef extern from "sys/epoll.h":
int EPOLLHUP, EPOLLPRI, EPOLLERR, EPOLLET, EPOLLONESHOT, EPOLLRDHUP
int EPOLLIN, EPOLLOUT
cdef struct fake_epoll_event:
cdef struct shrapnel_epoll_event:
uint32_t events
epoll_data_t data
int op
......@@ -75,6 +75,7 @@ cdef struct fake_epoll_event:
int err
cdef int SECS_TO_MILLISECS = 1000
cdef double NSECS_TO_MILLISECS = 1000000.0
class EV:
......@@ -105,31 +106,6 @@ cdef enum:
EVENT_STATUS_FIRED
EVENT_STATUS_ABORTED
# - ONESHOT: Used to indicate that this target should be removed from the
# event map after it fires.
# - CLOSED: For file-descriptor events, this indicates that the file
# descriptor has been closed, and that the event key has been
# removed from the event map.
cdef enum:
TARGET_FLAG_ONESHOT = 1
TARGET_CLOSED = 2
cdef class event_target:
cdef public int status
cdef public int index
cdef public object target
cdef public int flags
def __cinit__ (self, target, int index):
self.status = EVENT_STATUS_NEW
self.index = index
self.target = target
self.flags = 0
def __repr__ (self):
return '<event_target status=%r index=%r target=%r flags=%r>' % (self.status,
self.index, self.target, self.flags)
cdef class py_event:
"""Representation of a epoll event.
......@@ -146,7 +122,7 @@ cdef class py_event:
# cinit cannot take a C struct.
# It would be nice to support fast-constructor semantics in Cython.
cdef __fast_init__ (self, fake_epoll_event *e):
cdef __fast_init__ (self, shrapnel_epoll_event *e):
self.events = e.events
self.data.fd = e.data.fd
self.flags = e.flags
......@@ -196,20 +172,10 @@ cdef class event_key:
cdef public class queue_poller [ object queue_poller_object, type queue_poller_type ]:
cdef fake_epoll_event * change_list
cdef int change_list_index
cdef int ep_fd
cdef object event_map
def __cinit__ (self):
# XXX EVENT_SCALE should be a parameter.
self.change_list = <fake_epoll_event *>PyMem_Malloc (sizeof (fake_epoll_event) * EVENT_SCALE)
def __dealloc__ (self):
PyMem_Free (self.change_list)
def __init__ (self):
self.change_list_index = 0
self.ep_fd = -1
self.event_map = {}
......@@ -224,28 +190,60 @@ cdef public class queue_poller [ object queue_poller_object, type queue_poller_t
self.ep_fd = -1
cdef object set_wait_for (self, event_key ek):
cdef fake_epoll_event *e
cdef coro me
cdef event_target et
if self.change_list_index < EVENT_SCALE:
if PyDict_Contains (self.event_map, ek):
# Should be impossible to have KeyError due to previous line.
et = self.event_map[ek]
raise SimultaneousError (the_scheduler._current, et.target, ek)
else:
me = the_scheduler._current
target = event_target (me, self.change_list_index)
target.flags = TARGET_FLAG_ONESHOT
self.event_map[ek] = target
e = &(self.change_list[self.change_list_index])
e.data.fd = ek.fd
e.op = EPOLL_CTL_MOD
e.events = ek.events
e.flags = EPOLLONESHOT
self.change_list_index = self.change_list_index + 1
return target
cdef unsigned flag = 0
if PyDict_Contains (self.event_map, ek):
# Should be impossible to have KeyError due to previous line.
et = self.event_map[ek]
raise SimultaneousError (the_scheduler._current, et.target, ek)
else:
raise SystemError, "too many events in change_list"
ek1 = event_key (EPOLLOUT, ek.fd)
ek2 = event_key (EPOLLIN, ek.fd)
if ( (PyDict_Contains (self.event_map, ek2) and ek.events == EPOLLOUT) or
(PyDict_Contains (self.event_map, ek1) and ek.events == EPOLLIN)):
flags = EPOLLOUT | EPOLLIN | EPOLLET
else:
flags = EPOLLET
me = the_scheduler._current
target = me
self.event_map[ek] = target
self._register_event(ek, flags)
return target
def set_handler (self, object event, object handler, int flags=0, unsigned int fflags=0):
return
cdef notify_of_close (self, int fd):
return
cdef _register_event(self, event_key ek, unsigned int flags):
cdef int r
cdef epoll_event org_e
org_e.data.fd = ek.fd
org_e.events = ek.events | flags
r = epoll_ctl (
self.ep_fd,
EPOLL_CTL_MOD,
org_e.data.fd,
&org_e
)
# if fd doesn't exist in epoll, add it
if r == -1 and (libc.errno == libc.ENOENT):
r = epoll_ctl (
self.ep_fd,
EPOLL_CTL_ADD,
org_e.data.fd,
&org_e
)
if r == -1 and (libc.errno != libc.EEXIST):
raise_oserror()
cdef _wait_for_with_eof (self, int fd, int events):
cdef py_event event
......@@ -261,44 +259,11 @@ cdef public class queue_poller [ object queue_poller_object, type queue_poller_t
cdef _wait_for_write (self, int fd):
return self._wait_for_with_eof(fd, EPOLLOUT)
def wait_for_read (self, int fd):
return self._wait_for_with_eof(fd, EPOLLIN)
def wait_for_write (self, int fd):
return self._wait_for_with_eof(fd, EPOLLOUT)
cdef py_event _wait_for (self, int fd, int events):
cdef event_target et
cdef fake_epoll_event *e
cdef event_key ek
ek = event_key (events, fd)
et = self.set_wait_for (ek)
try:
return _YIELD()
finally:
if et.status == EVENT_STATUS_NEW:
# still in the change list
e = &self.change_list[et.index]
# event() will ignore this entry
e.events = 0
e.data.fd = 0
et.status = EVENT_STATUS_ABORTED
if not et.flags & TARGET_CLOSED:
# remove from event map
del self.event_map[ek]
#W ('wait_for() cleanup: (%d, %d) STATUS_NEW\n' % (ident, filter))
elif et.status == EVENT_STATUS_SUBMITTED:
# event submitted, delete it.
et.status = EVENT_STATUS_ABORTED
if not et.flags & TARGET_CLOSED:
self.delete_event (fd, events)
# remove from event map
del self.event_map[ek]
#W ('wait_for() cleanup: (%d, %d) STATUS_SUBMITTED\n' % (ident, filter))
elif et.status == EVENT_STATUS_FIRED:
# event already fired! do nothing.
#W ('wait_for() cleanup: (%d, %d) STATUS_FIRED\n' % (ident, filter))
pass
return _YIELD()
def wait_for (self, int fd, int events):
"""Wait for an event.
......@@ -325,184 +290,53 @@ cdef public class queue_poller [ object queue_poller_object, type queue_poller_t
#if r < 0:
# raise_oserror()
def set_handler (self, object event, object handler, int flags=EPOLLONESHOT, int op=EPOLL_CTL_MOD):
"""Add a event handler.
This is a low-level interface to register a event handler.
:Parameters:
- `event`: A tuple of ``(ident, filter)`` of the event to handle.
- `handler`: The handler to use, a callable object which will be
called with one argument, a `py_event` object.
- `flags`: Kevent flags to use. Defaults to ``EV_ADD|EV_ONESHOT``.
:Exceptions:
- `SimultaneousError`: There is already a handler set for this
event.
"""
cdef int events
cdef int fd
cdef fake_epoll_event * e
cdef event_key ek
assert callable(handler)
fd = PySequence_GetItem(event, 0)
events = PySequence_GetItem(event, 1)
#events |= flags
ek = event_key (events, fd)
if PyDict_Contains (self.event_map, ek):
# Should be impossible to have KeyError due to previous line.
et = self.event_map[ek]
raise SimultaneousError (the_scheduler._current, et.target, ek)
else:
if self.change_list_index < EVENT_SCALE:
e = &(self.change_list[self.change_list_index])
e.data.fd = fd
e.events = events
e.op = op
e.flags = flags
self.change_list_index = self.change_list_index + 1
et = event_target (handler, self.change_list_index)
self.event_map[ek] = et
else:
raise SystemError, "too many events in change_list"
cdef set_event_target (self, object event, event_target et):
cdef int filter
cdef int fd
cdef event_key ek
fd = PySequence_GetItem(event, 0)
filter = PySequence_GetItem(event, 1)
ek = event_key (filter, fd)
self.event_map[ek] = et
cdef notify_of_close (self, int fd):
cdef event_target et
cdef coro co
cdef event_key ek
cdef epoll_event e
ek = event_key (EPOLLIN, fd)
if PyDict_Contains(self.event_map, ek):
et = self.event_map[ek]
et.flags = et.flags | TARGET_CLOSED
del self.event_map[ek]
W ('(notify_of_close (%d) [read])\n' % (fd,))
co = et.target
try:
co.__interrupt (ClosedError(the_scheduler._current))
except ScheduleError:
W ('notify_of_close (%d) [read]: unable to interrupt thread: %r\n' % (fd, co))
ek = event_key (EPOLLOUT, fd)
if PyDict_Contains(self.event_map, ek):
et = self.event_map[ek]
et.flags = et.flags | TARGET_CLOSED
del self.event_map[ek]
W ('(notify_of_close (%d) [write])\n' % (fd,))
co = et.target
try:
co.__interrupt (ClosedError(the_scheduler._current))
except ScheduleError:
W ('notify_of_close (%d) [write]: unable to interrupt thread: %r\n' % (fd, co))
def poll (self, timeout=(30,0), int nevents=2000):
cdef timespec ts
cdef int r, i
cdef epoll_event * events, ee
cdef fake_epoll_event *e, new_e
cdef epoll_event org_e
cdef epoll_event * events
cdef shrapnel_epoll_event new_e
cdef coro co
cdef event_target et
cdef event_key ek
cdef py_event _py_event
ts.tv_sec, ts.tv_nsec = timeout
# poll() is only called from <main>, so alloca() is OK.
events = <epoll_event *> libc.alloca (sizeof (epoll_event) * nevents)
for i from 0 <= i < self.change_list_index:
e = &(self.change_list[i])
org_e.events = e.events | e.flags
org_e.data.fd = e.data.fd
if e.events != 0:
r = epoll_ctl (
self.ep_fd,
e.op,
org_e.data.fd,
&org_e
)
if r == -1 and (libc.errno == libc.ENOENT):
r = epoll_ctl (
self.ep_fd,
EPOLL_CTL_ADD,
org_e.data.fd,
&org_e
)
#print 'epoll_ctl event >>>>>> %s for %s' % (org_e.events, org_e.data.fd)
if r == -1 and (libc.errno != libc.EEXIST):
raise_oserror()
r = epoll_wait (self.ep_fd, events, nevents, timeout[0] * SECS_TO_MILLISECS)
if the_scheduler.profiling:
the_profiler.charge_wait()
if r == -1:
raise_oserror()
else:
for i from 0 <= i < self.change_list_index:
e = &(self.change_list[i])
# We mark events with a filter of 0 when we want them ignored
# (see EVENT_STATUS_NEW in _wait_for).
if e.events != 0:
ek = event_key (e.events, e.data.fd)
try:
et = self.event_map[ek]
except KeyError:
# This should never happen.
P('Missing event from dictionary for events=%r fd=%r (this should never happen).' % (
e.events, e.data.fd))
else:
et.status = EVENT_STATUS_SUBMITTED
r = epoll_wait (self.ep_fd, events, nevents, timeout[0] * SECS_TO_MILLISECS + (timeout[1] / NSECS_TO_MILLISECS))
#W ('{%d}' % (r,))
#P('mapsize = %i' % len(self.event_map))
for i from 0 <= i < r:
new_e.data.fd = events[i].data.fd
new_e.events = events[i].events
new_e.err = 0
#print 'epoll_wait event >>>>>> %s for %s' % (new_e.events, new_e.data.fd)
tmp = (EPOLLIN, EPOLLOUT)
if events[i].events & tmp[0] and events[i].events & tmp[1]:
pass
else:
tmp = [0]
for j from 0 <= j < len(tmp):
if len(tmp) == 2:
if events[i].events & tmp[j]:
new_e.events = events[i].events & ~(tmp[j])
self.change_list_index = 0
#W ('{%d}' % (r,))
#P('mapsize = %i' % len(self.event_map))
for i from 0 <= i < r:
new_e.data.fd = events[i].data.fd
new_e.events = events[i].events
new_e.err = 0
#print 'epoll_wait event >>>>>> %s for %s' % (new_e.events, new_e.data.fd)
if new_e.events & EPOLLERR or new_e.events & EPOLLHUP:
#print 'epoll_wait event >>>>>> %s for %s' % (new_e.events, new_e.data.fd)
new_e.events = new_e.events & ~(EPOLLHUP)
new_e.events = new_e.events & ~(EPOLLERR)
new_e.err = 104
# epoll doesn't specify the last event we had registered so make a guess
if new_e.events == 0:
new_e.events = EPOLLIN
try:
et = self.event_map[event_key(EPOLLIN, new_e.data.fd)]
except KeyError:
new_e.events = EPOLLOUT
_py_event = py_event()
_py_event.__fast_init__(&new_e)
ek = event_key (new_e.events, new_e.data.fd)
try:
et = self.event_map[ek]
co = self.event_map[ek]
except KeyError:
W ('un-handled event: fd=%s events=%s\n' % (new_e.data.fd, new_e.events))
pass
#W ('un-handled event: fd=%s events=%s\n' % (new_e.data.fd, new_e.events))
else:
assert et.status != EVENT_STATUS_ABORTED
try:
et.status = EVENT_STATUS_FIRED
if isinstance (et.target, coro):
co = et.target
co._schedule (_py_event)
else:
# assumes kt.target is a callable object
_spawn(et.target, (_py_event,), {})
finally:
if et.flags & TARGET_FLAG_ONESHOT:
del self.event_map[ek]
if isinstance (co, coro):
co._schedule (_py_event)
else:
# assumes kt.target is a callable object
_spawn(co, (_py_event,), {})
del self.event_map[ek]
coro/profiler.py
View file @ e921e61e
......@@ -25,17 +25,18 @@
Introduction
============
This profiler is coro-aware. It produces output to a binary file on disk. You
then use the `coro.print_profile` module to convert it to an HTML file.
then use the :mod:`coro.print_profile` module to convert it to an HTML file.
Using The Profiler
==================
There are two ways to run the profiler. One is to use the `go` function where
you give it a python function to run. Profiling will start and call the
function, and then the profiler will automatically stop when the function
exits.
There are two ways to run the profiler. One is to use the
:func:`coro.profiler.go` function where you give it a python function to run.
Profiling will start and call the function, and then the profiler will
automatically stop when the function exits.
The other method is to call `start` to start the profiler and `stop` when you
want to stop profiling. This can be conveniently done from the backdoor.
The other method is to call :func:`coro.profiler.start` to start the profiler
and :func:`coro.profiler.stop` when you want to stop profiling. This can be
conveniently done from the backdoor.
Rendering Output
================
......@@ -49,12 +50,13 @@ Then view the profile output in your web browser.
Profiler Types
==============
The profiler supports different ways of gathering statistics. This is done by
specifying the "bench" object to use (see `go` and `start`). They default to
the "rusage" method of gathering statistics about every function call (see the
getrusage man page for more detail). If you want a higher performance profile,
you can use the `coro.bench` object instead which simply records TSC values for
every function call. If you want to define your own method of gathering
statistics, subclass `coro.bench` and implement your own techniques.
specifying the "bench" object to use (see :func:`go` and :func:`start`). They
default to the "rusage" method of gathering statistics about every function
call (see the getrusage man page for more detail). If you want a higher
performance profile, you can use the :class:`coro.bench` object instead which
simply records TSC values for every function call. If you want to define your
own method of gathering statistics, subclass :class:`coro.bench` and implement
your own techniques.
"""
......@@ -98,14 +100,12 @@ def go (fun, *args, **kwargs):
This will display the results to stdout after the function is finished.
:Parameters:
- `fun`: The function to call.
:param fun: The function to call.
:Keywords:
- `profile_filename`: The name of the file to save the profile data.
:keyword profile_filename: The name of the file to save the profile data.
Defaults to '/tmp/coro_profile.bin'.
- `profile_bench`: The bench object type to use. Defaults to
`coro.rusage_bench`.
:keyword profile_bench: The bench object type to use. Defaults to
:class:`coro.rusage_bench`.
"""
if kwargs.has_key('profile_filename'):
profile_filename = kwargs['profile_filename']
......
coro/read_stream.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
class socket_producer:
def __init__ (self, conn, buffer_size=8000):
self.conn = conn
self.buffer_size = buffer_size
def next (self):
return self.conn.recv (self.buffer_size)
def sock_stream (sock):
return buffered_stream (socket_producer (sock).next)
class buffered_stream:
def __init__ (self, producer):
self.producer = producer
self.buffer = ''
def gen_read_until (self, delim):
"generate pieces of input up to and including <delim>, then StopIteration"
ld = len(delim)
m = 0
while 1:
if not self.buffer:
self.buffer = self.producer()
if not self.buffer:
# eof
yield ''
return
i = 0
while i < len (self.buffer):
if self.buffer[i] == delim[m]:
m += 1
if m == ld:
result, self.buffer = self.buffer[:i+1], self.buffer[i+1:]
yield result
return
else:
m = 0
i += 1
block, self.buffer = self.buffer, ''
yield block
def gen_read_until_dfa (self, dfa):
"generate pieces of input up to and including a match on <dfa>, then StopIteration"
m = 0
while 1:
if not self.buffer:
self.buffer = self.producer()
if not self.buffer:
# eof
yield ''
return
i = 0
while i < len (self.buffer):
if dfa.consume (self.buffer[i]):
result, self.buffer = self.buffer[:i+1], self.buffer[i+1:]
yield result
return
i += 1
block, self.buffer = self.buffer, ''
yield block
def gen_read_exact (self, size):
"generate pieces of input up to <size> bytes, then StopIteration"
remain = size
while remain:
if len (self.buffer) >= remain:
result, self.buffer = self.buffer[:remain], self.buffer[remain:]
yield result
return
else:
piece, self.buffer = self.buffer, self.producer()
remain -= len (piece)
yield piece
if not self.buffer:
# eof
yield ''
return
def read_until (self, delim, join=True):
"read until <delim>. return a list of parts unless <join> is True"
result = ( x for x in self.gen_read_until (delim) )
if join:
return ''.join (result)
else:
return result
def read_exact (self, size, join=True):
"read exactly <size> bytes. return a list of parts unless <join> is True"
result = ( x for x in self.gen_read_exact (size) )
if join:
return ''.join (result)
else:
return result
def flush (self):
"flush this stream's buffer"
result, self.buffer = self.buffer, ''
return result
def read_line (self, delim='\r\n'):
"read a CRLF-delimited line from this stream"
return self.read_until (delim)
def read_all (self):
"read from self.producer until the stream terminates"
if self.buffer:
yield self.flush()
while 1:
block = self.producer()
if not block:
return
else:
yield block
coro/socket.pyx
View file @ e921e61e
......@@ -207,20 +207,19 @@ cdef _readv_compute(size_list, buffer_tuple, int n, int received, iovec * iov,
into consideration what has been received so far will create an iovec array
for the readv function.
:Parameters:
- `size_list`: A Python object that should be a sequence of integers
:param size_list: A Python object that should be a sequence of integers
that indicate which blocks are being requested.
- `buffer_tuple`: A tuple of Python strings (should be already
:param buffer_tuple: A tuple of Python strings (should be already
allocated and should be the same length as size_list).
- `n`: The length of size_list and buffer_tuple.
- `received`: The number of bytes received so far.
- `iov`: The ``iovec`` array. This should have `n` elements.
- `left`: OUTPUT: The number of bytes left to read.
- `iov_pos`: OUTPUT: The number of elements added to ``iov``.
- `complete_index`: The index of the last element in the buffer tuple
:param n: The length of size_list and buffer_tuple.
:param received: The number of bytes received so far.
:param iov: The ``iovec`` array. This should have ``n`` elements.
:param left: OUTPUT: The number of bytes left to read.
:param iov_pos: OUTPUT: The number of elements added to ``iov``.
:param complete_index: The index of the last element in the buffer tuple
that has been *completely* received. -1 if nothing has been
completely received.
- `partial_index`: The index of the element in the buffer tuple that
:param partial_index: The index of the element in the buffer tuple that
has partially received some data. -1 if none of the elements have
partial data.
"""
......@@ -254,7 +253,8 @@ cdef _readv_compute(size_list, buffer_tuple, int n, int received, iovec * iov,
cdef public class sock [ object sock_object, type sock_type ]:
"""Coro socket object.
"""
Coro socket object.
This is typically used for network sockets, but can also be used for
coro-safe IO on any file descriptor that supports kqueue non-blocking
......@@ -262,20 +262,19 @@ cdef public class sock [ object sock_object, type sock_type ]:
The constructor takes the following parameters:
- `domain`: The socket domain family, defaults to AF_INET (see `AF`).
- `stype`: The socket type, defaults to SOCK_STREAM (see `SOCK`).
- `protocol`: The socket protocol (normally not used, defaults to 0).
- `fd`: The file descriptor to use. Creates a new socket file
:param domain: The socket domain family, defaults to AF_INET (see :class:`AF`).
:param stype: The socket type, defaults to SOCK_STREAM (see :class:`SOCK`).
:param protocol: The socket protocol (normally not used, defaults to 0).
:param fd: The file descriptor to use. Creates a new socket file
descriptor if not specified.
:IVariables:
- `fd`: The file descriptor number. Set to -1 when the socket is
:ivar fd: The file descriptor number. Set to -1 when the socket is
closed.
- `orig_fd`: The original file descriptor number. This is left for
:ivar orig_fd: The original file descriptor number. This is left for
debugging purposes to determine which file descriptor was in use
before the socket was closed.
- `domain`: The socket domain (AF_INET, AF_UNIX, AF_UNSPEC).
- `stype`: The socket type (SOCK_STREAM, SOCK_DGRAM)
:ivar domain: The socket domain (AF_INET, AF_UNIX, AF_UNSPEC).
:ivar stype: The socket type (SOCK_STREAM, SOCK_DGRAM)
"""
cdef public int fd, orig_fd, domain, stype
......@@ -323,8 +322,7 @@ cdef public class sock [ object sock_object, type sock_type ]:
def get_fileno (self):
"""Get the current file descriptor.
:Return:
Returns the current file descriptor number. Returns -1 if the
:returns: The current file descriptor number. Returns -1 if the
socket is closed.
"""
warnings.warn('socket.get_fileno() is deprecated, use fileno() instead.', DeprecationWarning)
......@@ -333,18 +331,17 @@ cdef public class sock [ object sock_object, type sock_type ]:
def fileno (self):
"""Get the current file descriptor.
:Return:
Returns the current file descriptor number. Returns -1 if the
:returns: The current file descriptor number. Returns -1 if the
socket is closed.
"""
return self.fd
cdef _set_reuse_addr (self):
cdef int old
cdef int old = 0
cdef socklen_t optlen
optlen = sizeof (old);
optlen = sizeof (old)
getsockopt (self.fd, SOL_SOCKET, SO_REUSEADDR, <void*> &old, &optlen)
old = old | 1;
old = old | 1
setsockopt (self.fd, SOL_SOCKET, SO_REUSEADDR, <void*> &old, optlen)
def set_reuse_addr (self):
......@@ -362,19 +359,16 @@ cdef public class sock [ object sock_object, type sock_type ]:
def getsockopt (self, int level, int optname, socklen_t buflen=0):
"""Get a socket option.
:Parameters:
- `level`: The socket level to get (see `SOL`).
- `optname`: The socket option to get (see `SO`).
- `buflen`: The size of the buffer needed to retrieve the value. If
:param level: The socket level to get (see :class:`SOL`).
:param optname: The socket option to get (see :class:`SO`).
:param buflen: The size of the buffer needed to retrieve the value. If
not specified, it assumes the result is an integer and will
return an integer. Otherwise, it will create a new string with
the result, and you may use the struct module to decode it.
:Return:
Returns an integer if `buflen` is zero, otherwise returns a string.
:returns: An integer if ``buflen`` is zero, otherwise returns a string.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef int flag, r
cdef socklen_t flagsize
......@@ -397,13 +391,11 @@ cdef public class sock [ object sock_object, type sock_type ]:
def setsockopt (self, int level, int optname, value):
"""Set a socket option.
:Parameters:
- `level`: The socket level to set (see `SOL`).
- `optname`: The socket option to set (see `SO`).
- `value`: The value to set. May be an integer, or a struct-packed string.
:param level: The socket level to set (see :class:`SOL`).
:param optname: The socket option to set (see :class:`SO`).
:param value: The value to set. May be an integer, or a struct-packed string.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef int flag, r
cdef socklen_t optlen
......@@ -421,8 +413,7 @@ cdef public class sock [ object sock_object, type sock_type ]:
It is safe to call this if the socket is already closed.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef int r
if self.fd != -1:
......@@ -442,15 +433,12 @@ cdef public class sock [ object sock_object, type sock_type ]:
This will repeatedly call write to ensure all data has been sent. This
will raise OSError if it is unable to send all data.
:Parameters:
- `data`: The data to send.
:param data: The data to send.
:Return:
Returns the number of bytes sent, which should always be the length
of `data`.
:returns: The number of bytes sent, which should always be the length
of ``data``.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef char * buffer
cdef int r, left, sent
......@@ -482,35 +470,32 @@ cdef public class sock [ object sock_object, type sock_type ]:
def sendall(self, data):
"""Send all data.
This is an alias for the `send` method.
This is an alias for the :meth:`send` method.
"""
return self.send(data)
def write (self, data):
"""Write data.
This is an alias for the `send` method.
This is an alias for the :meth:`send` method.
"""
return self.send(data)
def sendto (self, data, address, int flags=0):
"""Send data to a specific address.
:Parameters:
- `data`: The data to send.
- `address`: The address to send to. For unix-domain sockets, this
:param data: The data to send.
:param address: The address to send to. For unix-domain sockets, this
is a string. For IP sockets, this is a tuple ``(IP, port)``
where IP is a string.
Port is always an integer.
- `flags`: sendto flags to use (defaults to 0) (see sendto(2)
:param flags: sendto flags to use (defaults to 0) (see sendto(2)
manpage).
:Return:
Returns the number of bytes sent which may be less than the send
:returns: The number of bytes sent which may be less than the send
requested.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef char * buffer
cdef sockaddr_storage sa
......@@ -536,23 +521,19 @@ cdef public class sock [ object sock_object, type sock_type ]:
else:
return r
# - `flags`: recv flags to use (defaults to 0) (see recv(2) manpage).
def recv (self, int buffer_size):
"""Receive data.
This may return less data than you request if the socket buffer is not
large enough. Use `recv_exact` to ensure you receive exactly the
large enough. Use :meth:`recv_exact` to ensure you receive exactly the
amount requested.
:Parameters:
- `buffer_size`: The number of bytes to receive.
:param buffer_size: The number of bytes to receive.
:Return:
Returns a string of data. Returns the empty string when the end of
:returns: A string of data. Returns the empty string when the end of
the stream is reached.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef buffer
cdef int r, new_buffer_size
......@@ -584,7 +565,7 @@ cdef public class sock [ object sock_object, type sock_type ]:
def read (self, buffer_size):
"""Read data.
This is an alias for the `recv` method.
This is an alias for the :meth:`recv` method.
"""
return self.recv(buffer_size)
......@@ -594,20 +575,17 @@ cdef public class sock [ object sock_object, type sock_type ]:
This may return less data than you request if the socket buffer is not
large enough.
:Parameters:
- `buffer_size`: The number of bytes to receive.
- `flags`: Socket flags to set (defaults to 0) (see recvfrom(2)
:param buffer_size: The number of bytes to receive.
:param flags: Socket flags to set (defaults to 0) (see recvfrom(2)
manpage).
:Return:
Returns a tuple ``(data, address)`` where data is a string and
:returns: A tuple ``(data, address)`` where data is a string and
address is the address of the remote side (string for unix-domain,
tuple of ``(IP, port)`` for IP where IP is a string and port is an
integer). Data is the empty string when the end of the stream is
reached.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef buffer
cdef sockaddr_storage sa
......@@ -654,15 +632,12 @@ cdef public class sock [ object sock_object, type sock_type ]:
This will repeatedly call read until all data is received.
:Parameters:
- `bytes`: The number of bytes to receive.
:param bytes: The number of bytes to receive.
:Return:
Returns the data as a string.
:returns: The data as a string.
:Exceptions:
- `OSError`: OS-level error.
- `EOFError`: Not all data could be read. The first argument
:raises OSError: OS-level error.
:raises EOFError: Not all data could be read. The first argument
includes any partial data read as a string.
"""
cdef char * p, * p0
......@@ -696,16 +671,13 @@ cdef public class sock [ object sock_object, type sock_type ]:
of the stream is reached before all data is received, then the result
tuple will only contain the elements competely or partially received.
:Parameters:
- `size_list`: A sequence of integers that indicates the buffer
:param size_list: A sequence of integers that indicates the buffer
sizes to read.
:Return:
Returns a tuple of strings corresponding to the sizes requested in
`size_list`.
:returns: A tuple of strings corresponding to the sizes requested in
``size_list``.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef int n, i
cdef int iov_pos
......@@ -791,15 +763,12 @@ cdef public class sock [ object sock_object, type sock_type ]:
This will repeatedly call writev until all data is sent. If it is
unable to send all data, it will raise an OSError exception.
:Parameters:
- `data`: A sequence of strings to write.
:param data: A sequence of strings to write.
:Return:
Returns the number of bytes sent which should always be the sum of
:returns: The number of bytes sent which should always be the sum of
the lengths of all the strings in the data sequence.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef char * buffer
cdef int r, left, size, sent
......@@ -867,20 +836,17 @@ cdef public class sock [ object sock_object, type sock_type ]:
This is for the Python buffer interface. If you don't know what that
is, move along. This method is for Python socket compatibility.
:Parameters:
- `buffer`: A writeable Python buffer object. Must be a contiguous
:param buffer: A writeable Python buffer object. Must be a contiguous
segment.
- `nbytes`: Number of bytes to read. Must be less than or equal to
:param nbytes: Number of bytes to read. Must be less than or equal to
the size of the buffer. Defaults to 0 which means the size of
`buffer`.
- `flags`: Flags for the recv system call (see recv(2) manpage).
``buffer``.
:param flags: Flags for the recv system call (see recv(2) manpage).
Defaults to 0.
:Return:
Returns the number of bytes read.
:returns: The number of bytes read.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef void *cbuf
cdef Py_ssize_t cbuflen
......@@ -917,22 +883,19 @@ cdef public class sock [ object sock_object, type sock_type ]:
This is for the Python buffer interface. If you don't know what that
is, move along. This method is for Python socket compatibility.
:Parameters:
- `buffer`: A writeable Python buffer object. Must be a contiguous
:param buffer: A writeable Python buffer object. Must be a contiguous
segment.
- `nbytes`: Number of bytes to read. Must be less than or equal to
:param nbytes: Number of bytes to read. Must be less than or equal to
the size of the buffer. Defaults to 0 which means the size of
`buffer`.
- `flags`: Flags for the recv system call (see recvfrom(2) manpage).
``buffer``.
:param flags: Flags for the recv system call (see recvfrom(2) manpage).
Defaults to 0.
:Return:
Returns a tuple ``(nbytes, address)`` where ``bytes`` is the number
:returns: A tuple ``(nbytes, address)`` where ``bytes`` is the number
of bytes read and ``address`` then it is the address of the remote
side.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef sockaddr_storage sa
cdef void *cbuf
......@@ -969,21 +932,19 @@ cdef public class sock [ object sock_object, type sock_type ]:
address = self.unparse_address(&sa, addr_len)
return r, address
cdef parse_address (self, object address, sockaddr_storage * sa, socklen_t * addr_len):
cdef parse_address (self, object address, sockaddr_storage * sa, socklen_t * addr_len, bint resolve=False):
"""Parse a Python socket address and set the C structure values.
:Parameters:
- `address`: The Python address to parse. For IP, it should be a
:param address: The Python address to parse. For IP, it should be a
``(IP, port)`` tuple where the IP is a string. Use the empty
string to indicate INADDR_ANY.
The port should always be a host-byte-order integer.
For Unix-domain sockets, the address should be a string.
- `sa`: OUTPUT: The sockaddr_storage C-structure to store the
:param sa: OUTPUT: The sockaddr_storage C-structure to store the
result in.
- `addr_len`: OUTPUT: The size of the structure placed into `sa`.
:param addr_len: OUTPUT: The size of the structure placed into ``sa``.
:Exceptions:
- `ValueError`: The value could not be parsed.
:raises ValueError: The value could not be parsed.
"""
cdef sockaddr_in * sin
cdef sockaddr_in6 *sin6
......@@ -996,7 +957,7 @@ cdef public class sock [ object sock_object, type sock_type ]:
ip = PySequence_GetItem(address, 0)
port = PySequence_GetItem(address, 1)
if not PyString_Check (ip):
raise ValueError, "IP address be a string"
raise ValueError, "IP address must be a string"
if PyString_Size (ip) == 0:
if self.domain == AF_INET:
ip = "0.0.0.0"
......@@ -1012,7 +973,12 @@ cdef public class sock [ object sock_object, type sock_type ]:
sin.sin_port = htons(port)
r = inet_pton(AF_INET, PyString_AsString(ip), &sin.sin_addr)
if r != 1:
raise ValueError, "not a valid IPv4 address"
if resolve:
return self.parse_address (
(the_resolver.resolve_ipv4 (ip), port), sa, addr_len, False
)
else:
raise ValueError ("not a valid IPv4 address")
elif self.domain == AF_INET6:
sin6.sin6_family = AF_INET6
IF UNAME_SYSNAME == "FreeBSD":
......@@ -1021,10 +987,15 @@ cdef public class sock [ object sock_object, type sock_type ]:
sin6.sin6_port = htons(port)
r = inet_pton(AF_INET6, PyString_AsString(ip), &sin6.sin6_addr)
if r != 1:
raise ValueError, "not a valid IPv6 address"
if resolve:
return self.parse_address (
(the_resolver.resolve_ipv6 (ip), port), sa, addr_len, False
)
else:
raise ValueError ("not a valid IPv6 address")
else:
raise ValueError, "Unsupported address family: %d" % self.domain
elif PyString_Check (address):
elif PyString_Check (address) and address[0] == '/':
# AF_UNIX
# +1 to grab the NUL char
l = PyString_Size (address) + 1
......@@ -1043,12 +1014,10 @@ cdef public class sock [ object sock_object, type sock_type ]:
cdef object unparse_address (self, sockaddr_storage *sa, socklen_t addr_len):
"""Unpack a C-socket address structure and generate a Python address object.
:Parameters:
- `sa`: The sockaddr_storage structure to unpack.
- `addr_len`: The length of the `sa` structure.
:param sa: The sockaddr_storage structure to unpack.
:param addr_len: The length of the ``sa`` structure.
:Return:
Returns a ``(IP, port)`` tuple for IP addresses where IP is a
:returns: A ``(IP, port)`` tuple for IP addresses where IP is a
string in canonical format for the given address family . Returns a
string for UNIX-domain sockets. Returns None for unknown socket
domains.
......@@ -1077,13 +1046,11 @@ cdef public class sock [ object sock_object, type sock_type ]:
This will block until there is data available to be read.
:Return:
Returns the amount "readable". For different sockets, this may be
:returns: The amount "readable". For different sockets, this may be
different values, see the EVFILT_READ section of the kevent manpage
for details.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
return self._wait_for_read()
......@@ -1092,11 +1059,9 @@ cdef public class sock [ object sock_object, type sock_type ]:
This will block until it is possible to write to the socket.
:Return:
Returns the number of bytes writeable on the socket.
:returns: The number of bytes writeable on the socket.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
return self._wait_for_write()
......@@ -1106,17 +1071,15 @@ cdef public class sock [ object sock_object, type sock_type ]:
cdef _wait_for_write (self):
return the_poller._wait_for_write (self.fd)
def connect (self, address):
cpdef connect_addr (self, address, bint resolve=False):
"""Connect the socket.
:Parameters:
- `address`: The address to connect to. For IP, it should be a
:param address: The address to connect to. For IP, it should be a
``(IP, port)`` tuple where the IP is a string.
The port should always be a host-byte-order integer. For
Unix-domain sockets, the address should be a string.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef sockaddr_storage sa
cdef socklen_t addr_len
......@@ -1124,7 +1087,7 @@ cdef public class sock [ object sock_object, type sock_type ]:
cdef coro me
me = <coro>the_scheduler._current
memset (&sa, 0, sizeof (sockaddr_storage))
self.parse_address (address, &sa, &addr_len)
self.parse_address (address, &sa, &addr_len, resolve)
while 1:
r = connect (self.fd, <sockaddr*>&sa, addr_len)
if r == -1:
......@@ -1136,18 +1099,19 @@ cdef public class sock [ object sock_object, type sock_type ]:
else:
return None
cpdef connect (self, address):
return self.connect_addr (address, True)
def bind (self, address):
"""Bind the socket.
:Parameters:
- `address`: The address to bind to. For IP, it should be a
:param address: The address to bind to. For IP, it should be a
``(IP, port)`` tuple where the IP is a string. Use the empty
string to indicate INADDR_ANY.
The port should always be a host-byte-order integer.
For Unix-domain sockets, the address should be a string.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef sockaddr_storage sa
cdef socklen_t addr_len
......@@ -1163,11 +1127,9 @@ cdef public class sock [ object sock_object, type sock_type ]:
def listen (self, backlog):
"""Set the socket to listen for connections.
:Parameters:
- `backlog`: The maximum size of the queue for pending connections.
:param backlog: The maximum size of the queue for pending connections.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef int r
r = listen (self.fd, backlog)
......@@ -1177,14 +1139,12 @@ cdef public class sock [ object sock_object, type sock_type ]:
def accept (self):
"""Accept a connection.
:Return:
Returns a tuple ``(socket, address)`` where ``socket`` is a socket
:returns: A tuple ``(socket, address)`` where ``socket`` is a socket
object and ``address`` is an ``(IP, port)`` tuple for IP
addresses or a string for UNIX-domain sockets. IP addresses are
returned as strings.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef sockaddr_storage sa
cdef socklen_t addr_len
......@@ -1214,20 +1174,17 @@ cdef public class sock [ object sock_object, type sock_type ]:
def accept_many (self, int max=0):
"""Accept multiple connections.
This will accept up to `max` connections for any connections available
This will accept up to ``max`` connections for any connections available
on the listen queue. This will block if there are no connections
waiting.
:Parameters:
- `max`: The maximum number of connections to accept. If not
:param max: The maximum number of connections to accept. If not
specified, defaults to infinity (accept all pending connections).
:Return:
Returns a list of ``(socket, address)`` tuples (see `accept` method
:returns: A list of ``(socket, address)`` tuples (see :meth:`accept` method
for information on return format).
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef sockaddr_storage sa
cdef socklen_t addr_len
......@@ -1267,11 +1224,9 @@ cdef public class sock [ object sock_object, type sock_type ]:
def shutdown (self, int how):
"""Shutdown the socket.
:Parameters:
- `how`: How to shut down the socket (see the shutdown(2) manpage).
:param how: How to shut down the socket (see the shutdown(2) manpage).
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef int r
r = shutdown (self.fd, how)
......@@ -1283,12 +1238,10 @@ cdef public class sock [ object sock_object, type sock_type ]:
def getpeername (self):
"""Get the remote-side address.
:Return:
Returns a ``(IP, port)`` tuple for IP addresses where IP is a
:returns: A ``(IP, port)`` tuple for IP addresses where IP is a
string. Returns a string for UNIX-domain sockets.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef sockaddr_storage sa
cdef socklen_t addr_len
......@@ -1305,8 +1258,7 @@ cdef public class sock [ object sock_object, type sock_type ]:
def getsockname (self):
"""Get the local address of the socket.
:Return:
Returns a ``(IP, port)`` tuple for IP addresses where IP is a
:returns: A ``(IP, port)`` tuple for IP addresses where IP is a
string or an empty string for INADDR_ANY. Returns a
string for UNIX-domain sockets (empty string if not bound).
"""
......@@ -1327,17 +1279,15 @@ cdef public class sock [ object sock_object, type sock_type ]:
The mode and bufsize arguments are as for the built-in open() function.
The underlying socket is duplicated via `sock.dup` to emulate Python's
The underlying socket is duplicated via ``sock.dup`` to emulate Python's
reference counting behavior.
:Parameters:
- `mode`: The mode of the file, defaults to 'r'.
- `bufsize`: The buffer size (0 is no buffering, 1 is line
:param mode: The mode of the file, defaults to 'r'.
:param bufsize: The buffer size (0 is no buffering, 1 is line
buffering, greater than 1 is the explicit buffer size).
Defaults to -1 (does not change the default buffering).
:Return:
Returns a file-like object that wraps the socket.
:returns: A file-like object that wraps the socket.
"""
# Probably unwise to access an underscore private value from the
# socket module, but it should work OK for the foreseeable future.
......@@ -1353,8 +1303,7 @@ cdef public class sock [ object sock_object, type sock_type ]:
def dup(self):
"""Duplicate the socket object using the OS dup() call.
:Return:
Returns a new sock instance that holds the new file descriptor.
:returns: A new sock instance that holds the new file descriptor.
"""
cdef sock new_sock
cdef int new_fd
......@@ -1366,12 +1315,38 @@ cdef public class sock [ object sock_object, type sock_type ]:
return sock(self.domain, self.stype, 0, new_fd)
class NameError (Exception):
pass
cdef class dummy_resolver:
"blocking name resolver uses socket.getaddrinfo()"
def resolve_ipv4 (self, bytes address):
addrs = __socketmodule.getaddrinfo (address, None, __socketmodule.AF_INET)
if not addrs:
raise NameError ("unable to resolve host: %r" % address)
else:
return addrs[0][4][0]
def resolve_ipv6 (self, bytes address):
addrs = __socketmodule.getaddrinfo (address, None, __socketmodule.AF_INET6)
if not addrs:
raise NameError ("unable to resolve host: %r" % address)
else:
return addrs[0][4][0]
the_resolver = dummy_resolver()
def set_resolver (resolver):
"replace the default resolver - return previous value"
global the_resolver
old_value = the_resolver
the_resolver = resolver
return old_value
def get_live_sockets():
"""Get the number of live socket objects. This includes socket objects
that are closed.
:Return:
Returns the number of socket objects.
:returns: The number of socket objects.
"""
global live_sockets
return live_sockets
......@@ -1436,7 +1411,7 @@ cdef class file_sock(sock):
The constructor takes one argument:
- ``fileobj``: A Python-like file object. Currently only needs to
:param fileobj: A Python-like file object. Currently only needs to
implement the ``fileno`` method.
When the object is deallocated, the file descriptor is closed.
......@@ -1459,7 +1434,7 @@ cdef class fd_sock(sock):
The constructor takes one argument:
- ``fd``: A file descriptor.
:param fd: A file descriptor.
When the object is deallocated, the file descriptor is closed.
"""
......@@ -1471,55 +1446,45 @@ cdef class fd_sock(sock):
def tcp_sock():
"""Create a streaming IPv4 socket.
:Return:
Returns a socket object.
:returns: A socket object.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
return sock (AF_INET, SOCK_STREAM)
def udp_sock():
"""Create a datagram IPv4 socket.
:Return:
Returns a socket object.
:returns: A socket object.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
return sock (AF_INET, SOCK_DGRAM)
def tcp6_sock():
"""Create a streaming IPv6 socket.
:Return:
Returns a socket object.
:returns: A socket object.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
return sock (AF_INET6, SOCK_STREAM)
def udp6_sock():
"""Create a datagram IPv6 socket.
:Return:
Returns a socket object.
:returns: A socket object.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
return sock (AF_INET6, SOCK_DGRAM)
def unix_sock():
"""Create a streaming unix-domain socket.
:Return:
Returns a socket object.
:returns: A socket object.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
return sock (AF_UNIX, SOCK_STREAM)
......@@ -1529,26 +1494,19 @@ def make_socket (int domain, int stype):
This is a backwards-compatibility wrapper around the sock object
constructor.
:Parameters:
- `domain`: The socket domain family (see `AF`).
- `stype`: The socket type (see `SOCK`).
:param domain: The socket domain family (see :class:`AF`).
:param stype: The socket type (see :class:`SOCK`).
:Return:
Returns a socket object.
:returns: A socket object.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
return sock (domain, stype)
def has_ipv6():
"""Whether or not this system can create an IPv6 socket.
:Return:
Returns True if this system can create an IPv6 socket, False otherwise
:Exceptions:
- None
:returns: True if this system can create an IPv6 socket, False otherwise
"""
cdef int s
......@@ -1563,16 +1521,13 @@ def has_ipv6():
def socketpair(int domain=AF_UNIX, int stype=SOCK_STREAM, int protocol=0):
"""Create an unnamed pair of connected sockets.
:Parameters:
- `domain`: The socket domain family (defaults to AF_UNIX).
- `stype`: The socket type (defaults to SOCK_STREAM).
- `protocol`: The socket protocol (normally not used, defaults to 0).
:param domain: The socket domain family (defaults to AF_UNIX).
:param stype: The socket type (defaults to SOCK_STREAM).
:param protocol: The socket protocol (normally not used, defaults to 0).
:Return:
Returns a tuple of 2 connected sockets.
:returns: A tuple of 2 connected sockets.
:Exceptions:
- `OSError`: OS-level error.
:raises OSError: OS-level error.
"""
cdef int sv[2]
cdef int rc
......
coro/sync.pyx
View file @ e921e61e
......@@ -49,22 +49,19 @@ class LockError (Exception):
# Semaphore
# ===========================================================================
"""A semaphore is a locking primitive that corresponds with a set of
resources. A semphore is essentially a counter. Whenever a resource is
aquired, the count is lowered. If the count goes below 0, then it blocks
until it goes above zero. Once you are done with a resource, you raise
the counter."""
cdef class semaphore:
"""Semaphore lock object.
"""
A semaphore is a locking primitive that corresponds with a set of
resources. A semphore is essentially a counter. Whenever a resource is
aquired, the count is lowered. If the count goes below 0, then it blocks
until it goes above zero. Once you are done with a resource, you raise
the counter.
The constructor takes one parameter, the value to start the semaphore with.
:param value: The value to start the semaphore with (an integer).
:IVariables:
- `avail`: The current value of the semaphore. Also available via
__int__.
- `_waiting`: A fifo of ``(value, co)`` tuples of coroutines waiting
:ivar avail: The current value of the semaphore. Also available via __int__.
:ivar _waiting: A fifo of ``(value, co)`` tuples of coroutines waiting
for the semaphore. ``value`` is the value being requested, and ``co``
is the coroutine object. (C only.)
"""
......@@ -87,8 +84,7 @@ cdef class semaphore:
if the requested number of resource elements are not available (if the
value would go negative).
:Parameters:
- `value`: The number of resource elements.
:param value: The number of resource elements.
"""
cdef coro me
me = the_scheduler._current
......@@ -116,8 +112,7 @@ cdef class semaphore:
def release(self, int value):
"""Release a number of resource elements.
:Parameters:
- `value`: The number of resource elements to release (add to the
:param value: The number of resource elements to release (add to the
sempahore).
"""
cdef coro ci
......@@ -174,21 +169,18 @@ cdef class semaphore:
cdef class inverted_semaphore:
"""Inverted semaphore.
"""
An inverted semaphore works very much like a regular semaphore, except
threads block _until_ the value reaches zero. For example, if you want a
thread to wait for 1 or more events to finish, you can have each event
raise the value (always nonblocking) and have your waiter thread call
block_till_zero.
The constructor takes one optional parameter, the value to start the
semaphore with. It defaults to 0.
:param value: The value to start the semaphore with. It defaults to 0.
:IVariables:
- `value`: The value of the inverted semaphore. Also available via
:ivar value: The value of the inverted semaphore. Also available via
__int__.
- `_waiting`: A fifo of coroutine objects waiting for the semaphore to
:ivar _waiting: A fifo of coroutine objects waiting for the semaphore to
reach zero. (C only).
"""
......@@ -207,8 +199,7 @@ cdef class inverted_semaphore:
This never blocks.
:Parameters:
- `value`: The number of resource elements to acquire (add to the
:param value: The number of resource elements to acquire (add to the
semaphore). Defaults to 1.
"""
self.value = self.value + value
......@@ -218,8 +209,7 @@ cdef class inverted_semaphore:
This never blocks. This may wake up waiting threads.
:Parameters:
- `value`: The number of resource elements to release (subtract
:param value: The number of resource elements to release (subtract
from the semaphore). Defaults to 1.
"""
cdef coro co
......@@ -258,16 +248,16 @@ cdef class inverted_semaphore:
cdef class mutex:
"""Mutual Exclusion lock object.
"""
Mutual Exclusion lock object.
A single thread may acquire the mutex multiple times, but it must release
the lock an equal number of times.
:IVariables:
- `_locked`: Count of how many locks on the mutex are currently held.
- `_owner`: The coroutine object that owns the lock (None if no owner).
:ivar _locked: Count of how many locks on the mutex are currently held.
:ivar _owner: The coroutine object that owns the lock (None if no owner).
(C only.)
- `_waiting`: A fifo of coroutine objects waiting for the lock.
:ivar _waiting: A fifo of coroutine objects waiting for the lock.
"""
cdef public int _locked
......@@ -290,8 +280,7 @@ cdef class mutex:
A coro thread may lock the mutex multiple times. It must call unlock
the same number of times to release it.
:Return:
Returns True if it blocked, False if the mutex was acquired
:returns: True if it blocked, False if the mutex was acquired
immediately.
"""
cdef coro me
......@@ -317,8 +306,7 @@ cdef class mutex:
def trylock(self):
"""Try to lock the mutex.
:Return:
Returns True if it is already locked by another coroutine thread.
:returns: True if it is already locked by another coroutine thread.
Returns False if the lock was successfully acquired.
"""
cdef coro me
......@@ -333,20 +321,17 @@ cdef class mutex:
def locked (self):
"""Determine if the mutex is currently locked.
:Return:
Returns True if the mutex is locked, otherwise False.
:returns: True if the mutex is locked, otherwise False.
"""
return (self._locked > 0)
def has_lock (self, thread=None):
"""Determine if a particular coroutine has the lock.
:Parameters:
- `thread`: The coroutine object to check if it owns the lock. If
:param thread: The coroutine object to check if it owns the lock. If
not specified, defaults to the current thread.
:Return:
Returns True if the specified thread has the lock, otherwise
:returns: True if the specified thread has the lock, otherwise
returns False.
"""
if thread is None:
......@@ -358,8 +343,7 @@ cdef class mutex:
The thread unlocking must be the thread that initially locked it.
:Return:
Returns True if another thread was waiting for the lock, otherwise
:returns: True if another thread was waiting for the lock, otherwise
it returns False.
"""
cdef coro me, co
......@@ -397,7 +381,8 @@ cdef class mutex:
cdef class rw_lock:
"""A many-reader single-writer lock.
"""
A many-reader single-writer lock.
This lock allows multiple "readers" to own the lock simultaneously. A
"writer" can only acquire a lock if there are no other "readers" or
......@@ -411,14 +396,13 @@ cdef class rw_lock:
way around (holding a read lock and trying to acquire a write lock will
cause a deadlock).
:IVariables:
- `_writer`: Count of the number of write locks. (C only.)
- `_writer_id`: Thread ID of the current write lock owner (0 if there
:ivar _writer: Count of the number of write locks. (C only.)
:ivar _writer_id: Thread ID of the current write lock owner (0 if there
is no owner). (C only.)
- `_reader`: Count of the number of read locks. (C only.)
- `_waiting_writers`: A fifo of coroutine objects waiting for a write
:ivar _reader: Count of the number of read locks. (C only.)
:ivar _waiting_writers: A fifo of coroutine objects waiting for a write
lock. (C only.)
- `_waiting_readers`: A fifo of coroutine objects waiting for a read
:ivar _waiting_readers: A fifo of coroutine objects waiting for a read
lock. (C only.)
"""
......@@ -443,7 +427,7 @@ cdef class rw_lock:
thread.
A coro thread may acquire multiple read locks, but it must call
`read_unlock` an equal number of times.
:meth:`read_unlock` an equal number of times.
"""
cdef coro me
me = the_scheduler._current
......@@ -463,12 +447,11 @@ cdef class rw_lock:
def try_read_lock(self):
"""Attempt to acquire a read lock.
This is the same as `read_lock` except it does not block if it cannot
This is the same as :meth:`read_lock` except it does not block if it cannot
acquire the lock.
:Return:
Returns True if it cannot acquire the lock.
Returns False if it successfully acquired the lock.
:returns: True if it cannot acquire the lock.
False if it successfully acquired the lock.
"""
cdef coro me
me = the_scheduler._current
......@@ -484,7 +467,7 @@ cdef class rw_lock:
This blocks if there are any other readers or writers holding the lock.
A coro thread may acquire multiple write locks, but it must call
`write_unlock` an equal number of times.
:meth:`write_unlock` an equal number of times.
Attempting to acquire a read lock while holding a write lock will cause
a deadlock.
......@@ -518,12 +501,11 @@ cdef class rw_lock:
def try_write_lock(self):
"""Attempt to acquire a write lock.
This is the same as `write_lock` except it does not block if it cannot
This is the same as :meth:`write_lock` except it does not block if it cannot
acquire the lock.
:Return:
Returns True if it cannot acquire the lock.
Returns False if it successfully acquired the lock.
:returns: True if it cannot acquire the lock.
False if it successfully acquired the lock.
"""
cdef coro me
me = the_scheduler._current
......@@ -602,13 +584,11 @@ cdef class rw_lock:
cdef class condition_variable:
"""Condition variable.
"""
This locking primitive provides a method to "trigger" an event for other
threads.
:IVariables:
- `_waiting`: A fifo of coroutine objects waiting for the lock. (C only.)
:ivar _waiting: A fifo of coroutine objects waiting for the lock. (C only.)
"""
cdef readonly _fifo _waiting
......@@ -634,8 +614,7 @@ cdef class condition_variable:
def wait (self):
"""Wait for the condition variable to be triggered.
:Return:
Returns the arguments given to the wake call (defaults to the empty
:returns: The arguments given to the wake call (defaults to the empty
tuple).
"""
return self._wait()
......@@ -663,20 +642,17 @@ cdef class condition_variable:
If there are no threads waiting, this does nothing.
:Parameters:
- `args`: The arguments to wake the thread with. Defaults to the
:param args: The arguments to wake the thread with. Defaults to the
empty tuple.
:Return:
Returns True if a thread was awoken, False if not.
:returns: True if a thread was awoken, False if not.
"""
return self._wake_one (args)
def wake_all (self, args=()):
"""Wake all waiting threads.
:Parameters:
- `args`: The arguments to wake the thread with. Defaults to the
:param args: The arguments to wake the thread with. Defaults to the
empty tuple.
"""
cdef coro co
......@@ -690,13 +666,11 @@ cdef class condition_variable:
def wake_n (self, int count, args=()):
"""Wake a specific number of threads.
:Parameters:
- `count`: The number of threads to wake up.
- `args`: The arguments to wake the thread with. Defaults to the
:param count: The number of threads to wake up.
:param args: The arguments to wake the thread with. Defaults to the
empty tuple.
:Return:
Returns the total number of threads actually awoken.
:returns: The total number of threads actually awoken.
"""
cdef coro co
cdef int total
......@@ -715,8 +689,7 @@ cdef class condition_variable:
def raise_all (self, the_exception):
"""Raise an exception on all waiting threads.
:Parameters:
- `the_exception`: The exception to raise on all waiting threads.
:param the_exception: The exception to raise on all waiting threads.
"""
cdef coro co
while self._waiting.size:
......@@ -732,13 +705,13 @@ cdef class condition_variable:
cdef class fifo:
"""First-in First-Out container.
"""
First-in First-Out container.
This uses a linked list.
:IVariables:
- `fifo`: The fifo object. (C only.)
- `cv`: A condition variable. (C only.)
:ivar fifo: The fifo object. (C only.)
:ivar cv: A condition variable. (C only.)
"""
cdef _fifo fifo
......@@ -754,8 +727,7 @@ cdef class fifo:
def push (self, thing):
"""Push an object to the end of the FIFO.
:Parameters:
- `thing`: The thing to add to the FIFO.
:param thing: The thing to add to the FIFO.
"""
self.fifo._push (thing)
self.cv.wake_one()
......@@ -765,8 +737,7 @@ cdef class fifo:
This blocks if the FIFO is empty.
:Return:
Returns the next object from the FIFO.
:returns: The next object from the FIFO.
"""
while self.fifo.size == 0:
self.cv._wait()
......@@ -778,8 +749,7 @@ cdef class fifo:
This will block if the fifo is empty and wait until there is an element
to pop.
:Return:
Returns a list of objects. Returns an empty list if the FIFO is
:returns: A list of objects. Returns an empty list if the FIFO is
empty.
"""
cdef int i
......
coro/zstack_lz4.pyx
View file @ e921e61e
......@@ -24,6 +24,9 @@ cdef class zstack:
if not self.buffer:
raise MemoryError
self.buffer_size = size
def __dealloc__ (self):
if self.buffer:
PyMem_Free (self.buffer)
cdef size_t deflate (self, void * base, size_t size):
return lz4.LZ4_compress (<char*>base, self.buffer, size)
cdef size_t inflate (self, void * dst, size_t dsize, void * src, size_t ssize):
......
coroutine/httpd/coro_httpd.py deleted 100644 → 0
View file @ 7ba1e191
# -*- Mode: Python; tab-width: 4 -*-
# Copyright 1999 by eGroups, Inc.
#
# All Rights Reserved
#
# Permission to use, copy, modify, and distribute this software and
# its documentation for any purpose and without fee is hereby
# granted, provided that the above copyright notice appear in all
# copies and that both that copyright notice and this permission
# notice appear in supporting documentation, and that the name of
# eGroups not be used in advertising or publicity pertaining to
# distribution of the software without specific, written prior
# permission.
#
# EGROUPS DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE,
# INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN
# NO EVENT SHALL EGROUPS BE LIABLE FOR ANY SPECIAL, INDIRECT OR
# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS
# OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
# NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
VERSION_STRING = '$Id$'
__split_vers__ = VERSION_STRING.split()
__version__ = (len(__split_vers__) > 2) and __split_vers__[2] or '1.0'
#
# coro_ehttpd
# This is an infrastructure for having a http server using coroutines.
# There are three major classes defined here:
# http_client
# This is a descendent of coro.Thread. It handles the connection
# to the client, spawned by http_server. Its run method goes through
# the stages of reading the request, filling out a http_request and
# finding the right handler, etc.
# http_request
# This object collects all of the data for a request. It is initialized
# from the http_client thread with the http request data, and is then
# passed to the handler to receive data. It attempts to enforce a valid
# http protocol on the response
# http_server
# This is a thread which just sits accepting on a socket, and spawning
# http_clients to handle incoming requests
#
# Additionally, the server expects http handler classes which respond
# to match and handle_request. There is an example class,
# http_file_handler, which is a basic handler to respond to GET requests
# to a document root. It'll return any file which exists.
#
# To use, implement your own handler class which responds to match and
# handle_request. Then, create a server, add handlers to the server,
# and start it. You then need to call the event_loop yourself.
# Something like:
#
# server = http_server(args = (('0.0.0.0', 7001),))
# file_handler = http_file_handler ('/home/htdocs/')
# server.push_handler (file_handler)
# server.start()
# coro.event_loop(30.0)
#
# we would like to handle persistent connections correctly.
# Here's the control flow:
# <server> : <-- connection
# create <client>
# while get_request(): <-- request
# create <request>
# get_handler().process_request()
#
# The difficulty (compared to medusa) is that the handler
# actually sends all the data, including the header. So
# the request object doesn't necessarily know what's going
# on.
# 1) we should force handlers to use an interface for setting headers. [i.e., CGI
# won't call "send('Content-Type: text/html')", it will use a method on the
# request object].
# 2) <request> should monitor changes to the headers.
# 3) a few standardized headers should have specific methods to improve performance.
#
# This still doesn't necessarily solve the difficult issue of persistence.
# 1) Will we pipeline? Only if we buffer on read. [I think we do].
# 2) what about chunking? We could just always use chunking. [no, that won't work
# because then browsers won't know how big files are]
#
# Suggestion: how about a dynamic-inheritance feature for setting headers?
# For example, use __getattr__ to catch anything beginning with {set,get}_header_xxx.
# This would allow subclasses to override specific methods. [could/should we use an
# inheritance to handle http/1.0 vs http/1.1?]
#
# [...]
# Ok, most of that is now done, although a bit of a mess. Who is reponsible for
# closing, in the event of http/1.0 and conn:close??
#
# some nastiness: important to understand exactly which coroutine each piece of code
# needs to run in, and whether it runs parallel or as a subroutine of some other coroutine.
#
# coroutines:
# 1) server (accept loop)
# 2) client (request loop)
# 3) session (request loop)
#
# For example, i'm pretty sure that the 'session' coroutine needs to execute as
# 'subroutine' of the 'client' coroutine; i.e. when the session yields(), it does
# so back to the client coroutine, not to main.
import coro
import coro_ssl
import errno
import http_date
import mime_type_table
import os
import re
import read_stream
import socket
import sslip
import stat
import sys
import time
try:
import qlog
except:
class Qlog:
def __call__(self, *args, **kw):
# print "Calling", self.name
return None
def __getattr__(self, name):
self.name = name
return self
def __repr__(self):
return 'null log daemon'
qlog = Qlog()
ssl_ctx = None
def init_ssl(protocol=sslip.SSLV23_SERVER_METHOD):
global ssl_ctx
if not ssl_ctx:
ssl_ctx = coro_ssl.ssl_ctx(protocol)
ssl_ctx.set_ciphers ('RC4-SHA:RC4-MD5:ALL')
def update_cert_key(cert, key, passwd='', chain=()):
global ssl_ctx
if cert and key:
cert_obj = sslip.read_pem_cert(cert)
ssl_ctx.use_cert(cert_obj, chain)
key_obj = sslip.read_pem_key(key, passwd)
ssl_ctx.use_key(key_obj)
class http_client:
def __init__ (self, group=None, target=None, name=None, logfp=sys.stderr, args=(), kwargs={}):
self.stream = None
self.buffer = ''
self._bytes = 0
self.logfp = logfp
def run (self, conn, peer, server_obj, handlers):
self.conn = conn
self.server = server_obj
self.peer = peer
# Note that peer could be a fake address, and server_obj can be None.
# These indicate a "backdoor" request from the gui.
try:
try:
count = 0
qlog.write('WEBUI.CONN_INIT', 'http', id(self), peer[0], peer[1])
while 1:
if self.server and self.server.shutdown_flag:
break
try:
# We use self.stream to read the header line-by-line
# and then switch to reading directly from the socket
# for the body (if needed). Reuse the previous
# instance if it exists, to support HTTP pipelining.
if not self.stream:
self.stream = read_stream.stream_reader(self.conn.recv)
request_line = self.read_line()
if not request_line:
break
except socket.error:
qlog.write('WEBUI.CONN_ERROR', 'http', id(self), 'socket error')
break
count = count + 1
headers = self.read_header()
#print '\n'.join (headers) + '\n\n'
request = http_request (self, request_line, headers)
request.read_body()
if request._error:
# Bad Request
request.error (400)
return
else:
try:
try:
handler = self.pick_handler (handlers, request)
if handler:
handler.handle_request (request)
else:
self.not_found (request)
if not request._done:
request.done()
except OSError, err:
if err[0] == errno.EPIPE:
pass # ignore broken pipe error
else:
raise # process exception in outer try
# These exceptions are used inside the coro
# stuff and shouldn't be thrown away
except (coro.TimeoutError, coro.Interrupted):
raise
except:
tb = coro.compact_traceback()
## sys.stderr.write (repr(tb))
request.error (500, tb)
qlog.write('COMMON.APP_FAILURE',
tb + ' request: ' + `request`)
tb = None
if request._close:
# ok, this gets interesting. the connection needs to close
# here. the finally clause below isn't getting hit because
# the session and client are running in the same coroutine.
# that's bad, I think.
conn.close()
break
# this should be a policy decision of the owner of logfp
# self.logfp.flush()
except read_stream.BufferOverflow:
# Indicates a request header that exceeded the line
# buffer, which may indicate an attack on the server.
# We just close the connection without a response.
# TODO:lrosenstein - log this since it may be an attack?
qlog.write('WEBUI.CONN_ERROR', 'http',
id(self), 'line buffer limit exceeded')
pass
except sslip.Error, why:
# Most likely a problem with SSL negotiation
qlog.write('WEBUI.CONN_ERROR',
'https',
id(self),
why[1])
pass
except OSError, err:
# We got some kind of I/O error that wasn't handled
# elsewhere. Since this seem to happen because the
# client closed the connection, it is safe to ignore
# the exception.
qlog.write('WEBUI.CONN_ERROR',
'http', id(self), 'OS error %s' % str(err[1]))
pass
except coro.TimeoutError:
# Either a timeout from coro_ssl or a timeout
# on a backdoor GUI request (see gui.py)
pass
finally:
conn.close()
def not_found (self, request):
request.error (404)
def pick_handler (self, handlers, request):
for handler in handlers:
if handler.match (request):
return handler
return None
# This is for handlers that process PUT/POST themselves. This whole
# thing needs to be redone with a file-like interface to 'stdin' for
# requests, and we need to think about HTTP/1.1 and pipelining,
# etc...
def read (self, size):
if self.stream:
self.buffer = self.stream.drain_buffer()
self.stream = None
while len(self.buffer) < size:
result = self.conn.recv(size-len(self.buffer))
if result:
self.buffer = self.buffer + result
else:
break # connection closed
result = self.buffer[:size]
self.buffer = self.buffer[size:]
return result
def read_line (self):
try:
return coro.with_timeout(300, self._read_line)
except coro.TimeoutError:
return '' # EOF
def _read_line (self):
"""Read a line of input. Return '' on EOF or error.
TODO:lrosenstein - we should probably distinguish EOF/error
from blank lines. This would affect read_header(), which
could return an incomplete set of headers if the connection
closed prematurely."""
while 1:
try:
(ln, eof) = self.stream.read_line()
if eof:
return '' # throw away incomplete lines
else:
return ln
except coro.TimeoutError: # ssl sockets timeout
# Ignored to fix bug 3185. The problem was that httpd
# was closing the connection after 30 sec, but IE/Win
# would try to use the connection and fail with a wierd
# error. Now, we have a 5 min timeout in read_line
# above, which applies to SSL and non-SSL connections
# to prevent clients from tying up server resources
# indefinitely.
continue
except OSError, why:
if why[0] == errno.ECONNRESET:
return '' # signal an eof to the caller
else:
raise
def read_header (self):
header = []
while 1:
l = self.read_line()
if not l:
break
else:
header.append (l)
return header
def send (self, data):
return self.conn.send (data)
def close (self):
self.conn.close()
class http_request:
request_count = 0
# <path>;<params>?<query>#<fragment>
path_re = re.compile ('(/[^;?#]*)(;[^?#]*)?(\?[^#]*)?(#.*)?')
# <method> <uri> HTTP/<version>
request_re = re.compile ('([^ ]+) ([^ ]+) *(HTTP/([0-9.]+))?')
# shadowed instance variable
_chunking = 0
_close = 0
_done = 0
_sent_headers = 0
_error = 0
_user_id = '-'
_session_id = '-'
def __init__ (self, client, request, headers):
self._reply_headers = {}
self._reply_cookies = ()
self._reply_code = 200
http_request.request_count = http_request.request_count + 1
self._request_number = http_request.request_count
self._request = request
self._request_headers = headers
self._client = client
self._server = client.server
self._tstart = time.time()
self._sent_bytes = 0
self._whom = client.conn.getpeername()
m = http_request.request_re.match (request)
if m:
(self._method, self._uri, ver, self._version) = m.groups()
self._method = self._method.lower()
if not self._version:
self._version = "0.9"
m = http_request.path_re.match (self._uri)
if m:
(self._path, self._params, self._query, self._frag) = m.groups()
if self._query and self._query[0] == '?':
self._query = self._query[1:]
else:
self._error = 1
else:
self._version = "1.0"
self._error = 1
def read_body(self):
"""Read the message body, if any, so that it's cleared from
the input stream. This avoids problems with keep-alives if
the request handler doesn't read the body itself.
This used to be done in the __init__method, but that can
lead to a fatal error in the Python interpreter (see bug 3367).
The ultimate solution is to fix the way connections are handled
to ensure that we don't reuse the connection if the body wasn't
fully read by the request handler."""
self._body = ''
clen = self.get_request_header('Content-Length')
if clen:
try:
clen = int(clen)
self._body = coro.with_timeout(
60,
self._client.read,
clen)
if len(self._body) < clen:
qlog.write('WEBUI.CONN_ERROR',
'http', id(self),
'Truncated body (%d<%d) (req:%s)' % \
(len(self._body), clen, self._request))
self._error = 1 # didn't get the body we were promised
except coro.TimeoutError:
qlog.write('WEBUI.CONN_ERROR',
'http', id(self),
'Body read timeout (req:%s)' % self._request)
self._error = 1
except ValueError:
qlog.write('WEBUI.CONN_ERROR',
'http', id(self),
'Invalid Content-Length (%s) (req:%s)' % \
(clen, self._request)
)
self._error = 1
def is_secure(self):
return self._client.server and self._client.server.is_secure()
# --------------------------------------------------
# request header management
# --------------------------------------------------
_header_re = re.compile (r'([^: ]+): (.*)')
def get_request_header (self, header):
header = header.lower()
for h in self._request_headers:
m = self._header_re.match (h)
if m:
name, value = m.groups()
if name.lower() == header:
return value
return ''
# --------------------------------------------------
# reply header management
# --------------------------------------------------
# header names are case-insensitive, and we need to be
# able to reliably query a request for certain headers,
# thus the sprinkling of key.lower() calls.
def __setitem__ (self, key, value):
self._reply_headers[key.lower()] = value
def __getitem__ (self, key):
return self._reply_headers[key.lower()]
def has_key (self, key):
return self._reply_headers.has_key (key.lower())
# TODO:lrosenstein - it's legal and necessary to have multiple
# TODO:lrosenstein - Set-Cookie headers. Handle these as a special case.
def set_reply_cookies(self, cookies):
"""Set sequence of cookies to be used in the response."""
self._reply_cookies = cookies
# --------------------------------------------------
# reading request
# --------------------------------------------------
def read (self, size):
data = self._body[:size]
self._body = self._body[size:]
return data
# --------------------------------------------------
# sending response
# --------------------------------------------------
def send (self, data):
self._sent_bytes = self._sent_bytes + len(data)
return self._client.send (data)
# chunking works thus:
# <data>
# becomes:
# <hex-length><CRLF>
# <data><CRLF>
# when done, signal with
# 0<CRLF><CRLF>
# ok, I admit this is now something of a mess.
# this could maybe be better if we could:
# 1) distinguish replies that have content
# [we could even detect it automatically?]
# 2) be more explicit about buffering the header
def push (self, data):
if not self._sent_headers:
self._sent_headers = 1
headers = self.get_headers()
else:
headers = ''
if data:
if self._chunking:
self.send (headers + '%x\r\n%s\r\n' % (len(data), data))
else:
self.send (headers + data)
else:
self.send (headers)
def done (self, with_body=1):
if not self._sent_headers:
self.send_headers()
if with_body and self._chunking:
# note: there's an invisible 'footer' between the pair of CRLF's.
# it can be used to send certain additional types of headers.
self.send ('0\r\n\r\n')
if self._close:
self._client.close ()
self._done = 1
qlog.write('WEBUI.HTTP_REQUEST',
self._client.peer[0],
self._user_id,
self._session_id,
self._reply_code,
self._request,
'' # XXX: User agent
)
#self._client.logfp.write (self.log())
def get_headers (self):
chunking = 0
# here is were we decide things like keep-alive, 1.0 vs 1.1, chunking, etc.
connection = self.get_request_header('connection').lower()
connection_tokens = [ x.strip() for x in connection.split(',')]
close_it = 0
if self._version == '1.0':
if 'keep-alive' in connection_tokens:
if not self.has_key ('content-length'):
close_it = 1
else:
self['Connection'] = 'Keep-Alive'
else:
close_it = 1
elif self._version == '1.1':
if 'close' in connection_tokens:
close_it = 1
elif not self.has_key ('content-length'):
if self.has_key ('transfer-encoding'):
if self['Transfer-Encoding'] == 'chunked':
chunking = 1
else:
close_it = 1
else:
self['Transfer-Encoding'] = 'chunked'
chunking = 1
elif self._version == '0.9':
close_it = 1
if close_it:
self['Connection'] = 'close'
self._close = 1
self._chunking = chunking
self['Server'] = 'IronPort httpd/%s' % __version__
self['Date'] = http_date.build_http_date (coro.now_usec / coro.microseconds)
headers = [self.response (self._reply_code)] + [
('%s: %s' % x) for x in self._reply_headers.items()
] + [
(x.output()) for x in self._reply_cookies
] + ['\r\n']
#print '\n'.join (headers) + '\n\n'
return '\r\n'.join (headers)
def send_headers (self):
# this will force the headers to be sent...
self.push ('')
def response (self, code=200):
message = self.responses[code]
self._reply_code = code
return 'HTTP/%s %d %s' % (self._version, code, message)
def error (self, code, reason=None, with_body=1):
self._reply_code = code
if with_body:
message = self.responses[code]
s = self.DEFAULT_ERROR_MESSAGE % {
'code': code, 'message': message, 'reason': reason
}
self['Content-Length'] = len(s)
self['Content-Type'] = 'text/html'
self.push (s)
self.done (with_body)
else:
self.done (with_body)
self._error = 1
def set_user_id(self, user_id, session_id=None):
self._user_id = user_id or '-'
self._session_id = session_id or '-'
def log_date_string (self, when):
return time.strftime (
'%d/%b/%Y:%H:%M:%S ',
time.gmtime(when)
) + tz_for_log
def log (self):
tend = time.time()
whom = '%s:%d ' % self._whom
return '%s - - [%s] "%s" %d %d %0.2f\n' % (
whom,
self.log_date_string (tend),
self._request,
self._reply_code,
self._sent_bytes,
tend - self._tstart
)
responses = {
100: "Continue",
101: "Switching Protocols",
200: "OK",
201: "Created",
202: "Accepted",
203: "Non-Authoritative Information",
204: "No Content",
205: "Reset Content",
206: "Partial Content",
300: "Multiple Choices",
301: "Moved Permanently",
302: "Moved Temporarily",
303: "See Other",
304: "Not Modified",
305: "Use Proxy",
400: "Bad Request",
401: "Unauthorized",
402: "Payment Required",
403: "Forbidden",
404: "Not Found",
405: "Method Not Allowed",
406: "Not Acceptable",
407: "Proxy Authentication Required",
408: "Request Time-out",
409: "Conflict",
410: "Gone",
411: "Length Required",
412: "Precondition Failed",
413: "Request Entity Too Large",
414: "Request-URI Too Large",
415: "Unsupported Media Type",
500: "Internal Server Error",
501: "Not Implemented",
502: "Bad Gateway",
503: "Service Unavailable",
504: "Gateway Time-out",
505: "HTTP Version not supported"
}
# Default error message
DEFAULT_ERROR_MESSAGE = '\r\n'.join ([
'<html>',
'<head>',
'<title>Error response</title>',
'</head>',
'<body>',
'<h1>Error response</h1>',
'<p>Error code %(code)d.',
'<p>Message: %(message)s.',
'<p>Reason: %(reason)s.',
'</body>',
'</html>',
''
])
import pprint
class coro_status_handler:
def match (self, request):
return request._path.split ('/')[1] == 'status'
def clean (self, s):
s = s.replace ('<','&lt;')
s = s.replace ('>','&gt;')
return s
def handle_request (self, request):
request['Content-Type'] = 'text/html'
request.push ('<p>Listening on\r\n')
request.push ( repr(request._server.addr) )
request.push ('</p>\r\n')
request.push ('<p>Request dump</p><PRE>\r\n')
request.push ( pprint.pformat(request) )
request.push ('</PRE>\r\n')
request.push ('<ul>\r\n')
all_threads = map(lambda x: (x[1], coro.where(x[1])), coro.all_threads.items())
for thread, traceback in all_threads:
request.push ('<li>%s\r\n' % self.clean (repr(thread)))
request.push ('<pre>\r\n')
for level in traceback[1:-1].split ('|'):
[file, fun, line] = level.split(':')
request.push ('<b>%20s</b>:%03d %s\r\n' % (fun,int(line),file))
request.push ('</pre>')
request.push ('</ul>\r\n')
request.done()
class http_file_handler:
def __init__ (self, doc_root):
self.doc_root = doc_root
self.logfp = sys.stderr
def match (self, request):
path = request._path
filename = os.path.join (self.doc_root, path[1:])
if os.path.exists (filename):
return 1
return 0
crack_if_modified_since = re.compile ('([^;]+)(; length=([0-9]+))?$', re.IGNORECASE)
def handle_request (self, request):
path = request._path
filename = os.path.join (self.doc_root, path[1:])
if request._method not in ('get', 'head'):
request.error (405)
return
if os.path.isdir (filename):
filename = os.path.join (filename, 'index.html')
if not os.path.isfile (filename):
request.error (404)
else:
stat_info = os.stat (filename)
mtime = stat_info[stat.ST_MTIME]
file_length = stat_info[stat.ST_SIZE]
ims = request.get_request_header ('if-modified-since')
if ims:
length_match = 1
m = self.crack_if_modified_since.match (ims)
if m:
length = m.group (3)
if length:
if int(length) != file_length:
length_match = 0
ims_date = http_date.parse_http_date (m.group(1))
if length_match and ims_date:
if mtime <= ims_date:
request.error (304, with_body=0)
return
base, ext = os.path.splitext (filename)
ext = ext[1:].lower()
request['Content-Type'] = mime_type_table.content_type_map.get (ext, 'text/plain')
request['Last-Modified'] = http_date.build_http_date (mtime)
if request._method == 'get':
f = open (filename, 'rb')
block = f.read (32768)
if not block:
request.error (204) # no content
else:
while 1:
request.push (block)
block = f.read (32768)
if not block:
break
elif request._method == 'head':
pass
else:
# should be impossible
request.error (405)
class http_server:
def __init__ (self):
self._handlers = []
self.shutdown_flag = 0
self.thread_id = None
self.addr = ()
self.config = {}
self._qlog_code = 'http'
def set_config (self, name, value):
self.config[name] = value
def get_config (self, name):
return self.config.get(name, None)
def is_secure(self):
return hasattr(self, 'cert')
def push_handler (self, handler):
self._handlers.append (handler)
def _make_socket(self):
server_s = coro.make_socket (socket.AF_INET, socket.SOCK_STREAM)
server_s.set_reuse_addr()
return server_s
def start (self, addr, retries=5):
"""Start the web server listening on addr in a new coroutine.
Try up to retries time to bind to that address.
Raises an exception if the bind fails."""
server_s = self._make_socket()
done = 0
save_errno = 0
self.addr = addr
while not done:
for x in xrange (retries):
try:
was_eaddrinuse = 0
server_s.bind (addr)
except OSError, why:
if why.errno not in (errno.EADDRNOTAVAIL, errno.EADDRINUSE):
raise
else:
save_errno = 0
if why.errno == errno.EADDRINUSE:
was_eaddrinuse = 1
else:
done = 1
break
coro.sleep_relative(1) # ... and retry
else:
coro.print_stderr ("cannot bind to %s:%d after 5 attempts, errno = %d\n" % (addr[0], addr[1], save_errno))
if was_eaddrinuse:
qlog.write('WEBUI.PORT_IN_USE',
addr[0], str(addr[1]))
coro.sleep_relative(15)
server_s.listen (1024)
c = coro.spawn(self._run, server_s)
c.set_name('http_server (%s:%d)' % addr)
return 1 # in case the caller is expecting TRUE on success
def _run (self, server_s):
secure = self.is_secure()
self.thread_id = coro.current().thread_id()
while not self.shutdown_flag:
try:
conn, addr = server_s.accept()
client = http_client()
coro.spawn (client.run, conn, addr, self, self._handlers)
except coro.Shutdown:
# server-shutdown
break
except:
qlog.write('COMMON.APP_FAILURE',
('%s accept handler error %s' %
(self.__class__.__name__, coro.compact_traceback())))
coro.sleep_relative(0.25)
continue
server_s.close()
return None
def shutdown(self):
self.shutdown_flag = 1
try:
thread = coro.get_thread_by_id(self.thread_id)
thread.shutdown()
except KeyError:
return # already exited
class https_server (http_server):
def __init__ (self):
http_server.__init__(self)
self._qlog_code = 'https'
def _make_socket (self):
global ssl_ctx
ssl_sock = coro_ssl.ssl_sock(ssl_ctx)
ssl_sock.create()
return ssl_sock
# Copied from medusa/http_server.py
def compute_timezone_for_log ():
if time.daylight:
tz = time.altzone
else:
tz = time.timezone
if tz > 0:
neg = 1
else:
neg = 0
tz = -tz
h, rem = divmod (tz, 3600)
m, rem = divmod (rem, 60)
if neg:
return '-%02d%02d' % (h, m)
else:
return '+%02d%02d' % (h, m)
# if you run this program over a TZ change boundary, this will be invalid.
tz_for_log = compute_timezone_for_log()
if __name__ == '__main__':
import backdoor, grp, os
if len (sys.argv) > 1:
doc_root = sys.argv[1]
else:
doc_root = '.'
import coro_httpd
init_ssl()
update_cert_key(coro_ssl.CERT, coro_ssl.KEY)
coro_httpd.init_ssl()
coro_httpd.update_cert_key(coro_ssl.CERT, coro_ssl.KEY)
# server = https_server()
server = http_server()
file_handler = http_file_handler (doc_root)
server.push_handler (coro_status_handler())
server.push_handler (file_handler)
#coro.spawn (server._run, (('0.0.0.0', 9001)))
coro.spawn(server.start, ('0.0.0.0', 9001))
# server.start((('0.0.0.0', 9001)))
coro.spawn (backdoor.serve)
coro.event_loop (30.0)
coroutine/httpd/mime_type_table.py deleted 100644 → 0
View file @ 7ba1e191
# -*- Python -*-
# Converted by ./convert_mime_type_table.py from:
# /usr/src2/apache_1.2b6/conf/mime.types
#
content_type_map = \
{
'ai': 'application/postscript',
'aif': 'audio/x-aiff',
'aifc': 'audio/x-aiff',
'aiff': 'audio/x-aiff',
'au': 'audio/basic',
'avi': 'video/x-msvideo',
'bcpio': 'application/x-bcpio',
'bin': 'application/octet-stream',
'cdf': 'application/x-netcdf',
'class': 'application/octet-stream',
'cpio': 'application/x-cpio',
'cpt': 'application/mac-compactpro',
'csh': 'application/x-csh',
'dcr': 'application/x-director',
'dir': 'application/x-director',
'dms': 'application/octet-stream',
'doc': 'application/msword',
'dvi': 'application/x-dvi',
'dxr': 'application/x-director',
'eps': 'application/postscript',
'etx': 'text/x-setext',
'exe': 'application/octet-stream',
'gif': 'image/gif',
'gtar': 'application/x-gtar',
'gz': 'application/x-gzip',
'hdf': 'application/x-hdf',
'hqx': 'application/mac-binhex40',
'htm': 'text/html',
'html': 'text/html',
'ice': 'x-conference/x-cooltalk',
'ief': 'image/ief',
'jpe': 'image/jpeg',
'jpeg': 'image/jpeg',
'jpg': 'image/jpeg',
'kar': 'audio/midi',
'latex': 'application/x-latex',
'lha': 'application/octet-stream',
'lzh': 'application/octet-stream',
'man': 'application/x-troff-man',
'me': 'application/x-troff-me',
'mid': 'audio/midi',
'midi': 'audio/midi',
'mif': 'application/x-mif',
'mov': 'video/quicktime',
'movie': 'video/x-sgi-movie',
'mp2': 'audio/mpeg',
'mpe': 'video/mpeg',
'mpeg': 'video/mpeg',
'mpg': 'video/mpeg',
'mpga': 'audio/mpeg',
'mp3': 'audio/mpeg',
'ms': 'application/x-troff-ms',
'nc': 'application/x-netcdf',
'oda': 'application/oda',
'pbm': 'image/x-portable-bitmap',
'pdb': 'chemical/x-pdb',
'pdf': 'application/pdf',
'pgm': 'image/x-portable-graymap',
'png': 'image/png',
'pnm': 'image/x-portable-anymap',
'ppm': 'image/x-portable-pixmap',
'ppt': 'application/powerpoint',
'ps': 'application/postscript',
'qt': 'video/quicktime',
'ra': 'audio/x-realaudio',
'ram': 'audio/x-pn-realaudio',
'ras': 'image/x-cmu-raster',
'rgb': 'image/x-rgb',
'roff': 'application/x-troff',
'rpm': 'audio/x-pn-realaudio-plugin',
'rtf': 'application/rtf',
'rtx': 'text/richtext',
'sgm': 'text/x-sgml',
'sgml': 'text/x-sgml',
'sh': 'application/x-sh',
'shar': 'application/x-shar',
'sit': 'application/x-stuffit',
'skd': 'application/x-koan',
'skm': 'application/x-koan',
'skp': 'application/x-koan',
'skt': 'application/x-koan',
'snd': 'audio/basic',
'src': 'application/x-wais-source',
'sv4cpio': 'application/x-sv4cpio',
'sv4crc': 'application/x-sv4crc',
't': 'application/x-troff',
'tar': 'application/x-tar',
'tcl': 'application/x-tcl',
'tex': 'application/x-tex',
'texi': 'application/x-texinfo',
'texinfo': 'application/x-texinfo',
'tif': 'image/tiff',
'tiff': 'image/tiff',
'tr': 'application/x-troff',
'tsv': 'text/tab-separated-values',
'txt': 'text/plain',
'ustar': 'application/x-ustar',
'vcd': 'application/x-cdlink',
'vrml': 'x-world/x-vrml',
'wav': 'audio/x-wav',
'wrl': 'x-world/x-vrml',
'xbm': 'image/x-xbitmap',
'xpm': 'image/x-xpixmap',
'xwd': 'image/x-xwindowdump',
'xyz': 'chemical/x-pdb',
'zip': 'application/zip',
}
coroutine/httpd/rfc2616.txt deleted 100644 → 0
View file @ 7ba1e191
This source diff could not be displayed because it is too large. You can view the blob instead.
coroutine/httpd/session_handler.py deleted 100644 → 0
View file @ 7ba1e191
# -*- Mode: Python; tab-width: 4 -*-
import coro
import string
import re
import time
h_re = re.compile (r'([^: ]+): (.*)')
def get_header (header, headers):
for h in headers:
m = h_re.match (h)
if m:
name, value = m.groups()
if string.lower (name) == header:
return value
return None
def extract_session (cookie):
parts = string.split (cookie, ';')
for part in parts:
pair = string.split (part, '=')
if len(pair) == 2:
if pair[0] == 'session':
return pair[1]
return None
class session_handler:
def __init__ (self, name, function):
self.name = name
self.function = function
self.sessions = {}
def match (self, request):
path = string.split (request._path, '/')
if (len(path) > 1) and (path[1] == self.name):
return 1
else:
return 0
def get_next_request (self):
return coro._yield()
def find_session (self, request):
cookie = get_header ('cookie', request._request_headers)
if cookie:
sid = extract_session (cookie)
return sid, self.sessions.get (sid, None)
else:
return None, None
def gen_session_id (self):
import random
import sys
sid = None
while self.sessions.has_key (sid):
n = random.randint (0,sys.maxint-1)
sid = hex(n)[2:]
return sid
expires_delta = 100 * 86400
def handle_request (self, request):
sid, c = self.find_session (request)
# The sid=='None' test is temporary hack, can probably remove it
if (not sid) or (sid=='None'):
sid = self.gen_session_id()
if c and c.isAlive():
# is c already running?
# hack, must grok this
coro.schedule (c, request)
request._done = 1
else:
# login
c = coro.new (self.function, self, request, sid)
# Wdy, DD-Mon-YYYY HH:MM:SS GMT
expires = time.strftime ('%a, %d-%b-%Y 00:00:00 GMT', time.gmtime (int (time.time()) + self.expires_delta))
request['Set-Cookie'] = 'session=%s; path=/; expires=%s' % (sid, expires)
# hack, must grok this
request._done = 1
c.start()
distribute_setup.py 0 → 100644
View file @ e921e61e
#!python
"""Bootstrap distribute installation
If you want to use setuptools in your package's setup.py, just include this
file in the same directory with it, and add this to the top of your setup.py::
from distribute_setup import use_setuptools
use_setuptools()
If you want to require a specific version of setuptools, set a download
mirror, or use an alternate download directory, you can do so by supplying
the appropriate options to ``use_setuptools()``.
This file can also be run as a script to install or upgrade setuptools.
"""
import os
import sys
import time
import fnmatch
import tempfile
import tarfile
from distutils import log
try:
from site import USER_SITE
except ImportError:
USER_SITE = None
try:
import subprocess
def _python_cmd(*args):
args = (sys.executable,) + args
return subprocess.call(args) == 0
except ImportError:
# will be used for python 2.3
def _python_cmd(*args):
args = (sys.executable,) + args
# quoting arguments if windows
if sys.platform == 'win32':
def quote(arg):
if ' ' in arg:
return '"%s"' % arg
return arg
args = [quote(arg) for arg in args]
return os.spawnl(os.P_WAIT, sys.executable, *args) == 0
DEFAULT_VERSION = "0.6.26"
DEFAULT_URL = "http://pypi.python.org/packages/source/d/distribute/"
SETUPTOOLS_FAKED_VERSION = "0.6c11"
SETUPTOOLS_PKG_INFO = """\
Metadata-Version: 1.0
Name: setuptools
Version: %s
Summary: xxxx
Home-page: xxx
Author: xxx
Author-email: xxx
License: xxx
Description: xxx
""" % SETUPTOOLS_FAKED_VERSION
def _install(tarball, install_args=()):
# extracting the tarball
tmpdir = tempfile.mkdtemp()
log.warn('Extracting in %s', tmpdir)
old_wd = os.getcwd()
try:
os.chdir(tmpdir)
tar = tarfile.open(tarball)
_extractall(tar)
tar.close()
# going in the directory
subdir = os.path.join(tmpdir, os.listdir(tmpdir)[0])
os.chdir(subdir)
log.warn('Now working in %s', subdir)
# installing
log.warn('Installing Distribute')
if not _python_cmd('setup.py', 'install', *install_args):
log.warn('Something went wrong during the installation.')
log.warn('See the error message above.')
finally:
os.chdir(old_wd)
def _build_egg(egg, tarball, to_dir):
# extracting the tarball
tmpdir = tempfile.mkdtemp()
log.warn('Extracting in %s', tmpdir)
old_wd = os.getcwd()
try:
os.chdir(tmpdir)
tar = tarfile.open(tarball)
_extractall(tar)
tar.close()
# going in the directory
subdir = os.path.join(tmpdir, os.listdir(tmpdir)[0])
os.chdir(subdir)
log.warn('Now working in %s', subdir)
# building an egg
log.warn('Building a Distribute egg in %s', to_dir)
_python_cmd('setup.py', '-q', 'bdist_egg', '--dist-dir', to_dir)
finally:
os.chdir(old_wd)
# returning the result
log.warn(egg)
if not os.path.exists(egg):
raise IOError('Could not build the egg.')
def _do_download(version, download_base, to_dir, download_delay):
egg = os.path.join(to_dir, 'distribute-%s-py%d.%d.egg'
% (version, sys.version_info[0], sys.version_info[1]))
if not os.path.exists(egg):
tarball = download_setuptools(version, download_base,
to_dir, download_delay)
_build_egg(egg, tarball, to_dir)
sys.path.insert(0, egg)
import setuptools
setuptools.bootstrap_install_from = egg
def use_setuptools(version=DEFAULT_VERSION, download_base=DEFAULT_URL,
to_dir=os.curdir, download_delay=15, no_fake=True):
# making sure we use the absolute path
to_dir = os.path.abspath(to_dir)
was_imported = 'pkg_resources' in sys.modules or \
'setuptools' in sys.modules
try:
try:
import pkg_resources
if not hasattr(pkg_resources, '_distribute'):
if not no_fake:
_fake_setuptools()
raise ImportError
except ImportError:
return _do_download(version, download_base, to_dir, download_delay)
try:
pkg_resources.require("distribute>="+version)
return
except pkg_resources.VersionConflict:
e = sys.exc_info()[1]
if was_imported:
sys.stderr.write(
"The required version of distribute (>=%s) is not available,\n"
"and can't be installed while this script is running. Please\n"
"install a more recent version first, using\n"
"'easy_install -U distribute'."
"\n\n(Currently using %r)\n" % (version, e.args[0]))
sys.exit(2)
else:
del pkg_resources, sys.modules['pkg_resources'] # reload ok
return _do_download(version, download_base, to_dir,
download_delay)
except pkg_resources.DistributionNotFound:
return _do_download(version, download_base, to_dir,
download_delay)
finally:
if not no_fake:
_create_fake_setuptools_pkg_info(to_dir)
def download_setuptools(version=DEFAULT_VERSION, download_base=DEFAULT_URL,
to_dir=os.curdir, delay=15):
"""Download distribute from a specified location and return its filename
`version` should be a valid distribute version number that is available
as an egg for download under the `download_base` URL (which should end
with a '/'). `to_dir` is the directory where the egg will be downloaded.
`delay` is the number of seconds to pause before an actual download
attempt.
"""
# making sure we use the absolute path
to_dir = os.path.abspath(to_dir)
try:
from urllib.request import urlopen
except ImportError:
from urllib2 import urlopen
tgz_name = "distribute-%s.tar.gz" % version
url = download_base + tgz_name
saveto = os.path.join(to_dir, tgz_name)
src = dst = None
if not os.path.exists(saveto): # Avoid repeated downloads
try:
log.warn("Downloading %s", url)
src = urlopen(url)
# Read/write all in one block, so we don't create a corrupt file
# if the download is interrupted.
data = src.read()
dst = open(saveto, "wb")
dst.write(data)
finally:
if src:
src.close()
if dst:
dst.close()
return os.path.realpath(saveto)
def _no_sandbox(function):
def __no_sandbox(*args, **kw):
try:
from setuptools.sandbox import DirectorySandbox
if not hasattr(DirectorySandbox, '_old'):
def violation(*args):
pass
DirectorySandbox._old = DirectorySandbox._violation
DirectorySandbox._violation = violation
patched = True
else:
patched = False
except ImportError:
patched = False
try:
return function(*args, **kw)
finally:
if patched:
DirectorySandbox._violation = DirectorySandbox._old
del DirectorySandbox._old
return __no_sandbox
def _patch_file(path, content):
"""Will backup the file then patch it"""
existing_content = open(path).read()
if existing_content == content:
# already patched
log.warn('Already patched.')
return False
log.warn('Patching...')
_rename_path(path)
f = open(path, 'w')
try:
f.write(content)
finally:
f.close()
return True
_patch_file = _no_sandbox(_patch_file)
def _same_content(path, content):
return open(path).read() == content
def _rename_path(path):
new_name = path + '.OLD.%s' % time.time()
log.warn('Renaming %s into %s', path, new_name)
os.rename(path, new_name)
return new_name
def _remove_flat_installation(placeholder):
if not os.path.isdir(placeholder):
log.warn('Unkown installation at %s', placeholder)
return False
found = False
for file in os.listdir(placeholder):
if fnmatch.fnmatch(file, 'setuptools*.egg-info'):
found = True
break
if not found:
log.warn('Could not locate setuptools*.egg-info')
return
log.warn('Removing elements out of the way...')
pkg_info = os.path.join(placeholder, file)
if os.path.isdir(pkg_info):
patched = _patch_egg_dir(pkg_info)
else:
patched = _patch_file(pkg_info, SETUPTOOLS_PKG_INFO)
if not patched:
log.warn('%s already patched.', pkg_info)
return False
# now let's move the files out of the way
for element in ('setuptools', 'pkg_resources.py', 'site.py'):
element = os.path.join(placeholder, element)
if os.path.exists(element):
_rename_path(element)
else:
log.warn('Could not find the %s element of the '
'Setuptools distribution', element)
return True
_remove_flat_installation = _no_sandbox(_remove_flat_installation)
def _after_install(dist):
log.warn('After install bootstrap.')
placeholder = dist.get_command_obj('install').install_purelib
_create_fake_setuptools_pkg_info(placeholder)
def _create_fake_setuptools_pkg_info(placeholder):
if not placeholder or not os.path.exists(placeholder):
log.warn('Could not find the install location')
return
pyver = '%s.%s' % (sys.version_info[0], sys.version_info[1])
setuptools_file = 'setuptools-%s-py%s.egg-info' % \
(SETUPTOOLS_FAKED_VERSION, pyver)
pkg_info = os.path.join(placeholder, setuptools_file)
if os.path.exists(pkg_info):
log.warn('%s already exists', pkg_info)
return
log.warn('Creating %s', pkg_info)
f = open(pkg_info, 'w')
try:
f.write(SETUPTOOLS_PKG_INFO)
finally:
f.close()
pth_file = os.path.join(placeholder, 'setuptools.pth')
log.warn('Creating %s', pth_file)
f = open(pth_file, 'w')
try:
f.write(os.path.join(os.curdir, setuptools_file))
finally:
f.close()
_create_fake_setuptools_pkg_info = _no_sandbox(_create_fake_setuptools_pkg_info)
def _patch_egg_dir(path):
# let's check if it's already patched
pkg_info = os.path.join(path, 'EGG-INFO', 'PKG-INFO')
if os.path.exists(pkg_info):
if _same_content(pkg_info, SETUPTOOLS_PKG_INFO):
log.warn('%s already patched.', pkg_info)
return False
_rename_path(path)
os.mkdir(path)
os.mkdir(os.path.join(path, 'EGG-INFO'))
pkg_info = os.path.join(path, 'EGG-INFO', 'PKG-INFO')
f = open(pkg_info, 'w')
try:
f.write(SETUPTOOLS_PKG_INFO)
finally:
f.close()
return True
_patch_egg_dir = _no_sandbox(_patch_egg_dir)
def _before_install():
log.warn('Before install bootstrap.')
_fake_setuptools()
def _under_prefix(location):
if 'install' not in sys.argv:
return True
args = sys.argv[sys.argv.index('install')+1:]
for index, arg in enumerate(args):
for option in ('--root', '--prefix'):
if arg.startswith('%s=' % option):
top_dir = arg.split('root=')[-1]
return location.startswith(top_dir)
elif arg == option:
if len(args) > index:
top_dir = args[index+1]
return location.startswith(top_dir)
if arg == '--user' and USER_SITE is not None:
return location.startswith(USER_SITE)
return True
def _fake_setuptools():
log.warn('Scanning installed packages')
try:
import pkg_resources
except ImportError:
# we're cool
log.warn('Setuptools or Distribute does not seem to be installed.')
return
ws = pkg_resources.working_set
try:
setuptools_dist = ws.find(pkg_resources.Requirement.parse('setuptools',
replacement=False))
except TypeError:
# old distribute API
setuptools_dist = ws.find(pkg_resources.Requirement.parse('setuptools'))
if setuptools_dist is None:
log.warn('No setuptools distribution found')
return
# detecting if it was already faked
setuptools_location = setuptools_dist.location
log.warn('Setuptools installation detected at %s', setuptools_location)
# if --root or --preix was provided, and if
# setuptools is not located in them, we don't patch it
if not _under_prefix(setuptools_location):
log.warn('Not patching, --root or --prefix is installing Distribute'
' in another location')
return
# let's see if its an egg
if not setuptools_location.endswith('.egg'):
log.warn('Non-egg installation')
res = _remove_flat_installation(setuptools_location)
if not res:
return
else:
log.warn('Egg installation')
pkg_info = os.path.join(setuptools_location, 'EGG-INFO', 'PKG-INFO')
if (os.path.exists(pkg_info) and
_same_content(pkg_info, SETUPTOOLS_PKG_INFO)):
log.warn('Already patched.')
return
log.warn('Patching...')
# let's create a fake egg replacing setuptools one
res = _patch_egg_dir(setuptools_location)
if not res:
return
log.warn('Patched done.')
_relaunch()
def _relaunch():
log.warn('Relaunching...')
# we have to relaunch the process
# pip marker to avoid a relaunch bug
if sys.argv[:3] == ['-c', 'install', '--single-version-externally-managed']:
sys.argv[0] = 'setup.py'
args = [sys.executable] + sys.argv
sys.exit(subprocess.call(args))
def _extractall(self, path=".", members=None):
"""Extract all members from the archive to the current working
directory and set owner, modification time and permissions on
directories afterwards. `path' specifies a different directory
to extract to. `members' is optional and must be a subset of the
list returned by getmembers().
"""
import copy
import operator
from tarfile import ExtractError
directories = []
if members is None:
members = self
for tarinfo in members:
if tarinfo.isdir():
# Extract directories with a safe mode.
directories.append(tarinfo)
tarinfo = copy.copy(tarinfo)
tarinfo.mode = 448 # decimal for oct 0700
self.extract(tarinfo, path)
# Reverse sort directories.
if sys.version_info < (2, 4):
def sorter(dir1, dir2):
return cmp(dir1.name, dir2.name)
directories.sort(sorter)
directories.reverse()
else:
directories.sort(key=operator.attrgetter('name'), reverse=True)
# Set correct owner, mtime and filemode on directories.
for tarinfo in directories:
dirpath = os.path.join(path, tarinfo.name)
try:
self.chown(tarinfo, dirpath)
self.utime(tarinfo, dirpath)
self.chmod(tarinfo, dirpath)
except ExtractError:
e = sys.exc_info()[1]
if self.errorlevel > 1:
raise
else:
self._dbg(1, "tarfile: %s" % e)
def _build_install_args(argv):
install_args = []
user_install = '--user' in argv
if user_install and sys.version_info < (2,6):
log.warn("--user requires Python 2.6 or later")
raise SystemExit(1)
if user_install:
install_args.append('--user')
return install_args
def main(argv, version=DEFAULT_VERSION):
"""Install or upgrade setuptools and EasyInstall"""
tarball = download_setuptools()
_install(tarball, _build_install_args(argv))
if __name__ == '__main__':
main(sys.argv[1:])
docs/Makefile 0 → 100644
View file @ e921e61e
# Makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
PAPER =
BUILDDIR = _build
# Internal variables.
PAPEROPT_a4 = -D latex_paper_size=a4
PAPEROPT_letter = -D latex_paper_size=letter
ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
# the i18n builder cannot share the environment and doctrees with the others
I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext
help:
@echo "Please use \`make <target>' where <target> is one of"
@echo " html to make standalone HTML files"
@echo " dirhtml to make HTML files named index.html in directories"
@echo " singlehtml to make a single large HTML file"
@echo " pickle to make pickle files"
@echo " json to make JSON files"
@echo " htmlhelp to make HTML files and a HTML help project"
@echo " qthelp to make HTML files and a qthelp project"
@echo " devhelp to make HTML files and a Devhelp project"
@echo " epub to make an epub"
@echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
@echo " latexpdf to make LaTeX files and run them through pdflatex"
@echo " text to make text files"
@echo " man to make manual pages"
@echo " texinfo to make Texinfo files"
@echo " info to make Texinfo files and run them through makeinfo"
@echo " gettext to make PO message catalogs"
@echo " changes to make an overview of all changed/added/deprecated items"
@echo " linkcheck to check all external links for integrity"
@echo " doctest to run all doctests embedded in the documentation (if enabled)"
clean:
-rm -rf $(BUILDDIR)/*
html:
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
dirhtml:
$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
singlehtml:
$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
@echo
@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
pickle:
$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
@echo
@echo "Build finished; now you can process the pickle files."
json:
$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
@echo
@echo "Build finished; now you can process the JSON files."
htmlhelp:
$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
@echo
@echo "Build finished; now you can run HTML Help Workshop with the" \
".hhp project file in $(BUILDDIR)/htmlhelp."
qthelp:
$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
@echo
@echo "Build finished; now you can run "qcollectiongenerator" with the" \
".qhcp project file in $(BUILDDIR)/qthelp, like this:"
@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Shrapnel.qhcp"
@echo "To view the help file:"
@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Shrapnel.qhc"
devhelp:
$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
@echo
@echo "Build finished."
@echo "To view the help file:"
@echo "# mkdir -p $$HOME/.local/share/devhelp/Shrapnel"
@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Shrapnel"
@echo "# devhelp"
epub:
$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
@echo
@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
latex:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo
@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
@echo "Run \`make' in that directory to run these through (pdf)latex" \
"(use \`make latexpdf' here to do that automatically)."
latexpdf:
$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
@echo "Running LaTeX files through pdflatex..."
$(MAKE) -C $(BUILDDIR)/latex all-pdf
@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
text:
$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
@echo
@echo "Build finished. The text files are in $(BUILDDIR)/text."
man:
$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
@echo
@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
texinfo:
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
@echo
@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
@echo "Run \`make' in that directory to run these through makeinfo" \
"(use \`make info' here to do that automatically)."
info:
$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
@echo "Running Texinfo files through makeinfo..."
make -C $(BUILDDIR)/texinfo info
@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
gettext:
$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
@echo
@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
changes:
$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
@echo
@echo "The overview file is in $(BUILDDIR)/changes."
linkcheck:
$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
@echo
@echo "Link check complete; look for any errors in the above output " \
"or in $(BUILDDIR)/linkcheck/output.txt."
doctest:
$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
@echo "Testing of doctests in the sources finished, look at the " \
"results in $(BUILDDIR)/doctest/output.txt."
docs/conf.py 0 → 100644
View file @ e921e61e
# -*- coding: utf-8 -*-
#
# Shrapnel documentation build configuration file, created by
# sphinx-quickstart on Fri Apr 13 18:44:49 2012.
#
# This file is execfile()d with the current directory set to its containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
import sys, os
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#sys.path.insert(0, os.path.abspath('.'))
# -- General configuration -----------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = ['sphinx.ext.autodoc', 'sphinx.ext.todo']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'Shrapnel'
copyright = u'2012, Sam Rushing'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = '1.0.2'
# The full version, including alpha/beta/rc tags.
release = '1.0.2'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['_build']
# The reST default role (used for this markup: `text`) to use for all documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
#modindex_common_prefix = []
# -- Options for HTML output ---------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'default'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
html_theme_options = {'collapsiblesidebar': True}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
#html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
#html_domain_indices = True
# If false, no index is generated.
#html_use_index = True
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, links to the reST sources are added to the pages.
#html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = None
# Output file base name for HTML help builder.
htmlhelp_basename = 'Shrapneldoc'
# -- Options for LaTeX output --------------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#'preamble': '',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass [howto/manual]).
latex_documents = [
('index', 'Shrapnel.tex', u'Shrapnel Documentation',
u'Sam Rushing', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
#latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#latex_use_parts = False
# If true, show page references after internal links.
#latex_show_pagerefs = False
# If true, show URL addresses after external links.
#latex_show_urls = False
# Documents to append as an appendix to all manuals.
#latex_appendices = []
# If false, no module index is generated.
#latex_domain_indices = True
# -- Options for manual page output --------------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
('index', 'shrapnel', u'Shrapnel Documentation',
[u'Sam Rushing'], 1)
]
# If true, show URL addresses after external links.
#man_show_urls = False
# -- Options for Texinfo output ------------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
('index', 'Shrapnel', u'Shrapnel Documentation',
u'Sam Rushing', 'Shrapnel', 'One line description of project.',
'Miscellaneous'),
]
# Documents to append as an appendix to all manuals.
#texinfo_appendices = []
# If false, no module index is generated.
#texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
#texinfo_show_urls = 'footnote'
autodoc_default_flags = ['members', 'show-inheritance']
docs/coro.rst
View file @ e921e61e
......@@ -3,8 +3,7 @@ Shrapnel/Coro
=============
:Date: $Date: 2008/05/06 $
:Revision: $Revision: #1 $
:Author: Sam Rushing <rushing@ironport.com>
:Author: Sam Rushing
.. contents::
:depth: 2
......@@ -13,6 +12,15 @@ Shrapnel/Coro
Shrapnel/Coro is a cooperative thread facility built on top of Python.
.. note::
This document was originally written for internal use at
IronPort. It refers to several facilities that unfortunately have not
(yet) been open-sourced, (e.g., the dns resolver and sntp client).
It also references and describes things that are specific to the
IronPort mail appliance. Much of the advice in here is good, though,
and I hope to revisit it soon.
Threads
=======
......
docs/index.rst 0 → 100644
View file @ e921e61e
.. Shrapnel documentation master file, created by
sphinx-quickstart on Fri Apr 13 18:44:49 2012.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to Shrapnel's documentation!
====================================
Contents:
* :doc:`Installation <installation>`
* :doc:`Tutorial <tutorial>`
* :doc:`Reference Manual <ref/index>`
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
docs/installation.rst 0 → 100644
View file @ e921e61e
============
Installation
============
Supported Platforms
===================
Shrapnel currently works on FreeBSD, Linux, and Mac OS X with x86 32- or 64-bit platforms.
It supports Python 2.7 (TODO: and 2.6?).
Prerequisites
=============
pip
---
To make installation easy, you can use `pip <http://www.pip-installer.org/>`_.
This is a tool which will fetch Python packages from `PyPi
<http://pypi.python.org/>`_ and install them.
Visit http://www.pip-installer.org/en/latest/installing.html for information
on how to install pip if you don't already have it installed.
Cython
------
You need version 0.12.1 or newer of `Cython <http://cython.org/>`_. If you
already have Cython installed, you can check your current version by running
``cython -V``.
To install Cython, run:
pip install cython
Distribute
----------
You need version 0.6.16 or newer of `distribute <http://pypi.python.org/pypi/distribute>`_.
Distribute is a build and packaging tool for Python (a replacement for setuptools).
To install distribute, run:
pip install distribute
Shrapnel
--------
Finally, you can install Shrapnel, run:
pip install shrapnel
Alternatively you can download it from https://github.com/ironport/shrapnel
and do the usual ``python setup.py install`` procedure.
docs/ref/clocks.rst 0 → 100644
View file @ e921e61e
======
Clocks
======
Shrapnel needs to keep track of time to manage scheduling of sleeps and
timeouts. Because Shrapnel is intended to support thousands of coroutines,
and each coroutine may be making many timeout calls per second, Shrapnel needs
to use a timing facility that is relatively high performance. It also needs
one that is monotonic, so it does not need to deal with system clock changes.
The ``clocks`` subpackage is intended to provide a variety of different time
facilities. Currently it only supports using the x86 TSC timer. This is a
timer built in to the CPU, and thus is very fast.
TSC Time
========
Support for TSC time is implemented in the ``coro.clocks.tsc_time`` module.
.. automodule:: coro.clocks.tsc_time
docs/ref/coroutines.rst 0 → 100644
View file @ e921e61e
==========
Coroutines
==========
The central concept of Shrapnel is the coroutine. You can think of a coroutine
like it is a thread. When it runs out of work to do, it yields and allows other
coroutines to run. Scheduling of coroutines is handled by the scheduler which
runs an "event loop".
Event Loop
==========
The event loop is a loop that runs forever until the program ends. Every
Shrapnel program needs to start the event loop as one of the first things it
does. A typical example would be::
import coro
def main():
print 'Hello world!'
# This will cause the process to exit.
coro.set_exit(0)
if __name__ == '__main__':
coro.spawn(main)
coro.event_loop()
Coroutines
==========
Every coroutine thread is created with either the :func:`new` function (which
does NOT automatically start the thread) or the :func:`spawn` function (which
DOES automatically start it).
Every thread has a unique numeric ID. You may also set the name of the thread
when you create it.
.. autoclass:: coro.coro
Timeouts
========
The shrapnel timeout facility allows you to execute a function which will be
interrupted if it does not finish within a specified period of time. The
:class:`TimeoutError` exception will be raised if the timeout expires. See the
:func:`with_timeout` docstring for more detail.
If the event loop is not running (such as in a non-coro process), a custom
version of `with_timeout` is installed that will operate using SIGALRM so that
you may use `with_timeout` in code that needs to run in non-coro processes
(though this is not recommended and should be avoided if possible).
.. autofunction:: coro.with_timeout
Parallel Execution
==================
XXX
.. autofunction:: coro.in_parallel
.. autoexception:: coro.InParallelError
Thread Local Storage
====================
There is a thread-local storage interface available for storing global data that
is thread-specific. You instantiate a :class:`ThreadLocal` instance and you can
assign attributes to it that will be specific to that thread. From a design
perspective, it is generally discouraged to use thread-local storage. But
nonetheless, it can be useful at times.
.. autoclass:: coro.ThreadLocal
Functions
=========
The coro module defines the following functions:
.. autofunction:: coro.get_thread_by_id
.. autofunction:: coro.coro_is_running
.. autofunction:: coro.event_loop
.. autofunction:: coro.new
.. autofunction:: coro.spawn
.. autofunction:: coro.waitpid
.. autofunction:: coro.yield_slice
.. autofunction:: coro.schedule
.. autofunction:: coro.current
.. autofunction:: coro.set_exit
.. autofunction:: coro.set_print_exit_string
.. autofunction:: coro.sleep_absolute
.. autofunction:: coro.sleep_relative
Variables
=========
.. py:data:: coro.all_threads
A dictionary of all live coroutine objects. The key is the coroutine ID,
and the value is the coroutine object.
Exceptions
==========
The coro module defines the following exceptions:
.. autoexception:: coro.ScheduleError
.. autoexception:: coro.DeadCoroutine
.. autoexception:: coro.ClosedError
.. autoexception:: coro.NotStartedError
.. autoexception:: coro.TimeoutError
.. autoexception:: coro.SimultaneousError
.. autoexception:: coro.Shutdown
.. autoexception:: coro.WakeUp
docs/ref/debugging.rst 0 → 100644
View file @ e921e61e
=========
Debugging
=========
There are a variety of features available to help with debugging in Shrapnel.
Backdoor
========
A very powerful feature of Shrapnel is the ability to access a running process
via a backdoor. You can telnet to a socket (typically a unix-domain socket)
and get a Python prompt. At this point, you can interact with anything in
your Shrapnel process.
As an example of something you can do in the backdoor is call
:func:`coro.where_all`. This will return a dictionary of every coroutine that
is running with a string describing the call stack of where that coroutine is
currently blocked.
To enable the backdoor, you typically start a backdoor coroutine before starting
the event loop with the following code:
.. sourcecode:: python
import coro.backdoor
coro.spawn(coro.backdoor.serve)
By default this will listen on all IP's on the lowest port available from 8023
to 8033. This isn't a very safe or secure thing to do. It's best to specify a
unix-domain socket with the ``unix_path`` parameter. See
:func:`coro.backdoor.serve` for details.
By default, the globals available in a backdoor session is a copy of the
globals from your applications ``__main__`` module.
.. autofunction:: coro.backdoor.serve
Stderr Output
=============
Shrapnel provides some functions for printing debug information to stderr. The
:func:`coro.print_stderr` function will print a string with a timestamp and
the thread number. The :func:`coro.write_stderr` function writes the string
verbatim with no newline.
Shrapnel keeps a reference to the "real" stderr (in ``saved_stderr``) and the
``print_stderr`` and ``write_stderr`` functions always use the real stderr
value. A particular reason for doing this is the backdoor module replaces
sys.stderr and sys.stdout, but we do not want debug output to go to the
interactive session.
.. autofunction:: coro.write_stderr
.. autofunction:: coro.print_stderr
Exceptions
==========
Tracebacks
----------
As a convenience, Shrapnel has a module for printing stack traces in a
condensed format. The ``coro.tb`` module has the :func:`coro.tb.stack_string`
function for printing the current stack, and :func:`coro.tb.traceback_string`
for getting a traceback in an exception handler.
.. autofunction:: coro.tb.stack_string
.. autofunction:: coro.tb.traceback_string
Exception Notifications
-----------------------
If an exception is raised in a coroutine and is never caught, then Shrapnel
will by default display the exception to stderr. If you want to change this
behavior, use :func:`coro.set_exception_notifier`.
.. autofunction:: coro.set_exception_notifier
Latency
=======
Shrapnel will keep track of how long a coroutine runs before it yields.
This is helpful to track down coroutines which are running for too long, or are
potentially calling blocking calls. Here is an example of the output that would
be sent to stderr when this happens::
Sat Apr 14 20:55:39 2012 High Latency: (3.884s)
for <coro #1 name='<function my_func at 0x800fd32a8>'
dead=0 started=1 scheduled=0 at 0x801424720>
You can change the threshold that will trigger this warning with the
:func:`coro.set_latency_warning` function. However, doing this to silence
warnings isn't a good idea. It is best to fix whatever code is causing the
warnings. You can either call :func:`coro.yield_slice` periodically to let
other coroutines run, or make sure you are not calling any blocking
operations.
.. autofunction:: coro.set_latency_warning
Functions
=========
The ``coro`` module defines the following functions:
.. autofunction:: coro.where
.. autofunction:: coro.where_all
.. autofunction:: coro.get_live_coros
docs/ref/dns.rst 0 → 100644
View file @ e921e61e
===
DNS
===
TODO
.. autofunction:: coro.set_resolver
docs/ref/emulation.rst 0 → 100644
View file @ e921e61e
=========
Emulation
=========
Because Shrapnel is essentially its own threading system, code written with
the intention of using Python's standard threads will not work. Things like
Python's socket class will block and hang the entire program. To solve this
problem, Shrapnel includes some code that will monkeypatch some of Python's
standard classes to work with Shrapnel. You must manually enable this
behavior by calling :func:`coro.install_thread_emulation`.
.. autofunction:: coro.install_thread_emulation
docs/ref/index.rst 0 → 100644
View file @ e921e61e
*************************
Shrapnel Reference Manaul
*************************
This reference manual describes all of the basic concepts of Shrapnel along with all of the APIs.
.. toctree::
:maxdepth: 1
:numbered:
coroutines.rst
sockets.rst
synchronization.rst
clocks.rst
dns.rst
emulation.rst
debugging.rst
signals.rst
selfishness.rst
profiling.rst
oserrors.rst
docs/ref/oserrors.rst 0 → 100644
View file @ e921e61e
========
OSErrors
========
As a convenience, Shrapnel wraps all OSError exceptions that it raises with a
subclass that is specific to the errno code. For example, an OSError with an
errno of ENOENT will be raised as the ENOENT exception. All exceptions derive
from OSError, so it is compatible with regular OSError handling.
All of the exceptions are defined in the ``coro.oserrors`` module.
For example, instead of doing this:
.. sourcecode:: python
try:
data = sock.recv(1024)
except OSError, e:
if e.errno == errno.ECONNRESET:
# Handle connection reset.
else:
# Handle other unknown error.
You can do this:
.. sourcecode:: python
try:
data = sock.recv(1024):
except ECONNRESET:
# Handle connection reset.
docs/ref/profiling.rst 0 → 100644
View file @ e921e61e
Profiling
=========
.. automodule:: coro.profiler
docs/ref/selfishness.rst 0 → 100644
View file @ e921e61e
===========
Selfishness
===========
Shrapnel maintains a concept called "selfishness". This mechanism is used to
prevent a coroutine from yielding too often (or from running for too long).
This is currently only relevant to socket objects and socket I/O.
Each coroutine is given a set number of "free passes" each time it tries to do
I/O on a socket. If there is data immediately available on the socket, then
the coroutine may immediately receive that data. If Shrapnel did not
implement any "selfishness" limits, and that coroutine is in a loop repeatedly
calling ``read`` and there is always data available to the socket, then that
coroutine would run continuously without letting its fellow coroutines a
chance to run.
By default, every coroutine has a selfishness limit of 4. That means it is
allowed to do 4 I/O operations before it is forced to yield. Of course, if it
attempts to do an I/O operation that would block (such as if there is no data
available on a socket), then it will yield immediately.
You can set the default selfishness limit for all new coroutines with the
:func:`coro.set_selfishness` function. You can also change a coroutine's
limit with the :meth:`coro.coro.set_max_selfish_acts` method.
Functions
=========
The following functions are available in the ``coro`` module:
.. autofunction:: coro.set_selfishness
docs/ref/signals.rst 0 → 100644
View file @ e921e61e
=======
Signals
=======
Shrapnel provides a way to handle signals. Youn can register a function to
receive signals with :func:`coro.signal_handler.register`.
By default when you start the event loop, two signal handlers are installed
(for SIGTERM and SIGINT). The default signal handler will exit the event loop.
You can change this behavior by setting ``coro.install_signal_handlers`` to False
before starting the event loop.
Additionally, there is a signal handler installed for SIGINFO. It prints the
name of the coroutine that is currently running. On a typical terminal, you
can trigger this with CTRL-T.
.. autofunction:: coro.signal_handler.register
docs/ref/sockets.rst 0 → 100644
View file @ e921e61e
=======
Sockets
=======
Most Shrapnel programs make heavy use of sockets. The ``coro`` package
implements its own socket class, which is nearly identical to the socket class
in Python. Indeed, if you use :func:`coro.install_thread_emulation` then the
socket class will be monkey-patched into Python's socket module.
Creating Sockets
================
Though you are free to directly instantiate the :class:`coro.sock` object, there are
a variety of functions to assist in creating socket objects with a little more clarity.
.. autofunction:: coro.tcp6_sock
.. autofunction:: coro.tcp_sock
.. autofunction:: coro.udp6_sock
.. autofunction:: coro.udp_sock
.. autofunction:: coro.unix_sock
.. autofunction:: coro.socketpair
.. autofunction:: coro.has_ipv6
Socket Classes
==============
.. autoclass:: coro.sock
.. autoclass:: coro.file_sock
.. autoclass:: coro.fd_sock
Socket Functions
================
The coro module offers the following functions related to sockets.
.. autofunction:: coro.get_live_sockets
Socket Constants
================
The following classes provide a variety of constants often used with sockets.
.. autoclass:: coro.AF
.. autoclass:: coro.PF
.. autoclass:: coro.SHUT
.. autoclass:: coro.SO
.. autoclass:: coro.SOCK
.. autoclass:: coro.SOL
docs/ref/synchronization.rst 0 → 100644
View file @ e921e61e
===============
Synchronization
===============
You typically do not need to use synchronization primitives with Shrapnel
because coroutines are cooperative. However, there are situations where they
can be useful. For example, if you manipulate multiple shared data structures
that need to remain consistent, and you have potentially context-switch calls
interspersed (such as socket I/O).
Synchronization Classes
=======================
.. autoclass:: coro.condition_variable
.. autoclass:: coro.fifo
.. autoclass:: coro.inverted_semaphore
.. autoclass:: coro.LockError
.. autoclass:: coro.mutex
.. autoclass:: coro.semaphore
docs/shrapnel.svg 0 → 100644
View file @ e921e61e
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Created with Inkscape (http://www.inkscape.org/) -->
<svg
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
width="1052.3622"
height="744.09448"
id="svg2"
version="1.1"
inkscape:version="0.48.1 r9760"
sodipodi:docname="shrike.svg">
<defs
id="defs4">
<inkscape:perspective
sodipodi:type="inkscape:persp3d"
inkscape:vp_x="2.6642081 : 450.19734 : 1"
inkscape:vp_y="0 : 1000 : 0"
inkscape:vp_z="1047.9219 : 410.23422 : 1"
inkscape:persp3d-origin="526.18109 : 248.03149 : 1"
id="perspective3001" />
</defs>
<sodipodi:namedview
id="base"
pagecolor="#ffffff"
bordercolor="#666666"
borderopacity="1.0"
inkscape:pageopacity="0.0"
inkscape:pageshadow="2"
inkscape:zoom="1.187804"
inkscape:cx="526.18109"
inkscape:cy="372.04724"
inkscape:document-units="px"
inkscape:current-layer="layer1"
showgrid="false"
inkscape:window-width="1618"
inkscape:window-height="1252"
inkscape:window-x="103"
inkscape:window-y="110"
inkscape:window-maximized="0" />
<metadata
id="metadata7">
<rdf:RDF>
<cc:Work
rdf:about="">
<dc:format>image/svg+xml</dc:format>
<dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
<dc:title></dc:title>
</cc:Work>
</rdf:RDF>
</metadata>
<g
inkscape:label="Layer 1"
inkscape:groupmode="layer"
id="layer1"
transform="translate(0,-308.2677)">
<rect
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect3913"
width="41.102364"
height="47.480316"
x="17.716541"
y="473.38577" />
<rect
style="fill:#b3b3b3;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.14044118;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect3911"
width="41.10236"
height="52.440941"
x="17.007874"
y="691.6535" />
<rect
style="fill:#b3b3b3;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.22939622;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect3017"
width="160.49298"
height="541.52734"
x="69.828453"
y="402.52829"
ry="0" />
<rect
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.22939622;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect3029"
width="149.5751"
height="169.22731"
x="75.287399"
y="409.07901" />
<rect
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.22939622;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect3031"
width="137.56543"
height="39.304409"
x="81.292229"
y="416.72153" />
<rect
y="462.00809"
x="81.292229"
height="39.304409"
width="137.56543"
id="rect3035"
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.22939622;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.84482175;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect3039"
width="137.56543"
height="18.560415"
x="81.292229"
y="506.77145" />
<rect
y="529.69904"
x="80.200439"
height="39.304409"
width="139.74901"
id="rect3043"
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.23911488;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
ry="0"
y="405.80365"
x="287.49896"
height="541.52734"
width="160.49298"
id="rect3813"
style="fill:#b3b3b3;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.22939622;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
y="411.2626"
x="292.95792"
height="148.48332"
width="149.5751"
id="rect3815"
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.15158355;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
y="417.81335"
x="298.96277"
height="19.652199"
width="137.56543"
id="rect3817"
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.86931431;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.22939622;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect3819"
width="137.56543"
height="39.304409"
x="298.96277"
y="443.44769" />
<rect
y="488.21106"
x="298.96277"
height="57.864822"
width="137.56543"
id="rect3821"
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.4916898;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.22939622;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect3845"
width="149.5751"
height="169.22731"
x="632.32953"
y="329.88934" />
<rect
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.22939622;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect3847"
width="137.56543"
height="39.304409"
x="638.33441"
y="337.53186" />
<rect
y="382.81845"
x="638.33441"
height="39.304409"
width="137.56543"
id="rect3849"
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.22939622;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.84482175;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect3851"
width="137.56543"
height="18.560415"
x="638.33441"
y="427.58176" />
<rect
y="450.50937"
x="637.24261"
height="39.304409"
width="139.74901"
id="rect3853"
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.23911488;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
y="469.14532"
x="805.21356"
height="61.140198"
width="149.5751"
id="rect3855"
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.73895848;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
y="476.78784"
x="811.21838"
height="25.111151"
width="137.56543"
id="rect3857"
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.98266286;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.67957491;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect3859"
width="137.56543"
height="12.0097"
x="812.31018"
y="507.8811" />
<rect
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.02145195;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect3865"
width="149.5751"
height="116.82144"
x="825.68091"
y="584.77936" />
<rect
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.00379777;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect3867"
width="137.56543"
height="26.202938"
x="831.68573"
y="592.42194" />
<rect
y="623.5152"
x="831.68573"
height="53.497658"
width="137.56543"
id="rect3869"
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.43429542;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.61469811;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect3871"
width="137.56543"
height="9.8261023"
x="831.68573"
y="682.47186" />
<text
xml:space="preserve"
style="font-size:49.1758461px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="149.66675"
y="640.53827"
id="text3875"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
x="149.66675"
y="640.53827"
id="tspan3898"
style="text-align:center;text-anchor:middle">main</tspan><tspan
sodipodi:role="line"
x="149.66675"
y="702.00806"
style="text-align:center;text-anchor:middle"
id="tspan4014">stack</tspan></text>
<text
xml:space="preserve"
style="font-size:49.1758461px;font-style:normal;font-weight:normal;text-align:center;line-height:125%;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="366.40079"
y="642.7218"
id="text3879"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3881"
x="366.40079"
y="642.7218">coro</tspan><tspan
sodipodi:role="line"
x="366.40079"
y="704.19159"
id="tspan3902">stack</tspan></text>
<text
xml:space="preserve"
style="font-size:24.58792305px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="672.9906"
y="365.34204"
id="text3906"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3908"
x="672.9906"
y="365.34204">coro#0</tspan></text>
<text
xml:space="preserve"
style="font-size:24.58792305px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="846.93707"
y="497.5014"
id="text3910"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3912"
x="846.93707"
y="497.5014">coro#1</tspan></text>
<text
xml:space="preserve"
style="font-size:24.58792305px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="866.41394"
y="613.68134"
id="text3914"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3916"
x="866.41394"
y="613.68134">coro#2</tspan></text>
<text
xml:space="preserve"
style="font-size:19.67033958px;font-style:normal;font-weight:normal;text-align:center;line-height:125%;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="260.99069"
y="972.79584"
id="text3918"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3920"
x="260.99069"
y="972.79584">swap() switches between running on the main</tspan><tspan
sodipodi:role="line"
x="260.99069"
y="997.38379"
id="tspan3922">stack and the coro stack.</tspan></text>
<text
xml:space="preserve"
style="font-size:17.21154594px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="234.41489"
y="634.64331"
id="text3924"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3926"
x="234.41489"
y="634.64331">swap()</tspan></text>
<path
id="path3930"
d="m 276.09164,651.43827 -6.15489,-4.89396 c -0.007,1.44704 -0.009,1.81022 -0.0131,2.53459 l -24.37018,0 c -0.004,-0.72445 -0.007,-1.08979 -0.0131,-2.5375 l -6.1549,4.89687 6.1549,4.89396 c 0.005,-1.54238 0.009,-1.97524 0.0131,-2.69227 l 24.37018,0 c 0.004,0.71703 0.009,1.14989 0.0131,2.69227 l 6.15489,-4.89396 z"
style="fill:#000000;fill-opacity:1;stroke:#000000;stroke-width:0.10477623;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1"
inkscape:connector-curvature="0" />
<text
xml:space="preserve"
style="font-size:24.58792305px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="102.32178"
y="490.02835"
id="text3932"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3934"
x="102.32178"
y="490.02835">scheduler</tspan></text>
<text
xml:space="preserve"
style="font-size:39.34067917px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="573.12317"
y="567.94855"
id="text3940"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3942"
x="573.12317"
y="567.94855">stack slice copies</tspan></text>
<text
xml:space="preserve"
style="font-size:24.58792305px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="605.67572"
y="602.10406"
id="text3944"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3946"
x="605.67572"
y="602.10406">(potentially millions)</tspan></text>
<text
sodipodi:linespacing="125%"
id="text3952"
y="471.25784"
x="333.23474"
style="font-size:24.58792305px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
xml:space="preserve"><tspan
y="471.25784"
x="333.23474"
id="tspan3954"
sodipodi:role="line">coro#n</tspan></text>
<text
xml:space="preserve"
style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="654.14832"
y="162.34866"
id="text3956"
sodipodi:linespacing="125%"
transform="translate(0,308.2677)"><tspan
sodipodi:role="line"
id="tspan3958"
x="654.14832"
y="162.34866" /></text>
<rect
y="814.81781"
x="516.99066"
height="169.22731"
width="149.5751"
id="rect3968"
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.22939622;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
y="822.46033"
x="522.99554"
height="39.304409"
width="137.56543"
id="rect3970"
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.22939622;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.22939622;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect3972"
width="137.56543"
height="39.304409"
x="522.99554"
y="867.74695" />
<rect
y="912.51025"
x="522.99554"
height="18.560415"
width="137.56543"
id="rect3974"
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.84482175;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:1.23911488;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect3976"
width="139.74901"
height="39.304409"
x="521.90375"
y="935.43787" />
<text
xml:space="preserve"
style="font-size:12px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="672.66986"
y="844.98657"
id="text3982"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3984"
x="672.66986"
y="844.98657">PyObject_Call (func=0x1002d8dd0, arg=0x1002eaec0, kw=0x0)</tspan></text>
<text
xml:space="preserve"
style="font-size:12px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="672.66986"
y="890.27319"
id="text3986"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3988"
x="672.66986"
y="890.27319">PyEval_CallObjectWithKeywords (func=0x1002d8dd0, ...)</tspan></text>
<text
xml:space="preserve"
style="font-size:12px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="672.66986"
y="924.66156"
id="text3990"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3992"
x="672.66986"
y="924.66156">PyEval_EvalFrameEx (f=0x10031f070, ...)</tspan></text>
<text
xml:space="preserve"
style="font-size:12px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="672.66986"
y="957.96118"
id="text3994"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3996"
x="672.66986"
y="957.96118">PyEval_EvalCodeEx (co=0x10040a7b0, ...)</tspan><tspan
sodipodi:role="line"
x="672.66986"
y="972.96118"
id="tspan3998" /></text>
<text
xml:space="preserve"
style="font-size:32px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="517.28607"
y="800.63715"
id="text4000"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan4002"
x="517.28607"
y="800.63715">stack frame</tspan></text>
<text
xml:space="preserve"
style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="109.74291"
y="926.07874"
id="text4004"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan4006"
x="109.74291"
y="926.07874">4MB</tspan></text>
<text
sodipodi:linespacing="125%"
id="text4008"
y="926.07874"
x="327.41342"
style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
xml:space="preserve"><tspan
y="926.07874"
x="327.41342"
id="tspan4010"
sodipodi:role="line">4MB</tspan></text>
<g
id="g4067"
transform="matrix(0.92221294,-0.38668242,0.38668242,0.92221294,-113.90628,266.46282)">
<path
inkscape:connector-curvature="0"
style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.20497175;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
d="m 471.24729,456.37726 31.18686,10.23254 -6.99407,-7.32144 90.71753,0 0,-6.33834 -90.57701,0 6.85355,-7.52256 -31.18686,10.9498 z"
id="path4026" />
<text
sodipodi:linespacing="125%"
id="text4032"
y="444.51779"
x="511.86896"
style="font-size:20px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
xml:space="preserve"><tspan
y="444.51779"
x="511.86896"
id="tspan4034"
sodipodi:role="line">resume</tspan></text>
</g>
<g
id="g3951">
<path
id="path4030"
d="m 604.83658,382.91221 -24.7521,21.55594 3.59606,-9.46514 -83.57577,35.28118 -2.46506,-5.83935 83.44631,-35.22653 -9.23962,-4.26491 32.99018,-2.04119 z"
style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.20497175;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
inkscape:connector-curvature="0" />
<text
transform="matrix(0.9212747,-0.38891248,0.38891248,0.9212747,0,0)"
sodipodi:linespacing="125%"
id="text4036"
y="576.13635"
x="317.1749"
style="font-size:20px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
xml:space="preserve"><tspan
y="576.13635"
x="317.1749"
id="tspan4038"
sodipodi:role="line">yield</tspan></text>
</g>
<g
id="g4048">
<flowRoot
style="fill:black;stroke:none;stroke-opacity:1;stroke-width:1px;stroke-linejoin:miter;stroke-linecap:butt;fill-opacity:1;font-family:Times New Roman;font-style:normal;font-weight:normal;font-size:12px;line-height:125%;letter-spacing:0px;word-spacing:0px;-inkscape-font-specification:Times New Roman"
id="flowRoot4040"
xml:space="preserve"><flowRegion
id="flowRegion4042"><rect
y="103.41638"
x="467.24881"
height="78.295746"
width="123.75779"
id="rect4044" /></flowRegion><flowPara
id="flowPara4046" /></flowRoot> </g>
<flowRoot
xml:space="preserve"
id="flowRoot4054"
style="fill:black;stroke:none;stroke-opacity:1;stroke-width:1px;stroke-linejoin:miter;stroke-linecap:butt;fill-opacity:1;font-family:Times New Roman;font-style:normal;font-weight:normal;font-size:12px;line-height:125%;letter-spacing:0px;word-spacing:0px;-inkscape-font-specification:Times New Roman"><flowRegion
id="flowRegion4056"><rect
id="rect4058"
width="140.59558"
height="90.082199"
x="460.5137"
y="93.313705" /></flowRegion><flowPara
id="flowPara4060" /></flowRoot> <rect
y="665.60077"
x="698.55554"
height="12.273481"
width="15.714643"
id="rect4072"
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.10731567;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.16018762;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect4074"
width="23.456882"
height="18.320341"
x="738.12439"
y="651.28864" />
<rect
y="720.32361"
x="701.9231"
height="22.529789"
width="28.846548"
id="rect4076"
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.1969938;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.13074268;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect4078"
width="19.145149"
height="14.952783"
x="733.073"
y="695.06689" />
<rect
y="691.69934"
x="772.64178"
height="10.958413"
width="14.030864"
id="rect4080"
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.0958171;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.29129282;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect4082"
width="42.655113"
height="33.314583"
x="766.7486"
y="722.84924" />
<rect
y="720.26825"
x="832.41602"
height="43.835133"
width="56.125347"
id="rect4084"
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.38328138;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
y="646.2373"
x="716.23523"
height="7.1422563"
width="9.1447563"
id="rect4086"
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.06244976;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
y="626.03192"
x="739.80817"
height="5.8271875"
width="7.4609771"
id="rect4088"
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.05095119;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.05095119;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect4090"
width="7.4609771"
height="5.8271875"
x="711.1839"
y="628.55762" />
<rect
y="633.60895"
x="695.18799"
height="3.3015182"
width="4.2271767"
id="rect4092"
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02886749;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
y="732.11011"
x="854.30518"
height="59.572937"
width="76.275627"
id="rect4094"
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.52088809;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
y="736.00745"
x="857.36731"
height="13.362153"
width="70.151306"
id="rect4096"
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.51188534;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.73141706;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect4098"
width="70.151306"
height="27.281059"
x="857.36731"
y="751.8634" />
<rect
y="781.92828"
x="857.36731"
height="5.010808"
width="70.151306"
id="rect4100"
style="fill:#ff6600;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.31346449;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<text
sodipodi:linespacing="125%"
id="text4102"
y="746.84863"
x="875.0769"
style="font-size:12.53857899px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
xml:space="preserve"><tspan
y="746.84863"
x="875.0769"
id="tspan4104"
sodipodi:role="line">coro#3</tspan></text>
<rect
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.04877656;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect4106"
width="7.1425371"
height="5.5784788"
x="679.19208"
y="653.81433" />
<rect
y="641.9082"
x="681.87097"
height="3.486217"
width="4.4636602"
id="rect4108"
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.03048244;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02031904;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect4110"
width="2.9753954"
height="2.3238494"
x="682.46625"
y="629.4068" />
<rect
y="625.23969"
x="687.52637"
height="2.3238494"
width="2.9753954"
id="rect4112"
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02031904;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02031904;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect4114"
width="2.9753954"
height="2.3238494"
x="698.53955"
y="625.83496" />
<rect
y="621.07251"
x="695.56305"
height="2.3238494"
width="2.9753954"
id="rect4116"
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02031904;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<rect
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02031904;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
id="rect4118"
width="2.9753954"
height="2.3238494"
x="714.91052"
y="618.09601" />
<rect
y="616.01245"
x="701.81378"
height="2.3238494"
width="2.9753954"
id="rect4120"
style="fill:#aa0000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02031904;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" />
<text
xml:space="preserve"
style="font-size:12px;font-style:normal;font-weight:normal;text-align:center;line-height:125%;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="36.372467"
y="486.14166"
id="text3125"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3127"
x="36.372467"
y="486.14166">Used</tspan><tspan
sodipodi:role="line"
x="36.372467"
y="501.14166"
id="tspan3129">Stack</tspan><tspan
sodipodi:role="line"
x="36.372467"
y="516.14166"
id="tspan3131">Area</tspan></text>
<text
xml:space="preserve"
style="font-size:12px;font-style:normal;font-weight:normal;text-align:center;line-height:125%;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="36.735748"
y="706.5354"
id="text3133"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3135"
x="36.735748"
y="706.5354">Unused</tspan><tspan
sodipodi:role="line"
x="36.735748"
y="721.5354"
id="tspan3137">Stack</tspan><tspan
sodipodi:role="line"
x="36.735748"
y="736.53546"
id="tspan3139">Area</tspan><tspan
sodipodi:role="line"
x="36.735748"
y="751.53546"
id="tspan3141" /></text>
<text
sodipodi:linespacing="125%"
id="text3938"
y="396"
x="109.33276"
style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
xml:space="preserve"><tspan
y="396"
x="109.33276"
id="tspan3940"
sodipodi:role="line">0MB</tspan></text>
<text
xml:space="preserve"
style="font-size:14px;font-style:normal;font-weight:normal;text-align:center;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="569.5769"
y="-32.028229"
id="text3942"
sodipodi:linespacing="125%"
transform="matrix(0,1,-1,0,0,0)"><tspan
sodipodi:role="line"
id="tspan3944"
x="569.5769"
y="-32.028229">Stack Growth</tspan></text>
<path
id="path3949"
d="m 34.496003,649.45247 -6.4113,-9.75108 4.61419,2.14152 0.15907,-28.63263 4.00522,-0.0572 -0.15883,28.58826 4.76555,-2.23095 -6.9739,9.94202 z"
style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.09153501;stroke-linecap:round;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none"
inkscape:connector-curvature="0" />
<text
xml:space="preserve"
style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Times New Roman;-inkscape-font-specification:Times New Roman"
x="327.0033"
y="395.29135"
id="text3957"
sodipodi:linespacing="125%"><tspan
sodipodi:role="line"
id="tspan3959"
x="327.0033"
y="395.29135">0MB</tspan></text>
</g>
</svg>
docs/style.css 0 → 100644
View file @ e921e61e
body {
margin-left: 1.0em;
margin-top: 1.0em;
background-color: #b8ccb8;
font-family: Helvetica, Arial, FreeSans, san-serif;
color: #00000;
}
container {
margin: 0 auto;
width: 700px;
}
h1 { font-size: 3.0em; color: #102750; margin-bottom: 3px; }
h1 .small { font-size: 0.4em; }
h1 a { text-decoration: none }
h2 { font-size: 1.5em; color: #102750; }
h3 { text-align: center; color: #102750; }
a { color: #102750; }
.description { font-size: 1.2em; margin-bottom: 30px; margin-top: 30px; font-style: italic;}
.download { float: right; }
pre { background: #fff; color: #000; padding: 15px;}
hr { border: 0; width: 80%; border-bottom: 1px solid #aaa}
.footer { text-align:center; padding-top:30px; font-style: italic; }
docs/tutorial.rst 0 → 100644
View file @ e921e61e
=============================
Getting Started with Shrapnel
=============================
:Date: $Date: 2012/03/30 $
:Author: Sam Rushing
.. contents::
:depth: 2
:backlinks: top
.. section-numbering::
Shrapnel/Coro is a highly scalable cooperative threading facility for CPython.
Intro
=====
Shrapnel is a library for high-performance concurrency. It uses
coroutines to implement user threads on top of either kqueue (FreeBSD,
OS X) or /dev/epoll (linux), and is written mostly in Pyrex/Cython,
supporting both 32-bit and 64-bit platforms. It is the culmination of
about 8 years of work at IronPort Systems, a provider of high-speed
mail appliances. It was open-sourced by Cisco Systems in late 2011.
Features
--------
* Lightweight threads, event-driven scheduler.
* Underneath: non-blocking operations on descriptors, like sockets and pipes.
* On top, synchronous API for straight-line, simple code.
* Highly scalable - tens or hundreds of thousands of connections/threads.
* Thread synchronization primitives, like mutexes, semaphores, etc...
* Wait on kqueue events like file/directory changes, signals, processes, etc... [kqueue only]
* DNS stub resolver (full-fledged resolver may be forthcoming)
* HTTP server and client
* RPC system
* Support for TLS via tlslite (openssl interface may be forthcoming)
* other protocols/codecs: ldap, asn1, ftp, mysql, postgres, AMQP_.
* `MIT License`_.
.. _MIT License: http://www.opensource.org/licenses/mit-license.html
.. _AMQP: https://github.com/samrushing/amqp-shrapnel
The Name
--------
`Shrapnel` is the name of the implementation. The python package that
it implements is ``coro``. It's actually the third implementation of
the coro package written at IronPort - previous versions were written
in C using a painful type of continuation-passing style, and relied on
variants of Stackless Python.
Installing
----------
Requirements:
* Cython (>=0.12.1)
* distribute (>=0.6.16)
Install with::
$ pip install coro
*or* alternatively download shrapnel from
https://github.com/ironport/shrapnel and do the usual ``setup.py`` procedure::
$ git clone git://github.com/ironport/shrapnel.git
$ cd shrapnel
$ python setup.py build
$ [sudo] python setup.py install
Basics
======
Event Loop
----------
Everything runs in the event loop. So the last thing your ``__main__`` script will do is always::
coro.event_loop()
Calling this function starts the scheduler, which enters a loop continually running threads that are ready to run, feeding new waitable events to the operating system, and waiting for events to be triggered. When an event is triggered, the thread waiting on that event is awakened.
Interaction
-----------
Because everything runs in the event loop, you can't interact with the normal python prompt. But you can interact via the 'backdoor'. The backdoor interface is crucial to development with shrapnel. It allows you to open one or more telnet sessions into the system and interact with a python prompt there. So a common pattern in __main__ is this:
.. sourcecode:: python
if __name__ == '__main__':
import coro
import coro.backdoor
coro.spawn (coro.backdoor.serve, unix_path='/tmp/xx.bd')
coro.event_loop()
Once this is running, open another terminal window and::
$ telnet /tmp/xx.bd
[It's important to use the full path, you can't cd into /tmp and ``"$telnet xx.bd"``.]
Go ahead and run ``shrapnel/docs/tutorial/t0.py`` now, we'll use it in the following demonstration::
$ cd shrapnel/docs/tutorial/
$ python t0.py
1: Sat Mar 31 16:28:44 2012 Backdoor started on unix socket /tmp/xx.bd
In another window::
$ telnet /tmp/xx.bd
Trying /tmp/xx.bd...
Connected to (null).
Escape character is '^]'.
Python 2.7.2 (default, Mar 10 2012, 12:30:07)
[GCC 4.2.1 Compatible Clang 3.1 (trunk 149115)]
Copyright (c) 2001-2011 Python Software Foundation.
All Rights Reserved. [etc]
>>>
Output
------
Note the behavior of I/O to sys.stdout/stderr when using the back door::
>>> print "Howdy"
Howdy
and::
>>> sys.stdout.write ("Hello\n")
Hello
>>>
i.e, they print to the terminal, not the main console.
To send debugging output to the *console*, use ``coro.write_stderr()`` and ``coro.print_stderr()``.
Skullduggery
------------
From the back door you can poke around in the internals of the system while it's running::
>>> coro.event_map
{<kevent_key filter=-1 ident=4>: <kevent_target status=1 index=0 target=<coro #1 name='<function serve at 0x1007e1de8>' dead=0 started=1 scheduled=0 at 0x1005c1500> flags=0>, ...}
>>>
>>>
>>> coro.where_all()
{1: ('<function serve at 0x1007e1de8>', <coro #1 name='<function serve at 0x1007e1de8>' dead=0 started=1 scheduled=0 at 0x1005c1500>, '[coro/backdoor.py serve|224]'), ...}
>>>
>>> player_db['annoying_guy23'].demote()
<player annoying_guy23 id=394203 level=peon>
>>>
Sleeping
--------
The sleep_relative() function allow you to put a thread to sleep for a time::
>>> coro.sleep_relative (5)
[5 second pause]
>>>
Starting a New Thread
---------------------
Start a new thread with ``coro.spawn()``::
>>> def thing():
... for x in range (10):
... coro.write_stderr ('%d\n' % (x,))
... coro.sleep_relative (1)
...
>>> coro.spawn (thing)
<coro #5 name='<function thing at 0x10152af50>' dead=0 started=0 scheduled=1 at 0x1005f97b0>
>>>
In the main window you should see a new digit printed every second.
``coro.spawn()`` takes a callable object, args, and keyword args:
.. sourcecode:: python
coro.spawn (fun0, arg0, arg1, kwd0=val0, kwd1=val1)
Starting A *Lot* of New Threads
-------------------------------
Start up 1000 threads::
>>> import random
>>> def thing():
... t = random.randrange (0, 15)
... coro.sleep_relative (t)
... coro.write_stderr ('*')
...
>>> for x in range (1000):
... coro.spawn (thing)
...
>>>
Over the next 15 seconds you should see groups of ``'*'`` characters sent to the main window.
Servers and Clients
===================
Echo Server
-----------
Creating a server is easy (see `docs/tutorial/t1.py`_):
.. sourcecode:: python
import coro
import coro.backdoor
def session (conn, addr):
while 1:
block = conn.recv (1000)
if not block:
break
else:
conn.send (block)
def serve (port=9000):
s = coro.tcp_sock()
s.bind (('', port))
s.listen (50)
while 1:
conn, addr = s.accept()
coro.spawn (session, conn, addr)
if __name__ == '__main__':
coro.spawn (coro.backdoor.serve, unix_path='/tmp/xx.bd')
coro.spawn (serve)
coro.event_loop()
.. _docs/tutorial/t1.py: tutorial/t1.py
You can telnet into that server::
$ telnet localhost 9000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
asdf
asdf
asdf
asdf
asdf
[Hit ``Ctrl-], c, <return>`` to close the connection]
Timeouts
--------
One of the nicer features of shrapnel is ``with_timeout()``. Use
``with_timeout()`` around any function call:
.. sourcecode:: python
result = thing.db.query ("SELECT * FROM CANDYBIN;")
with a 30-second timeout becomes:
.. sourcecode:: python
result = coro.with_timeout (30, thing.db.query, "SELECT * FROM CANDYBIN;")
If the function hasn't returned within 30 seconds, it will raise
``coro.TimeoutError``:
.. sourcecode:: python
try:
r = coro.with_timeout (30, function, arg0, arg1, ...)
except coro.TimeoutError:
coro.write_stderr ("Hey, that took too long!\n")
Multiple layers of timeouts work as expected.
.. note::
Try editing the echo server from above, adding a timeout to the
``conn.recv (1000)`` call. Make it so that it exits the loop and
closes the connection if nothing is typed within 10 seconds.
Exiting The Event Loop
----------------------
You can tell the system to exit::
>>> coro.set_exit()
>>> Connection closed by foreign host.
.. note::
Try editing the echo server from above, so that it'll exit the
event loop when it receives the string 'quit\\r\\n'.
Echo Client
-----------
It's difficult to really beat on that server with your own fingers (and telnet).
How about a client that'll exercise it a little (see `docs/tutorial/t2.py`_):
.. sourcecode:: python
import coro
def client (ip='127.0.0.1', port=9000):
global alive
alive += 1
try:
s = coro.tcp_sock()
s.connect ((ip, port))
for i in range (10):
s.send ('howdy there\r\n')
assert (s.recv_exact (13) == 'howdy there\r\n')
coro.write_stderr ('.')
s.close()
finally:
alive -= 1
if alive == 0:
coro.write_stderr ('\ndone.\n')
coro.set_exit()
if __name__ == '__main__':
alive = 0
for i in range (100):
coro.spawn (client)
coro.event_loop()
.. _docs/tutorial/t2.py: tutorial/t2.py
Hit Ctrl-C to exit.
You should just see a hundred dots in the main window. You might get connection reset errors if the listen() parameter in the server wasn't high enough. If so, you could put some calls to sleep_relative() in there to stagger the creation of the clients.
Proxy Server
------------
This is a handy little server that lets you 'spy' on protocols. It's
very handy when implementing protocols. See `docs/tutorial/proxy.py`_.
.. _docs/tutorial/proxy.py: tutorial/proxy.py
.. sourcecode:: python
import coro
W = coro.write_stderr
class session:
counter = 0
def __init__ (self, conn, addr, saddr):
self.conn = conn
self.addr = addr
self.saddr = saddr
self.id = session.counter
session.counter += 1
self.proxy = coro.tcp_sock()
self.proxy.connect (saddr)
coro.spawn (self.feed, self.conn, self.proxy, '<==')
coro.spawn (self.feed, self.proxy, self.conn, '==>')
def feed (self, c0, c1, dir):
try:
while 1:
block = c0.recv (1000)
W ('%s %d %r\n' % (dir, self.id, block))
if not block:
break
else:
c1.send (block)
finally:
c0.close()
def serve (saddr):
ip, port = saddr
s = coro.tcp_sock()
s.bind (('0.0.0.0', port + 9000))
s.listen (5)
while 1:
conn, caddr = s.accept()
coro.spawn (session, conn, caddr, saddr)
if __name__ == '__main__':
import sys
if len (sys.argv) < 3:
print 'Usage: %s <server-host> <server-port>' % sys.argv[0]
else:
coro.spawn (serve, (sys.argv[1], int (sys.argv[2])))
coro.event_loop()
Let's say you want to spy on an HTTP connection::
$ python proxy.py 72.52.84.226 80
The proxy works by adding 9000 to the port number you're connecting to.
Try this link: http://localhost:9080/tutorial_hello.html
Profiler
========
Shrapnel comes with an efficient and comprehensive profiler that
accounts for the resources used by each thread. On most platforms it
uses the RDTSC instruction to gather accurate timings with low
overhead. It profiles both Python and Cython code:
.. sourcecode:: python
import coro.profiler
#coro.event_loop()
coro.profiler.go (coro.event_loop)
By default it collects data from the ``rusage()`` facility and RDTSC,
see the documentation for details.
When the function you are profiling has exited, it will dump a binary
file containing the results (default: ``/tmp/coro_profile.bin``),
which you can post-process using the ``coro.print_profile`` module::
$ python coro/print_profile.py /tmp/coro_profile.bin > /tmp/p0.html
Pull that up in your browser, you'll find aggregate and non-aggregate tables, along with a call graph.
Non-Aggregate Timings
+------+--------------------+---------------+-------------------+---------------------+-------------------+------------------+---------------+---------+---------+--------+--------+-------------------------------------+
|calls | ticks |ticks/call | utime | utime/call | stime | stime/call | minflt | majflt | oublock | msgsnd | msgrcv | Function |
+======+====================+===============+===================+=====================+===================+==================+===============+=========+=========+========+========+=====================================+
|0 | 29552069532 | 29552069532 |0.022320 | 0.02232 | 0.031529 | 0.031529 | 0 | 0 | 0 | 0 | 0 | <wait> |
+------+--------------------+---------------+-------------------+---------------------+-------------------+------------------+---------------+---------+---------+--------+--------+-------------------------------------+
|4540 |299809652 (15.86%) | 66037 |0.118307 (16.70%) | 0.000026 | 0.013881 (10.17%) | 0.000003 | 2 (4.88%) | 0 | 0 | 0 | 0 | python/worms.py:move:141 |
+------+--------------------+---------------+-------------------+---------------------+-------------------+------------------+---------------+---------+---------+--------+--------+-------------------------------------+
|0 | 227071913 (12.01%) | 227071913 |0.093381 (13.18%) | 0.093381 | 0.010697 (7.84%) | 0.010697 | 3 (7.32%) | 0 | 0 | 0 | 0 | <main> |
+------+--------------------+---------------+-------------------+---------------------+-------------------+------------------+---------------+---------+---------+--------+--------+-------------------------------------+
|4783 |192461416 (10.18%) | 40238 |0.072527 (10.24%) | 0.000015 | 0.012591 (9.22%) | 0.000003 | 1 (2.44%) | 0 | 0 | 0 | 0 | python2.7/random.py:randrange:173 |
+------+--------------------+---------------+-------------------+---------------------+-------------------+------------------+---------------+---------+---------+--------+--------+-------------------------------------+
|2420 | 171570517 (9.08%) | 70896 |0.069622 (9.83%) | 0.000029 | 0.006048 (4.43%) | 0.000002 | 1 (2.44%) | 0 | 0 | 0 | 0 | python/worms.py:draw:185 |
+------+--------------------+---------------+-------------------+---------------------+-------------------+------------------+---------------+---------+---------+--------+--------+-------------------------------------+
From the call graph section::
__builtin__:dict.has_key -- ticks=15234 utime=6e-06 stime=1e-06
3/3 (100.0%) coro/__init__.py:spawn:337
3 __builtin__:dict.has_key
__builtin__:file.write -- ticks=263623 utime=4.1e-05 stime=7.5e-05
7/7 (100.0%) python/worms.py:status:230
7 __builtin__:file.write
__builtin__:len -- ticks=28508261 utime=0.010467 stime=0.002547
14/4884 (00.3%) python/worms.py:status:230
330/4884 (06.8%) python2.7/random.py:choice:272
4540/4884 (93.0%) python/worms.py:move:141
4884 __builtin__:len
[...]
Example of the full `profiler output`_. Note: each graph may be [re]sorted by clicking on a column header.
.. _profiler output: sample_profile.html
The Killer Demo
===============
See `docs/tutorial/worms.py`_ for a fun demo. Run the script from
one terminal, and telnet into it from another terminal with a nice
large window (your terminal needs to support ANSI escape codes).
Each worm is its own thread, and each socket client has a separate
view into the shared 'arena'. This demo can easily handle hundreds of
separate worms (though things tend to get crowded)::
+=========================================================================+
| **********d|
| 9 |
| 9 |
| 9 |
| 9 |
| 9 |
| 9 |
| 9 |
| 9 |
| 9 |
| 9 |
| 9 |
| |
| |
| b a |
| b a |
| b a |
| b a 1 |
| eeeeee b a 1 |
| e b a 1 |
| e b a1111111 |
| e b a1 |
| e b a |
| e b a |
| b a |
| f |
| f |
| f |
| f |
| f |
| ffffff |
+=========================================================================+
keys: [q]uit [r]edraw [n]ew [c]ull [l]engthen [h]offa
.. _docs/tutorial/worms.py: tutorial/worms.py
Here's the code controlling each worm's movement:
.. sourcecode:: python
def go (self):
try:
while not self.exit:
coro.sleep_relative (self.speed / 10000.0)
if random.randrange (0,20) == 10:
if not self.turn():
return
else:
nx, ny = self.update()
while self.arena[(nx,ny)] != ' ':
if not self.turn():
return
nx, ny = self.update()
self.move ((nx, ny))
finally:
self.arena.worms.remove (self)
To come: a separate tutorial on hardening servers against attack. I
think this would be a great example to work with.
Details
=======
Exceptions
----------
What happens when there's an unhandled exception in a thread?::
>>> def thing():
... return 1/0
...
>>> coro.spawn (thing)
<coro #205 name='<function thing at 0x1007e6758>' dead=0 started=0 scheduled=1 at 0x1005cf040>
>>>
You should see something like this in the main window::
205: Sat Mar 31 17:29:06 2012 thread 205 (<function thing at 0x1007e6758>): error
'(\'<coro.backdoor.backdoor instance at 0x1007decf8> thing|2\',
"<type \'exceptions.ZeroDivisionError\'>", \'integer division or modulo by zero\',
\'[_coro.pyx coro._coro._wrap1 (coro/_coro.c:8821)|800] [<coro.backdoor.backdoor instance at 0x1007decf8> thing|2]\')'
The default exception handler for a thread prints a timestamp, some info about the thread that crashed, and a compact, one-line traceback.
You can replace the default handler with ``coro.set_exception_notifier()``.
Latency Warnings
----------------
It's important that no thread monopolizes the CPU for too long. This
can happen if you inadvertently call a blocking system function (e.g.,
filesystem I/O). To assist you in finding bugs that do this, the
scheduler will print out a warning like this::
Wed Apr 4 00:29:01 2012 High Latency: (5.449s) for <coro #4 name='mp4 encoder' at 0x1003ceaa0>
Any thread that holds the CPU for more than 0.2s will trigger the
warning. You can change the trigger value with ``coro.set_latency_warning()``.
SimultaneousError
-----------------
If two threads try to perform the same I/O operation (technically,
wait on the same kevent), this will trigger a ``SimultaneousError``::
>>> coro.x.recv (100)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/coro/backdoor.py", line 144, in parse
result = eval (co, env)
File "<coro.backdoor.backdoor instance at 0x100624ef0>", line 1, in <module>
File "socket.pyx", line 580, in coro._coro.sock.recv (coro/_coro.c:20208)
File "socket.pyx", line 1113, in coro._coro.sock._wait_for_read (coro/_coro.c:23549)
File "poller.pyx", line 326, in coro._coro.queue_poller._wait_for_read (coro/_coro.c:15292)
File "poller.pyx", line 318, in coro._coro.queue_poller._wait_for_with_eof (coro/_coro.c:15204)
File "poller.pyx", line 342, in coro._coro.queue_poller._wait_for (coro/_coro.c:15516)
File "poller.pyx", line 304, in coro._coro.queue_poller.set_wait_for (coro/_coro.c:15056)
SimultaneousError: <SimultaneousError co=<coro #6 name='backdoor session' dead=0 started=1 scheduled=0 at 0x1003ceaa0> other=<coro #5 name='backdoor session' dead=0 started=1 scheduled=0 at 0x1003d0080> event=<kevent_key filter=-1 ident=0>>
>>>
You can easily avoid this problem by isolating particular events to
their own thread. For example, you can have one thread that *reads*
from a socket, while another *writes* to it. You can combine
identical events from multiple threads by using one of the
synchronization primitives. A common idiom uses a ``coro.fifo``:
.. sourcecode:: python
def writer (self):
while not self.exit:
data = self.fifo.pop()
if data is None:
break
else:
self.conn.send (data)
In this example we use a sentinel (``None``) to force the fifo to wake
up and exit the loop. This is similar to a generator's use of ``StopIteration``.
Things To Avoid
---------------
Blocking calls. Slow file I/O. Not closing descriptors. *Threads*. etc.
How it Works
------------
Shrapnel works by using two (or more) C stacks. The first stack (the
default one from libc) runs the scheduler, which is responsible for
switching out coro threads, managing the timed-events queue, and
calling kevent (or hitting /dev/epoll). The second stack is where
coro threads run. When it's time for a thread to run, its stack
contents are copied from the heap onto the second stack, and a small
amount of assembly code (similar to the ``ucontext`` facility)
resumes it. When a thread yields, the portion of the stack used by
that thread is evacuated into the heap.
.. image:: shrapnel.svg
This design allows Shrapnel to work with a completely stock CPython.
It has been used continuously with Python 2.3 to 2.7, and can usually
be linked as a shared library against the platform's OEM install of
Python.
Credits
=======
[Get a comprehensive list of everyone that's contributed to shrapnel,
maybe with home page links?]
..
Local Variables:
compile-command: "rst2html.py --embed-stylesheet --stylesheet-path=style.css tutorial.rst tutorial.html"
End:
docs/tutorial/proxy.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
import coro
W = coro.write_stderr
class session:
counter = 0
def __init__ (self, conn, addr, saddr):
self.conn = conn
self.addr = addr
self.saddr = saddr
self.id = session.counter
session.counter += 1
self.proxy = coro.tcp_sock()
self.proxy.connect (saddr)
coro.spawn (self.feed, self.conn, self.proxy, '<==')
coro.spawn (self.feed, self.proxy, self.conn, '==>')
def feed (self, c0, c1, dir):
try:
while 1:
block = c0.recv (1000)
W ('%s %d %r\n' % (dir, self.id, block))
if not block:
break
else:
c1.send (block)
finally:
c0.close()
def serve (saddr):
ip, port = saddr
s = coro.tcp_sock()
s.bind (('0.0.0.0', port + 9000))
s.listen (5)
while 1:
conn, caddr = s.accept()
coro.spawn (session, conn, caddr, saddr)
if __name__ == '__main__':
import sys
if len (sys.argv) < 3:
print 'Usage: %s <server-host> <server-port>' % sys.argv[0]
else:
coro.spawn (serve, (sys.argv[1], int (sys.argv[2])))
coro.event_loop()
docs/tutorial/t0.py 0 → 100644
View file @ e921e61e
import coro
import coro.backdoor
coro.spawn (coro.backdoor.serve, unix_path='/tmp/xx.bd')
coro.event_loop()
docs/tutorial/t1.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
import coro
import coro.backdoor
def session (conn, addr):
while 1:
block = conn.recv (1000)
if not block:
break
else:
conn.send (block)
def serve (port=9000):
s = coro.tcp_sock()
s.bind (('', port))
s.listen (50)
while 1:
conn, addr = s.accept()
coro.spawn (session, conn, addr)
if __name__ == '__main__':
coro.spawn (coro.backdoor.serve, unix_path='/tmp/xx.bd')
coro.spawn (serve)
coro.event_loop()
docs/tutorial/t2.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
import coro
def client (ip='127.0.0.1', port=9000):
global alive
alive += 1
try:
s = coro.tcp_sock()
s.connect ((ip, port))
for i in range (10):
s.send ('howdy there\r\n')
assert (s.recv_exact (13) == 'howdy there\r\n')
coro.write_stderr ('.')
s.close()
finally:
alive -= 1
if alive == 0:
coro.write_stderr ('\ndone.\n')
coro.set_exit()
if __name__ == '__main__':
alive = 0
for i in range (100):
coro.spawn (client)
coro.event_loop()
docs/tutorial/t3.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
import coro
import coro.backdoor
import sys
import curses
import pyximport
pyximport.install()
sys.path.append ('/Users/rushing/src/cython')
import newterm
def session (conn, addr):
while 1:
block = conn.recv (1000)
if not block:
break
else:
conn.send (block)
def serve (port=9000):
s = coro.tcp_sock()
s.bind (('', port))
s.listen (50)
while 1:
conn, addr = s.accept()
coro.spawn (session, conn, addr)
if __name__ == '__main__':
coro.spawn (coro.backdoor.serve, unix_path='/tmp/xx.bd')
coro.spawn (serve)
coro.event_loop()
docs/tutorial/worms.py 0 → 100644
View file @ e921e61e
# -*- Mode: Python -*-
# http://en.wikipedia.org/wiki/ANSI_escape_code
import array
import coro
import math
W = coro.write_stderr
CSI = '\x1B['
at_format = CSI + '%d;%dH%s' # + CSI + '0m'
def at (x, y, ch):
return at_format % (y, x, ch)
import random
class arena:
def __init__ (self, w=150, h=53):
self.w = w
self.h = h
self.data = [ array.array ('c', " " * w) for y in range (h) ]
# put some walls around the outside
for i in range (w):
self.data[0][i] = '='
self.data[-1][i] = '='
for j in range (h):
self.data[j][0] = '|'
self.data[j][-1] = '|'
self.data[0][0] = '+'
self.data[0][-1] = '+'
self.data[-1][0] = '+'
self.data[-1][-1] = '+'
self.worms = []
self.listeners = []
def random_pos (self):
while 1:
x = random.randrange (1, self.w-1)
y = random.randrange (1, self.h-1)
if self[x,y] == ' ':
break
return x, y
def render (self):
return '\n'.join (x.tostring() for x in self.data)
def __getitem__ (self, (x, y)):
return self.data[y][x]
def draw (self, pos, chr='*'):
x, y = pos
if self.data[y][x] in "=|+":
import pdb; pdb.set_trace()
self.data[y][x] = chr
for lx in self.listeners:
lx.draw (pos, chr)
def erase (self, pos):
x, y = pos
self.data[y][x] = ' '
for lx in self.listeners:
lx.erase (pos)
def populate (self, n=5):
for i in range (n):
x, y = self.random_pos()
self.worms.append (worm (self, x, y))
def cull (self, hoffa=False):
for worm in self.worms:
if worm.stunned:
W ('culling worm %r\n' % (worm.chr,))
worm.kill (hoffa)
class worm:
counter = 0
# up down left right
movex = [0,0,-1,1]
movey = [-1,1,0,0]
def __init__ (self, arena, x, y, length=10):
self.arena = arena
self.head = x, y
self.tail = []
self.dir = random.randrange (0, 4)
self.length = length
worm.counter += 1
self.chr = '0123456789abcdefghijklmnopqrstuvwxyz'[worm.counter%36]
self.speed = random.randrange (200, 400)
self.exit = False
self.stunned = False
W ('new worm %r @ %d, %d speed=%d\n' % (self.chr, x, y, self.speed))
coro.spawn (self.go)
def go (self):
try:
while not self.exit:
coro.sleep_relative (self.speed / 10000.0)
if random.randrange (0,20) == 10:
if not self.turn():
return
else:
nx, ny = self.update()
while self.arena[(nx,ny)] != ' ':
if not self.turn():
return
nx, ny = self.update()
self.move ((nx, ny))
finally:
self.arena.worms.remove (self)
def update (self):
x, y = self.head
return (
x + self.movex[self.dir],
y + self.movey[self.dir]
)
def turn (self):
while not self.exit:
x, y = self.head
a = self.arena
choices = []
for i in range (4):
nx = x + self.movex[i]
ny = y + self.movey[i]
if a[nx,ny] == ' ':
choices.append (i)
if not choices:
return self.stun()
else:
self.dir = random.choice (choices)
return True
def stun (self):
self.stunned = True
for pos in self.tail:
self.arena.draw (pos, '*')
coro.sleep_relative (5)
self.stunned = False
return not self.exit
def move (self, pos):
self.tail.append (self.head)
self.head = pos
self.arena.draw (pos, self.chr)
if len (self.tail) > self.length:
tpos = self.tail.pop (0)
self.arena.erase (tpos)
def kill (self, hoffa=True):
self.arena.erase (self.head)
for pos in self.tail:
if hoffa:
self.arena.draw (pos, '#')
else:
self.arena.erase (pos)
self.exit = True
self.head = (None, None)
self.tail = []
class terminal:
def __init__ (self, conn):
self.conn = conn
self.conn.send (
'\xff\xfc\x01' # IAC WONT ECHO
'\xff\xfb\x03' # IAC WILL SUPPRESS_GO_AHEAD
'\xff\xfc"' # IAC WONT LINEMODE
)
# turn off the cursor
self.conn.send (CSI + '?25l')
self.fifo = coro.fifo()
self.redraw()
self.t0 = coro.spawn (self.listen)
self.t1 = coro.spawn (self.writer)
self.exit = False
def redraw (self):
self.fifo.push (''.join ([
# clear the screen
CSI + '2J',
# move to home
CSI + '1;1H',
# draw the arena
the_arena.render(),
'\n keys: [q]uit [r]edraw [n]ew [c]ull [l]engthen [h]offa\n',
]))
def draw (self, (x, y), chr):
chr = (
CSI
+ '%dm' % (40+ord(chr)%8,)
+ CSI
+ '%dm' % (30+ord(chr)%7,)
+ chr
+ CSI
+ '0m'
)
self.fifo.push (at (x+1, y+1, chr))
def erase (self, (x, y)):
self.fifo.push (at (x+1, y+1, ' '))
def writer (self):
while not self.exit:
data = self.fifo.pop()
if data is None:
break
else:
self.conn.send (data)
def listen (self):
while not self.exit:
byte = self.conn.recv (1)
if byte == '\xff':
# telnet stuff, dump it
self.conn.recv (2)
elif byte == 'r':
self.redraw()
elif byte == 'n':
the_arena.populate (1)
elif byte == 'c':
the_arena.cull()
elif byte == 'l':
for worm in the_arena.worms:
worm.length += 1
elif byte == 'h':
the_arena.cull (hoffa=True)
elif byte == 'q':
self.exit = True
# turn the cursor back on...
self.conn.send (CSI + '?25h')
the_arena.listeners.remove (self)
self.conn.close()
self.fifo.push (None)
def status():
while 1:
coro.sleep_relative (2)
coro.write_stderr ('%5d worms %5d threads\n' % (len(the_arena.worms), len(coro.all_threads)))
def serve():
s = coro.tcp_sock()
s.bind (('', 9001))
s.listen (5)
while 1:
c, a = s.accept()
t = terminal (c)
the_arena.listeners.append (t)
if __name__ == '__main__':
import coro.backdoor
the_arena = arena()
the_arena.populate (10)
coro.spawn (status)
coro.spawn (serve)
coro.spawn (coro.backdoor.serve, unix_path='/tmp/xx.bd')
#import coro.profiler
#coro.profiler.go (coro.event_loop)
coro.event_loop()
setup.py
View file @ e921e61e
# $Header: //prod/main/ap/shrapnel/setup.py#17 $
#!/usr/bin/env python
from distutils.core import setup
from Cython.Distutils import build_ext
from Cython.Distutils.extension import Extension
#from Cython.Distutils import build_ext
#from Cython.Distutils.extension import Extension
import sys
import glob
import os
from distribute_setup import use_setuptools
use_setuptools()
from setuptools import setup
try:
from Cython.Distutils import build_ext
from Cython.Distutils.extension import Extension
except ImportError:
sys.stderr.write (
'\nThe Cython compiler is required to build Shrapnel.\n'
' Try "pip install cython"\n'
' *or* "easy_install cython"\n'
)
sys.exit (-1)
include_dir = os.getcwd()
def newer(x, y):
......@@ -39,16 +49,25 @@ def check_lio():
setup (
name='coro',
version='1.0.0-000',
version='1.0.2-000',
description='IronPort Coroutine/Threading Library',
author='Sam Rushing, Eric Huss, IronPort Engineering',
author_email='sam-coro@rushing.nightmare.com',
license = "MIT",
url = "http://github.com/ironport/shrapnel",
ext_modules = [
Extension(
'coro.event_queue',
['coro/event_queue.pyx'],
language='c++',
depends=[os.path.join(include_dir, 'pyrex', 'python.pxi'),],
pyrex_include_dirs=[
os.path.join(include_dir, '.'),
os.path.join(include_dir, 'pyrex'),
],),
Extension (
'coro._coro',
['coro/_coro.pyx', 'coro/swap.c', 'coro/event_queue.cc'],
['coro/_coro.pyx', 'coro/swap.c'],
extra_compile_args = ['-Wno-unused-function'],
depends=(glob.glob('coro/*.pyx') +
glob.glob('coro/*.pxi') +
......@@ -76,15 +95,16 @@ setup (
# },
# to enable LZO|LZ4 for stack compression, set COMPILE_LZO|COMPILE_LZ4 in coro/_coro.pyx
# and uncomment one of the following:
#libraries=['lzo2']
#libraries=['lz4']
#libraries=['lzo2', 'z']
#libraries=['lz4', 'z'],
libraries=['z']
),
Extension(
'coro.oserrors',
['coro/oserrors.pyx', ],
),
Extension(
Extension ('coro.dns.packet', ['coro/dns/packet.pyx', ],),
Extension (
'coro.clocks.tsc_time',
['coro/clocks/tsc_time.pyx', ],
pyrex_include_dirs=[os.path.join(include_dir, 'pyrex')],
......@@ -94,13 +114,15 @@ setup (
],
),
],
packages=['coro', 'coro.clocks'],
packages=['coro', 'coro.clocks', 'coro.http', 'coro.dns'],
package_dir = {
'': 'coroutine',
# '': 'coroutine',
'coro': 'coro',
'coro.clocks': 'coro/clocks'
'coro.clocks': 'coro/clocks',
'coro.dns': 'coro/dns',
},
py_modules = ['backdoor', 'coro_process', 'coro_unittest'],
install_requires = ['cython>=0.12.1', 'pyrex>=0.9.8.6'],
py_modules = ['backdoor', 'coro.read_stream', 'coro_process', 'coro_unittest',],
download_url = 'http://github.com/ironport/shrapnel/tarball/master#egg=coro-1.0.2',
install_requires = ['Cython>=0.12.1', 'distribute>=0.6.16'],
cmdclass={'build_ext': build_ext},
)
test/test_event_queue.py 0 → 100644
View file @ e921e61e
# Copyright (c) 2002-2011 IronPort Systems and Cisco Systems
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
"""Unittests for event queue wrapper."""
__version__ = '$Revision: #1 $'
import unittest
import coro
import coro_unittest
class Test(unittest.TestCase):
def setUp(self):
self.q = coro.event_queue()
def test_insert(self):
data = [(3, "3"), (2, "21"), (1, "1"), (2, "22")]
res = ["1", "21", "22", "3"]
for i in data:
self.q.insert(*i)
self.assertEquals(len(data), len(self.q))
for j in res:
self.assertEquals(self.q.top(), j)
self.assertEquals(self.q.pop(), j)
def test_remove(self):
data = [(3, "3"), (2, "21"), (1, "1"), (2, "22")]
for i in data:
self.q.insert(*i)
self.assertRaises(IndexError, self.q.remove, 1, "2")
self.assertRaises(IndexError, self.q.remove, 10, "2")
for i in data:
self.q.remove(*i)
self.assertEquals(0, len(self.q))
def test_empty(self):
self.assertRaises(IndexError, self.q.top)
self.assertRaises(IndexError, self.q.pop)
self.assertRaises(IndexError, self.q.remove, 1, "2")
if __name__ == '__main__':
coro_unittest.run_tests()
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7