Merge branch 'master' of github.com:ironport/shrapnel

.gitignore

@@ -5,5 +5,7 @@ build/
 coro/_coro.[ch]
 coro/oserrors.[ch]
 coro/clocks/tsc_time.c
+coro/dns/packet.c
+coro/event_queue.cpp
 *.pyc
 *.so

README.rst

@@ -11,7 +11,7 @@ connections.
 API Documentation
 =================
 
-See http://ironport.github.com/api/shrapnel/frames.html
+See http://ironport.github.com/shrapnel/
 
 
 Short History Of Python Coroutine Implementations

coro/__init__.py

@@ -20,113 +20,7 @@
 
 # $Header: //prod/main/ap/shrapnel/coro/__init__.py#31 $
 
-"""Coroutine threading library.
-
-Introduction
-============
-Shrapnel is a cooperative threading library.
-
-Getting Started
-===============
-When your process starts up, you must spawn a thread to do some work, and then
-start the event loop.  The event loop runs forever processing events until the
-process exits.  An example::
-
-    import coro
-
-    def main():
-        print 'Hello world!'
-        # This will cause the process to exit.
-        coro.set_exit(0)
-
-    coro.spawn(main)
-    coro.event_loop()
-
-Coroutines
-==========
-Every coroutine thread is created with either the `new` function (which does
-NOT automatically start the thread) or the `spawn` function (which DOES
-automatically start it).
-
-Every thread has a unique numeric ID.  You may also set the name of the thread
-when you create it.
-
-Timeouts
-========
-The shrapnel timeout facility allows you to execute a function which will be
-interrupted if it does not finish within a specified period of time.  The
-`coro.TimeoutError` exception will be raised if the timeout expires.  See the
-`with_timeout` docstring for more detail.
-
-If the event loop is not running (such as in a non-coro process), a custom
-version of `with_timeout` is installed that will operate using SIGALRM so that
-you may use `with_timeout` in code that needs to run in non-coro processes
-(though this is not recommended and should be avoided if possible).
-
-Thread Local Storage
-====================
-There is a thread-local storage interface available for storing global data that
-is thread-specific.  You instantiate a `ThreadLocal` instance and you can
-assign attributes to it that will be specific to that thread.  See the
-`ThreadLocal` docs for more detail.
-
-Signal Handlers
-===============
-By default when you start the event loop, two signal handlers are installed
-(for SIGTERM and SIGINT). The default signal handler will exit the event loop.
-You can change this behavior by setting `install_signal_handlers` to False
-before starting the event loop.
-
-See `coro.signal_handler` for more detail on setting coro signal handlers.
-
-Selfishness
-===========
-Certain socket operations are allowed to try to execute without blocking if
-they are able to (such as send/receiving data on a local socket or on a
-high-speed network). However, there is a limit to the number of times a thread
-is allowed to do this. The default is 4.  The default may be changed
-(`set_selfishness`) and the value on a per-thread may be changed
-(`coro.coro.set_max_selfish_acts`).
-
-Time
-====
-Shrapnel uses the `tsc_time` module for handling time.  It uses the TSC
-value for a stable and high-resolution unit of time.  See that module's
-documentation for more detail.
-
-A thread is always created when you start the event loop that will
-resynchronize the TSC relationship to accomodate any clock drift (see
-`tick_updater` and `tsc_time.update_time_relation`).
-
-Exception Notifier
-==================
-When a thread exits due to an exception, by default a stack trace is printed to
-stderr.  You may install your own callback to handle this situation.  See the
-`set_exception_notifier` function for more detail.
-
-Debug Output
-============
-The shrapnel library provides a mechanism for printing debug information to
-stderr.  The `print_stderr` function will print a string with a timestamp
-and the thread number.  The `write_stderr` function writes the string verbatim.
-
-Shrapnel keeps a reference to the "real" stderr (in `saved_stderr`) and the
-`print_stderr` and `write_stderr` functions always use the real stderr value. A
-particular reason for doing this is the backdoor module replaces sys.stderr and
-sys.stdout, but we do not want debug output to go to the interactive session.
-
-Profiling
-=========
-Shrapnel has its own profiler that is coro-aware.  See `coro.profiler` for
-details on how to run the profiler.
-
-:Variables:
-    - `all_threads`: A dictionary of all live coroutine objects.  The key is
-      the coroutine ID, and the value is the coroutine object.
-    - `saved_stderr`: The actual stderr object for the process.  This normally
-      should not be used.  An example of why this exists is because the
-      backdoor replaces sys.stderr while executing user code.
-"""
+"""Coroutine threading library."""
 
 from coro._coro import *
 from coro._coro import _yield
@@ -166,13 +60,12 @@ set_exception_notifier (default_exception_notifier)
 
 class InParallelError (Exception):
 
-    """An error occurred in the `in_parallel` function.
+    """An error occurred in the :func:`in_parallel` function.
 
-    :IVariables:
-        - `result_list`: A list of ``(status, result)`` tuples. ``status`` is
-          either `SUCCESS` or `FAILURE`.  For success, the result is the return
+    :ivar result_list: A list of ``(status, result)`` tuples. ``status`` is
+          either :data:`SUCCESS` or :data:`FAILURE`.  For success, the result is the return
           value of the function.  For failure, it is the output from
-          `sys.exc_info`.
+          ``sys.exc_info``.
     """
 
     def __init__(self, result_list):
@@ -195,17 +88,14 @@ def in_parallel (fun_arg_list):
 
     This will block until all functions have returned or raised an exception.
 
-    If one or more functions raises an exception, then the `InParallelError`
+    If one or more functions raises an exception, then the :exc:`InParallelError`
     exception will be raised.
 
-    :Parameters:
-        - `fun_arg_list`: A list of ``(fun, args)`` tuples.
+    :param fun_arg_list: A list of ``(fun, args)`` tuples.
 
-    :Return:
-        Returns a list of return values from the functions.
+    :returns: A list of return values from the functions.
 
-    :Exceptions:
-        - `InParallelError`: One or more of the functions raised an exception.
+    :raises InParallelError: One or more of the functions raised an exception.
     """
 #         InParallelError, [(SUCCESS, result0), (FAILURE, exc_info1), ...]
 
@@ -257,14 +147,11 @@ def tick_updater():
 def waitpid (pid):
     """Wait for a process to exit.
 
-    :Parameters:
-        - `pid`: The process ID to wait for.
+    :param pid: The process ID to wait for.
 
-    :Return:
-        Returns a tuple ``(pid, status)`` of the process.
+    :returns: A tuple ``(pid, status)`` of the process.
 
-    :Exceptions:
-        - `SimultaneousError`: Something is already waiting for this process
+    :raises SimultaneousError: Something is already waiting for this process
           ID.
     """
     if UNAME == "Linux":
@@ -290,14 +177,11 @@ def waitpid (pid):
 def get_thread_by_id (thread_id):
     """Get a coro thread by ID.
 
-    :Parameters:
-        - `thread_id`: The thread ID.
+    :param thread_id: The thread ID.
 
-    :Return:
-        Returns the coroutine object.
+    :returns: The coroutine object.
 
-    :Exceptions:
-        - `KeyError`: The coroutine does not exist.
+    :raises KeyError: The coroutine does not exist.
     """
     return all_threads[thread_id]
 
@@ -305,11 +189,9 @@ def where (co):
     """Return a string indicating where the given coroutine thread is currently
     running.
 
-    :Parameters:
-        - `co`: The coroutine object.
+    :param co: The coroutine object.
 
-    :Return:
-        Returns a string displaying where the coro thread is currently
+    :returns: A string displaying where the coro thread is currently
         executing.
     """
     f = co.get_frame()
@@ -318,8 +200,7 @@ def where (co):
 def where_all():
     """Get a dictionary of where all coroutines are currently executing.
 
-    :Return:
-        Returns a dictionary mapping the coroutine ID to a tuple of ``(name,
+    :returns: A dictionary mapping the coroutine ID to a tuple of ``(name,
         coro, where)`` where ``where`` is a string representing where the
         coroutine is currently running.
     """
@@ -339,13 +220,11 @@ def spawn (fun, *args, **kwargs):
 
     Additional arguments and keyword arguments will be passed to the given function.
 
-    :Parameters:
-        - `fun`: The function to call when the coroutine starts.
-        - `thread_name`: The name of the thread.  Defaults to the name of the
+    :param fun: The function to call when the coroutine starts.
+    :param thread_name: The name of the thread.  Defaults to the name of the
           function.
 
-    :Return:
-        Returns the new coroutine object.
+    :returns: The new coroutine object.
     """
     if kwargs.has_key('thread_name'):
         thread_name = kwargs['thread_name']
@@ -364,13 +243,11 @@ def new (fun, *args, **kwargs):
     This will not start the coroutine.  Call the ``start`` method on the
     coroutine to schedule it to run.
 
-    :Parameters:
-        - `fun`: The function to call when the coroutine starts.
-        - `thread_name`: The name of the thread.  Defaults to the name of the
+    :param fun: The function to call when the coroutine starts.
+    :param thread_name: The name of the thread.  Defaults to the name of the
           function.
 
-    :Return:
-        Returns the new coroutine object.
+    :returns: The new coroutine object.
     """
     if kwargs.has_key('thread_name'):
         thread_name = kwargs['thread_name']
@@ -457,8 +334,7 @@ event_loop_is_running = False
 def coro_is_running():
     """Determine if the coro event loop is running.
 
-    :Return:
-        Returns True if the event loop is running, otherwise False.
+    :returns: True if the event loop is running, otherwise False.
     """
     return event_loop_is_running
 
@@ -468,8 +344,7 @@ def sigterm_handler (*_unused_args):
 def event_loop (timeout=30):
     """Start the event loop.
 
-    :Parameters:
-        - `timeout`: The amount of time to wait for kevent to return
+    :param timeout: The amount of time to wait for kevent to return
           events. You should probably *not* set this value.
     """
     global event_loop_is_running, with_timeout, sleep_relative

coro/profiler.py

@@ -25,17 +25,18 @@
 Introduction
 ============
 This profiler is coro-aware.  It produces output to a binary file on disk. You
-then use the `coro.print_profile` module to convert it to an HTML file.
+then use the :mod:`coro.print_profile` module to convert it to an HTML file.
 
 Using The Profiler
 ==================
-There are two ways to run the profiler.  One is to use the `go` function where
-you give it a python function to run.  Profiling will start and call the
-function, and then the profiler will automatically stop when the function
-exits.
+There are two ways to run the profiler.  One is to use the
+:func:`coro.profiler.go` function where you give it a python function to run.
+Profiling will start and call the function, and then the profiler will
+automatically stop when the function exits.
 
-The other method is to call `start` to start the profiler and `stop` when you
-want to stop profiling.  This can be conveniently done from the backdoor.
+The other method is to call :func:`coro.profiler.start` to start the profiler
+and :func:`coro.profiler.stop` when you want to stop profiling.  This can be
+conveniently done from the backdoor.
 
 Rendering Output
 ================
@@ -49,12 +50,13 @@ Then view the profile output in your web browser.
 Profiler Types
 ==============
 The profiler supports different ways of gathering statistics.  This is done by
-specifying the "bench" object to use (see `go` and `start`).  They default to
-the "rusage" method of gathering statistics about every function call (see the
-getrusage man page for more detail).  If you want a higher performance profile,
-you can use the `coro.bench` object instead which simply records TSC values for
-every function call.  If you want to define your own method of gathering
-statistics, subclass `coro.bench` and implement your own techniques.
+specifying the "bench" object to use (see :func:`go` and :func:`start`).  They
+default to the "rusage" method of gathering statistics about every function
+call (see the getrusage man page for more detail).  If you want a higher
+performance profile, you can use the :class:`coro.bench` object instead which
+simply records TSC values for every function call.  If you want to define your
+own method of gathering statistics, subclass :class:`coro.bench` and implement
+your own techniques.
 
 """
 
@@ -98,14 +100,12 @@ def go (fun, *args, **kwargs):
 
     This will display the results to stdout after the function is finished.
 
-    :Parameters:
-        - `fun`: The function to call.
+    :param fun: The function to call.
 
-    :Keywords:
-        - `profile_filename`: The name of the file to save the profile data.
+    :keyword profile_filename: The name of the file to save the profile data.
           Defaults to '/tmp/coro_profile.bin'.
-        - `profile_bench`: The bench object type to use.  Defaults to
-          `coro.rusage_bench`.
+    :keyword profile_bench: The bench object type to use.  Defaults to
+          :class:`coro.rusage_bench`.
     """
     if kwargs.has_key('profile_filename'):
         profile_filename = kwargs['profile_filename']

docs/.gitignore

+_build/

docs/Makefile

+# Makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+PAPER         =
+BUILDDIR      = _build
+
+# Internal variables.
+PAPEROPT_a4     = -D latex_paper_size=a4
+PAPEROPT_letter = -D latex_paper_size=letter
+ALLSPHINXOPTS   = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
+# the i18n builder cannot share the environment and doctrees with the others
+I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
+
+.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext
+
+help:
+	@echo "Please use \`make <target>' where <target> is one of"
+	@echo "  html       to make standalone HTML files"
+	@echo "  dirhtml    to make HTML files named index.html in directories"
+	@echo "  singlehtml to make a single large HTML file"
+	@echo "  pickle     to make pickle files"
+	@echo "  json       to make JSON files"
+	@echo "  htmlhelp   to make HTML files and a HTML help project"
+	@echo "  qthelp     to make HTML files and a qthelp project"
+	@echo "  devhelp    to make HTML files and a Devhelp project"
+	@echo "  epub       to make an epub"
+	@echo "  latex      to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
+	@echo "  latexpdf   to make LaTeX files and run them through pdflatex"
+	@echo "  text       to make text files"
+	@echo "  man        to make manual pages"
+	@echo "  texinfo    to make Texinfo files"
+	@echo "  info       to make Texinfo files and run them through makeinfo"
+	@echo "  gettext    to make PO message catalogs"
+	@echo "  changes    to make an overview of all changed/added/deprecated items"
+	@echo "  linkcheck  to check all external links for integrity"
+	@echo "  doctest    to run all doctests embedded in the documentation (if enabled)"
+
+clean:
+	-rm -rf $(BUILDDIR)/*
+
+html:
+	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
+	@echo
+	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
+
+dirhtml:
+	$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
+	@echo
+	@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
+
+singlehtml:
+	$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
+	@echo
+	@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
+
+pickle:
+	$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
+	@echo
+	@echo "Build finished; now you can process the pickle files."
+
+json:
+	$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
+	@echo
+	@echo "Build finished; now you can process the JSON files."
+
+htmlhelp:
+	$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
+	@echo
+	@echo "Build finished; now you can run HTML Help Workshop with the" \
+	      ".hhp project file in $(BUILDDIR)/htmlhelp."
+
+qthelp:
+	$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
+	@echo
+	@echo "Build finished; now you can run "qcollectiongenerator" with the" \
+	      ".qhcp project file in $(BUILDDIR)/qthelp, like this:"
+	@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Shrapnel.qhcp"
+	@echo "To view the help file:"
+	@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Shrapnel.qhc"
+
+devhelp:
+	$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
+	@echo
+	@echo "Build finished."
+	@echo "To view the help file:"
+	@echo "# mkdir -p $$HOME/.local/share/devhelp/Shrapnel"
+	@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Shrapnel"
+	@echo "# devhelp"
+
+epub:
+	$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
+	@echo
+	@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
+
+latex:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo
+	@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
+	@echo "Run \`make' in that directory to run these through (pdf)latex" \
+	      "(use \`make latexpdf' here to do that automatically)."
+
+latexpdf:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo "Running LaTeX files through pdflatex..."
+	$(MAKE) -C $(BUILDDIR)/latex all-pdf
+	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
+
+text:
+	$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
+	@echo
+	@echo "Build finished. The text files are in $(BUILDDIR)/text."
+
+man:
+	$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
+	@echo
+	@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
+
+texinfo:
+	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
+	@echo
+	@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
+	@echo "Run \`make' in that directory to run these through makeinfo" \
+	      "(use \`make info' here to do that automatically)."
+
+info:
+	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
+	@echo "Running Texinfo files through makeinfo..."
+	make -C $(BUILDDIR)/texinfo info
+	@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
+
+gettext:
+	$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
+	@echo
+	@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
+
+changes:
+	$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
+	@echo
+	@echo "The overview file is in $(BUILDDIR)/changes."
+
+linkcheck:
+	$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
+	@echo
+	@echo "Link check complete; look for any errors in the above output " \
+	      "or in $(BUILDDIR)/linkcheck/output.txt."
+
+doctest:
+	$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
+	@echo "Testing of doctests in the sources finished, look at the " \
+	      "results in $(BUILDDIR)/doctest/output.txt."

docs/conf.py

+# -*- coding: utf-8 -*-
+#
+# Shrapnel documentation build configuration file, created by
+# sphinx-quickstart on Fri Apr 13 18:44:49 2012.
+#
+# This file is execfile()d with the current directory set to its containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+import sys, os
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#sys.path.insert(0, os.path.abspath('.'))
+
+# -- General configuration -----------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+#needs_sphinx = '1.0'
+
+# Add any Sphinx extension module names here, as strings. They can be extensions
+# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
+extensions = ['sphinx.ext.autodoc', 'sphinx.ext.todo']
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix of source filenames.
+source_suffix = '.rst'
+
+# The encoding of source files.
+#source_encoding = 'utf-8-sig'
+
+# The master toctree document.
+master_doc = 'index'
+
+# General information about the project.
+project = u'Shrapnel'
+copyright = u'2012, Sam Rushing'
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The short X.Y version.
+version = '1.0.2'
+# The full version, including alpha/beta/rc tags.
+release = '1.0.2'
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#language = None
+
+# There are two options for replacing |today|: either, you set today to some
+# non-false value, then it is used:
+#today = ''
+# Else, today_fmt is used as the format for a strftime call.
+#today_fmt = '%B %d, %Y'
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+exclude_patterns = ['_build']
+
+# The reST default role (used for this markup: `text`) to use for all documents.
+#default_role = None
+
+# If true, '()' will be appended to :func: etc. cross-reference text.
+#add_function_parentheses = True
+
+# If true, the current module name will be prepended to all description
+# unit titles (such as .. function::).
+#add_module_names = True
+
+# If true, sectionauthor and moduleauthor directives will be shown in the
+# output. They are ignored by default.
+#show_authors = False
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = 'sphinx'
+
+# A list of ignored prefixes for module index sorting.
+#modindex_common_prefix = []
+
+
+# -- Options for HTML output ---------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+html_theme = 'default'
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+html_theme_options = {'collapsiblesidebar': True}
+
+# Add any paths that contain custom themes here, relative to this directory.
+#html_theme_path = []
+
+# The name for this set of Sphinx documents.  If None, it defaults to
+# "<project> v<release> documentation".
+#html_title = None
+
+# A shorter title for the navigation bar.  Default is the same as html_title.
+#html_short_title = None
+
+# The name of an image file (relative to this directory) to place at the top
+# of the sidebar.
+#html_logo = None
+
+# The name of an image file (within the static path) to use as favicon of the
+# docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
+# pixels large.
+#html_favicon = None
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
+# using the given strftime format.
+#html_last_updated_fmt = '%b %d, %Y'
+
+# If true, SmartyPants will be used to convert quotes and dashes to
+# typographically correct entities.
+#html_use_smartypants = True
+
+# Custom sidebar templates, maps document names to template names.
+#html_sidebars = {}
+
+# Additional templates that should be rendered to pages, maps page names to
+# template names.
+#html_additional_pages = {}
+
+# If false, no module index is generated.
+#html_domain_indices = True
+
+# If false, no index is generated.
+#html_use_index = True
+
+# If true, the index is split into individual pages for each letter.
+#html_split_index = False
+
+# If true, links to the reST sources are added to the pages.
+#html_show_sourcelink = True
+
+# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
+#html_show_sphinx = True
+
+# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
+#html_show_copyright = True
+
+# If true, an OpenSearch description file will be output, and all pages will
+# contain a <link> tag referring to it.  The value of this option must be the
+# base URL from which the finished HTML is served.
+#html_use_opensearch = ''
+
+# This is the file name suffix for HTML files (e.g. ".xhtml").
+#html_file_suffix = None
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'Shrapneldoc'
+
+
+# -- Options for LaTeX output --------------------------------------------------
+
+latex_elements = {
+# The paper size ('letterpaper' or 'a4paper').
+#'papersize': 'letterpaper',
+
+# The font size ('10pt', '11pt' or '12pt').
+#'pointsize': '10pt',
+
+# Additional stuff for the LaTeX preamble.
+#'preamble': '',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title, author, documentclass [howto/manual]).
+latex_documents = [
+  ('index', 'Shrapnel.tex', u'Shrapnel Documentation',
+   u'Sam Rushing', 'manual'),
+]
+
+# The name of an image file (relative to this directory) to place at the top of
+# the title page.
+#latex_logo = None
+
+# For "manual" documents, if this is true, then toplevel headings are parts,
+# not chapters.
+#latex_use_parts = False
+
+# If true, show page references after internal links.
+#latex_show_pagerefs = False
+
+# If true, show URL addresses after external links.
+#latex_show_urls = False
+
+# Documents to append as an appendix to all manuals.
+#latex_appendices = []
+
+# If false, no module index is generated.
+#latex_domain_indices = True
+
+
+# -- Options for manual page output --------------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+    ('index', 'shrapnel', u'Shrapnel Documentation',
+     [u'Sam Rushing'], 1)
+]
+
+# If true, show URL addresses after external links.
+#man_show_urls = False
+
+
+# -- Options for Texinfo output ------------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+  ('index', 'Shrapnel', u'Shrapnel Documentation',
+   u'Sam Rushing', 'Shrapnel', 'One line description of project.',
+   'Miscellaneous'),
+]
+
+# Documents to append as an appendix to all manuals.
+#texinfo_appendices = []
+
+# If false, no module index is generated.
+#texinfo_domain_indices = True
+
+# How to display URL addresses: 'footnote', 'no', or 'inline'.
+#texinfo_show_urls = 'footnote'
+
+autodoc_default_flags = ['members', 'show-inheritance']

docs/index.rst

+.. Shrapnel documentation master file, created by
+   sphinx-quickstart on Fri Apr 13 18:44:49 2012.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to Shrapnel's documentation!
+====================================
+
+Contents:
+
+* :doc:`Installation <installation>`
+* :doc:`Tutorial <tutorial>`
+* :doc:`Reference Manual <ref/index>`
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
+

docs/installation.rst

+============
+Installation
+============
+
+Supported Platforms
+===================
+Shrapnel currently works on FreeBSD, Linux, and Mac OS X with x86 32- or 64-bit platforms. 
+It supports Python 2.7 (TODO: and 2.6?).
+
+Prerequisites
+=============
+pip
+---
+To make installation easy, you can use `pip <http://www.pip-installer.org/>`_.
+This is a tool which will fetch Python packages from `PyPi
+<http://pypi.python.org/>`_ and install them.
+
+Visit http://www.pip-installer.org/en/latest/installing.html for information
+on how to install pip if you don't already have it installed.
+
+Cython
+------
+You need version 0.12.1 or newer of `Cython <http://cython.org/>`_.  If you
+already have Cython installed, you can check your current version by running
+``cython -V``.
+
+To install Cython, run:
+
+	pip install cython
+
+Distribute
+----------
+You need version 0.6.16 or newer of `distribute <http://pypi.python.org/pypi/distribute>`_.
+Distribute is a build and packaging tool for Python (a replacement for setuptools).
+
+To install distribute, run:
+
+	pip install distribute
+
+Shrapnel
+--------
+Finally, you can install Shrapnel, run:
+
+	pip install shrapnel
+
+Alternatively you can download it from https://github.com/ironport/shrapnel
+and do the usual ``python setup.py install`` procedure.

docs/ref/clocks.rst

+======
+Clocks
+======
+
+Shrapnel needs to keep track of time to manage scheduling of sleeps and
+timeouts.  Because Shrapnel is intended to support thousands of coroutines,
+and each coroutine may be making many timeout calls per second, Shrapnel needs
+to use a timing facility that is relatively high performance.  It also needs
+one that is monotonic, so it does not need to deal with system clock changes.
+
+The ``clocks`` subpackage is intended to provide a variety of different time
+facilities. Currently it only supports using the x86 TSC timer.  This is a
+timer built in to the CPU, and thus is very fast.
+
+TSC Time
+========
+Support for TSC time is implemented in the ``coro.clocks.tsc_time`` module.
+
+.. automodule:: coro.clocks.tsc_time

docs/ref/coroutines.rst

+==========
+Coroutines
+==========
+
+The central concept of Shrapnel is the coroutine.  You can think of a coroutine
+like it is a thread.  When it runs out of work to do, it yields and allows other
+coroutines to run.  Scheduling of coroutines is handled by the scheduler which
+runs an "event loop".
+
+Event Loop
+==========
+
+The event loop is a loop that runs forever until the program ends.  Every
+Shrapnel program needs to start the event loop as one of the first things it
+does.  A typical example would be::
+
+    import coro
+
+    def main():
+        print 'Hello world!'
+        # This will cause the process to exit.
+        coro.set_exit(0)
+
+    if __name__ == '__main__':
+    	coro.spawn(main)
+    	coro.event_loop()
+
+Coroutines
+========== 
+
+Every coroutine thread is created with either the :func:`new` function (which
+does NOT automatically start the thread) or the :func:`spawn` function (which
+DOES automatically start it).
+
+Every thread has a unique numeric ID.  You may also set the name of the thread
+when you create it.
+
+.. autoclass:: coro.coro
+
+Timeouts
+========
+The shrapnel timeout facility allows you to execute a function which will be
+interrupted if it does not finish within a specified period of time.  The
+:class:`TimeoutError` exception will be raised if the timeout expires.  See the
+:func:`with_timeout` docstring for more detail.
+
+If the event loop is not running (such as in a non-coro process), a custom
+version of `with_timeout` is installed that will operate using SIGALRM so that
+you may use `with_timeout` in code that needs to run in non-coro processes
+(though this is not recommended and should be avoided if possible).
+
+.. autofunction:: coro.with_timeout
+
+Parallel Execution
+==================
+XXX
+
+.. autofunction:: coro.in_parallel
+.. autoexception:: coro.InParallelError
+
+Thread Local Storage
+====================
+There is a thread-local storage interface available for storing global data that
+is thread-specific.  You instantiate a :class:`ThreadLocal` instance and you can
+assign attributes to it that will be specific to that thread.  From a design
+perspective, it is generally discouraged to use thread-local storage.  But
+nonetheless, it can be useful at times.
+
+.. autoclass:: coro.ThreadLocal
+
+Functions
+=========
+The coro module defines the following functions:
+
+.. autofunction:: coro.get_thread_by_id
+.. autofunction:: coro.coro_is_running
+.. autofunction:: coro.event_loop
+.. autofunction:: coro.new
+.. autofunction:: coro.spawn
+.. autofunction:: coro.waitpid
+.. autofunction:: coro.yield_slice
+.. autofunction:: coro.schedule
+.. autofunction:: coro.current
+.. autofunction:: coro.set_exit
+.. autofunction:: coro.set_print_exit_string
+.. autofunction:: coro.sleep_absolute
+.. autofunction:: coro.sleep_relative
+
+Variables
+=========
+.. py:data:: coro.all_threads
+
+	A dictionary of all live coroutine objects.  The key is the coroutine ID,
+	and the value is the coroutine object.
+
+Exceptions
+==========
+The coro module defines the following exceptions:
+
+.. autoexception:: coro.ScheduleError
+.. autoexception:: coro.DeadCoroutine
+.. autoexception:: coro.ClosedError
+.. autoexception:: coro.NotStartedError
+.. autoexception:: coro.TimeoutError
+.. autoexception:: coro.SimultaneousError
+.. autoexception:: coro.Shutdown
+.. autoexception:: coro.WakeUp

docs/ref/debugging.rst

+=========
+Debugging
+=========
+
+There are a variety of features available to help with debugging in Shrapnel.
+
+Backdoor
+========
+
+A very powerful feature of Shrapnel is the ability to access a running process
+via a backdoor. You can telnet to a socket (typically a unix-domain socket)
+and get a Python prompt.  At this point, you can interact with anything in
+your Shrapnel process.
+
+As an example of something you can do in the backdoor is call
+:func:`coro.where_all`.  This will return a dictionary of every coroutine that
+is running with a string describing the call stack of  where that coroutine is
+currently blocked.
+
+To enable the backdoor, you typically start a backdoor coroutine before starting
+the event loop with the following code:
+
+.. sourcecode:: python
+
+    import coro.backdoor
+    coro.spawn(coro.backdoor.serve)
+
+By default this will listen on all IP's on the lowest port available from 8023
+to 8033. This isn't a very safe or secure thing to do.  It's best to specify a
+unix-domain socket with the ``unix_path`` parameter.  See
+:func:`coro.backdoor.serve` for details.
+
+By default, the globals available in a backdoor session is a copy of the
+globals from your applications ``__main__`` module.
+
+.. autofunction:: coro.backdoor.serve
+
+Stderr Output
+=============
+
+Shrapnel provides some functions for printing debug information to stderr. The
+:func:`coro.print_stderr` function will print a string with a timestamp and
+the thread number.  The :func:`coro.write_stderr` function writes the string
+verbatim with no newline.
+
+Shrapnel keeps a reference to the "real" stderr (in ``saved_stderr``) and the
+``print_stderr`` and ``write_stderr`` functions always use the real stderr
+value. A particular reason for doing this is the backdoor module replaces
+sys.stderr and sys.stdout, but we do not want debug output to go to the
+interactive session.
+
+.. autofunction:: coro.write_stderr
+.. autofunction:: coro.print_stderr
+
+Exceptions
+==========
+
+Tracebacks
+----------
+
+As a convenience, Shrapnel has a module for printing stack traces in a
+condensed format. The ``coro.tb`` module has the :func:`coro.tb.stack_string`
+function for printing the current stack, and :func:`coro.tb.traceback_string`
+for getting a traceback in an exception handler.
+
+.. autofunction:: coro.tb.stack_string
+.. autofunction:: coro.tb.traceback_string
+
+Exception Notifications
+-----------------------
+
+If an exception is raised in a coroutine and is never caught, then Shrapnel
+will by default display the exception to stderr.  If you want to change this
+behavior, use :func:`coro.set_exception_notifier`.
+
+.. autofunction:: coro.set_exception_notifier
+
+Latency
+=======
+
+Shrapnel will keep track of how long a coroutine runs before it yields.
+This is helpful to track down coroutines which are running for too long, or are
+potentially calling blocking calls.  Here is an example of the output that would
+be sent to stderr when this happens::
+
+    Sat Apr 14 20:55:39 2012 High Latency: (3.884s) 
+        for <coro #1 name='<function my_func at 0x800fd32a8>' 
+             dead=0 started=1 scheduled=0 at 0x801424720>
+
+You can change the threshold that will trigger this warning with the
+:func:`coro.set_latency_warning` function.  However, doing this to silence
+warnings isn't a good idea.  It is best to fix whatever code is causing the
+warnings.  You can either call :func:`coro.yield_slice` periodically to let
+other coroutines run, or make sure you are not calling any blocking
+operations.
+
+.. autofunction:: coro.set_latency_warning
+
+Functions
+=========
+The ``coro`` module defines the following functions:
+
+.. autofunction:: coro.where
+.. autofunction:: coro.where_all
+.. autofunction:: coro.get_live_coros

docs/ref/dns.rst

+===
+DNS
+===
+
+TODO
+
+.. autofunction:: coro.set_resolver
+

docs/ref/emulation.rst

+=========
+Emulation
+=========
+
+Because Shrapnel is essentially its own threading system, code written with
+the intention of using Python's standard threads will not work.  Things like
+Python's socket class will block and hang the entire program.  To solve this
+problem, Shrapnel includes some code that will monkeypatch some of Python's
+standard classes to work with Shrapnel.  You must manually enable this
+behavior by calling :func:`coro.install_thread_emulation`.
+
+.. autofunction:: coro.install_thread_emulation

docs/ref/index.rst

+*************************
+Shrapnel Reference Manaul
+*************************
+
+This reference manual describes all of the basic concepts of Shrapnel along with all of the APIs.
+
+.. toctree::
+	:maxdepth: 1
+	:numbered:
+
+	coroutines.rst
+	sockets.rst
+	synchronization.rst
+	clocks.rst
+	dns.rst
+	emulation.rst
+	debugging.rst
+	signals.rst
+	selfishness.rst
+	profiling.rst
+	oserrors.rst
+

docs/ref/oserrors.rst

+========
+OSErrors
+========
+
+As a convenience, Shrapnel wraps all OSError exceptions that it raises with a
+subclass that is specific to the errno code. For example, an OSError with an
+errno of ENOENT will be raised as the ENOENT exception.  All exceptions derive
+from OSError, so it is compatible with regular OSError handling.
+
+All of the exceptions are defined in the ``coro.oserrors`` module.
+
+For example, instead of doing this:
+
+.. sourcecode:: python
+
+    try:
+        data = sock.recv(1024)
+    except OSError, e:
+        if e.errno == errno.ECONNRESET:
+            # Handle connection reset.
+        else:
+            # Handle other unknown error.
+
+You can do this:
+
+.. sourcecode:: python
+
+    try:
+        data = sock.recv(1024):
+    except ECONNRESET:
+        # Handle connection reset.

docs/ref/profiling.rst

+Profiling
+=========
+
+.. automodule:: coro.profiler
+

docs/ref/selfishness.rst

+===========
+Selfishness
+===========
+
+Shrapnel maintains a concept called "selfishness". This mechanism is used to
+prevent a coroutine from yielding too often (or from running for too long).
+This is currently only relevant to socket objects and socket I/O.  
+
+Each coroutine is given a set number of "free passes" each time it tries to do
+I/O on a socket.  If there is data immediately available on the socket, then
+the coroutine may immediately receive that data.   If Shrapnel did not
+implement any "selfishness" limits, and that coroutine is in a loop repeatedly
+calling ``read`` and there is always data available to the socket, then that
+coroutine would run continuously without letting its fellow coroutines a
+chance to run.
+
+By default, every coroutine has a selfishness limit of 4.  That means it is
+allowed to do 4 I/O operations before it is forced to yield.  Of course, if it
+attempts to do an I/O operation that would block (such as if there is no data
+available on a socket), then it will yield immediately.
+
+You can set the default selfishness limit for all new coroutines with the
+:func:`coro.set_selfishness` function.  You can also change a coroutine's
+limit with the :meth:`coro.coro.set_max_selfish_acts` method.
+
+Functions
+=========
+The following functions are available in the ``coro`` module:
+
+.. autofunction:: coro.set_selfishness

docs/ref/signals.rst

+=======
+Signals
+=======
+
+Shrapnel provides a way to handle signals.  Youn can register a function to
+receive signals with :func:`coro.signal_handler.register`.
+
+By default when you start the event loop, two signal handlers are installed
+(for SIGTERM and SIGINT). The default signal handler will exit the event loop.
+You can change this behavior by setting ``coro.install_signal_handlers`` to False
+before starting the event loop.
+
+Additionally, there is a signal handler installed for SIGINFO.  It prints the
+name of the coroutine that is currently running.  On a typical terminal, you
+can trigger this with CTRL-T.
+
+.. autofunction:: coro.signal_handler.register

docs/ref/sockets.rst

+=======
+Sockets
+=======
+
+Most Shrapnel programs make heavy use of sockets.  The ``coro`` package
+implements its own socket class, which is nearly identical to the socket class
+in Python.  Indeed, if you use :func:`coro.install_thread_emulation` then the
+socket class will be monkey-patched into Python's socket module.
+
+Creating Sockets
+================
+Though you are free to directly instantiate the :class:`coro.sock` object, there are
+a variety of functions to assist in creating socket objects with a little more clarity.
+
+.. autofunction:: coro.tcp6_sock
+.. autofunction:: coro.tcp_sock
+.. autofunction:: coro.udp6_sock
+.. autofunction:: coro.udp_sock
+.. autofunction:: coro.unix_sock
+.. autofunction:: coro.socketpair
+.. autofunction:: coro.has_ipv6
+
+Socket Classes
+==============
+.. autoclass:: coro.sock
+.. autoclass:: coro.file_sock
+.. autoclass:: coro.fd_sock
+
+Socket Functions
+================
+The coro module offers the following functions related to sockets.
+
+.. autofunction:: coro.get_live_sockets
+
+Socket Constants
+================
+The following classes provide a variety of constants often used with sockets.
+
+.. autoclass:: coro.AF
+.. autoclass:: coro.PF
+.. autoclass:: coro.SHUT
+.. autoclass:: coro.SO
+.. autoclass:: coro.SOCK
+.. autoclass:: coro.SOL

docs/ref/synchronization.rst

+===============
+Synchronization
+===============
+You typically do not need to use synchronization primitives with Shrapnel
+because coroutines are cooperative. However, there are situations where they
+can be useful.  For example, if you manipulate multiple shared data structures
+that need to remain consistent, and you have potentially context-switch calls
+interspersed (such as socket I/O).
+
+Synchronization Classes
+=======================
+.. autoclass:: coro.condition_variable
+.. autoclass:: coro.fifo
+.. autoclass:: coro.inverted_semaphore
+.. autoclass:: coro.LockError
+.. autoclass:: coro.mutex
+.. autoclass:: coro.semaphore

setup.py

@@ -111,7 +111,7 @@ setup (
         ],
     packages=['coro', 'coro.clocks', 'coro.http', 'coro.dns'],
     package_dir = {
-        '': 'coroutine',
+#        '': 'coroutine',
         'coro': 'coro',
         'coro.clocks': 'coro/clocks',
         'coro.dns': 'coro/dns',