Merge pull request #57 from iodide-project/docs

Add some real documentation

README.md

@@ -4,47 +4,52 @@
 
 The Python scientific stack, compiled to WebAssembly.
 
-It provides transparent conversion of objects between Javascript and Python and
-a sharing of global namespaces. When inside a browser, this means Python has
-full access to the Web APIs.
+It provides transparent conversion of objects between Javascript and Python.
+When inside a browser, this means Python has full access to the Web APIs.
+
+**While closely related to the [iodide project](https://iodide.io), Pyodide may
+be used standalone in any context where you want to run Python inside a web
+browser.**
 
 See [the demo](https://iodide.io/pyodide-demo/python.html)
 
 # Building
 
-These instructions were tested on Linux. OSX should be substantively the same.
+These instructions were tested on Linux. OSX should be mostly the same.
 
-Make sure the prerequisites for [emsdk](https://github.com/juj/emsdk) are installed.
+Make sure the prerequisites for [emsdk](https://github.com/juj/emsdk) are
+installed. Pyodide will build a custom, patched version of emsdk, so there is no
+need to build it yourself prior.
 
 Additional build prerequisites are:
 
+- A working native compiler toolchain, enough to build CPython.
+- A native Python 3.6 or later to run the build scripts.
+- PyYAML
 - [lessc](https://lesscss.org/) to compile less to css.
 - [uglifyjs](https://github.com/mishoo/UglifyJS) to minify Javascript builds.
 
-Type `make`.
-
-(The build downloads and builds a local, patched version of emsdk, then
-downloads and builds Python and third-party packages).
+`make`
 
 # Testing
 
-1. Install the following dependencies into the default Python installation:
+Install the following dependencies into the default Python installation:
 
    `pip install pytest selenium`
 
-2. Install [geckodriver](https://github.com/mozilla/geckodriver/releases) somewhere
-   on your `PATH`.
+Install [geckodriver](https://github.com/mozilla/geckodriver/releases) somewhere
+on your `PATH`.
 
-3. `make test`
+`make test`
 
 # Benchmarking
 
-1. Install the same dependencies as for testing.
+Install the same dependencies as for testing.
 
-2. `make benchmark`
+`make benchmark`
 
 # Linting
 
-1. Python is linted with `flake8`.  C and Javascript are linted with `clang-format`.
+Python is linted with `flake8`.  C and Javascript are linted with `clang-format`.
 
-2. `make lint`
+`make lint`

docs/README.md

+# Pyodide documentation
+
+## Using Pyodide
+
+- [Using Pyodide directly from Javascript](using_pyodide_from_javascript.md)
+- [Using Pyodide from Iodide](using_pyodide_from_iodide.md)
+- [Type conversions](type_conversions.md): describes how data types are shared between Python and Javascript
+
+## Developing Pyodide
+
+- [Making new packages](new_packages.md)
+- [Development instructions](../README.md)
+- [Contributing](../CONTRIBUTING.md)
+- [Code of Conduct](../CODE-OF-CONDUCT.md)

docs/index.md

+# Pyodide documentation
+
+## Using Pyodide
+
+- [Using Pyodide directly from Javascript](using_pyodide_from_javascript.md)
+- [Using Pyodide from Iodide](using_pyodide_from_iodide.md)
+
+## Developing Pyodide
+
+- [Type conversions](type_conversions.md): describes how data types are shared between Python and Javascript
+- [Making new packages](new_packages.md)
+- [Development instructions](../README.md)
+- [Contributing](../CONTRIBUTING.md)
+- [Code of Conduct](../CODE-OF-CONDUCT.md)

docs/new_packages.md

+# Creating a Pyodide package
+
+Pyodide includes a set of automatic tools to make it easier to add new
+third-party Python libraries to the build.
+
+These tools automate the following steps to build a package:
+
+- Download a source tarball (usually from PyPI)
+- Confirm integrity of the package by comparing it to a checksum
+- Apply patches, if any, to the source distribution
+- Add extra files, if any, to the source distribution
+- If the package includes C/C++/Cython extensions:
+  - Build the package natively, keeping track of invocations of the native
+    compiler and linker
+  - Rebuild the package using emscripten to target WebAssembly
+- If the package is pure Python:
+  - Run the `setup.py` script to get the built package
+- Package the results into an emscripten virtual filesystem package, which
+  comprises:
+  - A `.data` file containing the file contents of the whole package,
+    concatenated together
+  - A `.js` file which contains metadata about the files and installs them into
+    the virtual filesystem.
+
+Lastly, a `packages.json` file is output containing the dependency tree of all
+packages, so `pyodide.loadPackage` can load a package's dependencies
+automatically.
+
+## The meta.yaml file
+
+Packages are defined by writing a `meta.yaml` file. The format of these files is
+based on the `meta.yaml` files used to build [Conda
+packages](https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html),
+though it is much more limited. The most important limitation is that Pyodide
+assumes there will only be one version of a given library available, whereas
+Conda allows the user to specify the versions of each package that they want to
+install. Despite the limitations, keeping the file format as close as possible
+to conda's should make it easier to use existing conda package definitions as a
+starting point to create Pyodide packages. In general, however, one should not
+expect Conda packages to "just work" with Pyodide. (In the longer term, Pyodide
+may use conda as its packaging system, and this should hopefully ease that
+transition.)
+
+The supported keys in the `meta.yaml` file are described below.
+
+### `package`
+
+#### `package/name`
+
+The name of the package. It must match the name of the package used when
+expanding the tarball, which is sometimes different from the name of the package
+in the Python namespace when installed. It must also match the name of the
+directory in which the `meta.yaml` file is placed.
+
+#### `package/version`
+
+The version of the package.
+
+### `source`
+
+#### `source/url`
+
+The url of the source tarball.
+
+The tarball may be in any of the formats supported by Python's
+`shutil.unpack_archive`: `tar`, `gztar`, `bztar`, `xztar`, and `zip`.
+
+#### `source/md5`
+
+The MD5 checksum of the tarball.  (TODO: More hash types should be supported in the future).
+
+#### `source/patches`
+
+A list of patch files to apply after expanding the tarball. These are applied
+using `patch -p1` from the root of the source tree.
+
+#### `source/extras`
+
+Extra files to add to the source tree. This should be a list where each entry is
+a pair of the form `(src, dst)`. The `src` path is relative to the directory in
+which the `meta.yaml` file resides. The `dst` path is relative to the root of
+source tree (the expanded tarball).
+
+### `build`
+
+#### `build/cflags`
+
+Extra arguments to pass to the compiler when building for WebAssembly.
+
+(This key is not in the Conda spec).
+
+#### `build/ldflags`
+
+Extra arguments to pass to the linker when building for WebAssembly.
+
+(This key is not in the Conda spec).
+
+#### `build/post`
+
+Shell commands to run after building the library. These are run inside of
+`bash`, and there are two special environment variables defined:
+
+- `$BUILD`: The root of the built package. (`build/lib.XXX/` inside of the
+  source directory). This is what will be installed into Python site-packages.
+- `$PKGDIR`: The directory in which the `meta.yaml` file resides.
+
+(This key is not in the Conda spec).
+
+### `requirements`
+
+#### `requirements/run`
+
+A list of required packages.
+
+(Unlike conda, this only supports package names, not versions).

docs/type_conversions.md

+# Type conversions
+
+Python to Javascript conversions occur:
+
+- when returning the final expression from a `pyodide.runPython` call (evaluating a Python cell in Iodide)
+- using `pyodide.pyimport`
+- passing arguments to a Javascript function from Python
+
+Javascript to Python conversions occur:
+
+- when using the `from js import ...` syntax
+- returning the result of a Javascript function to Python
+
+## Basic types
+
+The following basic types are implicitly converted between Javascript and
+Python. The values are copied and any connection to the original object is lost.
+
+| Python          | Javascript          |
+|-----------------|---------------------|
+| `int`, `float`  | `Number`            |
+| `str`           | `String`            |
+| `True`          | `true`              |
+| `False`         | `false`             |
+| `None`          | `undefined`, `null` |
+| `list`, `tuple` | `Array`             |
+| `dict`          | `Object`            |
+
+Additionally, Python `bytes` and `buffer` objects are converted to/from Javascript
+`Uint8ClampedArray` typed arrays.  In this case, however, the underlying data is
+not copied, and is shared between the Python and Javascript sides.  This makes
+passing raw memory between the languages (which in practice can be quite large)
+very efficient.
+
+Aside: This is the technology on which matplotlib images are passed to
+Javascript to render in a canvas, and will be the basis of sharing Numpy arrays
+with n-dimensional array data structures in Javascript.
+
+## Class instances
+
+Any of the types not listed above are shared between languages using proxies
+that allow methods and some operators to be called on the object from the other
+language.
+
+
+When passing a Javascript object to Python, an extension type is used to
+delegate Python operations to the Javascript side. The following operations are
+currently supported. (More should be possible in the future -- work in ongoing
+to make this more complete):
+
+| Python         | Javascript     |
+|----------------|----------------|
+| `repr(x)`      | `x.toString()` |
+| `x.foo`        | `x.foo`        |
+| `x.foo = bar`  | `x.foo = bar`  |
+| `x(...)`       | `x(...)`       |
+| `x.foo(...)`   | `x.foo(...)`   |
+| `X.new(...)`   | `new X(...)`   |
+| `len(x)`       | `x.length`     |
+| `x[foo]`       | `x[foo]`       |
+| `x[foo] = bar` | `x[foo] = bar` |
+
+When passing a Python object to Javascript, the Javascript [Proxy
+API](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy)
+is used to delegate Javascript operations to the Python side. In general, the
+Proxy API is more limited than what can be done with a Python extension, so
+there are certain operations that are impossible or more cumbersome when using
+Python from Javascript than vice versa. The most notable limitation is that
+while Python has distinct ways of accessing attributes and items (`x.foo` and
+`x[foo]`), Javascript conflates these two concepts. The following operations are
+currently supported:
+
+| Javascript     | Python                   |
+|----------------|--------------------------|
+| `foo in x`     | `hasattr(x, 'foo')`      |
+| `x.foo`        | `getattr(x, 'foo')`      |
+| `x.foo = bar`  | `setattr(x, 'foo', bar)` |
+| `delete x.foo` | `delattr(x, 'foo')`      |
+| `x.ownKeys()`  | `dir(x)`                 |
+| `x(...)`       | `x(...)`                 |
+| `x.foo(...)`   | `x.foo(...)`             |
+
+An additional limitation is that when passing a Python object to Javascript,
+there is no way for Javascript to automatically garbage collect that object.
+Therefore, Python objects must be manually free'd when passed to Javascript, or
+they will leak. (TODO: There isn't currently a way to do this, but it will be
+implemented soon).
+
+## Using Python objects from Javascript
+
+A Python object (in global scope) can be brought over to Javascript using the
+`pyodide.pyimport` function. It takes a string giving the name of the variable,
+and returns the object, converted to Javascript (See [type
+conversions](type_conversions.md)).
+
+```javascript
+var sys = pyodide.pyimport('sys');
+```
+
+## Using Javascript objects from Python
+
+Javascript objects can be accessed from Python using the `from js import ...`
+syntax. The object must be in the global (`window`) namespace.
+
+```python
+from js import document
+document.title = 'New window title'
+```

docs/using_pyodide_from_iodide.md

+# Using Pyodide from Iodide
+
+This document describes using Pyodide inside Iodide. For information
+about using Pyodide directly from Javascript, see [Using Pyodide from
+Javascript](using_pyodide_from_javascript.md).
+
+**NOTE:** The details of how this works on the Iodide side is likely to change
+in the near future.
+
+## Startup
+
+The first step is to tell Iodide you want to import support for a new programming language.
+
+Create a "language plugin cell" by selecting "plugin" from the cell type dropdown and insert the following JSON:
+
+```json
+{
+  "languageId": "py",
+  "displayName": "python",
+  "codeMirrorMode": "python",
+  "keybinding": "p",
+  "url": "https://iodide.io/pyodide-demo/pyodide.js",
+  "module": "pyodide",
+  "evaluator": "runPython",
+  "pluginType": "language"
+}
+```
+
+Evaluate the cell (Shift+Enter) to load Pyodide and set up the Python environment.
+
+## Running basic Python
+
+Create a Python cell, by choosing Python from the cell type dropdown.
+
+Insert some Python into the cell, and press Shift+Enter to evaluate it. If the
+last clause in the cell is an expression, that expression is evaluated,
+converted to Javascript and displayed in the output cell like all other output
+in Javascript. See [type conversions](type_conversions.md) for more information
+about how data types are converted between Python and Javascript.
+
+```python
+import sys
+sys.version
+```
+
+## Loading packages
+
+Only the Python standard library and `six` are available after importing
+Pyodide. To use other libraries, you'll need to load their package using
+`pyodide.loadPackage`. This is a Javascript API, so importantly, it must be run
+from a Javascript cell. This downloads the file data over the network (as a
+`.data` and `.js` index file) and installs the files in the virtual filesystem.
+
+When you request a package, all of that package's dependencies are also loaded.
+
+`pyodide.loadPackage` returns a `Promise`.
+
+```javascript
+pyodide.loadPackage('matplotlib')
+```

docs/using_pyodide_from_javascript.md

+# Using Pyodide from Javascript
+
+This document describes using Pyodide directly from Javascript. For information
+about using Pyodide from Iodide, see [Using Pyodide from
+Iodide](using_pyodide_from_iodide.md).
+
+## Startup
+
+Include `pyodide.js` in your project.
+
+This has a single function which bootstraps the Python environment:
+`languagePluginLoader`. Since this must happen asynchronously, it returns a
+`Promise`, which you must call `then` on to complete initialization. When the
+promise resolves, pyodide will have installed a namespace in global scope:
+`pyodide`.
+
+```javascript
+languagePluginLoader().then(() => {
+  // pyodide is now ready to use...
+  console.log(pyodide.runPython('import sys\nsys.version'));
+});
+```
+
+## Running Python code
+
+Python code is run using the `pyodide.runPython` function. It takes as input a
+string of Python code. If the code ends in an expression, it returns the result
+of the expression, converted to Javascript objects (See [type
+conversions](type_conversions.md)).
+
+```javascript
+pyodide.runPython('import sys\nsys.version'));
+```
+
+## Loading packages
+
+Only the Python standard library and `six` are available after importing
+Pyodide. To use other libraries, you'll need to load their package using
+`pyodide.loadPackage`. This downloads the file data over the network (as a
+`.data` and `.js` index file) and installs the files in the virtual filesystem.
+
+When you request a package, all of that package's dependencies are also loaded.
+
+`pyodide.loadPackage` returns a `Promise`.
+
+```javascript
+pyodide.loadPackage('matplotlib').then(() => {
+  // matplotlib is not available
+});
+```
+
+## Complete example
+
+TODO