Skip to content

J
jio
Commit f9b88bd7 authored by Romain Courteaud's avatar Romain Courteaud
Browse files
Options

Documentation moved to https://jio.nexedi.com/

parent 60187866
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 2710 deletions
docs/authors.rst deleted 100644 → 0
View file @ 60187866
Authors
=======
* Francois Billioud
* Tristan Cavelier
* Sven Franck
* Romain Courteaud
docs/available_storages.rst deleted 100644 → 0
View file @ 60187866
This diff is collapsed.
docs/developers.rst deleted 100644 → 0
View file @ 60187866
For developers
==============
Quick start
-----------
The source repository includes ready-to-use files, so in case you do
not want to build jIO yourself, just use *sha256.amd.js*, *rsvp.js*, *jio.js*
plus the storages and dependencies you need and you will be good to go.
If you want to modify or build jIO yourself, you need to
* Clone from a repository
``$ git clone https://lab.nexedi.com/nexedi/jio.git``
* Install `NodeJS <http://nodejs.org/>`_ (including ``npm``)
* Install the Grunt command line with ``npm``.
``# npm -g install grunt-cli``
* Install the dependencies.
``$ npm install``
* Compile the JS/CC parser.
``$ make`` (until we find out how to compile it with grunt)
* Run build.
``$ grunt``
Naming Conventions
------------------
All the code follows this :ref:`JavaScript Style Guide <style-guide>`.
How to design your own jIO Storage Library
------------------------------------------
Create a constructor:
.. code-block:: javascript
function MyStorage(storage_description) {
this._value = storage_description.value;
if (typeof this._value !== 'string') {
throw new TypeError("'value' description property " +
"is not a string");
}
}
Your storage must be added to jIO. This is done
with the ``jIO.addStorage()`` method, which requires two parameters: the storage
type (string) and a constructor (function). It is called like this:
.. code-block:: javascript
// add custom storage to jIO
jIO.addStorage('mystoragetype', MyStorage);
Please refer to *localstorage.js* implementation for a good example on how to
setup a storage and what methods are required.
docs/getting_started.rst deleted 100644 → 0
View file @ 60187866
Getting started
===============
#. :ref:`Download <download-fork>` the core jIO, the storages you need and the
dependencies required for them.
#. Add the scripts to your HTML page in the following order:
.. code-block:: html
<!-- jio core + dependencies -->
<script src="rsvp.js"></script>
<script src="jio.js"></script>
<script ...>
#. jIO connects to a number of storages and allows adding handlers (or
functions) to specific storages.
You can use both handlers and available storages to build a storage
tree across which all documents will be maintained and managed by jIO.
.. code-block:: javascript
// create your jio instance
var my_jio = jIO.createJIO(storage_description);
You have to provide a ``storage_description`` object, providing location
and credentials.
Its format depends on the type of storage,
see :ref:`List of Available Storages <list-of-available-storages>`.
#. The jIO API provides ten main methods to manage documents across the storage(s) specified in your jIO storage tree.
For details on the ``document`` and ``attachment`` objects, see :ref:`What is a document? <what-is-a-document>`
======================= ======================================================
Method Example
======================= ======================================================
``.post()`` | ``my_jio.post(document, [options]);``
| Creates a new document
``.put()`` | ``my_jio.put(document, [options]);``
| Creates/Updates a document
``.putAttachment()`` | ``my_jio.putAttachement(attachment, [options]);```
| Updates/Adds an attachment to a document
``.get()`` | ``my_jio.get(document, [options]);``
| Reads a document
``.getAttachment()`` | ``my_jio.getAttachment(attachment, [options]);``
| Reads a document attachment
``.remove()`` | ``my_jio.remove(document, [options]);``
| Deletes a document and its attachments
``.removeAttachment()`` | ``my_jio.removeAttachment(attachment, [options]);``
| Deletes a document's attachment
``.allDocs()`` | ``my_jio.allDocs([options]);``
| Retrieves a list of existing documents
``.repair()`` | ``my_jio.repair(document, [options]);``
| Repairs the document
======================= ======================================================
.. _download-fork:
Download & Fork
---------------
You can get latest jIO release by using on of those links:
`Full download <https://lab.nexedi.com/nexedi/jio/raw/master/dist/jio-latest.js>`_
`Minified download <https://lab.nexedi.com/nexedi/jio/raw/master/dist/jio-latest.min.js>`_
You can get latest RSVP release by using on of those links:
`Full download <https://lab.nexedi.com/nexedi/rsvp.js/raw/master/dist/rsvp-2.0.4.js>`_
`Minified download <https://lab.nexedi.com/nexedi/rsvp.js/raw/master/dist/rsvp-2.0.4.min.js>`_
Fork jIO
^^^^^^^^
| Feel free to use the Gitlab repository:
| `GitLab <https://lab.nexedi.com/nexedi/jio.git>`_: ``git clone https://lab.nexedi.com/nexedi/jio.git``
docs/index.rst
View file @ f9b88bd7
Welcome to jIO
==============
jIO is a JavaScript library that allows to manage JSON documents on local or
remote storages in asynchronous fashion. jIO is an abstracted API mapped after
CouchDB, that offers connectors to multiple storages, special handlers to
enhance functionality (replication, revisions, indexing) and a query module to
retrieve documents and specific information across storage trees.
How does it work?
-----------------
jIO is composed of two parts - jIO core and storage libraries. The core
makes use of storage libraries (connectors) to interact with the associated remote
storage servers. Some queries can be used on top of the ``.allDocs()`` method to
query documents based on defined criteria.
jIO uses a job management system, so each method call adds a job into a
queue. The queue is copied in the browser's local storage (by default), so it
can be restored in case of browser crash. Jobs are invoked
asynchronously and ongoing jobs are not able to re-trigger to prevent
conflicts.
Copyright and license
---------------------
jIO is open source and is licensed under the `LGPL <http://en.wikipedia.org/wiki/GNU_Lesser_General_Public_License>`_ license.
jIO documentation
-----------------
.. toctree::
:maxdepth: 2
getting_started
manage_documents
replicate_storage
available_storages
query
keys
metadata
developers
style_guide
authors
jIO documentation moved to `jio.nexedi.com <https://jio.nexedi.com/>`_.
docs/keys.rst deleted 100644 → 0
View file @ 60187866
Search Keys
===========
Features like case insensitive, accent-removing, full-text searches and more can be implemented
by customizing jIO's query behavior.
Let's start with a simple search:
.. code-block:: javascript
var query = {
type: 'simple',
key: 'someproperty',
value: comparison_value,
operator: '='
}
Each of the ``.someproperty`` attribute in objects' metadata is compared with
``comparison_value`` through a function defined by the '=' operator.
You can provide your own function to be used as '=' operator:
.. code-block:: javascript
var strictEqual = function (object_value, comparison_value) {
return comparison_value === object_value;
};
var query = {
type: 'simple',
key: {
read_from: 'someproperty',
equal_match: strictEqual
},
value: comparison_value
}
Inside ``equal_match``, you can decide to interpret the wildcard character ``%``
or just ignore it, as in this case.
If you need to convert or preprocess the values before comparison, you can provide
a conversion function:
.. code-block:: javascript
var numberType = function (obj) {
return parseFloat('3.14');
};
var query = {
type: 'simple',
key: {
read_from: 'someproperty',
cast_to: numberType
},
value: comparison_value
}
In this case, the operator is still the default '=' that works with strings.
You can combine ``cast_to`` and ``equal_match``:
.. code-block:: javascript
var query = {
type: 'simple',
key: {
read_from: 'someproperty',
cast_to: numberType,
equal_match: strictEqual
},
value: comparison_value
}
Now the query returns all objects for which the following is true:
.. code-block:: javascript
strictEqual(numberType(metadata.someproperty),
numberType(comparison_value))
For a more useful example, the following function removes the accents
from any string:
.. code-block:: javascript
var accentFold = function (s) {
var map = [
[new RegExp('[àáâãäå]', 'gi'), 'a'],
[new RegExp('æ', 'gi'), 'ae'],
[new RegExp('ç', 'gi'), 'c'],
[new RegExp('[èéêë]', 'gi'), 'e'],
[new RegExp('[ìíîï]', 'gi'), 'i'],
[new RegExp('ñ', 'gi'), 'n'],
[new RegExp('[òóôõö]', 'gi'), 'o'],
[new RegExp('œ', 'gi'), 'oe'],
[new RegExp('[ùúûü]', 'gi'), 'u'],
[new RegExp('[ýÿ]', 'gi'), 'y']
];
map.forEach(function (o) {
var rep = function (match) {
if (match.toUpperCase() === match) {
return o[1].toUpperCase();
}
return o[1];
};
s = s.replace(o[0], rep);
});
return s;
};
...
cast_to: accentFold
...
A more robust solution to manage diacritics is recommended for production
environments, with unicode normalization, like (untested):
https://github.com/walling/unorm/
Overriding operators and sorting
--------------------------------
The advantage of providing an ``equal_match`` function is that it can work with basic types;
you can keep the values as strings or, if you use a ``cast_to`` function, it can return strings,
numbers, arrays... and that's fine if all you need is the '=' operator.
It's also possible to customize the behavior of the other operators: <, >, !=...
To do that, the object returned by ``cast_to`` must contain a ``.cmp``
property, that behaves like the ``compareFunction`` described in
`Array.prototype.sort() <https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort>`_:
.. code-block:: javascript
function myType (...) {
...
return {
...
'cmp': function (b) {
if (a < b) {
return -1;
}
if (a > b) {
return +1;
}
return 0;
}
};
}
...
cast_to: myType
...
If the < or > comparison makes no sense for the objects, the function should return ``undefined``.
The ``.cmp()`` property is also used, if present, by the sorting feature of queries.
Partial Date/Time match
-----------------------
As a real life example, consider a list of documents that have a *start_task* property.
The value of ``start_task`` can be an `ISO 8601 <http://en.wikipedia.org/wiki/ISO_8601>`_ string
with date and time information including fractions of a second. Which is, honestly, a bit too
much for most queries.
By using a ``cast_to`` function with custom operators, it is possible to perform queries like
"start_task > 2010-06", or "start_task != 2011". Partial time can be used as well, so
we can ask for projects started after noon of a given day: ``start_task = "2011-04-05" AND start_task > "2011-04-05 12"``
The JIODate type has been implemented on top of the `Moment.js <http://momentjs.com/>`_ library, which
has a rich API with support for multiple languages and timezones. No special support for timezones
is present (yet) in JIODate.
To use JIODate, include the ``jiodate.js`` and ``moment.js`` files in your
application, then set ``cast_to = jiodate.JIODate``.
Key Schemas
-----------
Instead of providing the key object for each attribute you want to filter,
you can group all of them in a schema object for reuse:
.. code-block:: javascript
var key_schema = {
key_set: {
date_day: {
read_from: 'date',
cast_to: 'dateType',
equal_match: 'sameDay'
},
date_month: {
read_from: 'date',
cast_to: 'dateType',
equal_match: 'sameMonth'
}
},
cast_lookup: {
dateType: function (str) {
return new Date(str);
}
},
match_lookup: {
sameDay: function (a, b) {
return (
(a.getFullYear() === b.getFullYear()) &&
(a.getMonth() === b.getMonth()) &&
(a.getDate() === b.getDate())
);
},
sameMonth: function (a, b) {
return (
(a.getFullYear() === b.getFullYear()) &&
(a.getMonth() === b.getMonth())
);
}
}
}
With this schema, we have created two 'virtual' metadata attributes,
``date_day`` and ``date_month``. When queried, they match values that
happen to be in the same day, ignoring the time, or the same month, ignoring
both time and day.
A key_schema object can have three properties:
* ``key_set`` - required.
* ``cast_lookup`` - optional, an object of the form ``{name: function}`` that is
used if cast_to is a string. If cast_lookup is not provided,
then cast_to must be a function.
* ``match_lookup`` - optional, an object of the form ``{name: function}`` that is
used if ``equal_match`` is a string. If match_lookup is not provided,
then ``equal_match`` must be a function.
Using a schema
--------------
A schema can be used:
* In a query constructor. The same schema will be applied to all the sub-queries:
.. code-block:: javascript
jIO.QueryFactory.create({...}, key_schema).exec(...);
* In the ``jIO.createJIO()`` method. The same schema will be used
by all the queries created with the ``.allDocs()`` method:
.. code-block:: javascript
var jio = jIO.createJIO({
type: 'local',
username: '...',
application_name: '...',
key_schema: key_schema
});
docs/manage_documents.rst deleted 100644 → 0
View file @ 60187866
How to manage documents?
========================
jIO is mapped after the CouchDB APIs and extends them to provide unified, scalable
and high performance access via JavaScript to a wide variety of storage backends.
If you are not familiar with `Apache CouchDB <http://couchdb.apache.org/>`_:
it is a scalable, fault-tolerant, and schema-free document-oriented database.
It is used in large and small organizations for a variety of applications where
traditional SQL databases are not the best solution for the problem at hand.
CouchDB provides a RESTful HTTP/JSON API accessible by many programming
libraries and tools (like `curl <http://curl.haxx.se/>`_ or `Pouchdb <http://pouchdb.com/>`_)
and has its own conflict management system.
.. _what-is-a-document:
What is a document?
-------------------
A document is an association of metadata and attachment(s). The metadata is the
set of properties of the document and the attachments are binary (or text) objects
that represent the content of the document.
In jIO, the metadata is a dictionary with keys and values (a JSON object), and
attachments are simple strings.
.. code-block:: javascript
{
// document metadata
title: 'A Title!',
creator: 'Mr.Author'
}
:ref:`Here <metadata-head>` is a draft about metadata to use with jIO.
Basic Methods
-------------
Below you can see examples of the main jIO methods.
.. code-block:: javascript
// Create a new jIO instance
var jio_instance = jIO.createJIO(storage_description);
// create and store new document
jio_instance.post({title: 'my document'}).
then(function (response) {
// console.log(response);
});
// create or update an existing document
jio_instance.put('document_name', {title: 'another document'}).
then(function (response) {
// console.log(response);
});
// add an attachment to a document
jio_instance.putAttachment('document_name',
'attachment_name',
new Blob([data], {'type' : data_mimetype});
).
then(function (response) {
// console.log(response);
});
// read a document
jio_instance.get('document_name').
then(function (response) {
// console.log(response);
});
// read an attachment
jio_instance.getAttachment('document_name',
'attachment_name').
then(function (response) {
// console.log(response);
});
// delete a document and its attachment(s)
jio_instance.remove('document_name').
then(function (response) {
// console.log(response);
});
// delete an attachment
jio_instance.removeAttachment('document_name',
'attachment_name').
then(function (response) {
// console.log(response);
});
// get all documents
jio_instance.allDocs().then(function (response) {
// console.log(response);
});
Promises
--------
Each jIO method (with the exception of ``.createJIO()``) returns a Promise object, which allows us to get responses into
callback parameters and to chain callbacks with other returned values.
jIO uses a custom version of `RSVP.js <https://lab.nexedi.com/nexedi/rsvp.js>`_, adding canceler and progression features.
You can read more about promises:
* `RSVP.js <https://lab.nexedi.com/nexedi/rsvp.js>`_ on GitHub
* `Promises/A+ <http://promisesaplus.com/>`_
* `CommonJS Promises <http://wiki.commonjs.org/wiki/Promises>`_
Method Options and Callback Responses
-------------------------------------
To retrieve jIO responses, you have to provide callbacks like this:
.. code-block:: javascript
jio_instance.post(metadata, [options]).
then([responseCallback], [errorCallback], [progressionCallback]);
* On command success, ``responseCallback`` is called with the jIO response as first parameter.
* On command error, ``errorCallback`` is called with the jIO error as first parameter.
* On command notification, ``progressionCallback`` is called with the storage notification.
Here is a list of responses returned by jIO according to methods and options:
============================================== ================== ===============================================
Available for Option Response (Callback first parameter)
============================================== ================== ===============================================
``.post()``, ``.put()``, ``.remove()`` Any id of the document affected (string)
``.putAttachment()``, ``.removeAttachment()`` Any no specific value
``.get()`` Any document_metadata (object)
``.getAttachment()`` Any .. code-block:: javascript
new Blob([data], {"type": content_type})
``.allDocs()`` No option .. code-block:: javascript
{
total_rows: 1,
rows: [{
id: 'mydoc',
value: {},
}]
}
``.allDocs()`` include_docs: true .. code-block:: javascript
{
total_rows: 1,
rows: [{
id: 'mydoc',
value: {
// Here, 'mydoc' metadata
}
}]
}
============================================== ================== ===============================================
In case of error, the ``errorCallback`` first parameter looks like:
.. code-block:: javascript
{
status_code: 404,
message: 'Unable to get the requested document'
}
How to store binary data
------------------------
The following example creates a new jIO in localStorage and then posts a document with two attachments.
.. code-block:: javascript
// create a new jIO
var jio_instance = jIO.createJIO({type: 'indexeddb'});
// post the document 'myVideo'
jio_instance.put( 'metadata', {
title : 'My Video',
type : 'MovingImage',
format : 'video/ogg',
description : 'Images Compilation'
})
.push(undefined, function(err) {
// pushes error handler with RSVP.Queue push method
// nothing to do with JIO itself
return alert('Error posting the document metadata');
});
// post a thumbnail attachment
jio_instance.putAttachment('metadata',
'thumbnail',
new Blob([my_image], {type: 'image/jpeg'})
).push(undefined, function(err) {
return alert('Error attaching thumbnail');
});
// post video attachment
jio_instance.putAttachment('metadata',
'video',
new Blob([my_video], {type: 'video/ogg'})
).push(undefined, function(err) {
return alert('Error attaching video');
});
alert('Video Stored');
indexedDB Storage now contains:
.. code-block:: javascript
{
"/myVideo/": {
"title": "My Video",
"type": "MovingImage",
"format": "video/ogg",
"description": "Images Compilation",
"_attachments":{
"thumbnail":{
"digest": "md5-3ue...",
"content_type": "image/jpeg",
"length": 17863
},
"video":{
"digest": "md5-0oe...",
"content_type": "video/ogg",
"length": 2840824
}
}
},
"/myVideo/thumbnail": "...",
"/myVideo/video": "..."
}
docs/metadata.rst deleted 100644 → 0
View file @ 60187866
This diff is collapsed.
docs/query.rst deleted 100644 → 0
View file @ 60187866
jIO Query
=========
What are Queries?
-----------------
In jIO, a query can ask a storage server to select, filter, sort, or limit a
document list before sending it back. If the server is not able to do so, the
jio query tool can do the filtering by itself on the client. Only the
``.allDocs()`` method can use jio queries.
A query can either be a string (using a specific language useful for writing
queries), or it can be a tree of objects (useful to browse queries). To handle
queries, jIO uses a parsed grammar file which is compiled using `JSCC
<http://jscc.phorward-software.com/>`_.
Why use JIO Queries?
--------------------
JIO queries can be used like database queries, for tasks such as:
* search a specific document
* sort a list of documents in a certain order
* avoid retrieving a list of ten thousand documents
* limit the list to show only N documents per page
For some storages (like localStorage), jio queries can be a powerful tool to
query accessible documents. When querying documents on a distant storage, some
server-side logic should be run to avoid returning too many documents to the
client. If distant storages are static, an alternative would be to use an
indexStorage with appropriate indices as jio queries will always try to run the
query on the index before querying documents itself.
How to use Queries with jIO?
----------------------------
Queries can be triggered by including the option named **query** in the ``.allDocs()`` method call.
Example:
.. code-block:: javascript
var options = {};
// search text query
options.query = '(creator:"John Doe") AND (format:"pdf")';
// OR query tree