Skip to content

jio_mebibou
jio_mebibou
Commit 2d5c99d2 authored by lucas.parsy's avatar lucas.parsy Committed by Romain Courteaud
Browse files
Options

Update documentation. 

Squashed commit of the following:

commit 3ddc5e522969d8e5270dc408eec71fc0847c40ce
Author: Lucas Parsy <lucas.parsy@nexedi.com>
Date:   Thu Dec 17 16:44:19 2015 +0100

    corrected misspellings and ambiguous examples in manage_document file.
    deleted gid and revision storage documentation.

commit 85d30999afdd227c8d4085fcd2df329f0d95333e
Author: Lucas Parsy <lucas.parsy@nexedi.com>
Date:   Tue Dec 15 15:13:23 2015 +0100

    updated documentation
    renamed "revision_storage.rst" "replicate_storage.rst"
    you need to "compile" the files (go in doc/ directory and type "make")
    to have the html files.
parent 0a7b8a91
Show whitespace changes
Inline Side-by-side
Showing with 472 additions and 688 deletions
docs/available_storages.rst
View file @ 2d5c99d2
...@@ -16,67 +16,101 @@ When building storage trees, there is no limit on the number of storages you ...@@ -16,67 +16,101 @@ When building storage trees, there is no limit on the number of storages you
can use. The only thing you have to be aware of is compatibility of simple and can use. The only thing you have to be aware of is compatibility of simple and
revision based storages. revision based storages.
Connectors Connectors
---------- ----------
LocalStorage LocalStorage
^^^^^^^^^^^^ ^^^^^^^^^^^^
Three methods are provided: This storage has only one document, so **post**, **put**, **remove** and **get** method are useless on it.
* ``.createDescription(username, [application_name], [mode='localStorage'])`` =============== ========== ========== ============================================================
* ``.createLocalDescription(username, [application_name])`` parameter required? type description
* ``.createMemoryDescription(username, [application_name])`` =============== ========== ========== ============================================================
``type`` yes string name of the storage type (here: "local")
``sessiononly`` no boolean | false: create a storage with unlimited duration.
| true: the storage duration is limited to the user session.
| (default to false)
=============== ========== ========== ============================================================
All parameters are strings.
Examples: Example:
.. code-block:: javascript .. code-block:: javascript
// to work on browser localStorage var jio = jIO.createJIO({
var jio = jIO.createJIO(local_storage.createDescription('me')); type: "local",
sessiononly: true
});
MemoryStorage
^^^^^^^^^^^^^
| Stores the data in a Javascript object, in memory.
| The storage's data isn't saved when your web page is closed or reloaded.
| The storage doesn't take any argument at creation.
Example:
.. code-block:: javascript
var jio = jIO.createJIO({type: "memory"});
IndexedDB
^^^^^^^^^^^^
================= ========== ========== ==========================================================
parameter required? type description
================= ========== ========== ==========================================================
``type`` yes string name of the storage type (here: "indexeddb")
``database`` yes string name of the database.
================= ========== ========== ==========================================================
// to work on browser memory Example:
var jio = jIO.createJIO(local_storage.createMemoryDescription('me'));
.. code-block:: javascript
// or
{ {
"type": "local", "type": "indexeddb",
"username": "me", "database": "mydb"
"application_name": "my app name", // optional
"mode": "memory" // optional, "localStorage" by default
} }
DavStorage WebSQL
^^^^^^^^^^ ^^^^^^^^^^^^
The method ``dav_storage.createDescription()`` generates a DAV storage description for ================= ========== ========== ==========================================================
*none*, *basic* or *digest* authentication. parameter required? type description
================= ========== ========== ==========================================================
``type`` yes string name of the storage type (here: "websql")
``database`` yes string name of the database.
================= ========== ========== ==========================================================
NB: digest **is not implemented yet**.
Example:
.. code-block:: javascript .. code-block:: javascript
dav_storage.createDescription(url, auth_type, {
[realm], [username], [password]); "type": "websql",
"database": "mydb"
}
All parameters are strings. DavStorage
^^^^^^^^^^
============= ======================== ================ ========== ========== ==========================================================
parameter required? parameter required? type description
============= ======================== ================ ========== ========== ==========================================================
``url`` yes ``type`` yes string name of the storage type (here: "dav")
``auth_type`` yes ``url`` yes string url of your webdav server
``realm`` if auth_type == 'digest' ``basic_login`` no string | login and password of your dav, base64 encoded like this:
``username`` if auth_type != 'none' | ``btoa(username + ":" + password)``
``password`` if auth-type != 'none' ================ ========== ========== ==========================================================
============= ========================
If ``auth_type`` is the string ``"none"``, then ``realm``, ``username`` and ``password`` are never used.
Descriptions: Example:
.. code-block:: javascript .. code-block:: javascript
...@@ -89,7 +123,7 @@ Descriptions: ...@@ -89,7 +123,7 @@ Descriptions:
// Basic authentication // Basic authentication
{ {
"type": "dav", "type": "dav",
"url": "url, "url": url,
"basic_login": btoa(username + ":" + password) "basic_login": btoa(username + ":" + password)
} }
...@@ -98,169 +132,251 @@ Descriptions: ...@@ -98,169 +132,251 @@ Descriptions:
**Be careful**: The generated description never contains a readable password, but **Be careful**: The generated description never contains a readable password, but
for basic authentication, the password is just base64 encoded. for basic authentication, the password is just base64 encoded.
S3Storage
^^^^^^^^^
Live tests OK! Dropbox
^^^^^^^
================= ========== ========== ==========================================================
parameter required? type description
================= ========== ========== ==========================================================
``type`` yes string name of the storage type (here: "dropbox")
``access_token`` yes string access token for your account.
See specific documentation on how to retreive it.
``root`` no string | "dropbox" for full access to account files,
| "sandbox" for app limited file access.
| default to "dropbox".
================= ========== ========== ==========================================================
Here is a basic description for jIO. Documentation comming soon.
Example:
.. code-block:: javascript .. code-block:: javascript
{ {
"type": "s3", "type": "dropbox",
"AWSIdentifier": "my aws identifier", "access_token": "sample_token"
"password": "my password", "root": "dropbox"
"server": "bucket_name"
} }
XWikiStorage Google Drive
^^^^^^^^^^^^ ^^^^^^^^^^^^
Work is in progress. ================= ========== ========== ==========================================================
parameter required? type description
================= ========== ========== ==========================================================
``type`` yes string name of the storage type (here: "gdrive")
``access_token`` yes string access token for your account.
See specific documentation on how to retreive it.
``trashing`` no boolean | true: sends files to the trash bin when doing a "remove"
| false: deletes permanently files when doing a "remove"
| default to true.
================= ========== ========== ==========================================================
Example:
.. code-block:: javascript
Searchable Encryption Storage {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ "type": "gdrive",
"access_token": "sample_token"
"trashing": true
}
Comes with a specific server with can query encrypted documents. ERP5Storage
^^^^^^^^^^^
=========================== ========== ========== ==========================================================
parameter required? type description
=========================== ========== ========== ==========================================================
``type`` yes string name of the storage type (here: "erp5")
``url`` yes string url of your erp5 account.
``default_view_reference`` no string | reference of the action used
| for the delivering of the document
=========================== ========== ========== ==========================================================
Work is in progress. Documentation comming soon. Example:
.. code-block:: javascript .. code-block:: javascript
{ {
"type": "searchableencryption", "type": "erp5",
"password": "your password", "url": erp5_url
"url": "http://your/url"
} }
Handlers Handlers
-------- --------
IndexStorage Zipstorage
^^^^^^^^^^^^ ^^^^^^^^^^
This handler indexes documents metadata into a database (which is a simple This handler compresses and decompresses files to reduce network and storage usage.
document) to increase the speed of ``.allDocs()`` requests. However, it is not able to
manage the ``include_docs`` option.
Here is the description: Usage:
.. code-block:: javascript .. code-block:: javascript
{ {
type: 'index', "type": "zip",
indices: [{ "sub_storage": <your storage>
// doc id where to store indices
id: 'index_title_subject.json',
// metadata to index
index: ['title', 'subject'],
attachment: 'db.json', // default 'body'
// additional metadata to add to database, default undefined
metadata: {
type: 'Dataset',
format: 'application/json',
title: 'My index database',
creator: 'Me'
},
// default equal to parent sub_storage field
sub_storage: <sub storage where to store index>
}, {
id: 'index_year.json',
index: 'year'
...
}],
sub_storage: <sub storage description>
} }
ShaStorage
GIDStorage
^^^^^^^^^^ ^^^^^^^^^^
:ref:`Full description here <gid-storage>`. This handler provides a post method that creates a document that has for name the SHA-1 hash of his parameters.
SplitStorage .. code-block:: javascript
^^^^^^^^^^^^
{
"type": "sha",
"sub_storage": <your storage>
}
Work is in progress. The interoperability is not enabled yet. UUIDStorage
^^^^^^^^^^^
This storage splits metadata and attachment data to *n* parts where *n* is the This handler provides a post method to create a document that has a unique ID for name.
number of sub storages. Each parts are stored on one sub storage only.
.. code-block:: javascript .. code-block:: javascript
{ {
type: 'split', "type": "uuid",
storage_list: [ "sub_storage": <your storage>
<sub storage description>,
...
]
} }
Other split modes will be added later. QueryStorage
^^^^^^^^^^^^
Replicate Storage
^^^^^^^^^^^^^^^^^
Work is in progress. This handler provides an allDocs method with queries support to the substorage.
.. code-block:: javascript .. code-block:: javascript
{ {
type: 'replicate', "type": "query",
storage_list: [ "sub_storage": <your storage>
<sub storage description>,
...
]
} }
CryptStorage
^^^^^^^^^^^^
Revision Based Handlers | This handler encrypts and decrypts attachments before storing them.
----------------------- | You need to generate a Crypto key at the JSON format to use the handler.
| (see https://developer.mozilla.org/fr/docs/Web/API/Window/crypto for more informations)
A revision based handler is a storage which is able to do some document Usage:
versioning using simple storages listed above.
On jIO command parameter, ``_id`` is still used to identify a document, but .. code-block:: javascript
another id ``_rev`` must be defined to use a specific revision of that document.
On command responses, you will find another field ``rev`` which will represent the var key,
new revision produced by your action. All the document history is kept unless jsonKey,
you decide to delete older revisions. jio;
Other fields ``conflicts``, ``revisions`` and ``revs_info`` can be returned if the //creation of an encryption/decryption key.
options **conflicts: true**, **revs: true** or **revs_info: true** are set.
crypto.subtle.generateKey({name: "AES-GCM",length: 256},
(true), ["encrypt", "decrypt"])
.then(function(res){key = res;});
window.crypto.subtle.exportKey("jwk", key)
.then(function(res){jsonKey = res})
//creation of the storage
jio = jIO.createJIO({
{
"type": "crypt",
"key": json_key
"sub_storage": <your storage>
}
Revision Storage
^^^^^^^^^^^^^^^^
This backend uses its sub storage to manage document and their revision. For UnionStorage
more information, :ref:`see here <revision-storages-conflicts-and-resolution>`. ^^^^^^^^^^^^
This handler takes in argument an array of storages.
When using a method, UnionStorage tries it on the first storage of the array,
and, in case of failure, tries with the next storage,
and repeats the operation until success, or end of storage's array.
.. code-block:: javascript
{
"type": "union",
"storage_list": [
sub_storage_description_1,
sub_storage_description_2,
sub_storage_description_X
]
}
Description: FileSystemBridgeStorage
^^^^^^^^^^^^^^^^^^^^^^^
This handler adds an abstraction level on top of the webDav Jio storage,
ensuring each document has only one attachment, and limiting the storage to one repertory.
.. code-block:: javascript .. code-block:: javascript
{ {
"type": "revision", "type": "drivetojiomapping",
"sub_storage": <sub storage description> "sub_storage": <your dav storage>
} }
Document Storage
^^^^^^^^^^^^^^^^
This handler creates a storage from a document in a storage,
by filling his attachments with a new jIO storage.
Replicate Revision Storage ====================== ========== ========== ============================================================
^^^^^^^^^^^^^^^^^^^^^^^^^^ parameter required? type description
====================== ========== ========== ============================================================
``type`` yes string name of the storage type (here: "document")
``document_id`` no string id of the document to use.
``repair_attachment`` no boolean verify if the document is in good state. (default to false)
====================== ========== ========== ============================================================
Replicate revisions across multiple revision based storages. Replicate Storage
^^^^^^^^^^^^^^^^^
Description: Replicate Storage synchronizes documents between a local and a remote storage.
=============================== ========== ========== ============================================================
parameter required? type description
=============================== ========== ========== ============================================================
``type`` yes string name of the storage type (here: "replicate")
``local_sub_storage`` yes object local sub_storage description.
``remote_sub_storage`` yes object remote sub_storage description.
``query_options`` no object query object to limit the synchronisation to specific files.
``use_remote_post`` no boolean | true: at file modification, modifies the local file id.
| false: at file modification, modifies the remote file id.
| default to false.
``conflict_handling`` no number | 0: no conflict resolution (throws error)
| 1: keep the local state.
| 2: keep the remote state.
| 3: keep both states (no signature update)
| default to 0.
``check_local_modification`` no boolean synchronise when local files are modified.
``check_local_creation`` no boolean synchronise when local files are created.
``check_local_deletion`` no boolean synchronise when local files are deleted.
``check_remote_modification`` no boolean synchronise when remote files are modified.
``check_remote_creation`` no boolean synchronise when local files are created.
``check_remote_deletion`` no boolean synchronise when local files are deleted.
=============================== ========== ========== ============================================================
synchronisation parameters are set by default to true.
.. code-block:: javascript .. code-block:: javascript
{ {
"type": "revision", type: 'replicate',
"storage_list": [ local_sub_storage: { 'type': 'local'}
<revision based sub storage description>, remote_sub_storage: {
... 'type': 'dav',
] 'url': 'http://mydav.com',
'basic_login': 'aGFwcHkgZWFzdGVy'
}
use_remote_post: false,
conflict_handling : 2,
check_local_creation: false,
check_remote_deletion: false
} }
docs/developers.rst
View file @ 2d5c99d2
...@@ -12,7 +12,7 @@ If you want to modify or build jIO yourself, you need to ...@@ -12,7 +12,7 @@ If you want to modify or build jIO yourself, you need to
* Clone from a repository * Clone from a repository
``$ git clone https://github.com/nexedi/jio.git`` ``$ git clone https://lab.nexedi.com/nexedi/jio.git``
* Install `NodeJS <http://nodejs.org/>`_ (including ``npm``) * Install `NodeJS <http://nodejs.org/>`_ (including ``npm``)
...@@ -53,8 +53,8 @@ Create a constructor: ...@@ -53,8 +53,8 @@ Create a constructor:
} }
} }
Create 10 methods: ``post``, ``put``, ``putAttachment``, ``get``, ``getAttachment``, Create 9 methods: ``post``, ``put``, ``putAttachment``, ``get``, ``getAttachment``,
``remove``, ``removeAttachment``, ``allDocs``, ``check`` and ``repair``. ``remove``, ``removeAttachment``, ``allDocs``. ``repair`` method is optional.
.. code-block:: javascript .. code-block:: javascript
...@@ -81,66 +81,26 @@ Create 10 methods: ``post``, ``put``, ``putAttachment``, ``get``, ``getAttachmen ...@@ -81,66 +81,26 @@ Create 10 methods: ``post``, ``put``, ``putAttachment``, ``get``, ``getAttachmen
(To help you design your methods, some tools are provided by jIO.util.) (To help you design your methods, some tools are provided by jIO.util.)
The first parameter command provides some methods to act on the jIO job:
* ``success``, to tell jIO that the job is successfully terminated
``command.success(status[Text], [{custom key to add to the response}]);``
* ``resolve``, is equal to success
* ``error``, to tell jIO that the job cannot be done
``command.error(status[Text], [reason], [message], [{custom key to add to the response}])``
* ``retry``, to tell jIO that the job cannot be done now, but can be retried later. (same API than error)
* ``reject``, to tell jIO that the job cannot be done, let jIO to decide whether to retry or not. (same API than error)
The second parameter ``metadata`` or ``param`` is the first parameter provided by the jIO user. The second parameter ``metadata`` or ``param`` is the first parameter provided by the jIO user.
The third parameter ``option`` is the option parameter provided by the jIO user. The third parameter ``option`` is the option parameter provided by the jIO user.
Methods should return the following objects: Methods should return:
* **post()** --> ``success("created", {"id": new_generated_id})`` * **post()**, **put()**, **remove()** --> id of the document affected (string)
* **put()**, **remove()**, **putAttachment()** or **removeAttachment()** --> ``success(204)`` * **putAttachment()** or **removeAttachment()** --> no specific value
* **get()** --> ``success("ok", {"data": document_metadata})`` * **get()** --> document_metadata (object)
* **getAttachment()** --> * **getAttachment()** -->
.. code-block:: javascript .. code-block:: javascript
success("ok", {"data": binary_string, "content_type": content_type}) new Blob([data], {"type": content_type})
// or
success("ok", {"data": new Blob([data], {"type": content_type})})
* **allDocs()** --> ``success("ok", {"data": row_object})``
* **check()** -->
.. code-block:: javascript
// if metadata provides "_id" -> check document state
// if metadata doesn't promides "_id" -> check storage state
success("no_content")
// or
error("conflict", "corrupted", "incoherent document or storage")
* **repair()** -->
.. code-block:: javascript * **allDocs()** --> list of all documents (restricted by a query, if given). (object)
// if metadata provides "_id" -> repair document state
// if metadata doesn't promides "_id" -> repair storage state
success("no_content")
// or
error("conflict", "corrupted", "impossible to repair document or storage")
// DON'T DESIGN STORAGES IF THERE IS NO WAY
// TO REPAIR INCOHERENT STATES
After creating all methods, your storage must be added to jIO. This is done After creating all methods, your storage must be added to jIO. This is done
with the ``jIO.addStorage()`` method, which requires two parameters: the storage with the ``jIO.addStorage()`` method, which requires two parameters: the storage
...@@ -154,160 +114,3 @@ type (string) and a constructor (function). It is called like this: ...@@ -154,160 +114,3 @@ type (string) and a constructor (function). It is called like this:
Please refer to *localstorage.js* implementation for a good example on how to Please refer to *localstorage.js* implementation for a good example on how to
setup a storage and what methods are required. setup a storage and what methods are required.
Also keep in mind that jIO is a job-based library: whenever you trigger a method,
a job is created, which will later return a response, after being processed.
Job rules
---------
The jIO job manager follows several rules set at the creation of a new jIO
instance. When you try to call a method, jIO will create a job and will make
sure the job is really necessary and will be executed. Thanks to these job
rules, jIO knows what to do with the new job before adding it to the queue. You
can also add your own rules, as we're going to see now.
These are the jIO **default rules**:
.. code-block:: javascript
var jio_instance = jIO.createJIO(storage_description, {
"job_rules": [{
"code_name": "readers update",
"conditions": [
"sameStorageDescription",
"areReaders",
"sameMethod",
"sameParameters",
"sameOptions"
],
"action": "update"
}, {
"code_name": "metadata writers update",
"conditions": [
"sameStorageDescription",
"areWriters",
"useMetadataOnly",
"sameMethod",
"haveDocumentIds",
"sameParameters"
],
"action": "update"
}, {
"code_name": "writers wait",
"conditions": [
"sameStorageDescription",
"areWriters",
"haveDocumentIds",
"sameDocumentId"
],
"action": "wait"
}]
});
The following actions can be used:
* ``ok`` - accept the job
* ``wait`` - wait until the end of the selected job
* ``update`` - bind the selected job to this one
* ``deny`` - reject the job
The following condition functions can be used:
* ``sameStorageDescription`` - check if the storage descriptions are different.
* ``areWriters`` - check if the commands are ``post``, ``put``, ``putAttachment``, ``remove``, ``removeAttachment``, or ``repair``.
* ``areReaders`` - check if the commands are ``get``, ``getAttachment``, ``allDocs`` or ``check``.
* ``useMetadataOnly`` - check if the commands are ``post``, ``put``, ``get``, ``remove`` or ``allDocs``.
* ``sameMethod`` - check if the commands are equal.
* ``sameDocumentId`` - check if the document ids are equal.
* ``sameParameters`` - check if the metadata or param are equal (deep comparison).
* ``sameOptions`` - check if the command options are equal.
* ``haveDocumentIds`` - test if the two commands contain document ids.
Create Job Condition
--------------------
You can create two types of function: job condition, and job comparison.
.. code-block:: javascript
// Job Condition
// Check if the job is a get command
jIO.addJobRuleCondition("isGetMethod", function (job) {
return job.method === 'get';
});
// Job Comparison
// Check if the jobs have the same 'title' property
// only if they are strings
jIO.addJobRuleCondition("sameTitleIfString",
function (job, selected_job) {
if (typeof job.kwargs.title === 'string' &&
typeof selected_job.kwargs.title === 'string') {
return job.kwargs.title === selected_job.kwargs.title;
}
return false;
});
Add job rules
-------------
You just have to define job rules in the jIO options:
.. code-block:: javascript
// Do not accept to post or put a document which title is equal
// to another already running post or put document title
var jio_instance = jIO.createJIO(storage_description, {
"job_rules": [{
"code_name": "avoid similar title",
"conditions": [
"sameStorageDescription",
"areWriters",
"sameTitleIfString"
],
"action": "deny",
"before": "writers update" // optional
// "after": also exists
}]
});
Clear/Replace default job rules
-------------------------------
If a job's ``code_name`` is equal to ``readers update``, then adding this rule will replace it:
.. code-block:: javascript
var jio_instance = jIO.createJIO(storage_description, {
"job_rules": [{
"code_name": "readers update",
"conditions": [
"sameStorageDescription",
"areReaders",
"sameMethod",
"haveDocumentIds"
"sameParameters"
// sameOptions is removed
],
"action": "update"
}]
});
Or you can just clear all rules before adding new ones:
.. code-block:: javascript
var jio_instance = jIO.createJIO(storage_description, {
"clear_job_rules": true,
"job_rules": [{
// ...
}]
});
docs/getting_started.rst
View file @ 2d5c99d2
...@@ -79,8 +79,6 @@ Getting started ...@@ -79,8 +79,6 @@ Getting started
| Deletes a document's attachment | Deletes a document's attachment
``.allDocs()`` | ``my_jio.allDocs([options]);`` ``.allDocs()`` | ``my_jio.allDocs([options]);``
| Retrieves a list of existing documents | Retrieves a list of existing documents
``.check()`` | ``my_jio.check(document, [options]);``
| Checks the document state
``.repair()`` | ``my_jio.repair(document, [options]);`` ``.repair()`` | ``my_jio.repair(document, [options]);``
| Repairs the document | Repairs the document
======================= ====================================================== ======================= ======================================================
...@@ -122,29 +120,29 @@ Storage dependencies ...@@ -122,29 +120,29 @@ Storage dependencies
Storage connectors Storage connectors
^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^
* localstorage.js * Localstorage
* davstorage.js * MemoryStorage
* searchableencryptionstorage.js (depends on sjcl) (WIP) * IndexedDB
* s3storage.js (depends on sha1, jQuery) (WIP) * WebSQL
* DavStorage
* Dropbox
* Google Drive
* ERP5Storage
* s3storage.js (WIP)
* xwikistorage.js (depends on jQuery) (WIP) * xwikistorage.js (depends on jQuery) (WIP)
* erp5storage.js (WIP)
* restsqlstorage.js (depends on jQuery) (WIP)
* mioga2storage.js (depends on jQuery) (WIP)
Storage handlers Storage handlers
^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^
* indexstorage.js * Zipstorage
* gidstorage.js * ShaStorage
* splitstorage.js (WIP) * UUIDStorage
* replicatestorage.js (WIP) * QueryStorage
* CryptStorage
Revision based storage handlers * UnionStorage
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * FileSystemBridgeStorage
* Document Storage
* revisionstorage.js (depends on sha256) * Replicate Storage
* replicaterevisionstorage.js
Unit tests Unit tests
^^^^^^^^^^ ^^^^^^^^^^
...@@ -155,9 +153,5 @@ the test suite with each release. ...@@ -155,9 +153,5 @@ the test suite with each release.
Fork jIO Fork jIO
^^^^^^^^ ^^^^^^^^
The same source code is kept in three synchronized repositories. | Feel free to use the Gitlab repository:
Feel free to use any of them. | `GitLab <https://lab.nexedi.com/nexedi/jio.git>`_: ``git clone https://lab.nexedi.com/nexedi/jio.git``
* `GitHub <https://github.com/nexedi/jio>`_: ``git clone https://github.com/nexedi/jio.git``
* `Gitorius <https://gitorious.org/nexedi/jio>`_: ``git clone https://git.gitorious.org/nexedi/jio.git``
* `Git Erp5 <http://git.erp5.org/gitweb/jio.git>`_ (read only): ``git clone http://git.erp5.org/repos/jio.git``
docs/gid_storage.rst deleted 100644 → 0
View file @ 0a7b8a91
.. _gid-storage:
jIO GIDStorage
==============
A storage to enable interoperability between all kind of storages.
A global ID (GID) is a document id which represents a unique document. This ID
is then used to find this unique document on all types of backends.
This storage uses sub storage ``.allDocs()`` and queries to find unique
documents, and converts their ids to gids.
Where it can be used
--------------------
When you want to duplicate / synchronize / split / edit data in different kind of storages (ERP5 + XWiki + Dav + ...).
Storage Description
-------------------
* ``type`` - ``"gid"``
* ``sub_storage`` - the sub storage description.
* ``constraints`` - the constraints to use to generate a gid by defining metadata types for some kind of document.
Example:
.. code-block:: javascript
{
"type": "gid",
"sub_storage": {<storage description>},
"constraints": {
"default": { // constraints for all kind of documents
// "document metadata": "type of metadata"
"type": "list"
"title": "string"
},
"Text": { // document of type 'Text' additional constraints
"language": "string"
}
}
}
This description tells the *GIDStorage* to use 2 metadata attributes (``type``, ``title``) to define a
document as unique in the default case. If the document is of type ``Text``, then
the handler will use 3 metadata (``type``, ``title``, ``language``).
If these constraints are not respected, then the storage returns an error telling us to
review the document metadata. Here are samples of document respecting the above
constraints:
.. code-block:: javascript
{
"type": "Text",
"title": "Hello World!",
"language": "en"
}
{
"type": ["Text", "Web Page"],
"title": "My Web Page Title",
"language": "en-US",
"format": "text/html"
}
{
"type": "Image",
"title": "My Image Title"
}
Available metadata types are:
* ``"json"`` - The json value of the metadata.
* ``"string"`` - The value as string if it is not a list.
* ``"list"`` - The value as list.
* ``"date"`` - The value if it can be converted to a date (as string).
* ``"DCMIType"`` - A value matching one of the DCMIType Vocabulary (as string).
* ``"contentType"`` - A value which is a content type (as string).
* ``["DCMIType", "list"]`` - The value which contains a DCMIType (as list).
* ``[...]`` - make your own combination.
Document Requirements
---------------------
A metadata value must be a string. This string can be placed in an attribute within
a ``"content"`` key. The object can contains custom keys with string values. A
metadata object can contain several values. Example:
.. code-block:: javascript
{
"key": "value",
// or
"key": ["value1", "value2"],
// or
"key": {
"attribute name": "attribute value",
"content": "value"
},
// or
"key": [
{"scheme": "DCTERMS.URI", "content": "http://foo.com/bar"},
"value2",
"value3",
...
],
...
}
Metadata attributes which names begin with an underscore can contain anything.
.. code-block:: javascript
{
"_key": {"whatever": ["blue", []], "a": null}
}
Storage Requirements
--------------------
* This storage is not compatible with *RevisionStorage* and *ReplicateRevisionStorage*.
* Sub storages have to support options for queries and ``include_docs``.
Dependencies
------------
No dependency.
Suggested storage tree
----------------------
Replication between storages::
Replicate Storage
+-- GID Storage
| `-- Local Storage
+-- GID Storage
| `-- Remote Storage 1
`-- GID Storage
`-- Remote Storage 2
**CAUTION: All gid storage must have the same description!**
Offline application usage::
Replicate Storage
+-- Index Storage with DB in Local Storage
| `-- GID Storage
| `-- ERP5 Storage
`-- GID Storage
`-- Local Storage
**CAUTION: All gid storage must have the same description!**
docs/index.rst
View file @ 2d5c99d2
...@@ -38,9 +38,8 @@ jIO documentation ...@@ -38,9 +38,8 @@ jIO documentation
getting_started getting_started
manage_documents manage_documents
revision_storages replicate_storage
available_storages available_storages
gid_storage
query query
keys keys
metadata metadata
......
docs/manage_documents.rst
View file @ 2d5c99d2
...@@ -29,39 +29,16 @@ attachments are simple strings. ...@@ -29,39 +29,16 @@ attachments are simple strings.
{ {
// document metadata // document metadata
_id: 'Identifier',
title: 'A Title!', title: 'A Title!',
creator: 'Mr.Author' creator: 'Mr.Author'
} }
You can also retrieve document attachment metadata in this object.
.. code-block:: javascript
{
// document metadata
_id : 'Identifier',
title : 'A Title!',
creator: 'Mr.Author',
_attachments: {
// attachment metadata
'body.html': {
length: 12893,
digest: 'sha256-XXXX...',
content_type: 'text/html'
}
}
}
:ref:`Here <metadata-head>` is a draft about metadata to use with jIO. :ref:`Here <metadata-head>` is a draft about metadata to use with jIO.
Basic Methods Basic Methods
------------- -------------
Below you can see examples of the main jIO methods. All examples are using Below you can see examples of the main jIO methods.
revisions (as in revision storage or replicated revision storage), so you can
see how method calls should be made with either of these storages.
.. code-block:: javascript .. code-block:: javascript
...@@ -69,48 +46,48 @@ see how method calls should be made with either of these storages. ...@@ -69,48 +46,48 @@ see how method calls should be made with either of these storages.
var jio_instance = jIO.createJIO(storage_description); var jio_instance = jIO.createJIO(storage_description);
// create and store new document // create and store new document
jio_instance.post({title: 'some title'}). jio_instance.post({title: 'my document'}).
then(function (response) { then(function (response) {
// console.log(response); // console.log(response);
}); });
// create or update an existing document // create or update an existing document
jio_instance.put({_id: 'my_document', title: 'New Title'}). jio_instance.put('document_name', {title: 'another document'}).
then(function (response) { then(function (response) {
// console.log(response); // console.log(response);
}); });
// add an attachment to a document // add an attachment to a document
jio_instance.putAttachment({_id: 'my_document', jio_instance.putAttachment('document_name',
_attachment: 'its_attachment', 'attachment_name',
_data: 'abc', new Blob([data], {'type' : data_mimetype});
_mimetype: 'text/plain'}). ).
then(function (response) { then(function (response) {
// console.log(response); // console.log(response);
}); });
// read a document // read a document
jio_instance.get({_id: 'my_document'}). jio_instance.get('document_name').
then(function (response) { then(function (response) {
// console.log(response); // console.log(response);
}); });
// read an attachment // read an attachment
jio_instance.getAttachment({_id: 'my_document', jio_instance.getAttachment('document_name',
_attachment: 'its_attachment'}). 'attachment_name').
then(function (response) { then(function (response) {
// console.log(response); // console.log(response);
}); });
// delete a document and its attachment(s) // delete a document and its attachment(s)
jio_instance.remove({_id: 'my_document'}). jio_instance.remove('document_name').
then(function (response) { then(function (response) {
// console.log(response); // console.log(response);
}); });
// delete an attachment // delete an attachment
jio_instance.removeAttachment({_id: 'my_document', jio_instance.removeAttachment('document_name',
_attachment: 'its_attachment'}). 'attachment_name').
then(function (response) { then(function (response) {
// console.log(response); // console.log(response);
}); });
...@@ -157,87 +134,37 @@ Here is a list of responses returned by jIO according to methods and options: ...@@ -157,87 +134,37 @@ Here is a list of responses returned by jIO according to methods and options:
============================================== ================== =============================================== ============================================== ================== ===============================================
Available for Option Response (Callback first parameter) Available for Option Response (Callback first parameter)
============================================== ================== =============================================== ============================================== ================== ===============================================
``.post()``, ``.put()``, ``.remove()`` Any .. code-block:: javascript ``.post()``, ``.put()``, ``.remove()`` Any id of the document affected (string)
{ ``.putAttachment()``, ``.removeAttachment()`` Any no specific value
result: 'success',
method: 'post',
// or put or remove
id: 'my_doc_id',
status: 204,
statusText: 'No Content'
}
``.putAttachment()``, ``.removeAttachment()`` Any .. code-block:: javascript
{
result: 'success',
method: 'putAttachment',
// or removeAttachment
id: 'my_doc_id',
attachment: 'my_attachment_id',
status: 204,
statusText: 'No Content'
}
``.get()`` Any .. code-block:: javascript
{ ``.get()`` Any document_metadata (object)
result: 'success',
method: 'get',
id: 'my_doc_id',
status: 200,
statusText: 'Ok',
data: {
// Here, the document metadata
}
}
``.getAttachment()`` Any .. code-block:: javascript ``.getAttachment()`` Any .. code-block:: javascript
{ new Blob([data], {"type": content_type})
result: 'success',
method: 'getAttachment',
id: 'my_doc_id',
attachment: 'my_attachment_id',
status: 200,
statusText: 'Ok',
data: Blob // Here, the attachment content
}
``.allDocs()`` No option .. code-block:: javascript ``.allDocs()`` No option .. code-block:: javascript
{ {
result: 'success',
method: 'allDocs',
id: 'my_doc_id',
status: 200,
statusText: 'Ok',
data: {
total_rows: 1, total_rows: 1,
rows: [{ rows: [{
id: 'mydoc', id: 'mydoc',
key: 'mydoc', // optional
value: {}, value: {},
}] }]
} }
}
``.allDocs()`` include_docs: true .. code-block:: javascript ``.allDocs()`` include_docs: true .. code-block:: javascript
{ {
result: 'success',
method: 'allDocs',
id: 'my_doc_id',
status: 200,
statusText: 'Ok',
data: {
total_rows: 1, total_rows: 1,
rows: [{ rows: [{
id: 'mydoc', id: 'mydoc',
key: 'mydoc', // optional value: {
value: {},
doc: {
// Here, 'mydoc' metadata // Here, 'mydoc' metadata
} }
}] }]
} }
}
============================================== ================== =============================================== ============================================== ================== ===============================================
...@@ -248,12 +175,7 @@ In case of error, the ``errorCallback`` first parameter looks like: ...@@ -248,12 +175,7 @@ In case of error, the ``errorCallback`` first parameter looks like:
.. code-block:: javascript .. code-block:: javascript
{ {
result: 'error', status_code: 404,
method: 'get',
status: 404,
statusText: 'Not Found',
error: 'not_found',
reason: 'document missing',
message: 'Unable to get the requested document' message: 'Unable to get the requested document'
} }
...@@ -267,59 +189,42 @@ The following example creates a new jIO in localStorage and then posts a documen ...@@ -267,59 +189,42 @@ The following example creates a new jIO in localStorage and then posts a documen
.. code-block:: javascript .. code-block:: javascript
// create a new jIO // create a new jIO
var jio_instance = jIO.createJIO({ var jio_instance = jIO.createJIO({type: 'indexeddb'});
type: 'local',
username: 'usr',
application_name: 'app'
});
// post the document 'metadata' // post the document 'myVideo'
jio_instance.post({ jio_instance.put( 'metadata', {
title : 'My Video', title : 'My Video',
type : 'MovingImage', type : 'MovingImage',
format : 'video/ogg', format : 'video/ogg',
description : 'Images Compilation' description : 'Images Compilation'
}, function (err, response) { })
var id; .push(undefined, function(err) {
if (err) { return alert('Error posting the document metadata');
return alert('Error posting the document meta'); });
}
id = response.id;
// post a thumbnail attachment // post a thumbnail attachment
jio_instance.putAttachment({ jio_instance.putAttachment('metadatda',
_id: id, 'thumbnail',
_attachment: 'thumbnail', new Blob([my_image], {type: 'image/jpeg'})
_data: my_image, ).push(undefined, function(err) {
_mimetype: 'image/jpeg'
}, function (err, response) {
if (err) {
return alert('Error attaching thumbnail'); return alert('Error attaching thumbnail');
} });
// post video attachment // post video attachment
jio_instance.putAttachment({ jio_instance.putAttachment('metadatda',
_id: id, 'video',
_attachment: 'video', new Blob([my_video], {type: 'video/ogg'})
_data: my_video, ).push(undefined, function(err) {
_mimetype: 'video/ogg' return alert('Error attaching video');
}, function (err, response) {
if (err) {
return alert('Error attaching the video');
}
alert('Video Stored');
}); });
}); alert('Video Stored');
});
localStorage now contains: indexedDB Storage now contains:
.. code-block:: javascript .. code-block:: javascript
{ {
"jio/local/usr/app/12345678-1234-1234-1234-123456789012": { "/myVideo/": {
"_id": "12345678-1234-1234-1234-123456789012",
"title": "My Video", "title": "My Video",
"type": "MovingImage", "type": "MovingImage",
"format": "video/ogg", "format": "video/ogg",
...@@ -337,7 +242,7 @@ localStorage now contains: ...@@ -337,7 +242,7 @@ localStorage now contains:
} }
} }
}, },
"jio/local/usr/app/myVideo/thumbnail": "/9j/4AAQSkZ...", "/myVideo/thumbnail": "...",
"jio/local/usr/app/myVideo/video": "..." "/myVideo/video": "..."
} }
docs/metadata.rst
View file @ 2d5c99d2
...@@ -58,11 +58,6 @@ List of metadata to use ...@@ -58,11 +58,6 @@ List of metadata to use
Identification Identification
^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^
* **_id**
A specific jIO metadata which helps the storage to find a document
(can be a real path name, a dc:identifier, a uuid, ...). **String Only**
* **identifier** * **identifier**
| ``{"identifier": "http://domain/jio_home_page"}`` | ``{"identifier": "http://domain/jio_home_page"}``
......
docs/revision_storages.rst → docs/replicate_storage.rst
View file @ 2d5c99d2
.. _revision-storages-conflicts-and-resolution: .. _replicate-storage-conflicts-and-resolution:
Revision Storages: Conflicts and Resolution Replicate Storage: Conflicts and Resolution
=========================================== ===========================================
...@@ -14,25 +14,29 @@ defines a conflict as multiple versions of a document existing in a storage ...@@ -14,25 +14,29 @@ defines a conflict as multiple versions of a document existing in a storage
tree and a user trying to save on a version that does not match the latest tree and a user trying to save on a version that does not match the latest
version of the document. version of the document.
To keep track of document versions a revision storage must be used. When doing To keep track of document versions a replicate storage must be used. When doing
so, jIO creates a document tree file for every document. This file contains all so, jIO creates a document tree file for every document. This file contains all
existing versions and their status and is modified whenever a version is existing versions and their status and is modified whenever a version is
added/updated/removed or when storages are being synchronized. added/updated/removed or when storages are being synchronized.
How to solve conflicts How conflicts are handled
---------------------- -------------------------
The RemoteStorage takes in parameter two substorages, one "local" and one "remote".
The "local" storage can be remote, but it will be used for all the requests
like **get()**, **getAttachment()**, **allDocs()**...
Using the document tree, jIO tries to make every version of a document Using the document tree, jIO tries to make every version of a document
available on every storage. When multiple versions of a document exist, jIO available on the two storages. When multiple versions of a document exist,
will select the **latest**, **left-most** version on the document tree, along with the Jio will follow the rule set by the conflict_handling option, given at storage creation.
conflicting versions (when option **conflicts: true** is set in order for This option can one of the following numbers:
developers to setup a routine to solve conflicts.
* 0: no conflict resolution (throws an error when conflict is occuring)
* 1: keep the local state. (overwrites the remote document with local content)
* 2: keep the remote state. (overwrites the local document with remote content)
* 3: keep both copies (leave documents untouched, no signature update)
Technically, a conflict is solved by deleting alternative versions of a document
("cutting leaves off from the document tree"). When a user decides to keep a
version of a document and manually deletes all conflicting versions, the
storage tree is updated accordingly and the document is available in a single
version on all storages.
Simple Conflict Example Simple Conflict Example
----------------------- -----------------------
...@@ -45,123 +49,85 @@ your PC with your new email adress. Someone else changes this email from your PC ...@@ -45,123 +49,85 @@ your PC with your new email adress. Someone else changes this email from your PC
and once your smartphone is recharged, you go back online and the previous and once your smartphone is recharged, you go back online and the previous
update is executed. update is executed.
#. Set up the storage tree: #. Set up the replicate storage:
.. code-block:: javascript .. code-block:: javascript
var jio_instance = jIO.createJIO({ var jio_instance = jIO.createJIO({
// replicate revision storage // replicate storage
type: 'replicaterevision', type: 'replicate',
storagelist:[{ local_sub_storage : {
type: 'revision', type: 'local',
sub_storage: {
type: 'dav',
... ...
} }
}, { remote_sub_storage: {
type: 'revision', type: 'dav',
sub_storage: {
type: 'local',
... ...
} }
}] conflict_handling: ...
}); });
#. Create the namecard on your smartphone: #. 1) Create the namecard on your smartphone:
.. code-block:: javascript .. code-block:: javascript
jio_instance.post({ jio_instance.put("myNameCard", {
_id: 'myNameCard', email: 'jb@td.com'
email: 'me@web.com'
}).then(function (response) { }).then(function (response) {
// response.id -> 'myNameCard' // response -> 'myNameCard'
// response.rev -> '1-5782E71F1E4BF698FA3793D9D5A96393'
}); });
This will create the document on your WebDAV and local storage This will create the document on your WebDAV and local storage
#. Someone else updates your shared namecard on WebDAV: #. 2) Someone else updates your shared namecard on WebDAV:
.. code-block:: javascript
jio_instance.put({
email: 'my_new_me@web.com',
_id: 'myNameCard'
_rev: '1-5782E71F1E4BF698FA3793D9D5A96393'
}).then(function (response) {
// response.id -> 'myNameCard'
// response.rev -> '2-068E73F5B44FEC987B51354DFC772891'
});
Your smartphone is offline, so now you will have one version (1-578...) on
your smartphone and another version on WebDAV (2-068...) on your PC.
#. You modify the namecard while being offline:
.. code-block:: javascript .. code-block:: javascript
jio_instance.get({_id: 'myNameCard'}).then(function (response) { jio_instance.put(myNameCard, {
// response.id -> 'myNameCard' email: 'kyle@td.com',
// response.rev -> '1-5782E71F1E4BF698FA3793D9D5A96393'
// response.data.email -> 'me@web.com'
return jio_instance.put({
_id: 'myNameCard',
email: 'me_again@web.com'
});
}).then(function (response) { }).then(function (response) {
// response.id -> 'myNameCard' // response -> 'myNameCard'
// response.rev -> '2-3753476B70A49EA4D8C9039E7B04254C'
}); });
Your smartphone is offline, so now you will have one version on
your smartphone and another version on WebDAV on your PC.
#. Later, your smartphone is online and you retrieve the other version of the namecard: #. 3) Later, your smartphone is online and you modify your email:
.. code-block:: javascript .. code-block:: javascript
jio_instance.get({_id: 'myNameCard'}).then(function (response) { jio_instance.get('myNameCard').then(function (response) {
// response.id -> 'myNameCard' // response.email -> 'jb@td.com'
// response.rev -> '2-3753476B70A49EA4D8C9039E7B04254C' // the get() method checks only on your local storage
// response.data.email -> 'me_again@web.com' // and doesn't warn you about remote modifications.
});
When multiple versions of a document are available, jIO returns the latest,
left-most version on the document tree (2-375... and labels all other
versions as conflicting 2-068...).
#. Retrieve conflicts by setting option:
.. code-block:: javascript return jio_instance.put('myNameCard', {
email: 'jack@td.com'
jio_instance.get({_id: 'myNameCard'}, { })
conflicts: true .then(function (response) {
}).then(function (response) { // response -> 'myNameCard'
// response.id -> 'myNameCard'
// response.rev -> '2-3753476B70A49EA4D8C9039E7B04254C',
// response.conflicts -> ['2-068E73F5B44FEC987B51354DFC772891']
}); });
The conflicting version (*2-068E...*) is displayed, because **{conflicts: true}** was | Your latest modification of the email is: "jack@td.com"
specified in the GET call. Deleting either version will solve the conflict. | The modification from the other user is: "kyle@td.com"
#. Delete the conflicting version: If your conflict_handling option was:
.. code-block:: javascript * | 0: the email is:
| -"kyle@td.com" on WebDAV
| -"jack@td.com" on your local storage
| The storage rejects your latest modification,
| you get an error because local and remote documents are desynchronized.
| The documents in local and remote state are left untouched.
jio_instance.remove({ * | 1: the email is: "jack@td.com" on both storages
_id: 'myNameCard', | The storage pushes the local modification, which is yours.
_rev: '2-068E73F5B44FEC987B51354DFC772891'
}).then(function (response) {
// response.id -> 'myNameCard'
// response.rev -> '3-28910A4937537B5168E772896B70EC98'
});
When deleting the conflicting version of your namecard, jIO removed it * | 2: the email is: "kyle@td.com" on both storages
from all storages and set the document tree leaf of that version to | The storage keeps the remote modification, which is from the other user.
*deleted*. All storages now contain just a single version of the namecard | Your local storage is modified to fit the state of the remote storage.
(2-3753...). Note that, on the document tree, removing a revison will
create a new revision with status set to *deleted*.
* | 3: the email is: "jack@td.com" on both storages
| The storage doesn't do synchronization, and pushes your modification
| without checking if the remote storage has been changed or not
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7