Skip to content

jio
jio
Commit 2d5c99d2 authored by lucas.parsy's avatar lucas.parsy Committed by Romain Courteaud
1 Browse files
Options

Update documentation. 

Squashed commit of the following:

commit 3ddc5e522969d8e5270dc408eec71fc0847c40ce
Author: Lucas Parsy <lucas.parsy@nexedi.com>
Date:   Thu Dec 17 16:44:19 2015 +0100

    corrected misspellings and ambiguous examples in manage_document file.
    deleted gid and revision storage documentation.

commit 85d30999afdd227c8d4085fcd2df329f0d95333e
Author: Lucas Parsy <lucas.parsy@nexedi.com>
Date:   Tue Dec 15 15:13:23 2015 +0100

    updated documentation
    renamed "revision_storage.rst" "replicate_storage.rst"
    you need to "compile" the files (go in doc/ directory and type "make")
    to have the html files.
parent 0a7b8a91
Hide whitespace changes
Inline Side-by-side
Showing with 472 additions and 688 deletions
docs/available_storages.rst
View file @ 2d5c99d2
......@@ -16,80 +16,114 @@ When building storage trees, there is no limit on the number of storages you
can use. The only thing you have to be aware of is compatibility of simple and
revision based storages.
Connectors
----------
LocalStorage
^^^^^^^^^^^^
Three methods are provided:
This storage has only one document, so **post**, **put**, **remove** and **get** method are useless on it.
* ``.createDescription(username, [application_name], [mode='localStorage'])``
* ``.createLocalDescription(username, [application_name])``
* ``.createMemoryDescription(username, [application_name])``
=============== ========== ========== ============================================================
parameter required? type description
=============== ========== ========== ============================================================
``type`` yes string name of the storage type (here: "local")
``sessiononly`` no boolean | false: create a storage with unlimited duration.
| true: the storage duration is limited to the user session.
| (default to false)
=============== ========== ========== ============================================================
All parameters are strings.
Examples:
Example:
.. code-block:: javascript
// to work on browser localStorage
var jio = jIO.createJIO(local_storage.createDescription('me'));
var jio = jIO.createJIO({
type: "local",
sessiononly: true
});
// to work on browser memory
var jio = jIO.createJIO(local_storage.createMemoryDescription('me'));
MemoryStorage
^^^^^^^^^^^^^
| Stores the data in a Javascript object, in memory.
| The storage's data isn't saved when your web page is closed or reloaded.
| The storage doesn't take any argument at creation.
// or
{
"type": "local",
"username": "me",
"application_name": "my app name", // optional
"mode": "memory" // optional, "localStorage" by default
}
Example:
.. code-block:: javascript
var jio = jIO.createJIO({type: "memory"});
IndexedDB
^^^^^^^^^^^^
================= ========== ========== ==========================================================
parameter required? type description
================= ========== ========== ==========================================================
``type`` yes string name of the storage type (here: "indexeddb")
``database`` yes string name of the database.
================= ========== ========== ==========================================================
DavStorage
^^^^^^^^^^
The method ``dav_storage.createDescription()`` generates a DAV storage description for
*none*, *basic* or *digest* authentication.
Example:
.. code-block:: javascript
{
"type": "indexeddb",
"database": "mydb"
}
WebSQL
^^^^^^^^^^^^
================= ========== ========== ==========================================================
parameter required? type description
================= ========== ========== ==========================================================
``type`` yes string name of the storage type (here: "websql")
``database`` yes string name of the database.
================= ========== ========== ==========================================================
NB: digest **is not implemented yet**.
Example:
.. code-block:: javascript
dav_storage.createDescription(url, auth_type,
[realm], [username], [password]);
{
"type": "websql",
"database": "mydb"
}
All parameters are strings.
DavStorage
^^^^^^^^^^
============= ========================
parameter required?
============= ========================
``url`` yes
``auth_type`` yes
``realm`` if auth_type == 'digest'
``username`` if auth_type != 'none'
``password`` if auth-type != 'none'
============= ========================
================ ========== ========== ==========================================================
parameter required? type description
================ ========== ========== ==========================================================
``type`` yes string name of the storage type (here: "dav")
``url`` yes string url of your webdav server
``basic_login`` no string | login and password of your dav, base64 encoded like this:
| ``btoa(username + ":" + password)``
================ ========== ========== ==========================================================
If ``auth_type`` is the string ``"none"``, then ``realm``, ``username`` and ``password`` are never used.
Descriptions:
Example:
.. code-block:: javascript
// No authentication
{
"type": "dav",
"url": url
"url": url
}
// Basic authentication
{
"type": "dav",
"url": "url,
"type": "dav",
"url": url,
"basic_login": btoa(username + ":" + password)
}
......@@ -98,169 +132,251 @@ Descriptions:
**Be careful**: The generated description never contains a readable password, but
for basic authentication, the password is just base64 encoded.
S3Storage
^^^^^^^^^
Live tests OK!
Dropbox
^^^^^^^
Here is a basic description for jIO. Documentation comming soon.
================= ========== ========== ==========================================================
parameter required? type description
================= ========== ========== ==========================================================
``type`` yes string name of the storage type (here: "dropbox")
``access_token`` yes string access token for your account.
See specific documentation on how to retreive it.
``root`` no string | "dropbox" for full access to account files,
| "sandbox" for app limited file access.
| default to "dropbox".
================= ========== ========== ==========================================================
Example:
.. code-block:: javascript
{
"type": "s3",
"AWSIdentifier": "my aws identifier",
"password": "my password",
"server": "bucket_name"
"type": "dropbox",
"access_token": "sample_token"
"root": "dropbox"
}
XWikiStorage
Google Drive
^^^^^^^^^^^^
Work is in progress.
================= ========== ========== ==========================================================
parameter required? type description
================= ========== ========== ==========================================================
``type`` yes string name of the storage type (here: "gdrive")
``access_token`` yes string access token for your account.
See specific documentation on how to retreive it.
``trashing`` no boolean | true: sends files to the trash bin when doing a "remove"
| false: deletes permanently files when doing a "remove"
| default to true.
================= ========== ========== ==========================================================
Example:
.. code-block:: javascript
Searchable Encryption Storage
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
{
"type": "gdrive",
"access_token": "sample_token"
"trashing": true
}
Comes with a specific server with can query encrypted documents.
ERP5Storage
^^^^^^^^^^^
=========================== ========== ========== ==========================================================
parameter required? type description
=========================== ========== ========== ==========================================================
``type`` yes string name of the storage type (here: "erp5")
``url`` yes string url of your erp5 account.
``default_view_reference`` no string | reference of the action used
| for the delivering of the document
=========================== ========== ========== ==========================================================
Work is in progress. Documentation comming soon.
Example:
.. code-block:: javascript
{
"type": "searchableencryption",
"password": "your password",
"url": "http://your/url"
"type": "erp5",
"url": erp5_url
}
Handlers
--------
IndexStorage
^^^^^^^^^^^^
Zipstorage
^^^^^^^^^^
This handler indexes documents metadata into a database (which is a simple
document) to increase the speed of ``.allDocs()`` requests. However, it is not able to
manage the ``include_docs`` option.
This handler compresses and decompresses files to reduce network and storage usage.
Here is the description:
Usage:
.. code-block:: javascript
{
type: 'index',
indices: [{
// doc id where to store indices
id: 'index_title_subject.json',
// metadata to index
index: ['title', 'subject'],
attachment: 'db.json', // default 'body'
// additional metadata to add to database, default undefined
metadata: {
type: 'Dataset',
format: 'application/json',
title: 'My index database',
creator: 'Me'
},
// default equal to parent sub_storage field
sub_storage: <sub storage where to store index>
}, {
id: 'index_year.json',
index: 'year'
...
}],
sub_storage: <sub storage description>
}
{
"type": "zip",
"sub_storage": <your storage>
}
GIDStorage
ShaStorage
^^^^^^^^^^
:ref:`Full description here <gid-storage>`.
This handler provides a post method that creates a document that has for name the SHA-1 hash of his parameters.
SplitStorage
^^^^^^^^^^^^
.. code-block:: javascript
{
"type": "sha",
"sub_storage": <your storage>
}
Work is in progress. The interoperability is not enabled yet.
UUIDStorage
^^^^^^^^^^^
This storage splits metadata and attachment data to *n* parts where *n* is the
number of sub storages. Each parts are stored on one sub storage only.
This handler provides a post method to create a document that has a unique ID for name.
.. code-block:: javascript
{
type: 'split',
storage_list: [
<sub storage description>,
...
]
}
{
"type": "uuid",
"sub_storage": <your storage>
}
QueryStorage
^^^^^^^^^^^^
Other split modes will be added later.
This handler provides an allDocs method with queries support to the substorage.
.. code-block:: javascript
Replicate Storage
^^^^^^^^^^^^^^^^^
{
"type": "query",
"sub_storage": <your storage>
}
CryptStorage
^^^^^^^^^^^^
Work is in progress.
| This handler encrypts and decrypts attachments before storing them.
| You need to generate a Crypto key at the JSON format to use the handler.
| (see https://developer.mozilla.org/fr/docs/Web/API/Window/crypto for more informations)
Usage:
.. code-block:: javascript
{
type: 'replicate',
storage_list: [
<sub storage description>,
...
]
}
var key,
jsonKey,
jio;
//creation of an encryption/decryption key.
Revision Based Handlers
-----------------------
crypto.subtle.generateKey({name: "AES-GCM",length: 256},
(true), ["encrypt", "decrypt"])
.then(function(res){key = res;});
window.crypto.subtle.exportKey("jwk", key)
.then(function(res){jsonKey = res})
A revision based handler is a storage which is able to do some document
versioning using simple storages listed above.
//creation of the storage
On jIO command parameter, ``_id`` is still used to identify a document, but
another id ``_rev`` must be defined to use a specific revision of that document.
jio = jIO.createJIO({
{
"type": "crypt",
"key": json_key
"sub_storage": <your storage>
}
On command responses, you will find another field ``rev`` which will represent the
new revision produced by your action. All the document history is kept unless
you decide to delete older revisions.
Other fields ``conflicts``, ``revisions`` and ``revs_info`` can be returned if the
options **conflicts: true**, **revs: true** or **revs_info: true** are set.
UnionStorage
^^^^^^^^^^^^
Revision Storage
^^^^^^^^^^^^^^^^
This handler takes in argument an array of storages.
When using a method, UnionStorage tries it on the first storage of the array,
and, in case of failure, tries with the next storage,
and repeats the operation until success, or end of storage's array.
.. code-block:: javascript
{
"type": "union",
"storage_list": [
sub_storage_description_1,
sub_storage_description_2,
sub_storage_description_X
]
}
This backend uses its sub storage to manage document and their revision. For
more information, :ref:`see here <revision-storages-conflicts-and-resolution>`.
FileSystemBridgeStorage
^^^^^^^^^^^^^^^^^^^^^^^
Description:
This handler adds an abstraction level on top of the webDav Jio storage,
ensuring each document has only one attachment, and limiting the storage to one repertory.
.. code-block:: javascript
{
"type": "revision",
"sub_storage": <sub storage description>
"type": "drivetojiomapping",
"sub_storage": <your dav storage>
}
Document Storage
^^^^^^^^^^^^^^^^
This handler creates a storage from a document in a storage,
by filling his attachments with a new jIO storage.
Replicate Revision Storage
^^^^^^^^^^^^^^^^^^^^^^^^^^
====================== ========== ========== ============================================================
parameter required? type description
====================== ========== ========== ============================================================
``type`` yes string name of the storage type (here: "document")
``document_id`` no string id of the document to use.
``repair_attachment`` no boolean verify if the document is in good state. (default to false)
====================== ========== ========== ============================================================
Replicate revisions across multiple revision based storages.
Replicate Storage
^^^^^^^^^^^^^^^^^
Description:
Replicate Storage synchronizes documents between a local and a remote storage.
=============================== ========== ========== ============================================================
parameter required? type description
=============================== ========== ========== ============================================================
``type`` yes string name of the storage type (here: "replicate")
``local_sub_storage`` yes object local sub_storage description.
``remote_sub_storage`` yes object remote sub_storage description.
``query_options`` no object query object to limit the synchronisation to specific files.
``use_remote_post`` no boolean | true: at file modification, modifies the local file id.
| false: at file modification, modifies the remote file id.
| default to false.
``conflict_handling`` no number | 0: no conflict resolution (throws error)
| 1: keep the local state.
| 2: keep the remote state.
| 3: keep both states (no signature update)
| default to 0.
``check_local_modification`` no boolean synchronise when local files are modified.
``check_local_creation`` no boolean synchronise when local files are created.
``check_local_deletion`` no boolean synchronise when local files are deleted.
``check_remote_modification`` no boolean synchronise when remote files are modified.
``check_remote_creation`` no boolean synchronise when local files are created.
``check_remote_deletion`` no boolean synchronise when local files are deleted.
=============================== ========== ========== ============================================================
synchronisation parameters are set by default to true.
.. code-block:: javascript
{
"type": "revision",
"storage_list": [
<revision based sub storage description>,
...
]
}
{
type: 'replicate',
local_sub_storage: { 'type': 'local'}
remote_sub_storage: {
'type': 'dav',
'url': 'http://mydav.com',
'basic_login': 'aGFwcHkgZWFzdGVy'
}
use_remote_post: false,
conflict_handling : 2,
check_local_creation: false,
check_remote_deletion: false
}
docs/developers.rst
View file @ 2d5c99d2
......@@ -12,7 +12,7 @@ If you want to modify or build jIO yourself, you need to
* Clone from a repository
``$ git clone https://github.com/nexedi/jio.git``
``$ git clone https://lab.nexedi.com/nexedi/jio.git``
* Install `NodeJS <http://nodejs.org/>`_ (including ``npm``)
......@@ -53,8 +53,8 @@ Create a constructor:
}
}
Create 10 methods: ``post``, ``put``, ``putAttachment``, ``get``, ``getAttachment``,
``remove``, ``removeAttachment``, ``allDocs``, ``check`` and ``repair``.
Create 9 methods: ``post``, ``put``, ``putAttachment``, ``get``, ``getAttachment``,
``remove``, ``removeAttachment``, ``allDocs``. ``repair`` method is optional.
.. code-block:: javascript
......@@ -81,66 +81,26 @@ Create 10 methods: ``post``, ``put``, ``putAttachment``, ``get``, ``getAttachmen
(To help you design your methods, some tools are provided by jIO.util.)
The first parameter command provides some methods to act on the jIO job:
* ``success``, to tell jIO that the job is successfully terminated
``command.success(status[Text], [{custom key to add to the response}]);``
* ``resolve``, is equal to success
* ``error``, to tell jIO that the job cannot be done
``command.error(status[Text], [reason], [message], [{custom key to add to the response}])``
* ``retry``, to tell jIO that the job cannot be done now, but can be retried later. (same API than error)
* ``reject``, to tell jIO that the job cannot be done, let jIO to decide whether to retry or not. (same API than error)
The second parameter ``metadata`` or ``param`` is the first parameter provided by the jIO user.
The third parameter ``option`` is the option parameter provided by the jIO user.
Methods should return the following objects:
Methods should return:
* **post()** --> ``success("created", {"id": new_generated_id})``
* **post()**, **put()**, **remove()** --> id of the document affected (string)
* **put()**, **remove()**, **putAttachment()** or **removeAttachment()** --> ``success(204)``
* **putAttachment()** or **removeAttachment()** --> no specific value
* **get()** --> ``success("ok", {"data": document_metadata})``
* **get()** --> document_metadata (object)
* **getAttachment()** -->
.. code-block:: javascript
success("ok", {"data": binary_string, "content_type": content_type})
// or
success("ok", {"data": new Blob([data], {"type": content_type})})
new Blob([data], {"type": content_type})
* **allDocs()** --> ``success("ok", {"data": row_object})``
* **check()** -->
.. code-block:: javascript
* **allDocs()** --> list of all documents (restricted by a query, if given). (object)
// if metadata provides "_id" -> check document state
// if metadata doesn't promides "_id" -> check storage state
success("no_content")
// or
error("conflict", "corrupted", "incoherent document or storage")
* **repair()** -->
.. code-block:: javascript
// if metadata provides "_id" -> repair document state
// if metadata doesn't promides "_id" -> repair storage state
success("no_content")
// or
error("conflict", "corrupted", "impossible to repair document or storage")
// DON'T DESIGN STORAGES IF THERE IS NO WAY
// TO REPAIR INCOHERENT STATES
After creating all methods, your storage must be added to jIO. This is done
with the ``jIO.addStorage()`` method, which requires two parameters: the storage
......@@ -154,160 +114,3 @@ type (string) and a constructor (function). It is called like this:
Please refer to *localstorage.js* implementation for a good example on how to
setup a storage and what methods are required.
Also keep in mind that jIO is a job-based library: whenever you trigger a method,
a job is created, which will later return a response, after being processed.
Job rules
---------
The jIO job manager follows several rules set at the creation of a new jIO
instance. When you try to call a method, jIO will create a job and will make
sure the job is really necessary and will be executed. Thanks to these job
rules, jIO knows what to do with the new job before adding it to the queue. You
can also add your own rules, as we're going to see now.
These are the jIO **default rules**:
.. code-block:: javascript
var jio_instance = jIO.createJIO(storage_description, {
"job_rules": [{
"code_name": "readers update",
"conditions": [
"sameStorageDescription",
"areReaders",
"sameMethod",
"sameParameters",
"sameOptions"
],
"action": "update"
}, {
"code_name": "metadata writers update",
"conditions": [
"sameStorageDescription",
"areWriters",
"useMetadataOnly",
"sameMethod",
"haveDocumentIds",
"sameParameters"
],
"action": "update"
}, {
"code_name": "writers wait",
"conditions": [
"sameStorageDescription",
"areWriters",
"haveDocumentIds",
"sameDocumentId"
],
"action": "wait"
}]
});
The following actions can be used:
* ``ok`` - accept the job
* ``wait`` - wait until the end of the selected job
* ``update`` - bind the selected job to this one
* ``deny`` - reject the job
The following condition functions can be used:
* ``sameStorageDescription`` - check if the storage descriptions are different.
* ``areWriters`` - check if the commands are ``post``, ``put``, ``putAttachment``, ``remove``, ``removeAttachment``, or ``repair``.
* ``areReaders`` - check if the commands are ``get``, ``getAttachment``, ``allDocs`` or ``check``.
* ``useMetadataOnly`` - check if the commands are ``post``, ``put``, ``get``, ``remove`` or ``allDocs``.
* ``sameMethod`` - check if the commands are equal.
* ``sameDocumentId`` - check if the document ids are equal.
* ``sameParameters`` - check if the metadata or param are equal (deep comparison).
* ``sameOptions`` - check if the command options are equal.
* ``haveDocumentIds`` - test if the two commands contain document ids.
Create Job Condition
--------------------
You can create two types of function: job condition, and job comparison.
.. code-block:: javascript
// Job Condition
// Check if the job is a get command
jIO.addJobRuleCondition("isGetMethod", function (job) {
return job.method === 'get';
});
// Job Comparison
// Check if the jobs have the same 'title' property
// only if they are strings
jIO.addJobRuleCondition("sameTitleIfString",
function (job, selected_job) {
if (typeof job.kwargs.title === 'string' &&
typeof selected_job.kwargs.title === 'string') {
return job.kwargs.title === selected_job.kwargs.title;
}
return false;
});
Add job rules
-------------
You just have to define job rules in the jIO options:
.. code-block:: javascript
// Do not accept to post or put a document which title is equal
// to another already running post or put document title
var jio_instance = jIO.createJIO(storage_description, {
"job_rules": [{
"code_name": "avoid similar title",
"conditions": [
"sameStorageDescription",
"areWriters",
"sameTitleIfString"
],
"action": "deny",
"before": "writers update" // optional
// "after": also exists
}]
});
Clear/Replace default job rules
-------------------------------
If a job's ``code_name`` is equal to ``readers update``, then adding this rule will replace it:
.. code-block:: javascript
var jio_instance = jIO.createJIO(storage_description, {
"job_rules": [{
"code_name": "readers update",
"conditions": [
"sameStorageDescription",
"areReaders",
"sameMethod",
"haveDocumentIds"
"sameParameters"
// sameOptions is removed
],
"action": "update"
}]
});
Or you can just clear all rules before adding new ones:
.. code-block:: javascript
var jio_instance = jIO.createJIO(storage_description, {
"clear_job_rules": true,
"job_rules": [{
// ...
}]
});
docs/getting_started.rst
View file @ 2d5c99d2
......@@ -79,8 +79,6 @@ Getting started
| Deletes a document's attachment
``.allDocs()`` | ``my_jio.allDocs([options]);``
| Retrieves a list of existing documents
``.check()`` | ``my_jio.check(document, [options]);``
| Checks the document state
``.repair()`` | ``my_jio.repair(document, [options]);``
| Repairs the document
======================= ======================================================
......@@ -122,29 +120,29 @@ Storage dependencies
Storage connectors
^^^^^^^^^^^^^^^^^^
* localstorage.js
* davstorage.js
* searchableencryptionstorage.js (depends on sjcl) (WIP)
* s3storage.js (depends on sha1, jQuery) (WIP)
* Localstorage
* MemoryStorage
* IndexedDB
* WebSQL
* DavStorage
* Dropbox
* Google Drive
* ERP5Storage
* s3storage.js (WIP)
* xwikistorage.js (depends on jQuery) (WIP)
* erp5storage.js (WIP)
* restsqlstorage.js (depends on jQuery) (WIP)
* mioga2storage.js (depends on jQuery) (WIP)
Storage handlers
^^^^^^^^^^^^^^^^
* indexstorage.js
* gidstorage.js
* splitstorage.js (WIP)
* replicatestorage.js (WIP)
Revision based storage handlers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* revisionstorage.js (depends on sha256)
* replicaterevisionstorage.js
* Zipstorage
* ShaStorage
* UUIDStorage
* QueryStorage
* CryptStorage
* UnionStorage
* FileSystemBridgeStorage
* Document Storage
* Replicate Storage
Unit tests
^^^^^^^^^^
......@@ -155,9 +153,5 @@ the test suite with each release.
Fork jIO
^^^^^^^^
The same source code is kept in three synchronized repositories.
Feel free to use any of them.
* `GitHub <https://github.com/nexedi/jio>`_: ``git clone https://github.com/nexedi/jio.git``
* `Gitorius <https://gitorious.org/nexedi/jio>`_: ``git clone https://git.gitorious.org/nexedi/jio.git``
* `Git Erp5 <http://git.erp5.org/gitweb/jio.git>`_ (read only): ``git clone http://git.erp5.org/repos/jio.git``
| Feel free to use the Gitlab repository:
| `GitLab <https://lab.nexedi.com/nexedi/jio.git>`_: ``git clone https://lab.nexedi.com/nexedi/jio.git``
docs/gid_storage.rst deleted 100644 → 0
View file @ 0a7b8a91
.. _gid-storage:
jIO GIDStorage
==============
A storage to enable interoperability between all kind of storages.
A global ID (GID) is a document id which represents a unique document. This ID
is then used to find this unique document on all types of backends.
This storage uses sub storage ``.allDocs()`` and queries to find unique
documents, and converts their ids to gids.
Where it can be used
--------------------
When you want to duplicate / synchronize / split / edit data in different kind of storages (ERP5 + XWiki + Dav + ...).
Storage Description
-------------------
* ``type`` - ``"gid"``
* ``sub_storage`` - the sub storage description.
* ``constraints`` - the constraints to use to generate a gid by defining metadata types for some kind of document.
Example:
.. code-block:: javascript
{
"type": "gid",
"sub_storage": {<storage description>},
"constraints": {
"default": { // constraints for all kind of documents
// "document metadata": "type of metadata"
"type": "list"
"title": "string"
},
"Text": { // document of type 'Text' additional constraints
"language": "string"
}
}
}
This description tells the *GIDStorage* to use 2 metadata attributes (``type``, ``title``) to define a
document as unique in the default case. If the document is of type ``Text``, then
the handler will use 3 metadata (``type``, ``title``, ``language``).
If these constraints are not respected, then the storage returns an error telling us to
review the document metadata. Here are samples of document respecting the above
constraints:
.. code-block:: javascript
{
"type": "Text",
"title": "Hello World!",
"language": "en"
}
{
"type": ["Text", "Web Page"],
"title": "My Web Page Title",
"language": "en-US",
"format": "text/html"
}
{
"type": "Image",
"title": "My Image Title"
}
Available metadata types are:
* ``"json"`` - The json value of the metadata.
* ``"string"`` - The value as string if it is not a list.
* ``"list"`` - The value as list.
* ``"date"`` - The value if it can be converted to a date (as string).
* ``"DCMIType"`` - A value matching one of the DCMIType Vocabulary (as string).
* ``"contentType"`` - A value which is a content type (as string).
* ``["DCMIType", "list"]`` - The value which contains a DCMIType (as list).
* ``[...]`` - make your own combination.
Document Requirements
---------------------
A metadata value must be a string. This string can be placed in an attribute within
a ``"content"`` key. The object can contains custom keys with string values. A
metadata object can contain several values. Example:
.. code-block:: javascript
{
"key": "value",
// or
"key": ["value1", "value2"],
// or
"key": {
"attribute name": "attribute value",
"content": "value"
},
// or
"key": [
{"scheme": "DCTERMS.URI", "content": "http://foo.com/bar"},
"value2",
"value3",
...
],
...
}
Metadata attributes which names begin with an underscore can contain anything.
.. code-block:: javascript
{
"_key": {"whatever": ["blue", []], "a": null}
}
Storage Requirements
--------------------
* This storage is not compatible with *RevisionStorage* and *ReplicateRevisionStorage*.
* Sub storages have to support options for queries and ``include_docs``.
Dependencies
------------
No dependency.
Suggested storage tree
----------------------
Replication between storages::
Replicate Storage
+-- GID Storage
| `-- Local Storage
+-- GID Storage
| `-- Remote Storage 1
`-- GID Storage
`-- Remote Storage 2
**CAUTION: All gid storage must have the same description!**
Offline application usage::
Replicate Storage
+-- Index Storage with DB in Local Storage
| `-- GID Storage
| `-- ERP5 Storage
`-- GID Storage
`-- Local Storage
**CAUTION: All gid storage must have the same description!**
docs/index.rst
View file @ 2d5c99d2
......@@ -38,9 +38,8 @@ jIO documentation
getting_started
manage_documents
revision_storages
replicate_storage
available_storages
gid_storage
query
keys
metadata
......
docs/manage_documents.rst
View file @ 2d5c99d2
......@@ -29,39 +29,16 @@ attachments are simple strings.
{
// document metadata
_id: 'Identifier',
title: 'A Title!',
creator: 'Mr.Author'
}
You can also retrieve document attachment metadata in this object.
.. code-block:: javascript
{
// document metadata
_id : 'Identifier',
title : 'A Title!',
creator: 'Mr.Author',
_attachments: {
// attachment metadata
'body.html': {
length: 12893,
digest: 'sha256-XXXX...',
content_type: 'text/html'
}
}
}
:ref:`Here <metadata-head>` is a draft about metadata to use with jIO.
Basic Methods
-------------
Below you can see examples of the main jIO methods. All examples are using
revisions (as in revision storage or replicated revision storage), so you can
see how method calls should be made with either of these storages.
Below you can see examples of the main jIO methods.
.. code-block:: javascript
......@@ -69,48 +46,48 @@ see how method calls should be made with either of these storages.
var jio_instance = jIO.createJIO(storage_description);
// create and store new document
jio_instance.post({title: 'some title'}).
jio_instance.post({title: 'my document'}).
then(function (response) {
// console.log(response);
});
// create or update an existing document
jio_instance.put({_id: 'my_document', title: 'New Title'}).
jio_instance.put('document_name', {title: 'another document'}).
then(function (response) {
// console.log(response);
});
// add an attachment to a document
jio_instance.putAttachment({_id: 'my_document',
_attachment: 'its_attachment',
_data: 'abc',
_mimetype: 'text/plain'}).
jio_instance.putAttachment('document_name',
'attachment_name',
new Blob([data], {'type' : data_mimetype});
).
then(function (response) {
// console.log(response);
});
// read a document
jio_instance.get({_id: 'my_document'}).
jio_instance.get('document_name').
then(function (response) {
// console.log(response);
});
// read an attachment
jio_instance.getAttachment({_id: 'my_document',
_attachment: 'its_attachment'}).
jio_instance.getAttachment('document_name',
'attachment_name').
then(function (response) {
// console.log(response);
});
// delete a document and its attachment(s)
jio_instance.remove({_id: 'my_document'}).
jio_instance.remove('document_name').
then(function (response) {
// console.log(response);
});
// delete an attachment
jio_instance.removeAttachment({_id: 'my_document',
_attachment: 'its_attachment'}).
jio_instance.removeAttachment('document_name',
'attachment_name').
then(function (response) {
// console.log(response);
});
......@@ -157,87 +134,37 @@ Here is a list of responses returned by jIO according to methods and options:
============================================== ================== ===============================================
Available for Option Response (Callback first parameter)
============================================== ================== ===============================================
``.post()``, ``.put()``, ``.remove()`` Any .. code-block:: javascript
{
result: 'success',
method: 'post',
// or put or remove
id: 'my_doc_id',
status: 204,
statusText: 'No Content'
}
``.putAttachment()``, ``.removeAttachment()`` Any .. code-block:: javascript
{
result: 'success',
method: 'putAttachment',
// or removeAttachment
id: 'my_doc_id',
attachment: 'my_attachment_id',
status: 204,
statusText: 'No Content'
}
``.get()`` Any .. code-block:: javascript
{
result: 'success',
method: 'get',
id: 'my_doc_id',
status: 200,
statusText: 'Ok',
data: {
// Here, the document metadata
}
}
``.post()``, ``.put()``, ``.remove()`` Any id of the document affected (string)
``.putAttachment()``, ``.removeAttachment()`` Any no specific value
``.get()`` Any document_metadata (object)
``.getAttachment()`` Any .. code-block:: javascript
{
result: 'success',
method: 'getAttachment',
id: 'my_doc_id',
attachment: 'my_attachment_id',
status: 200,
statusText: 'Ok',
data: Blob // Here, the attachment content
}
new Blob([data], {"type": content_type})
``.allDocs()`` No option .. code-block:: javascript
{
result: 'success',
method: 'allDocs',
id: 'my_doc_id',
status: 200,
statusText: 'Ok',
data: {
total_rows: 1,
rows: [{
id: 'mydoc',
key: 'mydoc', // optional
value: {},
}]
}
}
``.allDocs()`` include_docs: true .. code-block:: javascript
{
result: 'success',
method: 'allDocs',
id: 'my_doc_id',
status: 200,
statusText: 'Ok',
data: {
total_rows: 1,
rows: [{
id: 'mydoc',
key: 'mydoc', // optional
value: {},
doc: {
value: {
// Here, 'mydoc' metadata
}
}]
}
}
============================================== ================== ===============================================
......@@ -248,12 +175,7 @@ In case of error, the ``errorCallback`` first parameter looks like:
.. code-block:: javascript
{
result: 'error',
method: 'get',
status: 404,
statusText: 'Not Found',
error: 'not_found',
reason: 'document missing',
status_code: 404,
message: 'Unable to get the requested document'
}
......@@ -267,59 +189,42 @@ The following example creates a new jIO in localStorage and then posts a documen
.. code-block:: javascript
// create a new jIO
var jio_instance = jIO.createJIO({
type: 'local',
username: 'usr',
application_name: 'app'
});
var jio_instance = jIO.createJIO({type: 'indexeddb'});
// post the document 'metadata'
jio_instance.post({
// post the document 'myVideo'
jio_instance.put( 'metadata', {
title : 'My Video',
type : 'MovingImage',
format : 'video/ogg',
description : 'Images Compilation'
}, function (err, response) {
var id;
if (err) {
return alert('Error posting the document meta');
}
id = response.id;
})
.push(undefined, function(err) {
return alert('Error posting the document metadata');
});
// post a thumbnail attachment
jio_instance.putAttachment({
_id: id,
_attachment: 'thumbnail',
_data: my_image,
_mimetype: 'image/jpeg'
}, function (err, response) {
if (err) {
return alert('Error attaching thumbnail');
}
// post video attachment
jio_instance.putAttachment({
_id: id,
_attachment: 'video',
_data: my_video,
_mimetype: 'video/ogg'
}, function (err, response) {
if (err) {
return alert('Error attaching the video');
}
alert('Video Stored');
});
jio_instance.putAttachment('metadatda',
'thumbnail',
new Blob([my_image], {type: 'image/jpeg'})
).push(undefined, function(err) {
return alert('Error attaching thumbnail');
});
});
// post video attachment
jio_instance.putAttachment('metadatda',
'video',
new Blob([my_video], {type: 'video/ogg'})
).push(undefined, function(err) {
return alert('Error attaching video');
});
alert('Video Stored');
localStorage now contains:
indexedDB Storage now contains:
.. code-block:: javascript
{
"jio/local/usr/app/12345678-1234-1234-1234-123456789012": {
"_id": "12345678-1234-1234-1234-123456789012",
"/myVideo/": {
"title": "My Video",
"type": "MovingImage",
"format": "video/ogg",
......@@ -337,7 +242,7 @@ localStorage now contains:
}
}
},
"jio/local/usr/app/myVideo/thumbnail": "/9j/4AAQSkZ...",
"jio/local/usr/app/myVideo/video": "..."
"/myVideo/thumbnail": "...",
"/myVideo/video": "..."
}
docs/metadata.rst
View file @ 2d5c99d2
......@@ -58,11 +58,6 @@ List of metadata to use
Identification
^^^^^^^^^^^^^^
* **_id**
A specific jIO metadata which helps the storage to find a document
(can be a real path name, a dc:identifier, a uuid, ...). **String Only**
* **identifier**
| ``{"identifier": "http://domain/jio_home_page"}``
......
docs/revision_storages.rst → docs/replicate_storage.rst
View file @ 2d5c99d2
.. _revision-storages-conflicts-and-resolution:
.. _replicate-storage-conflicts-and-resolution:
Revision Storages: Conflicts and Resolution
Replicate Storage: Conflicts and Resolution
===========================================
......@@ -14,25 +14,29 @@ defines a conflict as multiple versions of a document existing in a storage
tree and a user trying to save on a version that does not match the latest
version of the document.
To keep track of document versions a revision storage must be used. When doing
To keep track of document versions a replicate storage must be used. When doing
so, jIO creates a document tree file for every document. This file contains all
existing versions and their status and is modified whenever a version is
added/updated/removed or when storages are being synchronized.
How to solve conflicts
----------------------
How conflicts are handled
-------------------------
The RemoteStorage takes in parameter two substorages, one "local" and one "remote".
The "local" storage can be remote, but it will be used for all the requests
like **get()**, **getAttachment()**, **allDocs()**...
Using the document tree, jIO tries to make every version of a document
available on every storage. When multiple versions of a document exist, jIO
will select the **latest**, **left-most** version on the document tree, along with the
conflicting versions (when option **conflicts: true** is set in order for
developers to setup a routine to solve conflicts.
available on the two storages. When multiple versions of a document exist,
Jio will follow the rule set by the conflict_handling option, given at storage creation.
This option can one of the following numbers:
* 0: no conflict resolution (throws an error when conflict is occuring)
* 1: keep the local state. (overwrites the remote document with local content)
* 2: keep the remote state. (overwrites the local document with remote content)
* 3: keep both copies (leave documents untouched, no signature update)
Technically, a conflict is solved by deleting alternative versions of a document
("cutting leaves off from the document tree"). When a user decides to keep a
version of a document and manually deletes all conflicting versions, the
storage tree is updated accordingly and the document is available in a single
version on all storages.
Simple Conflict Example
-----------------------
......@@ -45,123 +49,85 @@ your PC with your new email adress. Someone else changes this email from your PC
and once your smartphone is recharged, you go back online and the previous
update is executed.
#. Set up the storage tree:
#. Set up the replicate storage:
.. code-block:: javascript
var jio_instance = jIO.createJIO({
// replicate revision storage
type: 'replicaterevision',
storagelist:[{
type: 'revision',
sub_storage: {
type: 'dav',
// replicate storage
type: 'replicate',
local_sub_storage : {
type: 'local',
...
}
}, {
type: 'revision',
sub_storage: {
type: 'local',
remote_sub_storage: {
type: 'dav',
...
}
}]
conflict_handling: ...
});
#. Create the namecard on your smartphone:
#. 1) Create the namecard on your smartphone:
.. code-block:: javascript
jio_instance.post({
_id: 'myNameCard',
email: 'me@web.com'
jio_instance.put("myNameCard", {
email: 'jb@td.com'
}).then(function (response) {
// response.id -> 'myNameCard'
// response.rev -> '1-5782E71F1E4BF698FA3793D9D5A96393'
// response -> 'myNameCard'
});
This will create the document on your WebDAV and local storage
#. Someone else updates your shared namecard on WebDAV:
#. 2) Someone else updates your shared namecard on WebDAV:
.. code-block:: javascript
jio_instance.put({
email: 'my_new_me@web.com',
_id: 'myNameCard'
_rev: '1-5782E71F1E4BF698FA3793D9D5A96393'
jio_instance.put(myNameCard, {
email: 'kyle@td.com',
}).then(function (response) {
// response.id -> 'myNameCard'
// response.rev -> '2-068E73F5B44FEC987B51354DFC772891'
// response -> 'myNameCard'
});
Your smartphone is offline, so now you will have one version (1-578...) on
your smartphone and another version on WebDAV (2-068...) on your PC.
Your smartphone is offline, so now you will have one version on
your smartphone and another version on WebDAV on your PC.
#. You modify the namecard while being offline:
#. 3) Later, your smartphone is online and you modify your email:
.. code-block:: javascript
jio_instance.get({_id: 'myNameCard'}).then(function (response) {
// response.id -> 'myNameCard'
// response.rev -> '1-5782E71F1E4BF698FA3793D9D5A96393'
// response.data.email -> 'me@web.com'
jio_instance.get('myNameCard').then(function (response) {
// response.email -> 'jb@td.com'
// the get() method checks only on your local storage
// and doesn't warn you about remote modifications.
return jio_instance.put({
_id: 'myNameCard',
email: 'me_again@web.com'
});
}).then(function (response) {
// response.id -> 'myNameCard'
// response.rev -> '2-3753476B70A49EA4D8C9039E7B04254C'
return jio_instance.put('myNameCard', {
email: 'jack@td.com'
})
.then(function (response) {
// response -> 'myNameCard'
});
| Your latest modification of the email is: "jack@td.com"
| The modification from the other user is: "kyle@td.com"
#. Later, your smartphone is online and you retrieve the other version of the namecard:
.. code-block:: javascript
jio_instance.get({_id: 'myNameCard'}).then(function (response) {
// response.id -> 'myNameCard'
// response.rev -> '2-3753476B70A49EA4D8C9039E7B04254C'
// response.data.email -> 'me_again@web.com'
});
If your conflict_handling option was:
When multiple versions of a document are available, jIO returns the latest,
left-most version on the document tree (2-375... and labels all other
versions as conflicting 2-068...).
* | 0: the email is:
| -"kyle@td.com" on WebDAV
| -"jack@td.com" on your local storage
| The storage rejects your latest modification,
| you get an error because local and remote documents are desynchronized.
| The documents in local and remote state are left untouched.
#. Retrieve conflicts by setting option:
.. code-block:: javascript
jio_instance.get({_id: 'myNameCard'}, {
conflicts: true
}).then(function (response) {
// response.id -> 'myNameCard'
// response.rev -> '2-3753476B70A49EA4D8C9039E7B04254C',
// response.conflicts -> ['2-068E73F5B44FEC987B51354DFC772891']
});
The conflicting version (*2-068E...*) is displayed, because **{conflicts: true}** was
specified in the GET call. Deleting either version will solve the conflict.
#. Delete the conflicting version:
.. code-block:: javascript
jio_instance.remove({
_id: 'myNameCard',
_rev: '2-068E73F5B44FEC987B51354DFC772891'
}).then(function (response) {
// response.id -> 'myNameCard'
// response.rev -> '3-28910A4937537B5168E772896B70EC98'
});
* | 1: the email is: "jack@td.com" on both storages
| The storage pushes the local modification, which is yours.
When deleting the conflicting version of your namecard, jIO removed it
from all storages and set the document tree leaf of that version to
*deleted*. All storages now contain just a single version of the namecard
(2-3753...). Note that, on the document tree, removing a revison will
create a new revision with status set to *deleted*.
* | 2: the email is: "kyle@td.com" on both storages
| The storage keeps the remote modification, which is from the other user.
| Your local storage is modified to fit the state of the remote storage.
* | 3: the email is: "jack@td.com" on both storages
| The storage doesn't do synchronization, and pushes your modification
| without checking if the remote storage has been changed or not
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7