See merge request !436

Also change places where xml_marshaller were used directly to always use
the wrapper from utils (except in tests for simplicity)

On python3, deleting a chmoded folder causes this error:

    Traceback (most recent call last):
      File "/usr/lib/python3.7/shutil.py", line 423, in _rmtree_safe_fd
        dirfd = os.open(entry.name, os.O_RDONLY, dir_fd=topfd)
    PermissionError: [Errno 13] Permission denied: 'directory'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "slapos.core/slapos/grid/slapgrid.py", line 607, in processSoftwareReleaseList
        software.destroy()
      File "slapos.core/slapos/grid/SlapObject.py", line 360, in destroy
        rmtree(self.software_path)
      File "slapos.core/slapos/util.py", line 267, in rmtree
        shutil.rmtree(path, onerror=chmod_retry)
      File "/usr/lib/python3.7/shutil.py", line 491, in rmtree
        _rmtree_safe_fd(fd, path, onerror)
      File "/usr/lib/python3.7/shutil.py", line 425, in _rmtree_safe_fd
        onerror(os.open, fullname, sys.exc_info())
      File "slapos.core/slapos/util.py", line 264, in chmod_retry
        func(failed_path)
    TypeError: open() missing required argument 'flags' (pos 2)

Our error handler was assuming that on python 3 the EACCESS for the
directory would happen first on an os.lstat call [1], which is true when the
top level folder is not listable, but it did not support the case where
the top level folder is listable but contain another folder that is not listable.
In that case, the first error is an os.open [2]

Tests from erp5.util (where this function came from) only covered the 1 case but
new tests in slapos.core covered the 2.

Anyway, do the "chmod folder and retry from beginning" for both 1 or 2 cases.

[1]: https://github.com/python/cpython/blob/3.6/Lib/shutil.py#L475

[2]: https://github.com/python/cpython/blob/3.6/Lib/shutil.py#L418

software releases expects _ parameter to be a string with JSON inside
not a python dict so we partially revert 2acef14b
and we do the nice printing of JSON parameters in getInformation

 * xml2dict and dict2xml support the JSON format of parameter dict
 * since xml2dict doesn't support unicode, we convert unicode to str in
getInformation

follow up of f2798c25

it was still crashing when "create_tap = True"

connection_dict generated by client can be different in details from the
server side, so pass it thorugh a way how it is treat on the server.

Also, as we are going to be compatible with py3, calculate hash from
sorted items of a dict, instead of relying on side effect of py2 "ordered dict"

https://portingguide.readthedocs.io/en/latest/dicts.html#changed-key-order

Test are focused on client side, and the tricky cases are covered, they
somehow contain the protocol of client <--> server comparision.

All tools from the ecosystem shall use the same way to convert between python's
dict and XML.

Additionally stabilise generated XMLs by sorting the dictionary.

/reviewed-on !71

Each tap will have a subnetwork with 16 more bits than the interface.
The IPv6 of the tap is the last of the subnetwork.

Note: the example IPv6 address is the address used in documentation (see https://en.wikipedia.org/wiki/IPv6_address#Documentation)

Because e.g. for the following instance parameters

    <?xml version='1.0' encoding='utf-8'?>
    <instance>
      <parameter id="text">Привет Мир!</parameter>
    </instance>

we get the following DB query

    'UPDATE %s SET slap_state="busy" ,software_release=? ,partition_reference=? ,requested_by=? ,software_type=? ,xml=? WHERE
reference=? AND computer_reference=?',
    ['http://sr//', 'myinstance', 'slappart0', 'default',
     '<?xml version=\'1.0\' encoding=\'utf-8\'?>\n<instance>\n  <parameter id="text">\xd0\x9f\xd1\x80\xd0\xb8\xd0\xb2\xd0\xb5\xd1\x82\xd0\x9c\xd0\xb8\xd1\x80!</parameter>\n</instance>\n',
     'slappart6', 'computer']

which fails the following way:

  File ".../site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File ".../site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File ".../site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File ".../site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File ".../site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "slapos/proxy/views.py", line 393, in requestComputerPartition
    software_instance = requestNotSlave(**parsed_request_dict)
  File "slapos/proxy/views.py", line 609, in requestNotSlave
    execute_db('partition', q, args)
  File "slapos/proxy/views.py", line 142, in execute_db
    cur = db.execute(query, args)
ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

our instance parameters is always XML UTF-8 encoded, and thus it is not
unicode and we cannot go the 'use unicode always' way.

So autotune sqlite connections on open to support 8-bit strings.

Currently it is just sqlite3.connect(), but we'll want to add connection
autotuning in the next patch, so before that let's consolidate all
sqlite opening places in the code into one.

Used to decrease writes done by slapos.format.
Also disable specific chgrp for webrunner.