Skip to content

erp5
erp5
Commit 751c2516 authored by Rafael Monnerat's avatar Rafael Monnerat
Browse files
Options

erp5_disaster_recovery: Initial public release 

   This business template introduces a way to adjust mariadb and data.fs after restore from backup. In the occasion that the backups are unsync (which is normal) it helps to push both on sync instead of fully reindex the site.
parent 38483df3
Pipeline #33492 failed with stage
    in 0 seconds
    Hide whitespace changes
    Inline Side-by-side
    Showing with 697 additions and 0 deletions
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery.xml 0 → 100644
    View file @ 751c2516
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="Folder" module="OFS.Folder"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>_objects</string> </key>
    <value>
    <tuple/>
    </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>erp5_disaster_recovery</string> </value>
    </item>
    <item>
    <key> <string>title</string> </key>
    <value> <string></string> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_checkDeletedDocumentList.py 0 → 100644
    View file @ 751c2516
    portal = context.getPortalObject()
    document_list = portal.portal_catalog(
    limit=limit,
    uid={'query': min_uid, 'range': 'nlt'},
    sort_on=(('uid', 'ASC'),),
    )
    result_count = len(document_list)
    if result_count:
    if result_count == limit:
    portal.portal_activities.activate(activity='SQLQueue', priority=3).ERP5Site_checkDeletedDocumentList(document_list[-1].uid, limit, packet_size)
    column_list = [(x.path, x.uid) for x in document_list]
    for i in xrange(0, result_count, packet_size):
    portal.portal_activities.activate(activity='SQLQueue').ERP5Site_unindexDeletedDocumentList(column_list[i:i+packet_size])
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_checkDeletedDocumentList.xml 0 → 100644
    View file @ 751c2516
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>_bind_names</string> </key>
    <value>
    <object>
    <klass>
    <global name="_reconstructor" module="copy_reg"/>
    </klass>
    <tuple>
    <global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
    <global name="object" module="__builtin__"/>
    <none/>
    </tuple>
    <state>
    <dictionary>
    <item>
    <key> <string>_asgns</string> </key>
    <value>
    <dictionary>
    <item>
    <key> <string>name_container</string> </key>
    <value> <string>container</string> </value>
    </item>
    <item>
    <key> <string>name_context</string> </key>
    <value> <string>context</string> </value>
    </item>
    <item>
    <key> <string>name_m_self</string> </key>
    <value> <string>script</string> </value>
    </item>
    <item>
    <key> <string>name_subpath</string> </key>
    <value> <string>traverse_subpath</string> </value>
    </item>
    </dictionary>
    </value>
    </item>
    </dictionary>
    </state>
    </object>
    </value>
    </item>
    <item>
    <key> <string>_params</string> </key>
    <value> <string>min_uid, limit, packet_size</string> </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>ERP5Site_checkDeletedDocumentList</string> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_checkLatestModifiedDocumentList.py 0 → 100644
    View file @ 751c2516
    from Products.ZSQLCatalog.SQLCatalog import Query
    portal = context.getPortalObject()
    document_list = portal.portal_catalog(
    limit=limit,
    uid={'query': min_uid, 'range': 'nlt'},
    indexation_timestamp=Query(**{'indexation_timestamp': (before, now), 'range': 'minngt'}),
    sort_on=(('uid', 'ASC'),),
    )
    result_count = len(document_list)
    if result_count:
    if result_count == limit:
    portal.portal_activities.activate(activity='SQLQueue', priority=3).ERP5Site_checkLatestModifiedDocumentList(document_list[-1].uid, limit, packet_size, before, now)
    column_list = [(x.path, x.uid) for x in document_list]
    for i in xrange(0, result_count, packet_size):
    portal.portal_activities.activate(activity='SQLQueue').ERP5Site_reindexOrUnindexDocumentList(column_list[i:i+packet_size])
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_checkLatestModifiedDocumentList.xml 0 → 100644
    View file @ 751c2516
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>_bind_names</string> </key>
    <value>
    <object>
    <klass>
    <global name="_reconstructor" module="copy_reg"/>
    </klass>
    <tuple>
    <global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
    <global name="object" module="__builtin__"/>
    <none/>
    </tuple>
    <state>
    <dictionary>
    <item>
    <key> <string>_asgns</string> </key>
    <value>
    <dictionary>
    <item>
    <key> <string>name_container</string> </key>
    <value> <string>container</string> </value>
    </item>
    <item>
    <key> <string>name_context</string> </key>
    <value> <string>context</string> </value>
    </item>
    <item>
    <key> <string>name_m_self</string> </key>
    <value> <string>script</string> </value>
    </item>
    <item>
    <key> <string>name_subpath</string> </key>
    <value> <string>traverse_subpath</string> </value>
    </item>
    </dictionary>
    </value>
    </item>
    </dictionary>
    </state>
    </object>
    </value>
    </item>
    <item>
    <key> <string>_params</string> </key>
    <value> <string>min_uid, limit, packet_size, before, now</string> </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>ERP5Site_checkLatestModifiedDocumentList</string> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_recoverFromRestoration.py 0 → 100644
    View file @ 751c2516
    portal = context.getPortalObject()
    now = DateTime()
    before = now - int(days_before)
    strfstring = '%Y-%m-%d %H:%M:%S'
    portal.ERP5Site_checkLatestModifiedDocumentList(0, 1000, 100,
    before.strftime(strfstring), now.strftime(strfstring))
    # Force reindexation of recently created document
    # This expect module to use HBTree
    # The following category could be consider as "non-optimistic", but
    # it is minimal compared to a whole reindex and it is safe to prevent
    # Minor inconsistencies.
    for module_id in portal.objectIds(("ERP5 Folder",)):
    if module_id.endswith("_module"):
    portal[module_id].recurseCallMethod(
    'recursiveReindexObject',
    max_depth=1,
    min_depth=1,
    max_retry=0,
    activity_count=100,
    min_id=before.strftime("%Y%m%d"),
    )
    portal.ERP5Site_checkDeletedDocumentList(0, 1000, 100)
    # Whenever we trust that the catalog is consistent and more recent them the
    # ZODB, it isn't required to trigger the whole reindexation of the site.
    # If the Mariadb is eventually inconsitent or older them the current ZODB,
    # It's required reindex the whole site.
    if not optimistic:
    for module_id in [
    'portal_preferences',
    'portal_categories',
    'portal_alarms',
    'portal_simulation']+portal.objectIds(("ERP5 Folder",)):
    portal[module_id].recurseCallMethod(
    'immediateReindexObject',
    min_depth=1,
    max_depth=10000,
    activate_kw=dict(
    group_method_id='portal_catalog/catalogObjectList',
    alternate_method_id='alternateReindexObject',
    group_method_cost=1,
    priority=6,
    ),
    max_retry=0,
    activity_count=100,
    )
    return "OK"
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_recoverFromRestoration.xml 0 → 100644
    View file @ 751c2516
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>_bind_names</string> </key>
    <value>
    <object>
    <klass>
    <global name="_reconstructor" module="copy_reg"/>
    </klass>
    <tuple>
    <global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
    <global name="object" module="__builtin__"/>
    <none/>
    </tuple>
    <state>
    <dictionary>
    <item>
    <key> <string>_asgns</string> </key>
    <value>
    <dictionary>
    <item>
    <key> <string>name_container</string> </key>
    <value> <string>container</string> </value>
    </item>
    <item>
    <key> <string>name_context</string> </key>
    <value> <string>context</string> </value>
    </item>
    <item>
    <key> <string>name_m_self</string> </key>
    <value> <string>script</string> </value>
    </item>
    <item>
    <key> <string>name_subpath</string> </key>
    <value> <string>traverse_subpath</string> </value>
    </item>
    </dictionary>
    </value>
    </item>
    </dictionary>
    </state>
    </object>
    </value>
    </item>
    <item>
    <key> <string>_params</string> </key>
    <value> <string>optimistic=True, days_before=1</string> </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>ERP5Site_recoverFromRestoration</string> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_reindexOrUnindexDocumentList.py 0 → 100644
    View file @ 751c2516
    from erp5.component.module.Log import log
    for path, uid in column_list:
    try:
    ob = context.restrictedTraverse(path)
    except KeyError:
    log("object not found", path)
    context.portal_catalog.activate(activity='SQLQueue').uncatalog_object(uid=uid)
    else:
    ob.reindexObject()
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_reindexOrUnindexDocumentList.xml 0 → 100644
    View file @ 751c2516
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>_bind_names</string> </key>
    <value>
    <object>
    <klass>
    <global name="_reconstructor" module="copy_reg"/>
    </klass>
    <tuple>
    <global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
    <global name="object" module="__builtin__"/>
    <none/>
    </tuple>
    <state>
    <dictionary>
    <item>
    <key> <string>_asgns</string> </key>
    <value>
    <dictionary>
    <item>
    <key> <string>name_container</string> </key>
    <value> <string>container</string> </value>
    </item>
    <item>
    <key> <string>name_context</string> </key>
    <value> <string>context</string> </value>
    </item>
    <item>
    <key> <string>name_m_self</string> </key>
    <value> <string>script</string> </value>
    </item>
    <item>
    <key> <string>name_subpath</string> </key>
    <value> <string>traverse_subpath</string> </value>
    </item>
    </dictionary>
    </value>
    </item>
    </dictionary>
    </state>
    </object>
    </value>
    </item>
    <item>
    <key> <string>_params</string> </key>
    <value> <string>column_list</string> </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>ERP5Site_reindexOrUnindexDocumentList</string> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_unindexDeletedDocumentList.py 0 → 100644
    View file @ 751c2516
    from erp5.component.module.Log import log
    for path, uid in column_list:
    try:
    _ = context.restrictedTraverse(path)
    except KeyError:
    log("object not found", path)
    context.portal_catalog.activate(activity='SQLQueue').uncatalog_object(uid=uid)
    bt5/erp5_disaster_recovery/SkinTemplateItem/portal_skins/erp5_disaster_recovery/ERP5Site_unindexDeletedDocumentList.xml 0 → 100644
    View file @ 751c2516
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="PythonScript" module="Products.PythonScripts.PythonScript"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>_bind_names</string> </key>
    <value>
    <object>
    <klass>
    <global name="_reconstructor" module="copy_reg"/>
    </klass>
    <tuple>
    <global name="NameAssignments" module="Shared.DC.Scripts.Bindings"/>
    <global name="object" module="__builtin__"/>
    <none/>
    </tuple>
    <state>
    <dictionary>
    <item>
    <key> <string>_asgns</string> </key>
    <value>
    <dictionary>
    <item>
    <key> <string>name_container</string> </key>
    <value> <string>container</string> </value>
    </item>
    <item>
    <key> <string>name_context</string> </key>
    <value> <string>context</string> </value>
    </item>
    <item>
    <key> <string>name_m_self</string> </key>
    <value> <string>script</string> </value>
    </item>
    <item>
    <key> <string>name_subpath</string> </key>
    <value> <string>traverse_subpath</string> </value>
    </item>
    </dictionary>
    </value>
    </item>
    </dictionary>
    </state>
    </object>
    </value>
    </item>
    <item>
    <key> <string>_params</string> </key>
    <value> <string>column_list</string> </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>ERP5Site_unindexDeletedDocumentList</string> </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_disaster_recovery/TestTemplateItem/portal_components/test.erp5.testDisasterRecovery.py 0 → 100644
    View file @ 751c2516
    ##############################################################################
    #
    # Copyright (c) 2002-2024 Nexedi SA and Contributors. All Rights Reserved.
    #
    # WARNING: This program as such is intended to be used by professional
    # programmers who take the whole responsibility of assessing all potential
    # consequences resulting from its eventual inadequacies and bugs
    # End users who are looking for a ready-to-use solution with commercial
    # guarantees and support are strongly adviced to contract a Free Software
    # Service Company
    #
    # This program is Free Software; you can redistribute it and/or
    # modify it under the terms of the GNU General Public License
    # as published by the Free Software Foundation; either version 2
    # of the License, or (at your option) any later version.
    #
    # This program is distributed in the hope that it will be useful,
    # but WITHOUT ANY WARRANTY; without even the implied warranty of
    # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
    # GNU General Public License for more details.
    #
    # You should have received a copy of the GNU General Public License
    # along with this program; if not, write to the Free Software
    # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
    #
    ##############################################################################
    from Products.ERP5Type.tests.ERP5TypeTestCase import ERP5TypeTestCase
    class TestDisasterRecovery(ERP5TypeTestCase):
    """
    Test Disaster Recover
    """
    def test_missing_catalog(self):
    document = self.portal.person_module.newContent(
    portal_type='Person',
    title='Person for Disaster Recovery'
    )
    self.tic()
    _document = self.portal.portal_catalog.getResultValue(
    uid=document.getUid()
    )
    self.assertNotEqual(_document, None)
    self.assertEqual(document.getUid(), _document.getUid())
    self.portal.portal_catalog.uncatalog_object(uid=document.getUid())
    self.tic()
    _document = self.portal.portal_catalog.getResultValue(
    uid=document.getUid()
    )
    self.assertEqual(_document, None)
    self.portal.ERP5Site_recoverFromRestoration()
    self.tic()
    _document = self.portal.portal_catalog.getResultValue(
    uid=document.getUid()
    )
    self.assertNotEqual(_document, None)
    self.assertEqual(document.getUid(), _document.getUid())
    def test_catalog_but_deleted(self):
    if self.portal.person_module.getIdGenerator() != '_generatePerDayId':
    self.portal.person_module.setIdGenerator('_generatePerDayId')
    document = self.portal.person_module.newContent(
    portal_type='Person',
    title='Person for Disaster Recovery'
    )
    self.tic()
    _document = self.portal.portal_catalog.getResultValue(
    uid=document.getUid()
    )
    self.assertNotEqual(_document, None)
    self.assertEqual(document.getUid(), _document.getUid())
    # Force remove the object w/o trigger updates on catalog
    self.portal.person_module._objects = tuple([
    i for i in self.portal.person_module._objects
    if i['id'] != document.getId()])
    self.portal.person_module._delOb(document.getId())
    self.tic()
    connection = self.getSQLConnection()
    doc_list = connection.manage_test(
    "select * from catalog where path = '/%s/%s'" % (
    self.portal.getId(), document.getRelativeUrl()))
    self.assertEqual(len(doc_list), 1)
    doc_list = connection.manage_test("select * from catalog where uid = %s" % document.getUid())
    self.assertEqual(len(doc_list), 1)
    self.portal.ERP5Site_recoverFromRestoration()
    self.tic()
    ac = connection.manage_test("select * from catalog where uid = %s" % document.getUid())
    self.assertEqual(len(ac), 0)
    _document = self.portal.portal_catalog(uid=document.getUid())
    self.assertEqual(len(_document), 0)
    def test_cataloged_is_inconsistent(self):
    document = self.portal.person_module.newContent(
    portal_type='Person',
    title='Person for Disaster Recovery %s' % (
    str(self.portal.portal_ids.generateNewId(
    id_group=('erp5_disaster_recovery_test_id')))))
    self.tic()
    _document = self.portal.portal_catalog.getResultValue(
    uid=document.getUid()
    )
    self.assertNotEqual(_document, None)
    self.assertEqual(document.getUid(), _document.getUid())
    connection = self.getSQLConnection()
    connection.manage_test("update catalog set title = 'modified title' where uid = %s" % document.getUid())
    connection.manage_test("commit")
    self.commit()
    _document = self.portal.portal_catalog(title='modified title')
    self.assertEqual(len(_document), 1)
    _document = self.portal.portal_catalog(title=document.getTitle())
    self.assertEqual(len(_document), 0)
    self.portal.ERP5Site_recoverFromRestoration()
    self.tic()
    _document = self.portal.portal_catalog(title='modified title')
    self.assertEqual(len(_document), 0)
    _document = self.portal.portal_catalog(title=document.getTitle())
    self.assertEqual(len(_document), 1)
    _document = self.portal.portal_catalog(title=document.getTitle())
    self.assertEqual(_document[0].getUid(), document.getUid())
    bt5/erp5_disaster_recovery/TestTemplateItem/portal_components/test.erp5.testDisasterRecovery.xml 0 → 100644
    View file @ 751c2516
    <?xml version="1.0"?>
    <ZopeData>
    <record id="1" aka="AAAAAAAAAAE=">
    <pickle>
    <global name="Test Component" module="erp5.portal_type"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>default_reference</string> </key>
    <value> <string>testDisasterRecovery</string> </value>
    </item>
    <item>
    <key> <string>description</string> </key>
    <value>
    <none/>
    </value>
    </item>
    <item>
    <key> <string>id</string> </key>
    <value> <string>test.erp5.testDisasterRecovery</string> </value>
    </item>
    <item>
    <key> <string>sid</string> </key>
    <value>
    <none/>
    </value>
    </item>
    <item>
    <key> <string>text_content_error_message</string> </key>
    <value>
    <tuple/>
    </value>
    </item>
    <item>
    <key> <string>text_content_warning_message</string> </key>
    <value>
    <tuple/>
    </value>
    </item>
    <item>
    <key> <string>version</string> </key>
    <value> <string>erp5</string> </value>
    </item>
    <item>
    <key> <string>workflow_history</string> </key>
    <value>
    <persistent> <string encoding="base64">AAAAAAAAAAI=</string> </persistent>
    </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    <record id="2" aka="AAAAAAAAAAI=">
    <pickle>
    <global name="PersistentMapping" module="Persistence.mapping"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>data</string> </key>
    <value>
    <dictionary>
    <item>
    <key> <string>component_validation_workflow</string> </key>
    <value>
    <persistent> <string encoding="base64">AAAAAAAAAAM=</string> </persistent>
    </value>
    </item>
    </dictionary>
    </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    <record id="3" aka="AAAAAAAAAAM=">
    <pickle>
    <global name="WorkflowHistoryList" module="Products.ERP5Type.Workflow"/>
    </pickle>
    <pickle>
    <dictionary>
    <item>
    <key> <string>_log</string> </key>
    <value>
    <list>
    <dictionary>
    <item>
    <key> <string>action</string> </key>
    <value> <string>validate</string> </value>
    </item>
    <item>
    <key> <string>validation_state</string> </key>
    <value> <string>validated</string> </value>
    </item>
    </dictionary>
    </list>
    </value>
    </item>
    </dictionary>
    </pickle>
    </record>
    </ZopeData>
    bt5/erp5_disaster_recovery/bt/dependency_list 0 → 100644
    View file @ 751c2516
    erp5_core
    \ No newline at end of file
    bt5/erp5_disaster_recovery/bt/description 0 → 100644
    View file @ 751c2516
    Business template to help to sync mariadb and Datafs consistency, after a restoration.
    \ No newline at end of file
    bt5/erp5_disaster_recovery/bt/template_skin_id_list 0 → 100644
    View file @ 751c2516
    erp5_disaster_recovery
    \ No newline at end of file
    bt5/erp5_disaster_recovery/bt/template_test_id_list 0 → 100644
    View file @ 751c2516
    test.erp5.testDisasterRecovery
    \ No newline at end of file
    bt5/erp5_disaster_recovery/bt/test_dependency_list 0 → 100644
    View file @ 751c2516
    erp5_base
    erp5_full_text_mroonga_catalog
    \ No newline at end of file
    bt5/erp5_disaster_recovery/bt/title 0 → 100644
    View file @ 751c2516
    erp5_disaster_recovery
    \ No newline at end of file
    bt5/erp5_disaster_recovery/bt/version 0 → 100644
    View file @ 751c2516
    0.1
    \ No newline at end of file
    Markdown is supported
    0% or
    You are about to add 0 people to the discussion. Proceed with caution.
    Finish editing this message first!
    Please register or to comment
    GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7