I had a failure in testnode, then with the help of @jerome , it look like slapgrid rmtree is not correctly removing SR files.

So I reused the method rmtree imported from testnode worked. The method was moved to util.py and shared with grid/SlapObject.py and cli/prune.py.

Here was the failure:

2020-06-03 10:09:19 slapos[23730] INFO Destroying '/srv/slapgrid/slappart7/srv/testnode/day/soft/0eae2fcbeef6dc3cb65c827b1893e2ca/parts/slapos-repository/software/repman/software.cfg'...
2020-06-03 10:09:19 slapos[23730] INFO Removing path '/srv/slapgrid/slappart7/srv/testnode/day/inst/test0-0/tmp/soft/4ebaf0217f216898313f31a66df88e44'
2020-06-03 10:09:19 slapos[23730] ERROR 
Traceback (most recent call last):
  File "/srv/slapgrid/slappart7/srv/testnode/day/soft/0eae2fcbeef6dc3cb65c827b1893e2ca/parts/slapos.core-repository/slapos/grid/slapgrid.py", line 607, in processSoftwareReleaseList
    software.destroy()
  File "/srv/slapgrid/slappart7/srv/testnode/day/soft/0eae2fcbeef6dc3cb65c827b1893e2ca/parts/slapos.core-repository/slapos/grid/SlapObject.py", line 367, in destroy
    shutil.rmtree(self.software_path, onerror=retry)
  File "/srv/slapgrid/slappart7/srv/testnode/day/shared/python2.7/1007d4372da3e125dee8cba215f30d51/lib/python2.7/shutil.py", line 270, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/srv/slapgrid/slappart7/srv/testnode/day/shared/python2.7/1007d4372da3e125dee8cba215f30d51/lib/python2.7/shutil.py", line 270, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/srv/slapgrid/slappart7/srv/testnode/day/shared/python2.7/1007d4372da3e125dee8cba215f30d51/lib/python2.7/shutil.py", line 270, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/srv/slapgrid/slappart7/srv/testnode/day/shared/python2.7/1007d4372da3e125dee8cba215f30d51/lib/python2.7/shutil.py", line 270, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/srv/slapgrid/slappart7/srv/testnode/day/shared/python2.7/1007d4372da3e125dee8cba215f30d51/lib/python2.7/shutil.py", line 270, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/srv/slapgrid/slappart7/srv/testnode/day/shared/python2.7/1007d4372da3e125dee8cba215f30d51/lib/python2.7/shutil.py", line 270, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/srv/slapgrid/slappart7/srv/testnode/day/shared/python2.7/1007d4372da3e125dee8cba215f30d51/lib/python2.7/shutil.py", line 275, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "/srv/slapgrid/slappart7/srv/testnode/day/soft/0eae2fcbeef6dc3cb65c827b1893e2ca/parts/slapos.core-repository/slapos/grid/SlapObject.py", line 363, in retry
    func(path)
OSError: [Errno 13] Permission denied: '/srv/slapgrid/slappart7/srv/testnode/day/inst/test0-0/tmp/soft/4ebaf0217f216898313f31a66df88e44/go.work/pkg/mod/github.com/soheilhy/cmux@v0.1.4/patricia_test.go'
2020-06-03 10:09:19 slapos[23730] INFO Finished software releases.
ERROR