cPickleCache.c 26.5 KB
Newer Older
1
/*****************************************************************************
matt@zope.com's avatar
matt@zope.com committed
2

3 4
  Copyright (c) 2001, 2002 Zope Corporation and Contributors.
  All Rights Reserved.
5

matt@zope.com's avatar
matt@zope.com committed
6 7 8 9 10 11
  This software is subject to the provisions of the Zope Public License,
  Version 2.0 (ZPL).  A copy of the ZPL should accompany this distribution.
  THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
  WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
  WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
  FOR A PARTICULAR PURPOSE
12

13
 ****************************************************************************/
14

Toby Dickenson's avatar
Toby Dickenson committed
15 16 17 18 19 20 21 22 23
/*

Objects are stored under three different regimes:

Regime 1: Persistent Classes

Persistent Classes are part of ZClasses. They are stored in the
self->data dictionary, and are never garbage collected.

24 25 26
The klass_items() method returns a sequence of (oid,object) tuples for
every Persistent Class, which should make it possible to implement
garbage collection in Python if necessary.
Toby Dickenson's avatar
Toby Dickenson committed
27 28 29

Regime 2: Ghost Objects

30 31 32 33
There is no benefit to keeping a ghost object which has no external
references, therefore a weak reference scheme is used to ensure that
ghost objects are removed from memory as soon as possible, when the
last external reference is lost.
Toby Dickenson's avatar
Toby Dickenson committed
34

35 36 37
Ghost objects are stored in the self->data dictionary. Normally a
dictionary keeps a strong reference on its values, however this
reference count is 'stolen'.
Toby Dickenson's avatar
Toby Dickenson committed
38 39

This weak reference scheme leaves a dangling reference, in the
40 41 42 43 44 45 46
dictionary, when the last external reference is lost. To clean up this
dangling reference the persistent object dealloc function calls
self->cache->_oid_unreferenced(self->oid). The cache looks up the oid
in the dictionary, ensures it points to an object whose reference
count is zero, then removes it from the dictionary. Before removing
the object from the dictionary it must temporarily resurrect the
object in much the same way that class instances are resurrected
Toby Dickenson's avatar
Toby Dickenson committed
47 48
before their __del__ is called.

49 50 51 52
Since ghost objects are stored under a different regime to non-ghost
objects, an extra ghostify function in cPersistenceAPI replaces
self->state=GHOST_STATE assignments that were common in other
persistent classes (such as BTrees).
Toby Dickenson's avatar
Toby Dickenson committed
53 54 55

Regime 3: Non-Ghost Objects

56 57 58 59 60 61
Non-ghost objects are stored in two data structures: the dictionary
mapping oids to objects and a doubly-linked list that encodes the
order in which the objects were accessed.  The dictionary reference is
borrowed, as it is for ghosts.  The list reference is a new reference;
the list stores recently used objects, even if they are otherwise
unreferenced, to avoid loading the object from the database again.
Toby Dickenson's avatar
Toby Dickenson committed
62

63 64
The doubly-link-list nodes contain next and previous pointers linking
together the cache and all non-ghost persistent objects.
Toby Dickenson's avatar
Toby Dickenson committed
65 66

The node embedded in the cache is the home position. On every
67 68 69 70 71 72 73 74 75 76
attribute access a non-ghost object will relink itself just behind the
home position in the ring. Objects accessed least recently will
eventually find themselves positioned after the home position.

Occasionally other nodes are temporarily inserted in the ring as
position markers. The cache contains a ring_lock flag which must be
set and unset before and after doing so. Only if the flag is unset can
the cache assume that all nodes are either his own home node, or nodes
from persistent objects. This assumption is useful during the garbage
collection process.
Toby Dickenson's avatar
Toby Dickenson committed
77 78 79 80 81 82 83

The number of non-ghost objects is counted in self->non_ghost_count.
The garbage collection process consists of traversing the ring, and
deactivating (that is, turning into a ghost) every object until
self->non_ghost_count is down to the target size, or until it
reaches the home position again.

84 85 86
Note that objects in the sticky or changed states are still kept in
the ring, however they can not be deactivated. The garbage collection
process must skip such objects, rather than deactivating them.
Toby Dickenson's avatar
Toby Dickenson committed
87 88 89

*/

90
static char cPickleCache_doc_string[] =
Jeremy Hylton's avatar
Jeremy Hylton committed
91 92
"Defines the PickleCache used by ZODB Connection objects.\n"
"\n"
93
"$Id: cPickleCache.c,v 1.80 2003/04/02 16:50:49 jeremy Exp $\n";
Jim Fulton's avatar
Jim Fulton committed
94

95 96 97 98
#define ASSIGN(V,E) {PyObject *__e; __e=(E); Py_XDECREF(V); (V)=__e;}
#define UNLESS(E) if(!(E))
#define UNLESS_ASSIGN(V,E) ASSIGN(V,E) UNLESS(V)
#define OBJECT(O) ((PyObject*)O)
99

100
#define DONT_USE_CPERSISTENCECAPI
101
#include "cPersistence.h"
Jim Fulton's avatar
Jim Fulton committed
102
#include <time.h>
103
#include <stddef.h>
104 105
#undef Py_FindMethod

106
static PyObject *py__p_oid, *py_reload, *py__p_jar, *py__p_changed;
Jim Fulton's avatar
Jim Fulton committed
107

108 109 110 111 112 113 114
/* Do we want 'engine noise'.... abstract debugging output useful for
   visualizing cache behavior */
#if 0
#define ENGINE_NOISE(A) printf(A)
#else
#define ENGINE_NOISE(A) ((void)A)
#endif
115

116 117 118
/* This object is the pickle cache.  The CACHE_HEAD macro guarantees
   that layout of this struct is the same as the start of
   ccobject_head in cPersistence.c */
119
typedef struct {
120
    CACHE_HEAD
Toby Dickenson's avatar
Toby Dickenson committed
121
    int klass_count;                         /* count of persistent classes */
122 123 124
    PyObject *data;                          /* oid -> object dict */
    PyObject *jar;                           /* Connection object */
    PyObject *setklassstate;                 /* ??? */
Toby Dickenson's avatar
Toby Dickenson committed
125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143
    int cache_size;                          /* target number of items in cache */

    /* Most of the time the ring contains only:
       * many nodes corresponding to persistent objects
       * one 'home' node from the cache.
    In some cases it is handy to temporarily add other types
    of node into the ring as placeholders. 'ring_lock' is a boolean
    indicating that someone has already done this. Currently this
    is only used by the garbage collection code. */

    int ring_lock;

    /* 'cache_drain_resistance' controls how quickly the cache size will drop
    when it is smaller than the configured size. A value of zero means it will
    not drop below the configured size (suitable for most caches). Otherwise,
    it will remove cache_non_ghost_count/cache_drain_resistance items from
    the cache every time (suitable for rarely used caches, such as those
    associated with Zope versions. */

144
    int cache_drain_resistance;
Toby Dickenson's avatar
Toby Dickenson committed
145

146
} ccobject;
147

Jeremy Hylton's avatar
Jeremy Hylton committed
148
static int cc_ass_sub(ccobject *self, PyObject *key, PyObject *v);
149

Jim Fulton's avatar
Jim Fulton committed
150 151
/* ---------------------------------------------------------------- */

152 153 154
#define OBJECT_FROM_RING(SELF, HERE, CTX) \
    ((cPersistentObject *)(((char *)here) - offsetof(cPersistentObject, ring)))

155
static int
156
scan_gc_items(ccobject *self,int target)
157
{
158 159 160
    /* This function must only be called with the ring lock held,
       because it places a non-object placeholder in the ring.
    */
Toby Dickenson's avatar
Toby Dickenson committed
161

162 163
    cPersistentObject *object;
    int error;
164
    CPersistentRing placeholder;
165 166
    CPersistentRing *here = self->ring_home.next;

167 168 169 170
    /* Scan through the ring until we either find the ring_home (i.e. start
     * of the ring, or we've ghosted enough objects to reach the target
     * size.
     */
171 172 173
    while (1) {
	/* back to the home position. stop looking */
        if (here == &self->ring_home)
174 175
            return 0;

176 177 178 179 180
        /* At this point we know that the ring only contains nodes
	   from persistent objects, plus our own home node. We know
	   this because the ring lock is held.  We can safely assume
	   the current ring node is a persistent object now we know it
	   is not the home */
181
        object = OBJECT_FROM_RING(self, here, "scan_gc_items");
182 183
        if (!object) 
	    return -1;
184

185 186
	/* we are small enough */
        if (self->non_ghost_count <= target)
187
            return 0;
188
        else if (object->state == cPersistent_UPTODATE_STATE) {
189 190
            /* deactivate it. This is the main memory saver. */

191 192 193 194 195 196 197 198 199
            /* Add a placeholder; a dummy node in the ring.  We need
	       to do this to mark our position in the ring.  It is
	       possible that the PyObject_SetAttr() call below will
	       invoke an __setattr__() hook in Python.  If it does,
	       another thread might run; if that thread accesses a
	       persistent object and moves it to the head of the ring,
	       it might cause the gc scan to start working from the
	       head of the list.
	    */
Toby Dickenson's avatar
Toby Dickenson committed
200

201 202 203 204
            placeholder.next = here->next;
            placeholder.prev = here;
            here->next->prev = &placeholder;
            here->next = &placeholder;
205

206
            ENGINE_NOISE("G");
207

208
            /* In Python, "obj._p_changed = None" spells, ghostify */
209 210
            error = PyObject_SetAttr((PyObject *)object, py__p_changed, 
				     Py_None);
211

212 213 214 215 216 217

            /* unlink the placeholder */
            placeholder.next->prev = placeholder.prev;
            placeholder.prev->next = placeholder.next;

            here = placeholder.next;
218

219
            if (error)
220
                return -1; /* problem */
221 222
        }
        else {
223 224 225 226
            ENGINE_NOISE(".");
            here = here->next;
        }
    }
227
}
228

229
static PyObject *
230
lockgc(ccobject *self, int target_size)
231
{
232
    /* This is thread-safe because of the GIL, and there's nothing
233 234 235
     * in between checking the ring_lock and acquiring it that calls back
     * into Python.
     */
236
    if (self->ring_lock) {
237 238 239
        Py_INCREF(Py_None);
        return Py_None;
    }
240

241 242
    ENGINE_NOISE("<");
    self->ring_lock = 1;
243
    if (scan_gc_items(self, target_size)) {
244 245 246 247 248 249 250 251
        self->ring_lock = 0;
        return NULL;
    }
    self->ring_lock = 0;
    ENGINE_NOISE(">\n");

    Py_INCREF(Py_None);
    return Py_None;
252 253
}

254 255
static PyObject *
cc_incrgc(ccobject *self, PyObject *args)
Jim Fulton's avatar
Jim Fulton committed
256
{
257
    int n = 1;
258 259
    int starting_size = self->non_ghost_count;
    int target_size = self->cache_size;
Jim Fulton's avatar
Jim Fulton committed
260

261
    if (self->cache_drain_resistance >= 1) {
262 263
        /* This cache will gradually drain down to a small size. Check
           a (small) number of objects proportional to the current size */
264

265 266 267
        int target_size_2 = (starting_size - 1 
			     - starting_size / self->cache_drain_resistance);
        if (target_size_2 < target_size)
268
            target_size = target_size_2;
269
    }
Jim Fulton's avatar
Jim Fulton committed
270

271 272
    if (!PyArg_ParseTuple(args, "|i:incrgc", &n)) 
	return NULL;
273

274
    return lockgc(self, target_size);
Jim Fulton's avatar
Jim Fulton committed
275 276 277
}

static PyObject *
278
cc_full_sweep(ccobject *self, PyObject *args)
Jim Fulton's avatar
Jim Fulton committed
279
{
280 281 282
    int dt = 0;
    if (!PyArg_ParseTuple(args, "|i:full_sweep", &dt)) 
	return NULL;
Jeremy Hylton's avatar
Jeremy Hylton committed
283 284 285 286
    if (dt == 0)
	return lockgc(self, 0);
    else
	return cc_incrgc(self, args);
Jim Fulton's avatar
Jim Fulton committed
287 288
}

Jim Fulton's avatar
Jim Fulton committed
289
static PyObject *
Jeremy Hylton's avatar
Jeremy Hylton committed
290
cc_minimize(ccobject *self, PyObject *args)
Jim Fulton's avatar
Jim Fulton committed
291
{
Jeremy Hylton's avatar
Jeremy Hylton committed
292 293 294 295
    int ignored;
    if (!PyArg_ParseTuple(args, "|i:minimize", &ignored)) 
	return NULL;
    return lockgc(self, 0);
Jim Fulton's avatar
Jim Fulton committed
296 297
}

298 299
static void
_invalidate(ccobject *self, PyObject *key)
300
{
301
    PyObject *v = PyDict_GetItem(self->data, key);
302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320

    if (!v)
	return;
    if (PyExtensionClass_Check(v)) {
	if (v->ob_refcnt <= 1) {
	    self->klass_count--;
	    if (PyDict_DelItem(self->data, key) < 0)
		PyErr_Clear();
	}
	else {
	    v = PyObject_CallFunction(self->setklassstate, "O", v);
	    if (v) 
		Py_DECREF(v);
	    else 
		PyErr_Clear();
	}
    } else {
	if (PyObject_DelAttr(v, py__p_changed) < 0)
	    PyErr_Clear();
321 322
    }
}
323

324 325 326 327
static PyObject *
cc_invalidate(ccobject *self, PyObject *args)
{
  PyObject *inv, *key, *v;
328
  int i = 0;
Jeremy Hylton's avatar
Jeremy Hylton committed
329

330
  if (PyArg_ParseTuple(args, "O!", &PyDict_Type, &inv)) {
331 332
      while (PyDict_Next(inv, &i, &key, &v))
	  _invalidate(self, key);
Jeremy Hylton's avatar
Jeremy Hylton committed
333
      PyDict_Clear(inv);
334 335 336
  }
  else {
      PyErr_Clear();
Jeremy Hylton's avatar
Jeremy Hylton committed
337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356
      if (!PyArg_ParseTuple(args, "O:invalidate", &inv)) 
	  return NULL;
      if (PyString_Check(inv))
	  _invalidate(self, inv);
      else {
	  int l;
	  
	  PyErr_Clear();
	  l = PyObject_Length(inv);
	  if (l < 0)
	      return NULL;
	  for (i=l; --i >= 0; ) {
	      key = PySequence_GetItem(inv, i);
	      if (!key)
		  return NULL;
	      _invalidate(self, key);
	      Py_DECREF(key);
	  }
	  PySequence_DelSlice(inv, 0, l);
      }
357
  }
Jeremy Hylton's avatar
Jeremy Hylton committed
358
  
359 360 361
  Py_INCREF(Py_None);
  return Py_None;
}
362 363 364
  
static PyObject *
cc_get(ccobject *self, PyObject *args)
365
{
366
    PyObject *r, *key, *d = NULL;
367

368
    if (!PyArg_ParseTuple(args, "O|O:get", &key, &d)) 
Jeremy Hylton's avatar
Jeremy Hylton committed
369
	return NULL;
370

371
    r = PyDict_GetItem(self->data, key);
Jeremy Hylton's avatar
Jeremy Hylton committed
372
    if (!r) {
373 374 375
	if (d) {
	    r = d;
	} else {
Jeremy Hylton's avatar
Jeremy Hylton committed
376 377 378 379
	    PyErr_SetObject(PyExc_KeyError, key);
	    return NULL;
	}
    }
380
    Py_INCREF(r);
Jeremy Hylton's avatar
Jeremy Hylton committed
381
    return r;
382 383
}

384 385
static PyObject *
cc_klass_items(ccobject *self, PyObject *args)
386
{
387 388 389
    PyObject *l,*k,*v;
    int p = 0;

390 391
    if (!PyArg_ParseTuple(args, ":klass_items")) 
	return NULL;
392

393 394 395
    l = PyList_New(PyDict_Size(self->data));
    if (l == NULL) 
	return NULL;
396

397 398 399 400 401 402 403 404 405 406 407 408 409
    while (PyDict_Next(self->data, &p, &k, &v)) {
        if(PyExtensionClass_Check(v)) {
	    v = Py_BuildValue("OO", k, v);
	    if (v == NULL) {
		Py_DECREF(l);
		return NULL;
	    }
	    if (PyList_Append(l, v) < 0) {
		Py_DECREF(v);
		Py_DECREF(l);
		return NULL;
	    }
	    Py_DECREF(v);
410 411
        }
    }
412

413
    return l;
414 415 416
}

static PyObject *
417
cc_lru_items(ccobject *self, PyObject *args)
418
{
419 420
    PyObject *l;
    CPersistentRing *here;
421

422 423
    if (!PyArg_ParseTuple(args, ":lru_items")) 
	return NULL;
424

425
    if (self->ring_lock) {
426 427 428
	/* When the ring lock is held, we have no way of know which
	   ring nodes belong to persistent objects, and which a
	   placeholders. */
429 430
        PyErr_SetString(PyExc_ValueError,
		".lru_items() is unavailable during garbage collection");
431
        return NULL;
432
    }
433 434

    l = PyList_New(0);
435 436
    if (l == NULL) 
	return NULL;
437 438

    here = self->ring_home.next;
439
    while (here != &self->ring_home) {
440
        PyObject *v;
441
        cPersistentObject *object = OBJECT_FROM_RING(self, here, "cc_items");
442 443

        if (object == NULL) {
444 445 446
            Py_DECREF(l);
            return NULL;
        }
447 448
	v = Py_BuildValue("OO", object->oid, object);
	if (v == NULL) {
449 450
            Py_DECREF(l);
            return NULL;
451 452 453 454 455 456
	}
	if (PyList_Append(l, v) < 0) {
	    Py_DECREF(v);
            Py_DECREF(l);
            return NULL;
	}
457 458 459 460 461
        Py_DECREF(v);
        here = here->next;
    }

    return l;
462
}
463

464 465
static int
cc_oid_unreferenced(ccobject *self, PyObject *oid)
Jim Fulton's avatar
Jim Fulton committed
466
{
467 468 469 470 471 472 473 474
    /* This is called by the persistent object deallocation function
       when the reference count on a persistent object reaches
       zero. We need to fix up our dictionary; its reference is now
       dangling because we stole its reference count. Be careful to
       not release the global interpreter lock until this is
       complete. */

    PyObject *v;
Jim Fulton's avatar
Jim Fulton committed
475

476
    v = PyDict_GetItem(self->data, oid);
477 478
    if (v == NULL) {
	PyErr_SetObject(PyExc_KeyError, oid);
479
	return -1;
Jim Fulton's avatar
Jim Fulton committed
480 481
    }

482
    assert(v->ob_refcnt == 0);
483
    /* Need to be very hairy here because a dictionary is about
484 485
       to decref an already deleted object. 
    */
486 487

#ifdef Py_TRACE_REFS
488 489 490
    /* This is called from the deallocation function after the
       interpreter has untracked the reference.  Track it again.
     */
491
    _Py_NewReference(v);
492 493
    /* Don't increment total refcount as a result of the 
       shenanigans played in this function.  The _Py_NewReference()
494
       call above creates artificial references to v.
495
    */
496
    _Py_RefTotal--;
497 498 499
    /* XXX it may be a problem that v->ob_type is still NULL? 
       I don't understand what this comment means.  --jeremy */
    assert(v->ob_type);
500 501 502
#else
    Py_INCREF(v);
#endif
503
    assert(v->ob_refcnt == 1);
504 505 506 507
    /* Incremement the refcount again, because delitem is going to
       DECREF it.  If it's refcount reached zero again, we'd call back to
       the dealloc function that called us.
    */
508 509
    Py_INCREF(v);

510
    /* XXX Should we call _Py_ForgetReference() on error exit? */
511
    if (PyDict_DelItem(self->data, oid) < 0) 
512
	return -1;
513
    Py_DECREF((ccobject *)((cPersistentObject *)v)->cache);
514

515 516 517
    if (v->ob_refcnt != 1) {
        PyErr_SetString(PyExc_ValueError,
			"refcount is not 1 after removal from dict");
518
        return -1;
519 520
    }

521 522 523 524 525
    /* Undo the temporary resurrection.
       Don't DECREF the object, because this function is called from
       the object's dealloc function. If the refcnt reaches zero, it
       will all be invoked recursively.
     */
526 527
    _Py_ForgetReference(v);

528
    return 0;
Jim Fulton's avatar
Jim Fulton committed
529 530
}

531 532 533 534 535 536 537 538 539 540 541 542 543
static PyObject *
cc_ringlen(ccobject *self, PyObject *args)
{
    CPersistentRing *here;
    int c = 0;

    if (!PyArg_ParseTuple(args, ":ringlen"))
	return NULL;
    for (here = self->ring_home.next; here != &self->ring_home;
	 here = here->next)
	c++;
    return PyInt_FromLong(c);
}
544

Jim Fulton's avatar
Jim Fulton committed
545
static struct PyMethodDef cc_methods[] = {
546 547 548 549 550 551
  {"lru_items", (PyCFunction)cc_lru_items, METH_VARARGS,
   "List (oid, object) pairs from the lru list, as 2-tuples.\n"
   },
  {"klass_items", (PyCFunction)cc_klass_items, METH_VARARGS,
   "List (oid, object) pairs of cached persistent classes.\n"
   },
Jim Fulton's avatar
Jim Fulton committed
552
  {"full_sweep", (PyCFunction)cc_full_sweep, METH_VARARGS,
553
   "full_sweep([age]) -- Perform a full sweep of the cache\n\n"
Jeremy Hylton's avatar
Jeremy Hylton committed
554 555
   "Supported for backwards compatibility.  If the age argument is 0,\n"
   "behaves like minimize().  Otherwise, behaves like incrgc()."
556
   },
Jeremy Hylton's avatar
Jeremy Hylton committed
557 558 559 560
  {"minimize",	(PyCFunction)cc_minimize, METH_VARARGS,
   "minimize([ignored]) -- Remove as many objects as possible\n\n"
   "Ghostify all objects that are not modified.  Takes an optional\n"
   "argument, but ignores it."
561
   },
Jim Fulton's avatar
Jim Fulton committed
562
  {"incrgc", (PyCFunction)cc_incrgc, METH_VARARGS,
563 564 565
   "incrgc([n]) -- Perform incremental garbage collection\n\n"
   "Some other implementations support an optional parameter 'n' which\n"
   "indicates a repetition count; this value is ignored.\n"},
566 567
  {"invalidate", (PyCFunction)cc_invalidate, METH_VARARGS,
   "invalidate(oids) -- invalidate one, many, or all ids"},
Jim Fulton's avatar
Jim Fulton committed
568 569
  {"get", (PyCFunction)cc_get, METH_VARARGS,
   "get(key [, default]) -- get an item, or a default"},
570 571
  {"ringlen", (PyCFunction)cc_ringlen, METH_VARARGS,
   "ringlen() -- Returns number of non-ghost items in cache."},
Jim Fulton's avatar
Jim Fulton committed
572 573 574 575
  {NULL,		NULL}		/* sentinel */
};

static void
576
cc_dealloc(ccobject *self)
Jim Fulton's avatar
Jim Fulton committed
577
{
578 579 580 581
  Py_XDECREF(self->data);
  Py_XDECREF(self->jar);
  Py_XDECREF(self->setklassstate);
  PyMem_DEL(self);
Jim Fulton's avatar
Jim Fulton committed
582 583 584
}

static PyObject *
585
cc_getattr(ccobject *self, char *name)
Jim Fulton's avatar
Jim Fulton committed
586
{
587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604
  PyObject *r;

  if(*name=='c')
    {
      if(strcmp(name,"cache_age")==0)
	return PyInt_FromLong(0);   /* this cache does not use this value */
      if(strcmp(name,"cache_size")==0)
	return PyInt_FromLong(self->cache_size);
      if(strcmp(name,"cache_drain_resistance")==0)
	return PyInt_FromLong(self->cache_drain_resistance);
      if(strcmp(name,"cache_non_ghost_count")==0)
	return PyInt_FromLong(self->non_ghost_count);
      if(strcmp(name,"cache_klass_count")==0)
	return PyInt_FromLong(self->klass_count);
      if(strcmp(name,"cache_data")==0)
	{
	  /* now a copy of our data; the ring is too fragile */
	  return PyDict_Copy(self->data);
605
	}
606
    }
607 608 609 610 611 612 613
  if((*name=='h' && strcmp(name, "has_key")==0) ||
     (*name=='i' && strcmp(name, "items")==0) ||
     (*name=='k' && strcmp(name, "keys")==0)
     )
    return PyObject_GetAttrString(self->data, name);

  if((r=Py_FindMethod(cc_methods, (PyObject *)self, name)))
Jim Fulton's avatar
Jim Fulton committed
614
    return r;
615 616
  PyErr_Clear();
  return PyObject_GetAttrString(self->data, name);
Jim Fulton's avatar
Jim Fulton committed
617 618
}

Jim Fulton's avatar
Jim Fulton committed
619 620 621
static int
cc_setattr(ccobject *self, char *name, PyObject *value)
{
622 623
  if(value)
    {
Jim Fulton's avatar
Jim Fulton committed
624 625
      int v;

626 627 628
      if(strcmp(name,"cache_age")==0)
	{
	  /* this cache doesnt use the age */
629
	  return 0;
630
	}
Jim Fulton's avatar
Jim Fulton committed
631

632 633 634 635
      if(strcmp(name,"cache_size")==0)
	{
	  UNLESS(PyArg_Parse(value,"i",&v)) return -1;
	  self->cache_size=v;
636
	  return 0;
637 638 639 640 641 642 643 644 645
	}

      if(strcmp(name,"cache_drain_resistance")==0)
	{
	  UNLESS(PyArg_Parse(value,"i",&v)) return -1;
	  self->cache_drain_resistance=v;
	  return 0;
	}
    }
Jim Fulton's avatar
Jim Fulton committed
646 647 648 649
  PyErr_SetString(PyExc_AttributeError, name);
  return -1;
}

Jim Fulton's avatar
Jim Fulton committed
650
static int
651
cc_length(ccobject *self)
Jim Fulton's avatar
Jim Fulton committed
652
{
Jeremy Hylton's avatar
Jeremy Hylton committed
653
    return PyObject_Length(self->data);
Jim Fulton's avatar
Jim Fulton committed
654 655 656
}
  
static PyObject *
657
cc_subscript(ccobject *self, PyObject *key)
Jim Fulton's avatar
Jim Fulton committed
658
{
Jeremy Hylton's avatar
Jeremy Hylton committed
659
    PyObject *r;
Jim Fulton's avatar
Jim Fulton committed
660

661
    r = PyDict_GetItem(self->data, key);
Jeremy Hylton's avatar
Jeremy Hylton committed
662 663 664 665
    if (r == NULL) {
	PyErr_SetObject(PyExc_KeyError, key);
	return NULL;
    }
666
    Py_INCREF(r);
Jim Fulton's avatar
Jim Fulton committed
667

Jeremy Hylton's avatar
Jeremy Hylton committed
668
    return r;
Jim Fulton's avatar
Jim Fulton committed
669 670 671
}

static int
672
cc_add_item(ccobject *self, PyObject *key, PyObject *v)
Jim Fulton's avatar
Jim Fulton committed
673
{
674
    int result;
675
    PyObject *oid, *object_again, *jar;
676
    cPersistentObject *p;
677

678 679 680 681 682 683
    if (PyExtensionClass_Check(v)) {
        /* Its a persistent class, such as a ZClass. Thats ok. */
    }
    else if( PyExtensionInstance_Check(v) &&
             (((PyExtensionClass*)(v->ob_type))->class_flags & PERSISTENT_TYPE_FLAG) &&
             (v->ob_type->tp_basicsize >= sizeof(cPersistentObject)) ) {
684 685
        /* Its and instance of a persistent class, (ie Python classeses that
        derive from Persistence.Persistent, BTrees, etc). Thats ok. */
686
    }
687
    else {
688
	PyErr_SetString(PyExc_TypeError, 
689 690 691 692
			"Cache values must be persistent objects.");
	return -1;
    }

693 694 695
    /* Can't access v->oid directly because the object might be a
     *  persistent class.
     */
696
    oid = PyObject_GetAttr(v, py__p_oid);
697 698
    if (oid == NULL)
	return -1;
699 700 701 702 703 704 705 706 707
    if (!PyString_Check(oid)) {
        PyErr_Format(PyExc_TypeError,
                     "Cached object oid must be a string, not a %s",
		     oid->ob_type->tp_name);
	return -1;
    }
    /*  we know they are both strings.
     *  now check if they are the same string.
     */
Jeremy Hylton's avatar
Jeremy Hylton committed
708 709
    result = PyObject_Compare(key, oid);
    if (PyErr_Occurred()) {
710 711
	Py_DECREF(oid);
	return -1;
Jeremy Hylton's avatar
Jeremy Hylton committed
712
    } 
713 714
    Py_DECREF(oid);
    if (result) {
715
	PyErr_SetString(PyExc_ValueError, "Cache key does not match oid");
716 717
	return -1;
    }
Jeremy Hylton's avatar
Jeremy Hylton committed
718

719 720 721 722
    /* useful sanity check, but not strictly an invariant of this class */
    jar = PyObject_GetAttr(v, py__p_jar);
    if (jar == NULL)
        return -1;
723
    if (jar==Py_None) {
Toby Dickenson's avatar
Toby Dickenson committed
724
        Py_DECREF(jar);
725 726 727 728
        PyErr_SetString(PyExc_ValueError,
                        "Cached object jar missing");
	return -1;
    }
729
    Py_DECREF(jar);
730

731
    object_again = PyDict_GetItem(self->data, key);
732 733 734
    if (object_again) {
	if (object_again != v) {
	    PyErr_SetString(PyExc_ValueError,
735
		    "Can not re-register object under a different oid");
736 737 738 739 740 741
	    return -1;
	} else {
	    /* re-register under the same oid - no work needed */
	    return 0;
	}
    }
Jeremy Hylton's avatar
Jeremy Hylton committed
742

743
    if (PyExtensionClass_Check(v)) {
Jeremy Hylton's avatar
Jeremy Hylton committed
744
	if (PyDict_SetItem(self->data, key, v) < 0) 
745 746 747 748 749 750 751 752 753
	    return -1;
	self->klass_count++;
	return 0;
    } else {
	PerCache *cache = ((cPersistentObject *)v)->cache;
	if (cache) {
	    if (cache != (PerCache *)self)
		/* This object is already in a different cache. */
		PyErr_SetString(PyExc_ValueError, 
754
				"Cache values may only be in one cache.");
755 756 757 758 759 760 761 762
	    return -1;
	} 
	/* else:
	   
	   This object is already one of ours, which is ok.  It
	   would be very strange if someone was trying to register
	   the same object under a different key. 
	*/
763
    }
764
    
Jeremy Hylton's avatar
Jeremy Hylton committed
765
    if (PyDict_SetItem(self->data, key, v) < 0) 
766
	return -1;
767
    /* the dict should have a borrowed reference */
768
    Py_DECREF(v);
769 770
    
    p = (cPersistentObject *)v;
771
    Py_INCREF(self);
772 773 774
    p->cache = (PerCache *)self;
    if (p->state >= 0) {
	/* insert this non-ghost object into the ring just 
775
	   behind the home position. */
776 777 778 779 780
	self->non_ghost_count++;
	p->ring.next = &self->ring_home;
	p->ring.prev =  self->ring_home.prev;
	self->ring_home.prev->next = &p->ring;
	self->ring_home.prev = &p->ring;
781 782
	/* this list should have a new reference to the object */
	Py_INCREF(v);
783
    }
784
    return 0;
785
}
786

787 788 789 790 791 792 793
static int
cc_del_item(ccobject *self, PyObject *key)
{
    PyObject *v;
    cPersistentObject *p;

    /* unlink this item from the ring */
794
    v = PyDict_GetItem(self->data, key);
795 796 797 798 799 800 801 802 803 804 805 806 807
    if (v == NULL)
	return -1;

    if (PyExtensionClass_Check(v)) {
	self->klass_count--;
    } else {
	p = (cPersistentObject *)v;
	if (p->state >= 0) {
	    self->non_ghost_count--;
	    p->ring.next->prev = p->ring.prev;
	    p->ring.prev->next = p->ring.next;
	    p->ring.prev = NULL;
	    p->ring.next = NULL;
808 809
	    /* The DelItem below will account for the reference
	       held by the list. */
810 811 812 813 814 815 816 817
	} else {
	    /* This is a ghost object, so we havent kept a reference
	       count on it.  For it have stayed alive this long
	       someone else must be keeping a reference to
	       it. Therefore we need to temporarily give it back a
	       reference count before calling DelItem below */
	    Py_INCREF(v);
	}
818

819 820 821
	Py_DECREF((PyObject *)p->cache);
	p->cache = NULL;
    }
822

823 824
    if (PyDict_DelItem(self->data, key) < 0) {
	PyErr_SetString(PyExc_RuntimeError,
825
			"unexpectedly couldn't remove key in cc_ass_sub");
826 827
	return -1;
    }
828

829
    return 0;
Jim Fulton's avatar
Jim Fulton committed
830 831
}

832 833
static int
cc_ass_sub(ccobject *self, PyObject *key, PyObject *v)
834
{
835
    if (!PyString_Check(key)) {
836 837
	PyErr_Format(PyExc_TypeError,
                     "cPickleCache key must be a string, not a %s",
838
		     key->ob_type->tp_name);
839
	return -1;
840
    }
841 842 843 844 845
    if (v)
	return cc_add_item(self, key, v);
    else
	return cc_del_item(self, key);
}
846

Jim Fulton's avatar
Jim Fulton committed
847 848 849 850 851 852 853
static PyMappingMethods cc_as_mapping = {
  (inquiry)cc_length,		/*mp_length*/
  (binaryfunc)cc_subscript,	/*mp_subscript*/
  (objobjargproc)cc_ass_sub,	/*mp_ass_subscript*/
};

static PyTypeObject Cctype = {
Jeremy Hylton's avatar
Jeremy Hylton committed
854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871
    PyObject_HEAD_INIT(NULL)
    0,				/*ob_size*/
    "cPickleCache",		/*tp_name*/
    sizeof(ccobject),		/*tp_basicsize*/
    0,				/*tp_itemsize*/
    /* methods */
    (destructor)cc_dealloc,	/*tp_dealloc*/
    (printfunc)0,		/*tp_print*/
    (getattrfunc)cc_getattr,	/*tp_getattr*/
    (setattrfunc)cc_setattr,	/*tp_setattr*/
    (cmpfunc)0,			/*tp_compare*/
    (reprfunc)0,   		/*tp_repr*/
    0,				/*tp_as_number*/
    0,				/*tp_as_sequence*/
    &cc_as_mapping,		/*tp_as_mapping*/
    (hashfunc)0,		/*tp_hash*/
    (ternaryfunc)0,		/*tp_call*/
    (reprfunc)0,  		/*tp_str*/
Jim Fulton's avatar
Jim Fulton committed
872 873
};

Jeremy Hylton's avatar
Jeremy Hylton committed
874
static ccobject *
875
newccobject(PyObject *jar, int cache_size)
Jeremy Hylton's avatar
Jeremy Hylton committed
876
{
877
    ccobject *self;
Jeremy Hylton's avatar
Jeremy Hylton committed
878
  
879 880 881 882 883 884 885
    self = PyObject_NEW(ccobject, &Cctype);
    if (self == NULL)
	return NULL;
    self->setklassstate = self->jar = NULL;
    self->data = PyDict_New();
    if (self->data == NULL) {
	Py_DECREF(self);
Jeremy Hylton's avatar
Jeremy Hylton committed
886
	return NULL;
887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902
    }
    self->setklassstate = PyObject_GetAttrString(jar, "setklassstate");
    if (self->setklassstate == NULL) {
	Py_DECREF(self);
	return NULL;
    }
    self->jar = jar; 
    Py_INCREF(jar);
    self->cache_size = cache_size;
    self->non_ghost_count = 0;
    self->klass_count = 0;
    self->cache_drain_resistance = 0;
    self->ring_lock = 0;
    self->ring_home.next = &self->ring_home;
    self->ring_home.prev = &self->ring_home;
    return self;
Jeremy Hylton's avatar
Jeremy Hylton committed
903 904
}

Jim Fulton's avatar
Jim Fulton committed
905
static PyObject *
906
cCM_new(PyObject *self, PyObject *args)
Jim Fulton's avatar
Jim Fulton committed
907
{
908
    int cache_size=100;
909
    PyObject *jar;
Jim Fulton's avatar
Jim Fulton committed
910

911
    if (!PyArg_ParseTuple(args, "O|i", &jar, &cache_size))
912
	return NULL;
913
    return (PyObject*)newccobject(jar, cache_size);
Jim Fulton's avatar
Jim Fulton committed
914 915 916
}

static struct PyMethodDef cCM_methods[] = {
Jeremy Hylton's avatar
Jeremy Hylton committed
917 918
    {"PickleCache", (PyCFunction)cCM_new, METH_VARARGS},
    {NULL, NULL} /* sentinel */
Jim Fulton's avatar
Jim Fulton committed
919 920 921
};

void
Jeremy Hylton's avatar
Jeremy Hylton committed
922
initcPickleCache(void)
Jim Fulton's avatar
Jim Fulton committed
923
{
924 925
    PyObject *m, *d;
    cPersistenceCAPIstruct *capi;
926

927
    Cctype.ob_type = &PyType_Type;
928

929 930
    if (!ExtensionClassImported) 
	return;
931

932 933 934 935
    capi = (cPersistenceCAPIstruct *)PyCObject_Import("cPersistence", "CAPI");
    if (!capi)
	return;
    capi->percachedel = (percachedelfunc)cc_oid_unreferenced;
Jim Fulton's avatar
Jim Fulton committed
936

937 938
    m = Py_InitModule4("cPickleCache", cCM_methods, cPickleCache_doc_string,
		       (PyObject*)NULL, PYTHON_API_VERSION);
Jim Fulton's avatar
Jim Fulton committed
939

940 941 942 943
    py_reload = PyString_InternFromString("reload");
    py__p_jar = PyString_InternFromString("_p_jar");
    py__p_changed = PyString_InternFromString("_p_changed");
    py__p_oid = PyString_InternFromString("_p_oid");
944

945
    d = PyModule_GetDict(m);
946

947
    PyDict_SetItemString(d, "cache_variant", PyString_FromString("stiff/c"));
Jim Fulton's avatar
Jim Fulton committed
948
}