The ZODB package includes a number of related modules that provide useful data types such as BTrees.
The PersistentMapping class is a wrapper for mapping objects that will set the dirty bit when the mapping is modified by setting or deleting a key.
container = {}) |
PersistentMapping objects support all the same methods as Python dictionaries do.
The PersistentList class is a wrapper for mutable sequence objects, much as PersistentMapping is a wrapper for mappings.
initlist = []) |
PersistentList objects support all the same methods as Python lists do.
When programming with the ZODB, Python dictionaries aren't always what you need. The most important case is where you want to store a very large mapping. When a Python dictionary is accessed in a ZODB, the whole dictionary has to be unpickled and brought into memory. If you're storing something very large, such as a 100,000-entry user database, unpickling such a large object will be slow. BTrees are a balanced tree data structure that behave like a mapping but distribute keys throughout a number of tree nodes. The nodes are stored in sorted order (this has important consequences - see below). Nodes are then only unpickled and brought into memory as they're accessed, so the entire tree doesn't have to occupy memory (unless you really are touching every single key).
The BTrees package provides a large collection of related data structures. There are variants of the data structures specialized to handle integer values, which are faster and use less memory. There are four modules that handle the different variants. The first two letters of the module name specify the types of the keys and values in mappings - O for any object and I for integer. For example, the BTrees.IOBTree module provides a mapping with integer keys and arbitrary objects as values.
The four data structures provide by each module are a btree, a bucket, a tree set, and a set. The btree and bucket types are mappings and support all the usual mapping methods, e.g. update() and keys(). The tree set and set types are similar to mappings but they have no values; they support the methods that make sense for a mapping with no keys, e.g. keys() but not items(). The bucket and set types are the individual building blocks for btrees and tree sets, respectively. A bucket or set can be used when you are sure that it will have few elements. If the data structure will grow large, you should use a btree or tree set. Like Python lists, buckets and sets are allocated in one contiguous piece, and insertions and deletions can take time proportional to the number of existing elements. Btrees and tree sets are multi-level tree structures with much better (logarithmic) worst-case time bounds.
The four modules are named OOBTree, IOBTree, OIBTree, and IIBTree. The two letter prefixes are repeated in the data types names. The BTrees.OOBTree module defines the following types: OOBTree, OOBucket, OOSet, and OOTreeSet. Similarly, the other three modules each define their own variants of those four types.
The keys(), values(), and items() methods on btree and tree set types do not materialize a list with all of the data. Instead, they return lazy sequences that fetch data from the BTree as needed. They also support optional arguments to specify the minimum and maximum values to return, often called "range searching". Because all these types are stored in sorted order, range searching is very efficient.
The keys(), values(), and items() methods on bucket and set types do return lists with all the data. Starting in ZODB4, there are also iterkeys(), itervalues(), and iteritems() methods that return iterators (in the Python 2.2 sense).
A BTree object supports all the methods you would expect of a mapping with a few extensions that exploit the fact that the keys are sorted. The example below demonstrates how some of the methods work. The extra methods are minKey() and maxKey(), which find the minimum and maximum key value subject to an optional bound argument, and byValue(), which should probably be ignored (it's hard to explain exactly what it does, and as a result it's almost never used - best to consider it deprecated).
>>> from BTrees.OOBTree import OOBTree >>> t = OOBTree() >>> t.update({ 1: "red", 2: "green", 3: "blue", 4: "spades" }) >>> len(t) 4 >>> t[2] 'green' >>> s = t.keys() # this is a "lazy" sequence object >>> s <OOBTreeItems object at 0x0088AD20> >>> len(s) # it acts like a Python list 4 >>> s[-2] 3 >>> list(s) # materialize the full list [1, 2, 3, 4] >>> list(t.values()) ['red', 'green', 'blue', 'spades'] >>> list(t.values(1, 2)) ['red', 'green'] >>> list(t.values(2)) ['green', 'blue', 'spades'] >>> t.minKey() # smallest key 1 >>> t.minKey(1.5) # smallest key >= 1.5 2
Each of the modules also defines some functions that operate on BTrees - difference(), union(), and intersection(). The difference() function returns a bucket, while the other two methods return a set. If the keys are integers, then the module also defines multiunion(). If the values are integers, then the module also defines weightedIntersection() and weightedUnion(). The function doc strings describe each function briefly.
The BTree-based data structures differ from Python dicts in several fundamental ways. One of the most important is that while dicts require that keys support hash codes and equality comparison, the btree-based structures don't use hash codes and require a total ordering on keys.
Total ordering means three things:
x == x
is true.
x < y
, x == y
, and
x > y
is true.
x <= y
and
y <= z
, it's also true that
x <= z
.
The default comparison functions for most objects that come with Python
satisfy these rules, with some crucial cautions explained later. Complex
numbers are an example of an object whose default comparison function
does not satisfy these rules: complex numbers only support ==
and !=
comparisons, and raise an exception if you try to compare
them in any other way. They don't satisfy the trichotomy rule, and must
not be used as keys in btree-based data structures (although note that
complex numbers can be used as keys in Python dicts, which do not require
a total ordering).
Examples of objects that are wholly safe to use as keys in btree-based structures include ints, longs, floats, 8-bit strings, Unicode strings, and tuples composed (possibly recursively) of objects of wholly safe types.
It's important to realize that even if two types satisfy the
rules on their own, mixing objects of those types may not. For example,
8-bit strings and Unicode strings both supply total orderings, but mixing
the two loses trichotomy; e.g., 'x' < chr(255)
and
u'x' == 'x'
, but trying to compare chr(255)
to
u'x'
raises an exception. Partly for this reason (another is
given later), it can be dangerous to use keys with multiple types in
a single btree-based structure. Don't try to do that, and you don't
have to worry about it.
Another potential problem is mutability: when a key is inserted in a btree-based structure, it must retain the same order relative to the other keys over time. This is easy to run afoul of if you use mutable objects as keys. For example, lists supply a total ordering, and then
>>> L1, L2, L3 = [1], [2], [3] >>> from BTrees.OOBTree import OOSet >>> s = OOSet((L2, L3, L1)) # this is fine, so far >>> list(s.keys()) # note that the lists are in sorted order [[1], [2], [3]] >>> s.has_key([3]) # and [3] is in the set 1 >>> L2[0] = 5 # horrible -- the set is insane now >>> s.has_key([3]) # for example, it's insane this way 0 >>> s OOSet([[1], [5], [3]]) >>>
Key lookup relies on that the keys remain in sorted order (an efficient form of binary search is used). By mutating key L2 after inserting it, we destroyed the invariant that the OOSet is sorted. As a result, all future operations on this set are unpredictable.
A subtler variant of this problem arises due to persistence: by default, Python does several kinds of comparison by comparing the memory addresses of two objects. Because Python never moves an object in memory, this does supply a usable (albeit arbitrary) total ordering across the life of a program run (an object's memory address doesn't change). But if objects compared in this way are used as keys of a btree-based structure that's stored in a database, when the objects are loaded from the database again they will almost certainly wind up at different memory addresses. There's no guarantee then that if key K1 had a memory address smaller than the memory address of key K2 at the time K1 and K2 were inserted in a BTree, K1's address will also be smaller than K2's when that BTree is loaded from a database later. The result will be an insane BTree, where various operations do and don't work as expected, seemingly at random.
Now each of the types identified above as "wholly safe to use" never compares two instances of that type by memory address, so there's nothing to worry about here if you use keys of those types. The most common mistake is to use keys that are instances of a user-defined class that doesn't supply its own __cmp__() method. Python compares such instances by memory address. This is fine if such instances are used as keys in temporary btree-based structures used only in a single program run. It can be disastrous if that btree-based structure is stored to a database, though.
>>> class C: ... pass ... >>> a, b = C(), C() >>> print a < b # this may print 0 if you try it 1 >>> del a, b >>> a, b = C(), C() >>> print a < b # and this may print 0 or 1 0 >>>
That example illustrates that comparison of instances of classes that don't define __cmp__() yields arbitrary results (but consistent results within a single program run).
Another problem occurs with instances of classes that do define __cmp__(), but define it incorrectly. It's possible but rare for a custom __cmp__() implementation to violate one of the three required formal properties directly. It's more common for it to fall back" to address-based comparison by mistake. For example,
class Mine: def __cmp__(self, other): if other.__class__ is Mine: return cmp(self.data, other.data) else: return cmp(self.data, other)
It's quite possible there that the else clause allows a result to be computed based on memory address. The bug won't show up until a btree-based structure uses objects of class Mine as keys, and also objects of other types as keys, and the structure is loaded from a database, and a sequence of comparisons happens to execute the else clause in a case where the relative order of object memory addresses happened to change.
This is as difficult to track down as it sounds, so best to stay far away from the possibility.
You'll stay out of trouble by follwing these rules, violating them only with great care:
Any part of a comparison implementation that relies (explicitly or implicitly) on an address-based comparison result will eventually cause serious failure.