Reference

Interfaces

Interfaces for objects defined in the ZODB package.

interface nti.zodb.interfaces.INumericCounter

Extends: nti.zodb.interfaces.INumericValue

A counter that can be incremented. Conflicts are resolved by merging the numeric value of the difference in magnitude of changes. Intended to be used for monotonically increasing counters, typically integers.

interface nti.zodb.interfaces.INumericValue

Extends: zope.minmax.interfaces.IAbstractValue

A persistent numeric value with conflict resolution.

set(value)

Change the value of this object to the given value.

If the number is immutable, and the value is not the current value, this may raise NotImplementedError.

value

The numeric value of this object.

Implementation:nti.schema.field.Number
Read Only:False
Required:True
Default Value:None
Allowed Type:numbers.Number
increment(amount=1)

Increment the value by the specified amount (which should be non-negative).

Returns:The counter with the incremented value (this object).
__gt__(other)

These values are ordered like their values.

__hash__()

This object hashes like its value.

Caution

Do not place this object in a hash container and then mutate the value.

__lt__(other)

These objects are ordered like their values.

__eq__(other)

Is this object holding a value numerically equal to the other?

interface nti.zodb.interfaces.ITokenBucket

A token bucket is used in rate limiting applications. It has a maximum capacity and a rate at which tokens are regenerated (typically this is in terms of absolute time).

Clients attempt to consume tokens and are either allowed or denied based on whether there are enough tokens in the bucket.

tokens

The current number of tokens in the bucket at this instant.

Implementation:zope.schema.Float
Read Only:False
Required:True
Default Value:None
Allowed Type:float
fill_rate

The rate in tokens per second at which new tokens arrive.

Implementation:zope.schema.Float
Read Only:False
Required:True
Default Value:1.0
Allowed Type:float
wait_for_token()

Consume a single whole token from the bucket, blocking until one is available if need be. This is not meant to be used from multiple threads.

capacity

The maximum capacity of the token bucket.

Implementation:zope.schema.Float
Read Only:False
Required:True
Default Value:None
Allowed Type:float
consume(tokens=1)

Consume tokens from the bucket.

Parameters:tokens – The fractional number of tokens to consume. The default is to consume one whole token, which is probably what you want.
Returns:True if there were sufficient tokens, otherwise False. If True, then the value of tokens will have been reduced.
exception nti.zodb.interfaces.UnableToAcquireCommitLock

Bases: ZODB.POSException.StorageError

A ZODB storage (typically RelStorage) was unable to acquire the required commit lock.

This class is only used if RelStorage is not available; otherwise it is an alias for relstorage.adapters.interfaces.UnableToAcquireCommitLock.

nti.zodb.interfaces.ZODBUnableToAcquireCommitLock

alias of nti.zodb.interfaces.UnableToAcquireCommitLock

Utilities

Common helpers.

nti.zodb.isBroken(obj, uid=None)

Check if the object is broken (missing or an implementation of ZODB.interfaces.IBroken).

Parameters:uid (str) – A debugging aid, unless the obj is none, in which case it must be non-None to result in a True return. This makes no sense, so try to avoid passing objects that are None to this function.
nti.zodb.is_broken(obj, uid=None)

Check if the object is broken (missing or an implementation of ZODB.interfaces.IBroken).

Parameters:uid (str) – A debugging aid, unless the obj is none, in which case it must be non-None to result in a True return. This makes no sense, so try to avoid passing objects that are None to this function.
nti.zodb.readCurrent(obj, container=True)

Persistence safe wrapper around zodb connection readCurrent; also has some built in smarts about typical objects that need to be read together.

Activity Log

Logging of database connection activity. Activate this with ZCML:

<include package="nti.zodb" file="configure_activitylog.zcml" />

Originally based on code from the unreleased zc.zodbactivitylog.

nti.zodb.activitylog.AbstractActivityMonitor

alias of nti.zodb.activitylog.ComponentActivityMonitor

class nti.zodb.activitylog.ActivityMonitorComponent

Bases: object

One component of a chain of activity monitors.

class nti.zodb.activitylog.ActivityMonitorData(loads, stores, db_name, pool_all_count, pool_idle_count)

Bases: tuple

Create new instance of ActivityMonitorData(loads, stores, db_name, pool_all_count, pool_idle_count)

db_name

Alias for field number 2

loads

Alias for field number 0

pool_all_count

Alias for field number 3

pool_idle_count

Alias for field number 4

stores

Alias for field number 1

class nti.zodb.activitylog.ComponentActivityMonitor(base=None, components=())

Bases: object

Activity monitor that:

  • Can call another activity monitor; this is useful for adding on to an existing activity monitor.
  • Can call a sequence of callables with some data. This is useful for composing many activity monitors much more cheaply than if they were on their own.
Parameters:
class nti.zodb.activitylog.LogActivityComponent

Bases: nti.zodb.activitylog.ActivityMonitorComponent

An activity monitor component that logs connection transfer information and pool information.

Changed in version 1.3.0: Add min_loads and min_stores. These additional thresholds are tested in addition to min_loads_and_stores and if any threshold is reached the logging happens. The default value of each threshold is 10.

The thresholds may be configured in the environment (before loading this class) as integer strings using the values NTI_ZODB_LOG_MIN_LOADS, NTI_ZODB_LOG_MIN_STORES, and NTI_ZODB_LOG_MIN_ACTIVITY, respectively.

class nti.zodb.activitylog.LogActivityMonitor(base=None)

Bases: nti.zodb.activitylog.ComponentActivityMonitor

A pre-configured ComponentActivityMonitor that uses LogActivityComponent()

class nti.zodb.activitylog.StatsdActivityComponent

Bases: nti.zodb.activitylog.ActivityMonitorComponent

An activity monitor component that stores counters (guages) in statsd, if the statsd client is available.

The stats are:

  • ZODB.DB.<name>.loads How many loads the open connection performed.
  • ZODB.DB.<name>.stores How many stores the open connection performed.
  • ZODB.DB.<name>.total_connections All currently open connections, including those in use and those in the pool.
  • ZODB.DB.<name>.available_connections The connections sitting idle in the pool.
static statsd_client()

Return the current StatsdClient for the thread.

Returns the thread-local client if there is one, or the global client if there is one, or None.

class nti.zodb.activitylog.StatsdActivityMonitor(base=None)

Bases: nti.zodb.activitylog.ComponentActivityMonitor

A pre-configured ComponentActivityMonitor that uses LogActivityComponent()

nti.zodb.activitylog.register_subscriber(event)

Subscriber to the zope.processlifetime.IDatabaseOpenedWithRoot that registers an activity monitor.

Containers

Utilities for working with containers, particularly BTree containers.

nti.zodb.containers.bit64_int_to_time(value)

Convert a 64 bit integer to its floating point value.

Inverse of time_to_64bit_int().

nti.zodb.containers.time_to_64bit_int(value)

Given a Python floating point object (usually a time value), losslessly return a 64-bit long int that represents it. Useful for storing as the value in a OL tree, when you really want a float (since BTrees does not provide OF), or as a key in a Lx tree.

Copy

Things to assist with copying persistent objects.

nti.zodb.copy.wref_copy_factory(ref)

Weak references cannot typically be copied due to the presence of the Connection attribute (in the dm value). This factory makes them copyable.

Currently we assume that the reference can be resolved at copy time (since we cannot create a reference to None).

MinMax

Conflict resolving value/counter implementations for use on persistent objects.

class nti.zodb.minmax.AbstractNumericValue(value=0)

Bases: zope.minmax._minmax.AbstractValue

A numeric value that provides ordering operations. Defaults to zero.

nti.zodb.minmax.ConstantZeroValue(value=0)

Use this as a class attribute for a default values of zero. The value cannot be changed, and instances cannot be serialized.

class nti.zodb.minmax.MergingCounter(value=0)

Bases: nti.zodb.minmax.AbstractNumericValue

A zope.minmax item that resolves conflicts by merging the numeric value of the difference in magnitude of changes. Intented to be used for monotonically increasing counters.

As a special case, if the counter is reset to zero by both transactions, that becomes the new state.

Changed in version 1.2.0: Special case setting the counter to zero.

class nti.zodb.minmax.NumericMaximum(value=0)

Bases: nti.zodb.minmax.AbstractNumericValue, zope.minmax._minmax.Maximum

Maximizes the number during conflicts.

class nti.zodb.minmax.NumericMinimum(value=0)

Bases: nti.zodb.minmax.AbstractNumericValue, zope.minmax._minmax.Minimum

Minimizes the number during conflicts.

class nti.zodb.minmax.NumericPropertyDefaultingToZero(name, factory, as_number=False)

Bases: nti.zodb.persistentproperty.PropertyHoldingPersistent

In persistent objects (that extend nti.zodb.persistentproperty.PersistentPropertyHolder), use this to hold a merging counter or numeric minimum or maximum.

This is a data descriptor:

class Foo(PersistentPropertyHolder):

   a = NumericPropertyDefaultingToZero('a')

Creates a new property in a new-style class that does not use __slots__ (persistent classes shouldn’t use __slots__ anyway).

Parameters:
  • name (str) – The name of the property; this will be the key in the instance dictionary. This should match the name of the property (e.g., a = NumericPropertyDefaultingToZero( 'a',...)) but is not required to. It must be a native string (bytes on py2, str/unicode on py3).
  • factory (callable) – The value object factory that determines the type of conflict resolution used for this property. Typically NumericMaximum(), NumericMinimum or MergingCounter.
  • as_number (bool) – If set to True (not the default), then when an instance reads this property, the numeric value will be returned; otherwise the factory class instance will be returned and you will want to access its .value attribute. Setting this property always takes the (raw) numeric value.
class IncrementingZeroValue(name, holder, prop)

Bases: nti.zodb.minmax._ConstantZeroValue

Persistent Property

Classes for making property objects (actually, general descriptors) more convenient for working with in persistent.Persistent objects.

class nti.zodb.persistentproperty.PersistentPropertyHolder

Bases: persistent.Persistent

Lets you assign to a property without necessarily changing the _p_status of this object.

In a subclass of persistent.Persistent, the __setattr__ method sets _p_changed to True when called with a name argument that does not start with _p_ (properties of the persistent object itself) or _v_ (volatile properties). This makes it hard to use with conflict-reducing objects like nti.zodb.minmax.NumericMaximum: instead of being able to define a descriptor to access and mutate them directly, you must remember to go through their API, and replacing existing simple attributes (a plain number) with a property doesn’t actually reduce conflicts until all callers have been updated to use the API.

This superclass fixes that problem. When __setattr__() is called, it checks to see if the underlying attribute is actually a descriptor extending PropertyHoldingPersistent, and if so, delegates directly to that object. That object is responsible for managing the persistent state of that instance.

Caution

When you subclass this, you should not modify the type after the first instance is constructed by adding new PropertyHoldingPersistent instances. As an implementation note, the __new__ method caches the properties that are PropertyHoldingPersistent. Adding new ones will bypass the cache (and make the instance modified) but otherwise still behave correctly. Replacing one with a different type of property or deleting the property altogether may not function correctly.

class nti.zodb.persistentproperty.PropertyHoldingPersistent

Bases: object

Base class mixin for a property that, when installed in a PersistentPropertyHolder, can be used to hold another persistent object. This property object takes all responsibility for changing persistent state (of the instance it is installed in) if needed.

Token Bucket

Implementations of the token bucket algorithm.

class nti.zodb.tokenbucket.PersistentTokenBucket(capacity, fill_rate=1.0)

Bases: object

Persistent implementation of the token bucket algorithm. If the ZODB is used from multiple machines, relies on their clocks being relatively synchronized to be effective.

Initially based on an ActiveState recipe

(For efficiency, this object itself isn’t persistent, but the objects it holds are.)

Creates a new token bucket, initially full.

Parameters:
  • capacity – The max number of tokens in the bucket (also the initial number of tokens in the bucket.
  • fill_rate – The rate in fractional tokens per second that the bucket will fill. The default is to add one token per second.
consume(tokens=1)

Consume one or more tokens from the bucket. Returns True if there were sufficient tokens otherwise False.

tokens

The fractional number of tokens currently in the bucket.

wait_for_token()

Consume a single token from the bucket, blocking until one is available if need be.

Weak References

Utilities for weak references to persistent objects.

class nti.zodb.wref.CopyingWeakRef(ob)

Bases: persistent.wref.WeakRef

A weak ref that also stores a one-shot copy (using zope.copy.copy()) of its reference, as a fallback to return from __call__() if the weak reference cannot be resolved.

Use this if the object is expected to mutate after this reference is established and you normally want to have access to those changes, and you cannot tolerate the object going missing, but you also cannot keep a strong reference to the object. This may be particularly the case in cross-database refs.

Implements nti.wref.interfaces.ICachingWeakRef.

ZlibStorage

Utilities for working with zc.zlibstorage.

TODO: Add support for zodburi, which can be installed automatically as entry points in setup.py.

class nti.zodb.zlibstorage.ZlibStorageClientStorageURIResolver

Bases: repoze.zodbconn.resolvers.ClientStorageURIResolver

Wraps ZEO.ClientStorage.ClientStorage with zc.zlibstorage when using the zlibzeo URI scheme.

class nti.zodb.zlibstorage.ZlibStorageFileStorageURIResolver

Bases: repoze.zodbconn.resolvers.FileStorageURIResolver

Wraps ZODB.FileStorage.FileStorage with zc.zlibstorage when using the zlibfile URI scheme.

nti.zodb.zlibstorage.install_zlib_client_resolver()

Makes it possible for repoze.zodbconn.uri.db_from_uri() to connect to ZEO servers that are using zlib storage, through providing support for the use of the zlibzeo URI scheme, and likewise for zlibfile://.

Alternatively, you can use a ZConfig file and the zconfig:// URI scheme.

BTrees

Support for using BTrees with ZODB.

This module is primarily concerned with making BTrees even more efficient, especially at large scale and high concurrency.

family64LargeBuckets

This is a BTree family that’s useful for high-concurrency or large applications by increasing the size of individual buckets. This reduces the number of ZODB fetches required, and also reduces the amount of bucket splitting required on write-heavy workloads, thus reducing unresolvable conflicts. For example, an OOBTree defaults to a bucket size of 60, while this family uses 500. This can be used anywhere BTrees are used, for example in containers.

zope.index provides a particularly motivating example.

Underlying the indices are the same BTree data structures as under any zope.container.btree.BTreeContainer. They’re typically intid, based, though, so instead of the default limit of 60 items in a bucket that applies for an OO tree, they would have different limits. For the 64-bit IO tree, that limit is…also 60. (Any BTree with a n ‘O’ in its name defaults to 60.)

Most indices maintain two BTrees:

forward: OO = indexed value to sequence of intids
backward: IO = intid to indexed value

The forward index values are usually either a ISet or ITreeSet; attribute indices always use TreeSet, but keyword indices transition between Set and TreeSet at a fixed threshold. Sets should basically always be able to resolve conflicts (no splits to deal with) but will conflict on every addition (and of course have unbounded growth in pickle size). TreeSets may occasionally have un-resolvable conflicts, but any given insert is less likely to conflict because they’re spread across buckets.

Just as for BTreeContainers, using a BTree with increased bucket size, applied to the forward, backward, and sequence values, can serve to reduce conflicts. Since all the index constructors take a ‘family’ argument that’s used to determine the type of their forward, backward and sequence attributes, supplying the family64LargeBuckets will accomplish this.

Why wasn’t this done by default? Most importantly, the ability to customize those aspects of BTree sizes is a relatively recent addition (within the last few years), well after zope.index and zope.catalog were written.

Why use such small sizes for the defaults? My guess is that it’s a product of the time BTrees were designed: smaller disks, smaller memories, slower networks -> optimize for space and transfer time with smaller pickles.

However, there are some caveats.

First, the pickles for individual buckets will be larger by a factor of 8 (for an OO object). While there will be 8x fewer ZODB fetches required to traverse a complete tree, unpickling a single bucket could be up to 8x slower. If you store large keys or values, that may be a concern.

Conflict resolution in a bucket is O(n), so resolving conflicts will take some time longer.

The ZODB cache is configured by number of objects, so if the total number of BTree buckets goes down, then there’s space for some more objects. This is somewhat offset by the fact that fewer buckets means less overhead and thus slightly less memory usage. We’re talking about a 8x reduction of buckets; exactly how much difference that makes depends on what proportion of the cache is currently buckets.

The RelStorage cache is measured in bytes; overall pickle sizes should also be about the same, just slightly smaller.