Reference

Interfaces

Interfaces for objects defined in the ZODB package.

interface nti.zodb.interfaces.ITokenBucket

A token bucket is used in rate limiting applications. It has a maximum capacity and a rate at which tokens are regenerated (typically this is in terms of absolute time).

Clients attempt to consume tokens and are either allowed or denied based on whether there are enough tokens in the bucket.

fill_rate

The rate in tokens per second at which new tokens arrive.

Implementation:

zope.schema.Float

Read Only:

False

Required:

True

Default Value:

1.0

Allowed Type:

float

capacity

The maximum capacity of the token bucket.

Implementation:

zope.schema.Float

Read Only:

False

Required:

True

Default Value:

None

Allowed Type:

float

tokens

The current number of tokens in the bucket at this instant.

Implementation:

zope.schema.Float

Read Only:

False

Required:

True

Default Value:

None

Allowed Type:

float

consume(tokens=1)

Consume tokens from the bucket.

Parameters:

tokens – The fractional number of tokens to consume. The default is to consume one whole token, which is probably what you want.

Returns:

True if there were sufficient tokens, otherwise False. If True, then the value of tokens will have been reduced.

wait_for_token()

Consume a single whole token from the bucket, blocking until one is available if need be. This is not meant to be used from multiple threads.

interface nti.zodb.interfaces.INumericValue

Extends: zope.minmax.interfaces.IAbstractValue

A persistent numeric value with conflict resolution.

value

The numeric value of this object.

Implementation:

nti.schema.field.Number

Read Only:

False

Required:

True

Default Value:

None

Allowed Type:

numbers.Number

set(value)

Change the value of this object to the given value.

If the number is immutable, and the value is not the current value, this may raise NotImplementedError.

__eq__(other)

Is this object holding a value numerically equal to the other?

__hash__()

This object hashes like its value.

Caution

Do not place this object in a hash container and then mutate the value.

__lt__(other)

These objects are ordered like their values.

__gt__(other)

These values are ordered like their values.

increment(amount=1)

Increment the value by the specified amount (which should be non-negative).

Returns:

The counter with the incremented value (this object).

interface nti.zodb.interfaces.INumericCounter

Extends: nti.zodb.interfaces.INumericValue

A counter that can be incremented. Conflicts are resolved by merging the numeric value of the difference in magnitude of changes. Intended to be used for monotonically increasing counters, typically integers.

exception nti.zodb.interfaces.UnableToAcquireCommitLock

Bases: StorageError

A ZODB storage (typically RelStorage) was unable to acquire the required commit lock.

This class is only used if RelStorage is not available; otherwise it is an alias for relstorage.adapters.interfaces.UnableToAcquireCommitLock.

nti.zodb.interfaces.ZODBUnableToAcquireCommitLock

alias of UnableToAcquireCommitLock

interface nti.zodb.interfaces.IZODBProvider

Provides zero or more ZODB ZODB.interfaces.IDatabase objects to be registered as global components.

This is a low-level interface that most people won’t implement. Instead, see IZODBConfigProvider.

When this is implemented, it is intended to be registered as a global named utility. Some startup process is invoked to find all such utilities, invoke them, and register the resulting databases as global components. The API nti.zodb.config_providers.provideDatabases() is provided for this purpose.

Added in version 4.3.0.

getDatabases()

Return a mapping of {name: (database, discriminator)} objects.

Each database is meant to be registered as a global component with the given name.

Equal discriminator values imply equal database values, and such equal values may be collapsed into a single database value. If you cannot determine discriminators, you may use a simple object, but this will defeat the ability to detect multiples.

interface nti.zodb.interfaces.IZODBConfigProvider

Provides a ZODB configuration suitable for creating ZODB database object.

The format of this configuration, and how it is accessed, is not defined. You are expected to provide an adapter from (a sub-interface of) this interface to ZODB.interfaces.IDatabase.

This package provides an implementation of IZODBProvider that looks for named utilities of this interface, and adapts them to the databases to be returned from getDatabases.

Important

For these utilities to be found, you must be sure to register them to provide exactly this interface, not the sub-interface they actually implement.

This package provides one sub-interface, and the appropriate adapter; see IZODBZConfigProvider

Added in version 4.3.0.

getDiscriminator()

Return a hashable object that uniquely represents the database configuration this object holds.

The intent is that equivalent databases can be detected this way and collapsed into a single entry. In this way we can support aliases.

The return value is typically a string. If you are unable to create a unique discriminator, you may return an arbitrary hashable object.

Added in version 4.4.0.

interface nti.zodb.interfaces.IZODBZConfigProvider

Extends: nti.zodb.interfaces.IZODBConfigProvider

This package provides an implementation of this interface for a temporary (in-memory) database in config_providers as well as an implementation of the registration step in ().

By default, the temporary database config provider is registered in the configure_configprovider.zcml file with the name “mtemp”. No functionality in this package requires this, and you can use zope.configuration overrides to cancel this registration.

The discriminator should typically be the same as the ZConfig string, assuming that it is produced deterministically.

getZConfigString()

Return the ZConfig string to be passed to ZODB.config.databaseFromString().

Utilities

Common helpers.

nti.zodb.readCurrent(obj, container=True)

Persistence safe wrapper around zodb connection readCurrent; also has some built in smarts about typical objects that need to be read together.

nti.zodb.isBroken(obj, uid=None)

Check if the object is broken (missing or an implementation of ZODB.interfaces.IBroken).

Parameters:

uid (str) – A debugging aid, unless the obj is none, in which case it must be non-None to result in a True return. This makes no sense, so try to avoid passing objects that are None to this function.

nti.zodb.is_broken(obj, uid=None)

Check if the object is broken (missing or an implementation of ZODB.interfaces.IBroken).

Parameters:

uid (str) – A debugging aid, unless the obj is none, in which case it must be non-None to result in a True return. This makes no sense, so try to avoid passing objects that are None to this function.

Database Configuration

Implementations of, and helpers for, IZODBZConfigProvider

Added in version 4.3.0.

class nti.zodb.config_providers.DatabaseProvider

Bases: object

An aggregate IZODBProvider implementation based on all registered IZODBProvider objects.

Important

This object should not be registered in a component registry. Instead, use the module-level provideDatabases function.

If we find only a single database, then it is registered both under its own name and as the global default database, unless the value of REGISTER_SINGLE_AS_DEFAULT` is set to false (through an environment variable often).

If we register a database with the default name, whether because we were configured that way or because we copied a single named registration, the event zope.processlifetime.DatabaseOpened event will be sent for that database.

All databases found are registered with each other as a single multi-database. This includes existing databases.

Raises:

ValueError – If a database name is duplicated, either by providers or with existing databases.

Simple Usage

The typical use will look something like this. You’ll need:

• One or more named registered IZODBConfigProvider utilities. It is important it is registered for exactly this interface. If you can express your configuration in ZConfig (most configuration can be expressed that way), implement IZODBZConfigProvider.

• A registered IZODBConfigProvider to IDatabase adapter. If you implement IZODBZConfigProvider, this is already given.

• To call provideDatabases() during your process startup, after component registration.

You may also choose to just register one or more direct implementations of IZODBProvider if you cannot express your database any other way.

getDatabases()

See the documentation for DatabaseProvider.

Changed in version 4.4.0: Return the mapping of databases as expected.

nti.zodb.config_providers.provideDatabases()

See the documentation for DatabaseProvider.

Changed in version 4.4.0: Return the mapping of databases as expected.

class nti.zodb.config_providers.ZODBConfigProviderDBProvider

Bases: object

Uses registered IZODBConfigProvider utilities to implement IZODBProvider.

class nti.zodb.config_providers.InMemoryDemoStorageZConfigProvider

Bases: object

Provides the ZConfig for a temporary database.

This database will be a DemoStorage on top of an in-memory storage. It does not support blobs.

Activity Log

Logging of database connection activity. Activate this with ZCML:

<include package="nti.zodb" file="configure_activitylog.zcml" />

Originally based on code from the unreleased zc.zodbactivitylog.

class nti.zodb.activitylog.ComponentActivityMonitor(base=None, components=())

Bases: object

Activity monitor that:

• Can call another activity monitor; this is useful for adding on to an existing activity monitor.

• Can call a sequence of callables with some data. This is useful for composing many activity monitors much more cheaply than if they were on their own.

Parameters:

• base – An arbitrary activity monitor.

• components – An iterable of ActivityMonitorComponent instances (callables taking a ActivityMonitorData).

nti.zodb.activitylog.AbstractActivityMonitor

alias of ComponentActivityMonitor

class nti.zodb.activitylog.ActivityMonitorData(loads, stores, db_name, pool_all_count, pool_idle_count)

Bases: tuple

The data passed to ActivityMonitorComponent

db_name

Alias for field number 2

loads

Alias for field number 0

pool_all_count

Alias for field number 3

pool_idle_count

Alias for field number 4

stores

Alias for field number 1

class nti.zodb.activitylog.ActivityMonitorComponent

Bases: object

One component of a chain of activity monitors.

class nti.zodb.activitylog.LogActivityComponent

Bases: ActivityMonitorComponent

An activity monitor component that logs connection transfer information and pool information.

Changed in version 1.3.0: Add min_loads and min_stores. These additional thresholds are tested in addition to min_loads_and_stores and if any threshold is reached the logging happens. The default value of each threshold is 10.

The thresholds may be configured in the environment (before loading this class) as integer strings using the values NTI_ZODB_LOG_MIN_LOADS, NTI_ZODB_LOG_MIN_STORES, and NTI_ZODB_LOG_MIN_ACTIVITY, respectively.

min_loads_and_stores

Perform logging if the total of loads + stores is at least this many.

min_loads

Perform logging if the number of loads is at least this many.

min_stores

Perform logging if the number of stores is at least this many.

class nti.zodb.activitylog.LogActivityMonitor(base=None)

Bases: ComponentActivityMonitor

A pre-configured ComponentActivityMonitor that uses LogActivityComponent()

Parameters:

• base – An arbitrary activity monitor.

• components – An iterable of ActivityMonitorComponent instances (callables taking a ActivityMonitorData).

class nti.zodb.activitylog.StatsdActivityComponent

Bases: ActivityMonitorComponent

An activity monitor component that stores counters (guages) in statsd, if the statsd client is available.

The stats are:

• ZODB.DB.<name>.loads How many loads the open connection performed.

• ZODB.DB.<name>.stores How many stores the open connection performed.

• ZODB.DB.<name>.total_connections All currently open connections, including those in use and those in the pool.

• ZODB.DB.<name>.available_connections The connections sitting idle in the pool.

static statsd_client()

Return the current StatsdClient for the thread.

Returns the thread-local client if there is one, or the global client if there is one, or None.

class nti.zodb.activitylog.StatsdActivityMonitor(base=None)

Bases: ComponentActivityMonitor

A pre-configured ComponentActivityMonitor that uses LogActivityComponent()

Parameters:

• base – An arbitrary activity monitor.

• components – An iterable of ActivityMonitorComponent instances (callables taking a ActivityMonitorData).

nti.zodb.activitylog.register_subscriber(event)

Subscriber to the zope.processlifetime.IDatabaseOpenedWithRoot that registers an activity monitor.

Containers

Utilities for working with containers, particularly BTree containers.

nti.zodb.containers.time_to_64bit_int(value)

Given a Python floating point object (usually a time value), losslessly return a 64-bit long int that represents it. Useful for storing as the value in a OL tree, when you really want a float (since BTrees does not provide OF), or as a key in a Lx tree.

nti.zodb.containers.bit64_int_to_time(value)

Convert a 64 bit integer to its floating point value.

Inverse of time_to_64bit_int().

class nti.zodb.containers.CaseInsensitiveKey(key)

Bases: object

This class implements a dictionary key that preserves case, but compares case-insensitively. It works with unicode keys only (BTrees do not work if 8-bit and unicode are mixed).

This is a bit of a heavyweight solution. It is nonetheless optimized for comparisons only with other objects of its same type.

It must not be subclassed.

Added in version 4.2.0.

property comp_key

Returns the key to compare.

class nti.zodb.containers.CaseInsensitiveBTreeContainer

Bases: _CheckObjectOnSetMixin, BTreeContainer

A BTreeContainer that only works with string (unicode) keys, and treats them in a case-insensitive fashion. The original case of the key entered is preserved.

The underlying BTree is configured for large amounts of data.

Added in version 4.2.0.

get(key, default=None)

See interface IReadContainer

clear()

Convenience method to clear the entire tree at one time.

class nti.zodb.containers.CaseInsensitiveFolder

Bases: CaseInsensitiveBTreeContainer, Folder

A BTree folder that preserves case but otherwise ignores it.

Copy

Things to assist with copying persistent objects.

nti.zodb.copy.wref_copy_factory(ref)

Weak references cannot typically be copied due to the presence of the Connection attribute (in the dm value). This factory makes them copyable.

Currently we assume that the reference can be resolved at copy time (since we cannot create a reference to None).

MinMax

Conflict resolving value/counter implementations for use on persistent objects.

class nti.zodb.minmax.AbstractNumericValue(value=0)

Bases: AbstractValue

A numeric value that provides ordering operations. Defaults to zero.

nti.zodb.minmax.ConstantZeroValue(value=0)

Use this as a class attribute for a default values of zero. The value cannot be changed, and instances cannot be serialized.

class nti.zodb.minmax.NumericMaximum(value=0)

Bases: AbstractNumericValue, Maximum

Maximizes the number during conflicts.

class nti.zodb.minmax.NumericMinimum(value=0)

Bases: AbstractNumericValue, Minimum

Minimizes the number during conflicts.

class nti.zodb.minmax.MergingCounter(value=0)

Bases: AbstractNumericValue

A zope.minmax item that resolves conflicts by merging the numeric value of the difference in magnitude of changes. Intented to be used for monotonically increasing counters.

As a special case, if the counter is reset to zero by both transactions, that becomes the new state.

Changed in version 1.2.0: Special case setting the counter to zero.

class nti.zodb.minmax.NumericPropertyDefaultingToZero(name, factory, as_number=False)

Bases: PropertyHoldingPersistent

In persistent objects (that extend nti.zodb.persistentproperty.PersistentPropertyHolder), use this to hold a merging counter or numeric minimum or maximum.

This is a data descriptor:

class Foo(PersistentPropertyHolder):

   a = NumericPropertyDefaultingToZero('a')

Creates a new property in a new-style class that does not use __slots__ (persistent classes shouldn’t use __slots__ anyway).

Parameters:

• name (str) – The name of the property; this will be the key in the instance dictionary. This should match the name of the property (e.g., a = NumericPropertyDefaultingToZero( 'a',...)) but is not required to. It must be a native string (bytes on py2, str/unicode on py3).

• factory (callable) – The value object factory that determines the type of conflict resolution used for this property. Typically NumericMaximum(), NumericMinimum or MergingCounter.

• as_number (bool) – If set to True (not the default), then when an instance reads this property, the numeric value will be returned; otherwise the factory class instance will be returned and you will want to access its .value attribute. Setting this property always takes the (raw) numeric value.

class IncrementingZeroValue(name, holder, prop)

Bases: _ConstantZeroValue

Persistent Property

Classes for making property objects (actually, general descriptors) more convenient for working with in persistent.Persistent objects.

class nti.zodb.persistentproperty.PropertyHoldingPersistent

Bases: object

Base class mixin for a property that, when installed in a PersistentPropertyHolder, can be used to hold another persistent object. This property object takes all responsibility for changing persistent state (of the instance it is installed in) if needed.

class nti.zodb.persistentproperty.PersistentPropertyHolder(*args, **kwargs)

Bases: Persistent

Lets you assign to a property without necessarily changing the _p_status of this object.

In a subclass of persistent.Persistent, the __setattr__ method sets _p_changed to True when called with a name argument that does not start with _p_ (properties of the persistent object itself) or _v_ (volatile properties). This makes it hard to use with conflict-reducing objects like nti.zodb.minmax.NumericMaximum: instead of being able to define a descriptor to access and mutate them directly, you must remember to go through their API, and replacing existing simple attributes (a plain number) with a property doesn’t actually reduce conflicts until all callers have been updated to use the API.

This superclass fixes that problem. When __setattr__() is called, it checks to see if the underlying attribute is actually a descriptor extending PropertyHoldingPersistent, and if so, delegates directly to that object. That object is responsible for managing the persistent state of that instance.

Caution

When you subclass this, you should not modify the type after the first instance is constructed by adding new PropertyHoldingPersistent instances. As an implementation note, the __new__ method caches the properties that are PropertyHoldingPersistent. Adding new ones will bypass the cache (and make the instance modified) but otherwise still behave correctly. Replacing one with a different type of property or deleting the property altogether may not function correctly.

Token Bucket

Implementations of the token bucket algorithm.

class nti.zodb.tokenbucket.PersistentTokenBucket(capacity, fill_rate=1.0)

Bases: object

Persistent implementation of the token bucket algorithm. If the ZODB is used from multiple machines, relies on their clocks being relatively synchronized to be effective.

Initially based on an ActiveState recipe

(For efficiency, this object itself isn’t persistent, but the objects it holds are.)

Creates a new token bucket, initially full.

Parameters:

• capacity – The max number of tokens in the bucket (also the initial number of tokens in the bucket.

• fill_rate – The rate in fractional tokens per second that the bucket will fill. The default is to add one token per second.

consume(tokens=1)

Consume one or more tokens from the bucket. Returns True if there were sufficient tokens otherwise False.

wait_for_token()

Consume a single token from the bucket, blocking until one is available if need be.

property tokens

The fractional number of tokens currently in the bucket.

Weak References

Utilities for weak references to persistent objects.

class nti.zodb.wref.CopyingWeakRef(ob)

Bases: WeakRef

A weak ref that also stores a one-shot copy (using zope.copy.copy()) of its reference, as a fallback to return from __call__() if the weak reference cannot be resolved.

Use this if the object is expected to mutate after this reference is established and you normally want to have access to those changes, and you cannot tolerate the object going missing, but you also cannot keep a strong reference to the object. This may be particularly the case in cross-database refs.

Implements nti.wref.interfaces.ICachingWeakRef.

ZlibStorage

Utilities for working with zc.zlibstorage.

TODO: Add support for zodburi, which can be installed automatically as entry points in setup.py.

class nti.zodb.zlibstorage.ZlibStorageClientStorageURIResolver

Bases: ClientStorageURIResolver

Wraps ZEO.ClientStorage.ClientStorage with zc.zlibstorage when using the zlibzeo URI scheme.

class nti.zodb.zlibstorage.ZlibStorageFileStorageURIResolver

Bases: FileStorageURIResolver

Wraps ZODB.FileStorage.FileStorage with zc.zlibstorage when using the zlibfile URI scheme.

nti.zodb.zlibstorage.install_zlib_client_resolver()

Makes it possible for repoze.zodbconn.uri.db_from_uri() to connect to ZEO servers that are using zlib storage, through providing support for the use of the zlibzeo URI scheme, and likewise for zlibfile://.

Alternatively, you can use a ZConfig file and the zconfig:// URI scheme.

BTrees

Support for using BTrees with ZODB.

This module is primarily concerned with making BTrees even more efficient, especially at large scale and high concurrency.

family64LargeBuckets

This is a BTree family that’s useful for high-concurrency or large applications by increasing the size of individual buckets. This reduces the number of ZODB fetches required, and also reduces the amount of bucket splitting required on write-heavy workloads, thus reducing unresolvable conflicts. For example, an OOBTree defaults to a bucket size of 60, while this family uses 500. This can be used anywhere BTrees are used, for example in containers.

zope.index provides a particularly motivating example.

Underlying the indices are the same BTree data structures as under any zope.container.btree.BTreeContainer. They’re typically intid, based, though, so instead of the default limit of 60 items in a bucket that applies for an OO tree, they would have different limits. For the 64-bit IO tree, that limit is…also 60. (Any BTree with a n ‘O’ in its name defaults to 60.)

Most indices maintain two BTrees:

forward: OO = indexed value to sequence of intids
backward: IO = intid to indexed value

The forward index values are usually either a ISet or ITreeSet; attribute indices always use TreeSet, but keyword indices transition between Set and TreeSet at a fixed threshold. Sets should basically always be able to resolve conflicts (no splits to deal with) but will conflict on every addition (and of course have unbounded growth in pickle size). TreeSets may occasionally have un-resolvable conflicts, but any given insert is less likely to conflict because they’re spread across buckets.

Just as for BTreeContainers, using a BTree with increased bucket size, applied to the forward, backward, and sequence values, can serve to reduce conflicts. Since all the index constructors take a ‘family’ argument that’s used to determine the type of their forward, backward and sequence attributes, supplying the family64LargeBuckets will accomplish this.

Why wasn’t this done by default? Most importantly, the ability to customize those aspects of BTree sizes is a relatively recent addition (within the last few years), well after zope.index and zope.catalog were written.

Why use such small sizes for the defaults? My guess is that it’s a product of the time BTrees were designed: smaller disks, smaller memories, slower networks -> optimize for space and transfer time with smaller pickles.

However, there are some caveats.

First, the pickles for individual buckets will be larger by a factor of 8 (for an OO object). While there will be 8x fewer ZODB fetches required to traverse a complete tree, unpickling a single bucket could be up to 8x slower. If you store large keys or values, that may be a concern.

Conflict resolution in a bucket is O(n), so resolving conflicts will take some time longer.

The ZODB cache is configured by number of objects, so if the total number of BTree buckets goes down, then there’s space for some more objects. This is somewhat offset by the fact that fewer buckets means less overhead and thus slightly less memory usage. We’re talking about a 8x reduction of buckets; exactly how much difference that makes depends on what proportion of the cache is currently buckets.

The RelStorage cache is measured in bytes; overall pickle sizes should also be about the same, just slightly smaller.

nti.zodb.btrees.MAX_LEAF_SIZE = 500

The value used for max_leaf_size in all BTree classes available in family64LargeBuckets.

nti.zodb.btrees.MAX_INTERNAL_SIZE = 500

The value used for max_internal_size in all BTree classes available in family64LargeBuckets

nti.zodb.btrees.family64LargeBuckets = <nti.zodb.btrees._Family64LargeBuckets object>

A BTree family (BTrees.Interfaces.IBTreeFamily) where all modules have BTree and TreeSet objects that use larger buckets than the default.