Reference¶
Interfaces¶
Interfaces for objects defined in the ZODB package.
-
interface
nti.zodb.interfaces.
INumericCounter
¶ Extends:
nti.zodb.interfaces.INumericValue
A counter that can be incremented. Conflicts are resolved by merging the numeric value of the difference in magnitude of changes. Intended to be used for monotonically increasing counters, typically integers.
-
interface
nti.zodb.interfaces.
INumericValue
¶ Extends:
zope.minmax.interfaces.IAbstractValue
A persistent numeric value with conflict resolution.
-
set
(value)¶ Change the value of this object to the given value.
If the number is immutable, and the value is not the current value, this may raise
NotImplementedError
.
-
value
¶ The numeric value of this object.
Implementation: nti.schema.field.Number
Read Only: False Required: True Default Value: None Allowed Type: numbers.Number
-
increment
(amount=1)¶ Increment the value by the specified amount (which should be non-negative).
Returns: The counter with the incremented value (this object).
-
__gt__
(other)¶ These values are ordered like their values.
-
__hash__
()¶ This object hashes like its value.
Caution
Do not place this object in a hash container and then mutate the value.
-
__lt__
(other)¶ These objects are ordered like their values.
-
__eq__
(other)¶ Is this object holding a value numerically equal to the other?
-
-
interface
nti.zodb.interfaces.
ITokenBucket
¶ A token bucket is used in rate limiting applications. It has a maximum capacity and a rate at which tokens are regenerated (typically this is in terms of absolute time).
Clients attempt to consume tokens and are either allowed or denied based on whether there are enough tokens in the bucket.
-
tokens
¶ The current number of tokens in the bucket at this instant.
Implementation: zope.schema.Float
Read Only: False Required: True Default Value: None Allowed Type: float
-
fill_rate
¶ The rate in tokens per second at which new tokens arrive.
Implementation: zope.schema.Float
Read Only: False Required: True Default Value: 1.0 Allowed Type: float
-
wait_for_token
()¶ Consume a single whole token from the bucket, blocking until one is available if need be. This is not meant to be used from multiple threads.
-
capacity
¶ The maximum capacity of the token bucket.
Implementation: zope.schema.Float
Read Only: False Required: True Default Value: None Allowed Type: float
-
consume
(tokens=1)¶ Consume tokens from the bucket.
Parameters: tokens – The fractional number of tokens to consume. The default is to consume one whole token, which is probably what you want. Returns: True if there were sufficient tokens, otherwise False. If True, then the value of tokens
will have been reduced.
-
-
exception
nti.zodb.interfaces.
UnableToAcquireCommitLock
¶ Bases:
ZODB.POSException.StorageError
A ZODB storage (typically RelStorage) was unable to acquire the required commit lock.
This class is only used if RelStorage is not available; otherwise it is an alias for
relstorage.adapters.interfaces.UnableToAcquireCommitLock
.
-
nti.zodb.interfaces.
ZODBUnableToAcquireCommitLock
¶
Utilities¶
Common helpers.
-
nti.zodb.
isBroken
(obj, uid=None)¶ Check if the object is broken (missing or an implementation of
ZODB.interfaces.IBroken
).Parameters: uid (str) – A debugging aid, unless the obj is none, in which case it must be non-None to result in a True return. This makes no sense, so try to avoid passing objects that are None to this function.
-
nti.zodb.
is_broken
(obj, uid=None)¶ Check if the object is broken (missing or an implementation of
ZODB.interfaces.IBroken
).Parameters: uid (str) – A debugging aid, unless the obj is none, in which case it must be non-None to result in a True return. This makes no sense, so try to avoid passing objects that are None to this function.
-
nti.zodb.
readCurrent
(obj, container=True)¶ Persistence safe wrapper around zodb connection readCurrent; also has some built in smarts about typical objects that need to be read together.
Activity Log¶
Logging of database connection activity. Activate this with ZCML:
<include package="nti.zodb" file="configure_activitylog.zcml" />
Originally based on code from the unreleased zc.zodbactivitylog.
-
nti.zodb.activitylog.
AbstractActivityMonitor
¶
-
class
nti.zodb.activitylog.
ActivityMonitorComponent
¶ Bases:
object
One component of a chain of activity monitors.
-
class
nti.zodb.activitylog.
ActivityMonitorData
(loads, stores, db_name, pool_all_count, pool_idle_count)¶ Bases:
tuple
Create new instance of ActivityMonitorData(loads, stores, db_name, pool_all_count, pool_idle_count)
-
db_name
¶ Alias for field number 2
-
loads
¶ Alias for field number 0
-
pool_all_count
¶ Alias for field number 3
-
pool_idle_count
¶ Alias for field number 4
-
stores
¶ Alias for field number 1
-
-
class
nti.zodb.activitylog.
ComponentActivityMonitor
(base=None, components=())¶ Bases:
object
Activity monitor that:
- Can call another activity monitor; this is useful for adding on to an existing activity monitor.
- Can call a sequence of callables with some data. This is useful for composing many activity monitors much more cheaply than if they were on their own.
Parameters: - base – An arbitrary activity monitor.
- components – An iterable of
ActivityMonitorComponent
instances (callables taking aActivityMonitorData
).
-
class
nti.zodb.activitylog.
LogActivityComponent
¶ Bases:
nti.zodb.activitylog.ActivityMonitorComponent
An activity monitor component that logs connection transfer information and pool information.
Changed in version 1.3.0: Add
min_loads
andmin_stores
. These additional thresholds are tested in addition tomin_loads_and_stores
and if any threshold is reached the logging happens. The default value of each threshold is 10.The thresholds may be configured in the environment (before loading this class) as integer strings using the values
NTI_ZODB_LOG_MIN_LOADS
,NTI_ZODB_LOG_MIN_STORES
, andNTI_ZODB_LOG_MIN_ACTIVITY
, respectively.
-
class
nti.zodb.activitylog.
LogActivityMonitor
(base=None)¶ Bases:
nti.zodb.activitylog.ComponentActivityMonitor
A pre-configured
ComponentActivityMonitor
that usesLogActivityComponent()
-
class
nti.zodb.activitylog.
StatsdActivityComponent
¶ Bases:
nti.zodb.activitylog.ActivityMonitorComponent
An activity monitor component that stores counters (guages) in statsd, if the statsd client is available.
The stats are:
- ZODB.DB.<name>.loads How many loads the open connection performed.
- ZODB.DB.<name>.stores How many stores the open connection performed.
- ZODB.DB.<name>.total_connections All currently open connections, including those in use and those in the pool.
- ZODB.DB.<name>.available_connections The connections sitting idle in the pool.
-
static
statsd_client
()¶ Return the current StatsdClient for the thread.
Returns the thread-local client if there is one, or the global client if there is one, or None.
-
class
nti.zodb.activitylog.
StatsdActivityMonitor
(base=None)¶ Bases:
nti.zodb.activitylog.ComponentActivityMonitor
A pre-configured
ComponentActivityMonitor
that usesLogActivityComponent()
-
nti.zodb.activitylog.
register_subscriber
(event)¶ Subscriber to the
zope.processlifetime.IDatabaseOpenedWithRoot
that registers an activity monitor.
Containers¶
Utilities for working with containers, particularly BTree containers.
-
nti.zodb.containers.
bit64_int_to_time
(value)¶ Convert a 64 bit integer to its floating point value.
Inverse of
time_to_64bit_int()
.
-
nti.zodb.containers.
time_to_64bit_int
(value)¶ Given a Python floating point object (usually a time value), losslessly return a 64-bit long int that represents it. Useful for storing as the value in a OL tree, when you really want a float (since BTrees does not provide OF), or as a key in a Lx tree.
Copy¶
Things to assist with copying persistent objects.
-
nti.zodb.copy.
wref_copy_factory
(ref)¶ Weak references cannot typically be copied due to the presence of the Connection attribute (in the dm value). This factory makes them copyable.
Currently we assume that the reference can be resolved at copy time (since we cannot create a reference to None).
MinMax¶
Conflict resolving value/counter implementations for use on persistent objects.
-
class
nti.zodb.minmax.
AbstractNumericValue
(value=0)¶ Bases:
zope.minmax._minmax.AbstractValue
A numeric value that provides ordering operations. Defaults to zero.
-
nti.zodb.minmax.
ConstantZeroValue
(value=0)¶ Use this as a class attribute for a default values of zero. The value cannot be changed, and instances cannot be serialized.
-
class
nti.zodb.minmax.
MergingCounter
(value=0)¶ Bases:
nti.zodb.minmax.AbstractNumericValue
A
zope.minmax
item that resolves conflicts by merging the numeric value of the difference in magnitude of changes. Intented to be used for monotonically increasing counters.As a special case, if the counter is reset to zero by both transactions, that becomes the new state.
Changed in version 1.2.0: Special case setting the counter to zero.
-
class
nti.zodb.minmax.
NumericMaximum
(value=0)¶ Bases:
nti.zodb.minmax.AbstractNumericValue
,zope.minmax._minmax.Maximum
Maximizes the number during conflicts.
-
class
nti.zodb.minmax.
NumericMinimum
(value=0)¶ Bases:
nti.zodb.minmax.AbstractNumericValue
,zope.minmax._minmax.Minimum
Minimizes the number during conflicts.
-
class
nti.zodb.minmax.
NumericPropertyDefaultingToZero
(name, factory, as_number=False)¶ Bases:
nti.zodb.persistentproperty.PropertyHoldingPersistent
In persistent objects (that extend
nti.zodb.persistentproperty.PersistentPropertyHolder
), use this to hold a merging counter or numeric minimum or maximum.This is a data descriptor:
class Foo(PersistentPropertyHolder): a = NumericPropertyDefaultingToZero('a')
Creates a new property in a new-style class that does not use
__slots__
(persistent classes shouldn’t use__slots__
anyway).Parameters: - name (str) – The name of the property; this will be the key in the instance
dictionary. This should match the name of the property
(e.g.,
a = NumericPropertyDefaultingToZero( 'a',...)
) but is not required to. It must be a native string (bytes on py2, str/unicode on py3). - factory (callable) – The value object factory that determines the type of
conflict resolution used for this property. Typically
NumericMaximum()
,NumericMinimum
orMergingCounter
. - as_number (bool) – If set to
True
(not the default), then when an instance reads this property, the numeric value will be returned; otherwise thefactory
class instance will be returned and you will want to access its.value
attribute. Setting this property always takes the (raw) numeric value.
-
class
IncrementingZeroValue
(name, holder, prop)¶ Bases:
nti.zodb.minmax._ConstantZeroValue
- name (str) – The name of the property; this will be the key in the instance
dictionary. This should match the name of the property
(e.g.,
Persistent Property¶
Classes for making property
objects (actually, general descriptors)
more convenient for working with in persistent.Persistent
objects.
-
class
nti.zodb.persistentproperty.
PersistentPropertyHolder
¶ Bases:
persistent.Persistent
Lets you assign to a property without necessarily changing the
_p_status
of this object.In a subclass of
persistent.Persistent
, the__setattr__
method sets_p_changed
to True when called with aname
argument that does not start with_p_
(properties of the persistent object itself) or_v_
(volatile properties). This makes it hard to use with conflict-reducing objects likenti.zodb.minmax.NumericMaximum
: instead of being able to define a descriptor to access and mutate them directly, you must remember to go through their API, and replacing existing simple attributes (a plain number) with a property doesn’t actually reduce conflicts until all callers have been updated to use the API.This superclass fixes that problem. When
__setattr__()
is called, it checks to see if the underlying attribute is actually a descriptor extendingPropertyHoldingPersistent
, and if so, delegates directly to that object. That object is responsible for managing the persistent state of that instance.Caution
When you subclass this, you should not modify the type after the first instance is constructed by adding new
PropertyHoldingPersistent
instances. As an implementation note, the__new__
method caches the properties that arePropertyHoldingPersistent
. Adding new ones will bypass the cache (and make the instance modified) but otherwise still behave correctly. Replacing one with a different type of property or deleting the property altogether may not function correctly.
-
class
nti.zodb.persistentproperty.
PropertyHoldingPersistent
¶ Bases:
object
Base class mixin for a property that, when installed in a
PersistentPropertyHolder
, can be used to hold another persistent object. This property object takes all responsibility for changing persistent state (of the instance it is installed in) if needed.
Token Bucket¶
Implementations of the token bucket algorithm.
-
class
nti.zodb.tokenbucket.
PersistentTokenBucket
(capacity, fill_rate=1.0)¶ Bases:
object
Persistent implementation of the token bucket algorithm. If the ZODB is used from multiple machines, relies on their clocks being relatively synchronized to be effective.
Initially based on an ActiveState recipe
(For efficiency, this object itself isn’t persistent, but the objects it holds are.)
Creates a new token bucket, initially full.
Parameters: - capacity – The max number of tokens in the bucket (also the initial number of tokens in the bucket.
- fill_rate – The rate in fractional tokens per second that the bucket will fill. The default is to add one token per second.
-
consume
(tokens=1)¶ Consume one or more tokens from the bucket. Returns True if there were sufficient tokens otherwise False.
-
tokens
¶ The fractional number of tokens currently in the bucket.
-
wait_for_token
()¶ Consume a single token from the bucket, blocking until one is available if need be.
Weak References¶
Utilities for weak references to persistent objects.
-
class
nti.zodb.wref.
CopyingWeakRef
(ob)¶ Bases:
persistent.wref.WeakRef
A weak ref that also stores a one-shot copy (using
zope.copy.copy()
) of its reference, as a fallback to return from__call__()
if the weak reference cannot be resolved.Use this if the object is expected to mutate after this reference is established and you normally want to have access to those changes, and you cannot tolerate the object going missing, but you also cannot keep a strong reference to the object. This may be particularly the case in cross-database refs.
Implements
nti.wref.interfaces.ICachingWeakRef
.
ZlibStorage¶
Utilities for working with zc.zlibstorage.
TODO: Add support for zodburi, which can be installed automatically as entry points in setup.py.
-
class
nti.zodb.zlibstorage.
ZlibStorageClientStorageURIResolver
¶ Bases:
repoze.zodbconn.resolvers.ClientStorageURIResolver
Wraps
ZEO.ClientStorage.ClientStorage
with zc.zlibstorage when using thezlibzeo
URI scheme.
-
class
nti.zodb.zlibstorage.
ZlibStorageFileStorageURIResolver
¶ Bases:
repoze.zodbconn.resolvers.FileStorageURIResolver
Wraps
ZODB.FileStorage.FileStorage
with zc.zlibstorage when using thezlibfile
URI scheme.
-
nti.zodb.zlibstorage.
install_zlib_client_resolver
()¶ Makes it possible for
repoze.zodbconn.uri.db_from_uri()
to connect to ZEO servers that are using zlib storage, through providing support for the use of thezlibzeo
URI scheme, and likewise forzlibfile://
.Alternatively, you can use a ZConfig file and the zconfig:// URI scheme.
BTrees¶
Support for using BTrees
with ZODB.
This module is primarily concerned with making BTrees even more efficient, especially at large scale and high concurrency.
family64LargeBuckets
This is a BTree family that’s useful for high-concurrency or large
applications by increasing the size of individual buckets. This
reduces the number of ZODB fetches required, and also reduces the
amount of bucket splitting required on write-heavy workloads, thus
reducing unresolvable conflicts. For example, an OOBTree
defaults
to a bucket size of 60, while this family uses 500. This can be used
anywhere BTrees are used, for example in containers.
zope.index
provides a particularly motivating example.
Underlying the indices are the same BTree data structures as under any
zope.container.btree.BTreeContainer
. They’re typically intid,
based, though, so instead of the default limit of 60 items in a bucket
that applies for an OO tree, they would have different limits. For the
64-bit IO tree, that limit is…also 60. (Any BTree with a n ‘O’ in
its name defaults to 60.)
Most indices maintain two BTrees:
forward: OO = indexed value to sequence of intids
backward: IO = intid to indexed value
The forward index values are usually either a ISet or ITreeSet; attribute indices always use TreeSet, but keyword indices transition between Set and TreeSet at a fixed threshold. Sets should basically always be able to resolve conflicts (no splits to deal with) but will conflict on every addition (and of course have unbounded growth in pickle size). TreeSets may occasionally have un-resolvable conflicts, but any given insert is less likely to conflict because they’re spread across buckets.
Just as for BTreeContainers
, using a BTree with increased bucket
size, applied to the forward, backward, and sequence values, can serve
to reduce conflicts. Since all the index constructors take a ‘family’
argument that’s used to determine the type of their forward, backward
and sequence attributes, supplying the family64LargeBuckets
will
accomplish this.
Why wasn’t this done by default? Most importantly, the ability to customize those aspects of BTree sizes is a relatively recent addition (within the last few years), well after zope.index and zope.catalog were written.
Why use such small sizes for the defaults? My guess is that it’s a product of the time BTrees were designed: smaller disks, smaller memories, slower networks -> optimize for space and transfer time with smaller pickles.
However, there are some caveats.
First, the pickles for individual buckets will be larger by a factor of 8 (for an OO object). While there will be 8x fewer ZODB fetches required to traverse a complete tree, unpickling a single bucket could be up to 8x slower. If you store large keys or values, that may be a concern.
Conflict resolution in a bucket is O(n)
, so resolving conflicts
will take some time longer.
The ZODB cache is configured by number of objects, so if the total number of BTree buckets goes down, then there’s space for some more objects. This is somewhat offset by the fact that fewer buckets means less overhead and thus slightly less memory usage. We’re talking about a 8x reduction of buckets; exactly how much difference that makes depends on what proportion of the cache is currently buckets.
The RelStorage cache is measured in bytes; overall pickle sizes should also be about the same, just slightly smaller.