Operators and solvers for high-performance computing

The PyOperators package defines operators and solvers for high-performance computing. These operators are multi-dimensional functions that, if linear, behave like matrices with a sparse storage footprint. The chaining of the operations is optimised and controlled by a pool-based memory manager.

15 Oct 2014

  PyOperators 0.13

What’s new ?

• Python 3 support
• add settingerr context manager to contextually set error behaviour
• add environment variables PYOPERATORS_NO_MPI (to prevent importing mpi4py on supercomputers such as NERSC’s edison to avoid crashes on login nodes), PYOPERATORS_GC_NBYTES_THRESHOLD, PYOPERATORS_MEMORY_ALIGNMENT, PYOPERATORS_MEMORY_TOLERANCE and PYOPERATORS_VERBOSE (see config.py file)
• add helpers last, last_is_not, ilast and ilast_is_not, split
• improved ProxyOperator: operators in a proxy group may have different shapes, and can access the operator’s attributes
• Return algorithm instance in pcg ’s output
• add pool_threading context manager
• add MPI rank in strinfo
• in iterative algorithms, silence MPI processes with rank>0
• add timers to measure the time spent in Operator’s __call__ method and in MPI operations

API changes, deprecations:

• rename openmp_num_threads -> omp_num_threads
• deprecate distribute and distribute_slice
• change isclassattr calling sequence

Under the hood:

• common superclass for DiagonalOperator, DiagonalNumexprOperator and MaskOperator
• add Cartcomm communicator and Comm.Create_cart method in fake_mpi module
• morph MPIDistributionIdentityOperator into IdentityOperator is communicator size is 1
• allow implicit shape SparseBase subclasss
• update for scipy 0.14
• new decorator deprecated
• add setup.py’s clean command (contributed by G. Vaillant)
• automatic cythonization of files (contributed by G. Vaillant)

14 May 2014

  PyOperators 0.12

What’s new ?

• add a nbytes property, indicating the operator storage footprint.
• delete method for operators, to force deletion of an operator, alongside its associated operators. If the storage footprint of the deleted operators exceeds a certain threshold GC_NBYTES_THRESHOLD, a garbage collection is triggered.
• ProxyOperator for on-the-fly computations. In a group of proxy operators, only one actual operator is cached. The proxy operators inherit all the properties from the actual operators.
• add a context manager setting to change a global variable value.

Under the hood:

• Renamed operator flag ‘inplace_reduction’ -> ‘update_output’. And make it a regular flag, i.e. the ‘update_output’ is no more inferred from the direct method signature.
• allow generators in composite operators.

26 Feb 2014

  PyOperators 0.11

What’s new ?

• New rule manager: new mechanism to enable or disable simplification rules inside a with context. The rule trigger none is implemented and it inhibits all algebraic simplifications if set to True.
• API change: the multiplication sign * is now context-dependent: composition for linear operators and element-wise multiplication otherwise.
• New operator DenseBlockDiagonalOperator. Extra dimensions of arrays are interpreted as diagonal blocks. This operator is broadcastable:
  • against vectors: multiplying a DenseBlockDiagonalOperator initialised with an array of shape (L, 1, M, N) by a tensor of shape (P, N) gives a tensor of shape (L, P, M)
  • against other DenseBlockDiagonalOperators: multiplying two of them initialised with arrays of shapes (K, 1, M, N) and (L, N, P) gives a DenseBlockDiagonalOperator initialised with an array of shape (K, L, M, P)
• DenseOperator is now implicit shape and broadcastable: an M×N DenseOperator multiplied by a tensor of shape (P, Q, N) gives a tensor of shape (P, Q, M).
• Make asoperator return an explicit shape DiagonalOperator, DenseOperator or DenseBlockDiagonalOperator for arrays of dimensions 1, 2 or more if the keyword constant is not set to True.
• New operators: PowerOperator, ReciprocalOperator, SqrtOperator and SquareOperator.
• Helper functions:
  • reshape_broadcast to reshape an array to a broadcastable shape, by using 0-strides along the missing dimensions
  • broadcast_shapes to return the shape of the output obtained by broadcasting arrays of given shapes
  • add reverse option for first, first_is_not, ifirst and ifirst_is_not.
• Improved __str__ for non-linear operators.
• New ufunc abs2, which returns the square modulus of a complex input.
• DEPRECATED: use isscalarlike instead of isscalar.

Under the hood:

• Add PyOperatorsWarning and PyOperatorsDeprecationWarnings.
• Rename the decorators module -> flags.
• Move operator’s rules -> rules module.
• Add CopyOperator.

16 Jan 2014

  PyOperators 0.10

What’s new ?

• SparseOperator. The sparse storage can be anyone from the scipy.sparse package (except the LIL format, which is not suited for matrix-vector multiplication)
• Change binary rule priorities to favour specialized rules and subclasses
• Add mechanism to prevent CompositeOperator from morphing into their unique component. This mechanism allow GroupOperator to only have one operand
• Remove DiagonalNumexprNonSeparableOperator, it’s not possible to avoid calling get_data in __init__ and it complicates the broadcasting operators too much
• API change: the mask convention for PackOperator and UnpackOperator is changed. True means kept (similar to Fortran’s pack & unpack). Make PackOperator and UnpackOperator subclass BroadcastingBase

Under the hood:

• Handle infinity in assert_same
• Improve rule’s __str__ when the predicate in a lambda function
• Add ‘_reset’ method for Operators
• Add ‘broadcast’ keyword to strshape
• Add debugging for the composition and commutative rules
• Cleanup broadcasting operators, including shape and dtype. Improved testing
• Fix ‘square’ flag when morphing from a DiagonalOperator or a MaskOperator to ZeroOperator
• In ‘todense’, use dtype=int when the operator’s dtype is None
• Hack to add shapein and shapeout to Operator’s __repr__
• In CommutativeCompositeOperator, get info from the operands before applying the simplification rules
• Add ZeroOperator rule for MultiplicationOperator. Make sure a copy of the other operator is done for AdditionOperator
• In ‘uninterruptible’ contextmanager, ensure that the SIGINT handler is always put back in

14 Oct 2013

  PyOperators 0.9

What’s new ?

• Rotation2dOperator for rotations in a plane
• Rotation3dOperator, about 1 axis (3 conventions), 2 axes (6 conventions, including intrinsic and extrinsic rotations) or 3 axes (12 conventions)
• Spherical2CartesianOperator & Cartesian2SphericalOperator with conventions ‘zenith,azimuth’, ‘azimuth,zenith’, ‘elevation,azimuth’ and ‘azimuth,elevation’
• NormalizeOperator to obtain unit vectors
• DegreesOperator & RadiansOperator for degrees/radians conversions
• Implement __ne__ for Operators
• Add helper function float_dtype. Rename complex_dtype_for->complex_dtype
• Add helper functions one, pi and zero, which return a value of the specified dtype.
• API change in Operator’s rule:
  • in unary rules, use ‘C’, ‘T’, ‘H’ or ‘I’ instead of ‘.C’, ‘.T’, ‘.H’ or ‘.I’
  • binary rule subjects are now specified by a 2-tuple
  • use Operator subclass type instead of the string ‘{MyOperator}’
• API change: IntegrationTrapezeWeightOperator -> IntegrationTrapezeOperator

Under the hood:

• benchmark and memory_usage functions moved to a distinct distribution pybenchmarks
• Add test and coverage commands in setup.py
• Automatically compute adjoint from transpose, transpose from adjoint, inverse_adjoint from inverse_transpose and inverse_tranpose from inverse_adjoint
• Fix attribute copy during scalar absorption
• In binary rules, use ‘==’ instead of ‘is’ in subject matching
• In assert_same, allow broadcasting for integer types
• Remove obsolete Operator’s isalias method
• Make shape_input, shape_output and inplace_reduction non-settable flags
• Take forward FFTW instance dtype into account for the backward instance
• Remove DirectOperatorFactory and ReverseOperatorFactory
• DenseOperator output and input shapes can now be implicit. Don’t rely on this feature yet, it will be changed in future versions.