Motivation

NumPy has been very impactful in the Python ecosystem, providing efficient array representations and computations. It is still heavily used today and many recent Python libraries either use it internally or were influenced by its API.

However, with the explosion of machine learning and data analysis work happening in Python, many new libraries have implemented their own typed arrays and computation layers, because the NumPy model is not flexible enough. It is hard to extend NumPy with new types and get efficient computation on a variety of hardware devices. It can also be difficult to use just the type system of NumPy or just the computational infrastructure of NumPy.

We are building XND to recreate the foundations of NumPy as a number of smaller libraries, combining the lessons learned in the past twenty years of array computing in Python with the needs of newer applications. This is not a replacement of NumPy. Eventually, NumPy could use XND as could Pandas, Dask, and other libraries.

In fact, we are actively working on using XND in Numba and are also very interested in integrating it with a variety of libraries including Dask, xarray, Numba, Chainer, PyTorch, Tensorflow, PyMC4, TVM/NNVM, Plasma Store, Apache Arrow, and Tensor Comprehensions.

Please reach out if you have ideas or want to help, on either of the Gitter channels below:

XND XND integrations

We also have weekly public XND meetings that you are welcome to attend:

Public Meeting Calendar

If you want to understand more about XND and Plures projects, we invite you to check out this introductory notebook:

Introduction Notebook (Git repo) XND Notebooks (Binder)

Packages

gumath is a library for function dispatch.

libgumath is a C library supporting a general dispatch mechanism for general memory containers as well as a specific implementation for NumPy-like containers of a composable, generalized function concept.

gumath is a Python wrapper for libgumath.

Code C Docs Python Docs

$ conda install -c xnd/label/dev gumath
# or
$ python3 -m pip install gumath

>>> from gumath import functions as fn
>>> from xnd import xnd
>>> fn.sin(xnd([[1, 2], [3, 4]], type='2 * 2 * float64'))
xnd([[0.8414709848078965, 0.9092974268256817], [0.1411200080598672, -0.7568024953079282]], type='2 * 2 * float64'))

xnd is a library for typed memory blocks.

libxnd implements support for typed memory blocks using the libndtypes type library.

The xnd module implements a container type that maps most Python values relevant for scientific computing directly to typed memory.

Code C Docs Python Docs

>>> from xnd import xnd
>>> xnd([{'b': [10.2, 232.3]}, {'b': [0.2, 0.23]}])
xnd([{'b': [10.2, 232.3]}, {'b': [0.2, 0.23]}], type='2 * {b : 2 * float64}')

>>> from xnd import xnd
>>> import numpy as np
>>> xnd.from_buffer(np.arange(6).reshape(3, 2))
xnd([[0, 1], [2, 3], [4, 5]], type='3 * 2 * int64')

ndtypes is a library for typing memory blocks.

libndtypes implements the type part of a compiler frontend. It can describe C types needed for array computing and additionally includes symbolic types for the purposes of dynamic type checking.

The ndtypes Python module implements bindings for libndtypes. Its purpose is to support the type part required for the xnd container module.

Code Docs

>>> from ndtypes import ndt
>>> # fixed dimension type
>>> ndt("fixed(shape=10) * uint64")
ndt("10 * uint64")

>>> from ndtypes import ndt
>>> # variable dimension type
>>> ndt("var * float32")
ndt("var * float32")

xndtools is a tool to help make gumath kernels.

xndtools provides a script that builds gumath kernels from C prototypes of the computational functions.

Code Docs

# Generate kernels configuration file (mkl_blas-kernels.cfg)
$ xnd_tools config $CONDA_PREFIX/include/mkl_blas.h

# Modify mkl_blas-kernels.cfg
[MODULE mkl_blas]
typemaps:
  MKL_INT: int64
includes:
  mkl_blas.h
kinds: Xnd, C
ellipses: none, ...
[KERNEL dot]
prototypes:
  double ddot(MKL_INT *n, double *x, MKL_INT *incx, double *y, MKL_INT *incy);
  float sdot(MKL_INT *n, float *x, MKL_INT *incx, float *y, MKL_INT *incy);
description: Compute dot product of two arrays
input_arguments: x(n), y(n)
hide_arguments: n = len(x), incx = 1, incy = 1

# Generate kernels (mkl_blas-kernels.c)
# and Python extension module (mkl_blas-python.c)
$ xnd_tools module mkl_blas-kernels.cfg

>>> # Build Python extension and use:
>>> import mkl_blas
>>> from xnd import xnd
>>> x = xnd([[1,2], [3,4]], dtype = 'float64')
>>> mkl_blas.dot(x, x)
xnd([5.0, 25.0], type='2 * float64')