Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python.announce > #2400

bcolz, a column store for Python, 1.1.2 released

Show key headers only | View raw

======================
Announcing bcolz 1.1.2
======================

What's new
==========

This is a maintenance release that brings quite a lot of improvements.
Here are the highlights:

- Zstd is a supported codec now.  Fixes #331.

- C-Blosc updated to 1.11.2.

- Added a new `defaults_ctx` context so that users can select defaults
  easily without changing global behaviour. For example::

   with bcolz.defaults_ctx(vm="python", cparams=bcolz.cparams(clevel=0)):
      cout = bcolz.eval("(x + 1) < 0")

- Fixed a crash occurring in `ctable.todataframe()` when both `columns`
  and `orient='columns'` were specified.  PR #311.  Thanks to Peter
  Quackenbush.

- Use `pkg_resources.parse_version()` to test for version of packages.
  Fixes #322 (PY27 bcolz with dask unicode error).

- New package recipe for conda-forge.  Now you can install bcolz with:
`conda install -c conda-forge bcolz`.  Thanks to Alistair Miles.

For a more detailed change log, see:

https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst

For some comparison between bcolz and other compressed data containers,
see:

https://github.com/FrancescAlted/DataContainersTutorials

specially chapters 3 (in-memory containers) and 4 (on-disk containers).


What it is
==========

*bcolz* provides **columnar and compressed** data containers that can
live either on-disk or in-memory.  The compression is carried out
transparently by Blosc, an ultra fast meta-compressor that is optimized
for binary data.  Compression is active by default.

Column storage allows for efficiently querying tables with a large
number of columns.  It also allows for cheap addition and removal of
columns.  Lastly, high-performance iterators (like ``iter()``,
``where()``) for querying the objects are provided.

bcolz can use diffent backends internally (currently numexpr,
Python/NumPy or dask) so as to accelerate many vector and query
operations (although it can use pure NumPy for doing so too).  Moreover,
since the carray/ctable containers can be disk-based, it is possible to
use them for seamlessly performing out-of-memory computations.

While NumPy is used as the standard way to feed and retrieve data from
bcolz internal containers, but it also comes with support for
high-performance import/export facilities to/from `HDF5/PyTables tables
<http://www.pytables.org>`_ and `pandas dataframes
<http://pandas.pydata.org>`_.

Have a look at how bcolz and the Blosc compressor, are making a better
use of the memory without an important overhead, at least for some real
scenarios:

http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/querying-ep14.ipynb#Plots

bcolz has minimal dependencies (NumPy is the only strict requisite),
comes with an exhaustive test suite, and it is meant to be used in
production. Example users of bcolz are Visualfabriq
(http://www.visualfabriq.com/), Quantopian (https://www.quantopian.com/)
and scikit-allel:

* Visualfabriq:

  * *bquery*, A query and aggregation framework for Bcolz:
  * https://github.com/visualfabriq/bquery

* Quantopian:

  * Using compressed data containers for faster backtesting at scale:
  * https://quantopian.github.io/talks/NeedForSpeed/slides.html

* scikit-allel:

  * Exploratory analysis of large scale genetic variation data.
  * https://github.com/cggh/scikit-allel


Resources
=========

Visit the main bcolz site repository at:
http://github.com/Blosc/bcolz

Manual:
http://bcolz.blosc.org

Home of Blosc compressor:
http://blosc.org

User's mail list:
bcolz@googlegroups.com
http://groups.google.com/group/bcolz

License is the new BSD:
https://github.com/Blosc/bcolz/blob/master/LICENSES/BCOLZ.txt

Release notes can be found in the Git repository:
https://github.com/Blosc/bcolz/blob/master/RELEASE_NOTES.rst

----

  **Enjoy data!**

-- 
Francesc Alted

Back to comp.lang.python.announce | Previous | Next | Find similar | Unroll thread

Thread

bcolz, a column store for Python, 1.1.2 released Francesc Alted <faltet@gmail.com> - 2017-02-10 09:51 +0100

csiph-web