Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #15692

Bug#970021: Seeking a small group to package Apache Arrow (was: Bug#970021: RFP: apache-arrow -- cross-language development platform for in-memory analytics)

Cross-posted to 3 groups.

Show key headers only | View raw

Julian,

Arrow is a complicated and large package. We use it at work (where there is a
fair amount of Python, also to Conda etc) and do have issues with more
complex builds especially because it is 'data infrastructure' and can come in
from different parts. I would recommend against packaging at old one -- we
also have seen issues with different (py)arrow version biting.

Have you seen https://github.com/apache/arrow-nanoarrow ?

It works via the C API to Arrow which interchanges data via two void* to the
the two structs for arrow array and schema -- and avoids linkage issue. (In
user space the pyarrow or R arrow packages can still be used also interfacing
via these.)  I have been using it for R package bindings for some time and we
plan to expand that (again, at work) -- as do others. It is already use by
duckdb, by the Arrow 'ADBC' interfaces (which are generic in the ODBC/JDBC
sense but for Arrow, and also by a python interface to snowflake.

Dirk

-- 
dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org

Back to linux.debian.maint.python | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

Seeking a small group to package Apache Arrow (was: Bug#970021: RFP:  apache-arrow -- cross-language development platform for in-memory analytics) Julian Gilbey <jdg@debian.org> - 2024-03-25 19:20 +0100
  Bug#970021: Seeking a small group to package Apache Arrow (was: Bug#970021: RFP: apache-arrow -- cross-language development platform for in-memory analytics) Rene Engelhard <rene@debian.org> - 2024-03-29 21:00 +0100
  Bug#970021: Seeking a small group to package Apache Arrow (was: Bug#970021: RFP: apache-arrow -- cross-language development platform for in-memory analytics) Julian Gilbey <julian@d-and-j.net> - 2024-03-30 21:30 +0100
    Bug#970021: Seeking a small group to package Apache Arrow (was: Bug#970021: RFP: apache-arrow -- cross-language development platform for in-memory analytics) Julian Gilbey <julian@d-and-j.net> - 2024-03-31 13:30 +0200
      Bug#970021: Seeking a small group to package Apache Arrow (was: Bug#970021: RFP: apache-arrow -- cross-language development platform for in-memory analytics) Dirk Eddelbuettel <edd@debian.org> - 2024-03-31 14:00 +0200
  Bug#970021: Seeking a small group to package Apache Arrow (was: Bug#970021: RFP: apache-arrow -- cross-language development platform for in-memory analytics) Richard Duivenvoorde <richard@duiv.nl> - 2024-04-06 09:52 +0200

csiph-web