Unified arrow API

[evertlammerts]

Problem

The Arrow API of the Python client regularly causes confusion. The most important issues seem to be that:

• The Relational API has a total of six functions, four of which are aliases, three of which are unique to the Relational API
• The naming of the two core functions is not as intuitive as it could be.

Also see #97

Core Functions

The Connection API (and with it, the duckdb module) and the Relational API have two core functions to create Arrow objects:

• fetch_record_batch() -> pyarrow.lib.RecordBatchReader
• fetch_arrow_table() -> pyarrow.lib.Table

The Connection API has another function to create a Relation from an Arrow object:

• arrow(arrow_object, connection = None) -> DuckDBPyRelation

Aliases

The Connection and Relational APIs both have an alias for fetch_record_batch():

• arrow() -> pyarrow.lib.RecordBatchReader This function was the first we exposed in the API and is probably most often used. It changed return type over the course of 1.4.X, from Table to RecordBatchReader, which caused a number of issues.

The Relational API has three more aliases:

• to_arrow_table() -> pyarrow.lib.Table
• fetch_arrow_reader() -> pyarrow.lib.RecordBatchReader
• record_batch() -> pyarrow.lib.RecordBatchReader (this has been deprecated for a while)

Changes

v1.5.0 API

The Connection and Relational APIs will have the following functions:

• to_arrow_reader() -> pyarrow.lib.RecordBatchReader
• to_arrow_table() -> pyarrow.lib.Table
• arrow() -> pyarrow.lib.RecordBatchReader Note: we will not deprecate this function in v1.5.0, but we will discourage its use in both the documentation and the docstring. We encourage users to use to_arrow_reader() instead.

The Connection API will keep this function:

• arrow(arrow_object, connection = None) -> DuckDBPyRelation

v1.5.0 Deprecated API

The fetch_* functions will be deprecated in v1.5.0 (which will be emitted as a DeprecationWarning) and removed in v1.6.0:

• fetch_record_batch() -> pyarrow.lib.RecordBatchReader
• fetch_arrow_table() -> pyarrow.lib.Table
• fetch_arrow_reader() -> pyarrow.lib.RecordBatchReader

v1.5.0 Removed API

• Relation::record_batch() -> pyarrow.lib.RecordBatchReader will be removed.

What's in a Name

Arrow's ADBC Driver Manager API uses the fetch_* naming convention (docs):

• fetch_arrow_table()
• fetch_record_reader()
• fetch_df()

There are lots of examples of the to_* naming convention as well:

• Vortex API
• Pandas I/O functions
• etc