Implement single point for scalar conversion from python objects #7
Conversation
|
@kosiew I tried taking a stab at moving all of the scalar value conversion to a single point and supporting all of the libraries that I know about. I didn't add unit tests yet, though. |
|
@kosiew ready for review |
| import arro3.core | ||
| import nanoarrow |
There was a problem hiding this comment.
This should also be included in pyproject.toml's dev dependencies.
| let field = Arc::new(Field::new_list_field( | ||
| array.data_type().clone(), | ||
| array.nulls().is_some(), |
There was a problem hiding this comment.
this logic which is duplicated later can be extracted to a helper array_to_list_scalar to keep behaviour consistent and make future fixes easier.
| let field = Arc::new(Field::new_list_field( | ||
| array.data_type().clone(), | ||
| array.nulls().is_some(), | ||
| )); |
There was a problem hiding this comment.
duplicated logic which can be extracted to a helper array_to_list_scalar to keep behaviour consistent and make future fixes easier.
| let type_name = value.get_type().repr()?; | ||
| if type_name.contains("nanoarrow")? && type_name.contains("Scalar")? { |
There was a problem hiding this comment.
can we use is_instance instead of repr?
if let Ok(scalar_type) = na.getattr("Scalar") {
if value.is_instance(&scalar_type)? {
| // Does it have a PyCapsule interface but isn't one of our known libraries? | ||
| // If so do our "best guess". Try checking type name, and if that fails | ||
| // return a single value if the length is 1 and return a List value otherwise | ||
| if value.hasattr("__arrow_c_array__")? { |
There was a problem hiding this comment.
I could not find tests for the unknown __arrow_c_array__ path in python/tests/test_expr.py
2 participants
This PR is a proposal for adding a single point where we do python object to scalar value conversion. It attempts to handle three known arrow libraries: pyarrow, nanoarrow, and arro3. It includes trying to convert any library that produces a pycapsule arrow interface. There is a fallback to take any regular Python object and try turning it into a pyarrow scalar value and then importing it.