Skip to content

[Java] Minor types don't account for nullable FieldType flag #429

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
asfimport opened this issue May 10, 2017 · 4 comments
Open

[Java] Minor types don't account for nullable FieldType flag #429

asfimport opened this issue May 10, 2017 · 4 comments
Labels
Type: enhancement New feature or request

Comments

@asfimport
Copy link

Calling e.g. `FLOAT4.getNewVector("foo", new FieldType(false, ...), ...)" returns a NullableFloat4Vector instead of a Float4Vector.
edit: Float4Vector doesn't implement FieldVector, so can't currently be a top-level vector. I'm confused as to what the nullable flag is supposed to represent then.

Reporter: Emilio Lahr-Vivaz / @elahrvivaz

Note: This issue was originally created as ARROW-999. Please see the migration documentation for further details.

@asfimport
Copy link
Author

Emilio Lahr-Vivaz / @elahrvivaz:
So it seems from the layout (https://github.com/apache/arrow/blob/master/format/Layout.md) that all vectors have a null bitarray, however it doesn't have to be populated if there are no null values. So is the intention that the Field nullable flag should just control the creation of the null bitarray?

@asfimport
Copy link
Author

Jacques Nadeau / @jacques-n:
You probably need to update the vectors so that they set all the bits to null if they have "no-nulls". The goal was to only have nullable types in the java implementation as we found maintaining two parallel sets of vectors an extra level of complexity.

@asfimport
Copy link
Author

Emilio Lahr-Vivaz / @elahrvivaz:
Thanks, maybe I should update this issue to be 'implement non-nullable FieldVectors'? Is that something that would be desirable to implement, or would it introduce too much complexity? It seems like it would provide a performance improvement in the non-nullable case.

@asfimport
Copy link
Author

Jacques Nadeau / @jacques-n:
I've previously found that it adds excessive complexity (earlier versions of Arrow maintained them more as alternatives to the nullable versions). We've generally found that once you do vectorized operations, word-wise or wider bitmap evals are rarely the performance bottleneck.

@assignUser assignUser transferred this issue from apache/arrow Nov 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant