repeat_arrs_from_indices

Function repeat_arrs_from_indices 

Source
fn repeat_arrs_from_indices(
    batch: &[ArrayRef],
    indices: &PrimitiveArray<Int64Type>,
    repeat_mask: &[bool],
) -> Result<Vec<Arc<dyn Array>>>
Expand description

Create a batch of arrays based on an input batch and a indices array. The indices array is used by the take kernel to repeat values in the arrays that are marked with true in the repeat_mask. Arrays marked with false in the repeat_mask will be replaced with arrays filled with nulls of the appropriate length.

For example if we have the following batch:

c1: [1], null, [2, 3, 4], null, [5, 6]
c2: 'a', 'b',  'c', null, 'd'

then the unnested_list_arrays contains the unnest column that will replace c1 in the final batch if preserve_nulls is true:

c1: 1, null, 2, 3, 4, null, 5, 6

And the indices array contains the indices that are used by take kernel to repeat the values in c2:

0, 1, 2, 2, 2, 3, 4, 4

so that the final batch will look like:

c1: 1, null, 2, 3, 4, null, 5, 6
c2: 'a', 'b', 'c', 'c', 'c', null, 'd', 'd'

The repeat_mask determines whether an array’s values are repeated or replaced with nulls. For example, if the repeat_mask is:

[true, false]

The final batch will look like:

c1: 1, null, 2, 3, 4, null, 5, 6  // Repeated using `indices`
c2: null, null, null, null, null, null, null, null  // Replaced with nulls