Profiling & optimization

[florian-huber]

Currently the key data part is implemented using numpy structured arrays.
An alternative approach I tried used pandas multi-index DataFrames (#4 ).

This issue is to explore/discuss different implementations and their performance.

[florian-huber]

I tried several implementations including Numpy (multiple variants), Pandas, Polars, Numba (multiple variants).

Numpy structured array

Pro:

• Fast slicing
• Allows storing different datatypes.

Cons:

• Requires several more complex utils functions.
• Currently really slow when it comes to stacking more layers --> join data step is a serious bottleneck!

Pandas multi-index DataFrames (#4 )

Pro:

• Fast addition of additional data (joins/merges)
• Rather accessible implementation under the hood (when compared to the structured arrays)
• Allows storing different datatypes.

Cons:

• Slow when it comes to slicing. Since this is a very common action I would consider this to be critical.

Polars DataFrames

Pro:

• Very fast addition of additional data (joins/merges)
• Rather accessible implementation under the hood (when compared to the structured arrays)
• Allows storing different datatypes.

Cons:

• Notably slower than Numpy when it comes to slicing. Since this is a very common action I would consider this to be critical.

Numba

Pro:

• Can be used together with Numpy Arrays hence keeping the fast slicing part
• Avoids additional dependencies (Pandas or Polars)

Cons:

• I did not get merging to be as fast as with Polars!

[florian-huber]

Here results from a profiling script I made: