[SEDONA-756] feat: raster Python serde and with_bands() support#2956
Open
prantogg wants to merge 1 commit into
Open
[SEDONA-756] feat: raster Python serde and with_bands() support#2956prantogg wants to merge 1 commit into
prantogg wants to merge 1 commit into
Conversation
prantogg
force-pushed
the
pranav/feature/raster-python-serde
branch
from
ed31473 to
782a8e1
Compare
prantogg
changed the title
prantogg
force-pushed
the
pranav/feature/raster-python-serde
branch
2 times, most recently
from
3a9890f to
782a8e1
Compare
Add Python-side serialize() for InDbSedonaRaster, enabling Python UDFs to return raster objects directly instead of the lossy .tolist() + RS_MakeRaster workaround. Rasters now round-trip as contiguous bytes preserving native dtypes and all metadata (CRS, nodata, affine, etc.). Add with_bands() to InDbSedonaRaster for replacing pixel data (NumPy array) while preserving spatial metadata. Band count and dtype may differ from the source raster. Add reconcileColorModel() to DeepCopiedRenderedImage (JVM) to fix colorModel/sampleModel mismatches at deserialization when Python UDFs change band count or dtype. Cherry-picked from wherobots/wherobots-compute@e08bde1da08 with vectorized UDF wiring excluded.
prantogg
force-pushed
the
pranav/feature/raster-python-serde
branch
from
782a8e1 to
288bfe7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Did you read the Contributor Guide?
Is this PR related to a ticket?
[SEDONA-XXX] my subject.What changes were proposed in this PR?
Returning raster data from Python UDFs currently requires
.tolist()+RS_MakeRaster, which forces Float64 promotion, creates 262K Python float objects per 512×512 tile, and loses all raster metadata (CRS, nodata, affine transform).This PR adds:
raster_serde.serialize()— WritesInDbSedonaRasterto Sedona's binary format, byte-compatible with JVMSerde.deserialize(). Uses cache-and-replay for opaque Kryo blobs (categories, properties, colorModel).InDbSedonaRaster.with_bands()— Creates a new raster with replaced pixel data (NumPy array) but preserved spatial metadata. Band count and dtype may differ from the source.RasterType.serialize()— Delegates toraster_serde.serialize()instead of raisingNotImplementedError.DeepCopiedRenderedImage.reconcileColorModel()(JVM) — Fixes colorModel/sampleModel mismatches at deserialization time when Python UDFs change band count or dtype.KryoUtil.skipUTF8String()andGridSampleDimensionSerializer.skip()— Utility methods for navigating Kryo streams without full deserialization.Benchmarked on Apple M2 Pro, 4-band rasters, median of 50 iterations:
.tolist()(ms)serialize()(ms)How was this patch tested?
with_bands()tests (band count changes, dtype changes, metadata survival)Did this PR include necessary documentation updates?
docs/tutorial/raster.mdto show the new raster-to-raster UDF pattern usingwith_bands().