spark 读取 tfrecord文件，解析tf.train.Feature(bytes_list=tf.train.BytesList(np.array(feature, dtype=np.float32).tobytes())) 文件字节码异常

[shaoshuaig]

1. 如下图所示：用spark直接去读取tfrecord数据，string类型的字段可以直接读取，但是adc_traj等numpy.array的数据会是乱码的形式。

2. 原因是spark加载tfrecord数据的时候，会默认把bytes类型的数据转成str，目前不清楚字符编码。而adc_traj存储的时候是用tf.train.Feature(bytes_list=tf.train.BytesList(np.array(feature, dtype=np.float32).tobytes()))方式存储的。 3. 用 np.frombuffer(str_y., dtype=np.float32)去解析数据 数据还是会报错： ValueError: buffer size must be a multiple of element size

[catdollar]

Hello, I think your issue is related to spark encoding, can you try the charset option? This stackoverflow thread may be helpful to you: https://stackoverflow.com/questions/51957742/read-a-bytes-column-in-spark

[jasonmusespresso]

it can be resolved by explicitly providing a schema when we load the tfrecords. e.g.

from pyspark.sql.types import StructType, StructField, StringType, BinaryType

schema = StructType(
    [
        StructField("my_string_col", StringType(), True),
        StructField("my_bytes_col", BinaryType(), False),   # implicitly set it to be binary 
    ]
)
df = spark.read.format("tfrecord").option("recordType", "Example").schema(custom_schema).load("/path/to/the/tfrecord/file")