Data error while using given dat file and copy book from this repo

[chelika2000]

Data read error: Downloads\COMP.DETAILS.SEP30.DATA.dat that are NOT DIVISIBLE by the RECORD SIZE calculated from the copybook

I am new to Cobrix and trying to explore for my use case: Read Mainframe data using copy book
Question
I am trying to use the data file and copy that presents in this repo.
examples/example_data/companies_data ------------ as the data file
examples/example_data/companies_copybook.cpy --- as the copy book
I am using spark 3.5.8 and using local machine.
Here is my code that I am trying:

spark = get_spark_session()

    # Parse the file using Cobrix
    df = spark.read.format("cobol") \
        .option("copybook", copybook_path) \
        .option("is_record_sequence", "true") \
        .option("is_rdw_big_endian", "true") \
        .option("debug", "true") \
        .load(datafile_path)

    # Show the Schema
    st.subheader("Parsed Copybook Schema:")
    # We extract the schema as a readable tree-like list
    schema_fields = [f"{field.name} ({field.dataType.simpleString()})" for field in df.schema.fields]
    st.write(schema_fields)

    # Show the Data
    st.subheader(f"Data Preview (First {num_rows} records):")

[yruslan]

Hi, @chelika2000
correct options to pass to cobol data format for the example companies_data folder are:

    val df = spark
      .read
      .format("cobol")                               // Alternatively can use "za.co.absa.cobrix.spark.cobol.source"
      .option("copybook", copybook_path)             // Or as a path name in HDFS. For local filesystem use file:// prefix
      .option("record_format", "V")                        // This file is a sequence of records with 4 byte record headers
      .option("segment_field", "SEGMENT_ID")               // The SEGMENT_ID field contains IDs of segments
      // Specifies a mapping between segment IDs and corresponding REDEFINE fields
      .option("redefine_segment_id_map:1", "STATIC-DETAILS => C")
      .option("redefine-segment-id-map:2", "CONTACTS => P")
      // Specifies the parent-child relationship between segments
      .option("segment-children:1", "STATIC-DETAILS => CONTACTS")
      .load(datafile_path)                   // Location of data file(s)

as per example:

cobrix/examples/spark-cobol-app/src/main/scala/com/example/spark/cobol/app/SparkCobolHierarchical.scala

Line 81 in e7bece4

val df = spark

The reason is the example file has variable-record length format, e.g. each record has different size. The size of each record is determined by RDW headers (first 4 bytes of each record). Here is more about mainframe record formats:

https://www.ibm.com/docs/en/zos-basic-skills?topic=set-data-record-formats

[chelika2000]

This is fantastic, ty for the tip. now it is working . I have another question , how do I know the example datasets in this repo are fixed or variable length etc. How do I know these properties so that I can use the options carefully so that I can see the power of cobrix. This is perfectly works for my use case and I am going to discuss my observations with my peers. Thank you so much again Cheli

…

On Tue, Mar 10, 2026 at 3:14 AM Ruslan Yushchenko ***@***.***> wrote: *yruslan* left a comment (AbsaOSS/cobrix#832) <#832 (comment)> Hi, @chelika2000 <https://github.com/chelika2000> correct options to pass to cobol data format for the example companies_data folder are: val df = spark .read .format("cobol") // Alternatively can use "za.co.absa.cobrix.spark.cobol.source" .option("copybook", copybook_path) // Or as a path name in HDFS. For local filesystem use file:// prefix .option("record_format", "V") // This file is a sequence of records with 4 byte record headers .option("segment_field", "SEGMENT_ID") // The SEGMENT_ID field contains IDs of segments // Specifies a mapping between segment IDs and corresponding REDEFINE fields .option("redefine_segment_id_map:1", "STATIC-DETAILS => C") .option("redefine-segment-id-map:2", "CONTACTS => P") // Specifies the parent-child relationship between segments .option("segment-children:1", "STATIC-DETAILS => CONTACTS") .load(datafile_path) // Location of data file(s) as per example: https://github.com/AbsaOSS/cobrix/blob/e7bece4274fbb594d3fbd5368a7d77e824273d74/examples/spark-cobol-app/src/main/scala/com/example/spark/cobol/app/SparkCobolHierarchical.scala#L81 The reason is the example file has variable-record length format, e.g. each record has different size. The size of each record is determined by RDW headers (first 4 bytes of each record). Here is more about mainframe record formats: https://www.ibm.com/docs/en/zos-basic-skills?topic=set-data-record-formats — Reply to this email directly, view it on GitHub <#832 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AL6Z3QOPEPFBLSBZGHM3W434P7FIBAVCNFSM6AAAAACWMGR2W2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DAMRZGQ4TGOJXGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[chelika2000]

I have another question I am using very small copy book and small data file 01 DCLRC-CB-EVENT. 10 RCEV-PART-KEY PIC S9(5)V COMP-3. 10 RCEV-LP-ID PIC S9(18)V COMP-3. 10 RCEV-CB-EVENT-DT PIC X(10). 10 RCEV-CB-INFO-REQ-IND PIC X(1). And I am getting schema like this , PARSED COPYBOOK SCHEMA: root |-- RCEV_PART_KEY: integer (nullable = true) |-- RCEV_LP_ID: long (nullable = true) |-- RCEV_CB_EVENT_DT: string (nullable = true) |-- RCEV_CB_INFO_REQ_IND: string (nullable = true) And getting this 26/03/10 20:25:24 WARN FileStreamer: End of stream reached: Requested 4608 bytes, received 796. 26/03/10 20:25:24 WARN FileStreamer: End of stream reached: Requested 4608 bytes, received 796. Total Records Found: 1 ================================================================================ DATA PREVIEW (First 100 records): 26/03/10 20:25:26 WARN FileStreamer: End of stream reached: Requested 4608 bytes, received 796. +--------------------+--------------------+--------------------+--------------------+ Why I am not seeing my 2 records and always getting blanks And the code that I am using is # Parse the file using Cobrix print("Loading COBOL data with Cobrix...") df = spark.read.format("cobol") \ .option("copybook", copybook_path) \ .option("is_record_sequence", "true") \ .option("allow_short_records", "true") \ .option("is_variable_size", "false") \ .option("ebcdic_code_page", "cp037") \ .option("debug", "false") \ .load(datafile_path) What Ami missing ? I can not upload the test data due to firewall issue . But it is dummy data. Any help would be appreciated Cheli

…

On Tue, Mar 10, 2026 at 7:37 AM W Wildwood ***@***.***> wrote: This is fantastic, ty for the tip. now it is working . I have another question , how do I know the example datasets in this repo are fixed or variable length etc. How do I know these properties so that I can use the options carefully so that I can see the power of cobrix. This is perfectly works for my use case and I am going to discuss my observations with my peers. Thank you so much again Cheli On Tue, Mar 10, 2026 at 3:14 AM Ruslan Yushchenko < ***@***.***> wrote: > *yruslan* left a comment (AbsaOSS/cobrix#832) > <#832 (comment)> > > Hi, @chelika2000 <https://github.com/chelika2000> > correct options to pass to cobol data format for the example > companies_data folder are: > > val df = spark > .read > .format("cobol") // Alternatively can use "za.co.absa.cobrix.spark.cobol.source" > .option("copybook", copybook_path) // Or as a path name in HDFS. For local filesystem use file:// prefix > .option("record_format", "V") // This file is a sequence of records with 4 byte record headers > .option("segment_field", "SEGMENT_ID") // The SEGMENT_ID field contains IDs of segments > // Specifies a mapping between segment IDs and corresponding REDEFINE fields > .option("redefine_segment_id_map:1", "STATIC-DETAILS => C") > .option("redefine-segment-id-map:2", "CONTACTS => P") > // Specifies the parent-child relationship between segments > .option("segment-children:1", "STATIC-DETAILS => CONTACTS") > .load(datafile_path) // Location of data file(s) > > as per example: > https://github.com/AbsaOSS/cobrix/blob/e7bece4274fbb594d3fbd5368a7d77e824273d74/examples/spark-cobol-app/src/main/scala/com/example/spark/cobol/app/SparkCobolHierarchical.scala#L81 > > The reason is the example file has variable-record length format, e.g. > each record has different size. The size of each record is determined by > RDW headers (first 4 bytes of each record). Here is more about mainframe > record formats: > > https://www.ibm.com/docs/en/zos-basic-skills?topic=set-data-record-formats > > — > Reply to this email directly, view it on GitHub > <#832 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AL6Z3QOPEPFBLSBZGHM3W434P7FIBAVCNFSM6AAAAACWMGR2W2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DAMRZGQ4TGOJXGU> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[chelika2000]

FB.data.html

Not sure why it is ending with html it should be like fb.data and you can rename like that

[yruslan]

Hi @chelika2000 , I was able to fetch the example file (800 bytes), will take a look.

Generally, there is 100% proof automated way to detect the file format. If it has fixed record length, variable record length, with RDWs is an additional info that user provides to Cobrix. The file format is not determined by copybook alone, unfortunately. But usually, by looking at the file in a HEX editor it can be determined if files have RDWs, and other properties. Also, there is 'debug=true' option that is helpful when data is shifted and does not perfectly align to the copybook.

I'm going to take a look at your example and let you know the options to use.

[yruslan]

Your copybook has the length of 24:

-------- FIELD LEVEL/NAME --------- --ATTRIBS--    FLD  START     END  LENGTH

1 DCLRC_CB_EVENT                                      1      1     24     24
  10 RCEV_PART_KEY                                    2      1      3      3
  10 RCEV_LP_ID                                       3      4     13     10
  10 RCEV_CB_EVENT_DT                                 4     14     23     10
  10 RCEV_CB_INFO_REQ_IND                             5     24     24      1

but the actual record size in the file is 80 (determined by looking in the file using a HEX editor).
Solution: either add a FILLER column to the copybook to make its side 80, or add the option to Cobrix specifying the record size:

    val df = spark
      .read
      .format("cobol")
      .option("copybook", copybook_path)
      .option("record_format", "F")
      .option("debug", "true")    // Optionally
      .option("record_length", 80)
      .load("FB.data")

    df.show(false)

[chelika2000]

This is fantastic and you the best , it worked very great Thank you so much Cheli

…

On Wed, Mar 11, 2026 at 6:09 AM Ruslan Yushchenko ***@***.***> wrote: *yruslan* left a comment (AbsaOSS/cobrix#832) <#832 (comment)> Your copybook has the length of 24: -------- FIELD LEVEL/NAME --------- --ATTRIBS-- FLD START END LENGTH 1 DCLRC_CB_EVENT 1 1 24 24 10 RCEV_PART_KEY 2 1 3 3 10 RCEV_LP_ID 3 4 13 10 10 RCEV_CB_EVENT_DT 4 14 23 10 10 RCEV_CB_INFO_REQ_IND 5 24 24 1 but the actual record size in the file is 80 (determined by looking in the file using a HEX editor). Solution: either add a FILLER column to the copybook to make its side 80, or add the option to Cobrix specifying the record size: val df = spark .read .format("cobol") .option("copybook", copybook_path) .option("record_format", "F") .option("debug", "true") // Optionally .option("record_length", 80) .load("FB.data") df.show(false) — Reply to this email directly, view it on GitHub <#832 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AL6Z3QIQ7AKUFRXQIHHRACT4QFCNTAVCNFSM6AAAAACWMGR2W2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DAMZYGQ2DSNBWGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>