Better handling of signed bytes MRC files

[rdrighetto]

Hi @colinpalmer, hope you are doing well!

I see that mrcfile reads MRC files with signed bytes 'literally'. A tomogram mask generated with IMOD and saved as signed bytes (not intentionally) has the 0's represented as -128 and 1's represented as -127. Of course, bad things happen down the line if you read this mask with mrcfile and naively multiply your tomogram with it.

I recently made a PR to pytom-match-pick to be able to handle this case:
SBC-Utrecht/pytom-match-pick#270

But as @sroet suggested, maybe this is something that mrcfile should be able to handle properly by itself? Or am I missing something?

Below I leave the header of one such file (from IMOD's header command):

RO image file on unit   1 : 28.mrc     Size=     736561 K

 Number of columns, rows, sections .....    1023    1440     512
 Map mode ..............................    0   (bytes - signed in file)
 Start cols, rows, sects, grid x,y,z ...    0     0     0    1023   1440    512
 Pixel spacing (Angstroms)..............   10.74      10.74      10.74
 Cell angles ...........................   90.000   90.000   90.000
 Fast, medium, slow axes ...............    X    Y    Z
 Origin on x,y,z .......................    0.000       0.000       0.000
 Minimum density .......................   0.0000     (  -128.00     in file)
 Maximum density .......................   1.0000     (  -127.00     in file)
 Mean density ..........................  0.52643E-03 (  -128.00     in file)
 tilt angles (original,current) ........   0.0   0.0   0.0   0.0   0.0   0.0
 Space group,# extra bytes,idtype,lens .        0        0        0        0

     2 Titles :
Created by mrcfile.py                                       2024-10-22 19:08:37
clip: Subtract                                          03-Mar-25  10:46:13

It became a signed bytes file after the IMOD clip command. But IMOD somehow knows that this is a binary mask, while other programs don't 🤔

Any thoughts on how to go about this?

Thanks in advance!

[colinpalmer]

Hi @rdrighetto, thanks for raising this and sorry for the slow reply.

My inclination is that mrcfile should generally just comply with the published MRC2014 standard and not make special exceptions for other software handling things incorrectly. In this case it looks like IMOD is doing something strange with the files, and so it'd be a problem for IMOD to fix. (Specifically, if a file is created with 0 and 1 values, stored as signed bytes according to the standard, and then running it through an IMOD command causes the values to change to -128 and -127, then that's a mistake by IMOD. Whether IMOD chooses to use signed or unsigned bytes internally, it shouldn't be changing it when it processes a file.)

Of course, downstream users of mrcfile are welcome to implement their own workarounds for cases like this, as you have in pytom-match-pick. But I think for mrcfile as an I/O library, it'd risk causing more confusion if the numeric values or data type of a file become unpredictable and dependent on what the range of the values is.

I would consider adding handling for this case if it turns out that IMOD sets some other flag in the header to indicate whether the bytes are intended to be signed or unsigned. Then we can deviate from the standard specifically for files that are flagged as such. But I don't think changing it based purely on the values of the bytes themselves is a good idea.

[colinpalmer]

And just to follow up on this - I've had a quick glance at IMOD's MRC format doc here: https://bio3d.colorado.edu/imod/doc/mrc_format.txt

It does have a flag in imodFlags to indicate whether bytes are stored as signed or not. So, in your case it'd be interesting to check if those flags are set correctly or not in the file, so that we could interpret the byte signed-ness without reference to the actual values. If so, we could add some handling for that into mrcfile.

But it still looks like there's a problem with IMOD itself. If a file was created outside IMOD (i.e. imodStamp is not set) then it should assume that bytes are signed as per MRC2014. It's fine for it to use unsigned bytes when imodStamp is set because then the file is indicating that it's aware of IMOD's bytes convention, but for files without that indication it shouldn't be treating the bytes as unsigned.