Use mmap for reading the input file#762
Conversation
|
I initially wrote binwalkv3 to use Ultimately I decided to just read-everything-into-memory, as it was simple and I'd rather throw an "out of memory" error than have the entire process mysteriously die mid-analysis. For reference, when I say "large files", the files in question were several gigs (specifically an Ubuntu installer ISO, which is just under 6GB), but still had plenty of room to fit into memory (24GB of RAM). After switching over to reading everything into memory these issues went away, as there was indeed enough free memory to fit the file in question. This also brought into question whether the issue with mmaping had anything to do with the OOM killer, or if it was some issue in I tested your PR against the same Ubuntu ISO and got the same result (process killed mid-extraction, and the terminal window it was running in closed as well). Unfortunately this is probably hard to test/replicate, since it is going to depend a lot on the system that binwalk is being run on. Any suggestions or fixes would be welcome, as I would love for this to work. :) |
This pull request changes how Binwalk reads the input files. Instead of reading the whole file into memory, files are mapped into memory, using the
memmap2crate. This approach improves the experience of using Binwalk with large files, especially if the file size is greater that the amount of memory on your system.