Allow to customize raw CSV file reading

[mlell]

While CSVReader has numerous options to skip header or footer lines, or skip lines based on prefixes, it is cumbersome to read lines with varying lines before or after the actual statements. Users would need to override CSVReader.read(). Simply extending using super().read(...) is possible but requires creating a temporary file because read() needs a physical file path. Simply adding an open(str) -> file-like method to CSVReader allows to keep the useful functionality of read() while still being able to preprocess the file, for example, skipping a variable number of lines based on a pattern.

This would fix #196.

[johannesjh]

I second the proposition. It would also help with #197

Instead of strictly requiring a file-like object, the importer could allow more general Iterable[String] type of objects. Users could still pass file-like objects, but also generator expressions. Generator expressions could be a convenient and idiomatic way of manipulating / filtering CSV lines.

[dnicolodi]

This change does not belong to this commit or this PR. I also find it unnecessary.

[mlell]

The rationale was that there would be two ways to customize importer behaviour which partially overlap. Overriding "open()" would render the member "encoding" non-functional, for example. Will remove it.

[mlell]

done in a5561e3

[dnicolodi]

What does "special file formats" mean here? This class supports a declarative way to define an importer for CSV files. Why would you want to use it for something else?

[mlell]

I see that this is unclearly phrased. I wanted to refer to the fact that CSV has a large number of strange variants.

[mlell]

Also, there are grey areas I think, like compressed CSV or XLSX which are not strictly CSV but are conceptually similar and can easily be converted

[dnicolodi]

With the iterlines() idea exposed above, this would become simply:

from itertools import dropwhile

class Reader(CSVReader):
    first = Column("First")
    second = Column("Second")

    def iterlines(self, fd):
        return dropwhile(lambda line: "First" not in line, fd)

which seems much better to me.

[mlell]

Reader class is now:

        class Reader(CSVReader):
            first = Column("First")
            second = Column("Second")


            def open(self, filepath):
                """Skip lines until we find the column headers."""
                lines = super().open(filepath)
                return dropwhile(lambda line: "First" not in line, lines)

[mlell]

• Changed the return type of open() to be an iterable (e.g., a generator) to allow for e.g. generator expressions
• Removed the change to the docstring of read()
• Updated the unit test to use the dropwhile approach

While it is true that this is a reader for CSV data only, I felt that there might be some formats, like compressed data or XLSX, where it might be useful to first open in binary mode, preprocess and then hand over to read(). This would not be possible with the iterlines(). Approach. However, you might have experience with more different bank statement formats that I have, if you think noone is going to need this, the iterlines() approach would be simpler.

[dnicolodi]

Better but not needs some tweaks.

[mlell]

I have added your suggestions and rebased the branch to the latest version! What do you think @dnicolodi ?