io_tail

privex.helpers.common.io_tail(f: BinaryIO, nlines: int = 20, bsz: int = 4096) → Generator[List[str], None, None][source]
NOTE: If you’re only loading a small amount of lines, e.g. less than 1MB, consider using the much easier tail()

function - it only requires one call and returns the lines as a singular, correctly ordered list.

This is a generator function which works similarly to tail on UNIX systems. It efficiently retrieves lines in reverse order using the passed file handle f.

WARNING: This function is a generator which returns “chunks” of lines - while the lines within each chunk are in the correct order, the chunks themselves are backwards, i.e. each chunk retrieves lines prior to the previous chunk.

This function was designed as a generator to allow for memory efficient handling of large files, and tailing large amounts of lines. It only loads bsz bytes from the file handle into memory with each iteration, allowing you to process each chunk of lines as they’re read from the file, instead of having to load all nlines lines into memory at once.

To ensure your retrieved lines are in the correct order, with each iteration you must PREPEND the outputted chunk to your final result, rather than APPEND. Example:

>>> from privex.helpers import io_tail
>>> lines = []
>>> with open('/tmp/example', 'rb') as fp:
...     # We prepend each chunk from 'io_tail' to our result variable 'lines'
...     for chunk in io_tail(fp, nlines=10):
...         lines = chunk + lines
>>> print('\n'.join(lines))

Modified to be more memory efficient, but originally based on this SO code snippet: https://stackoverflow.com/a/136354

Parameters

  • f (BinaryIO) – An open file handle for the file to tail, must be in binary mode (e.g. rb)

  • nlines (int) – Total number of lines to retrieve from the end of the file

  • bsz (int) – Block size (in bytes) to load with each iteration (default: 4096 bytes). DON’T CHANGE UNLESS YOU UNDERSTAND WHAT THIS MEANS.

Return Generator chunks

Generates chunks (in reverse order) of correctly ordered lines as List[str]