Common/General Helpers¶

Common functions and classes that don’t fit into a specific category

Copyright:

    +===================================================+
    |                 © 2019 Privex Inc.                |
    |               https://www.privex.io               |
    +===================================================+
    |                                                   |
    |        Originally Developed by Privex Inc.        |
    |        License: X11 / MIT                         |
    |                                                   |
    |        Core Developer(s):                         |
    |                                                   |
    |          (+)  Chris (@someguy123) [Privex]        |
    |          (+)  Kale (@kryogenic) [Privex]          |
    |                                                   |
    +===================================================+

Copyright 2019     Privex Inc.   ( https://www.privex.io )

Attributes

`ALPHANUM`	All characters from a-z, A-Z, and 0-9 - for random strings where there’s no risk of user font confusion
`SAFE_CHARS`	Characters that shouldn’t be mistaken, avoiding users confusing an o with a 0 or an l with a 1 or I
`IS_XARGS`	Pre-compiled regex for matching catch-all positional argument parameter names like `*args`
`IS_XKWARGS`	Pre-compiled regex for matching catch-all keyword argument parameter names like `**args`
`T_PARAM`	alias of `inspect.Parameter`
`T_PARAM_DICT`	Type alias for dict’s mapping parameter names to `inspect.Parameter`’s, `DictObject`’s, and dict’s mapping classes to dict’s mapping parameter names to `inspect.Parameter`’s.
`T_PARAM_LIST`	Type alias for dict’s containing strings mapped to `inspect.Parameter`’s, lists of just `inspect.Parameter`’s, and any iterable of `inspect.Parameter`
`INS_EMPTY`	alias of `inspect._empty`

Functions

`_filter_params`(params[, ignore_xargs, …])	Filter an iterable containing `inspect.Parameter`’s, returning a `DictObject` containing parameter names mapped to their `inspect.Parameter` object.
`almost`(compare, *numbers[, tolerance])	Compare two or more numbers, returning `True` if all `numbers` are no more than `tolerance` greater or smaller than than `compare` - otherwise `False`.
`auto_list`(obj[, conv, force_wrap, force_iter])	Used for painless conversion of various data types into list-like objects (`list` / `tuple` / `set` etc.)
`byteify`(data[, encoding, if_none])	Convert a piece of data into bytes if it isn’t already.
`call_sys`(proc, *args[, write])	A small wrapper around `subprocess.Popen` which allows executing processes, while optionally piping data (`write`) into the process’s stdin, then finally returning the process’s output and error results.
`camel_to_snake`(name)	Convert `name` from camel case (`HelloWorld`) to snake case (`hello_world`).
`chunked`(iterable, n)	Split iterable into `n` iterables of similar size
`construct_dict`(cls, kwargs[, args, …])	Removes keys from the passed dict `data` which don’t exist on `cls` (thus would get rejected as kwargs) using `get_function_params()`.
`dec_round`(amount[, dp, rounding])	Round a Decimal to x decimal places using `quantize` (`dp` must be >= 1 and the default dp is 2)
`empty`(v[, zero, itr])	Quickly check if a variable is empty or not.
`empty_if`(v[, is_empty, not_empty])	Syntactic sugar for `x if empty(y) else z`.
`env_bool`(env_key[, env_default])	Obtains an environment variable `env_key`, if it’s empty or not set, `env_default` will be returned.
`env_cast`(env_key, cast[, env_default])	Obtains an environment variable `env_key`, if it’s empty or not set, `env_default` will be returned.
`env_csv`(env_key[, env_default, csvsplit])	Quick n’ dirty parsing of simple CSV formatted environment variables, with fallback to user specified `env_default` (defaults to None)
`env_decimal`(env_key[, env_default])	Alias for `env_cast()` with `Decimal` casting
`env_int`(env_key[, env_default])	Alias for `env_cast()` with `int` casting
`env_keyval`(env_key[, env_default, valsplit, …])	Parses an environment variable containing `key:val,key:val` into a list of tuples [(key,val), (key,val)]
`extract_settings`(prefix[, _settings, …])	Extract prefixed settings from a given module, dictionary, class, or instance.
`extract_type`(tp, **kwargs)	Attempt to identify the `type` of a given value, or for functions/methods - identify their RETURN value type.
`filter_form`(form, *keys[, cast])	Extract the keys `keys` from the dict-like `form` if they exist and return a dictionary containing the keys and values found.
`get_function_params`(obj[, check_parents])	Extracts a function/method’s signature (or class constructor signature if a class is passed), and returns it as a dictionary.
`get_return_type`(f)	Extract the return type for a function/method.
`human_name`(class_name)	This function converts a class/function name into a Title Case name.
`inject_items`(items, dest_list, position)	Inject a list `items` after a certain element in `dest_list`.
`io_tail`(f[, nlines, bsz])	NOTE: If you’re only loading a small amount of lines, e.g. less than 1MB, consider using the much easier `tail()`
`is_false`(v[, chk_none])	Warning: Unless you specifically need to verify a value is Falsey, it’s usually safer to check for truth `is_true()` and invert the result, i.e. `if not is_true(v)`.
`is_true`(v)	Check if a given bool/str/int value is some form of `True`:
`parse_csv`(line[, csvsplit])	Quick n’ dirty parsing of a simple comma separated line, with automatic whitespace stripping of both the `line` itself, and the values within the commas.
`parse_keyval`(line[, valsplit, csvsplit])	Parses a csv with key:value pairs such as.
`random_str`([size, chars])	Generate a random string of arbitrary length using a given character set (string / list / tuple).
`reverse_io`(f[, blocksize])	Read file as series of blocks from end of file to start.
`shell_quote`(*args)	Takes command line arguments as positional args, and properly quotes each argument to make it safe to pass on the command line.
`stringify`(data[, encoding, if_none])	Convert a piece of data into a string (from bytes) if it isn’t already.
`strip_null`(value[, conv, nullc])	Small convenience function which `stringify()`’s `value` then strips it of whitespace and null bytes, with two passes for good measure.
`tail`(filename[, nlines, bsz])	Pure python equivalent of the UNIX `tail` command.
`typing_to_base`(tp[, fail, return_orig, …])	Attempt to extract one or more native Python base types from a `typing` type, including generics such as `List[str]`, and combined types such as `Union[list, str]`

Classes

`ErrHelpParser`([prog, usage, description, …])	ErrHelpParser - Use this instead of `argparse.ArgumentParser` to automatically get full help output as well as the error message when arguments are invalid, instead of just an error message.
`LayeredContext`(wrapped_class, max_layers, fail)	A wrapper class for context manager classes / functions which allows you to control how many `with` layers that a context manager can have - and allow for the previous layer’s context manager `__enter__` / `yield` result to be passed down when `max_layers` is hit.

privex.helpers.common.ALPHANUM = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvwxyz'¶: All characters from a-z, A-Z, and 0-9 - for random strings where there’s no risk of user font confusion

class privex.helpers.common.ErrHelpParser(prog=None, usage=None, description=None, epilog=None, parents=[], formatter_class=<class 'argparse.HelpFormatter'>, prefix_chars='-', fromfile_prefix_chars=None, argument_default=None, conflict_handler='error', add_help=True, allow_abbrev=True)[source]¶

ErrHelpParser - Use this instead of argparse.ArgumentParser to automatically get full help output as well as the error message when arguments are invalid, instead of just an error message.

>>> parser = ErrHelpParser(description='My command line app')
>>> parser.add_argument('nums', metavar='N', type=int, nargs='+')

error(message: string)[source]¶

Prints a usage message incorporating the message to stderr and exits.

If you override this in a subclass, it should not return – it should either exit or raise an exception.

privex.helpers.common.INS_EMPTY¶: alias of inspect._empty

privex.helpers.common.IS_XARGS = re.compile('^\\*([a-zA-Z0-9_])+$')¶: Pre-compiled regex for matching catch-all positional argument parameter names like *args

privex.helpers.common.IS_XKWARGS = re.compile('^\\*\\*([a-zA-Z0-9_])+$')¶: Pre-compiled regex for matching catch-all keyword argument parameter names like **args

class privex.helpers.common.LayeredContext(wrapped_class: K, max_layers: Optional[int] = 1, fail: bool = False)[source]¶

A wrapper class for context manager classes / functions which allows you to control how many with layers that a context manager can have - and allow for the previous layer’s context manager __enter__ / yield result to be passed down when max_layers is hit.

(context managers are classes/functions with the methods __enter__ / __exit__ / __aenter__ / __aexit__ etc.)

Works with context manager classes, asyncio context manager classes, and contextlib.contextmanager() functions.

By default, LayeredContext sets max_layers to 1, meaning after 1 layer of with or async with statements, all additional layers will simply get given the same context result as the 1st layer, plus both __enter__ and __exit__ will only be called once (at the start and end of the first layer).

Using with class-based context managers:

>>> class Hello:
...     def __enter__(self):
...         print('entering Hello')
...         return self
...     def __exit__(self, exc_type, exc_val, exc_tb):
...         print('exiting Hello')
>>> ctx_a = LayeredContext(Hello())
>>> with ctx_a as a:
...     print('class manager layer 1')
...     with ctx_a as b:
...         print('class manager layer 2')
...     print('back to class layer 1')
entering Hello
class manager layer 1
class manager layer 2
back to class layer 1
exiting Hello

We can see that entering Hello and exiting Hello were only outputted at the end of the first context block with ctx_a as a, showing that Hello was only entered/exited as a context manager for the first with block.

Using with function-based :func:`contextlib.contextmanager` context managers:

>>> from contextlib import contextmanager
>>> @contextmanager
>>> def lorem():
...     print('entering lorem contextmanager')
...     yield 'hello world'
...     print('exiting lorem contextmanager')
>>> ctx_b = LayeredContext(lorem())
>>> with ctx_b as c:
...     print('function manager layer 1 - context is:', c)
...     with ctx_b as d:
...         print('function manager layer 2 - context is:', d)
...     print('back to function layer 1')
entering lorem contextmanager
function manager layer 1 - context is: hello world
function manager layer 2 - context is: hello world
back to function layer 1
exiting lorem contextmanager

We can see the default max_layers of 1 was respected, as the 2nd layer with ctx_b as d only printed function manager layer 2 (thus lorem’s enter/exit methods were not called), and it shows the context is still hello world (the context yielded by lorem in layer 1).

Example usage

First we need an example class which can be used as a context manager, so we create Example with a very simple __enter__ and __exit__ method, which simply adds and subtracts from self.ctx_layer respectively:

>>> class Example:
...     def __init__(self):
...         self.ctx_layer = 0
...     def __enter__(self):
...         self.ctx_layer += 1
...         return self
...     def __exit__(self, exc_type, exc_val, exc_tb):
...         if self.ctx_layer <= 0: raise ValueError('ctx_layer <= 0 !!!')
...         self.ctx_layer -= 1
...         return None

If we then create an instance of Example, and use it as a context manager in a 2 layer nested with exp, we can see ctx_layer gets increased each time we use it as a context manager, and decreases after the context manager block:

>>> exp = Example()
>>> with exp as x:
...     print(x.ctx_layer)       # prints: 1
...     with exp as y:
...         print(y.ctx_layer)   # prints: 2
...     print(x.ctx_layer)       # prints: 1
>>> exp.ctx_layer
0

Now, lets wrap it with LayeredContext, and set the maximum amount of layers to 1. If we start using ctx as a context manager, it works as if we used the example instance exp as a context manager. But, unlike the real instance, __enter__ is only really called for the first with block, and __exit__ is only really called once we finish the first layer with ctx as x

>>> ctx = LayeredContext(exp, max_layers=1)
>>> with ctx as x:
...     print(x.ctx_layer)             # prints: 1
...     with ctx as y:
...         print(y.ctx_layer)         # prints: 1
...         print(ctx.virtual_layer)   # prints: 2
...     print(x.ctx_layer)         # prints: 1
...     print(ctx.virtual_layer)   # prints: 1
>>> exp.ctx_layer
0
>>> print(ctx.layer, ctx.virtual_layer)
0 0

async aenter() → Union[K, Any][source]¶

async aexit(exc_type=None, exc_val=None, exc_tb=None) → Any[source]¶

property class_name¶

current_context: Optional[Union[K, Any]]¶

enter() → Union[K, Any][source]¶

exit(exc_type=None, exc_val=None, exc_tb=None) → Any[source]¶

fail: bool¶

layer: int¶

layer_contexts: List[Any]¶

max_layers: Optional[int]¶

virtual_layer: int¶

wrapped_class: K¶

privex.helpers.common.SAFE_CHARS = 'abcdefhkmnprstwxyz23456789ACDEFGHJKLMNPRSTWXYZ'¶: Characters that shouldn’t be mistaken, avoiding users confusing an o with a 0 or an l with a 1 or I

privex.helpers.common.T_PARAM¶: alias of inspect.Parameter

privex.helpers.common.T_PARAM_DICT¶

Type alias for dict’s mapping parameter names to inspect.Parameter’s, DictObject’s, and dict’s mapping classes to dict’s mapping parameter names to inspect.Parameter’s.

alias of Union[Dict[str, inspect.Parameter], privex.helpers.collections.DictObject, Dict[type, Dict[str, inspect.Parameter]]]

privex.helpers.common.T_PARAM_LIST¶

Type alias for dict’s containing strings mapped to inspect.Parameter’s, lists of just inspect.Parameter’s, and any iterable of inspect.Parameter

alias of Union[Dict[str, inspect.Parameter], Mapping[str, inspect.Parameter], List[inspect.Parameter], Iterable[inspect.Parameter]]

privex.helpers.common.almost(compare: Union[decimal.Decimal, int, float, str], *numbers: Union[decimal.Decimal, int, float, str], tolerance: Union[decimal.Decimal, int, float, str] = Decimal('0.01'), **kwargs) → bool [source]¶

Compare two or more numbers, returning True if all numbers are no more than tolerance greater or smaller than than compare - otherwise False.

Works similarly to unittest.TestCase.assertAlmostEqual()

Basic usage with two numbers + default tolerance (0.01):

>>> almost('5', '5.001')
True
>>> almost('5', '5.5')
False

Multiple numbers + custom tolerance:

>>> almost('5', '5.14', '4.85', '5.08', tolerance=Decimal('0.2'))
True
>>> almost('5', '5.3', '4.85', '5.08', tolerance=Decimal('0.2'))
False

Using fail or test:

>>> # By passing ``fail=True``, a descriptive AssertionError is raised when the tolerance check fails.
>>> almost('5', '5.01', fail=True)
True
>>> almost('5', '5.02', fail=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "privex/helpers/common.py", line 1044, in almost
    raise AssertionError(
AssertionError: Number at position 0 (val: 5.02) failed tolerance (0.01) check against 5
>>> # By passing ``test=True``, a standard ``assert`` will be used to compare the numbers.
>>> almost('5', '5.01', test=True)
True
>>> almost('5', '5.02', test=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "privex/helpers/common.py", line 1041, in almost
    assert (x - tolerance) <= compare <= (x + tolerance)
AssertionError

Parameters

compare (Decimal|int|float|str) – The base number which all numbers will be compared against.
numbers (Decimal|int|float|str) – One or more numbers to compare against compare
tolerance (Decimal|int|float|str) – (kwarg only) Amount that each numbers can be greater/smaller than compare before returning False.
fail (bool) – (default: False) If true, will raise AssertionError on failed tolerance check, instead of returning False. (mutually exclusive with assert)
test (bool) – (default: False) If true, will use assert instead of testing with if. Useful in unit tests. (mutually exclusive with raise)

Raises

AttributeError – When less than 1 number is present in numbers
AssertionError – When kwarg raise is True and one or more numbers failed the tolerance check.

Return bool is_almost

True if all numbers are within tolerance of compare, False if one or more numbers is outside of the tolerance.

privex.helpers.common.auto_list(obj: V, conv: Union[Type[T], Callable[[V], T]] = <class 'list'>, force_wrap=False, force_iter=False, **kw) → T[source]¶

Used for painless conversion of various data types into list-like objects (list / tuple / set etc.)

Ensure object obj is a list-like object of type conv, if it isn’t, then attempt to convert it into an instance of conv via either list wrapping, or list iterating, depending on the type that obj is detected to be.

Examples:

>>> auto_list('hello world')
['hello world']
>>> auto_list('lorem ipsum', conv=set)
{'lorem ipsum'}
>>> auto_list('hello world', force_iter=True)
['h', 'e', 'l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd']

>>> auto_list(('this', 'is', 'a', 'test',))
['this', 'is', 'a', 'test']
>>> auto_list(('this', 'is', 'a', 'test',), force_wrap=True)
[('this', 'is', 'a', 'test')]

List Wrapping

The list wrapping conversion method is when we wrap an object with list brackets, i.e. [obj]. which makes the obj a single item inside of a new list.

This is important for simple single-value data types such as str, bytes, integers, floats etc. - since using list() might simply iterate over their contents, e.g. turnining "hello" into ['h', 'e', 'l', 'l', 'o'], which is rarely what you intend when you want to convert an object into a list.

This method is used by default for the types:

str, bytes, int, float, Decimal, bool, dict

To force conversion via List Wrapping, set the argument force_wrap=True

List Iteration / Iterating

The list iteration method is when we call list(obj) to convert obj ‘s contents into a list, rather than making obj an item inside of the list.

This is important for other list-like data types such as list / set / tuple etc., since with the List Wrapping method, it would result in for example, a set {'hello', 'world'} simply being wrapped by a list [{'hello', 'world'}], instead of converting it into a list.

To force conversion via List Iteration, set the argument force_iter=True

This method is used bt default for the types:

list, set, tuple, range

any object which didn't match the list wrapping type checks and has the method: __iter__

Parameters

obj (V|any) – An object of practically any type, to convert into an instance type of conv
conv (T|type|callable) – A type which is also callable with obj as the first positional argument, to convert obj into a conv instance.
force_wrap (bool) – When set to True, obj will always be converted into conv using the list wrapping method conv([obj]), regardless of whether it’s a type that should or shouldn’t be wrapped.
force_iter (bool) – When set to True, obj will always be converted into conv using the list iterator method, i.e. conv(list(obj)), regardless of whether it’s a type that should or shouldn’t be iterated.
zero (bool) – Passthru argument to empty() (treat the number 0 as empty)
itr (bool) – Passthru argument to empty() (treat zero-length iterables as empty)

Return T|list|set|tuple data

The object obj after converting it into a conv instance

privex.helpers.common.byteify(data: Optional[Union[str, bytes]], encoding='utf-8', if_none=None) → bytes [source]¶

Convert a piece of data into bytes if it isn’t already:

>>> byteify("hello world")
b"hello world"

By default, if data is None, then a TypeError will be raised by bytes().

If you’d rather convert None into a blank bytes string, use if_node="", like so:

>>> byteify(None)
TypeError: encoding without a string argument
>>> byteify(None, if_none="")
b''

privex.helpers.common.call_sys(proc, *args, write: Union[bytes, str] = None, **kwargs) → Union[Tuple[bytes, bytes], Tuple[str, str]][source]¶

A small wrapper around subprocess.Popen which allows executing processes, while optionally piping data (write) into the process’s stdin, then finally returning the process’s output and error results. Designed to be easier to use than using subprocess.Popen directly.

Using AsyncIO? - there’s a native python asyncio version of this function available in call_sys_async(), which uses the native asyncio.subprocess.create_subprocess_shell(), avoiding blocking IO.

By default, stdout and stdin are set to subprocess.PIPE while stderr defaults to subprocess.STDOUT. You can override these by passing new values as keyword arguments.

NOTE: The first positional argument is executed, and all other positional arguments are passed to the process in the order specified. To use call_sys’s arguments write, stdout, stderr and/or stdin, you MUST specify them as keyword arguments, otherwise they’ll just be passed to the process you’re executing.

Any keyword arguments not specified in the :param or :key pydoc specifications will simply be forwarded to the subprocess.Popen constructor.

Simple Example:

>>> # All arguments are automatically quoted if required, so spaces are completely fine.
>>> folders, _ = call_sys('ls', '-la', '/tmp/spaces are fine/hello world')
>>> print(stringify(folders))
backups  cache   lib  local  lock  log  mail  opt  run  snap  spool  tmp

Piping data into a process:

>>> data = "hello world"
>>> # The data "hello world" will be piped into wc's stdin, and wc's stdout + stderr will be returned
>>> out, _ = call_sys('wc', '-c', write=data)
>>> int(out)
11

Parameters

proc (str) – The process to execute.
args (str) – Any arguments to pass to the process proc as positional arguments.
write (bytes|str) – If this is not None, then this data will be piped into the process’s STDIN.

Key stdout

The subprocess file descriptor for stdout, e.g. subprocess.PIPE or subprocess.STDOUT

Key stderr

The subprocess file descriptor for stderr, e.g. subprocess.PIPE or subprocess.STDOUT

Key stdin

The subprocess file descriptor for stdin, e.g. subprocess.PIPE or subprocess.STDIN

Key cwd

Set the current/working directory of the process to this path, instead of the CWD of your calling script.

Return tuple output

A tuple containing the process output of stdout and stderr

privex.helpers.common.camel_to_snake(name: Union[bytes, str]) → str [source]¶

Convert name from camel case (HelloWorld) to snake case (hello_world).

name can be either a str or bytes.

Example:

>>> camel_to_snake("HelloWorldLoremIpsum")
'hello_world_lorem_ipsum'

Parameters: name (str|bytes) – A camel case (class style) name, e.g. HelloWorld
Return str snake_case: name converted to snake case hello_world

privex.helpers.common.chunked(iterable, n)[source]¶

Split iterable into n iterables of similar size

Examples::

>>> l = [1, 2, 3, 4]
>>> list(chunked(l, 4))
[[1], [2], [3], [4]]

>>> l = [1, 2, 3]
>>> list(chunked(l, 4))
[[1], [2], [3], []]

>>> l = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> list(chunked(l, 4))
[[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]

Taken from: https://stackoverflow.com/a/24484181/2648583

privex.helpers.common.construct_dict(cls: Union[Type[T], C], kwargs: dict, args: Iterable = None, check_parents=True) → Union[T, Any][source]¶

Removes keys from the passed dict data which don’t exist on cls (thus would get rejected as kwargs) using get_function_params(). Then create and return an instance of cls, passing the filtered kwargs dictionary as keyword args.

Ensures that any keys in your dictionary which don’t exist on cls are automatically filtered out, instead of causing an error due to unexpected keyword arguments.

Example - User class which only takes specific arguments

First let’s define a class which only takes three arguments in it’s constructor - username, first_name, last_name.

>>> class User:
...    def __init__(self, username, first_name=None, last_name=None):
...        self.username = username
...        self.first_name, self.last_name = first_name, last_name
...

Now we’ll create a dictionary which has those three arguments, but also the excess address and phone.

>>> data = dict(username='johndoe123', first_name='John', last_name='Doe',
...             address='123 Example St', phone='+1-123-000-1234')

If we tried to directly pass data as keyword args, we’d get an error:

>>> john = User(**data)
TypeError: __init__() got an unexpected keyword argument 'address'

But by using construct_dict(), we’re able to construct a User, as this helper function detects that the excess address and phone are not valid parameters for User’s constructor.

>>> from privex.helpers import construct_dict
>>> john = construct_dict(User, data)
>>> print(john.username, john.first_name, john.last_name)
johndoe123 John Doe

Example - A function/method which only takes specific arguments

Not only can construct_dict() be used for classes, but it can also be used for any function/method.

Here’s an example using a simple “factory function” which creates user objects:

>>> def create_user(username, first_name=None, last_name=None):
...     return User(username, first_name, last_name)
>>>
>>> data = dict(
...     username='johndoe123', first_name='John', last_name='Doe',
...     address='123 Example St', phone='+1-123-000-1234'
... )
>>> # We can't just pass data as kwargs due to the extra keys.
>>> create_user(**data)
TypeError: create_user() got an unexpected keyword argument 'address'
>>> # But we can call the function using construct_dict, which filters out the excess dict keys :)
>>> john = construct_dict(create_user, data)
>>> print(john.username, john.first_name, john.last_name)
johndoe123 John Doe

Parameters

cls (Type[T]|callable) – A class (not an instance) or callable (function / lambda) to extract and filter the parameter’s from, then call using filtered kwargs and args.
kwargs (dict) – A dictionary containing keyword arguments to filter and use to call / construct cls.
args (list|set) – A list of positional arguments (NOT FILTERED!) to pass when calling/constructing cls.
check_parents (bool) – (Default: True) If obj is a class and this is True, will recursively grab the constructor parameters for all parent classes of cls and merge them into the returned dict.

Return Any func_result

If cls was a function/method, the return result will be the returned data/object from the function passed.

Return T cls_instance

If cls was a class, then the return result will be an instance of the class.

privex.helpers.common.dec_round(amount: decimal.Decimal, dp: int = 2, rounding=None) → decimal.Decimal [source]¶

Round a Decimal to x decimal places using quantize (dp must be >= 1 and the default dp is 2)

If you don’t specify a rounding option, it will use whatever rounding has been set in decimal.getcontext() (most python versions have this default to ROUND_HALF_EVEN)

Basic Usage:

>>> from decimal import Decimal, getcontext, ROUND_FLOOR
>>> x = Decimal('1.9998')
>>> dec_round(x, 3)
Decimal('2.000')

Custom Rounding as an argument:

>>> dec_round(x, 3, rounding=ROUND_FLOOR)
Decimal('1.999')

Override context rounding to set the default:

>>> getcontext().rounding = ROUND_FLOOR
>>> dec_round(x, 3)
Decimal('1.999')

Parameters

amount (Decimal) – The amount (as a Decimal) to round
dp (int) – Number of decimal places to round amount to. (Default: 2)
rounding (str) – A decimal rounding option, e.g. ROUND_HALF_EVEN or ROUND_FLOOR

Return Decimal rounded

The rounded Decimal amount

privex.helpers.common.empty(v, zero: bool = False, itr: bool = False) → bool [source]¶

Quickly check if a variable is empty or not. By default only ‘’ and None are checked, use itr and zero to test for empty iterable’s and zeroed variables.

Returns True if a variable is None or '', returns False if variable passes the tests

Example usage:

>>> x, y = [], None
>>> if empty(y):
...     print('Var y is None or a blank string')
...
>>> if empty(x, itr=True):
...     print('Var x is None, blank string, or an empty dict/list/iterable')

Parameters

v – The variable to check if it’s empty
zero – if zero=True, then return True if the variable is int 0 or str '0'
itr – if itr=True, then return True if the variable is [], {}, or is an iterable and has 0 length

Return bool is_blank

True if a variable is blank (None, '', 0, [] etc.)

Return bool is_blank

False if a variable has content (or couldn’t be checked properly)

privex.helpers.common.empty_if(v: V, is_empty: K = None, not_empty: T = <class 'privex.helpers.types.UseOrigVar'>, **kwargs) → Union[T, K, V][source]¶

Syntactic sugar for x if empty(y) else z. If not_empty isn’t specified, then the original value v will be returned if it’s not empty.

Example 1:

>>> def some_func(name=None):
...     name = empty_if(name, 'John Doe')
...     return name
>>> some_func("")
John Doe
>>> some_func("Dave")
Dave

Example 2:

>>> empty_if(None, 'is empty', 'is not empty')
is empty
>>> empty_if(12345, 'is empty', 'is not empty')
is not empty

Parameters

v (Any) – The value to test for emptiness
is_empty – The value to return if v is empty (defaults to None)
not_empty – The value to return if v is not empty (defaults to the original value v)
kwargs – Any additional kwargs to pass to empty()

Key zero

if zero=True, then v is empty if it’s int 0 or str '0'

Key itr

if itr=True, then v is empty if it’s [], {}, or is an iterable and has 0 length

Return V orig_var

The original value v is returned if not_empty isn’t specified.

Return K is_empty

The value specified as is_empty is returned if v is empty

Return T not_empty

The value specified as not_empty is returned if v is not empty (and not_empty was specified)

privex.helpers.common.env_bool(env_key: str, env_default=None) → Optional[bool][source]¶

Obtains an environment variable env_key, if it’s empty or not set, env_default will be returned. Otherwise, it will be converted into a boolean using is_true()

Example:

>>> os.environ['HELLO_WORLD'] = '1'
>>> env_bool('HELLO_WORLD')
True
>>> env_bool('HELLO_NOEXIST')
None
>>> env_bool('HELLO_NOEXIST', 'error')
'error'

Parameters

env_key (str) – Environment var to attempt to load
env_default (any) – Fallback value if the env var is empty / not set (Default: None)

privex.helpers.common.env_cast(env_key: str, cast: callable, env_default=None)[source]¶

Obtains an environment variable env_key, if it’s empty or not set, env_default will be returned. Otherwise, it will be converted into a type of your choice using the callable cast parameter

Example:

>>> os.environ['HELLO'] = '1.234'
>>> env_cast('HELLO', Decimal, Decimal('0'))
Decimal('1.234')

Parameters

cast (callable) – A function to cast the user’s env data such as int str or Decimal etc.
env_key (str) – Environment var to attempt to load
env_default (any) – Fallback value if the env var is empty / not set (Default: None)

privex.helpers.common.env_csv(env_key: str, env_default=None, csvsplit=',') → List[str][source]¶

Quick n’ dirty parsing of simple CSV formatted environment variables, with fallback to user specified env_default (defaults to None)

Example:

>>> import os
>>> os.environ['EXAMPLE'] = '  hello ,  world, test')
>>> env_csv('EXAMPLE', [])
['hello', 'world', 'test']
>>> env_csv('NONEXISTANT', [])
[]

Parameters

env_key (str) – Environment var to attempt to load
env_default (any) – Fallback value if the env var is empty / not set (Default: None)
csvsplit (str) – A character (or several) used to terminate each value in the list. Default: comma ,

Return List[str] parsed_data

A list of str values parsed from the env var

privex.helpers.common.env_decimal(env_key: str, env_default=None) → decimal.Decimal [source]¶: Alias for env_cast() with Decimal casting

privex.helpers.common.env_int(env_key: str, env_default=None) → int [source]¶: Alias for env_cast() with int casting

privex.helpers.common.env_keyval(env_key: str, env_default=None, valsplit=':', csvsplit=',') → List[Tuple[str, str]][source]¶

Parses an environment variable containing key:val,key:val into a list of tuples [(key,val), (key,val)]

See parse_keyval()

Parameters

env_key (str) – Environment var to attempt to load
env_default (any) – Fallback value if the env var is empty / not set (Default: None)
valsplit (str) – A character (or several) used to split the key from the value (default: colon :)
csvsplit (str) – A character (or several) used to terminate each keyval pair (default: comma ,)

privex.helpers.common.extract_settings(prefix: str, _settings=<module 'privex.helpers.settings' from '/home/docs/checkouts/readthedocs.org/user_builds/python-helpers/checkouts/latest/privex/helpers/settings.py'>, defaults=None, merge_conf=None, **kwargs) → dict [source]¶

Extract prefixed settings from a given module, dictionary, class, or instance.

This helper function searches the object _settings for keys starting with prefix, and for any matching keys, it removes the prefix from each key, converts the remaining portion of each key to lowercase (unless you’ve set _case_sensitive=True), and then returns the keys their linked values as a dict.

For example, if you had a file called myapp/settings.py which contained REDIS_HOST = 'localhost' and REDIS_PORT = 6379, you could then run:

>>> # noinspection PyUnresolvedReferences
>>> from myapp import settings
>>> extract_settings('REDIS_', settings)
{'host': 'localhost', 'port': 6379}

Example uses

Example settings module at myapp/settings.py

from os.path import dirname, abspath, join

BASE_DIR = dirname(dirname(dirname(abspath(__file__))))
VERSION_FILE = join(BASE_DIR, 'privex', 'helpers', '__init__.py')

REDIS_HOST = 'localhost'
REDIS_PORT = 6379
REDIS_DB = 0

DEFAULT_CACHE_TIMEOUT = 300

Example - Extract Redis settings:

>>> # noinspection PyUnresolvedReferences
>>> from myapp import settings
>>> from privex.helpers import extract_settings
>>>
>>> # All keyword arguments (apart from _settings_mod and _keys_lower) are converted into a dictionary
>>> # and merged with the extracted settings
>>> # noinspection PyTypeChecker
>>> extract_settings('REDIS_', _settings=settings, port=6479, debug=True)
{'host': 'localhost', 'port': 6379, 'db': 0, 'debug': True}
>>> extract_settings('REDIS_', _settings=settings, merge_conf=dict(port=6479))
{'host': 'localhost', 'port': 6479, 'db': 0}

Example - Extract Redis settings - case sensitive mode:

>>> extract_settings('REDIS_', _settings=settings, _case_sensitive=True)
{'HOST': 'localhost', 'PORT': 6379, 'DB': 0}

Example - Extract database settings from the environment

The below dict comprehension is just so you can see the original environment keys before we run extract_settings:

>>> import os
>>> from privex.helpers import extract_settings
>>>
>>> {k: v for k,v in os.environ.items() if 'DB_' in k}
{'DB_USER': 'root',
 'DB_PASS': 'ExamplePass',
 'DB_NAME': 'example_db'}

We’ll now call extract_settings using os.environ converted into a dictionary, and attempt to quickly obtain the database settings - with lowercase keys, and without their DB_ prefix.

Below, you’ll see extract_settings extracted all keys starting with DB_, removed the DB_ prefix, converted the remaining portion of the key to lowercase, and also merged in the default setting ‘host’ since DB_HOST didn’t exist.

The outputted dictionary is perfect for passing to many database library constructors:

>>> extract_settings('DB_', dict(os.environ), host='localhost')
{'user': 'root',
 'pass': 'ExamplePass',
 'name': 'example_db',
 'host': 'localhost'}

Parameters

prefix (str) – The prefix (including the first underscore (_) or other separator) to search for in the settings
_settings (Module|dict|object) –
The object to extract the settings from. The object can be one of the following:
- A module, for example passing settings after running from myapp import settings
- A dict, for example extract_settings('X_', dict(X_A=1, X_B=2))
- A class which has the desired settings defined on it’s .__dict__ (e.g. any standard user class - class MyClass:, with settings defined as static class attributes)
- An instance of a class, which has all desired settings defined inside of .__dict__ (e.g. any standard user class instance, with static and/or instance attributes for each setting)
- Any other type which supports being casted to a dictionary via dict(obj).
merge_conf (dict) – Optionally you may specify a dictionary of “override” settings to merge with the extracted settings. The values in this dictionary take priority over both defaults, and the keys from _settings.
defaults (dict) – Optionally you may specify a dictionary of default settings to merge before the extracted settings, meaning values are only used if the key wasn’t present in the extracted settings nor merge_conf.
kwargs – Additional settings as keyword arguments (see below). Any keyword argument keys which aren’t valid settings will be added to the defaults dictionary. This means that defaults can also be specified as kwargs - as long as they don’t clash with any used kwarg settings (see below).

Key _case_sensitive

(Default False) If True, prefix is compared against _settings keys case sensitively. If False, then both prefix and each _settings key is converted to lowercase before comparison.

Key _keys_lower

Defaults to True if _case_sensitive is False, and False if _case_sensitive is True. If True, each extracted settings key is converted to lowercase before returning them - otherwise they’re returned with the same case as they were in _settings.

Return dict config

The extracted configuration keys (without their prefixes) and values as a dictionary. Based on the extracted keys from _settings, the fallback settings in defaults (and excess kwargs), plus the override settings in merge_conf.

privex.helpers.common.extract_type(tp: Union[type, callable, object], **kwargs) → Optional[Union[type, object, callable, tuple, Tuple[type]]][source]¶

Attempt to identify the type of a given value, or for functions/methods - identify their RETURN value type.

This function can usually detect typing types, including generics such as List[str], and will attempt to extract their native Python base type, e.g. list.

For typing.Union based types (including typing.Optional), it can extract a tuple of base types, including from nested typing.Union’s - e.g. Union[str, list, Union[dict, set], int would be simplified down to (str, list, dict, set, int)

Attention

If you want to extract the original return type from a function/method, including generic types such as List[str], then you should use get_return_type() instead.

Example 1 - convert a generic type e.g. Dict[str, str] into it’s native type (e.g. dict):

>>> dtype = Dict[str, str]
>>> # noinspection PyTypeHints,PyTypeChecker
>>> isinstance({}, dtype)
TypeError: Subscripted generics cannot be used with class and instance checks
>>> extract_type(dtype)
dict
>>> isinstance({}, extract_type(dtype))
True

Example 2 - extract the return type of a function/method, and if the return type is a generic (e.g. List[str]), automatically convert it into the native type (e.g. list) for use in comparisons such as isinstance():

>>> def list_wrap(v: T) -> List[T]:
...     return [v]
>>>
>>> extract_type(list_wrap)
list
>>> isinstance([1, 2, 3], extract_type(list_wrap))
True

Example 3 - extract the type from an instantiated object, allowing for isinstance() comparisons:

>>> from privex.helpers import DictObject
>>> db = DictObject(hello='world', lorem='ipsum')
{'hello': 'world', 'lorem': 'ipsum'}
>>> type_db = extract_type(db)
privex.helpers.collections.DictObject
>>> isinstance(db, type_db)
True
>>> isinstance(DictObject(test=123), type_db)
True

Example 4 - extract a tuple of types from a typing.Union or typing.Optional (inc. return types)

>>> def hello(x) -> Optional[str]:
...     return x * 5
...
>>> extract_type(hello)
(str, NoneType)
>>> # Even with a Union[] containing a List[], another Union[] (containing a Tuple and set), and a Dict[],
>>> # extract_type is still able to recursively flatten and simplify it down to a tuple of base Python types
>>> extract_type(Union[
...     List[str],
...     Union[Tuple[str, int, str], set],
...     Dict[int, str]
... ])
(list, tuple, set, dict)

Return Types

A type will be returned for most calls where tp is either:

Already a native type e.g. list

A generic type such as List[str] (which are technically instances of object)

A function/method with a valid return type annotation, including generic return types

An instance of a class (an object), where the original type can be easily extracted via tp.__class__

If tp was an object and the type/class couldn’t be extracted, then it would be returned in it’s original object form.

If tp was an unusual function/method which couldn’t be detected as one, or issues occurred while extracting the return type, then tp may be returned in it’s original callable form.

Parameters: tp – The type/object/function etc. to extract the most accurate type from
Return type|object|callable ret: A type will be returned for most calls, but may be an object or callable if there were issues detecting the type.

privex.helpers.common.filter_form(form: Mapping, *keys, cast: callable = None) → Dict[str, Any][source]¶

Extract the keys keys from the dict-like form if they exist and return a dictionary containing the keys and values found.

Optionally, if cast isn’t None, then cast will be called to cast each form value to the desired type, e.g. int, Decimal, or str.

Example usage:

>>> a = dict(a=1, c=2, d=3)
>>> filter_form(a, 'a', 'c', 'e')
{'a': 1, 'c': 2}
>>> b = dict(lorem=1, ipsum='2', dolor=5.67)
>>> filter_form(b, 'lorem', 'ipsum', 'dolor', cast=int)
{'lorem': 1, 'ipsum': 2, 'dolor': 5}

Parameters

form (Mapping) – A dict-like object to extract key from.
keys (str|Any) – One or more keys to extract from form
cast (callable) – Cast the value of any extract form key using this callable

Return dict filtered_form

A dict containing the extracted keys and respective values from form

privex.helpers.common.get_function_params(obj: Union[type, callable], check_parents=False, **kwargs) → Union[Dict[str, inspect.Parameter], privex.helpers.collections.DictObject, Dict[type, Dict[str, inspect.Parameter]]][source]¶

Extracts a function/method’s signature (or class constructor signature if a class is passed), and returns it as a dictionary.

Primarily used by construct_dict() - but may be useful for other purposes.

If you’ve passed a class, you can set check_parents to True to obtain the signatures of the passed class’s constructor AND all of it’s parent classes, returned as a dictionary mapping classes to dictionaries of parameters.

If you’ve set check_parents to True, but you want the parameters to be a flat dictionary (just like when passing a function or class without check_parents), you can also pass merge=True, which merges each class’s constructor parameters into a dictionary mapping names to inspect.Parameter objects.

If any parameters conflict, children’s constructor parameters always take precedence over their parent’s version, much in the same way that Python’s inheritance works.

Basic (with functions):

>>> def some_func(x, y, z=123, *args, **kwargs):
...    pass

Get all normal parameters (positional and kwargs - excluding catch-all *args / **kwargs parameter types):

>>> params = get_function_params(some_func)
>>> params
{'x': <Parameter "x">, 'y': <Parameter "y">, 'z': <Parameter "z=123">}

Get raw parameter name and value (as written in signature) / access default values:

>>> str(params.z.name)     # You can also access it via params['z']
'z=123'
>>> params.z.default  # You can also access it via params['z']
123

Get only required parameters:

>>> get_function_params(some_func, ignore_defaults=True)
{'x': <Parameter "x">, 'y': <Parameter "y">}

Get only parameters with defaults:

>>> get_function_params(some_func, ignore_positional=True)
{'z': <Parameter "z=123">}

Example Usage (with classes and sub-classes):

>>> class BaseClass:
...     def __init__(self, a, b, c=1234, **kwargs):
...         pass

>>> class Example(BaseClass):
...     def __init__(self, d, e='hello', f=None, a='overridden', **kwargs):
...         super().__init__(a=a, d=d, e=e, f=f, **kwargs)

If we pass the class Example on it’s own, we get a dictionary of just it’s own parameters:

>>> get_function_params(Example)
{'d': <Parameter "d">, 'e': <Parameter "e='hello'">, 'f': <Parameter "f=None">}

However, if we set check_parents=True, we now get a dictionary containing Example’s constructor parameters, AND BaseClass’s (it’s parent class) constructor parameters, organised by class:

>>> get_function_params(Example, True)
{
    <class '__main__.Example'>: {
        'd': <Parameter "d">, 'e': <Parameter "e='hello'">, 'f': <Parameter "f=None">,
        'a': <Parameter "a='overridden'">
    },
    <class '__main__.BaseClass'>: {'a': <Parameter "a">, 'b': <Parameter "b">, 'c': <Parameter "c=1234">}
}

We can also add the optional kwarg merge=True, which merges the parameters of the originally passed class, and it’s parents.

This is done in reverse order, so that children’s conflicting constructor parameters take priority over their parents, as can be seen below with a which is shown as a='overridden' - the overridden parameter of the class Example with a default value, instead of the parent’s a which makes a mandatory:

>>> get_function_params(Example, True, merge=True)
{
    'a': <Parameter "a='overridden'">, 'b': <Parameter "b">, 'c': <Parameter "c=1234">,
    'd': <Parameter "d">, 'e': <Parameter "e='hello'">, 'f': <Parameter "f=None">
}

Parameters

obj (type|callable) – A class (not an instance) or callable (function / lambda) to extract and filter the parameter’s from. If a class is passed, the parameters of the constructor will be returned (__init__), excluding the initial self parameter.
check_parents (bool) – (Default: False) If obj is a class and this is True, will recursively grab the constructor parameters for all parent classes, and return the parameters as a dictionary of {<class X>: {'a': <Parameter 'a'>}, <class Y>: {'b': <Parameter 'b'>}, unless merge is also set to True.

Key bool ignore_xargs

(Default: True) Filter out any catch-all positional arguments (e.g. *args)

Key bool ignore_xkwargs

(Default: True) Filter out any catch-all keyword arguments (e.g. **kwargs)

Key bool ignore_defaults

(Default: False) Filter out any parameter which has a default value (e.g. args usable as kwargs)

Key bool ignore_positional

(Default: False) Filter out any parameter which doesn’t have a default value (mandatory args)

Key bool merge

(Default: False) If this is True, when check_parents is enabled, all parameters will be flatted into a singular dictionary, e.g. {'a': <Parameter 'a'>, 'b': <Parameter "b">}

Returns

privex.helpers.common.get_return_type(f: callable) → Optional[Union[type, object, callable]][source]¶

Extract the return type for a function/method. Note that this only works with functions/methods which have their return type annotated, e.g. def somefunc(x: int) -> float: return x * 2.1

Attention

If you want to extract a function/method return type and have any Generic typing types simplified down to their native Python base types (important to be able to compare with isinstance() etc.), then you should use extract_type() instead (handles raw types, objects, and function pointers)

Example 1 - Extracting a generic return type from a function:

>>> def list_wrap(v: T) -> List[T]:
...     return [v]
...
>>> rt = get_return_type(list_wrap)
typing.List[~T]
>>> rt._name            # We can get the string type name via _name
'List'
>>> l = rt.__args__[0]  # We can access the types inside of the [] via .__args__
~T
>>> l.__name__          # Get the name of 'l' - the type inside of the []
'T'

Example 2 - What happens if you use this on a function/method with no return type annotation?

The answer is: nothing - it will simply return None if the function/method has no return type annotation:

>>> def hello(x):
...     return x * 5
>>> repr(get_return_type(hello))
'None'

Parameters: f (callable) – A function/method to extract the return type from
Return return_type: The return type, usually either a type or a object

privex.helpers.common.human_name(class_name: Union[str, bytes, callable, Type[object]]) → str [source]¶

This function converts a class/function name into a Title Case name. It also directly accepts classes/functions.

Input names can be either snake case my_function, or InitialCaps MyClass - though mixtures of the two may work, such as some_functionName - however some_FunctionName will not (causes double spaces).

Examples

Using a plain string or bytes:

>>> human_name(b'_some_functionName')
'Some Function Name'
>>> human_name('SomeClassName')
'Some Class Name'

Using a reference to a function:

>>> def some_func():
...     pass
>>> human_name(some_func)
'Some Func'

Using a reference to a class, or an instance of a class:

>>> class MyExampleClass:
...     pass
>>> my_instance = MyExampleClass()
>>> human_name(MyExampleClass)
'My Example Class'
>>> human_name(my_instance)
'My Example Class'

Parameters: class_name – The name of a class/function specified either in InitialCaps or snake_case. You may also pass a function reference, class reference, or class instance. (see examples)
Return str human_name: The humanised Title Case name of class_name

privex.helpers.common.inject_items(items: list, dest_list: list, position: int) → List[str][source]¶

Inject a list items after a certain element in dest_list.

NOTE: This does NOT alter dest_list - it returns a NEW list with items injected after the given position in dest_list.

Example Usage:

>>> x = ['a', 'b', 'e', 'f', 'g']
>>> y = ['c', 'd']
>>> # Inject the list 'y' into list 'x' after element 1 (b)
>>> inject_items(y, x, 1)
['a', 'b', 'c', 'd', 'e', 'f', 'g']

Parameters

items (list) – A list of items to inject into dest_list
dest_list (list) – The list to inject items into
position (int) – Inject items after this element (0 = 1st item) in dest_list

Return List[str] injected

dest_list with the passed items list injected at position

privex.helpers.common.io_tail(f: BinaryIO, nlines: int = 20, bsz: int = 4096) → Generator[List[str], None, None][source]¶

NOTE: If you’re only loading a small amount of lines, e.g. less than 1MB, consider using the much easier tail(): function - it only requires one call and returns the lines as a singular, correctly ordered list.

This is a generator function which works similarly to tail on UNIX systems. It efficiently retrieves lines in reverse order using the passed file handle f.

WARNING: This function is a generator which returns “chunks” of lines - while the lines within each chunk are in the correct order, the chunks themselves are backwards, i.e. each chunk retrieves lines prior to the previous chunk.

This function was designed as a generator to allow for memory efficient handling of large files, and tailing large amounts of lines. It only loads bsz bytes from the file handle into memory with each iteration, allowing you to process each chunk of lines as they’re read from the file, instead of having to load all nlines lines into memory at once.

To ensure your retrieved lines are in the correct order, with each iteration you must PREPEND the outputted chunk to your final result, rather than APPEND. Example:

>>> from privex.helpers import io_tail
>>> lines = []
>>> with open('/tmp/example', 'rb') as fp:
...     # We prepend each chunk from 'io_tail' to our result variable 'lines'
...     for chunk in io_tail(fp, nlines=10):
...         lines = chunk + lines
>>> print('\n'.join(lines))

Modified to be more memory efficient, but originally based on this SO code snippet: https://stackoverflow.com/a/136354

Parameters

f (BinaryIO) – An open file handle for the file to tail, must be in binary mode (e.g. rb)
nlines (int) – Total number of lines to retrieve from the end of the file
bsz (int) – Block size (in bytes) to load with each iteration (default: 4096 bytes). DON’T CHANGE UNLESS YOU UNDERSTAND WHAT THIS MEANS.

Return Generator chunks

Generates chunks (in reverse order) of correctly ordered lines as List[str]

privex.helpers.common.is_false(v, chk_none: bool = True) → bool [source]¶

Warning: Unless you specifically need to verify a value is Falsey, it’s usually safer to check for truth is_true() and invert the result, i.e. if not is_true(v)

Check if a given bool/str/int value is some form of False:

bool: False

str: 'false', 'no', 'n', '0'

int: 0

If chk_none is True (default), will also consider the below values to be Falsey:

boolean: None // string: 'null', 'none', ''

(note: strings are automatically .lower()’d)

Usage:

>>> is_false(0)
True
>>> is_false('yes')
False

Parameters

v (Any) – The value to check for falseyness
chk_none (bool) – If True, treat None/'none'/'null' as Falsey (default True)

Return bool is_False

True if the value appears to be falsey, otherwise False.

privex.helpers.common.is_true(v) → bool [source]¶

Check if a given bool/str/int value is some form of True:

bool: True

str: 'true', 'yes', 'y', '1'

int: 1

(note: strings are automatically .lower()’d)

Usage:

>>> is_true('true')
True
>>> is_true('no')
False

Parameters: v (Any) – The value to check for truthfulness
Return bool is_true: True if the value appears to be truthy, otherwise False.

privex.helpers.common.parse_csv(line: str, csvsplit: str = ',') → List[str][source]¶

Quick n’ dirty parsing of a simple comma separated line, with automatic whitespace stripping of both the line itself, and the values within the commas.

Example:

>>> parse_csv('  hello ,  world, test')
['hello', 'world', 'test']
>>> parse_csv(' world  ;   test   ; example', csvsplit=';')
['world', 'test', 'example']

Parameters

line (str) – A string of columns separated by commas e.g. hello,world,foo
csvsplit (str) – A character (or several) used to terminate each value in the list. Default: comma ,

privex.helpers.common.parse_keyval(line: str, valsplit: str = ':', csvsplit=',') → List[Tuple[str, str]][source]¶

Parses a csv with key:value pairs such as:

John Alex:Doe,Jane Sarah:Doe

Into a list with tuple pairs (can be easily converted to a dict):

[
    ('John Alex', 'Doe'), 
    ('Jane Sarah', 'Doe')
]

By default, uses a colons : to split the key/value, and commas , to terminate the end of each keyval pair. This can be overridden by changing valsplit/csvsplit.

Parameters

line (str) – A string of key:value pairs separated by commas e.g. John Alex:Doe,Jane Sarah:Doe
valsplit (str) – A character (or several) used to split the key from the value (default: colon :)
csvsplit (str) – A character (or several) used to terminate each keyval pair (default: comma ,)

Return List[Tuple[str,str]] parsed_data

A list of (key, value) tuples that can easily be casted to a dict()

privex.helpers.common.random_str(size: int = 50, chars: Sequence = 'abcdefhkmnprstwxyz23456789ACDEFGHJKLMNPRSTWXYZ') → str [source]¶

Generate a random string of arbitrary length using a given character set (string / list / tuple). Uses Python’s SystemRandom class to provide relatively secure randomness from the OS. (On Linux, uses /dev/urandom)

By default, uses the character set SAFE_CHARS which contains letters a-z / A-Z and numbers 2-9 with commonly misread characters removed (such as 1, l, L, 0 and o). Pass ALPHANUM as chars if you need the full set of upper/lowercase + numbers.

Usage:

>>> from privex.helpers import random_str
>>> # Default random string - 50 character alphanum without easily mistaken chars
>>> password = random_str()
'MrCWLYMYtT9A7bHc5ZNE4hn7PxHPmsWaT9GpfCkmZASK7ApN8r'
>>> # Customised random string - 12 characters using only the characters `abcdef12345` 
>>> custom = random_str(12, chars='abcdef12345')
'aba4cc14a43d'

Warning: As this relies on the OS’s entropy features, it may not be cryptographically secure on non-Linux platforms:

> The returned data should be unpredictable enough for cryptographic applications, though its exact quality > depends on the OS implementation.

Parameters

size (int) – Length of random string to generate (default 50 characters)
chars (str) – Characterset to generate with ( default is SAFE_CHARS - a-z/A-Z/0-9 with often misread chars removed)

privex.helpers.common.reverse_io(f: BinaryIO, blocksize: int = 4096) → Generator[bytes, None, None][source]¶

Read file as series of blocks from end of file to start.

The data itself is in normal order, only the order of the blocks is reversed. ie. “hello world” -> [“ld”,”wor”, “lo “, “hel”] Note that the file must be opened in binary mode.

Original source: https://stackoverflow.com/a/136354

privex.helpers.common.shell_quote(*args: str) → str [source]¶

Takes command line arguments as positional args, and properly quotes each argument to make it safe to pass on the command line. Outputs a string containing all passed arguments properly quoted.

Uses shlex.join() on Python 3.8+, and a for loop of shlex.quote() on older versions.

Example:

>>> print(shell_quote('echo', '"orange"'))
echo '"orange"'

privex.helpers.common.stringify(data: Optional[Union[str, bytes]], encoding='utf-8', if_none=None) → str [source]¶

Convert a piece of data into a string (from bytes) if it isn’t already:

>>> stringify(b"hello world")
"hello world"

By default, if data is None, then None will be returned.

If you’d rather convert None into a blank string, use if_node="", like so:

>>> repr(stringify(None))
'None'
>>> stringify(None, if_none="")
''

privex.helpers.common.strip_null(value: Union[str, bytes], conv: Callable[[str], Union[str, bytes, T]] = <function stringify>, nullc='\x00') → Union[str, bytes, T][source]¶

Small convenience function which stringify()’s value then strips it of whitespace and null bytes, with two passes for good measure.

Parameters

value (str|bytes) – The value to clean whitespace/null bytes out of
conv (callable) – (Default stringify()) Optionally, you can override the casting function used after the stripping is completed
nullc (str) – (Default: \) Null characters to remove

Return str|bytes|T cleaned

The cleaned up value

privex.helpers.common.tail(filename: str, nlines: int = 20, bsz: int = 4096) → List[str][source]¶

Pure python equivalent of the UNIX tail command. Simply pass a filename and the number of lines you want to load from the end of the file, and a List[str] of lines (in forward order) will be returned.

This function is simply a wrapper for the highly efficient io_tail(), designed for usage with a small (<10,000) amount of lines to be tailed. To allow for the lines to be returned in the correct order, it must load all nlines lines into memory before it can return the data.

If you need to tail a large amount of data, e.g. 10,000+ lines of a logfile, you should consider using the lower level function io_tail() - which acts as a generator, only loading a certain amount of bytes into memory per iteration.

Example file /tmp/testing:

this is an example 1
this is an example 2
this is an example 3
this is an example 4
this is an example 5
this is an example 6

Example usage:

>>> from privex.helpers import tail
>>> lines = tail('/tmp/testing', nlines=3)
>>> print("\n".join(lines))
this is an example 4
this is an example 5
this is an example 6

Parameters

filename (str) – Path to file to tail. Relative or absolute path. Absolute path is recommended for safety.
nlines (int) – Total number of lines to retrieve from the end of the file
bsz (int) – Block size (in bytes) to load with each iteration (default: 4096 bytes). DON’T CHANGE UNLESS YOU UNDERSTAND WHAT THIS MEANS.

Return List[str] lines

The last ‘nlines’ lines of the file ‘filename’ - in forward order.

privex.helpers.common.typing_to_base(tp, fail=False, return_orig=True, clean_union=True) → Optional[Union[type, object, callable, tuple, Tuple[type]]][source]¶

Attempt to extract one or more native Python base types from a typing type, including generics such as List[str], and combined types such as Union[list, str]

>>> typing_to_base(List[str])
list
>>> typing_to_base(Union[str, Dict[str, list], int])
(str, dict, int)
>>> typing_to_base(Union[str, Dict[str, list], int], clean_union=False)
(str, typing.Dict[str, list], int)
>>> typing_to_base(str)
str
>>> typing_to_base(str, fail=True)
TypeError: Failed to extract base type for type object: <class 'str'>
>>> repr(typing_to_base(str, return_orig=False))
'None'

Parameters

tp – The typing type object to extract base/native type(s) from.
fail (bool) – (Default: False) If True, then raises TypeError if tp doesn’t appear to be a typing type.
return_orig (bool) – (Default: True) If True, returns tp as-is if it’s not a typing type. When False, non- typing types will cause None to be returned.
clean_union (bool) – (Default: True) If True, typing.Union’s will have each type converted/validated into a normal type using extract_type()

Return type_res

Either a type base type, a tuple of types, a typing type object, or something else depending on what type tp was.