Common/General Helpers¶
Common functions and classes that don’t fit into a specific category
Copyright:
+===================================================+
| © 2019 Privex Inc. |
| https://www.privex.io |
+===================================================+
| |
| Originally Developed by Privex Inc. |
| License: X11 / MIT |
| |
| Core Developer(s): |
| |
| (+) Chris (@someguy123) [Privex] |
| (+) Kale (@kryogenic) [Privex] |
| |
+===================================================+
Copyright 2019 Privex Inc. ( https://www.privex.io )
Attributes
All characters from a-z, A-Z, and 0-9 - for random strings where there’s no risk of user font confusion |
|
Characters that shouldn’t be mistaken, avoiding users confusing an o with a 0 or an l with a 1 or I |
|
Pre-compiled regex for matching catch-all positional argument parameter names like |
|
Pre-compiled regex for matching catch-all keyword argument parameter names like |
|
alias of |
|
Type alias for dict’s mapping parameter names to |
|
Type alias for dict’s containing strings mapped to |
|
alias of |
Functions
|
Filter an iterable containing |
|
Compare two or more numbers, returning |
|
Used for painless conversion of various data types into list-like objects ( |
|
Convert a piece of data into bytes if it isn’t already. |
|
A small wrapper around |
|
Convert |
|
Split iterable into |
|
Removes keys from the passed dict |
|
Round a Decimal to x decimal places using |
|
Quickly check if a variable is empty or not. |
|
Syntactic sugar for |
|
Obtains an environment variable |
|
Obtains an environment variable |
|
Quick n’ dirty parsing of simple CSV formatted environment variables, with fallback to user specified |
|
Alias for |
|
Alias for |
|
Parses an environment variable containing |
|
Extract prefixed settings from a given module, dictionary, class, or instance. |
|
Attempt to identify the |
|
Extract the keys |
|
Extracts a function/method’s signature (or class constructor signature if a class is passed), and returns it as a dictionary. |
Extract the return type for a function/method. |
|
|
This function converts a class/function name into a Title Case name. |
|
Inject a list |
|
NOTE: If you’re only loading a small amount of lines, e.g. less than 1MB, consider using the much easier |
|
Warning: Unless you specifically need to verify a value is Falsey, it’s usually safer to check for truth |
|
Check if a given bool/str/int value is some form of |
|
Quick n’ dirty parsing of a simple comma separated line, with automatic whitespace stripping of both the |
|
Parses a csv with key:value pairs such as. |
|
Generate a random string of arbitrary length using a given character set (string / list / tuple). |
|
Read file as series of blocks from end of file to start. |
|
Takes command line arguments as positional args, and properly quotes each argument to make it safe to pass on the command line. |
|
Convert a piece of data into a string (from bytes) if it isn’t already. |
|
Small convenience function which |
|
Pure python equivalent of the UNIX |
|
Attempt to extract one or more native Python base types from a |
Classes
|
ErrHelpParser - Use this instead of |
|
A wrapper class for context manager classes / functions which allows you to control how many |
-
privex.helpers.common.
ALPHANUM
= 'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvwxyz'¶ All characters from a-z, A-Z, and 0-9 - for random strings where there’s no risk of user font confusion
-
class
privex.helpers.common.
ErrHelpParser
(prog=None, usage=None, description=None, epilog=None, parents=[], formatter_class=<class 'argparse.HelpFormatter'>, prefix_chars='-', fromfile_prefix_chars=None, argument_default=None, conflict_handler='error', add_help=True, allow_abbrev=True)[source]¶ ErrHelpParser - Use this instead of
argparse.ArgumentParser
to automatically get full help output as well as the error message when arguments are invalid, instead of just an error message.>>> parser = ErrHelpParser(description='My command line app') >>> parser.add_argument('nums', metavar='N', type=int, nargs='+')
-
privex.helpers.common.
INS_EMPTY
¶ alias of
inspect._empty
-
privex.helpers.common.
IS_XARGS
= re.compile('^\\*([a-zA-Z0-9_])+$')¶ Pre-compiled regex for matching catch-all positional argument parameter names like
*args
-
privex.helpers.common.
IS_XKWARGS
= re.compile('^\\*\\*([a-zA-Z0-9_])+$')¶ Pre-compiled regex for matching catch-all keyword argument parameter names like
**args
-
class
privex.helpers.common.
LayeredContext
(wrapped_class: K, max_layers: Optional[int] = 1, fail: bool = False)[source]¶ A wrapper class for context manager classes / functions which allows you to control how many
with
layers that a context manager can have - and allow for the previous layer’s context manager__enter__
/yield
result to be passed down whenmax_layers
is hit.(context managers are classes/functions with the methods
__enter__
/__exit__
/__aenter__
/__aexit__
etc.)Works with context manager classes, asyncio context manager classes, and
contextlib.contextmanager()
functions.By default,
LayeredContext
setsmax_layers
to1
, meaning after 1 layer ofwith
orasync with
statements, all additional layers will simply get given the same context result as the 1st layer, plus both__enter__
and__exit__
will only be called once (at the start and end of the first layer).Using with class-based context managers:
>>> class Hello: ... def __enter__(self): ... print('entering Hello') ... return self ... def __exit__(self, exc_type, exc_val, exc_tb): ... print('exiting Hello') >>> ctx_a = LayeredContext(Hello()) >>> with ctx_a as a: ... print('class manager layer 1') ... with ctx_a as b: ... print('class manager layer 2') ... print('back to class layer 1') entering Hello class manager layer 1 class manager layer 2 back to class layer 1 exiting Hello
We can see that
entering Hello
andexiting Hello
were only outputted at the end of the first context blockwith ctx_a as a
, showing thatHello
was only entered/exited as a context manager for the firstwith
block.Using with function-based :func:`contextlib.contextmanager` context managers:
>>> from contextlib import contextmanager >>> @contextmanager >>> def lorem(): ... print('entering lorem contextmanager') ... yield 'hello world' ... print('exiting lorem contextmanager') >>> ctx_b = LayeredContext(lorem()) >>> with ctx_b as c: ... print('function manager layer 1 - context is:', c) ... with ctx_b as d: ... print('function manager layer 2 - context is:', d) ... print('back to function layer 1') entering lorem contextmanager function manager layer 1 - context is: hello world function manager layer 2 - context is: hello world back to function layer 1 exiting lorem contextmanager
We can see the default
max_layers
of1
was respected, as the 2nd layerwith ctx_b as d
only printedfunction manager layer 2
(thuslorem
’s enter/exit methods were not called), and it shows the context is stillhello world
(the context yielded bylorem
in layer 1).Example usage
First we need an example class which can be used as a context manager, so we create
Example
with a very simple__enter__
and__exit__
method, which simply adds and subtracts fromself.ctx_layer
respectively:>>> class Example: ... def __init__(self): ... self.ctx_layer = 0 ... def __enter__(self): ... self.ctx_layer += 1 ... return self ... def __exit__(self, exc_type, exc_val, exc_tb): ... if self.ctx_layer <= 0: raise ValueError('ctx_layer <= 0 !!!') ... self.ctx_layer -= 1 ... return None
If we then create an instance of
Example
, and use it as a context manager in a 2 layer nestedwith exp
, we can seectx_layer
gets increased each time we use it as a context manager, and decreases after the context manager block:>>> exp = Example() >>> with exp as x: ... print(x.ctx_layer) # prints: 1 ... with exp as y: ... print(y.ctx_layer) # prints: 2 ... print(x.ctx_layer) # prints: 1 >>> exp.ctx_layer 0
Now, lets wrap it with
LayeredContext
, and set the maximum amount of layers to1
. If we start usingctx
as a context manager, it works as if we used the example instanceexp
as a context manager. But, unlike the real instance,__enter__
is only really called for the firstwith
block, and__exit__
is only really called once we finish the first layerwith ctx as x
>>> ctx = LayeredContext(exp, max_layers=1) >>> with ctx as x: ... print(x.ctx_layer) # prints: 1 ... with ctx as y: ... print(y.ctx_layer) # prints: 1 ... print(ctx.virtual_layer) # prints: 2 ... print(x.ctx_layer) # prints: 1 ... print(ctx.virtual_layer) # prints: 1 >>> exp.ctx_layer 0 >>> print(ctx.layer, ctx.virtual_layer) 0 0
-
property
class_name
¶
-
current_context
: Optional[Union[K, Any]]¶
-
layer_contexts
: List[Any]¶
-
wrapped_class
: K¶
-
property
-
privex.helpers.common.
SAFE_CHARS
= 'abcdefhkmnprstwxyz23456789ACDEFGHJKLMNPRSTWXYZ'¶ Characters that shouldn’t be mistaken, avoiding users confusing an o with a 0 or an l with a 1 or I
-
privex.helpers.common.
T_PARAM
¶ alias of
inspect.Parameter
-
privex.helpers.common.
T_PARAM_DICT
¶ Type alias for dict’s mapping parameter names to
inspect.Parameter
’s,DictObject
’s, and dict’s mapping classes to dict’s mapping parameter names toinspect.Parameter
’s.alias of Union[Dict[str, inspect.Parameter], privex.helpers.collections.DictObject, Dict[type, Dict[str, inspect.Parameter]]]
-
privex.helpers.common.
T_PARAM_LIST
¶ Type alias for dict’s containing strings mapped to
inspect.Parameter
’s, lists of justinspect.Parameter
’s, and any iterable ofinspect.Parameter
alias of Union[Dict[str, inspect.Parameter], Mapping[str, inspect.Parameter], List[inspect.Parameter], Iterable[inspect.Parameter]]
-
privex.helpers.common.
almost
(compare: Union[decimal.Decimal, int, float, str], *numbers: Union[decimal.Decimal, int, float, str], tolerance: Union[decimal.Decimal, int, float, str] = Decimal('0.01'), **kwargs) → bool[source]¶ Compare two or more numbers, returning
True
if allnumbers
are no more thantolerance
greater or smaller than thancompare
- otherwiseFalse
.Works similarly to
unittest.TestCase.assertAlmostEqual()
Basic usage with two numbers + default tolerance (
0.01
):>>> almost('5', '5.001') True >>> almost('5', '5.5') False
Multiple numbers + custom tolerance:
>>> almost('5', '5.14', '4.85', '5.08', tolerance=Decimal('0.2')) True >>> almost('5', '5.3', '4.85', '5.08', tolerance=Decimal('0.2')) False
Using
fail
ortest
:>>> # By passing ``fail=True``, a descriptive AssertionError is raised when the tolerance check fails. >>> almost('5', '5.01', fail=True) True >>> almost('5', '5.02', fail=True) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "privex/helpers/common.py", line 1044, in almost raise AssertionError( AssertionError: Number at position 0 (val: 5.02) failed tolerance (0.01) check against 5 >>> # By passing ``test=True``, a standard ``assert`` will be used to compare the numbers. >>> almost('5', '5.01', test=True) True >>> almost('5', '5.02', test=True) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "privex/helpers/common.py", line 1041, in almost assert (x - tolerance) <= compare <= (x + tolerance) AssertionError
- Parameters
compare (Decimal|int|float|str) – The base number which all
numbers
will be compared against.numbers (Decimal|int|float|str) – One or more numbers to compare against
compare
tolerance (Decimal|int|float|str) – (kwarg only) Amount that each
numbers
can be greater/smaller thancompare
before returningFalse
.fail (bool) – (default:
False
) If true, will raiseAssertionError
on failed tolerance check, instead of returningFalse
. (mutually exclusive withassert
)test (bool) – (default:
False
) If true, will useassert
instead of testing withif
. Useful in unit tests. (mutually exclusive withraise
)
- Raises
AttributeError – When less than 1 number is present in
numbers
AssertionError – When kwarg
raise
isTrue
and one or more numbers failed the tolerance check.
- Return bool is_almost
True
if allnumbers
are withintolerance
ofcompare
,False
if one or morenumbers
is outside of the tolerance.
-
privex.helpers.common.
auto_list
(obj: V, conv: Union[Type[T], Callable[[V], T]] = <class 'list'>, force_wrap=False, force_iter=False, **kw) → T[source]¶ Used for painless conversion of various data types into list-like objects (
list
/tuple
/set
etc.)Ensure object
obj
is a list-like object of typeconv
, if it isn’t, then attempt to convert it into an instance ofconv
via either list wrapping, or list iterating, depending on the type thatobj
is detected to be.Examples:
>>> auto_list('hello world') ['hello world'] >>> auto_list('lorem ipsum', conv=set) {'lorem ipsum'} >>> auto_list('hello world', force_iter=True) ['h', 'e', 'l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd'] >>> auto_list(('this', 'is', 'a', 'test',)) ['this', 'is', 'a', 'test'] >>> auto_list(('this', 'is', 'a', 'test',), force_wrap=True) [('this', 'is', 'a', 'test')]
List Wrapping
The list wrapping conversion method is when we wrap an object with list brackets, i.e.
[obj]
. which makes theobj
a single item inside of a new list.This is important for simple single-value data types such as
str
,bytes
, integers, floats etc. - since usinglist()
might simply iterate over their contents, e.g. turnining"hello"
into['h', 'e', 'l', 'l', 'o']
, which is rarely what you intend when you want to convert an object into a list.This method is used by default for the types:
str, bytes, int, float, Decimal, bool, dict
To force conversion via List Wrapping, set the argument
force_wrap=True
List Iteration / Iterating
The list iteration method is when we call
list(obj)
to convertobj
‘s contents into a list, rather than makingobj
an item inside of the list.This is important for other list-like data types such as
list
/set
/tuple
etc., since with the List Wrapping method, it would result in for example, a set{'hello', 'world'}
simply being wrapped by a list[{'hello', 'world'}]
, instead of converting it into a list.To force conversion via List Iteration, set the argument
force_iter=True
This method is used bt default for the types:
list, set, tuple, range any object which didn't match the list wrapping type checks and has the method: __iter__
- Parameters
obj (V|any) – An object of practically any type, to convert into an instance type of
conv
conv (T|type|callable) – A
type
which is also callable withobj
as the first positional argument, to convertobj
into aconv
instance.force_wrap (bool) – When set to
True
,obj
will always be converted intoconv
using the list wrapping methodconv([obj])
, regardless of whether it’s a type that should or shouldn’t be wrapped.force_iter (bool) – When set to
True
,obj
will always be converted intoconv
using the list iterator method, i.e.conv(list(obj))
, regardless of whether it’s a type that should or shouldn’t be iterated.zero (bool) – Passthru argument to
empty()
(treat the number0
as empty)itr (bool) – Passthru argument to
empty()
(treat zero-length iterables as empty)
- Return T|list|set|tuple data
The object
obj
after converting it into aconv
instance
-
privex.helpers.common.
byteify
(data: Optional[Union[str, bytes]], encoding='utf-8', if_none=None) → bytes[source]¶ Convert a piece of data into bytes if it isn’t already:
>>> byteify("hello world") b"hello world"
By default, if
data
isNone
, then aTypeError
will be raised bybytes()
.If you’d rather convert
None
into a blank bytes string, useif_node=""
, like so:>>> byteify(None) TypeError: encoding without a string argument >>> byteify(None, if_none="") b''
-
privex.helpers.common.
call_sys
(proc, *args, write: Union[bytes, str] = None, **kwargs) → Union[Tuple[bytes, bytes], Tuple[str, str]][source]¶ A small wrapper around
subprocess.Popen
which allows executing processes, while optionally piping data (write
) into the process’s stdin, then finally returning the process’s output and error results. Designed to be easier to use than usingsubprocess.Popen
directly.Using AsyncIO? - there’s a native python asyncio version of this function available in
call_sys_async()
, which uses the nativeasyncio.subprocess.create_subprocess_shell()
, avoiding blocking IO.By default,
stdout
andstdin
are set tosubprocess.PIPE
while stderr defaults tosubprocess.STDOUT
. You can override these by passing new values as keyword arguments.NOTE: The first positional argument is executed, and all other positional arguments are passed to the process in the order specified. To use call_sys’s arguments
write
,stdout
,stderr
and/orstdin
, you MUST specify them as keyword arguments, otherwise they’ll just be passed to the process you’re executing.Any keyword arguments not specified in the
:param
or:key
pydoc specifications will simply be forwarded to thesubprocess.Popen
constructor.Simple Example:
>>> # All arguments are automatically quoted if required, so spaces are completely fine. >>> folders, _ = call_sys('ls', '-la', '/tmp/spaces are fine/hello world') >>> print(stringify(folders)) backups cache lib local lock log mail opt run snap spool tmp
Piping data into a process:
>>> data = "hello world" >>> # The data "hello world" will be piped into wc's stdin, and wc's stdout + stderr will be returned >>> out, _ = call_sys('wc', '-c', write=data) >>> int(out) 11
- Parameters
- Key stdout
The subprocess file descriptor for stdout, e.g.
subprocess.PIPE
orsubprocess.STDOUT
- Key stderr
The subprocess file descriptor for stderr, e.g.
subprocess.PIPE
orsubprocess.STDOUT
- Key stdin
The subprocess file descriptor for stdin, e.g.
subprocess.PIPE
orsubprocess.STDIN
- Key cwd
Set the current/working directory of the process to this path, instead of the CWD of your calling script.
- Return tuple output
A tuple containing the process output of stdout and stderr
-
privex.helpers.common.
camel_to_snake
(name: Union[bytes, str]) → str[source]¶ Convert
name
from camel case (HelloWorld
) to snake case (hello_world
).name
can be either astr
orbytes
.Example:
>>> camel_to_snake("HelloWorldLoremIpsum") 'hello_world_lorem_ipsum'
- Parameters
name (str|bytes) – A camel case (class style) name, e.g.
HelloWorld
- Return str snake_case
name
converted to snake casehello_world
-
privex.helpers.common.
chunked
(iterable, n)[source]¶ Split iterable into
n
iterables of similar size- Examples::
>>> l = [1, 2, 3, 4] >>> list(chunked(l, 4)) [[1], [2], [3], [4]]
>>> l = [1, 2, 3] >>> list(chunked(l, 4)) [[1], [2], [3], []]
>>> l = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] >>> list(chunked(l, 4)) [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]
Taken from: https://stackoverflow.com/a/24484181/2648583
-
privex.helpers.common.
construct_dict
(cls: Union[Type[T], C], kwargs: dict, args: Iterable = None, check_parents=True) → Union[T, Any][source]¶ Removes keys from the passed dict
data
which don’t exist oncls
(thus would get rejected as kwargs) usingget_function_params()
. Then create and return an instance ofcls
, passing the filteredkwargs
dictionary as keyword args.Ensures that any keys in your dictionary which don’t exist on
cls
are automatically filtered out, instead of causing an error due to unexpected keyword arguments.Example - User class which only takes specific arguments
First let’s define a class which only takes three arguments in it’s constructor - username, first_name, last_name.
>>> class User: ... def __init__(self, username, first_name=None, last_name=None): ... self.username = username ... self.first_name, self.last_name = first_name, last_name ...
Now we’ll create a dictionary which has those three arguments, but also the excess
address
andphone
.>>> data = dict(username='johndoe123', first_name='John', last_name='Doe', ... address='123 Example St', phone='+1-123-000-1234')
If we tried to directly pass data as keyword args, we’d get an error:
>>> john = User(**data) TypeError: __init__() got an unexpected keyword argument 'address'
But by using
construct_dict()
, we’re able to construct aUser
, as this helper function detects that the excessaddress
andphone
are not valid parameters forUser
’s constructor.>>> from privex.helpers import construct_dict >>> john = construct_dict(User, data) >>> print(john.username, john.first_name, john.last_name) johndoe123 John Doe
Example - A function/method which only takes specific arguments
Not only can
construct_dict()
be used for classes, but it can also be used for any function/method.Here’s an example using a simple “factory function” which creates user objects:
>>> def create_user(username, first_name=None, last_name=None): ... return User(username, first_name, last_name) >>> >>> data = dict( ... username='johndoe123', first_name='John', last_name='Doe', ... address='123 Example St', phone='+1-123-000-1234' ... ) >>> # We can't just pass data as kwargs due to the extra keys. >>> create_user(**data) TypeError: create_user() got an unexpected keyword argument 'address' >>> # But we can call the function using construct_dict, which filters out the excess dict keys :) >>> john = construct_dict(create_user, data) >>> print(john.username, john.first_name, john.last_name) johndoe123 John Doe
- Parameters
cls (Type[T]|callable) – A class (not an instance) or callable (function / lambda) to extract and filter the parameter’s from, then call using filtered
kwargs
andargs
.kwargs (dict) – A dictionary containing keyword arguments to filter and use to call / construct
cls
.args (list|set) – A list of positional arguments (NOT FILTERED!) to pass when calling/constructing
cls
.check_parents (bool) – (Default:
True
) Ifobj
is a class and this is True, will recursively grab the constructor parameters for all parent classes ofcls
and merge them into the returned dict.
- Return Any func_result
If
cls
was a function/method, the return result will be the returned data/object from the function passed.- Return T cls_instance
If
cls
was a class, then the return result will be an instance of the class.
-
privex.helpers.common.
dec_round
(amount: decimal.Decimal, dp: int = 2, rounding=None) → decimal.Decimal[source]¶ Round a Decimal to x decimal places using
quantize
(dp
must be >= 1 and the default dp is 2)If you don’t specify a rounding option, it will use whatever rounding has been set in
decimal.getcontext()
(most python versions have this default toROUND_HALF_EVEN
)Basic Usage:
>>> from decimal import Decimal, getcontext, ROUND_FLOOR >>> x = Decimal('1.9998') >>> dec_round(x, 3) Decimal('2.000')
Custom Rounding as an argument:
>>> dec_round(x, 3, rounding=ROUND_FLOOR) Decimal('1.999')
Override context rounding to set the default:
>>> getcontext().rounding = ROUND_FLOOR >>> dec_round(x, 3) Decimal('1.999')
-
privex.helpers.common.
empty
(v, zero: bool = False, itr: bool = False) → bool[source]¶ Quickly check if a variable is empty or not. By default only ‘’ and None are checked, use
itr
andzero
to test for empty iterable’s and zeroed variables.Returns
True
if a variable isNone
or''
, returnsFalse
if variable passes the testsExample usage:
>>> x, y = [], None >>> if empty(y): ... print('Var y is None or a blank string') ... >>> if empty(x, itr=True): ... print('Var x is None, blank string, or an empty dict/list/iterable')
- Parameters
v – The variable to check if it’s empty
zero – if
zero=True
, then returnTrue
if the variable is int0
or str'0'
itr – if
itr=True
, then returnTrue
if the variable is[]
,{}
, or is an iterable and has 0 length
- Return bool is_blank
True
if a variable is blank (None
,''
,0
,[]
etc.)- Return bool is_blank
False
if a variable has content (or couldn’t be checked properly)
-
privex.helpers.common.
empty_if
(v: V, is_empty: K = None, not_empty: T = <class 'privex.helpers.types.UseOrigVar'>, **kwargs) → Union[T, K, V][source]¶ Syntactic sugar for
x if empty(y) else z
. Ifnot_empty
isn’t specified, then the original valuev
will be returned if it’s not empty.Example 1:
>>> def some_func(name=None): ... name = empty_if(name, 'John Doe') ... return name >>> some_func("") John Doe >>> some_func("Dave") Dave
Example 2:
>>> empty_if(None, 'is empty', 'is not empty') is empty >>> empty_if(12345, 'is empty', 'is not empty') is not empty
- Parameters
v (Any) – The value to test for emptiness
is_empty – The value to return if
v
is empty (defaults toNone
)not_empty – The value to return if
v
is not empty (defaults to the original valuev
)kwargs – Any additional kwargs to pass to
empty()
- Key zero
if
zero=True
, then v is empty if it’s int0
or str'0'
- Key itr
if
itr=True
, then v is empty if it’s[]
,{}
, or is an iterable and has 0 length- Return V orig_var
The original value
v
is returned ifnot_empty
isn’t specified.- Return K is_empty
The value specified as
is_empty
is returned ifv
is empty- Return T not_empty
The value specified as
not_empty
is returned ifv
is not empty (and not_empty was specified)
-
privex.helpers.common.
env_bool
(env_key: str, env_default=None) → Optional[bool][source]¶ Obtains an environment variable
env_key
, if it’s empty or not set,env_default
will be returned. Otherwise, it will be converted into a boolean usingis_true()
Example:
>>> os.environ['HELLO_WORLD'] = '1' >>> env_bool('HELLO_WORLD') True >>> env_bool('HELLO_NOEXIST') None >>> env_bool('HELLO_NOEXIST', 'error') 'error'
- Parameters
env_key (str) – Environment var to attempt to load
env_default (any) – Fallback value if the env var is empty / not set (Default: None)
-
privex.helpers.common.
env_cast
(env_key: str, cast: callable, env_default=None)[source]¶ Obtains an environment variable
env_key
, if it’s empty or not set,env_default
will be returned. Otherwise, it will be converted into a type of your choice using the callablecast
parameterExample:
>>> os.environ['HELLO'] = '1.234' >>> env_cast('HELLO', Decimal, Decimal('0')) Decimal('1.234')
- Parameters
cast (callable) – A function to cast the user’s env data such as
int
str
orDecimal
etc.env_key (str) – Environment var to attempt to load
env_default (any) – Fallback value if the env var is empty / not set (Default: None)
-
privex.helpers.common.
env_csv
(env_key: str, env_default=None, csvsplit=',') → List[str][source]¶ Quick n’ dirty parsing of simple CSV formatted environment variables, with fallback to user specified
env_default
(defaults to None)Example:
>>> import os >>> os.environ['EXAMPLE'] = ' hello , world, test') >>> env_csv('EXAMPLE', []) ['hello', 'world', 'test'] >>> env_csv('NONEXISTANT', []) []
- Parameters
- Return List[str] parsed_data
A list of str values parsed from the env var
-
privex.helpers.common.
env_decimal
(env_key: str, env_default=None) → decimal.Decimal[source]¶ Alias for
env_cast()
withDecimal
casting
-
privex.helpers.common.
env_int
(env_key: str, env_default=None) → int[source]¶ Alias for
env_cast()
withint
casting
-
privex.helpers.common.
env_keyval
(env_key: str, env_default=None, valsplit=':', csvsplit=',') → List[Tuple[str, str]][source]¶ Parses an environment variable containing
key:val,key:val
into a list of tuples [(key,val), (key,val)]See
parse_keyval()
- Parameters
env_key (str) – Environment var to attempt to load
env_default (any) – Fallback value if the env var is empty / not set (Default: None)
valsplit (str) – A character (or several) used to split the key from the value (default: colon
:
)csvsplit (str) – A character (or several) used to terminate each keyval pair (default: comma
,
)
-
privex.helpers.common.
extract_settings
(prefix: str, _settings=<module 'privex.helpers.settings' from '/home/docs/checkouts/readthedocs.org/user_builds/python-helpers/checkouts/latest/privex/helpers/settings.py'>, defaults=None, merge_conf=None, **kwargs) → dict[source]¶ Extract prefixed settings from a given module, dictionary, class, or instance.
This helper function searches the object
_settings
for keys starting withprefix
, and for any matching keys, it removes the prefix from each key, converts the remaining portion of each key to lowercase (unless you’ve set_case_sensitive=True
), and then returns the keys their linked values as adict
.For example, if you had a file called
myapp/settings.py
which containedREDIS_HOST = 'localhost'
andREDIS_PORT = 6379
, you could then run:>>> # noinspection PyUnresolvedReferences >>> from myapp import settings >>> extract_settings('REDIS_', settings) {'host': 'localhost', 'port': 6379}
Example uses
Example settings module at
myapp/settings.py
from os.path import dirname, abspath, join BASE_DIR = dirname(dirname(dirname(abspath(__file__)))) VERSION_FILE = join(BASE_DIR, 'privex', 'helpers', '__init__.py') REDIS_HOST = 'localhost' REDIS_PORT = 6379 REDIS_DB = 0 DEFAULT_CACHE_TIMEOUT = 300
Example - Extract Redis settings:
>>> # noinspection PyUnresolvedReferences >>> from myapp import settings >>> from privex.helpers import extract_settings >>> >>> # All keyword arguments (apart from _settings_mod and _keys_lower) are converted into a dictionary >>> # and merged with the extracted settings >>> # noinspection PyTypeChecker >>> extract_settings('REDIS_', _settings=settings, port=6479, debug=True) {'host': 'localhost', 'port': 6379, 'db': 0, 'debug': True} >>> extract_settings('REDIS_', _settings=settings, merge_conf=dict(port=6479)) {'host': 'localhost', 'port': 6479, 'db': 0}
Example - Extract Redis settings - case sensitive mode:
>>> extract_settings('REDIS_', _settings=settings, _case_sensitive=True) {'HOST': 'localhost', 'PORT': 6379, 'DB': 0}
Example - Extract database settings from the environment
The below dict comprehension is just so you can see the original environment keys before we run
extract_settings
:>>> import os >>> from privex.helpers import extract_settings >>> >>> {k: v for k,v in os.environ.items() if 'DB_' in k} {'DB_USER': 'root', 'DB_PASS': 'ExamplePass', 'DB_NAME': 'example_db'}
We’ll now call
extract_settings
usingos.environ
converted into a dictionary, and attempt to quickly obtain the database settings - with lowercase keys, and without theirDB_
prefix.Below, you’ll see extract_settings extracted all keys starting with
DB_
, removed theDB_
prefix, converted the remaining portion of the key to lowercase, and also merged in the default setting ‘host’ sinceDB_HOST
didn’t exist.The outputted dictionary is perfect for passing to many database library constructors:
>>> extract_settings('DB_', dict(os.environ), host='localhost') {'user': 'root', 'pass': 'ExamplePass', 'name': 'example_db', 'host': 'localhost'}
- Parameters
prefix (str) – The prefix (including the first underscore (
_
) or other separator) to search for in the settings_settings (Module|dict|object) –
The object to extract the settings from. The object can be one of the following:
A
module
, for example passingsettings
after runningfrom myapp import settings
A
dict
, for exampleextract_settings('X_', dict(X_A=1, X_B=2))
A class which has the desired settings defined on it’s
.__dict__
(e.g. any standard user class -class MyClass:
, with settings defined as static class attributes)An instance of a class, which has all desired settings defined inside of
.__dict__
(e.g. any standard user class instance, with static and/or instance attributes for each setting)Any other type which supports being casted to a dictionary via
dict(obj)
.
merge_conf (dict) – Optionally you may specify a dictionary of “override” settings to merge with the extracted settings. The values in this dictionary take priority over both
defaults
, and the keys from_settings
.defaults (dict) – Optionally you may specify a dictionary of default settings to merge before the extracted settings, meaning values are only used if the key wasn’t present in the extracted settings nor
merge_conf
.kwargs – Additional settings as keyword arguments (see below). Any keyword argument keys which aren’t valid settings will be added to the
defaults
dictionary. This means that defaults can also be specified as kwargs - as long as they don’t clash with any used kwarg settings (see below).
- Key _case_sensitive
(Default
False
) IfTrue
,prefix
is compared against_settings
keys case sensitively. IfFalse
, then bothprefix
and each_settings
key is converted to lowercase before comparison.- Key _keys_lower
Defaults to
True
if _case_sensitive is False, andFalse
if _case_sensitive is True. IfTrue
, each extracted settings key is converted to lowercase before returning them - otherwise they’re returned with the same case as they were in_settings
.- Return dict config
The extracted configuration keys (without their prefixes) and values as a dictionary. Based on the extracted keys from
_settings
, the fallback settings indefaults
(and excesskwargs
), plus the override settings inmerge_conf
.
-
privex.helpers.common.
extract_type
(tp: Union[type, callable, object], **kwargs) → Optional[Union[type, object, callable, tuple, Tuple[type]]][source]¶ Attempt to identify the
type
of a given value, or for functions/methods - identify their RETURN value type.This function can usually detect
typing
types, including generics such asList[str]
, and will attempt to extract their native Python base type, e.g.list
.For
typing.Union
based types (includingtyping.Optional
), it can extract a tuple of base types, including from nestedtyping.Union
’s - e.g.Union[str, list, Union[dict, set], int
would be simplified down to(str, list, dict, set, int)
Attention
If you want to extract the original return type from a function/method, including generic types such as
List[str]
, then you should useget_return_type()
instead.Example 1 - convert a generic type e.g.
Dict[str, str]
into it’s native type (e.g.dict
):>>> dtype = Dict[str, str] >>> # noinspection PyTypeHints,PyTypeChecker >>> isinstance({}, dtype) TypeError: Subscripted generics cannot be used with class and instance checks >>> extract_type(dtype) dict >>> isinstance({}, extract_type(dtype)) True
Example 2 - extract the return type of a function/method, and if the return type is a generic (e.g.
List[str]
), automatically convert it into the native type (e.g.list
) for use in comparisons such asisinstance()
:>>> def list_wrap(v: T) -> List[T]: ... return [v] >>> >>> extract_type(list_wrap) list >>> isinstance([1, 2, 3], extract_type(list_wrap)) True
Example 3 - extract the type from an instantiated object, allowing for
isinstance()
comparisons:>>> from privex.helpers import DictObject >>> db = DictObject(hello='world', lorem='ipsum') {'hello': 'world', 'lorem': 'ipsum'} >>> type_db = extract_type(db) privex.helpers.collections.DictObject >>> isinstance(db, type_db) True >>> isinstance(DictObject(test=123), type_db) True
Example 4 - extract a tuple of types from a
typing.Union
ortyping.Optional
(inc. return types)>>> def hello(x) -> Optional[str]: ... return x * 5 ... >>> extract_type(hello) (str, NoneType) >>> # Even with a Union[] containing a List[], another Union[] (containing a Tuple and set), and a Dict[], >>> # extract_type is still able to recursively flatten and simplify it down to a tuple of base Python types >>> extract_type(Union[ ... List[str], ... Union[Tuple[str, int, str], set], ... Dict[int, str] ... ]) (list, tuple, set, dict)
Return Types
A
type
will be returned for most calls wheretp
is either:If
tp
was anobject
and the type/class couldn’t be extracted, then it would be returned in it’s original object form.If
tp
was an unusual function/method which couldn’t be detected as one, or issues occurred while extracting the return type, thentp
may be returned in it’s originalcallable
form.
-
privex.helpers.common.
filter_form
(form: Mapping, *keys, cast: callable = None) → Dict[str, Any][source]¶ Extract the keys
keys
from the dict-likeform
if they exist and return a dictionary containing the keys and values found.Optionally, if
cast
isn’tNone
, thencast
will be called to cast eachform
value to the desired type, e.g.int
,Decimal
, orstr
.Example usage:
>>> a = dict(a=1, c=2, d=3) >>> filter_form(a, 'a', 'c', 'e') {'a': 1, 'c': 2} >>> b = dict(lorem=1, ipsum='2', dolor=5.67) >>> filter_form(b, 'lorem', 'ipsum', 'dolor', cast=int) {'lorem': 1, 'ipsum': 2, 'dolor': 5}
- Parameters
form (Mapping) – A dict-like object to extract
key
from.keys (str|Any) – One or more keys to extract from
form
cast (callable) – Cast the value of any extract
form
key using this callable
- Return dict filtered_form
A dict containing the extracted keys and respective values from
form
-
privex.helpers.common.
get_function_params
(obj: Union[type, callable], check_parents=False, **kwargs) → Union[Dict[str, inspect.Parameter], privex.helpers.collections.DictObject, Dict[type, Dict[str, inspect.Parameter]]][source]¶ Extracts a function/method’s signature (or class constructor signature if a class is passed), and returns it as a dictionary.
Primarily used by
construct_dict()
- but may be useful for other purposes.If you’ve passed a class, you can set
check_parents
toTrue
to obtain the signatures of the passed class’s constructor AND all of it’s parent classes, returned as a dictionary mapping classes to dictionaries of parameters.If you’ve set
check_parents
toTrue
, but you want the parameters to be a flat dictionary (just like when passing a function or class without check_parents), you can also passmerge=True
, which merges each class’s constructor parameters into a dictionary mapping names toinspect.Parameter
objects.If any parameters conflict, children’s constructor parameters always take precedence over their parent’s version, much in the same way that Python’s inheritance works.
Basic (with functions):
>>> def some_func(x, y, z=123, *args, **kwargs): ... pass
Get all normal parameters (positional and kwargs - excluding catch-all
*args
/**kwargs
parameter types):>>> params = get_function_params(some_func) >>> params {'x': <Parameter "x">, 'y': <Parameter "y">, 'z': <Parameter "z=123">}
Get raw parameter name and value (as written in signature) / access default values:
>>> str(params.z.name) # You can also access it via params['z'] 'z=123' >>> params.z.default # You can also access it via params['z'] 123
Get only required parameters:
>>> get_function_params(some_func, ignore_defaults=True) {'x': <Parameter "x">, 'y': <Parameter "y">}
Get only parameters with defaults:
>>> get_function_params(some_func, ignore_positional=True) {'z': <Parameter "z=123">}
Example Usage (with classes and sub-classes):
>>> class BaseClass: ... def __init__(self, a, b, c=1234, **kwargs): ... pass >>> class Example(BaseClass): ... def __init__(self, d, e='hello', f=None, a='overridden', **kwargs): ... super().__init__(a=a, d=d, e=e, f=f, **kwargs)
If we pass the class
Example
on it’s own, we get a dictionary of just it’s own parameters:>>> get_function_params(Example) {'d': <Parameter "d">, 'e': <Parameter "e='hello'">, 'f': <Parameter "f=None">}
However, if we set
check_parents=True
, we now get a dictionary containingExample
’s constructor parameters, ANDBaseClass
’s (it’s parent class) constructor parameters, organised by class:>>> get_function_params(Example, True) { <class '__main__.Example'>: { 'd': <Parameter "d">, 'e': <Parameter "e='hello'">, 'f': <Parameter "f=None">, 'a': <Parameter "a='overridden'"> }, <class '__main__.BaseClass'>: {'a': <Parameter "a">, 'b': <Parameter "b">, 'c': <Parameter "c=1234">} }
We can also add the optional kwarg
merge=True
, which merges the parameters of the originally passed class, and it’s parents.This is done in reverse order, so that children’s conflicting constructor parameters take priority over their parents, as can be seen below with
a
which is shown asa='overridden'
- the overridden parameter of the classExample
with a default value, instead of the parent’sa
which makesa
mandatory:>>> get_function_params(Example, True, merge=True) { 'a': <Parameter "a='overridden'">, 'b': <Parameter "b">, 'c': <Parameter "c=1234">, 'd': <Parameter "d">, 'e': <Parameter "e='hello'">, 'f': <Parameter "f=None"> }
- Parameters
obj (type|callable) – A class (not an instance) or callable (function / lambda) to extract and filter the parameter’s from. If a class is passed, the parameters of the constructor will be returned (
__init__
), excluding the initialself
parameter.check_parents (bool) – (Default:
False
) Ifobj
is a class and this is True, will recursively grab the constructor parameters for all parent classes, and return the parameters as a dictionary of{<class X>: {'a': <Parameter 'a'>}, <class Y>: {'b': <Parameter 'b'>}
, unlessmerge
is also set toTrue
.
- Key bool ignore_xargs
(Default:
True
) Filter out any catch-all positional arguments (e.g.*args
)- Key bool ignore_xkwargs
(Default:
True
) Filter out any catch-all keyword arguments (e.g.**kwargs
)- Key bool ignore_defaults
(Default:
False
) Filter out any parameter which has a default value (e.g. args usable as kwargs)- Key bool ignore_positional
(Default:
False
) Filter out any parameter which doesn’t have a default value (mandatory args)- Key bool merge
(Default:
False
) If this is True, whencheck_parents
is enabled, all parameters will be flatted into a singular dictionary, e.g.{'a': <Parameter 'a'>, 'b': <Parameter "b">}
- Returns
-
privex.helpers.common.
get_return_type
(f: callable) → Optional[Union[type, object, callable]][source]¶ Extract the return type for a function/method. Note that this only works with functions/methods which have their return type annotated, e.g.
def somefunc(x: int) -> float: return x * 2.1
Attention
If you want to extract a function/method return type and have any Generic
typing
types simplified down to their native Python base types (important to be able to compare withisinstance()
etc.), then you should useextract_type()
instead (handles raw types, objects, and function pointers)Example 1 - Extracting a generic return type from a function:
>>> def list_wrap(v: T) -> List[T]: ... return [v] ... >>> rt = get_return_type(list_wrap) typing.List[~T] >>> rt._name # We can get the string type name via _name 'List' >>> l = rt.__args__[0] # We can access the types inside of the [] via .__args__ ~T >>> l.__name__ # Get the name of 'l' - the type inside of the [] 'T'
Example 2 - What happens if you use this on a function/method with no return type annotation?
The answer is: nothing - it will simply return
None
if the function/method has no return type annotation:>>> def hello(x): ... return x * 5 >>> repr(get_return_type(hello)) 'None'
-
privex.helpers.common.
human_name
(class_name: Union[str, bytes, callable, Type[object]]) → str[source]¶ This function converts a class/function name into a Title Case name. It also directly accepts classes/functions.
Input names can be either snake case
my_function
, or InitialCapsMyClass
- though mixtures of the two may work, such assome_functionName
- howeversome_FunctionName
will not (causes double spaces).Examples
Using a plain string or bytes:
>>> human_name(b'_some_functionName') 'Some Function Name' >>> human_name('SomeClassName') 'Some Class Name'
Using a reference to a function:
>>> def some_func(): ... pass >>> human_name(some_func) 'Some Func'
Using a reference to a class, or an instance of a class:
>>> class MyExampleClass: ... pass >>> my_instance = MyExampleClass() >>> human_name(MyExampleClass) 'My Example Class' >>> human_name(my_instance) 'My Example Class'
- Parameters
class_name – The name of a class/function specified either in InitialCaps or snake_case. You may also pass a function reference, class reference, or class instance. (see examples)
- Return str human_name
The humanised Title Case name of
class_name
-
privex.helpers.common.
inject_items
(items: list, dest_list: list, position: int) → List[str][source]¶ Inject a list
items
after a certain element indest_list
.NOTE: This does NOT alter
dest_list
- it returns a NEW list withitems
injected after the givenposition
indest_list
.Example Usage:
>>> x = ['a', 'b', 'e', 'f', 'g'] >>> y = ['c', 'd'] >>> # Inject the list 'y' into list 'x' after element 1 (b) >>> inject_items(y, x, 1) ['a', 'b', 'c', 'd', 'e', 'f', 'g']
-
privex.helpers.common.
io_tail
(f: BinaryIO, nlines: int = 20, bsz: int = 4096) → Generator[List[str], None, None][source]¶ - NOTE: If you’re only loading a small amount of lines, e.g. less than 1MB, consider using the much easier
tail()
function - it only requires one call and returns the lines as a singular, correctly ordered list.
This is a generator function which works similarly to
tail
on UNIX systems. It efficiently retrieves lines in reverse order using the passed file handlef
.WARNING: This function is a generator which returns “chunks” of lines - while the lines within each chunk are in the correct order, the chunks themselves are backwards, i.e. each chunk retrieves lines prior to the previous chunk.
This function was designed as a generator to allow for memory efficient handling of large files, and tailing large amounts of lines. It only loads
bsz
bytes from the file handle into memory with each iteration, allowing you to process each chunk of lines as they’re read from the file, instead of having to load allnlines
lines into memory at once.To ensure your retrieved lines are in the correct order, with each iteration you must PREPEND the outputted chunk to your final result, rather than APPEND. Example:
>>> from privex.helpers import io_tail >>> lines = [] >>> with open('/tmp/example', 'rb') as fp: ... # We prepend each chunk from 'io_tail' to our result variable 'lines' ... for chunk in io_tail(fp, nlines=10): ... lines = chunk + lines >>> print('\n'.join(lines))
Modified to be more memory efficient, but originally based on this SO code snippet: https://stackoverflow.com/a/136354
- Parameters
f (BinaryIO) – An open file handle for the file to tail, must be in binary mode (e.g.
rb
)nlines (int) – Total number of lines to retrieve from the end of the file
bsz (int) – Block size (in bytes) to load with each iteration (default: 4096 bytes). DON’T CHANGE UNLESS YOU UNDERSTAND WHAT THIS MEANS.
- Return Generator chunks
Generates chunks (in reverse order) of correctly ordered lines as
List[str]
- NOTE: If you’re only loading a small amount of lines, e.g. less than 1MB, consider using the much easier
-
privex.helpers.common.
is_false
(v, chk_none: bool = True) → bool[source]¶ Warning: Unless you specifically need to verify a value is Falsey, it’s usually safer to check for truth
is_true()
and invert the result, i.e.if not is_true(v)
Check if a given bool/str/int value is some form of
False
:bool:
False
str:
'false'
,'no'
,'n'
,'0'
int:
0
If
chk_none
is True (default), will also consider the below values to be Falsey:boolean: None // string: 'null', 'none', ''
(note: strings are automatically .lower()’d)
Usage:
>>> is_false(0) True >>> is_false('yes') False
- Parameters
v (Any) – The value to check for falseyness
chk_none (bool) – If
True
, treatNone
/'none'
/'null'
as Falsey (defaultTrue
)
- Return bool is_False
True
if the value appears to be falsey, otherwiseFalse
.
-
privex.helpers.common.
is_true
(v) → bool[source]¶ Check if a given bool/str/int value is some form of
True
:bool:
True
str:
'true'
,'yes'
,'y'
,'1'
int:
1
(note: strings are automatically .lower()’d)
Usage:
>>> is_true('true') True >>> is_true('no') False
- Parameters
v (Any) – The value to check for truthfulness
- Return bool is_true
True
if the value appears to be truthy, otherwiseFalse
.
-
privex.helpers.common.
parse_csv
(line: str, csvsplit: str = ',') → List[str][source]¶ Quick n’ dirty parsing of a simple comma separated line, with automatic whitespace stripping of both the
line
itself, and the values within the commas.Example:
>>> parse_csv(' hello , world, test') ['hello', 'world', 'test'] >>> parse_csv(' world ; test ; example', csvsplit=';') ['world', 'test', 'example']
-
privex.helpers.common.
parse_keyval
(line: str, valsplit: str = ':', csvsplit=',') → List[Tuple[str, str]][source]¶ Parses a csv with key:value pairs such as:
John Alex:Doe,Jane Sarah:Doe
Into a list with tuple pairs (can be easily converted to a dict):
[ ('John Alex', 'Doe'), ('Jane Sarah', 'Doe') ]
By default, uses a colons
:
to split the key/value, and commas,
to terminate the end of each keyval pair. This can be overridden by changing valsplit/csvsplit.- Parameters
- Return List[Tuple[str,str]] parsed_data
A list of (key, value) tuples that can easily be casted to a dict()
-
privex.helpers.common.
random_str
(size: int = 50, chars: Sequence = 'abcdefhkmnprstwxyz23456789ACDEFGHJKLMNPRSTWXYZ') → str[source]¶ Generate a random string of arbitrary length using a given character set (string / list / tuple). Uses Python’s SystemRandom class to provide relatively secure randomness from the OS. (On Linux, uses /dev/urandom)
By default, uses the character set
SAFE_CHARS
which contains letters a-z / A-Z and numbers 2-9 with commonly misread characters removed (such as1
,l
,L
,0
ando
). PassALPHANUM
as chars if you need the full set of upper/lowercase + numbers.Usage:
>>> from privex.helpers import random_str >>> # Default random string - 50 character alphanum without easily mistaken chars >>> password = random_str() 'MrCWLYMYtT9A7bHc5ZNE4hn7PxHPmsWaT9GpfCkmZASK7ApN8r' >>> # Customised random string - 12 characters using only the characters `abcdef12345` >>> custom = random_str(12, chars='abcdef12345') 'aba4cc14a43d'
Warning: As this relies on the OS’s entropy features, it may not be cryptographically secure on non-Linux platforms:
> The returned data should be unpredictable enough for cryptographic applications, though its exact quality > depends on the OS implementation.
-
privex.helpers.common.
reverse_io
(f: BinaryIO, blocksize: int = 4096) → Generator[bytes, None, None][source]¶ Read file as series of blocks from end of file to start.
The data itself is in normal order, only the order of the blocks is reversed. ie. “hello world” -> [“ld”,”wor”, “lo “, “hel”] Note that the file must be opened in binary mode.
Original source: https://stackoverflow.com/a/136354
-
privex.helpers.common.
shell_quote
(*args: str) → str[source]¶ Takes command line arguments as positional args, and properly quotes each argument to make it safe to pass on the command line. Outputs a string containing all passed arguments properly quoted.
Uses
shlex.join()
on Python 3.8+, and a for loop ofshlex.quote()
on older versions.Example:
>>> print(shell_quote('echo', '"orange"')) echo '"orange"'
-
privex.helpers.common.
stringify
(data: Optional[Union[str, bytes]], encoding='utf-8', if_none=None) → str[source]¶ Convert a piece of data into a string (from bytes) if it isn’t already:
>>> stringify(b"hello world") "hello world"
By default, if
data
isNone
, thenNone
will be returned.If you’d rather convert
None
into a blank string, useif_node=""
, like so:>>> repr(stringify(None)) 'None' >>> stringify(None, if_none="") ''
-
privex.helpers.common.
strip_null
(value: Union[str, bytes], conv: Callable[[str], Union[str, bytes, T]] = <function stringify>, nullc='\x00') → Union[str, bytes, T][source]¶ Small convenience function which
stringify()
’svalue
then strips it of whitespace and null bytes, with two passes for good measure.- Parameters
value (str|bytes) – The value to clean whitespace/null bytes out of
conv (callable) – (Default
stringify()
) Optionally, you can override the casting function used after the stripping is completednullc (str) – (Default:
\
) Null characters to remove
- Return str|bytes|T cleaned
The cleaned up
value
-
privex.helpers.common.
tail
(filename: str, nlines: int = 20, bsz: int = 4096) → List[str][source]¶ Pure python equivalent of the UNIX
tail
command. Simply pass a filename and the number of lines you want to load from the end of the file, and aList[str]
of lines (in forward order) will be returned.This function is simply a wrapper for the highly efficient
io_tail()
, designed for usage with a small (<10,000) amount of lines to be tailed. To allow for the lines to be returned in the correct order, it must load allnlines
lines into memory before it can return the data.If you need to
tail
a large amount of data, e.g. 10,000+ lines of a logfile, you should consider using the lower level functionio_tail()
- which acts as a generator, only loading a certain amount of bytes into memory per iteration.Example file
/tmp/testing
:this is an example 1 this is an example 2 this is an example 3 this is an example 4 this is an example 5 this is an example 6
Example usage:
>>> from privex.helpers import tail >>> lines = tail('/tmp/testing', nlines=3) >>> print("\n".join(lines)) this is an example 4 this is an example 5 this is an example 6
- Parameters
filename (str) – Path to file to tail. Relative or absolute path. Absolute path is recommended for safety.
nlines (int) – Total number of lines to retrieve from the end of the file
bsz (int) – Block size (in bytes) to load with each iteration (default: 4096 bytes). DON’T CHANGE UNLESS YOU UNDERSTAND WHAT THIS MEANS.
- Return List[str] lines
The last ‘nlines’ lines of the file ‘filename’ - in forward order.
-
privex.helpers.common.
typing_to_base
(tp, fail=False, return_orig=True, clean_union=True) → Optional[Union[type, object, callable, tuple, Tuple[type]]][source]¶ Attempt to extract one or more native Python base types from a
typing
type, including generics such asList[str]
, and combined types such asUnion[list, str]
>>> typing_to_base(List[str]) list >>> typing_to_base(Union[str, Dict[str, list], int]) (str, dict, int) >>> typing_to_base(Union[str, Dict[str, list], int], clean_union=False) (str, typing.Dict[str, list], int) >>> typing_to_base(str) str >>> typing_to_base(str, fail=True) TypeError: Failed to extract base type for type object: <class 'str'> >>> repr(typing_to_base(str, return_orig=False)) 'None'
- Parameters
tp – The
typing
type object to extract base/native type(s) from.fail (bool) – (Default:
False
) If True, then raisesTypeError
iftp
doesn’t appear to be atyping
type.return_orig (bool) – (Default:
True
) If True, returnstp
as-is if it’s not a typing type. WhenFalse
, non-typing
types will causeNone
to be returned.clean_union (bool) – (Default:
True
) If True,typing.Union
’s will have each type converted/validated into a normal type usingextract_type()
- Return type_res
Either a
type
base type, atuple
of types, atyping
type object, or something else depending on what typetp
was.