Privex Python Helpers’s documentation¶
Welcome to the documentation for Privex’s Python Helpers - a small, open source Python 3 package containing a variety of functions, classes, exceptions, decorators and more - each of which would otherwise be too small to maintain in an individual package.
This documentation is automatically kept up to date by ReadTheDocs, as it is automatically re-built each time a new commit is pushed to the Github Project
Contents
Quick install¶
Installing with Pipenv (recommended)
pipenv install privex-helpers
Installing with standard pip3
pip3 install privex-helpers
Python Module Overview¶
Privex’s Python Helpers is organised into various sub-modules to make it easier to find the functions/classes you want to use, and to avoid having to load the entire module (though it’s lightweight).
With the exception of privex.helpers.django (Django gets upset if certain django modules are imported before
it’s initialised), all functions/classes are imported within the __init__ file, allowing you to simply type:
from privex.helpers import empty, run_sync, asn_to_name
Instead of having to import the functions from each individual module:
from privex.helpers.common import empty
from privex.helpers.asyncx import run_sync
from privex.helpers.net import asn_to_name
Below is a listing of the sub-modules available in privex-helpers with a short description of what each module
contains.
Functions and classes related to working with Python’s native asyncio support |
|
This module contains somewhat risky code that uses app introspection e.g. |
|
Helper functions/classes related to caching. |
|
Common functions and classes that don’t fit into a specific category |
|
Functions, classes and/or types which either are, or are related to Python variable storage types ( |
|
Various functions/classes which convert/parse objects from one type into another. |
|
Cryptography related helper classes/functions |
|
Class Method / Function decorators |
|
This module file contains Django-specific helper functions, to help save time when developing with the Django framework. |
|
Exception classes used either by our helpers, or just generic exception names which are missing from the standard base exceptions in Python, and are commonly used across our projects. |
|
Various helper functions/classes which depend on a certain package being installed. |
|
Network related helper code |
|
This module handles connection objects for databases, APIs etc. |
|
Configuration options for helpers, and services they depend on, such as Redis. |
|
Helpers for setup.py, e.g. |
|
All Documentation¶
Installation¶
Download and install from PyPi using pip (recommended)¶
pip3 install privex-helpers
(Alternative) Manual install from Git¶
Option 1 - Use pip to install straight from Github
pip3 install git+https://github.com/Privex/python-helpers
Option 2 - Clone and install manually
# Clone the repository from Github
git clone https://github.com/Privex/python-helpers
cd python-helpers
# RECOMMENDED MANUAL INSTALL METHOD
# Use pip to install the source code
pip3 install .
# ALTERNATIVE MANUAL INSTALL METHOD
# If you don't have pip, or have issues with installing using it, then you can use setuptools instead.
python3 setup.py install
Example Usages¶
Boolean testing¶
The empty function¶
The empty() function in our opinion, is one of most useful functions in this library. It allows for a clean,
readable method of checking if a variable is “empty”, e.g. when checking keyword arguments to a function.
With a single argument, it simply tests if a variable is "" (empty string) or None.
The argument itr can be set to True if you consider an empty iterable such as [] or {} as “empty”. This
functionality also supports objects which implement __len__, and also checks to ensure __len__ is available,
avoiding an exception if an object doesn’t support it.
The argument zero can be set to True if you want to consider 0 (integer) and '0' (string) as “empty”.
from privex.helpers import empty
x, y = "", None
z, a = [], 0
empty(x) # True
empty(y) # True
empty(z) # False
empty(z, itr=True) # True
empty(a) # False
empty(a, zero=True) # True
The is_true and is_false functions¶
When handling user input, whether from an environment file (.env), or from data passed to a web API, it can be
a pain attempting to check for booleans.
A boolean True could be represented as the string 'true', '1', 'YES', as an integer 1, or even
an actual boolean True. Trying to test for all of those cases requires a rather long if statement…
Thus is_true() and is_false() were created.
from privex.helpers import is_true, is_false
is_true(0) # False
is_true(1) # True
is_true('1') # True
is_true('true') # True
is_true('false') # False
is_true('orange') # False
is_true('YeS') # True
is_false(0) # True
is_false('false') # True
is_false('true') # False
is_false(False) # True
Handling environmental variables in different formats¶
Using env_csv to support lists contained within an env var¶
The function env_csv() parses a CSV-like environment variable into a list
from privex.helpers import env_csv
import os
os.environ['EXAMPLE'] = "this, is, an,example "
env_csv('EXAMPLE', ['error'])
# returns: ['this', 'is', 'an', 'example']
env_csv('NOEXIST', ['non-existent'])
# returns: ['non-existent']
Using env_keyval to support dictionaries contained within an env var¶
The function env_keyval() parses an environment variable into a ordered list of tuple pairs, which can be
easily converted into a dictionary using dict().
from privex.helpers import env_keyval
import os
os.environ['EXAMPLE'] = "John: Doe , Jane : Doe, Aaron:Smith"
env_keyval('EXAMPLE')
# returns: [('John', 'Doe'), ('Jane', 'Doe'), ('Aaron', 'Smith')]
env_keyval('NOEXIST', {})
# returns: {}
Improved collections, including dict’s and namedtuple’s¶
In our privex.helpers.collections module (plus maybe a few things in privex.helpers.common),
we have various functions and classes designed to make working with Python’s storage types more painless, while
trying to keep compatibility with code that expects the native types.
Dictionaries with dot notation attribute read/write¶
Dictionaries (dict) are powerful, and easy to deal with. But why can’t you read or write dictionary items with
attribute dot notation!?
This is where DictObject comes in to save the day. It’s a child class of python’s native dict type, which
means it’s still compatible with functions/methods such as json.dumps(), and in most cases will be plug-n-play with
existing dict-using code.
Basic usage
from privex.helpers import DictObject
x = dict(hello='world', lorem='ipsum')
x['hello'] # This works with a normal dict
x.hello # But this raises: AttributeError: 'dict' object has no attribute 'hello'
# We can cast the dict 'x' into a DictObject
y = DictObject(x)
y['hello'] # Returns: 'world'
y.hello # Returns: 'world'
# Not only can you access dict keys via attributes, you can also set keys via attributes
y.example = 'testing'
y # We can see below that setting 'example' worked as expected.
# Output: {'hello': 'world', 'lorem': 'ipsum', 'example': 'testing'}
Type checking / Equality comparison
As DictObject is a subclass of dict, you can use isinstance() to check against dict
(e.g. isinstance(DictObject(), dict)) and it should return True.
You can also compare dictionary equality between a DictObject and a dict using == as normal.
y = DictObject(hello='world')
isinstance(y, dict) # You should always use isinstance instead of `type(x) == dict`
# Returns: True
# You can also use typing.Dict with isinstance when checking a DictObject
from typing import Dict
isinstance(y, Dict) # Returns: True
# You can compare equality between a DictObject and a dict with no problems
DictObject(hello='world') == dict(hello='world')
# Returns: True
DictObject(hello='world') == dict(hello='example')
# Returns: False
Type Masquerading
Also included is the class MockDictObj, which is a subclass of DictObject with it’s name, qualified name,
and module adjusted so that it appears to be the builtin dict type.
This may help in some cases, but sadly can’t fool a type(x) == dict check.
from privex.helpers import MockDictObj
z = MockDictObj(y)
type(z) # Returns: <class 'dict'>
z.__class__.__module__ # Returns: 'builtins'
Named Tuple’s (namedtuple) with dict-like key access, dict casting, and writable fields¶
A somewhat simpler version of dict’s are collections.namedtuple()’s
Unfortunately they have a few quirks that can make them annoying to deal with.
Person = namedtuple('Person', 'first_name last_name') # This is an existing namedtuple "type" or "class"
john = Person('John', 'Doe') # This is an existing namedtuple instance
john.first_name # This works on a standard namedtuple. Returns: John
john[1] # This works on a standard namedtuple. Returns: Doe
john['first_name'] # However, this would throw a TypeError.
dict(john) # This would throw a ValueError.
john.address = '123 Fake St' # This raises an AttributeError.
Thus, we created dictable_namedtuple() (and more), which creates namedtuples with additional functionality,
including item/key access of fields, easy casting into dictionaries, and ability to add new fields.
from privex.helpers import dictable_namedtuple
Person = dictable_namedtuple('Person', 'first_name last_name')
john = Person('John', 'Doe')
dave = Person(first_name='Dave', last_name='Smith')
print(dave['first_name']) # Prints: Dave
print(dave.first_name) # Prints: Dave
print(john[1]) # Prints: Doe
print(dict(john)) # Prints: {'first_name': 'John', 'last_name': 'Doe'}
john.address = '123 Fake St' # Unlike normal namedtuple, we can add new fields
print(john) # Prints: Person(first_name='John', last_name='Doe', address='123 Fake St')
You can use convert_dictable_namedtuple() to convert existing namedtuple instancess
into dictable_namedtuple instances:
Person = namedtuple('Person', 'first_name last_name') # This is an existing namedtuple "type" or "class"
john = Person('John', 'Doe') # This is an existing namedtuple instance
d_john = convert_dictable_namedtuple(john)
d_john.first_name # Returns: John
d_john[1] # Returns: Doe
d_john['first_name'] # Returns: 'John'
dict(d_john) # Returns: {'first_name': 'John', 'last_name': 'Doe'}
For more information, check out the module docs at privex.helpers.collections
Functions and classes related to working with Python’s native asyncio support |
|
This module contains somewhat risky code that uses app introspection e.g. |
|
Helper functions/classes related to caching. |
|
Common functions and classes that don’t fit into a specific category |
|
Functions, classes and/or types which either are, or are related to Python variable storage types ( |
|
Various functions/classes which convert/parse objects from one type into another. |
|
Cryptography related helper classes/functions |
|
Class Method / Function decorators |
|
This module file contains Django-specific helper functions, to help save time when developing with the Django framework. |
|
Exception classes used either by our helpers, or just generic exception names which are missing from the standard base exceptions in Python, and are commonly used across our projects. |
|
Various helper functions/classes which depend on a certain package being installed. |
|
Network related helper code |
|
This module handles connection objects for databases, APIs etc. |
|
Configuration options for helpers, and services they depend on, such as Redis. |
|
Helpers for setup.py, e.g. |
|
How to use the unit tests¶
This module contains test cases for Privex’s Python Helper’s (privex-helpers).
Testing pre-requisites¶
Ensure you have any mandatory requirements installed (see setup.py’s install_requires)
You should install
pytestto run the tests, it works much better than standard python unittest.You may wish to install any optional requirements listed in README.md for best results
Python 3.7 is recommended at the time of writing this. See README.md in-case this has changed.
For the best testing experience, it’s recommended to install the dev extra, which includes every optional
dependency, as well as development requirements such as pytest , coverage as well as requirements for
building the documentation.
Running via PyTest¶
To run the tests, we strongly recommend using the pytest tool (used by default for our Travis CI):
# Install PyTest if you don't already have it.
user@host: ~/privex-helpers $ pip3 install pytest
# We recommend adding the option ``-rxXs`` which will show information about why certain tests were skipped
# as well as info on xpass / xfail tests
# You can add `-v` for more detailed output, just like when running the tests directly.
user@host: ~/privex-helpers $ pytest -rxXs
# NOTE: If you're using a virtualenv, sometimes you may encounter strange conflicts between a global install
# of PyTest, and the virtualenv PyTest, resulting in errors related to packages not being installed.
# A simple workaround is just to call pytest as a module from the python3 executable:
user@host: ~/privex-helpers $ python3 -m pytest -rxXs
============================== test session starts ==============================
platform darwin -- Python 3.7.0, pytest-5.2.2, py-1.8.0, pluggy-0.13.0
rootdir: /home/user/privex-helpers
collected 99 items
tests/test_bool.py ......... [ 9%]
tests/test_cache.py ................ [ 25%]
tests/test_crypto.py ......................... [ 50%]
tests/test_general.py ................... [ 69%]
tests/test_net.py ssss.s [ 75%]
tests/test_parse.py .......... [ 85%]
tests/test_rdns.py .............. [100%]
============================ short test summary info ============================
SKIPPED [1] tests/test_net.py:76: Requires package 'dnspython'
SKIPPED [1] tests/test_net.py:83: Requires package 'dnspython'
SKIPPED [1] tests/test_net.py:66: Requires package 'dnspython'
SKIPPED [1] tests/test_net.py:71: Requires package 'dnspython'
SKIPPED [1] /home/user/privex-helpers/tests/test_net.py:56: Skipping test TestGeneral.test_ping_v6 as platform is
not supported: "privex.helpers.net.ping is not fully supported on platform 'Darwin'..."
================== 94 passed, 5 skipped, 1 warnings in 21.66s ===================
Running individual test modules¶
Some test modules such as test_cache can be quite slow, as sometimes it’s required to call sleep, e.g. sleep(2)
either to prevent interference from previous/following tests, or when testing that an expiration/timeout works.
Thankfully, PyTest allows you to run individual test modules like this:
user@host: ~/privex-helpers $ pytest -rxXs -v tests/test_parse.py
============================== test session starts ==============================
platform darwin -- Python 3.7.0, pytest-5.2.2, py-1.8.0, pluggy-0.13.0
cachedir: .pytest_cache
rootdir: /home/user/privex-helpers
plugins: cov-2.8.1
collected 10 items
tests/test_parse.py::TestParseHelpers::test_csv_single PASSED [ 10%]
tests/test_parse.py::TestParseHelpers::test_csv_spaced PASSED [ 20%]
tests/test_parse.py::TestParseHelpers::test_env_bool_false PASSED [ 30%]
tests/test_parse.py::TestParseHelpers::test_env_bool_true PASSED [ 40%]
tests/test_parse.py::TestParseHelpers::test_env_nonexist_bool PASSED [ 50%]
tests/test_parse.py::TestParseHelpers::test_kval_clean PASSED [ 60%]
tests/test_parse.py::TestParseHelpers::test_kval_custom_clean PASSED [ 70%]
tests/test_parse.py::TestParseHelpers::test_kval_custom_spaced PASSED [ 80%]
tests/test_parse.py::TestParseHelpers::test_kval_single PASSED [ 90%]
tests/test_parse.py::TestParseHelpers::test_kval_spaced PASSED [100%]
============================== 10 passed in 0.09s ===============================
Running directly using Python Unittest¶
Alternatively, you can run the tests by hand with python3.7 ( or just python3 ), however we strongly
recommend using PyTest as our tests use various PyTest functionality to allow for things such as skipping tests
when you don’t have a certain dependency installed.
Running via python unittest
user@the-matrix ~/privex-helpers $ python3.7 -m tests
............................
----------------------------------------------------------------------
Ran 28 tests in 0.001s
OK
For more verbosity, simply add -v to the end of the command:
user@the-matrix ~/privex-helpers $ python3 -m tests -v
test_empty_combined (__main__.TestBoolHelpers) ... ok
test_isfalse_truthy (__main__.TestBoolHelpers) ... ok
test_v4_arpa_boundary_16bit (__main__.TestIPReverseDNS)
Test generating 16-bit v4 boundary ... ok
test_v4_arpa_boundary_24bit (__main__.TestIPReverseDNS)
Test generating 24-bit v4 boundary ... ok
test_kval_single (__main__.TestParseHelpers)
Test that a single value still returns a list ... ok
test_kval_spaced (__main__.TestParseHelpers)
Test key:val csv parsing with excess outer whitespace, and value whitespace ... ok
# Truncated excess output in this PyDoc example, as there are many more lines showing
# the results of each individual testcase, wasting space and adding bloat...
----------------------------------------------------------------------
Ran 28 tests in 0.001s
OK
Copyright:
Copyright 2019 Privex Inc. ( https://www.privex.io )
License: X11 / MIT Github: https://github.com/Privex/python-helpers
+===================================================+
| © 2019 Privex Inc. |
| https://www.privex.io |
+===================================================+
| |
| Originally Developed by Privex Inc. |
| |
| Core Developer(s): |
| |
| (+) Chris (@someguy123) [Privex] |
| (+) Kale (@kryogenic) [Privex] |
| |
+===================================================+
Copyright 2019 Privex Inc. ( https://www.privex.io )
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to use,
copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Unit Test List / Overview¶
Various classes / functions / attributes used by test cases (no actual test cases in here) |
|
General test cases for various un-categorized functions / classes e.g. |
|
Test cases for boolean helper functions, such as |
|
Test cases for the cache decorator |
|
Test cases for |
|
Test cases for the |
|
Test cases for |
|
Test cases for parsing functions, such as |
|
A thorough test case for |
|
Test cases related to |