htools package

Submodules

htools.config module

htools.config.get_credentials(from_email)

Get the user’s password for a specified email address.

Parameters

from_email (str) – The email address to get the password for.

Returns

str or None – it as a string. Otherwise, return None.

Return type

If a password is found for the specified email address, return

htools.config.get_default_user()

Get user’s default email address. If one has not been set, user has the option to set it here.

Returns

str or None – declines to specify one, None is returned.

Return type

A string containing the default email address. If user

htools.core module

htools.core.Args(**kwargs)

Wrapper to easily create a named tuple of arguments. Functions sometimes return multiple values, and we have a few options to handle this: we can return them as a regular tuple, but it is often convenient to be able to reference items by name rather than position. If we want the output to be mutable, we can return a dictionary, but this still requires more space than a tuple and bracket notation is arguably less convenient than dot notation. We can create a new namedtuple inside the function, but this kind of seems like overkill to create a new type of namedtuple for each function.

Instead, this lets us create a namedtuple of Args on the fly just as easily as making a dictionary.

Examples

def math_summary(x, y):

sum_ = x + y prod = x * y diff = x - y quotient = x / y return Args(sum=sum_,

product=prod, difference=diff, quotient=quotient)

>>> results = math_summary(4, 2)
>>> results.product

8

>>> results.quotient

2

>>> results

Args(sum=6, product=8, difference=2, quotient=2)

exception htools.core.InvalidArgumentError

Bases: Exception

class htools.core.LambdaDict(default_function)

Bases: dict

Create a default dict where the default function can accept parameters. Whereas the defaultdict in Collections can set the default as int or list, here we can pass in any function where the key is the parameter.

htools.core.catch(func, *args, verbose=False)

Error handling for list comprehensions. In practice, it’s recommended to use the higher-level robust_comp() function which uses catch() under the hood.

Parameters
  • func (function) –

  • *args (any type) – Arguments to be passed to func.

  • verbose (bool) – If True, print the error message should one occur.

Returns

any type – Otherwise, return None.

Return type

If the function executes successfully, its output is returned.

Examples

[catch(lambda x: 1 / x, i) for i in range(3)] >>> [None, 1.0, 0.5]

# Note that the filtering method shown below also removes zeros which is # okay in this case. list(filter(None, [catch(lambda x: 1 / x, i) for i in range(3)])) >>> [1.0, 0.5]

htools.core.dict_sum(*args)
Given two or more dictionaries with numeric values, combine them into a

single dictionary. For keys that appear in multiple dictionaries, their corresponding values are added to produce the new value.

This differs from combining two dictionaries in the following manner:

{**d1, **d2}

The method shown above will combine the keys but will retain the value from d2, rather than adding the values from d1 and d2.

Parameters

*args (dicts) – 2 or more dictionaries with numeric values.

Returns

dict – passed in. The corresponding values from each dictionary containing a given key are summed to produce the new value.

Return type

Contains all keys which appear in any of the dictionaries that are

Examples

>>> d1 = {'a': 1, 'b': 2, 'c': 3}
>>> d2 = {'a': 10, 'c': -20, 'd': 30}
>>> d3 = {'c': 10, 'd': 5, 'e': 0}
>>> dict_sum(d1, d2)

{‘a’: 11, ‘b’: 2, ‘c’: -7, ‘d’: 35, ‘e’: 0}

htools.core.differences(obj1, obj2, methods=False, **kwargs)

Find the differences between two objects of the same type. This is a way to get more detail beyond whether two objects are equal or not.

Parameters
  • obj1 (any type) – An object.

  • obj2 (same type as obj1) – An object.

  • methods (bool) –

    If True, include methods in the comparison. If False, only attributes will be compared. Note that the output may not be particularly interpretable when using method=True; for instance when comparing two strings consisting of different characters, we get a lot of output that looks like this:

    {‘islower’: (<function str.islower()>, <function str.islower()>), ‘isupper’: (<function str.isupper()>, <function str.isupper()>),… ‘istitle’: (<function str.istitle()>, <function str.istitle()>)}

    These attributes all reflect the same difference: if obj1 is ‘abc’ and obj2 is ‘def’, then ‘abc’ != ‘def’ and ‘ABC’ != ‘DEF’ abd ‘Abc’ != ‘Def’.

    When method=False, we ignore all of these, such that differences(‘a’, ‘b’) returns {}. Therefore, it is important to carefully consider what differences you care about identifying.

  • **kwargs (bool) – Can pass args to hdir to include magics or internals.

Returns

dict[str, tuple] – first is the corresponding value for obj1 and the second is the corresponding value for obj2.

Return type

Maps attribute name to a tuple of values, where the

htools.core.eprint(arr, indent=2, spacing=1)

Enumerated print. Prints an iterable with one item per line accompanied by a number specifying its index in the iterable.

Parameters
  • arr (iterable) – The object to be iterated over.

  • indent (int) – Width to assign to column of integer indices. Default is 2, meaning columns will line up as long as <100 items are being printed, which is the expected use case.

  • spacing (int) – Line spacing. Default of 1 will print each item on a new line with no blank lines in between. Spacing of 2 will double space output, and so on for larger values.

Returns

Return type

None

htools.core.flatten(nested)

Flatten a nested sequence where the sub-items can be sequences or primitives. This differs slightly from itertools chain methods because those require all sub-items to be sequences. Here, items can be primitives, sequences, nested sequences, or any combination of these. This also returns a list rather than a generator.

Parameters

nested (sequence (list, tuple, set)) – Sequence where some or all of the items are also sequences.

Returns

list

Return type

Flattened version of nested.

htools.core.hdir(obj, magics=False, internals=False)

Print object methods and attributes, by default excluding magic methods.

Parameters
  • obj (any type) – The object to print methods and attributes for.

  • magics (bool) – Specifies whether to include magic methods (e.g. __name__, __hash__). Default False.

  • internals (bool) – Specifies whether to include internal methods (e.g. _dfs, _name). Default False.

Returns

Keys are method/attribute names, values are strings specifying whether the corresponding key is a ‘method’ or an ‘attr’.

Return type

dict

htools.core.hsplit(text, sep, group=True, attach=True)

Flexible string splitting that retains the delimiter rather, unlike the built-in str.split() method.

Parameters
  • text (str) – The input text to be split.

  • sep (str) – The delimiter to be split on.

  • group (bool) – Specifies whether to group consecutive delimiters together (True), or to separate them (False).

  • attach (bool) – Specifies whether to attach the delimiter to the string that preceeds it (True), or to detach it so it appears in the output list as its own item (False).

Returns

Return type

list[str]

Examples

text = “Score – Giants win 6-5” sep = ‘-‘

# Case 0.1: Delimiters are grouped together and attached to the preceding word. >> hsplit(text, sep, group=True, attach=True) >> [‘Score –’, ‘ Giants win 6-‘, ‘5’]

# Case 0.2: Delimiters are grouped together but are detached from the preceding word, instead appearing as their own item in the output list. >> hsplit(text, sep, group=True, attach=False) >> [‘Score ‘, ‘–’, ‘ Giants win 6’, ‘-‘, ‘5’]

Case 1.1: Delimiters are retained and attached to the preceding string. If the delimiter occurs multiple times consecutively, only the first occurrence is attached, and the rest appear as individual items in the output list. >> hsplit(text, sep, group=False, attach=True) >> [‘Score -‘, ‘-‘, ‘ Giants win 6-‘, ‘5’]

# Case 1.2: Delimiters are retained but are detached from the preceding string. Each instance appears as its own item in the output list. >> hsplit(text, sep, group=False, attach=False) >> [‘Score ‘, ‘-‘, ‘-‘, ‘ Giants win 6’, ‘-‘, ‘5’]

htools.core.load(path, verbose=True)

Wrapper to load pickled (optionally zipped) or json data.

Parameters
  • path (str) – File to load. File type will be inferred from extension.

  • verbose (bool, optional) – If True, will print message stating where object was loaded from.

Returns

object

Return type

The Python object that was pickled to the specified file.

htools.core.print_object_sizes(space, limit=None, exclude_underscore=True)

Print the object names and sizes of the currently defined objects.

Parameters
  • space (dict) – locals(), globals(), or vars()

  • limit (int or None) – Optionally limit the number of objects displayed (default None for no limit).

  • exclude_underscore (bool) – Determine whether to exclude objects whose names start with an underscore (default True).

htools.core.quickmail(subject, message, to_email, from_email=None)

Send an email.

Parameters
  • from_email (str) – Gmail address being used to send email.

  • to_email (str) – Recipient’s email.

  • subject (str) – Subject line of email.

  • message (str) – Body of email.

Returns

Return type

None

htools.core.robust_comp(func, gen)

List comprehension with error handling. Note that values of None will be removed from the resulting list.

Parameters
  • func (function) – Function to apply to each

  • gen (generator) – The sequence to iterate over. This could also be a list, set, etc.

Returns

Return type

list

Examples

# Notice that >>> robust_comp(lambda x: x/(x-2), range(4)) [-0.0, -1.0, 3.0]

htools.core.save(obj, path, verbose=True)

Wrapper to save data as pickle (optionally zipped) or json.

Parameters
  • obj (any) – Object to pickle.

  • path (str) – File name to save pickled object to. Should end with .pkl, .zip, or .json depending on desired output format. If .zip is used, object will be zipped and then pickled.

  • verbose (bool) – If True, print a message confirming that the data was pickled, along with its path.

Returns

Return type

None

htools.core.tdir(obj, **kwargs)

A variation of the built in dir function that shows the attribute names as well as their types. Methods are excluded as they can change the object’s state.

Parameters
  • obj (any type) – The object to examine.

  • kwargs (bool) – Additional arguments to be passed to hdir. Options are magics and internals. See hdir documentation for more information.

Returns

  • dict[str, type] (Dictionary mapping the name of the object’s attributes to)

  • the corresponding types of those attributes.

htools.magics module

htools.meta module

class htools.meta.AutoInit

Bases: object

Mixin class where child class has a long list of init arguments where the parameter name and the class attribute will be the same. Note that *args are not supported in the init method because each attribute that is defined in the resulting object must have a name. A variable length list of args can still be passed in as a single argument, of course, without the use of star unpacking.

This updated version of AutoInit is slightly more user friendly than in V1 (no more passing locals() to super()) but also slower and probably requires more testing (all because of the frame hack in the init method). Note that usage differs from the AutoInit present in htools<=2.0.0, so this is a breaking change.

Examples

Without AutoInit:

class Child:
def __init__(self, name, age, sex, hair, height, weight, grade, eyes):

self.name = name self.age = age self.sex = sex self.hair = hair self.height = height self.weight = weight self.grade = grade self.eyes = eyes

def __repr__(self):

return f’Child(name={self.name}, age={self.age}, sex={self.sex}, ‘ f’hair={self.hair}, weight={self.weight}, ‘ f’grade={self.grade}, eyes={self.eyes})’

With AutoInit:

class Child(AutoInit):
def __init__(self, name, age, sex, hair, height, weight, grade, eyes):

super().__init__()

Note that we could also use the following method, though this is less informative when constructing instances of the child class and does not have the built in __repr__ that comes with AutoInit:

class Child:
def __init__(self, **kwargs):

self.__dict__.update(kwargs)

class htools.meta.Callback

Bases: abc.ABC

Abstract base class for callback objects to be passed to @callbacks decorator. Children must implement a __call__ method that accepts function inputs and outputs.

class htools.meta.ReadOnly(name)

Bases: object

Descriptor to make an attribute read-only. This means that once a value has been set, the user cannot change or delete it. Note that read-only attributes must first be created as class variables (see example below). To allow more flexibility, we do allow the user to manually manipulate the instance dictionary.

Examples

class Dog:

breed = ReadOnly(‘breed’) def __init__(self, breed, age):

# Once breed is set in the line below, it cannot be changed. self.breed = breed self.age = age

>>> d = Dog('dalmatian', 'Arnold')
>>> d.breed

‘dalmatian’

>>> d.breed = 'labrador'

PermissionError: Attribute is read-only.

>>> del d.breed

PermissionError: Attribute is read-only.

exception htools.meta.TimeExceededError

Bases: Exception

htools.meta.assert_raises(error)

Context manager to assert that an error is raised. This can be nice if we don’t want to clutter up a notebook with error messages.

Parameters

error (class inheriting from Exception or BaseException) – The type of error to catch, e.g. ValueError.

Examples

# First example does not throw an error. >>> with assert_raises(TypeError) as ar: >>> a = ‘b’ + 6

# Second example throws an error. >>> with assert_raises(ValueError) as ar: >>> a = ‘b’ + 6

AssertionError: Wrong error raised. Expected PermissionError, got TypeError(can only concatenate str (not “int”) to str)

# Third example throws an error because the code inside the context manager # completed successfully. >>> with assert_raises(ValueError) as ar: >>> a = ‘b’ + ‘6’

AssertionError: No error raised, expected PermissionError.

htools.meta.block_timer()

Context manager to time a block of code. This works similarly to @timer but can be used on code outside of functions.

Examples

with block_timer() as bt:

# Code inside the context manager will be timed. arr = [str(i) for i in range(25_000_000)] first = None while first != ‘100’:

arr.pop(0)

class htools.meta.cached_property(func)

Bases: object

Decorator for computationally expensive methods that should only be computed once (i.e. they take zero arguments aside from self and are slow to execute). Lowercase name is used for consistency with more decorators. Heavily influenced by example in Python Cookbook by David Beazley and Brian K. Jones. Note that, as with the @property decorator, no parentheses are used when calling the decorated method.

Examples

class Vocab:

def __init__(self, tokens):

self.tokens = tokens

@cached_property def embedding_matrix(self):

print(‘Building matrix…’) # Slow computation to build and return a matrix of word embeddings. return matrix

# First call is slow. >>> v = Vocab(tokens) >>> v.embedding_matrix

Building matrix… [[.03, .5, .22, .01],

[.4, .13, .06, .55] [.77, .14, .05, .9]]

# Second call accesses attribute without re-computing # (notice no “Building matrix” message). >>> v = Vocab(tokens) >>> v.embedding_matrix

[[.03, .5, .22, .01],

[.4, .13, .06, .55] [.77, .14, .05, .9]]

htools.meta.callbacks(*, on_begin=None, on_end=None)

Decorator that attaches callbacks to a function. Callbacks should be defined as classes inheriting from abstract base class Callback that implement a __call__ method. This allows us to store states rather than just printing outputs or relying on global vars.

Parameters
  • on_begin (list) – Callbacks to be executed before the decorated function.

  • on_end (list) – Callbacks to be executed after the decorated function.

Examples

@callbacks(on_begin=[print_hyperparameters],

on_end=[plot_activation_hist, activation_means, print_output])

def train(**kwargs):

# Train model and return loss.

htools.meta.debug_call(func)

Decorator to help debug function calls. In general, this is not meant to permanently decorate a function, but rather to be used temporarily when debugging a function call. Note that a wrapped function that accepts *args will display a signature including an ‘args’ parameter even though it isn’t a named parameter, because the goal here is to explicitly show which values are being passed to which parameters. This does mean that the printed string won’t be executable code in this case, but that shouldn’t be necessary anyway since it would contain the same call that was just used.

Parameters

func (function) – Function being decorated.

Examples

Occasionally, you might pass arguments to different parameters than you intended. Throwing a debug_call decorator on the function helps you check that the arguments are matching up as expected. For example, the parameter names in the function below have an unexpected order, so you could easily make the following call and expect to get 8. The debug decorator helps catch that the third argument is being passed in as the x parameter.

@debug_call def f(a, b, x=0, y=None, z=4, c=2):

return a + b + c

>>> f(3, 4, 1)
f(a=3, b=4, x=1, y=None, z=4, c=9)
9
htools.meta.timebox(time, strict=True)

Try to execute code for specified amount of time before throwing error. If you don’t want to throw an error, use with a try/except block.

Parameters
  • time (int) – Max number of seconds before throwing error.

  • strict (bool) – If True, timeout will cause an error to be raised, halting execution of the entire program. If False, a warning message will be printed and the timeboxed operation will end, letting the program proceed to the next step.

Examples

with time_box(5) as tb:

x = computationally_expensive_code()

More permissive version: x = step_1() with timebox(5) as tb:

try:

x = slow_step_2()

except TimeExceededError:

pass

htools.meta.timebox_handler(time, frame)
htools.meta.timeboxed(time, strict=True)

Decorator version of timebox. Try to execute decorated function for time seconds before throwing exception.

Parameters
  • time (int) – Max number of seconds before throwing error.

  • strict (bool) – If True, timeout will cause an error to be raised, halting execution of the entire program. If False, a warning message will be printed and the timeboxed operation will end, letting the program proceed to the next step.

Examples

@timeboxed(5) def func(x, y):

# If function does not complete within 5 seconds, will throw error.

htools.meta.timer(func)

Provide conservative time estimate for a function to run. Behavior may not be interpretable for recursive functions.

Parameters

func (function) – The function to time.

Examples

import time

@timer def count_to(x):

for i in range(x):

time.sleep(0.5)

>>> count_to(10)
[TIMER]: count_to executed in approximately 5.0365 seconds.
htools.meta.typecheck(func_=None, **types)

Decorator to enforce type checking for a function or method. There are two ways to call this: either explicitly passing argument types to the decorator, or letting it infer them using type annotations in the function that will be decorated. We allow multiple both usage methods since older versions of Python lack type annotations, and also because I feel the annotation syntax can hurt readability.

Parameters
  • func_ (function) – The function to decorate. When using decorator with manually-specified types, this is None. Underscore is used so that func can still be used as a valid keyword argument for the wrapped function.

  • types (type) – Optional way to specify variable types. Use standard types rather than importing from the typing library, as subscripted generics are not supported (e.g. typing.List[str] will not work; typing.List will but at that point there is no benefit over the standard list).

Examples

In the first example, we specify types directly in the decorator. Notice that they can be single types or tuples of types. You can choose to specify types for all arguments or just a subset.

@typecheck(x=float, y=(int, float), iters=int, verbose=bool) def process(x, y, z, iters=5, verbose=True):

print(f’z = {z}’) for i in range(iters):

if verbose: print(f’Iteration {i}…’) x *= y

return x

>>> process(3.1, 4.5, 0, 2.0)
TypeError: iters must be <class 'int'>, not <class 'float'>.
>>> process(3.1, 4, 'a', 1, False)
z = a
12.4

Alternatively, you can let the decorator infer types using annotations in the function that is to be decorated. The example below behaves equivalently to the explicit example shown above. Note that annotations regarding the returned value are ignored.

@typecheck def process(x:float, y:(int, float), z, iters:int=5, verbose:bool=True):

print(f’z = {z}’) for i in range(iters):

if verbose: print(f’Iteration {i}…’) x *= y

return x

>>> process(3.1, 4.5, 0, 2.0)
TypeError: iters must be <class 'int'>, not <class 'float'>.
>>> process(3.1, 4, 'a', 1, False)
z = a
12.4
htools.meta.validating_property(func, allow_del=False)

Factory that makes properties that perform some user-specified validation when setting values. The returned function must be used as a descriptor to create a class variable before setting the instance attribute.

Parameters
  • func (function) – Function or lambda that accepts a single parameter. This will be used when attempting to set a value for the managed attribute. It should return True if the value is acceptable, False otherwise.

  • allow_del (bool) – If True, allow the attribute to be deleted.

Returns

function – will be used as a descriptor, so it must create a class variable as shown below. In the example, also notice that the name passed to LengthyInt mustt match the name of the variable it is assigned to.

Return type

A property with validation when setting values. Note that this

Examples

LengthyInt = validating_property(

lambda x: isinstance(x, int) and len(str(int)) > 4

)

class Foo:

long = LengthyInt(‘long’) def __init__(self, a, long):

self.a = a self.long = long

>>> foo = Foo(3, 4)

ValueError: Invalid value 4 for argument long.

# No error on instantiation because the argument is a valid LengthyInt. >>> foo = Foo(3, 543210) >>> foo.long

543210

>>> foo = Foo(3, 'abc')
ValueError: Invalid value 'abc' for argument long.

Module contents