utils

This module contains some utility functions

source

HiddenPrints

 HiddenPrints ()

To hide print command when called

source

create_dir

 create_dir (path_dir)

Create directory if needed

source

val2iterable

 val2iterable (val, lsize=1, t='list')

Convert an element (nonlist value) to an iterable. Currently support list and nparray

source

seed_everything

 seed_everything (seed=42)

source

seed_notorch

 seed_notorch (seed=42)

source

callable_name

 callable_name (any_callable:Callable[...,Any])

To get name of any callable

source

check_and_get_attribute

 check_and_get_attribute (obj, attr_name)

source

load_pickle

 load_pickle (fname, parent='pickle_files')

source

save_to_pickle

 save_to_pickle (my_list, fname, parent='pickle_files')

source

check_input_validation

 check_input_validation (df:pandas.core.frame.DataFrame, verbose=True)

source

check_text_leaking

 check_text_leaking (trn_txt:list, test_txt:list, verbose=True)

source

none2emptystr

 none2emptystr (x)

source

lambda_batch

 lambda_batch (inp, feature, func, is_batched)
Details
inp HuggingFace Dataset
feature Feature name.
func The function to apply
is_batched Whether batching is applied

source

lambda_map_batch

 lambda_map_batch (inp, feature, func, is_batched, output_feature='same',
                   is_func_batched=False)
Type Default Details
inp HuggingFace Dataset
feature Feature name.
func The function to apply
is_batched Whether batching is applied
output_feature str same New feature output, if different from ‘feature’. If none, use function’s output feature(s)
is_func_batched bool False Whether the func above only works with batch (mostly sklearn’s)

source

augmentation_stream_generator

 augmentation_stream_generator (dset, text_name, func)

source

func_all

 func_all (x, functions)

source

get_dset_col_names

 get_dset_col_names (dset)

source

hf_filter_dset

 hf_filter_dset (dset, func, is_batched=True, batch_size=1024, num_proc=1)

source

hf_map_dset

 hf_map_dset (dset, func, is_batched=True, batch_size=1024, num_proc=1)

source

resize_model_embeddings

 resize_model_embeddings (model, tokenizer)

source

sigmoid

 sigmoid (x)

*A numerically stable version of the logistic sigmoid function.

Source: assignment3 of cs231n*