b2luigi.core.utils module

b2luigi.core.utils.add_on_failure_function(task)

b2luigi.core.utils.create_cmd_from_task(task)

b2luigi.core.utils.create_output_dirs(task)

Create all output dicts if needed. Normally only used internally.

b2luigi.core.utils.create_output_file_name(task, base_filename, create_folder=False, result_path=None)

b2luigi.core.utils.fill_kwargs_with_lists(**kwargs)

Return the kwargs with each value mapped to [value] if not a list already.

Example: .. code-block:: python

>>> fill_kwargs_with_lists(arg_1=[1, 2], arg_2=3)
{'arg_1': [1, 2], 'arg_2': [3]}

Parameters:kwargs – The input keyword arguments Returns:Same as kwargs, but each value mapped to a list if not a list already

b2luigi.core.utils.filter_from_params(output_files, **kwargs)

b2luigi.core.utils.flatten_to_dict(inputs)

Return a whatever input structure into a dictionary. If it is a dict already, return this. If is is an iterable of dict or dict-like objects, return the merged dictionary. All non-dict values will be turned into a dictionary with value -> {value: value}

Example: .. code-block:: python

>>> flatten_to_dict([{"a": 1, "b": 2}, {"c": 3}, "d"])
{'a': 1, 'b': 2, 'c': 3, 'd': 'd'}

Parameters:inputs – The input structure Returns:A dict constructed as described above.

b2luigi.core.utils.flatten_to_file_paths(inputs)

Take in a structure of something and replace each luigi target by its corresponding path. For dicts, it will replace the value as well as the key. The key will however only by the basename of the path.

Parameters:inputs – A dict or a luigi target Returns:A dict with the keys replaced by the basename of the targets and the values by the full path

b2luigi.core.utils.flatten_to_list_of_dicts(inputs)

b2luigi.core.utils.get_all_output_files_in_tree(root_module, key=None)

b2luigi.core.utils.get_filled_params(task)

Helper function for getting the parameter list with each parameter set to its current value

b2luigi.core.utils.get_log_file_dir(task)

b2luigi.core.utils.get_serialized_parameters(task)

Get a string-typed ordered dict of key=value for the significant parameters

b2luigi.core.utils.get_task_from_file(file_name, task_name, **kwargs)

b2luigi.core.utils.on_failure(self, exception)

b2luigi.core.utils.product_dict(**kwargs)

Cross-product the given parameters and return a list of dictionaries.

Example: .. code-block:: python

>>> list(product_dict(arg_1=[1, 2], arg_2=[3, 4]))
[{'arg_1': 1, 'arg_2': 3}, {'arg_1': 1, 'arg_2': 4}, {'arg_1': 2, 'arg_2': 3}, {'arg_1': 2, 'arg_2': 4}]

The thus produced list can directly be used as inputs for a required tasks:

Parameters:kwargs – Each keyword argument should be an iterable Returns:A list of kwargs where each list of input keyword arguments is cross-multiplied with every other.

b2luigi.core.utils.remember_cwd()

Helper contextmanager to stay in the same cwd

b2luigi.core.utils.task_iterator(task, only_non_complete=False)