Run Modes

The run mode can be chosen by calling your python file with

python file.py --mode

or by calling b2luigi.process with a given mode set to True

b2luigi.process(.., mode=True)

where mode can be one of:

  • batch: Run the tasks on a batch system, as described in Quick Start. The maximal number of batch jobs to run in parallel (jobs in flight) is equal to the number of workers. This is 1 by default, so you probably want to change this. By default, LSF is used as a batch system. If you want to change this, set the corresponding batch_system setting-label to one of the supported systems.
  • dry-run: Similar to the dry-run funtionality of luigi, this will not start any tasks but just tell you, which tasks it would run. The exit code is 1 in case a task needs to run and 0 otherwise.
  • show-output: List all output files that this has produced/will produce. Files which already exist (where the targets define, what exists mean in this case) are marked as green whereas missing targets are marked red.
  • test: Run the tasks normally (no batch submission), but turn on debug logging of luigi. Also, do not dispatch any task (if requested) and print the output to the console instead of in log files.

Additional console arguments:

  • –scheduler-host and –scheduler-port: If you have set up a central scheduler, you can pass this information here easily. This works for batch or non-batch submission but is turned of for the test mode.