meticulous.experiment

class meticulous.experiment.Experiment(args: Dict, default_args: Dict = {}, project_directory: str = '', experiments_directory: str = 'experiments', experiment_id=None, description: str = '', norecord: bool = False)

Class to keep track and store an experiment’s configurations, the code version (via git) and the summary results

__init__(args: Dict, default_args: Dict = {}, project_directory: str = '', experiments_directory: str = 'experiments', experiment_id=None, description: str = '', norecord: bool = False)

Setup the experiment configuration

1. Find a git repo by looking at the project and its parent directories

2. Throws an error if the repo is dirty (has uncommitted tracked files)

3. Creates the experiments directory if it doesn’t exist

4. Extracts metadata from the git repo

5. If experiment_id is provided sets that as the current experiment

   If such an experiment exists, then checks if it exactly matches the arguments and the git sha. If not, throws an error. Otherwise, it resumes that experiment by setting it as the current experiment.

6. Saves experiment info

7. Redirects stdout and stderr to the experiment directory

8. Creates the STATUS file inside experiment directory

Parameters

• args (dict) – Arguments to the program

• default_args (dict) – Default values of the arguments. If supplied helps display experiments using differentiating arguments

• project_directory (str) – Path to the project directory, should be part of a git repo

• experiments_directory (str) – Path to the directory that stores experiments. If a relative path is specified then it is relative to the project directory. Created if it doesn’t exist

• experiment_id –

  Explicitly specified experiment id used for naming experiment folder. If the folder exists (i.e. experiment was run previously), then,

  checks for matching args and githead-sha before resuming,

  otherwise, creates a new experiment folder

• description (str) – Descriptor for the experiment

• norecord (bool) – If true, it skips the entire process and does not record the experiment

metadata

Metadata stored to metadata.json

Type

dict

curexpdir

Path to the directory for the current experiment

Type

str

static add_argument_group(parser, project_directory='', experiments_directory='experiments', experiment_id=None, description='', norecord=False)

Add the meticulous arguments to argparse as a separate group

Parameters

• parser – An argparse.ArgumentParser object

• project_directory – default for –project-directory argument

• experiments_directory – default for –experiments-directory argument

• experiment-id – default for –experiment-id argument

• description – default for –description argument

• norecord – default for –norecord argument

static extract_meticulous_args(parser, arg_list=None)

Extract meticulous specific arguments from argparse parser and return them as a dictionary

Parameters

• parser – An argparse.ArgumentParser object

• arg_list – list of arguments, default is sys.argv[1:]

Returns

Dictionary of meticulous specific arguments

classmethod from_parser(parser, arg_list=None, **default_meticulous_args)

Extract meticulous specific arguments from argparse parser and return an Experiment object

Parameters

• parser – argparse parser

• arg_list – list of arguments, default is sys.argv[1:]

• **meticulous_args – any other args for constructing Experiment object that may not be in the parser

Returns

Experiment object

summary(summary_dict: Dict)

Takes a dictionary object score and (over)writes it in the experiment directory

open(*args, **kwargs)

wrapper around the function open to redirect relative paths to experiment directory

exception meticulous.experiment.DirtyRepoException

Raised when the repo is dirty

exception meticulous.experiment.MismatchedArgsException

Raised when attempting to resume an experiment with different argument values

exception meticulous.experiment.MismatchedCommitException

Raised when attempting to resume an experiment with different git commit