Bookkeeper

The bookkeeper is a built-in ViPErLEED utility that automatically runs before and after each ViPErLEED calculation. It moves input and output files to a history directory, keeps the main folder organized, and tracks all relevant changes.

See viperleed bookkeeper for details on how to run the bookkeeper manually.

Most importantly, the bookkeeper automatically moves and updates files PARAMETERS, POSCAR, and VIBROCC. By default, these files are overwritten by the outputs of the previous calculation, so that a new calculation can be started without the need of manually copying the output files.

The bookkeeper has four archiving-related modes:

Archive

Stores the results of the previous calculation into the history directory, and overwrites the input files PARAMETERS, POSCAR, and VIBROCC with the results of the previous calculation stored in the OUT directory. The previous input files are renamed to PARAMETERS_ori, POSCAR_ori, and VIBROCC_ori, respectively. They may be used for comparison with the non-suffixed files. See History organization for more details on how results are stored in history.

Runs automatically at the end of every calculation.

Clear

Removes files belonging to a previous run.[2] This includes all *_ori files, *.log files, as well as the OUT and SUPP directories.

Runs automatically at the start of every calculation.

Discard

Discards the results of the previous run and restores the input files to their original state: PARAMETERS_ori, POSCAR_ori, and VIBROCC_ori are renamed to PARAMETERS, POSCAR, and VIBROCC, respectively. This is useful if the previous run was not successful and you want to start over. Input and output files of the previous run will still be stored in the history directory for reference. A DISCARDED line is added to the Notes section of the corresponding history.info entry.

Needs to be run manually with viperleed bookkeeper --discard.

Discard full

Same as Discard, but also removes the input and output files of the previous run from the history directory as if it had never happened. It also deletes any Tensors/Tensors_<index>.zip and Deltas/Deltas_<index>.zip files that were created during the last run, unless they had been previously used for other calculations. Users are required to explicitly confirm deletion of the files and folders, unless the -y command-line argument is specified.

Needs to be run manually with viperleed bookkeeper --discard-full.

Warning

Running bookkeeper in Discard full mode in a folder that has been previously Cleared will not restore the input files of the last non-removed execution. The files can be manually copied from the SUPP/original_inputs/ directory of the corresponding “main” history folder.

All the modes that store results to history (i.e., Archive, Clear, and Discard) check that the PARAMETERS, POSCAR, and VIBROCC files have not been edited after viperleed.calc was started. If any such file is found, it is renamed by adding an _edited suffix. bookkeeper will warn if any such file is found. It is up to you to decide whether to migrate these edits to the new input files, or whether to delete the *_edited files.

See Other bookkeeper modes for the description of non-archiving-related bookkeeper modes.

When bookkeeper is executed in the root directory of a multi-domain calculation, all the domain subfolders are processed in the same mode, except for those subfolders in which bookkeeper was manually invoked explicitly.

Changed in version 0.13.0: The bookkeeper behavior was overhauled and the names of the modes were changed. See Previous versions of bookkeeper for details concerning differences with respect to earlier versions.

Changed in version 0.13.0: bookkeeper automatically propagates to domain subfolders when executed in the root of a multi-domain calculation.

history.info and notes.txt

In addition to storing results to history, the bookkeeper also creates or updates a plain-text history.info file. The history.info file contains one entry per each viperleed.calc execution that was archived in history (see Listing 32 for an example). Entries are structured as follows:

TENSORS

Which Tensors/Tensors_<index>.zip file was created or used in the viperleed.calc run (may be multiple if reference calculations repeat). None if the run did not involve any Tensors file.

This value may not be accurate for a Full-dynamic optimization. In fact, no Tensors file is used or produced during a Full-dynamic optimization. However, the corresponding history.info entry will use the index of the most recent Tensors/Tensors_<index>.zip present in the root directory at the time bookkeeper runs.

JOB ID

A progressive identifier for each run. Increments by one every time the bookkeeper archives a new calculation to history. Resets to 1 when a new Tensors file is created, so a specific job is identified by the combined (TENSORS, JOB ID) pair (see also History organization). One JOB ID is present for each of the TENSORS.

RUN

List of viperleed.calc work segments that were executed in the run.

TIME

Starting time of the viperleed.calc execution that was archived to history. This serves as a reliable unique identifier, but is not as intuitive as the TENSORS and JOB ID numbers.

Changed in version 0.13.0: Changed the default format of this field to yyyy-mm-dd HH:mm::ss. Earlier versions used dd.mm.yy HH:mm:ss. bookkeeper will add new history.info entries keeping the format consistent, but will warn about the format change. history.info files created with earlier bookkeeper versions can be updated automatically to the new format by running bookkeeper in --fix mode.

R REF/R SUPER (if applicable)

\(R\) factor of the final reference calculation and superpos calculation, if any were run. Includes the total \(R\) factor (i.e., considering all beams), as well as those for integer- and fractional-order beams, if a distinction was specified via the SEARCH_BEAMS parameter.

FOLDER

The (main) folder created for this job in the history directory, i.e., where the bookkeeper moved the primary output files. See History organization for more details concerning how results are stored in history.

Notes

Added manually or inserted automatically. When the bookkeeper runs and a notes or notes.txt file is present in the folder, the contents of that file will be automatically inserted here, and the notes file will be cleared. This allows taking notes concerning a currently running job in advance, without having to wait for it to finish and the bookkeeper to run.

A run that has been --discarded has one DISCARDED line in the Notes field.

Changed in version 0.13.0: Earlier bookkeeper versions could also create a JOB NAME line (after JOB ID) with the string given by the user as the --name (before v0.12.0) or --job-name (v0.12.0–v0.12.2) command-line argument. The command-line argument has been removed in v0.13.0, but bookkeeper will still recognize one such line.

History organization

Whenever bookkeeper archives the results of a viperleed.calc execution to history, it creates one “main” folder with name format tTTT.rRRR_<timestamp>, and may produce additional folders named tTTT.rRRR.SSS_<segments>_<timestamp> containing Intermediate viperleed.calc results. Each folder is labeled by its TENSORS number (TTT), JOB ID (RRR), and the starting time of the viperleed.calc execution (<timestamp>). The latter is the same for both tTTT.rRRR_<timestamp>- and tTTT.rRRR.SSS_<segments>_<timestamp>-named folders.

An example of the history folder is shown in Listing 31, corresponding to a first viperleed.calc execution with RUN = 1-3 1. Listing 32 shows the corresponding entry added by bookkeeper to the history.info file.

The “main” history folder (t002.r001_250321-101512 in Listing 31) collects both the input files (in the root of the folder) as well as the final results of a viperleed.calc execution (i.e., the main *.log file and SUPP/OUT directories).

Listing 31 Example of the contents of history following a viperleed.calc execution with RUN = 1-3 1.
history/
├── t001.r001.001_RDS_250321-101512/  <-- after first search iteration
│   ├── OUT/                          <-- intermediate calc results
│   │   └── ...
│   └── SUPP/                         <-- intermediate calc results
│       └── ...
│
├── t001.r001.002_DS_250321-101512/  <-- after second search iteration
│   ├── OUT/                         <-- intermediate calc results
│   │   └── ...
│   └── SUPP/                        <-- intermediate calc results
│       └── ...
│
└── t002.r001_250321-101512/          <-- "main" history folder
    ├── SUPP/                         <-- calc results at end
    │   ├── original_inputs/
    │   │   ├── EXPBEAMS.csv
    │   │   ├── PARAMETERS
    │   │   ├── PHASESHIFTS
    │   │   └── POSCAR
    │   └── ...
    ├── OUT/                          <-- calc results at end
    │   └── ...
    ├── DISPLACEMENTS_from_root       <-- not found in original_inputs
    ├── EXPBEAMS.csv                  <-- original input file
    ├── PARAMETERS                    <-- original input file
    ├── PHASESHIFTS                   <-- original input file
    ├── POSCAR                        <-- original input file
    └── viperleed-calc_250321-101512.log

The input files stored in a “main” history folder are preferentially collected from the SUPP/original_inputs/ directory. If an input file is not found in SUPP/original_inputs/, it is copied from the directory in which bookkeeper executes and renamed by adding a _from_root suffix. This is meant to prevent obscuring potential user edits to the root files since viperleed.calc was started.

Note

bookkeeper will archive in the main history directory all the input files, irrespective of whether they were actually used during the corresponding calculation.

Listing 32 Contents of the history.info file corresponding to the history of Listing 31.
# TENSORS    1, 2
# JOB ID     1, 1
# RUN        0 1 11 2 3 31 12 2 3 31 12 1 11
# TIME       2025-03-21 10:15:12
# R REF      <R factor after last refcalc>
# R SUPER    <R factor after last superpos>
# FOLDER     t002.r001_250321-101512
Notes:  ...user notes...

###########

Intermediate viperleed.calc results

A similar system as the bookkeeper’s history is used by viperleed.calc during execution. When the order of execution loops “backwards” — i.e., when executing another reference calculation after a search, or running multiple searches consecutively — viperleed.calc creates a workhistory directory and moves there a snapshot of all relevant outputs.

When the bookkeeper runs and a workhistory directory is present, the workhistory contents will be incorporated into the history folder. Directories in the workhistory will be moved to history and renamed to tTTT.rRRR.SSS_<segments>_<timestamp>, following the same basic formatting of the “main” directories in history. SSS is an incremental number for workhistory directories generated during the same run, while <segments> gives a quick overview about what segments were performed before this snapshot was created: Reference calculation, Delta amplitudes, or Search.

Listing 31 shows an example of the history folder after executing viperleed.calc for the first time with RUN = 1-3 1 and with a DISPLACEMENTS file that defines two consecutive search blocks. When the first search is finished, and the job loops around to execute further delta-amplitudes calculations and searches, viperleed.calc collects in a first workhistory folder the intermediate output files. This snapshot appears as folder t001.r001.001_RDS_250321-101512 in history: .001 marks that it is the first such snapshot taken for job .r001, and RDS indicates that a reference calculation, a delta-amplitudes calculation, and a search were executed before the snapshot was taken. The reference calculation generated a Tensors/Tensors_001.zip file (t001). When viperleed.calc concludes the second search block and “loops back” to run the last reference calculation, a second snapshot is taken, yielding folder t001.r001.002_DS_250321-101512 in history: this second (.002) intermediate snapshot used the same Tensors_001.zip file as the first one (t001) but only executed delta-amplitudes and search segments (DS).

The last reference calculation produces a new a Tensors/Tensors_002.zip file. Its results constitute the final output of this viperleed.calc execution, and are stored in the “main” history folder, named t002.r001_250321-101512.

Changed in version 0.13.0: Earlier versions would assign the incorrect JOB ID to intermediate results of a viperleed.calc execution with RUN = 2-3 1 (and similar). See this comment for details.

Other bookkeeper modes

This section describes non-archiving-related modes for the bookkeeper utility. They require manual invocation of the bookkeeper (see also viperleed bookkeeper).

Fix

Edits the history folder and the history.info file to conform to the most recent format implemented in the bookkeeper, where possible. A list of the automatic edits that mode Fix can apply to the history.info file as of v0.13.0 can be found here. The only modification currently implemented for history folders consists in the addition of metadata (file bookkeeper_meta) used internally by bookkeeper for cross-referencing history folders.

Previous versions of bookkeeper

This section outlines differences that have been introduced in bookkeeper as of v0.13.0 compared to previous versions, and documents related changes in viperleed.calc. bookkeeper has been kept backward compatible as much as possible. However, some breaking changes have been introduced in viperleed.calc that make running bookkeeper in a folder that was created with earlier versions of viperleed.calc (or bookkeeper) not entirely consistent with the results of executing bookkeeper in a folder created with v0.13.0 and later versions.

The most important changes introduced in v0.13.0 concern the storage of user-given input files by viperleed.calc, the behavior of --discard-full mode, and the modification of the default interaction between viperleed.calc and bookkeeper.

Storage of user-given input files

While the user-given input files have been stored in SUPP/original_inputs/ since viperleed.calc v0.7.0, storage would happen too late. As a result, the files saved to SUPP/original_inputs/ had already been processed by viperleed.calc. See this and the following comments for more details.

Since bookkeeper v0.13.0 (and later versions) relies on the “originality” of the files in SUPP/original_inputs/, executing it in a tree created by earlier viperleed.calc versions may potentially (i) archive in history input files that were already processed by viperleed.calc, and (ii) restore the incorrect PARAMETERS, POSCAR, and VIBROCC files when executed in Discard/Discard Full modes. Especially in the latter case, we suggest to carefully validate that the contents of the PARAMETERS, POSCAR, and VIBROCC input files are as intended.

Behavior of --discard-full mode

bookkeeper v0.13.0 introduced a mechanism to cross-reference the “main” history folder to intermediate results (see History organization for details). Before running bookkeeper v0.13.0 (and later) in --discard-full mode in a tree created by earlier bookkeeper versions, make sure to manually invoke bookkeeper in --fix mode. This will add the necessary cross-reference information where possible. Failing to execute bookkeeper --fix before --discard-full will only delete from history the “main” folder of the most recent run, and remove from history.info the corresponding entry. This will leave intermediate-results folders untouched. Additionally, these folders will not have a corresponding history.info entry.

Interaction with viperleed.calc v0.1.0 – v0.11.0

Up to v0.11.0, bookkeeper would automatically execute only before a viperleed.calc run. It would, at that point, archive to history/history.info any results of earlier viperleed.calc executions. Then, it would remove any outputs of the previous run from the root folder. However, at that moment input files could have been modified relative to the ones originally given when the previous viperleed.calc execution started. Such edited input files would then be archived to history.

The intended workflow for v0.1.0 – v0.11.0 that would give the correct storage of files to history was:

  1. Execute viperleed.calc.

  2. Manually execute bookkeeper in either “default” (i.e., the current Clear mode), --discard (i.e., the current Discard Full mode), or --cont modes (i.e., the current Archive mode). This manual execution would need to be done before any edit to input files.

  3. Start a new viperleed.calc session.

Failing to manually execute bookkeeper after viperleed.calc would cause the following viperleed.calc execution to always start from the same inputs as the previous one. Manually executing bookkeeper after having introduced any edit to the input files would cause storage of the incorrect input files to history. bookkeeper v0.13.0 cannot detect the latter scenario.

Interaction with viperleed.calc v0.12.0 – v0.12.2

In v0.12, bookkeeper would still automatically execute before a viperleed.calc run as in previous versions. Additionally, it would also run in --cont mode after a viperleed.calc run.

The rationale behind this choice is that it was easy to forget to manually execute bookkeeper after viperleed.calc and before editing the input files in the root directory. As discussed in the previous sections, this would then likely store the incorrect files to history.

This choice, however, turned out to be fairly poor for a number of reasons:

  1. The results of a calculation would be “hidden away” inside the history subfolder, rather than being readily available in the folder where viperleed.calc was started.

  2. The PARAMETERS, POSCAR, and VIBROCC files in the root directory after execution would be the ones at the end, potentially after a structure optimization. This would make comparison of the progress of a calculation not straightforward.

  3. Manually running bookkeeper would have no effect after executing viperleed.calc with default arguments, irrespective of the bookkeeper mode.

Executing bookkeeper v0.13.0 (and later versions) in a tree created by versions 0.12.0–0.12.2 will not yield the expected results in modes --discard and --discard-full: the input files will not be restored to those at the beginning of the discarded run. These files can, however, be manually copied from the corresponding history folder.