LIBPSIO --- The New PSIO I/O Library

# Programmer's Manual for LIBPSIO: The New PSI I/O Library

T. Daniel Crawford
Version: October 22, 1998
crawdad@ccqc.uga.edu

### I. The structure and philosophy of the library

In order to access data in the original PSI I/O library (found in `libciomr` and `io`), the programmer is required to know the byte-by-byte layout of the given direct access file. Accordingly, the primary read and write functions [`wreadw()` and `wwritw()`, respectively] require as an argument a global bytewise file pointer to the beginning of the desired data. As a result, when this pointer is defined to be an unsigned four-byte integer (common on 32-bit computers), the total size of the direct access file is limited to 4 GB (232 bytes). Furthermore, in order to avoid code duplication, this I/O design requires that one construct specialized libraries of functions (e.g., `libfile30` or `libfile34`) for interaction with particularly complicated files such as file30 (the PSI checkpoint file) or file34 (the AO-/SO-basis integral file). Even slight modification of the file layout can easily require substantial changes to these libraries.

This new I/O library, `libpsio`, is intended to overcome these problems in two ways:

- Each file makes use of its own table of contents (TOC) which contains file-global starting and ending addresses for each data item.
- Addresses to data items in the TOC are provided to the standard read and write functions by the programmer as entry-relative page/offset pairs, rather than file-global bytewise file pointers.

Data items in the TOC are identified by keyword strings (e.g., `"Nuclear Repulsion Energy"`). If the programmer wishes to read or write an entire TOC entry, he/she is required to provide only the TOC keyword and the entry size (in bytes) to obtain the data; the entry's global starting address is supplied by the TOC. Furthermore, it is possible to read pieces of TOC entries (say a single buffer of a large list of two-electron integrals) by providing the appropriate TOC keyword, a size, and a starting address relative to the beginning of the TOC entry. In short, the TOC design hides all information about the global structure of the direct access file from the programmer and allows him/her to be concerned only with the structure of individual entries. The current TOC is written to the end of the file when it is closed.

The direct-access file itself is viewed as a series of pages, each of which contains an identical number of bytes. The global address of the beginning of a given entry is stored on the TOC as a page/offset pair comprised of the starting page and byte-offset on that page where the data reside. The entry-relative page/offset addresses which the programmer must provide work in exactly the same manner, but the 0/0 position is taken to be the beginning of the TOC entry rather than the beginning of the file.

### II. The user interface

All of the functions needed to carry out basic I/O are described in this subsection. Proper declarations of these routines are provided by the header file `psio.h`. Note that before any open/close functions may be called, the input parsing library, `libipv1` must be initialized so that the necessary file striping information may be read from user input. (See the PSI programmer's manual for details on the current version of the input parser.) Also note that `ULI` is used as an abbreviation for `unsigned long int` in the remainder of this manual.

`int psio_init(void)`: Before any files may be opened or the basic read/write functions of `libpsio` may be used, the global data needed by the library functions must be initialized using this function.

`int psio_done(void)`: When all interaction with the direct-access files is complete, this function is used to free the library's global memory.

`int psio_open(ULI unit, int status)`: Opens the direct access file identified by `unit`. The `status` flag is a boolean used to indicate if the file is new (0) or if it already exists and is being re-opened (1). If specified in the user input file, the file will be automatically opened as a multivolume (striped) file, and each page of data will be read from or written to each volume in succession.

`int psio_close(ULI unit, int keep)`: Closes a direct access file identified by `unit`. The `keep` flag is a boolean used to indicate if the file's volumes should be deleted (0) or retained (1) after being closed.

`int psio_read_entry(ULI unit, char *key, char *buffer, ULI size)`: Used to read an entire TOC entry identified by the string `key` from `unit` into the array `buffer`. The number of bytes to be read is given by `size`, but this value is only used to ensure that the read request does not exceed the end of the entry. If the entry does not exist, an error is printed to `stderr` and the program will exit.

`int psio_write_entry(ULI unit, char *key, char *buffer, ULI size)`: Used to write an entire TOC entry idenitified by the string `key` to `unit` into the array `buffer`. The number of bytes to be written is given by `size`. If the entry already exists and its data is being overwritten, the value of `size` is used to ensure that the write request does not exceed the end of the entry.

`int psio_read(ULI unit, char *key, char *buffer, ULI size, psio_address sadd, psio_address *eadd)`: Used to read a fragment of `size` bytes of a given TOC entry identified by `key` from `unit` into the array `buffer`. The starting address is given by the `sadd` and the ending address (that is, the entry-relative address of the next byte in the file) is returned in `*eadd`.

`int psio_write(ULI unit, char *key, char *buffer, ULI size, psio_address sadd, psio_address *eadd)`: Used to write a fragment of `size` bytes of a given TOC entry identified by `key` to `unit` into the array `buffer`. The starting address is given by the `sadd` and the ending address (that is, the entry-relative address of the next byte in the file) is returned in `*eadd`.

The page/offset address pairs required by the preceeding read and write functions are supplied via variables of the data type `psio_address`, defined by:

`typedef struct {
ULI page;
ULI offset;
} psio_address;
`
The `PSIO_ZERO` defined in macro provides a convenient input for the 0/0 page/offset.

### III. Manipulating the table of contents

In addition, to the basic open/close/read/write functions described above, the programmer also has a limited ability to directly manipulate or examine the data in the TOC itself.

`int psio_tocprint(ULI unit, FILE *outfile)`: Prints the TOC of `unit` in a readable form to `outfile`, including entry keywords and starting/ending addresses.

`int psio_toclen(ULI unit, FILE *outfile)`: Returns the number of entries in the TOC of `unit`.

`int psio_tocdel(ULI unit, char *key)`: Deletes the TOC entry corresponding to `key`. NB that this function only deletes the entry's reference from the TOC itself and does not remove the corresponding data from the file. Hence, it is possible to introduce data "holes" into the file.

`int psio_tocclean(ULI unit, char *key)`: Deletes the TOC entry corresponding to `key` and all subsequent entries. As with `psio_tocdel()`, this function only deletes the entry references from the TOC itself and does not remove the corresponding data from the file. This function is still under construction.

### IV. Some simple examples

The following code illustrates the basic use of the library, as well as when/how the `psio_init()` and `psio_done()` functions should be called in relation to initialization of `libipv1`.

    #include <stdio.h>
    #include <libipv1/ip_lib.h>
    #include <libpsio/psio.h>
    #include <libciomr/libciomr.h>

    FILE *infile, *outfile;

    int main()
    {
      int i, M, N;
      double enuc, *some_data;
      psio_address next;  /* Special page/offset structure */

      ffile(&infile,"input.dat",2);
      ffile(&outfile,"output.dat",1);
      ip_set_uppercase(1);
      ip_initialize(infile,outfile);
      ip_cwk_add(":DEFAULT");
      ip_cwk_add(progid);

      /* Initialize the I/O system */
      psio_init();

      /* Open the file and write an energy */
      psio_open(31, PSIO_OPEN_NEW);
      enuc = 12.3456789;
      psio_write_entry(31, "Nuclear Repulsion Energy", (char *) &enuc,
                       sizeof(double));
      psio_close(31,1);

      /* Read M rows of an MxN matrix from a file */
      some_data = init_matrix(M,N);

      psio_open(91, PSIO_OPEN_OLD);
      next = PSIO_ZERO;/* Note use of the special macro */
      for(i=0; i < M; i++)
          psio_read(91, "Some Coefficients", (char *) (some_data + i*N),
                    N*sizeof(double), next, &next);
      psio_close(91,0);

      /* Close the I/O system */
      psio_done();

      ip_done();
    }

    char *gprgid()
    {
       char *prgid = "CODE_NAME";
       return(prgid);
    }

### V. A few caveats

The interface to the new I/O system has been designed to mimic that of the old `wreadw()` and `wwritw()` routines of libciomr. The table of contents system introduces a few complications that users of the library should be aware of:

- As pointed out earlier, deletion of TOC entries is allowed using `psio_tocdel()` and `psio_tocclean()`. However, since only the TOC reference is removed from the file and the corresponding data is not, a data hole will be left in the file if the deleted entry was not the last one in the TOC. A utility function designed to "defrag" a PSI file may become necessary if such holes ever present a problem.
- One may append data to an existing TOC entry by simply writing beyond the entry's current boundary; the ending address data in the TOC will be updated automatically. However, no safety measures have been implemented to prevent one from overwriting data in a subsequent entry thereby corrupting the TOC. This feature/bug remains because (1) it is possible that such error checking functions may slow the I/O codes significantly; (2) it may be occasionally desirable to overwrite exiting data, regardless of its effect on the TOC. Eventually a utility function which checks the validity of the TOC may be needed if this becomes a problem, particularly for debugging purposes.

Last modified: Fri June 7 2002