def_file

Read and write *.def file from the IPSL/modipsl project.

The *.def file contains model parameters in the key-value format. The format is extremely simple in comparison to similar formats, like *.ini or *.toml, as it doesn't provide sections, nor standarized datatypes. Usually, model configuration files contain dozens or hundred of parameters with scalars (int, float, str), arrays, or special _AUTO_/_AUTOBLOCK_ values.

Examples usage:

from ipsl_common.modipsl.def_file import load, dump
with open("run_dynamico.def", "r") as f:
    parameters = load(f)
    // Modify loaded parameters
    parameters["start_file_name"] = "start2024.nc"
    with open("new_run_dynamico.def", "w") as g:
        dump(parameters, g)

Loading an example *.def file:

INCLUDEDEF=run_lmdz.def
INCLUDEDEF=run_dynamico.def
use_forcing=y
g=_AUTO_: DEFAULT=9.8
start_file_name=start2023
physics="always"

Gives the following dictionary:

{
    'INCLUDEDEF': ['run_lmdz.def', 'run_dynamico.def'],
    'g': ('_AUTO_', 9.8),
    'physics': '"always"',
    'start_file_name': 'start2023',
    'use_forcing': True
}

Loaded configuration can be altered and subsequently dumped onto a file or into a string. The configuration is easy to view and modify, because it is directly decoded into a Python dictionary. The decoding and encoding process is managed internally by DefFileDecoder and DefFileEncoder classes with decode() and encode() methods.

Classes

DefFileDecoder

DefFileDecoder(include_positions: bool = False)

Bases: Transformer

Decoder performs translation from *.def file to a dictionary.

The translation rules are:

*.def	Python	Comment
key-value	dict	Including many key-value pairs and INCLUDEDEF
array	list	Of at least 2 elements
string	str	Unquoted and quoted (single or double) strings
integer number	int	---
real number	float	Including scientific notation
true/y	True	Case insensitive
false/n	False	Case insensitive
_AUTO_	tuple	With optional default value
_AUTOBLOCKER_	tuple	With optional default value

The first step of the decoder parses *.def file. For this task lark Earley parser is used with a simple grammar expressed with EBNF notation. The *.def grammer doesn't parse well with LALR(1) parser. The parsing produces a parse tree.

The second step transforms the parse tree into Python dictionary using translation rules mentioned in the above table. This transformation is based on a automated visitor pattern called Transformer, which produces the dictionary in a bottom-up manner.

Examples:

from ipsl_common.modipsl.def_file import DefFileDecoder
dictionary = DefFileDecoder().decode(text)

If text contains this *.def file:

radius=6.371229E6
g=9.80665
omega=_AUTO_: DEFAULT=7.292E-5

Then, Python dictionary would look as follows:

{
    "radius": 6.371229e6,
    "g": 9.80665,
    "omega": ("_AUTO_", 7.292e-05),
}

Tip

By default, the result dictionary contains no information about textual layout of the file. However, by using the argument include_positions=True, it is possible to refine the dictionary with exact start/end positions of each value as follows:

{
    "radius": {"value": 6371229.0, "start_pos": 50, "end_pos": 60},
    "g": {"value": 9.80665, "start_pos": 99, "end_pos": 106},
    "omega": {"value": ("_AUTO_", 7.292e-05), "start_pos": 158, "end_pos": 182},
}

Initialize DefFileDecoder.

Parameters:

• include_positions

  (bool, default: False ) –

  include textual positions of values

Source code in ipsl_common/modipsl/def_file.py

def __init__(self, include_positions: bool = False) -> None:
    """Initialize DefFileDecoder.

    Args:
        include_positions: include textual positions of values
    """
    self._include_positions = include_positions

Functions

decode

decode(content: str) -> dict

Decode *.def content into dictionary.

Parameters:

• content

  (str) –

  content of the *.def file

Returns:

• dict ( dict ) –

  Decoded *.def file

Source code in ipsl_common/modipsl/def_file.py

def decode(self, content: str) -> dict:
    """Decode `*.def` content into dictionary.

    Args:
        content (str): content of the `*.def` file

    Returns:
        dict: Decoded `*.def` file
    """
    parse_tree = self._parser.parse(content)
    return self.transform(parse_tree)

DefFileEncoder

DefFileEncoder(
    truthy_value: str = "true", falsey_value: str = "false"
)

Encoder performs translation from a dictionary to *.def file.

The translation rules are:

Python	*.def	Comment
dict	key-value	Each INCLUDEDEF value translates to a single key-value
list	array	Of at least 2 elements
str	string	Quoted strings will contain explicit quote characters
int	integer number	---
float	real number	Including scientific notation
bool	true/false	Can be specified with encode arguments
tuple	_AUTO_/_AUTOBLOCKER_	With optional default value at second position in tuple

The translation is straightforward, based on Python type a specific conversion is performed. No grammar, nor parse tree is used during this step.

Example:

from ipsl_common.modipsl.def_file import DefFileEncoder
text = DefFileEncoder().encode(dictionary)

If dictionary contains:

{
    "radius": 6.371229e6,
    "g": 9.80665,
    "omega": ("_AUTO_", 7.292e-05),
}

Then, the encoded *.def file would look as follows:

radius = 6371229.0
g = 9.80665
omega = _AUTO_: DEFAULT=7.292e-05

Tip

Python representation of the *.def file doesn't contain any textual position of particular elements (keys, values, comments, whitespaces, etc.), thus, re-encoding of the exact input *.def file is impossible. In order to recreate the original file, or modify a file while keeping the original comments, whitespaces, and order of elements, use the designated modify functions.

Initialize DefFileEncoder.

Parameters:

• truthy_value

  (str, default: 'true' ) –

  label used to encode True

• falsey_value

  (str, default: 'false' ) –

  label used to encode False

Source code in ipsl_common/modipsl/def_file.py

def __init__(
    self,
    truthy_value: str = "true",
    falsey_value: str = "false",
):
    """Initialize DefFileEncoder.

    Args:
        truthy_value: label used to encode True
        falsey_value: label used to encode False
    """
    self._truthy_value = truthy_value
    self._falsey_value = falsey_value

Functions

encode

encode(obj: object) -> str

Encode dictionary or other Python object into *.def file.

This function works not only on full decoded *.def files, but it also work on particular Python object such as a list, or a tuple. In such case, it will take the given object and apply one of the encoding rules mentioned before.

Parameters:

• obj

  (object) –

  dictionary or Python object

Returns:

• str –

  Encoded text of a *.def file

Source code in ipsl_common/modipsl/def_file.py

def encode(self, obj: object) -> str:
    """Encode dictionary or other Python object into `*.def` file.

    This function works not only on full decoded `*.def` files,
    but it also work on particular Python object such as a list,
    or a tuple. In such case, it will take the given object and apply
    one of the encoding rules mentioned before.

    Args:
        obj: dictionary or Python object

    Returns:
        Encoded text of a `*.def` file
    """
    # bool must be tested before int, because it is a subclass of int class.
    if isinstance(obj, bool):
        return self._truthy_value if obj else self._falsey_value
    elif isinstance(obj, int | float):
        return str(obj)
    elif isinstance(obj, tuple):
        if len(obj) != 2:
            raise ValueError(
                f"Only tuples with len=2 are encoded. Invalid tuple: {obj}"
            )
        auto, default = obj
        if auto not in _AUTO_VALUES:
            raise ValueError(
                f"First element of a tuple must be one of {_AUTO_VALUES}. Invalid tuple: {obj}"
            )
        if not self._is_scalar(default):
            raise TypeError(
                f"Second element of the tuple must be a scalar: bool, int, float, str, or None. Invalid tuple: {obj}"
            )
        if default is not None:
            return f"{auto}: DEFAULT={self.encode(default)}"
        else:
            return str(auto)
    elif isinstance(obj, dict) and len(obj) == 1:
        # Using tuple assignment
        ((k, v),) = obj.items()
        if k == _INCLUDEDEF:
            return self._encode_include(v)
        else:
            return self._encode_entry(k, v)
    elif isinstance(obj, dict):
        return "\n".join([self.encode({k: v}) for k, v in obj.items()])
    elif isinstance(obj, list):
        if len(obj) < 2:
            raise ValueError("List must have at least two elements")
        if not all(map(self._is_scalar, obj)):
            raise TypeError(
                f"All array values must be simple scalars: bool, int, float, str, or None. Instead got: {obj}"
            )
        return ", ".join(map(self.encode, obj))
    else:
        return str(obj)

Functions

dump

dump(obj: dict, buffer: TextIOBase) -> None

Dump dictionary into *.def text/file buffer.

Parameters:

• obj

  (dict) –

  dictionary to dump to a file

• buffer

  (TextIOBase) –

  text or file buffer for storing *.def file

Source code in ipsl_common/modipsl/def_file.py

def dump(obj: dict, buffer: TextIOBase) -> None:
    """Dump dictionary into *.def text/file buffer.

    Args:
        obj: dictionary to dump to a file
        buffer: text or file buffer for storing *.def file
    """
    if not buffer.writable():
        raise ValueError("Text buffer (TextIOBase) must be writable")
    buffer.write(dumps(obj))

dumps

dumps(obj: dict) -> str

Dump dictionary into *.def string.

Parameters:

• obj

  (dict) –

  dictionary to dump to a file

Source code in ipsl_common/modipsl/def_file.py

def dumps(obj: dict) -> str:
    """Dump dictionary into *.def string.

    Args:
        obj: dictionary to dump to a file
    """
    return DefFileEncoder().encode(obj)

load

load(
    buffer: TextIOBase, include_positions: bool = False
) -> dict

Load *.def text/file buffer into dictionary.

Parameters:

• buffer

  (TextIOBase) –

  text or file buffer with the *.def file

• include_positions

  (bool, default: False ) –

  include textual positions of values

Returns:

• dict ( dict ) –

  Loaded *.def file

Source code in ipsl_common/modipsl/def_file.py

def load(buffer: TextIOBase, include_positions: bool = False) -> dict:
    """Load `*.def` text/file buffer into dictionary.

    Args:
        buffer: text or file buffer with the `*.def` file
        include_positions: include textual positions of values

    Returns:
        dict: Loaded `*.def` file
    """
    if not buffer.readable():
        raise ValueError("Text buffer (TextIOBase) must be readable")
    return loads(buffer.read(), include_positions)

loads

loads(text: str, include_positions: bool = False) -> dict

Load *.def file string into dictionary.

Parameters:

• text

  (str) –

  content of the *.def file

• include_positions

  (bool, default: False ) –

  include textual positions of values

Returns:

• dict ( dict ) –

  Loaded *.def file

Source code in ipsl_common/modipsl/def_file.py

def loads(text: str, include_positions: bool = False) -> dict:
    """Load `*.def` file string into dictionary.

    Args:
        text: content of the `*.def` file
        include_positions: include textual positions of values

    Returns:
        dict: Loaded `*.def` file
    """
    return DefFileDecoder(include_positions).decode(text)

modify

modify(
    obj: dict,
    fp: TextIOBase,
    fp_reference: TextIOBase | None = None,
) -> None

Modify *.def file with minimal changes.

If only fp is defined, the file will be changed in place. Otherwise, the content fp_in file is

Source code in ipsl_common/modipsl/def_file.py

def modify(obj: dict, fp: TextIOBase, fp_reference: TextIOBase | None = None) -> None:
    """Modify *.def file with minimal changes.

    If only fp is defined, the file will be changed in place.
    Otherwise, the content fp_in file is
    """
    # Modify file in place or create new
    dictionary = (
        load(fp, include_positions=True)
        if fp_reference is None
        else load(fp_reference, include_positions=True)
    )

modify_file

modify_file(
    file: Path,
    modifications: dict[str, str | int | float | bool],
) -> None

Modify *.def file in place.

TODO: modification of AUTO fields is not supported.

Source code in ipsl_common/modipsl/def_file.py

def modify_file(file: Path, modifications: dict[str, str | int | float | bool]) -> None:
    """
    Modify *.def file in place.

    TODO: modification of _AUTO_ fields is not supported.
    """
    with open(file, "r") as f:
        new_content = modify_text(f.read(), modifications)
        with open(file, "w") as f:
            f.write(new_content)

modify_text

modify_text(text: str, modifications: dict) -> str

Appends new keys at the end in alphabetical order.

Source code in ipsl_common/modipsl/def_file.py

def modify_text(text: str, modifications: dict) -> str:
    """
    Appends new keys at the end in alphabetical order.
    """
    tree = DEF_FILE_PARSER.parse(text)
    position_map = ValuePositionMap().transform(tree)
    existing_keys = list(position_map.keys())
    # 1. Replace values of existing keys
    new_content = __replace_in_str(modifications, position_map, text)
    # 2. Append new keys
    epilogue = []
    # Sort by keys to obtain an alphabetical order of inserted keys
    for key, value in sorted(modifications.items()):
        # Position map contains only keys existing in the text
        if key not in existing_keys:
            epilogue.append(f"{key}={value}")
    if epilogue:
        new_content += "\n"
        epilogue.insert(0, "# ~~~ Values added by TéléphériqueIPSL ~~~")
        new_content += "\n".join(epilogue)
        new_content += "\n"
    return new_content