Skip to content

modipsl

Modules

card_file

Classes

CardFileDecoder 
CardFileDecoder(include_positions: bool = False)

Bases: Transformer

Decoder performs translation from *.card file to a dictionary.

The translation rules are:

*.card Python Comment
section dict[str, dict] Top-level dicts are sections
key-value dict Each item is a sigle kv pair
list list Empty or with elements
nested list list[list] List of lists
range tuple[int] Integer range in the form: START:STEP:END
string str Unquoted (with chars: -./${}*) and double quoted
integer number int ---
real number float Including scientific notation
true/y True Case insensitive
false/n False Case insensitive
Empty/NONE None Empty value that acts like null
Warning

CardFileDecoder works on items not belonging to any section, e.g. top-level key-value pairs. However, the load(s) functions require that a valid *.card file contains all keys inside some section.

Initialize CardFileDecoder.

Parameters:

  • include_positions
    (bool, default: False ) –

    include textual positions of keys and values (offset from the start)

Source code in ipsl_common/modipsl/card_file.py 
275
276
277
278
279
280
281
def __init__(self, include_positions: bool = False) -> None:
    """Initialize CardFileDecoder.

    Args:
        include_positions: include textual positions of keys and values (offset from the start)
    """
    self._include_positions = include_positions
Methods:
decode 
decode(text: str) -> dict[str, Any]

Decode *.card text into dictionary.

Parameters:

  • text (str) –

    content of the *.card file

Returns:

  • dict ( dict[str, Any] ) –

    Decoded *.card file

Source code in ipsl_common/modipsl/card_file.py 
283
284
285
286
287
288
289
290
291
292
293
def decode(self, text: str) -> dict[str, Any]:
    """Decode `*.card` text into dictionary.

    Args:
        text: content of the `*.card` file

    Returns:
        dict: Decoded `*.card` file
    """
    parse_tree = self._parser.parse(text)
    return self.transform(parse_tree)
CardFileEncoder 
CardFileEncoder(
    truthy_value: str = "true",
    falsey_value: str = "false",
    encode_none_in_kv: bool = False,
)

Encoder translates Python dictionary to *.card file.

The translation rules are:

Python *.card Comment
dict[str, dict] section Top-level keys with dicts are sections
dict key-value Each item is a sigle kv pair
list list Empty or with elements
list[list] nested list List of lists
tuple[int] range Integer range in the form: START:STEP:END
str string Unquoted (with chars: -./${}*) and double quoted
int integer number ---
float real number Including scientific notation
bool true/false Case insensitive
None Empty/NONE Empty value that acts like null

The translation is straightforward, based on Python type a specific conversion is performed. No grammar, nor parse tree is used during this step. A null value, NONE in lists is always encoded as NONE, so that the number of list elements is preserved. However, a single key with NONE is encoded as a= by default. See :encode_none_in_kv argument for more information.

Example:

from ipsl_common.modipsl.card_file import CardFileEncoder
text = CardFileEncoder().encode(dictionary)

If dictionary contains:

{
    "UserChoices": {
        "LMDZ_Physics": "NPv6.2"
    },
    "BoundaryFiles": {
        "ListNonDel": [
            ["${R_IN}/ATM/INPUT_CE0L/Albedo4_deg.nc", "Albedo.nc"],
            ["${R_IN}/ATM/INPUT_CE0L/ECDYN4.nc", "ECDYN.nc"]
        ]
    }
}

Then, the encoded *.card file would look as follows:

[UserChoices]
LMDZ_Physics = NPv6.2
[BoundaryFiles]
ListNonDel = (${R_IN}/ATM/INPUT_CE0L/Albedo4_deg.nc, Albedo.nc),                      (${R_IN}/ATM/INPUT_CE0L/ECDYN4.nc, ECDYN.nc)
Tip

Python representation of the *.card file doesn't contain any textual position of particular elements (keys, values, comments, whitespaces, etc.), thus, re-encoding of the exact input *.card file is impossible. In order to recreate the original file, or modify a file while keeping the original comments, whitespaces, and order of elements, use the designated modify functions.

Initialize CardFileEncoder.

Parameters:

  • truthy_value
    (str, default: 'true' ) –

    label used to encode True

  • falsey_value
    (str, default: 'false' ) –

    label used to encode False

  • encode_none_in_kv
    (bool, default: False ) –

    encode null NONE with single key-values (e.g. a=NONE)

Source code in ipsl_common/modipsl/card_file.py 
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
def __init__(
    self,
    truthy_value: str = "true",
    falsey_value: str = "false",
    encode_none_in_kv: bool = False,
):
    """Initialize CardFileEncoder.

    Args:
        truthy_value: label used to encode True
        falsey_value: label used to encode False
        encode_none_in_kv: encode null `NONE` with single key-values (e.g. a=NONE)
    """
    self._truthy_value = truthy_value
    self._falsey_value = falsey_value
    self._encode_none_in_kv = encode_none_in_kv
Methods:
encode 
encode(obj: Any, _nested_list_indent=None) -> str

Encode dictionary or other Python object into *.card file.

Parameters:

  • obj (Any) –

    dictionary or Python object

  • _nested_list_indent –

    internal parameter used to indent nested lists, so that each sublist starts at the same position.

Returns:

  • str –

    Encoded text of a *.def file

Info

_nested_list_indent parameter is internal, even if set by the user, it will be set again during the encoding to match the length of a key with a nested list as its value.

Source code in ipsl_common/modipsl/card_file.py 
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
def encode(self, obj: Any, _nested_list_indent=None) -> str:
    """Encode dictionary or other Python object into `*.card` file.

    Args:
        obj: dictionary or Python object
        _nested_list_indent: internal parameter used to indent nested lists,
                             so that each sublist starts at the same position.

    Returns:
        Encoded text of a `*.def` file

    Info:
        `_nested_list_indent` parameter is internal, even if set by the user,
        it will be set again during the encoding to match the length of a key
        with a nested list as its value.
    """
    if isinstance(obj, dict):
        top_kv, sections = [], []
        for k, v in obj.items():
            if isinstance(v, dict):
                sections.append(self._encode_section(k, v))
            else:
                top_kv.append(self._encode_kv(k, v))
        top_kv_text = "\n".join(top_kv)
        section_text = "\n".join(sections)
        # Separate top-level keys and sections with a newline, if both aren't empty
        sep = "\n" if top_kv_text and section_text else ""
        return top_kv_text + sep + section_text
    elif isinstance(obj, list):
        elements_are_list = [isinstance(e, list) for e in obj]
        # Case 1: singular nested list doesn't make sense!
        if len(obj) == 1 and all(elements_are_list):
            raise ValueError("Single nested list [[...]] is prohibited")
        # Case 2: classic nested list with at least two nested element
        if len(obj) > 1 and all(elements_are_list):
            return self._encode_nested_list(
                obj, nested_list_indent=_nested_list_indent
            )
        # Case 3: invalid partial nested list
        elif any(elements_are_list):
            raise ValueError("List can't mix nested lists with scalar literals")
        # Case 4: classic flat list
        else:
            return self._encode_flat_list(obj)
    elif isinstance(obj, tuple):
        return self._encode_range(obj)
    else:
        return self._encode_scalar(obj)

Functions:

dump 
dump(obj: dict, buffer: TextIOBase) -> None

Dump dictionary into *.card text/file buffer.

Parameters:

  • obj
    (dict) –

    dictionary to dump to a file

  • buffer
    (TextIOBase) –

    text or file buffer for storing *.card file

Source code in ipsl_common/modipsl/card_file.py 
53
54
55
56
57
58
59
60
61
62
def dump(obj: dict, buffer: TextIOBase) -> None:
    """Dump dictionary into *.card text/file buffer.

    Args:
        obj: dictionary to dump to a file
        buffer: text or file buffer for storing *.card file
    """
    if not buffer.writable():
        raise ValueError("Text buffer (TextIOBase) must be writable")
    buffer.write(dumps(obj))
dumps 
dumps(obj: dict) -> str

Dump dictionary into *.card string.

Parameters:

  • obj
    (dict) –

    dictionary to dump to a file

Source code in ipsl_common/modipsl/card_file.py 
65
66
67
68
69
70
71
def dumps(obj: dict) -> str:
    """Dump dictionary into *.card string.

    Args:
        obj: dictionary to dump to a file
    """
    return CardFileEncoder().encode(obj)
load 
load(
    buffer: TextIOBase, include_positions: bool = False
) -> dict

Load *.card text/file buffer into dictionary.

Parameters:

  • buffer
    (TextIOBase) –

    text or file buffer with the *.card file

  • include_positions
    (bool, default: False ) –

    include textual positions of elements (section, key, value)

Returns:

  • dict ( dict ) –

    Loaded *.card file

Source code in ipsl_common/modipsl/card_file.py 
16
17
18
19
20
21
22
23
24
25
26
27
28
def load(buffer: TextIOBase, include_positions: bool = False) -> dict:
    """Load `*.card` text/file buffer into dictionary.

    Args:
        buffer: text or file buffer with the `*.card` file
        include_positions: include textual positions of elements (section, key, value)

    Returns:
        dict: Loaded `*.card` file
    """
    if not buffer.readable():
        raise ValueError("Text buffer (TextIOBase) must be readable")
    return loads(buffer.read(), include_positions)
loads 
loads(text: str, include_positions: bool = False) -> dict

Load *.card file string into dictionary.

Parameters:

  • text
    (str) –

    content of the *.card file

  • include_positions
    (bool, default: False ) –

    include textual positions of elements (section, key, value)

Returns:

  • dict ( dict ) –

    Loaded *.card file

Source code in ipsl_common/modipsl/card_file.py 
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
def loads(text: str, include_positions: bool = False) -> dict:
    """Load `*.card` file string into dictionary.

    Args:
        text: content of the `*.card` file
        include_positions: include textual positions of elements (section, key, value)

    Returns:
        dict: Loaded `*.card` file
    """
    card = CardFileDecoder(include_positions).decode(text)
    # Validate there are no top-level kv pairs
    for _, v in card.items():
        if (not include_positions and not isinstance(v, dict)) or (
            include_positions and not isinstance(v["value"], dict)
        ):
            raise ValueError(
                "*.card file must contain every key-value pairs inside a section"
            )
    return card
modify 
modify(
    buffer: TextIOBase,
    new_obj: dict,
    buffer_out: TextIOBase | None = None,
    insert_header: str = "",
) -> None

Modify *.card text/file buffer with minimal amount of changes.

The output text/file buffer follows changes made to new_obj representation of *.card file. The modifications are performed, so that the minimal amount of changes is applied. As a result, the diff between the old and new file content is minimal and no comments or whitespaces are lost beyond what is neccessary. The new_obj can remove, modify, and/or add new key-value pairs.

Parameters:

  • buffer
    (TextIOBase) –

    text or file buffer for reading the file (optionally to write to the file)

  • new_obj
    (dict) –

    modified representation of the *.card file content

  • buffer_out
    (TextIOBase | None, default: None ) –

    optional output buffer for writing the modified file content

  • insert_header
    (str, default: '' ) –

    optional header inserted before appended key-value pairs

Source code in ipsl_common/modipsl/card_file.py 
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
def modify(
    buffer: TextIOBase,
    new_obj: dict,
    buffer_out: TextIOBase | None = None,
    insert_header: str = "",
) -> None:
    """Modify `*.card` text/file buffer with minimal amount of changes.

    The output text/file buffer follows changes made to `new_obj` representation of `*.card` file.
    The modifications are performed, so that the minimal amount of changes is applied.
    As a result, the diff between the old and new file content is minimal and no comments
    or whitespaces are lost beyond what is neccessary. The `new_obj` can remove, modify,
    and/or add new key-value pairs.

    Args:
        buffer(TextIOBase): text or file buffer for reading the file (optionally to write to the file)
        new_obj(dict): modified representation of the *.card file content
        buffer_out(TextIOBase | None): optional output buffer for writing the modified file content
        insert_header(str): optional header inserted before appended key-value pairs
    """
    # Buffer must always be readable
    if not buffer.readable():
        raise ValueError("Text buffer (TextIOBase) must be readable")
    # Buffer must be writeable if buffer_out is not passed
    if buffer_out is None and not buffer.writable():
        raise ValueError("Text buffer (TextIOBase) must be writeable")
    # Otherwise, buffer_out must be writeable
    elif buffer_out is not None and not buffer_out.writable():
        raise ValueError("Text buffer_out (TextIOBase) must be writeable")

    target_buffer = buffer_out if buffer_out is not None else buffer
    target_buffer.write(modifys(buffer.read(), new_obj, insert_header=insert_header))
modifys 
modifys(
    text: str, new_obj: dict, insert_header: str = ""
) -> str

Modify *.card file string with minimal amount of changes.

The output text follows changes made to new_obj representation of *.card file. The modifications are performed, so that the minimal amount of changes is applied. As a result, the diff between the old and new file content is minimal and no comments or whitespaces are lost beyond what is neccessary. The new_obj can remove, modify, and/or add new key-value pairs.

Parameters:

  • text
    (str) –

    input text with *.card content to modify

  • new_obj
    (dict) –

    modified representation of the *.card file content

  • insert_header
    (str, default: '' ) –

    optional header inserted before appended key-value pairs

Returns:

  • str ( str ) –

    Modified text

Source code in ipsl_common/modipsl/card_file.py 
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
def modifys(text: str, new_obj: dict, insert_header: str = "") -> str:
    """Modify `*.card` file string with minimal amount of changes.

    The output text follows changes made to `new_obj` representation of `*.card` file.
    The modifications are performed, so that the minimal amount of changes is applied.
    As a result, the diff between the old and new file content is minimal and no comments
    or whitespaces are lost beyond what is neccessary. The `new_obj` can remove, modify,
    and/or add new key-value pairs.

    Args:
        text(str): input text with *.card content to modify
        new_obj(dict): modified representation of the *.card file content
        insert_header(str): optional header inserted before appended key-value pairs

    Returns:
        str: Modified text
    """
    old_obj = loads(text, include_positions=True)
    encoder = CardFileEncoder()
    replacements = []

    # Top-level can be only sections
    deleted_sections = old_obj.keys() - new_obj.keys()
    for k in deleted_sections:
        v = old_obj[k]
        replacements.append(
            (
                v["section_start_pos"],
                index_after_newline(text, v["section_end_pos"]),
                "",
            )
        )

    # Apply modification to existing keys
    for k, v in old_obj.items():
        if k not in new_obj:
            continue  # Skip, if key was removed
        # Find keys to delete
        s1 = v["value"]
        s2 = new_obj[k]
        deleted_keys = s1.keys() - s2.keys()
        for k1 in deleted_keys:
            v1 = s1[k1]
            replacements.append(
                (v1["key_start_pos"], index_after_newline(text, v1["end_pos"]), "")
            )
        # Find keys thath were modified
        common_keys = s1.keys() & s2.keys()
        for k1 in common_keys:
            if retrieve_values(s1[k1]["value"]) != s2[k1]:
                v1 = s1[k1]
                replacements.append(
                    (
                        v1["key_start_pos"],
                        v1["end_pos"],
                        encoder.encode({k1: s2[k1]}),
                    )
                )
        # Find new keys
        new_keys = []
        for k1, _ in s2.items():
            if k1 in s1:
                continue
            else:
                new_keys.append(k1)

        epilogue = ""
        if append_keys := encoder.encode({k: s2[k] for k in new_keys}):
            epilogue += f"{append_keys}"
            end_pos = index_after_newline(text, v["section_end_pos"])
            if end_pos == len(text):
                epilogue = f"\n{epilogue}"
            else:
                epilogue = f"{epilogue}\n"
            replacements.append((end_pos, end_pos, epilogue))

    new_text = replace_text(text, replacements)
    # Then, add new sections at the end
    # INFO: We cannot use .keys() to find the new keys, because this method
    # returns a set which doesn't keep keys order. Instead, we can iterate
    # over items in dictionaries which is guaranteed to preserve the
    # insertion order since Python 3.7
    # (https://docs.python.org/3/library/stdtypes.html#typesmapping)
    new_sections = []
    for k in new_obj:
        if k not in old_obj:
            new_sections.append(k)

    epilogue = ""
    # Append only if there is something to append
    if append_keys := encoder.encode({k: new_obj[k] for k in new_sections}):
        if insert_header:
            epilogue = f"\n# {insert_header}"
        epilogue += f"\n{append_keys}"
    return (new_text + epilogue).rstrip("\n")

common

Functions:

index_after_newline 
index_after_newline(text: str, position: int) -> int

Return new position with the newline eaten at the input position

Source code in ipsl_common/modipsl/common.py 
 7
 8
 9
10
11
12
def index_after_newline(text: str, position: int) -> int:
    """Return new position with the newline eaten at the input position"""
    # There are no newlines at the end!
    if position == len(text):
        return position
    return position + 1 if text[position] == "\n" else position
inject_value_position 
inject_value_position(func: Callable) -> Callable

Retrieve in-text position of values from their token(s).

This decorator works with grammar generated methods of the lark.Transformer. It works well on scalar (e.g. 1, 3.14, True) and compound (e.g. (1, 2, 3)) values, because those are represented using only lark.Tokens, each with a start_pos/end_pos attributes. The decorator converts the normal parsing method into the one that takes those position attributes and transform the parsed value to a dictionary comining the original value with its positions.

Parameters:

  • func
    (Callable) –

    lark.Transformer based method to decorate

Returns:

  • Callable ( Callable ) –

    decorated method returning dictionary with the parsed value and its positions in the text

Source code in ipsl_common/modipsl/common.py 
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
def inject_value_position(func: Callable) -> Callable:
    """Retrieve in-text position of values from their token(s).

    This decorator works with grammar generated methods of the lark.Transformer.
    It works well on scalar (e.g. `1`, `3.14`, `True`) and compound
    (e.g. `(1, 2, 3)`) values, because those are represented using only
    lark.Tokens, each with a start_pos/end_pos attributes.
    The decorator converts the normal parsing method into the one
    that takes those position attributes and transform the parsed value to
    a dictionary comining the original value with its positions.

    Args:
        func: lark.Transformer based method to decorate

    Returns:
        Callable: decorated method returning dictionary with the parsed value and its positions in the text
    """

    @functools.wraps(func)
    def wrapper_decorator(self, tokens):
        # Get scalar value
        value = func(self, tokens)
        if self._include_positions:
            # Retrive position from token(s)
            return {
                "value": value,
                "start_pos": tokens[0].start_pos,
                "end_pos": tokens[-1].end_pos,
            }
        else:
            return value

    return wrapper_decorator
is_empty_rule_branch 
is_empty_rule_branch(
    entity: Any, rule_name: str | None = None
) -> bool

Check if entity is an empty rule branch of an AST with an empty Token.

Such empty branch is created when an optional sub-rule (e.g. rule: optional? required) is not present.

Parameters:

  • entity
    (Any) –

    potential empty rule branch to check

  • rule_name
    (str | None, default: None ) –

    optional grammar rule name to validate in the empty rule branch

Returns:

  • bool ( bool ) –

    True, if entity is an empty rule branch

Source code in ipsl_common/modipsl/common.py 
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
def is_empty_rule_branch(entity: Any, rule_name: str | None = None) -> bool:
    """Check if entity is an empty rule branch of an AST with an empty Token.

    Such empty branch is created when an optional sub-rule
    (e.g. `rule: optional? required`) is not present.

    Args:
        entity: potential empty rule branch to check
        rule_name: optional grammar rule name to validate in the empty rule branch

    Returns:
        bool: True, if entity is an empty rule branch
    """
    if not isinstance(entity, Tree):
        return False
    rule_check = (
        entity.data == Token("RULE", rule_name) if rule_name is not None else True
    )
    return (
        isinstance(entity.data, Token)
        and not entity.children
        # Also, check if rule name matches if passed as an argument
        and rule_check
    )
retrieve_values 
retrieve_values(obj: Any) -> dict | list

Retrieve only values from a nested structure with position attributes.

The .def/.card decoding with inserting value/key/section positions convert those files into Python dictionaries which, apart from values, contains additional keys such as start_pos, end_pos, key_start_pos, section_start_pos and section_end_pos. Those keys encode the start/end positions of various elements of those file formats. This function removes them and it keeps only the original values from the files.

Example input dictionary:

{
    'section': {
        'b': {
            'value': 2,
            'start_pos': 16,
            'end_pos': 17,
            'key_start_pos': 14
        }
    }
}

And the result:

{
    'section': {
        'b': 2
    }
}

Parameters:

  • obj
    (Any) –

    object with values to retrive

Returns:

  • dict ( dict | list ) –

    a new object with only values

Source code in ipsl_common/modipsl/common.py 
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
def retrieve_values(obj: Any) -> dict | list:
    """Retrieve only values from a nested structure with position attributes.

    The *.def/*.card decoding with inserting value/key/section positions
    convert those files into Python dictionaries which, apart from values,
    contains additional keys such as `start_pos`, `end_pos`, `key_start_pos`,
    `section_start_pos` and `section_end_pos`. Those keys encode the start/end
    positions of various elements of those file formats. This function removes
    them and it keeps only the original values from the files.

    Example input dictionary:

        {
            'section': {
                'b': {
                    'value': 2,
                    'start_pos': 16,
                    'end_pos': 17,
                    'key_start_pos': 14
                }
            }
        }

    And the result:

        {
            'section': {
                'b': 2
            }
        }

    Args:
        obj: object with values to retrive

    Returns:
        dict: a new object with only values
    """
    if isinstance(obj, dict):
        if "value" in obj:
            return retrieve_values(obj["value"])
        return {k: retrieve_values(v) for k, v in obj.items()}
    elif isinstance(obj, list):
        return [retrieve_values(item) for item in obj]
    return obj

def_file

Read, write, and modify *.def file from the IPSL/modipsl project.

The *.def file contains model parameters in the key-value format. The format is extremely simple in comparison to similar formats, like *.ini or *.toml, as it doesn't provide sections, nor standarized datatypes. Usually, model configuration files contain dozens or hundred of parameters with scalars (int, float, str), arrays, or special _AUTO_/_AUTOBLOCK_ values.

The special "auto" values are used by the IPSL/modipsl/libIGCM projects to mark place where an external script, called component driver, should inject configuration values.

Examples usage:

// Follows Python convention of the JSON module with load(), dump(), etc. functions
from ipsl_common.modipsl.def_file import load, dump
with open("run_dynamico.def", "r") as f:
    parameters = load(f)
    // Modify loaded parameters
    parameters["start_file_name"] = "start2024.nc"
    with open("new_run_dynamico.def", "w") as g:
        dump(parameters, g)

Loading an example *.def file:

INCLUDEDEF=run_lmdz.def
INCLUDEDEF=run_dynamico.def
use_forcing=y
g=_AUTO_: DEFAULT=9.8
start_file_name=start2023
physics="always"

Gives the following dictionary:

{
    'INCLUDEDEF': ['run_lmdz.def', 'run_dynamico.def'],
    'g': ('_AUTO_', 9.8),
    'physics': '"always"',
    'start_file_name': 'start2023',
    'use_forcing': True
}

Loaded configuration can be altered and subsequently dumped onto a file or into a string. The configuration is easy to view and modify, because it is directly decoded into a Python dictionary. The decoding and encoding process is managed internally by DefFileDecoder and DefFileEncoder classes with decode() and encode() methods.

Classes

DefFileDecoder 
DefFileDecoder(include_positions: bool = False)

Bases: Transformer

Decoder performs translation from *.def file to a dictionary.

The translation rules are:

*.def Python Comment
key-value dict Including many key-value pairs and INCLUDEDEF
array list Collection of at least 2 elements
string str Unquoted and quoted (single or double) strings
integer number int Standard integer values
real number float Including scientific notation
true/y True Case insensitive
false/n False Case insensitive
_AUTO_ tuple With optional default value
_AUTOBLOCKER_ tuple With optional default value

The first step of the decoder is parsing of *.def file. For this task Lark Earley parser is used with a simple grammar expressed with EBNF notation. The *.def grammer doesn't parse well with LALR(1) parser. The parsing produces a parse tree.

The second step transforms the parse tree into Python dictionary using translation rules mentioned in the above table. This transformation is based on a automated visitor pattern called Transformer, which produces the dictionary in a bottom-up manner.

Examples:

from ipsl_common.modipsl.def_file import DefFileDecoder
dictionary = DefFileDecoder().decode(text)

If text contains this *.def file:

radius=6.371229E6
g=9.80665
omega=_AUTO_: DEFAULT=7.292E-5

Then, Python dictionary would look as follows:

{
    "radius": 6.371229e6,
    "g": 9.80665,
    "omega": ("_AUTO_", 7.292e-05),
}
Tip

By default, the result dictionary contains no information about textual layout of the file. However, by using the argument include_positions=True, it is possible to refine the dictionary with exact start/end positions of each value as follows:

{
    "radius": {"value": 6371229.0, "start_pos": 50, "end_pos": 60},
    "g": {"value": 9.80665, "start_pos": 99, "end_pos": 106},
    "omega": {"value": ("_AUTO_", 7.292e-05), "start_pos": 158, "end_pos": 182},
}

Initialize DefFileDecoder.

Parameters:

  • include_positions
    (bool, default: False ) –

    include textual positions of keys and values (offset from the start)

Source code in ipsl_common/modipsl/def_file.py 
349
350
351
352
353
354
355
def __init__(self, include_positions: bool = False) -> None:
    """Initialize DefFileDecoder.

    Args:
        include_positions: include textual positions of keys and values (offset from the start)
    """
    self._include_positions = include_positions
Methods:
decode 
decode(content: str) -> dict

Decode *.def content into dictionary.

Parameters:

  • content (str) –

    content of the *.def file

Returns:

  • dict ( dict ) –

    Decoded *.def file

Source code in ipsl_common/modipsl/def_file.py 
357
358
359
360
361
362
363
364
365
366
367
def decode(self, content: str) -> dict:
    """Decode `*.def` content into dictionary.

    Args:
        content (str): content of the `*.def` file

    Returns:
        dict: Decoded `*.def` file
    """
    parse_tree = self._parser.parse(content)
    return self.transform(parse_tree)
DefFileEncoder 
DefFileEncoder(
    truthy_value: str = "true", falsey_value: str = "false"
)

Encoder performs translation from a dictionary to *.def file.

The translation rules are:

Python *.def Comment
dict key-value Each INCLUDEDEF value translates to a single key-value
list array Of at least 2 elements
str string Quoted strings will contain explicit quote characters
int integer number ---
float real number Including scientific notation
bool true/false Can be specified with encode arguments
tuple _AUTO_/_AUTOBLOCKER_ With optional default value at second position in tuple

The translation is straightforward, based on Python type a specific conversion is performed. No grammar, nor parse tree is used during this step.

Example:

from ipsl_common.modipsl.def_file import DefFileEncoder
text = DefFileEncoder().encode(dictionary)

If dictionary contains:

{
    "radius": 6.371229e6,
    "g": 9.80665,
    "omega": ("_AUTO_", 7.292e-05),
}

Then, the encoded *.def file would look as follows:

radius = 6371229.0
g = 9.80665
omega = _AUTO_: DEFAULT=7.292e-05
Tip

Python representation of the *.def file doesn't contain any textual position of particular elements (keys, values, comments, whitespaces, etc.), thus, re-encoding of the exact input *.def file is impossible. In order to recreate the original file, or modify a file while keeping the original comments, whitespaces, and order of elements, use the designated modify functions.

Initialize DefFileEncoder.

Parameters:

  • truthy_value
    (str, default: 'true' ) –

    label used to encode True

  • falsey_value
    (str, default: 'false' ) –

    label used to encode False

Source code in ipsl_common/modipsl/def_file.py 
510
511
512
513
514
515
516
517
518
519
520
521
522
def __init__(
    self,
    truthy_value: str = "true",
    falsey_value: str = "false",
):
    """Initialize DefFileEncoder.

    Args:
        truthy_value: label used to encode True
        falsey_value: label used to encode False
    """
    self._truthy_value = truthy_value
    self._falsey_value = falsey_value
Methods:
encode 
encode(obj: Any) -> str

Encode dictionary or other Python object into *.def file.

This function works not only on full decoded *.def files, but it also work on particular Python object such as a list, or a tuple. In such case, it will take the given object and apply one of the encoding rules mentioned before.

Parameters:

  • obj (Any) –

    dictionary or Python object

Returns:

  • str –

    Encoded text of a *.def file

Source code in ipsl_common/modipsl/def_file.py 
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
def encode(self, obj: Any) -> str:
    """Encode dictionary or other Python object into `*.def` file.

    This function works not only on full decoded `*.def` files,
    but it also work on particular Python object such as a list,
    or a tuple. In such case, it will take the given object and apply
    one of the encoding rules mentioned before.

    Args:
        obj: dictionary or Python object

    Returns:
        Encoded text of a `*.def` file
    """
    # bool must be tested before int, because it is a subclass of int class.
    if isinstance(obj, bool):
        return self._truthy_value if obj else self._falsey_value
    elif isinstance(obj, int | float):
        return str(obj)
    elif isinstance(obj, tuple):
        if len(obj) != 2:
            raise ValueError(
                f"Only tuples with len=2 are encoded. Invalid tuple: {obj}"
            )
        auto, default = obj
        if auto not in _AUTO_VALUES:
            raise ValueError(
                f"First element of a tuple must be one of {_AUTO_VALUES}. Invalid tuple: {obj}"
            )
        if not self._is_scalar(default):
            raise TypeError(
                f"Second element of the tuple must be a scalar: bool, int, float, str, or None. Invalid tuple: {obj}"
            )
        if default is not None:
            return f"{auto}: DEFAULT={self.encode(default)}"
        else:
            return str(auto)
    elif isinstance(obj, dict) and len(obj) == 1:
        # Using tuple assignment
        ((k, v),) = obj.items()
        if k == _INCLUDEDEF:
            return self._encode_include(v)
        else:
            return self._encode_entry(k, v)
    elif isinstance(obj, dict):
        return "\n".join([self.encode({k: v}) for k, v in obj.items()])
    elif isinstance(obj, list):
        if len(obj) < 2:
            raise ValueError("List must have at least two elements")
        if not all(map(self._is_scalar, obj)):
            raise TypeError(
                f"All array values must be simple scalars: bool, int, float, str, or None. Instead got: {obj}"
            )
        return ", ".join(map(self.encode, obj))
    else:
        return str(obj)

Functions:

dump 
dump(obj: dict, buffer: TextIOBase) -> None

Dump dictionary into *.def text/file buffer.

Parameters:

  • obj
    (dict) –

    dictionary to dump to a file

  • buffer
    (TextIOBase) –

    text or file buffer for storing *.def file

Source code in ipsl_common/modipsl/def_file.py 
103
104
105
106
107
108
109
110
111
112
def dump(obj: dict, buffer: TextIOBase) -> None:
    """Dump dictionary into *.def text/file buffer.

    Args:
        obj: dictionary to dump to a file
        buffer: text or file buffer for storing *.def file
    """
    if not buffer.writable():
        raise ValueError("Text buffer (TextIOBase) must be writable")
    buffer.write(dumps(obj))
dumps 
dumps(obj: dict) -> str

Dump dictionary into *.def string.

Parameters:

  • obj
    (dict) –

    dictionary to dump to a file

Source code in ipsl_common/modipsl/def_file.py 
115
116
117
118
119
120
121
def dumps(obj: dict) -> str:
    """Dump dictionary into *.def string.

    Args:
        obj: dictionary to dump to a file
    """
    return DefFileEncoder().encode(obj)
load 
load(
    buffer: TextIOBase,
    include_positions: bool = False,
    return_text: bool = False,
) -> dict | tuple[dict, str]

Load *.def text/file buffer into dictionary.

Parameters:

  • buffer
    (TextIOBase) –

    text or file buffer with the *.def file

  • include_positions
    (bool, default: False ) –

    include textual positions of values

Returns:

  • dict ( dict | tuple[dict, str] ) –

    Loaded *.def file

Source code in ipsl_common/modipsl/def_file.py 
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
def load(
    buffer: TextIOBase, include_positions: bool = False, return_text: bool = False
) -> dict | tuple[dict, str]:
    """Load `*.def` text/file buffer into dictionary.

    Args:
        buffer: text or file buffer with the `*.def` file
        include_positions: include textual positions of values

    Returns:
        dict: Loaded `*.def` file
    """
    if not buffer.readable():
        raise ValueError("Text buffer (TextIOBase) must be readable")
    text = buffer.read()
    if return_text:
        return loads(text, include_positions), text
    else:
        return loads(text, include_positions)
loads 
loads(text: str, include_positions: bool = False) -> dict

Load *.def file string into dictionary.

Parameters:

  • text
    (str) –

    content of the *.def file

  • include_positions
    (bool, default: False ) –

    include textual positions of values

Returns:

  • dict ( dict ) –

    Loaded *.def file

Source code in ipsl_common/modipsl/def_file.py 
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
def loads(text: str, include_positions: bool = False) -> dict:
    """Load `*.def` file string into dictionary.

    Args:
        text: content of the `*.def` file
        include_positions: include textual positions of values

    Returns:
        dict: Loaded `*.def` file
    """
    return DefFileDecoder(include_positions).decode(text)
modify 
modify(
    buffer: TextIOBase,
    new_obj: dict,
    buffer_out: TextIOBase | None = None,
    insert_header: str = "",
) -> None

Modify *.def text/file buffer with minimal amount of changes.

The output text/file buffer follows changes made to new_obj representation of *.def file. The modifications are performed, so that the minimal amount of changes is applied. As a result, the diff between the old and new file content is minimal and no comments or whitespaces are lost beyond what is neccessary. The new_obj can remove, modify, and/or add new key-value pairs.

Parameters:

  • buffer
    (TextIOBase) –

    text or file buffer for reading the file (optionally to write to the file)

  • new_obj
    (dict) –

    modified representation of the *.def file content

  • buffer_out
    (TextIOBase | None, default: None ) –

    optional output buffer for writing the modified file content

  • insert_header
    (str, default: '' ) –

    optional header inserted before appended key-value pairs

Source code in ipsl_common/modipsl/def_file.py 
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
def modify(
    buffer: TextIOBase,
    new_obj: dict,
    buffer_out: TextIOBase | None = None,
    insert_header: str = "",
) -> None:
    """Modify `*.def` text/file buffer with minimal amount of changes.

    The output text/file buffer follows changes made to `new_obj` representation of `*.def` file.
    The modifications are performed, so that the minimal amount of changes is applied.
    As a result, the diff between the old and new file content is minimal and no comments
    or whitespaces are lost beyond what is neccessary. The `new_obj` can remove, modify,
    and/or add new key-value pairs.

    Args:
        buffer(TextIOBase): text or file buffer for reading the file (optionally to write to the file)
        new_obj(dict): modified representation of the *.def file content
        buffer_out(TextIOBase | None): optional output buffer for writing the modified file content
        insert_header(str): optional header inserted before appended key-value pairs
    """
    # Buffer must always be readable
    if not buffer.readable():
        raise ValueError("Text buffer (TextIOBase) must be readable")
    # Buffer must be writeable if buffer_out is not passed
    if buffer_out is None and not buffer.writable():
        raise ValueError("Text buffer (TextIOBase) must be writeable")
    # Otherwise, buffer_out must be writeable
    elif buffer_out is not None and not buffer_out.writable():
        raise ValueError("Text buffer_out (TextIOBase) must be writeable")

    target_buffer = buffer_out if buffer_out is not None else buffer
    target_buffer.write(modifys(buffer.read(), new_obj, insert_header=insert_header))
modifys 
modifys(
    text: str, new_obj: dict, insert_header: str = ""
) -> str

Modify *.def file string with minimal amount of changes.

The output text follows changes made to new_obj representation of *.def file. The modifications are performed, so that the minimal amount of changes is applied. As a result, the diff between the old and new file content is minimal and no comments or whitespaces are lost beyond what is neccessary. The new_obj can remove, modify, and/or add new key-value pairs.

Parameters:

  • text
    (str) –

    input text with *.def content to modify

  • new_obj
    (dict) –

    modified representation of the *.def file content

  • insert_header
    (str, default: '' ) –

    optional header inserted before appended key-value pairs

Returns:

  • str ( str ) –

    Modified text

Source code in ipsl_common/modipsl/def_file.py 
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
def modifys(text: str, new_obj: dict, insert_header: str = "") -> str:
    """Modify `*.def` file string with minimal amount of changes.

    The output text follows changes made to `new_obj` representation of `*.def` file.
    The modifications are performed, so that the minimal amount of changes is applied.
    As a result, the diff between the old and new file content is minimal and no comments
    or whitespaces are lost beyond what is neccessary. The `new_obj` can remove, modify,
    and/or add new key-value pairs.

    Args:
        text(str): input text with *.def content to modify
        new_obj(dict): modified representation of the *.def file content
        insert_header(str): optional header inserted before appended key-value pairs

    Returns:
        str: Modified text
    """
    old_obj = loads(text, include_positions=True)
    encoder = DefFileEncoder()
    replacements = []

    # Replace removed key-value pairs by empty strings
    deleted_keys = old_obj.keys() - new_obj.keys()
    for k in deleted_keys:
        v = old_obj[k]
        replacements.append(
            (v["key_start_pos"], index_after_newline(text, v["end_pos"]), "")
        )

    # TODO: refactor this part in the future, now I don't have enough strength to do so
    include_insert_pos = 0
    old_includes_with_pos = old_obj.pop("INCLUDEDEF", [])
    old_includes = [m["value"] for m in old_includes_with_pos]
    to_remove = old_includes.copy()
    for m in new_obj.pop("INCLUDEDEF", []):
        if m in old_includes:
            to_remove.remove(m)
            include_insert_pos = (
                old_includes_with_pos[old_includes.index(m)]["end_pos"] + 1
            )
        else:
            replacements.append(
                (include_insert_pos, include_insert_pos, f"INCLUDEDEF={m}\n")
            )
    # Whatever has left is to remove
    for m in to_remove:
        d = old_includes_with_pos[old_includes.index(m)]
        replacements.append(
            (d["key_start_pos"], index_after_newline(text, d["end_pos"]), "")
        )

    # Apply modification to existing keys
    for k, v in old_obj.items():
        if k in new_obj and v["value"] != new_obj[k]:
            replacements.append(
                (v["start_pos"], v["end_pos"], encoder.encode(new_obj[k]))
            )
    new_text = replace_text(text, replacements)
    # Then, add new keys at the end
    # INFO: We cannot use .keys() to find the new keys, because this method
    # returns a set which doesn't keep keys order. Instead, we can iterate
    # over items in dictionaries which is guaranteed to preserve the
    # insertion order since Python 3.7
    # (https://docs.python.org/3/library/stdtypes.html#typesmapping)
    new_keys = []
    for k in new_obj:
        if k not in old_obj:
            new_keys.append(k)

    epilogue = ""
    # Append only if there is something to append
    if append_keys := encoder.encode({k: new_obj[k] for k in new_keys}):
        if insert_header:
            epilogue = f"\n# {insert_header}"
        epilogue += f"\n{append_keys}"
    return new_text + epilogue

mod_file

Read mod.def file from the IPSL/modipsl project.

The mod.def file contains definitions of coupled model configurations. A single configuration consists of one or more components (e.g. athmospheric model, dynamics, I/O system, experiments).

Examples:

from ipsl_common.modipsl.mod_file import load
with open("mod.def", "r") as f:
    configs = load(f)

The loaded dictionary has a fixed schema. Using the following input:

#-H- GRISLI GRISLI stand-alone for Antarctica icesheets (prototype)
#-C- GRISLI trunk/libIGCM HEAD 10 libIGCM .
#-C- GRISLI branches/xios HEAD 26 GRISLI modeles
...
#-S- 7 git https://gitlab.in2p3.fr/ipsl/projets/nemogcm/nemo.git
#-S- 8 svn --username icmc_users https://forge.ipsl.fr/igcmg/svn

the output looks as follows:

{
    'configuration': {
        'GRISLI': {
            'description': ['GRISLI stand-alone for Antarctica icesheets (prototype)']
            'components': [
                {
                    'modipsl_dir': '.',
                    'name': 'libIGCM',
                    'repository': 10,
                    'revision': 'HEAD',
                    'variant': 'trunk/libIGCM'
                },
                {
                    'modipsl_dir': 'modeles',
                    'name': 'GRISLI',
                    'repository': 26,
                    'revision': 'HEAD',
                    'variant': 'branches/xios'
                },
                (...)
            ],
        },
        (...)
    },
    'repository': {
        7: {
            'clone_url': 'https://gitlab.in2p3.fr/ipsl/projets/nemogcm/nemo.git',
            'type': 'git'
        },
        8: {
            'clone_url': '--username icmc_users https://forge.ipsl.fr/igcmg/svn',
            'type': 'svn'
        },
        (...)
    }
}
Warning

Because this module uses iterative matching to patterns (with re.finditer), it doesn't have a capability to tell, when the *.mod file is not well formatted, nor invalid. It will siliently skip non-matched lines and move on!

Functions:

load 
load(buffer: TextIOBase) -> dict

Load mod.def file from a text/file buffer.

Parameters:

  • buffer
    (TextIOBase) –

    text or file buffer with the mod.def file

Returns:

  • dict ( dict ) –

    Loaded mod.def file

Source code in ipsl_common/modipsl/mod_file.py 
151
152
153
154
155
156
157
158
159
160
def load(buffer: io.TextIOBase) -> dict:
    """Load mod.def file from a text/file buffer.

    Args:
        buffer (io.TextIOBase): text or file buffer with the mod.def file

    Returns:
        dict: Loaded mod.def file
    """
    return loads(buffer.read())
loads 
loads(content: str) -> dict

Load mod.def file from a string.

Parameters:

  • content
    (str) –

    content of the mod.def file

Returns:

  • dict ( dict ) –

    Loaded mod.def file

Source code in ipsl_common/modipsl/mod_file.py 
163
164
165
166
167
168
169
170
171
172
173
174
175
176
def loads(content: str) -> dict:
    """Load mod.def file from a string.

    Args:
        content (str): content of the mod.def file

    Returns:
        dict: Loaded mod.def file
    """
    repositories = __parse_repositories(content)
    return {
        "repository": repositories,
        "configuration": __parse_configurations(content, repositories),
    }