card_file

Classes

CardFileDecoder

CardFileDecoder(include_positions: bool = False)

Bases: Transformer

Decoder performs translation from *.card file to a dictionary.

The translation rules are:

*.card	Python	Comment
section	dict[str, dict]	Top-level dicts are sections
key-value	dict	Each item is a sigle kv pair
list	list	Empty or with elements
nested list	list[list]	List of lists
range	tuple[int]	Integer range in the form: START:STEP:END
string	str	Unquoted (with chars: -./${}*) and double quoted
integer number	int	---
real number	float	Including scientific notation
true/y	True	Case insensitive
false/n	False	Case insensitive
Empty/NONE	None	Empty value that acts like null

Warning

CardFileDecoder works on items not belonging to any section, e.g. top-level key-value pairs. However, the load(s) functions require that a valid *.card file contains all keys inside some section.

Initialize CardFileDecoder.

Parameters:

• include_positions

  (bool, default: False ) –

  include textual positions of keys and values (offset from the start)

Source code in ipsl_common/modipsl/card_file.py

def __init__(self, include_positions: bool = False) -> None:
    """Initialize CardFileDecoder.

    Args:
        include_positions: include textual positions of keys and values (offset from the start)
    """
    self._include_positions = include_positions

Methods:

decode

decode(text: str) -> dict[str, Any]

Decode *.card text into dictionary.

Parameters:

• text

  (str) –

  content of the *.card file

Returns:

• dict ( dict[str, Any] ) –

  Decoded *.card file

Source code in ipsl_common/modipsl/card_file.py

def decode(self, text: str) -> dict[str, Any]:
    """Decode `*.card` text into dictionary.

    Args:
        text: content of the `*.card` file

    Returns:
        dict: Decoded `*.card` file
    """
    parse_tree = self._parser.parse(text)
    return self.transform(parse_tree)

CardFileEncoder

CardFileEncoder(
    truthy_value: str = "true",
    falsey_value: str = "false",
    encode_none_in_kv: bool = False,
)

Encoder translates Python dictionary to *.card file.

The translation rules are:

Python	*.card	Comment
dict[str, dict]	section	Top-level keys with dicts are sections
dict	key-value	Each item is a sigle kv pair
list	list	Empty or with elements
list[list]	nested list	List of lists
tuple[int]	range	Integer range in the form: START:STEP:END
str	string	Unquoted (with chars: -./${}*) and double quoted
int	integer number	---
float	real number	Including scientific notation
bool	true/false	Case insensitive
None	Empty/NONE	Empty value that acts like null

The translation is straightforward, based on Python type a specific conversion is performed. No grammar, nor parse tree is used during this step. A null value, NONE in lists is always encoded as NONE, so that the number of list elements is preserved. However, a single key with NONE is encoded as a= by default. See :encode_none_in_kv argument for more information.

Example:

from ipsl_common.modipsl.card_file import CardFileEncoder
text = CardFileEncoder().encode(dictionary)

If dictionary contains:

{
    "UserChoices": {
        "LMDZ_Physics": "NPv6.2"
    },
    "BoundaryFiles": {
        "ListNonDel": [
            ["${R_IN}/ATM/INPUT_CE0L/Albedo4_deg.nc", "Albedo.nc"],
            ["${R_IN}/ATM/INPUT_CE0L/ECDYN4.nc", "ECDYN.nc"]
        ]
    }
}

Then, the encoded *.card file would look as follows:

[UserChoices]
LMDZ_Physics = NPv6.2
[BoundaryFiles]
ListNonDel = (${R_IN}/ATM/INPUT_CE0L/Albedo4_deg.nc, Albedo.nc),                      (${R_IN}/ATM/INPUT_CE0L/ECDYN4.nc, ECDYN.nc)

Tip

Python representation of the *.card file doesn't contain any textual position of particular elements (keys, values, comments, whitespaces, etc.), thus, re-encoding of the exact input *.card file is impossible. In order to recreate the original file, or modify a file while keeping the original comments, whitespaces, and order of elements, use the designated modify functions.

Initialize CardFileEncoder.

Parameters:

• truthy_value

  (str, default: 'true' ) –

  label used to encode True

• falsey_value

  (str, default: 'false' ) –

  label used to encode False

• encode_none_in_kv

  (bool, default: False ) –

  encode null NONE with single key-values (e.g. a=NONE)

Source code in ipsl_common/modipsl/card_file.py

def __init__(
    self,
    truthy_value: str = "true",
    falsey_value: str = "false",
    encode_none_in_kv: bool = False,
):
    """Initialize CardFileEncoder.

    Args:
        truthy_value: label used to encode True
        falsey_value: label used to encode False
        encode_none_in_kv: encode null `NONE` with single key-values (e.g. a=NONE)
    """
    self._truthy_value = truthy_value
    self._falsey_value = falsey_value
    self._encode_none_in_kv = encode_none_in_kv

Methods:

encode

encode(obj: Any, _nested_list_indent=None) -> str

Encode dictionary or other Python object into *.card file.

Parameters:

• obj

  (Any) –

  dictionary or Python object

• _nested_list_indent

  –

  internal parameter used to indent nested lists, so that each sublist starts at the same position.

Returns:

• str –

  Encoded text of a *.def file

Info

_nested_list_indent parameter is internal, even if set by the user, it will be set again during the encoding to match the length of a key with a nested list as its value.

Source code in ipsl_common/modipsl/card_file.py

def encode(self, obj: Any, _nested_list_indent=None) -> str:
    """Encode dictionary or other Python object into `*.card` file.

    Args:
        obj: dictionary or Python object
        _nested_list_indent: internal parameter used to indent nested lists,
                             so that each sublist starts at the same position.

    Returns:
        Encoded text of a `*.def` file

    Info:
        `_nested_list_indent` parameter is internal, even if set by the user,
        it will be set again during the encoding to match the length of a key
        with a nested list as its value.
    """
    if isinstance(obj, dict):
        top_kv, sections = [], []
        for k, v in obj.items():
            if isinstance(v, dict):
                sections.append(self._encode_section(k, v))
            else:
                top_kv.append(self._encode_kv(k, v))
        top_kv_text = "\n".join(top_kv)
        section_text = "\n".join(sections)
        # Separate top-level keys and sections with a newline, if both aren't empty
        sep = "\n" if top_kv_text and section_text else ""
        return top_kv_text + sep + section_text
    elif isinstance(obj, list):
        elements_are_list = [isinstance(e, list) for e in obj]
        # Case 1: singular nested list doesn't make sense!
        if len(obj) == 1 and all(elements_are_list):
            raise ValueError("Single nested list [[...]] is prohibited")
        # Case 2: classic nested list with at least two nested element
        if len(obj) > 1 and all(elements_are_list):
            return self._encode_nested_list(
                obj, nested_list_indent=_nested_list_indent
            )
        # Case 3: invalid partial nested list
        elif any(elements_are_list):
            raise ValueError("List can't mix nested lists with scalar literals")
        # Case 4: classic flat list
        else:
            return self._encode_flat_list(obj)
    elif isinstance(obj, tuple):
        return self._encode_range(obj)
    else:
        return self._encode_scalar(obj)

Functions:

dump

dump(obj: dict, buffer: TextIOBase) -> None

Dump dictionary into *.card text/file buffer.

Parameters:

• obj

  (dict) –

  dictionary to dump to a file

• buffer

  (TextIOBase) –

  text or file buffer for storing *.card file

Source code in ipsl_common/modipsl/card_file.py

def dump(obj: dict, buffer: TextIOBase) -> None:
    """Dump dictionary into *.card text/file buffer.

    Args:
        obj: dictionary to dump to a file
        buffer: text or file buffer for storing *.card file
    """
    if not buffer.writable():
        raise ValueError("Text buffer (TextIOBase) must be writable")
    buffer.write(dumps(obj))

dumps

dumps(obj: dict) -> str

Dump dictionary into *.card string.

Parameters:

• obj

  (dict) –

  dictionary to dump to a file

Source code in ipsl_common/modipsl/card_file.py

def dumps(obj: dict) -> str:
    """Dump dictionary into *.card string.

    Args:
        obj: dictionary to dump to a file
    """
    return CardFileEncoder().encode(obj)

load

load(
    buffer: TextIOBase, include_positions: bool = False
) -> dict

Load *.card text/file buffer into dictionary.

Parameters:

• buffer

  (TextIOBase) –

  text or file buffer with the *.card file

• include_positions

  (bool, default: False ) –

  include textual positions of elements (section, key, value)

Returns:

• dict ( dict ) –

  Loaded *.card file

Source code in ipsl_common/modipsl/card_file.py

def load(buffer: TextIOBase, include_positions: bool = False) -> dict:
    """Load `*.card` text/file buffer into dictionary.

    Args:
        buffer: text or file buffer with the `*.card` file
        include_positions: include textual positions of elements (section, key, value)

    Returns:
        dict: Loaded `*.card` file
    """
    if not buffer.readable():
        raise ValueError("Text buffer (TextIOBase) must be readable")
    return loads(buffer.read(), include_positions)

loads

loads(text: str, include_positions: bool = False) -> dict

Load *.card file string into dictionary.

Parameters:

• text

  (str) –

  content of the *.card file

• include_positions

  (bool, default: False ) –

  include textual positions of elements (section, key, value)

Returns:

• dict ( dict ) –

  Loaded *.card file

Source code in ipsl_common/modipsl/card_file.py

def loads(text: str, include_positions: bool = False) -> dict:
    """Load `*.card` file string into dictionary.

    Args:
        text: content of the `*.card` file
        include_positions: include textual positions of elements (section, key, value)

    Returns:
        dict: Loaded `*.card` file
    """
    card = CardFileDecoder(include_positions).decode(text)
    # Validate there are no top-level kv pairs
    for _, v in card.items():
        if (not include_positions and not isinstance(v, dict)) or (
            include_positions and not isinstance(v["value"], dict)
        ):
            raise ValueError(
                "*.card file must contain every key-value pairs inside a section"
            )
    return card

modify

modify(
    buffer: TextIOBase,
    new_obj: dict,
    buffer_out: TextIOBase | None = None,
    insert_header: str = "",
) -> None

Modify *.card text/file buffer with minimal amount of changes.

The output text/file buffer follows changes made to new_obj representation of *.card file. The modifications are performed, so that the minimal amount of changes is applied. As a result, the diff between the old and new file content is minimal and no comments or whitespaces are lost beyond what is neccessary. The new_obj can remove, modify, and/or add new key-value pairs.

Parameters:

• buffer

  (TextIOBase) –

  text or file buffer for reading the file (optionally to write to the file)

• new_obj

  (dict) –

  modified representation of the *.card file content

• buffer_out

  (TextIOBase | None, default: None ) –

  optional output buffer for writing the modified file content

• insert_header

  (str, default: '' ) –

  optional header inserted before appended key-value pairs

Source code in ipsl_common/modipsl/card_file.py

def modify(
    buffer: TextIOBase,
    new_obj: dict,
    buffer_out: TextIOBase | None = None,
    insert_header: str = "",
) -> None:
    """Modify `*.card` text/file buffer with minimal amount of changes.

    The output text/file buffer follows changes made to `new_obj` representation of `*.card` file.
    The modifications are performed, so that the minimal amount of changes is applied.
    As a result, the diff between the old and new file content is minimal and no comments
    or whitespaces are lost beyond what is neccessary. The `new_obj` can remove, modify,
    and/or add new key-value pairs.

    Args:
        buffer(TextIOBase): text or file buffer for reading the file (optionally to write to the file)
        new_obj(dict): modified representation of the *.card file content
        buffer_out(TextIOBase | None): optional output buffer for writing the modified file content
        insert_header(str): optional header inserted before appended key-value pairs
    """
    # Buffer must always be readable
    if not buffer.readable():
        raise ValueError("Text buffer (TextIOBase) must be readable")
    # Buffer must be writeable if buffer_out is not passed
    if buffer_out is None and not buffer.writable():
        raise ValueError("Text buffer (TextIOBase) must be writeable")
    # Otherwise, buffer_out must be writeable
    elif buffer_out is not None and not buffer_out.writable():
        raise ValueError("Text buffer_out (TextIOBase) must be writeable")

    target_buffer = buffer_out if buffer_out is not None else buffer
    target_buffer.write(modifys(buffer.read(), new_obj, insert_header=insert_header))

modifys

modifys(
    text: str, new_obj: dict, insert_header: str = ""
) -> str

Modify *.card file string with minimal amount of changes.

The output text follows changes made to new_obj representation of *.card file. The modifications are performed, so that the minimal amount of changes is applied. As a result, the diff between the old and new file content is minimal and no comments or whitespaces are lost beyond what is neccessary. The new_obj can remove, modify, and/or add new key-value pairs.

Parameters:

• text

  (str) –

  input text with *.card content to modify

• new_obj

  (dict) –

  modified representation of the *.card file content

• insert_header

  (str, default: '' ) –

  optional header inserted before appended key-value pairs

Returns:

• str ( str ) –

  Modified text

Source code in ipsl_common/modipsl/card_file.py

def modifys(text: str, new_obj: dict, insert_header: str = "") -> str:
    """Modify `*.card` file string with minimal amount of changes.

    The output text follows changes made to `new_obj` representation of `*.card` file.
    The modifications are performed, so that the minimal amount of changes is applied.
    As a result, the diff between the old and new file content is minimal and no comments
    or whitespaces are lost beyond what is neccessary. The `new_obj` can remove, modify,
    and/or add new key-value pairs.

    Args:
        text(str): input text with *.card content to modify
        new_obj(dict): modified representation of the *.card file content
        insert_header(str): optional header inserted before appended key-value pairs

    Returns:
        str: Modified text
    """
    old_obj = loads(text, include_positions=True)
    encoder = CardFileEncoder()
    replacements = []

    # Top-level can be only sections
    deleted_sections = old_obj.keys() - new_obj.keys()
    for k in deleted_sections:
        v = old_obj[k]
        replacements.append(
            (
                v["section_start_pos"],
                index_after_newline(text, v["section_end_pos"]),
                "",
            )
        )

    # Apply modification to existing keys
    for k, v in old_obj.items():
        if k not in new_obj:
            continue  # Skip, if key was removed
        # Find keys to delete
        s1 = v["value"]
        s2 = new_obj[k]
        deleted_keys = s1.keys() - s2.keys()
        for k1 in deleted_keys:
            v1 = s1[k1]
            replacements.append(
                (v1["key_start_pos"], index_after_newline(text, v1["end_pos"]), "")
            )
        # Find keys thath were modified
        common_keys = s1.keys() & s2.keys()
        for k1 in common_keys:
            if retrieve_values(s1[k1]["value"]) != s2[k1]:
                v1 = s1[k1]
                replacements.append(
                    (
                        v1["key_start_pos"],
                        v1["end_pos"],
                        encoder.encode({k1: s2[k1]}),
                    )
                )
        # Find new keys
        new_keys = []
        for k1, _ in s2.items():
            if k1 in s1:
                continue
            else:
                new_keys.append(k1)

        epilogue = ""
        if append_keys := encoder.encode({k: s2[k] for k in new_keys}):
            epilogue += f"{append_keys}"
            end_pos = index_after_newline(text, v["section_end_pos"])
            if end_pos == len(text):
                epilogue = f"\n{epilogue}"
            else:
                epilogue = f"{epilogue}\n"
            replacements.append((end_pos, end_pos, epilogue))

    new_text = replace_text(text, replacements)
    # Then, add new sections at the end
    # INFO: We cannot use .keys() to find the new keys, because this method
    # returns a set which doesn't keep keys order. Instead, we can iterate
    # over items in dictionaries which is guaranteed to preserve the
    # insertion order since Python 3.7
    # (https://docs.python.org/3/library/stdtypes.html#typesmapping)
    new_sections = []
    for k in new_obj:
        if k not in old_obj:
            new_sections.append(k)

    epilogue = ""
    # Append only if there is something to append
    if append_keys := encoder.encode({k: new_obj[k] for k in new_sections}):
        if insert_header:
            epilogue = f"\n# {insert_header}"
        epilogue += f"\n{append_keys}"
    return (new_text + epilogue).rstrip("\n")