buhtzology.bandas

Add category label for missing values to a categorical data series.

Read and validate multiple files at once.

The argument read_this is a complex dictionary containing all relevant information about how to read and validate the files. The following example illustrate how to use it. The element specs_and_rules is described elsewhere in bandas.read_and_validate_csv().

read_this = {
    'Foo':  # name of resulting dataframe
        {
            'file': Path('foo.csv'),
            'specs_and_rules': {...},
            'encoding': 'utf-8',  # optional
            'no_header': True,  # optional
            'skiprows': 1,  # optional
            'skipfooter': 4,  # optional
        },
    'Bar':
        {
            'file': Path('bar.xlsx'),
            'sheet_name': 'Money',  # optional
            # ...
        },
    'FromArchive':
        {
            'file': zipfile.Path('data.zip', 'inzip_file.xlsx'),
            # ...
        },
     # ...
}

Create an ICD Catalog based on data from BfArM or WHO.

The data source can be a zip file (download from BfArM or WHO) or a download URL for that zip file (e.g. icd10gm2024syst-meta.zip).

The result looks like this:

CODE      TYP CHAPTER    BLOCK        TEXT         CODE_TEXT
0    A00.-     CODE      01  A00-A09   Cholera..   A00.- Cholera..
1    A00.0     CODE      01  A00-A09   Cholera..   A00.0 Cholera..
2  A00-A09    BLOCK      01  A00-A09  Infektiö..  A00-A09 Infekt..
3  A15-A19    BLOCK      01  A15-A19  Tuberkul..  A15-A19 Tuberk..
4       01  CHAPTER      01           Bestimmt..  01 Bestimmte i..
5       02  CHAPTER      02           Neubildu..  02 Neubildunge..

Cutting values into bins similar to pandas.cut().

See also cut_bins_via_interval_count() for a simplified wrapper. There are two major methods to specify the bins or intervals.

# Specify steps using "interval_range" to get intervals of
# same length:
cut_bins(the_values, interval_range=10)

# Or using "interval_list" to specify explicit each interval
# with pairs of start and begin value which are both included
# in that interval:
bins = [(0, 4), (5, 9), (10, 50)]
cut_bins(the_values, interval_list=bins)

The interval range arguments are ignored when interval_list is given.

Infinity starts and ends are created automatically when interval_range is used but not with interval_list. For the latter you have to specify them yourself like so:

bins = [(-math.inf, -1), (0, 9), (10, math.inf)]
cut_bins(the_values, interval_list=bins)

The wording of the labels can be modified via format_string and when using infinity ends also via begin_format_string and end_format_string. Or you can set explicit labels via labels.

Cutting values into specific number of bins of the same range.

If one of the values is not covered by the resulting intervals (bins) a ValueError is raised. NA values are an exception of that rule. This function is a wrapper around cut_bins().

Cut a dataframe into pieces row by row but keep groups together.

Groups are specified by the column name in group_column. The data will by sorted by group_column using the sort algorithm specified by sort_kind which is used by pandas/numpy. The default quicksort is the fastest. For unittesting stable should be used. The number of resulting parts is not guaranteed to be n_pieces.

Cut a dataframe horizontally into pieces of (nearly) the same size.

This pieces can be used for parallelization. To keep groups of rows together there is the alternative cut_by_row_keep_group(). The number of pieces is garantueed.

Create labels for the associated intervals.

The function is used by cut_bins(). Two ways exists to give the intervals in the bin argument.

# Pairs of intervals (both ends included)
bins = [
    (0, 4),
    (5, 9),
    (10, 20)
]
result = ['0 to 4', '5 to 9', '10 to 20']

# Or with infinity ends
bins = [
    (-math.inf, 4),
    (5, 9),
    (10, math.inf)
]
result = ['less than 5', '5 to 9', '10 and more']

# As list of edges (only right end included)
bins = [0, 5, 10]
result = ['1 to 5', '6 to 10']

# Or with infinity ends
bins = [-math.inf, 0, 5, 10, math.inf]
result = ['less than 1', '1 to 5', '6 to 10', '11 and more']

Use multiprocessing to working on dataframe row by row.

A dataframe is cut into multiple dataframes. Each of them is transferred to another process (not thread). This is fast because of using multiple CPU cores but costs a lot of RAM and some time for transffering the dataframe pieces (via pickle) to a process and back.

There are two options to cut a dataframe. By default or when using n_pieces it is cut into pieces with nearly the same number of rows. The function bandas.cut_into_pieces() is used in that case. When using group_column the function bandas.cut_by_row_keep_group() will be used.

Just return the result of the worker:

def the_worker(sub_dataframe):
    sub_dataframe.foo = 7

    return sub_dataframe

To add additional arguments to the worker use worker_args argument and the values in a tuple.

def the_worker(columns, sub_dataframe):
    sub_dataframe['Extra'] = sub_dataframe.loc[:, cols].apply(
        lambda row: row * 7, axis=1)

    return sub_dataframe

if __name__ == '__main__':
    result = bandas.parallelize_job_on_dataframe(
        data=df,
        worker_func=the_worker,
        group_column='group',
        worker_args=(['colA', 'colD', 'colT'], )
    )

Read and validate a CSV file.

Read a CSV file with respect to specifications about format and rules about valid values and return a pandas.DataFrame. You have to give specifications for all existing columns in the correct order. The following aspects can be specified:

• Columns to read and columns to ignore.
• The data type of a column. Types from pandas, numpy or Pythons builtins are valid.
• Missing values. They are converted to pandas.NA in the resulting data frame.
• Length of a data field in the raw CSV file.
• Valid values in a column (checked in the resulting data frame).
• Ignoring a column is also a specification.

specs_and_rules = {
    'ColumnA': 'str',
    'ColumnB': ('str', 'no answer'),
    'ColumnC': None,
    'ColumnD': (
        'Int16',
        -9,
        {
            'len': [1, 2, (4-8)],
            'val': [0, 1, (3-9)],
            'unique': True,
        }
    },
    -1: int,
}

specs_and_rules = {
    'ColumnA': None,
    'ColumnB': 'int',
    'ColumnC': (None, None, {'val': [1]}),
}

Read and validate an Excel file.

Read an Excel file with respect to specifications about format and rules about valid values and return a pandas.DataFrame. You have to give specifications for all existing columns in the correct order. But ignoring a column is also a specification. See read_and_validate_csv() for details and examples about usage of specs_and_rules.

Rearrange the columns of a DataFrame.

The columns named in this_columns are moved behind the column named via behind_this_column.

Validate the structure of a CSV file.

See read_and_validate_csv() for more details. Malformed or invalid rows are logged to a file (*.wrong.csv). The need for that function arises from the fact that the checks of pandas.read_csv() are quite sluggish.

Validate a dataframes content and format.

If the dataframe is valid nothing happens; no return value. If something is invalid a TypeError is raised.

The expected_columns is allowed to have negative integers. They are replaced using _name_unnamed_columns().

Validate if the DataFrame fit to length rules.

Validate if the DataFrame (df_to_check) fit to len_rules. Otherwise an ValueError is raised.

Validate if columns contain unique values.

Validate if the DataFrame fit to value rules.

Validate if the DataFrame (df_to_check) fit to value_rules. Otherwise an ValueError is raised.