whylogs.io.file_loader

Module Contents

Functions

valid_file(fname: str)

simple check if extension is part of the implemented ones

extension_file(path: str)

Check the enconding format based on the magic number

image_loader(path: str)

tries to load image using the PIL lib

json_loader(path: str = None) → Union[Dict, list]

Loads json or jsonl data

file_loader(path: str, valid_file: Callable[[str], bool] = valid_file) → Any

Factory for file data

Attributes

EXTENSIONS

IMAGE_EXTENSIONS

PD_EXCEL_FORMATS

whylogs.io.file_loader.EXTENSIONS = ['.csv', '.jpg', '.jpeg', '.png', '.ppm', '.bmp', '.jsonl', '.json', '.pgm', '.tif', '.tiff',...
whylogs.io.file_loader.IMAGE_EXTENSIONS = ['.jpg', '.jpeg', '.png', '.ppm', '.bmp', '.pgm', '.tif', '.tiff', '.webp', '.gif']
whylogs.io.file_loader.PD_EXCEL_FORMATS = ['.xls', '.xlsx', '.xlsm', '.xlsb', '.odf', '.ods', '.odt']
whylogs.io.file_loader.valid_file(fname: str)

simple check if extension is part of the implemented ones

Parameters

fname (str) – file path

Returns

bool

whylogs.io.file_loader.extension_file(path: str)

Check the enconding format based on the magic number if file has no magic number we simply use extension. More advance analytics of file content is needed, potentially extendind to a lib like libmagic

Parameters

path (str) – File path

Returns

str: extension of encoding data magic_data : dic : any magic data information available including

magic number : byte mime_type: str name : str

Return type

file_extension_given

whylogs.io.file_loader.image_loader(path: str)

tries to load image using the PIL lib

Parameters

path (str) – path to image files

Returns

image data and image encoding format

Return type

PIL.Image.Image

whylogs.io.file_loader.json_loader(path: str = None)Union[Dict, list]

Loads json or jsonl data

Parameters

path (str, optional) – path to file

Returns

Union[Dict, list]: Returns a list or dict of json data json_format : format of file (json or jsonl)

Return type

objs

whylogs.io.file_loader.file_loader(path: str, valid_file: Callable[[str], bool] = valid_file)Any

Factory for file data

Parameters
  • path (str) – path to file

  • valid_file (Callable[[str], bool], optional) – Optional valid file check,

Returns

Tuple( [] Dataframe or Image data (PIL format), or Dict], magic_data: Dict of magic number data)

Return type

data

Raises

NotImplementedError – Description