whylogs.io.file_loader

Module Contents

Functions

valid_file(fname: str)	simple check if extension is part of the implemented ones
extension_file(path: str)	Check the enconding format based on the magic number
image_loader(path: str)	tries to load image using the PIL lib
json_loader(path: str = None) → Union[Dict, list]	Loads json or jsonl data
file_loader(path: str, valid_file: Callable[[str], bool] = valid_file) → Any	Factory for file data

Attributes

EXTENSIONS
IMAGE_EXTENSIONS
PD_EXCEL_FORMATS

whylogs.io.file_loader.EXTENSIONS = ['.csv', '.jpg', '.jpeg', '.png', '.ppm', '.bmp', '.jsonl', '.json', '.pgm', '.tif', '.tiff',...

whylogs.io.file_loader.IMAGE_EXTENSIONS = ['.jpg', '.jpeg', '.png', '.ppm', '.bmp', '.pgm', '.tif', '.tiff', '.webp', '.gif']

whylogs.io.file_loader.PD_EXCEL_FORMATS = ['.xls', '.xlsx', '.xlsm', '.xlsb', '.odf', '.ods', '.odt']

whylogs.io.file_loader.valid_file(fname: str)

simple check if extension is part of the implemented ones

Parameters

fname (str) – file path

Returns

bool

whylogs.io.file_loader.extension_file(path: str)

Check the enconding format based on the magic number if file has no magic number we simply use extension. More advance analytics of file content is needed, potentially extendind to a lib like libmagic

Parameters

path (str) – File path

Returns

str: extension of encoding data magic_data : dic : any magic data information available including

magic number : byte mime_type: str name : str

Return type

file_extension_given

whylogs.io.file_loader.image_loader(path: str)

tries to load image using the PIL lib

Parameters

path (str) – path to image files

Returns

image data and image encoding format

Return type

PIL.Image.Image

whylogs.io.file_loader.json_loader(path: str = None) → Union[Dict, list]

Loads json or jsonl data

Parameters

path (str, optional) – path to file

Returns

Union[Dict, list]: Returns a list or dict of json data json_format : format of file (json or jsonl)

Return type

objs

whylogs.io.file_loader.file_loader(path: str, valid_file: Callable[[str], bool] = valid_file) → Any

Factory for file data

Parameters

• path (str) – path to file

• valid_file (Callable[[str], bool], optional) – Optional valid file check,

Returns

Tuple( [] Dataframe or Image data (PIL format), or Dict], magic_data: Dict of magic number data)

Return type

data

Raises

NotImplementedError – Description