`whylogs.app.writers`¶

Classes for writing whylogs output

Module Contents¶

Classes¶

`Writer`	Class for writing to disk
`LocalWriter`	whylogs Writer class that can write to disk.
`S3Writer`	whylogs Writer class that can write to S3.
`MlFlowWriter`	Class for writing to disk
`WhyLabsWriter`	Class for writing to disk

Functions¶

writer_from_config(config: whylogs.app.config.WriterConfig)

Construct a whylogs Writer from a WriterConfig

Attributes¶

`DEFAULT_PATH_TEMPLATE`
`DEFAULT_FILENAME_TEMPLATE`
`logger`

whylogs.app.writers.DEFAULT_PATH_TEMPLATE = $name/$session_id¶

whylogs.app.writers.DEFAULT_FILENAME_TEMPLATE = dataset_profile¶

whylogs.app.writers.logger¶

class whylogs.app.writers.Writer(output_path: str, formats: List[str], path_template: Optional[str] = None, filename_template: Optional[str] = None, transport_params: Optional[whylogs.app.config.TransportParameterConfig] = None)¶

Bases: abc.ABC

Class for writing to disk

Parameters

output_path (str) – Prefix of where to output files. A directory for type = ‘local’, or key prefix for type = ‘s3’
formats (list) – All output formats. See whylogs.app.config.ALL_SUPPORTED_FORMATS
path_template (str, optional) – Templatized path output using standard python string templates. Variables are accessed via $identifier or ${identifier}. See Writer.template_params() for a list of available identifers. Default = DEFAULT_PATH_TEMPLATE
filename_template (str, optional) – Templatized output filename using standardized python string templates. Variables are accessed via $identifier or ${identifier}. See Writer.template_params() for a list of available identifers. Default = DEFAULT_FILENAME_TEMPLATE

close(self)¶

abstract write(self, profile: whylogs.core.DatasetProfile, rotation_suffix: str = None)¶: Abstract method to write a dataset profile to disk. Must be implemented

path_suffix(self, profile: whylogs.core.DatasetProfile)¶: Generate a path string for an output path from a dataset profile by applying the path templating defined in self.path_template

file_name(self, profile: whylogs.core.DatasetProfile, file_extension: str, rotation_suffix: Optional[str] = None)¶: For a given DatasetProfile, generate an output filename based on the templating defined in self.filename_template

static template_params(profile: whylogs.core.DatasetProfile) → dict¶

Return a dictionary of dataset profile metadata which can be used for generating templatized variables or paths.

Parameters: profile (DatasetProfile) – The dataset profile
Returns: params – Variables which can be substituted into a template string.
Return type: dict

Notes

Template params:

name: name of the dataset
session_timestamp: session time in UTC epoch milliseconds
dataset_timestamp: timestamp for the data in UTC epoch ms
session_id: Unique identifier for the session

class whylogs.app.writers.LocalWriter(output_path: str, formats: List[str], path_template: str, filename_template: str)¶

Bases: Writer

whylogs Writer class that can write to disk.

See Writer for a description of arguments

write(self, profile: whylogs.core.DatasetProfile, rotation_suffix: Optional[str] = None)¶: Write a dataset profile to disk

_do_write(self, profile, rotation_suffix: Optional[str] = None, **kwargs)¶

_write_json(self, profile: whylogs.core.DatasetProfile, rotation_suffix: Optional[str] = None)¶: Write a JSON summary of the dataset profile to disk

_write_flat(self, profile: whylogs.core.DatasetProfile, indent: int = 4, rotation_suffix: Optional[str] = None)¶

Write output data for flat format

Parameters

profile (DatasetProfile) – the dataset profile to output
indent (int) – The JSON indentation to use. Default is 4

_write_protobuf(self, profile: whylogs.core.DatasetProfile, rotation_suffix: Optional[str] = None)¶: Write a protobuf serialization of the DatasetProfile to disk

ensure_path(self, suffix: str, addition_part: Optional[str] = None) → str¶: Ensure that a path exists, creating it if not

class whylogs.app.writers.S3Writer(output_path: str, formats: List[str], path_template: str = None, filename_template: str = None)¶

Bases: Writer

whylogs Writer class that can write to S3.

See Writer for a description of arguments

write(self, profile: whylogs.core.DatasetProfile, rotation_suffix: str = None)¶: Write a dataset profile to S3

_do_write(self, profile, rotation_suffix: str = None, **kwargs)¶

_write_json(self, profile: whylogs.core.DatasetProfile, rotation_suffix: Optional[str] = None, transport_params: Optional[dict] = None)¶: Write a dataset profile JSON summary to disk

_write_flat(self, profile: whylogs.core.DatasetProfile, indent: int = 4, rotation_suffix: Optional[str] = None, transport_params: Optional[dict] = None)¶

Write output data for flat format

Parameters

profile (DatasetProfile) – the dataset profile to output
indent (int) – The JSON indentation to use. Default is 4

_write_protobuf(self, profile: whylogs.core.DatasetProfile, rotation_suffix: Optional[str] = None, transport_params: Optional[dict] = None)¶: Write a datasetprofile protobuf serialization to S3

class whylogs.app.writers.MlFlowWriter(output_path: str, formats: List[str], path_template: str = None, filename_template: str = None)¶

Bases: Writer

Class for writing to disk

Parameters

output_path (str) – Prefix of where to output files. A directory for type = ‘local’, or key prefix for type = ‘s3’
formats (list) – All output formats. See whylogs.app.config.ALL_SUPPORTED_FORMATS
path_template (str, optional) – Templatized path output using standard python string templates. Variables are accessed via $identifier or ${identifier}. See Writer.template_params() for a list of available identifers. Default = DEFAULT_PATH_TEMPLATE
filename_template (str, optional) – Templatized output filename using standardized python string templates. Variables are accessed via $identifier or ${identifier}. See Writer.template_params() for a list of available identifers. Default = DEFAULT_FILENAME_TEMPLATE

write(self, profile: whylogs.core.DatasetProfile, rotation_suffix: str = None)¶: Write a dataset profile to MLFlow path

static _write_protobuf(profile: whylogs.core.DatasetProfile, rotation_suffix: str = None, **kwargs)¶: Write a protobuf the dataset profile to disk in binary format to MlFlow

class whylogs.app.writers.WhyLabsWriter(output_path='', formats=None)¶

Bases: Writer

Class for writing to disk

Parameters

output_path (str) – Prefix of where to output files. A directory for type = ‘local’, or key prefix for type = ‘s3’
formats (list) – All output formats. See whylogs.app.config.ALL_SUPPORTED_FORMATS
path_template (str, optional) – Templatized path output using standard python string templates. Variables are accessed via $identifier or ${identifier}. See Writer.template_params() for a list of available identifers. Default = DEFAULT_PATH_TEMPLATE
filename_template (str, optional) – Templatized output filename using standardized python string templates. Variables are accessed via $identifier or ${identifier}. See Writer.template_params() for a list of available identifers. Default = DEFAULT_FILENAME_TEMPLATE

write(self, profile: whylogs.core.DatasetProfile, rotation_suffix: str = None)¶: Write a dataset profile to WhyLabs

static _write_protobuf(profile: whylogs.core.DatasetProfile)¶: Write a protobuf profile to WhyLabs

whylogs.app.writers.writer_from_config(config: whylogs.app.config.WriterConfig)¶

Construct a whylogs Writer from a WriterConfig

Returns: writer – whylogs writer
Return type: Writer

whylogs.app.writers¶

Module Contents¶

Classes¶

Functions¶

Attributes¶

`whylogs.app.writers`¶