whylogs.app.writers

Classes for writing whylogs output

Module Contents

Classes

Writer

Class for writing to disk

LocalWriter

whylogs Writer class that can write to disk.

S3Writer

whylogs Writer class that can write to S3.

WhyLabsWriter

Class for writing to disk

Functions

writer_from_config(config: whylogs.app.config.WriterConfig)

Construct a whylogs Writer from a WriterConfig

Attributes

DEFAULT_PATH_TEMPLATE

DEFAULT_FILENAME_TEMPLATE

whylogs.app.writers.DEFAULT_PATH_TEMPLATE = $name/$session_id
whylogs.app.writers.DEFAULT_FILENAME_TEMPLATE = dataset_profile
class whylogs.app.writers.Writer(output_path: str, formats: List[str], path_template: Optional[str] = None, filename_template: Optional[str] = None)

Bases: abc.ABC

Class for writing to disk

Parameters
  • output_path (str) – Prefix of where to output files. A directory for type = ‘local’, or key prefix for type = ‘s3’

  • formats (list) – All output formats. See whylogs.app.config.ALL_SUPPORTED_FORMATS

  • path_template (str, optional) – Templatized path output using standard python string templates. Variables are accessed via $identifier or ${identifier}. See Writer.template_params() for a list of available identifers. Default = DEFAULT_PATH_TEMPLATE

  • filename_template (str, optional) – Templatized output filename using standardized python string templates. Variables are accessed via $identifier or ${identifier}. See Writer.template_params() for a list of available identifers. Default = DEFAULT_FILENAME_TEMPLATE

abstract write(self, profile: whylogs.core.DatasetProfile, rotation_suffix: str = None)

Abstract method to write a dataset profile to disk. Must be implemented

path_suffix(self, profile: whylogs.core.DatasetProfile)

Generate a path string for an output path from a dataset profile by applying the path templating defined in self.path_template

file_name(self, profile: whylogs.core.DatasetProfile, file_extension: str)

For a given DatasetProfile, generate an output filename based on the templating defined in self.filename_template

static template_params(profile: whylogs.core.DatasetProfile)dict

Return a dictionary of dataset profile metadata which can be used for generating templatized variables or paths.

Parameters

profile (DatasetProfile) – The dataset profile

Returns

params – Variables which can be substituted into a template string.

Return type

dict

Notes

Template params:

  • name: name of the dataset

  • session_timestamp: session time in UTC epoch milliseconds

  • dataset_timestamp: timestamp for the data in UTC epoch ms

  • session_id: Unique identifier for the session

class whylogs.app.writers.LocalWriter(output_path: str, formats: List[str], path_template: str, filename_template: str)

Bases: Writer

whylogs Writer class that can write to disk.

See Writer for a description of arguments

write(self, profile: whylogs.core.DatasetProfile, rotation_suffix: str = None)

Write a dataset profile to disk

_write_json(self, profile: whylogs.core.DatasetProfile)

Write a JSON summary of the dataset profile to disk

_write_flat(self, profile: whylogs.core.DatasetProfile, indent: int = 4)

Write output data for flat format

Parameters
  • profile (DatasetProfile) – the dataset profile to output

  • indent (int) – The JSON indentation to use. Default is 4

_write_protobuf(self, profile: whylogs.core.DatasetProfile)

Write a protobuf serialization of the DatasetProfile to disk

ensure_path(self, suffix: str, addition_part: Optional[str] = None)str

Ensure that a path exists, creating it if not

class whylogs.app.writers.S3Writer(output_path: str, formats: List[str], path_template: str = None, filename_template: str = None)

Bases: Writer

whylogs Writer class that can write to S3.

See Writer for a description of arguments

write(self, profile: whylogs.core.DatasetProfile, rotation_suffix: str = None)

Write a dataset profile to S3

_write_json(self, profile: whylogs.core.DatasetProfile)

Write a dataset profile JSON summary to disk

_write_flat(self, profile: whylogs.core.DatasetProfile, indent: int = 4)

Write output data for flat format

Parameters
  • profile (DatasetProfile) – the dataset profile to output

  • indent (int) – The JSON indentation to use. Default is 4

_write_protobuf(self, profile: whylogs.core.DatasetProfile)

Write a datasetprofile protobuf serialization to S3

class whylogs.app.writers.WhyLabsWriter(output_path: str, formats: List[str], path_template: Optional[str] = None, filename_template: Optional[str] = None)

Bases: Writer

Class for writing to disk

Parameters
  • output_path (str) – Prefix of where to output files. A directory for type = ‘local’, or key prefix for type = ‘s3’

  • formats (list) – All output formats. See whylogs.app.config.ALL_SUPPORTED_FORMATS

  • path_template (str, optional) – Templatized path output using standard python string templates. Variables are accessed via $identifier or ${identifier}. See Writer.template_params() for a list of available identifers. Default = DEFAULT_PATH_TEMPLATE

  • filename_template (str, optional) – Templatized output filename using standardized python string templates. Variables are accessed via $identifier or ${identifier}. See Writer.template_params() for a list of available identifers. Default = DEFAULT_FILENAME_TEMPLATE

write(self, profile: whylogs.core.DatasetProfile, rotation_suffix: str = None)

Write a dataset profile to WhyLabs

_write_protobuf(self, profile: whylogs.core.DatasetProfile)

Write a protobuf profile to WhyLabs

whylogs.app.writers.writer_from_config(config: whylogs.app.config.WriterConfig)

Construct a whylogs Writer from a WriterConfig

Returns

writer – whylogs writer

Return type

Writer