Release 0.21.3

Release 0.21.3

Major Features and Improvements

• Added support for model validation using either value threshold or diff
  threshold.
• Added a writer to output model validation result (ValidationResult).
• Added support for multi-model evaluation using EvalSavedModels.
• Added support for inserting model_names by default to metrics_specs.

Bug fixes and other changes

• Fixed issue with model_name not being set in keras metrics.

Breaking changes

• Populate TDistributionValue metric when confidence intervals is enabled in
  V2.
• Rename the writer MetricsAndPlotsWriter to MetricsPlotsAndValidationsWriter.

Deprecations

Release 0.21.2

Release 0.21.2

Major Features and Improvements

Bug fixes and other changes

• Adding SciPy dependency for both Python2 and Python3
• Increased table and tooltip font in Fairness Indicators.

Breaking changes

• tfma.BinarizeOptions.class_ids, tfma.BinarizeOptions.k_list,
  tfma.BinarizeOptions.top_k_list, and tfma.Options.disabled_outputs are
  now wrapped in an additional proto message.

Deprecations

Release 0.21.1

Release 0.21.1

Major Features and Improvements

Bug fixes and other changes

• Adding support to compute deterministic confidence intervals using a seed
  value in tfma.run_model_analysis API for testing or experimental purposes.
• Fixed calculation of tfma.metrics.CoefficientOfDiscrimination and
  tfma.metrics.RelativeCoefficientOfDiscrimination.

Breaking changes

Deprecations

Release 0.21.0

Release 0.21.0

Major Features and Improvements

• Added tfma.metrics.MinLabelPosition and tfma.metrics.QueryStatistics for
  use with V2 metrics API.
• Added tfma.metrics.CoefficientOfDiscrimination and
  tfma.metrics.RelativeCoefficientOfDiscrimination for use with V2 metrics
  API.
• Added support for using tf.keras.metrics.* metrics with V2 metrics API.
• Added support for default V2 MetricSpecs and creating specs from
  tf.kera.metrics.* and tfma.metrics.* metric classes.
• Added new MetricsAndPlotsEvaluator based on V2 infrastructure. Note this
  evaluator also supports query-based metrics.
• Add support for micro_average, macro_average, and weighted_macro_average
  metrics.
• Added support for running V2 extractors and evaluators. V2 extractors will
  be used whenever the default_eval_saved_model is created using a non-eval
  tag (e.g. tf.saved_model.SERVING). The V2 evaluator will be used whenever
  a tfma.EvalConfig is used containing metrics_specs.
• Added support for tfma.metrics.SquaredPearsonCorrelation for use with V2
  metrics API.
• Improved support for TPU autoscaling and handling batch_size related
  scaling.
• Added support for tfma.metrics.Specificity, tfma.metrics.FallOut, and
  tfma.metrics.MissRate for use with V2 metrics API. Renamed AUCPlot to
  ConfusionMatrixPlot, MultiClassConfusionMatrixAtThresholds to
  MultiClassConfusionMatrixPlot and MultiLabelConfusionMatrixAtThresholds
  to MultiLabelConfusionMatrixPlot.
• Added Jupyter support to Fairness Indicators. Currently does not support WIT
  integration.
• Added fairness indicators metrics
  tfma.addons.fairness.metrics.FairnessIndicators.
• Updated documentation for new metrics infrastructure and newly supported
  models (keras, etc).

Bug fixes and other changes

• Fixed error in tfma-multi-class-confusion-matrix-at-thresholds with
  default classNames value.
• Fairness Indicators
  • Compute ratio metrics with safe division.
  • Remove "post_export_metrics" from metric names.
  • Move threshold dropdown selector to a metric-by-metric basis, allowing
    different metrics to be inspected with different thresholds. Don't show
    thresholds for metrics that do not support them.
  • Slices are now displayed in alphabetic order.
  • Adding an option to "Select all" metrics in UI.

Breaking changes

• Updated proto config to remove input/output data specs in favor of passing
  them directly to the run_eval.

Deprecations

Release 0.15.4

Release 0.15.4

Major Features and Improvements

Bug fixes and other changes

• Fixed the bug that Fairness Indicator will skip metrics with NaN value.

Breaking changes

Deprecations

Release 0.15.3

Release 0.15.3

Major Features and Improvements

Bug fixes and other changes

• Updated vulcanized_tfma.js with UI changes in addons/fairness_indicators.

Breaking changes

Deprecations

Release 0.15.2

Release 0.15.2

Major Features and Improvements

Bug fixes and other changes

• Updated to use tf.io.gfile for reading config files (fixes issue with
  reading from GCS/HDFS in 0.15.0 and 0.15.1 releases).

Breaking changes

Deprecations

Release 0.15.1

Release 0.15.1

Major Features and Improvements

• Added support for defaulting to using class IDs when classes are not present
  in outputs for multi-class metrics (for use in keras model_to_estimator).
• Added example count metrics (tfma.metrics.ExampleCount and
  tfma.metrics.WeightedExampleCount) for use with V2 metrics API.
• Added calibration metrics (tfma.metrics.MeanLabel,
  tfma.metrics.MeanPrediction, and tfma.metrics.Calibration) for use with
  V2 metrics API.
• Added tfma.metrics.ConfusionMatrixAtThresholds for use with V2 metrics
  API.
• Added tfma.metrics.CalibrationPlot and tfma.metrics.AUCPlot for use with
  V2 metrics API.
• Added multi_class / multi_label plots (
  tfma.metrics.MultiClassConfusionMatrixAtThresholds,
  tfma.metrics.MultiLabelConfusionMatrixAtThresholds) for use with V2
  metrics API.
• Added tfma.metrics.NDCG metric for use with V2 metrics API.

Bug fixes and other changes

• Depends on tensorflow>=1.15,<3.0.
  • Starting from 1.15, package tensorflow comes with GPU support. Users
    won't need to choose between tensorflow and tensorflow-gpu.
  • Caveat: tensorflow 2.0.0 is an exception and does not have GPU
    support. If tensorflow-gpu 2.0.0 is installed before installing
    tensorflow_model_analysis, it will be replaced with tensorflow
    2.0.0. Re-install tensorflow-gpu 2.0.0 if needed.

Breaking changes

Deprecations

Release 0.15.0

Release 0.15.0

Major Features and Improvements

• Added V2 of PredictExtractor that uses TF 2.0 signature APIs and supports
  keras models (note: keras model evaluation not fully supported yet).
• tfma.run_model_analysis, tfma.default_extractors,
  tfma.default_evaluators, and tfma.default_writers now allow settings to
  be passed as an EvalConfig.
• tfma.run_model_analysis, tfma.default_extractors,
  tfma.default_evaluators, and tfma.default_writers now allow multiple
  models to be passed (note: multi-model support not fully implemented yet).
• Added InputExtractor for extracting labels, features, and example weights
  from tf.Examples.
• Added Fairness Indicator as an addon.

Bug fixes and other changes

• Enabled TF 2.0 support using compat.v1.
• Added support for slicing on native dicts of features in addition to FPL
  types.
• For multi-output and / or multi-class models, please provide output_name and
  / or class_id to tfma.view.render_plot.
• Replaced dependency on tensorflow-transform with tfx-bsl. If running
  with latest master, tfx-bsl must also be latest master.
• Depends on tfx-bsl>=0.15,<0.16.
• Slicing now supports conversion between int/floats and strings.
• Depends on apache-beam[gcp]>=2.16,<3.
• Depends on six==1.12.

Breaking changes

• tfma.EvalResult.slicing_metrics now contains nested dictionaries of output,
  class id and then metric names.
• Update config serialization to use JSON instead of pickling and reformat
  config to include input_data_specs, model_specs, output_data_specs, and
  metrics_specs.
• Requires pre-installed TensorFlow >=1.15,<3.

Deprecations

Release 0.14.0

Release 0.14.0

Major Features and Improvements

• Added documentation on architecture.
• Added an adapt_to_remove_metrics function to tfma.exporter which can be
  used to remove metrics incompatible with TFMA (e.g. py_func or streaming
  metrics) before exporting the TFMA EvalSavedModel.
• Added support for passing sparse int64 tensors to precision/recall@k.
• Added support for binarization of multiclass metrics that use labels of the
  from (N) in addition to (N, 1).
• Added support for using iterators with EvalInputReceiver.
• Improved performance of confidence interval computations by modifying the
  pipeline shape.
• Added QueryBasedMetricsEvaluator which supports computing query-based
  metrics (e.g. normalized discounted cumulative gain).
• Added support for merging metrics produced by different evaluators.
• Added support for blacklisting specified features from fetches.
• Added functionality to the FeatureExtractor to specify the features dict as
  a possible destination.
• Added support for label vocabularies for binary and multi-class estimators
  that support the new ALL_CLASSES prediction output.
• Move example parsing in aggregation into the graph for performance
  improvements in both standard and model_agnostic evaluation modes.

Bug fixes and other changes

• Upgraded codebase for TF 2.0 compatibility.
• Make metrics-related operations thread-safe by wrapping them with locks.
  This eliminates race conditions that were previously possible in
  multi-threaded runners which could result in incorrect metric values.
• More flexible FanoutSlices.
• Limit the number of sampling buckets to 20.
• Improved performance in Confidence Interval computation.
• Refactored poisson bootstrap code to be re-usable in other evaluators.
• Refactored k-anonymity code to be re-usable in other evaluators.
• Fixed slicer feature string value handling in Python3.
• Added support for example weight keys for multi-output models.
• Added option to set the desired batch size when calling run_model_analysis.
• Changed TFRecord compression type from UNCOMPRESSED to AUTO.
• Depends on apache-beam[gcp]>=2.14,<3.
• Depends on numpy>=1.16,<2.
• Depends on protobuf>=3.7,<4.
• Depends on scipy==1.1.0.

Breaking changes

• Removed uses of deprecated tf.contrib packages (where possible).
• tfma.default_writers now requires the eval_saved_model to be passed as
  an argument.
• Requires pre-installed TensorFlow >=1.14,<2.

Deprecations