Validation of the resolution model

This notebook was tested with the environment LHCb Analysis Facility as made available in the image landerlini/lhcbaf:v0p8.

This notebook is part of a pipeline, in particular it requires data preprocessed in the Preprocessing-GANs notebook and the model trained in the Resolution notebook.

Environment and libraries

As for other validation notebooks, we are using the GPU to process the data (selections and histogramming) which will make it unusable for evaluating the DNN models in tensorflow. For a discussion on this issue, refer to the acceptance-validation notebook.

The libraries for using the GPU for data analysis are:

• cupy, as a replacement for numpy with operations running on the GPU
• cudf, as a replacement for pandas with a DataFrame stored on GPU
• dask to implement lazy operations and streaming data from disk to the GPU memory, on demand.

/usr/local/miniconda3/lib/python3.9/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Loading data

We are using a dataset statistically equivalent to the une used for training, but never loaded in the training notebook. We will use our custom FeatherReader helper function, converting data to a dask dataframe while they are read from disk.

Loading model, preprocessing and postprocessing steps

Preprocessing steps are obtained from the same directory as the model with the standard naming convention:

• tX.pkl for the preprocessing step
• tY.pkl for the postprocessing step

Please note that tY encodes the transformation from physics variables to normally-distributed preprocessed features. In this notebook we are interested in the inverse transformation, mapping the output of the generator (normally distributed features) to the physics quantities.

WARNING:tensorflow:No training configuration found in save file, so the model was *not* compiled. Compile it manually.

Processing pipeline

We define a lazy pipeline using dask to:

• load the data from disk (implemented by FeatherReader)
• transform both the preprocessed conditions and target features into the original physics quantities
• generate the random noise
• evaluate the generator on the preprocessed conditions and generated noise
• apply the inverse preprocessing step to the generated features to retrieve physics quantities (postprocessing)
• upload the resulting dataset to device memory for further processing

WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.
INFO:tensorflow:Assets written to: ram://6719af3a-dcae-43b0-9350-fd8f04f64898/assets

Adding variables to the dataframe

We append to the pipeline the computation of variables that we wish using for binning the dataset but do not enter directly in the training dataset.

The list of the variables computed include:

• the components of the momentum (from momentum and the slopes)
• the pseudorapidity $\eta$
• the azimuthal angle $\phi$
• the relative error on the momentum $\frac{\Delta p}{p}$

Distribution of the output features in kinematic bins

The following histogram show a comparison of the distribution of the generated and reference output features in kinematic bins. The comparison is split per particle type (electron, muon and hadron) and track class (long, upstream and downstream), resulting into nine comparisons for each output feature.

Long tracks

Particle type: electron

Variable: dx for electrons with long tracks

Variable: dy for electrons with long tracks

Variable: dz for electrons with long tracks

Variable: dtx for electrons with long tracks

Variable: dty for electrons with long tracks

Variable: dp_rel for electrons with long tracks

Variable: chi2PerDoF for electrons with long tracks

Variable: nDoF_f for electrons with long tracks

Variable: ghostProb for electrons with long tracks

Particle type: muon

Variable: dx for muons with long tracks

Variable: dy for muons with long tracks

Variable: dz for muons with long tracks

Variable: dtx for muons with long tracks

Variable: dty for muons with long tracks

Variable: dp_rel for muons with long tracks

/tmp/ipykernel_10971/4039132164.py:50: RuntimeWarning: ks_2samp: Exact calculation unsuccessful. Switching to method=asymp.
  *ks_2samp(bin_df[label].values.get(), bin_df[f"predicted_{label}"].values.get())])

Variable: chi2PerDoF for muons with long tracks

Variable: nDoF_f for muons with long tracks

Variable: ghostProb for muons with long tracks

Particle type: hadron

Variable: dx for hadrons with long tracks

Variable: dy for hadrons with long tracks

Variable: dz for hadrons with long tracks

Variable: dtx for hadrons with long tracks

Variable: dty for hadrons with long tracks

Variable: dp_rel for hadrons with long tracks

Variable: chi2PerDoF for hadrons with long tracks

Variable: nDoF_f for hadrons with long tracks

Variable: ghostProb for hadrons with long tracks

Upstream tracks

Particle type: electron

Variable: dx for electrons with upstream tracks

Variable: dy for electrons with upstream tracks

Variable: dz for electrons with upstream tracks

Variable: dtx for electrons with upstream tracks

Variable: dty for electrons with upstream tracks

Variable: dp_rel for electrons with upstream tracks

Variable: chi2PerDoF for electrons with upstream tracks

Variable: nDoF_f for electrons with upstream tracks

/tmp/ipykernel_10971/4039132164.py:60: UserWarning: Data has no positive values, and therefore cannot be log-scaled.
  plt.yscale('log')
/tmp/ipykernel_10971/4039132164.py:60: UserWarning: Data has no positive values, and therefore cannot be log-scaled.
  plt.yscale('log')
/tmp/ipykernel_10971/4039132164.py:60: UserWarning: Data has no positive values, and therefore cannot be log-scaled.
  plt.yscale('log')
/tmp/ipykernel_10971/4039132164.py:60: UserWarning: Data has no positive values, and therefore cannot be log-scaled.
  plt.yscale('log')

Variable: ghostProb for electrons with upstream tracks

Particle type: muon

Variable: dx for muons with upstream tracks

Variable: dy for muons with upstream tracks

Variable: dz for muons with upstream tracks

Variable: dtx for muons with upstream tracks

/tmp/ipykernel_10971/4039132164.py:50: RuntimeWarning: ks_2samp: Exact calculation unsuccessful. Switching to method=asymp.
  *ks_2samp(bin_df[label].values.get(), bin_df[f"predicted_{label}"].values.get())])

Variable: dty for muons with upstream tracks

Variable: dp_rel for muons with upstream tracks

/tmp/ipykernel_10971/4039132164.py:60: UserWarning: Data has no positive values, and therefore cannot be log-scaled.
  plt.yscale('log')

Variable: chi2PerDoF for muons with upstream tracks

/tmp/ipykernel_10971/4039132164.py:50: RuntimeWarning: ks_2samp: Exact calculation unsuccessful. Switching to method=asymp.
  *ks_2samp(bin_df[label].values.get(), bin_df[f"predicted_{label}"].values.get())])

Variable: nDoF_f for muons with upstream tracks

/tmp/ipykernel_10971/4039132164.py:60: UserWarning: Data has no positive values, and therefore cannot be log-scaled.
  plt.yscale('log')
/tmp/ipykernel_10971/4039132164.py:60: UserWarning: Data has no positive values, and therefore cannot be log-scaled.
  plt.yscale('log')
/tmp/ipykernel_10971/4039132164.py:60: UserWarning: Data has no positive values, and therefore cannot be log-scaled.
  plt.yscale('log')
/tmp/ipykernel_10971/4039132164.py:60: UserWarning: Data has no positive values, and therefore cannot be log-scaled.
  plt.yscale('log')

Variable: ghostProb for muons with upstream tracks

Particle type: hadron

Variable: dx for hadrons with upstream tracks

Variable: dy for hadrons with upstream tracks

Variable: dz for hadrons with upstream tracks

Variable: dtx for hadrons with upstream tracks

Variable: dty for hadrons with upstream tracks

Variable: dp_rel for hadrons with upstream tracks

Variable: chi2PerDoF for hadrons with upstream tracks

Variable: nDoF_f for hadrons with upstream tracks

/tmp/ipykernel_10971/4039132164.py:60: UserWarning: Data has no positive values, and therefore cannot be log-scaled.
  plt.yscale('log')
/tmp/ipykernel_10971/4039132164.py:60: UserWarning: Data has no positive values, and therefore cannot be log-scaled.
  plt.yscale('log')

Variable: ghostProb for hadrons with upstream tracks

Downstream tracks

Particle type: electron

Variable: dx for electrons with downstream tracks

Variable: dy for electrons with downstream tracks

Variable: dz for electrons with downstream tracks

Variable: dtx for electrons with downstream tracks

Variable: dty for electrons with downstream tracks

Variable: dp_rel for electrons with downstream tracks

Variable: chi2PerDoF for electrons with downstream tracks

Variable: nDoF_f for electrons with downstream tracks

Variable: ghostProb for electrons with downstream tracks

Particle type: muon

Variable: dx for muons with downstream tracks

/tmp/ipykernel_10971/4039132164.py:60: UserWarning: Data has no positive values, and therefore cannot be log-scaled.
  plt.yscale('log')
/tmp/ipykernel_10971/4039132164.py:60: UserWarning: Data has no positive values, and therefore cannot be log-scaled.
  plt.yscale('log')

Variable: dy for muons with downstream tracks

/tmp/ipykernel_10971/4039132164.py:60: UserWarning: Data has no positive values, and therefore cannot be log-scaled.
  plt.yscale('log')
/tmp/ipykernel_10971/4039132164.py:60: UserWarning: Data has no positive values, and therefore cannot be log-scaled.
  plt.yscale('log')
/tmp/ipykernel_10971/4039132164.py:50: RuntimeWarning: ks_2samp: Exact calculation unsuccessful. Switching to method=asymp.
  *ks_2samp(bin_df[label].values.get(), bin_df[f"predicted_{label}"].values.get())])

Variable: dz for muons with downstream tracks

/tmp/ipykernel_10971/4039132164.py:50: RuntimeWarning: ks_2samp: Exact calculation unsuccessful. Switching to method=asymp.
  *ks_2samp(bin_df[label].values.get(), bin_df[f"predicted_{label}"].values.get())])
/tmp/ipykernel_10971/4039132164.py:60: UserWarning: Data has no positive values, and therefore cannot be log-scaled.
  plt.yscale('log')
/tmp/ipykernel_10971/4039132164.py:50: RuntimeWarning: ks_2samp: Exact calculation unsuccessful. Switching to method=asymp.
  *ks_2samp(bin_df[label].values.get(), bin_df[f"predicted_{label}"].values.get())])

Variable: dtx for muons with downstream tracks

Variable: dty for muons with downstream tracks

Variable: dp_rel for muons with downstream tracks

Variable: chi2PerDoF for muons with downstream tracks

Variable: nDoF_f for muons with downstream tracks

Variable: ghostProb for muons with downstream tracks

Particle type: hadron

Variable: dx for hadrons with downstream tracks

/tmp/ipykernel_10971/4039132164.py:50: RuntimeWarning: ks_2samp: Exact calculation unsuccessful. Switching to method=asymp.
  *ks_2samp(bin_df[label].values.get(), bin_df[f"predicted_{label}"].values.get())])

Variable: dy for hadrons with downstream tracks

Variable: dz for hadrons with downstream tracks

/tmp/ipykernel_10971/4039132164.py:50: RuntimeWarning: ks_2samp: Exact calculation unsuccessful. Switching to method=asymp.
  *ks_2samp(bin_df[label].values.get(), bin_df[f"predicted_{label}"].values.get())])

Variable: dtx for hadrons with downstream tracks

Variable: dty for hadrons with downstream tracks

Variable: dp_rel for hadrons with downstream tracks

Variable: chi2PerDoF for hadrons with downstream tracks

Variable: nDoF_f for hadrons with downstream tracks

Variable: ghostProb for hadrons with downstream tracks

Kolmogorov-Smirnov distance as a quality metric

To assess the overall quality of the dataset we are using the KS distance as computed from the histograms discussed above. For each kinematic bin with a sufficient number of samples (100 or more) and for each output feature, we compute the KS distance. The KS distances are used to fill a histogram that provides a visual summary of the fitting of the model to the reference sample across the kinematic region covered by the training sample.

In general, we aim at the lowest possible values of the KS distance. In practice, values below 0.1 are conventionally considered as very good and values below 0.3 are considered acceptable. These thresholds may change increasing the statistics of the test set.

KS distance across the kinematic space

In the plot below, we report the value of the KS distance as a function the kinematic bin. The purpose of this visualization is to identify regions where the model performs better and regions where it is weaker. In general, it is observed, as expected, that the model performs better in regions where the training dataset is more populated.

No handles with labels found to put in legend.

No handles with labels found to put in legend.

No handles with labels found to put in legend.

No handles with labels found to put in legend.

No handles with labels found to put in legend.

No handles with labels found to put in legend.

No handles with labels found to put in legend.

No handles with labels found to put in legend.

No handles with labels found to put in legend.

No handles with labels found to put in legend.

No handles with labels found to put in legend.

No handles with labels found to put in legend.

No handles with labels found to put in legend.

No handles with labels found to put in legend.

No handles with labels found to put in legend.

No handles with labels found to put in legend.

No handles with labels found to put in legend.

No handles with labels found to put in legend.

Conclusion

In this notebook we discussed the validation procedure of the resolution model, comparing the distributions of the predicted features with the reference test set.

Additional splitting and variables might be included in the future to provide a more comprhensive view on the performance of the model.

To ease the identification of regions where the model is weaker, we draw the KS distance obtained comparing the distribution of each variable, in each kinematic bin.