Mandatory checks

This page illustrates all the information regarding what to check before and while you push your modifications.

The development of this software is monitored by a Continuous Integration (CI) pipeline that takes care of all checks (some using external services).

Documentation

Each Pull Request (PR) has to have its own documentation updates (if any), according to what are the changes to be merged into master.

The documentation is generated by Sphinx with reStructuredText syntax.

To build and check your documentation locally,

  • cd docs

  • for big changes (or just to be sure), rm api/* && make clean && make html

  • for small changes, make html

The built documentation will be stored under docs/_build/html from which you can open index.html with your favorite browser.

You will have to fix any warning that appears during documentation building, because the documentation also runs on readthedocs with an option to treat warnings as errors.

Unit and integration tests

Note

This is a maintenance activity which has being long overdue and we need manpower for it, so if you have experience on this or you want to contribute please feel free to do so.

For more information on how to contribute to this effort check these issues,

Being protopipe based on ctapipe, all the tools imported from the latter have been already tested and approved (protopipe uses always a version of ctapipe which has been released on the Anaconda framework). Same for pyirf.

Warning

This is not true for,

  • hard-coded parts that had to be modified in anticipation of code migration,

  • protopipe functions themselves (which will eventually migrate to ctapipe)

Regarding the first point: given the difference in versions between the imported ctapipe and its development version, sometimes it’s possible that, in order to code a new feature, this has to be pull-requested to ctapipe and at the same time hardcoded in protopipe, until the new version of ctapipe is released.

For the moment there is only one integration test.

This test is in charge to detect if changes in the script protopipe.scripts.data_training and any code used by it, produce any fatal behaviour or crash.

For the moment it uses an old test file from ctapipe (a Prod2 CTA South array composed of LSTCam, FlashCam and ASTRICam with about ~100 simulated showers). Each test is expected to produce a non-empty HDF5 file.

The test can be executed directly from the main folder of protopipe by launching pytest. It is also automatically triggered by the CI every time a new pull-request is pushed to the repository, and its correct execution is a mandatory condition for merging.

Benchmarks

This documentation hosts a series of notebooks used for benchmarking.

Their contents follow the development triggered by the ongoing comparison between protopipe and CTA-MARS (see this issue and references therein for a summary).

The following tables summarize the analyses currently performed and planned with protopipe to evaluate its performance and that of the observatory. Links are provided for the simtel files lists and reference data from historical pipelines, which is used throughout the notebooks for comparison and performance monitoring.

Note

Also this is a heavy-load activity which could be shared.

If you’re interested in analysing any of the simulations listed here or you want to propose new ones you are more than welcome to do it and contribute to the benchmarking following the set of prescriptions described in this documentation.

PROD 3B

Site

Array

Zenith

Azimuth

simtel

CTAMARS

North (La Palma)

baseline

20°

180°

link

link

South (Paranal desert)

baseline

20°

180°

…

…
PROD 5

Site

Array

Zenith

Azimuth

simtel

EventDisplay

North (La Palma)

baseline

20°

180°

…

…

South (Paranal desert)

baseline

20°

180°

…

…

In the documentation we show only the full-pipeline performance results from the latest release. It is suggested to open the notebooks with jupyter lab from their location at docs/contribute/benchmarks.

The benchmarks are organised as follows,

In particular:

  • calibration requires ctapipe DL1a output (images without parameters),

  • all image cleaning and direction reconstruction benchmarks use protopipe TRAINING data without estimated energy,

  • all benchmarks for the energy estimator use protopipe TRAINING data without estimated energy,

  • benchmarks for the classifier use protopipe TRAINING data with energy as the only estimated DL2b parameter.

  • MODELS

These performances are obtained from a test portion of the TRAINING data,

  • Energy | benchmarks_MODELS_energy.ipynb

  • Particle type | benchmarks_MODELS_classification.ipynb

The DL3 folder contains also the CTA requirements, while the ASWG performance data is left to the user, being internal.

Note

This part of protopipe is not meant to be kept here in the end, in order to avoid divergences with ctaplot and cta-benchmarks.

Plots should be properly migrated to ctaplot and the single pipeline steps will ported to cta-benchmarks after the pipeline has been refactored using ctapipe’s stage tools.

Note

The storage of static versions of the benchmarks in this documentation is temporary. It is planned to run such benchmarks from the CTAO GitLab runner’s CI via an external IN2P3 data server in which test data, benchmarks and documentation will be stored.

For the moment the purpose of these tools is to help developers and testers to check if their changes improve or degrade previous performances stored in the documentation.