# ADD metrics computation and figure

## Requirements

Run the download content file: `./download_content.sh`, this downloads a simple scene with annotation rendered by NViSII and with DOPE predictions.

## How to run

If you downloaded the previous content you can execute the following:

```
python add_compute.py
```
and you should get the following displayed:
```
mean 0.0208515107260977 std 0.016006083915162977 ratio 17/22
auc at 0.02 : 0.5
auc at 0.04 : 0.6818181818181818
auc at 0.06 : 0.7272727272727273
auc 0.6115249999999999
```
This means the area under the curve, *auc* from 0 cm to 10 cm is 0.61. This script also produces graphs such as:

![example of graph](results/output.png)

These are the metrics we reported in the original DOPE paper. I will refer to the paper for explaining the graph.

## Assumptions
We make a few assumptions in this script.
1. We assume the folders structures are the same and there are only scenes in the folder. See `data/` folder example from downloading the content.
2. We assume the notation folder is in the opengl format and that it is using the nvisii outputs from the data generation pipeline. If you use a diffirent file format please update the script or your data.
3. We assume the inferences are from DOPE inference, _e.g._, the poses are in the opengl format. These conventions are easy to change, _e.g._, look for the line `visii_gu.get_transform().rotate_around` in `add_compute.py` to change the pose convention.

If the script takes to long to run, please run with `--cuboid`, instead of using the 3d models vertices to compare the metric, it uses the 3d cuboid of the 3d model to compute the metric.

## 2D image-based metric

If you do not have a 3d model of your model and you would prefer to just measure the quality of your detections with a simple euclidean distance for the predicted keypoints. You can use `python kpd_compute.py`, this is very similar to `add_compute.py` and it behaves very similarly.
The metric used here is the euclidean (L2) distance from predicted keypoint and the ground truth keypoint. Then we propose to use a threshold plot to evaluate the data, similar to the ADD metric.

# Rendering 3d predictions using NViSII

![example of overlay](overlay.png)

We added a script for you to add render of the 3d model to your predictions. It uses a version of NViSII that is not released yet. Please manually install this [wheel](https://www.dropbox.com/s/m85v7ts981xs090/nvisii-1.2.dev47%2Bgf122b5b.72-cp36-cp36m-manylinux2014_x86_64.whl?dl=0).
```
# for scenes with DOPE inference
python render_json.py --path_json data/table_dope_results/scene1/00300.json --scale 0.01 --opencv --contour --gray
# for scenes generated by nvisii
python render_json.py --path_json data/table_ground_truth/scene1/00100.json --scale 0.01 --contour --gray
```

`--gray` render the 3d model as a gray image and `--contour` adds the 3d model contour in green.

## Rendering BOP format on images

Using the same argument, you can use this script on the BOP annotation with 3d models. The script simply rebuilds the data structure that is needed to load the scene.

```
python render_json.py --path_json /PATH/TO/hope_bop/hope_val/val/000001/scene_gt.json --bop --objs_folder /PATH/TO/hope_bop/hope_models/models/ --gray --contour --bop_scene 0
```

Only `--bop` is needed to be passed to load a bop scene. You can pass which scene you want to load with `--bop_scene`. The rest is the same behaviour. This was only tested on the HOPE data.

## Assumptions

We assume that you have the intrinsics stored in the camera data. If you do not have them, the script uses 512 x 512 with a fov of 0.78. If the camera data is complete, like with NViSII data, it will use the camera intrinsics.