Evaluation

Benchmark Platform

We use Codabench (testA, testB) for evaluation the test subsets.

Submission Steps

Step 1 - Benchmark platform:

Sign up your account in Codabench and participate the benchmark (please note all participants should submit the form and send us the signed EULA in registration to get the dataset).

Step 2 - Prepare predictions:

Prepare your submission files with the format mentioned in details. Name your prediction files for Task 1 as "prediction.txt", prediction file for Task 2 as "prediction.json". You can include either or both of them based on the tasks you participate, then compress them as a zip file.

Step 3 - Submission:

In Codabench tab "My Submissions", section "Submission upload", select the tasks, then upload the zip file.

Classification

# Team TestA TestB Average
1 XJTU SunFlower Lab 0.9783 0.9568 0.9676
2 Pindrop Labs 0.9249 0.8936 0.9093
3 KLASS 0.9278 0.8654 0.8966
4 KETI 0.8799 0.8407 0.8603
5 AILS-NTUA 0.8238 0.7628 0.7933
- Baseline Xception 0.5509 0.5729 0.5619

Temporal Localization

# Team TestA TestB Average
1 Pindrop Labs 0.6720 0.4703 0.5712
2 KLASS 0.3536 0.2435 0.2986
- Baseline BA-TFD+ 0.1471 0.1515 0.1493
- Baseline BA-TFD 0.1354 0.1117 0.1236

Temporal Localization on TestA

# Team TestA Score AP@0.5 AP@0.75 AP@0.9 AP@0.95 AR@50 AR@30 AR@20 AR@10 AR@5
1 Pindrop Labs 0.6720 0.7794 0.6652 0.4466 0.3427 0.8030 0.8009 0.7959 0.7784 0.7493
2 KLASS 0.3536 0.5117 0.4017 0.1701 0.0416 0.4259 0.4259 0.4259 0.4259 0.4258
- Baseline BA-TFD+ 0.1471 0.1401 0.0235 0.0005 0.0000 0.3280 0.3017 0.2661 0.2088 0.1611
- Baseline BA-TFD 0.1354 0.0981 0.0129 0.0004 0.0000 0.3325 0.2916 0.2520 0.1924 0.1459

Temporal Localization on TestB

# Team TestB Score AP@0.5 AP@0.75 AP@0.9 AP@0.95 AR@50 AR@30 AR@20 AR@10 AR@5
1 Pindrop Labs 0.4703 0.5268 0.3558 0.1631 0.0901 0.6872 0.6799 0.6696 0.6439 0.6030
2 KLASS 0.2435 0.3606 0.2795 0.1061 0.0213 0.2950 0.2950 0.2950 0.2950 0.2950
- Baseline BA-TFD+ 0.1515 0.1620 0.0339 0.0012 0.0001 0.3245 0.2935 0.2651 0.2156 0.1698
- Baseline BA-TFD 0.1117 0.0463 0.0057 0.0002 0.0000 0.2871 0.2514 0.2203 0.1681 0.1250