Platform Updates - What's new ?

THIS IS A TEST!

Platform Updates - What's new ?

The first challenge hosted in the platform is now over. We have learnt a lot of things during these months and also received feedback from participants and the rest of the community. In the spirit of always improving and in order to provide the best possible user experience, we have been working hard on new features and changes to the platform. These are some of them:

New scoring mechanism

The biggest change made is to how we compute and rank teams based on their scores. We are aware that for the first challenge this was a little bit strange and cumbersome, involving multiple metrics where teams where positioned in the leaderboard according to their relative scores. For new and future challenges, we have changed this to take into account a single metric directly reflected on the leaderoboard.

Best submissions to the leaderboard

For the first challenge, only the last submission made to the platform was used for the leaderboard. This was in part due to the difficulty of the scoring mechanism previously explained. For new and future challenges, we have changed this to use the best submission (the one with the best public score, more on this in the next section).

Public and Private leaderboards

For the first challenge we had two different phases, one for tuning models with a dataset and a final short phase to use the models on new datasets for the final evaluation. We have changed this to use only one dataset during all the challenge, in a single phase. In order to avoid teams optimizing their submissions for the test data, the leaderboard is split into a public and a private leaderboard. The public leaderboard will reflect the evaluation metric obtained with a small random subset of the test data, while the private leaderboard is built with the scores obtained in the rest of the test data. This splits are hidden to the participants.

During the challenge, only the public leaderboard will be visible, as well as public scores in the submissions section. Once the challenge ends, the private scores will be visible, and the final ranking will be formed based on these scores.

FAQs

We also made available a list of the most common questions received during the first challenge in the support section. You are encouraged to take a look to these in case you have some doubts.

Other changes

You will also find some other changes, such as tag-based filtering mechanisms for challenges and the forum, a markdown-based text editor for your topics and comments on the forum and visual updates on some sections.

We hope you enjoy the use of the AI4EO challenges platform, and if you encounter issues do not hesitate to contact with us at support@ai4eo.eu.

Good luck, and have fun !