Submission to the DRAGON leaderboard¶
Submissions to the DRAGON benchmark should be open-sourced and accompanied by a method description. The open-source submission may be kept private temporarily to facilitate e.g. manuscript submissions. For more info, please see here.
Figure: Evaluation method for the DRAGON benchmark. Challenge participants must provide all resources necessary to process the reports and generate predictions for the test set. Any processing of reports is performed on the Grand Challenge platform, without any interaction with the participant.
Grand Challenge Algorithm Template¶
Submissions to the DRAGON challenge are to be made as training+inference containers. These containers are equivalent to Grand Challenge (GC) algorithms. They are Docker containers that encapsulate the training resources (e.g., fine-tuning strategy, pretrained model weights) and the components needed to generate predictions for the test cases.
The following code repository contains such a submission. You can use this repo as a template for your submission: DRAGON submission repository.
Before implementing your algorithm using this template, we recommend creating a GC algorithm based on the unaltered template. In the following sections, we will provide a walkthrough of how to do this.
Installing Docker Locally 🐳¶
You must have Docker installed and running on your system for the following steps to work. If you are using Windows, we recommend installing Windows Subsystem for Linux 2 (WSL2). For more details, you can watch the official tutorial by Microsoft for installing WSL 2 with GPU support.
Building and Testing Locally 🐳¶
We highly recommend first completing this general tutorial on creating GC algorithms: https://grand-challenge.org/documentation/create-your-own-algorithm/.
Note: The provided shell scripts (.sh files) stated in all following
commands should work on Linux, macOS, and Windows (WSL2).
Clone the template repo (see above) to your local system. Build and test
the Docker container by running build.sh, followed by test.sh.
Please note, that testing also runs a build, so running the build
command separately is optional (if you are certain that everything is
set up correctly). Testing involves fine-tuning your algorithm to nine
synthetic datasets provided in the ./test-input/ folder. This will
trigger nine training runs, so this can take some time. It will then
check the test predictions produced by your algorithm against the labels
and display the performance. Since training is typically not
deterministic (even with the same random seed when changing hardware),
we don't compare against pre-computed test predictions. Instead, we
check if all training runs perform better than random guessing. If all
tests have been completed successfully, you're good to go!
Adding Algorithm Container to Grand Challenge¶
Algorithm containers need to have the corresponding input/output
interfaces of the DRAGON challenge to function. To have these set up
automatically, navigate to the "Submit" page and complete the form "on this page":
Once complete, click the "Save" button. At this point, your GC algorithm has been created and you are on its homepage. Now, follow the steps here to link your forked DRAGON submission repository to your algorithm.
Once the algorithm is linked, tag your repository to start a build (see the documentation for more information).
It typically takes 20-60 minutes till your container is built and has
been activated (depending on the size of your container). After its
status is "Active", test your container with a synthetic dataset from
the
./test-input/ directory of the DRAGON submission repository by
navigating to the "Try-out Algorithm" page of your algorithm.
The required input interfaces for this challenge are:
- NLP Task Configuration
- NLP Training Dataset
- NLP Validation Dataset
- NLP Test Dataset
The required output interfaces for this challenge are:
- NLP Predictions Dataset
To learn more about GC input/output interfaces, visit https://grand-challenge.org/components/interfaces/algorithms/.
Create Containers for Your Own AI Algorithm¶
Following these same steps, you can encapsulate your own AI algorithm in
a training+inference container by altering the provided algorithm
template. You can implement your own solution by editing the functions
in ./process.py, there are two example alterations provided there too.
Any additional imported packages should be added to
./requirements.txt, and any additional files and folders should be
explicitly copied through commands in the ./Dockerfile. To update your
algorithm, you can test your new Docker container, after which you can
update the forked repository and tag it (following the instructions
here). Please note, you only have internet access
when building the Docker container on Grand Challenge. Your
algorithm container will not have access to the internet when they are
executed on grand-challenge.org, so all necessary model weights and
resources must be encapsulated in your container image apriori. You can
test whether this is true locally using the --network=none option of
docker run.
If something doesn't work for you, please feel free to make an issue in the DRAGON submission GitHub.
Submission to the DRAGON Challenge¶
Once you have your training+inference container uploaded as a fully-functional GC algorithm, you're now ready to make submissions to the challenge! Please follow each of the steps below to ensure a smooth submission process.
1. Make a submission to the Synthetic Leaderboard [PENDING]¶
Navigate to the "Submit" page and select the "Synthetic" phase. There, select your algorithm, and click "Save" to submit. If there are no errors and evaluation has completed successfully, your score will be up on the leaderboard (typically in less than 24 hours). This ensures your algorithm can handle all the task types in the DRAGON challenge and is a mandatory step! This way, we can identify potential issues with your algorithm submission early. This helps to keep costs low, allowing us to keep supporting the DRAGON challenge!
2. Make a submission to the Validation Leaderboard [PENDING]¶
Navigate to the "Submit" page and select the "Validation" phase. There, select your algorithm, and click "Save" to submit. If there are no errors and evaluation has completed successfully, your score will be up on the leaderboard (typically in less than 24 hours). This ensures your algorithm can handle all the tasks in the DRAGON challenge and is a mandatory step! This way, we can identify potential issues with your algorithm submission early. This helps to keep costs low, allowing us to keep supporting the DRAGON challenge!
3. Make a submission to the Test Leaderboard [PENDING]¶
Finally, navigate to the "Submit" page and select the "Test" phase. There, select your algorithm.
This submission also requires two aspects to be checked by the challenge organizers (as specified in more detail here):
- The submission is open-source
- The method is described in the supporting document
Please provide this when submitting your algorithm. Without a valid open-source license and supporting document, the algorithm will not be processed and no score will be shown on the leaderboard!
With all information provided, click "Save" to submit. If there are no errors and evaluation has completed successfully, your algorithm will be processed (typically in less than 24 hours). Your score will be shown on the leaderboard after a manual verification of the open-source license and supporting document.
Invalid submissions will be removed and teams repeatedly violating any/multiple rules will be disqualified.
We look forward to your submissions to the DRAGON challenge. All the best! 👍