Skip to content

How to submit runs

Important Rules

  • All development, training, and testing must be done on train/dev datasets
  • You cannot change/train/test your instruction method using test datasets
  • You cannot use qrels data of test datasets
  • You must generate log files using test datasets for run submission
  • You cannot edit the generated log files except session names

What is a run?

A run at AgenticInstruction Task consists of jsonl outputs generated by geniie-lab using your instruction method. If you developed two instruction methods, you will have two run-id. Submitted runs will be included in the task overview paper.

How many session files do we need to include per run?

Participants are expected to include at least two session files per run. One using BM25, and another using Sparse Encoder (denoted as splade) for your choice of language (English or Japanese).

If your method is applied to both English and Japanese, then four session files should be included per run (English + BM25, English + Sparse, Japanese + BM25, Japanese + Sparse).

How many runs can we submit?

We plan to accept up to THREE runs per group for NTCIR-19.

💡 You can perform additional experiments with more methods and report findings in your participant paper.

Session Name

See Common settings to learn how to set a session name in your experiments.

Format

[group-id]-[language]-[ir-model]-[run-id]

  • language: en or ja
  • ir-model: bm25 or splade (Sparse Encoder)
  • run-id: 1, 2, 3 (Your instruction method ID, up to 3)

Examples

Group ID Language IR Model Run ID Session Name
tsukuba48 English BM25 1 tsukuba48-en-bm25-1
tsukuba48 English BM25 2 tsukuba48-en-bm25-2
tsukuba48 English Sparse Encoder 1 tsukuba48-en-splade-1
tsukuba48 Japanese BM25 1 tsukuba48-ja-bm25-1

💡 It's okay to change the session names after the run files are generated.

Run File Name

geniie-lab saves log files in the logs folder: .jsonl and .log. Please rename each .jsonl file to its session name plus the .jsonl extension.

Example

A session named tsukuba48-en-bm25-1 is submitted as tsukuba48-en-bm25-1.jsonl.

Submission

Send your jsonl files to agenticinstruction-org at googlegroups dot com by the submission deadline. Follow the instruction below.

  • Subject: NTCIR-19 AgenticInstruction Task Submission
  • Body: Make sure to include your group ID and brief summary description of each of your runs

Questions?

Contact us via Discord server or email agenticinstruction-org at googlegroups dot com.