Task Description for NTCIR-19

Important Rules

All development, training, and testing must be done on train/dev datasets
You cannot change/train/test your instruction method using test datasets
You cannot use qrels data of test datasets
You must submit jsonl log files generated by test datasets
You cannot edit the generated log files except session names

Goal of AgenticInstruction Task at NTCIR-19

Goal of AgenticInstruction Task at NTCIR-19 is to develop instruction methods for an LLM to maximise a total number of unique relevant documents identified (recall) within five iterations of search sequences, while minimising a total number of tokens sent to and produced by the LLM.
A search sequence includes a query, ranking, click, and relevance judgement
A relevant document in this task is
- a document clicked (selected) from a SERP by the LLM; and
- a document judged as relevant by the LLM; and
- a document labeled as relevant by official qrels

How to change the size of iterations

Iteration is defined by plan parameter. Start with a small plan.

Iteration x 1Iteration x 2Iteration x 3 to 5For debugging or testing

plan=["query", "ranking", "click", "relevance"]

plan=["query", "ranking", "click", "relevance"] + ["reformulate", "ranking", "click", "relevance"]

# 3 iterations
plan=["query", "ranking", "click", "relevance"] + ["reformulate", "ranking", "click", "relevance"] * 2
# 4 iterations
plan=["query", "ranking", "click", "relevance"] + ["reformulate", "ranking", "click", "relevance"] * 3
# 5 iterations (Target)
plan=["query", "ranking", "click", "relevance"] + ["reformulate", "ranking", "click", "relevance"] * 4

plan=["query"]                        # Initial query formulation
plan=["query", "ranking"]             #   plus ranking
plan=["query", "ranking", "click"]    #   plus clicking

Index name

Look up the tables below to find an index name that matches your selection of dataset and IR model. You will need to use the index name in multiple places in the experimental settings of geniie-lab.

EnglishJapanese

Index name	Dataset	IR Model
`trec_robust_2004_bm25`	TREC Robust 2004	BM25
`trec_robust_2004_splade`	TREC Robust 2004	Sparse Encoder
`trec_robust_2005_bm25`	TREC Robust 2005	BM25
`trec_robust_2005_splade`	TREC Robust 2005	Sparse Encoder

Index name	Dataset	IR Model
`ntcir1_bm25`	NTCIR-1 AdHoc	BM25
`ntcir1_splade`	NTCIR-1 AdHoc	Sparse Encoder
`ntcir2_bm25`	NTCIR-2 AdHoc	BM25
`ntcir2_splade`	NTCIR-2 AdHoc	Sparse Encoder

Customisable parameters

This is your design space to explore. Making changes to any of these parameters could potentially change the output of a LLM-based search agent. Edit the experimental settings in run_session_experiment_ntcir19.py

System promptTopic fieldsTool descriptionInstruction for query, click, judge, and reformulation

models=[
    ModelDescription(
        ...
        system_prompt="You're a helpful assistant",
        ...
    )
]

topicset=TopicDescription(
    ...
    topic_class=FullTopic # TitleOnlyTopic, TitleDescriptionTopic, TitleNarrativeTopic
)

ToolDescription(
    ...
    description="It allows you to perform searches using keywords only and employs the BM25 ranking model to order results.",
)

stages={
    "query": StageConfig(
        instruction="""
            Review the provided descriptions of task, corpus, tool and search topic. Then, formulate a search query.
        """,
    ),
    "ranking": StageConfig(
        instruction="" # This is not used.
    ),
    "click": StageConfig(
        instruction="""
            Select a set of documents that are likely to contain relevant information to the search topic. Return an empty list if none of the results appears relevant.
        """,
    ),
    "relevance": StageConfig(
        instruction="""
            Evaluate the relevance of the document based on the search topic description and its narrative.
        """,
    ),
    "reformulate": StageConfig(
        instruction="""
            Formulate another search query to find new relevant documents.
        """,
    )
}

Fixed parameters

Don't change these parameters for your submitted runs. Parameters masked by ... means you can modify them.

Task DescriptionCorpus Description

task=TaskDescription(
    name="High-Recall Retrieval",
    description="Find as many different relevant documents as possible for a given search topic from a given document collection using a provided search tool.",
    ...
)

# Robust 2004 (Train/Dev)
corpus=CorpusDescription(
    name="Disk45",
    description="A document collection of about 528,000 English news documents. Sources include the Financial Times (1991-1994), the Federal Register (1994), the Foreign Broadcast Information Service (1996), and the Los Angeles Times (1989-1990).",
    ...
)
# Robust 2005 (Test)
corpus=CorpusDescription(
    name="Aquaint",
    description="A document collection of about 1M English newswire text. Sources include the Xinhua News Service (1996-2000), the New York Times News Service (1998-2000), and the Associated Press Worldstream News Service (1998-2000).",
    ...
)
# NTCIR-1 AdHoc (Train/Dev)
corpus=CorpusDescription(
    name="NTCIR1",
    description="A collection of about 340,000 Japanese academic documents. Sources include author abstracts of academic conference papers hosted by 65 Japanese academic societies (1988-1997).",
    ...
)
# NTCIR-2 AdHoc (Test)
corpus=CorpusDescription(
    name="NTCIR2",
    description="A collection of about 736,000 Japanese academic documents. Sources include author abstracts of academic conference papers hosted by Japanese academic societies (1997-1999) and extended summaries of grant reports (1986-1997).",
    ...
)

How to change IR model

BM25Sparse Encoder

tools=[
    ToolDescription(
        name="opensearch",
        ranking_model="bm25",
        index_name="[index name]",
        host=os.getenv("OPENSEARCH_HOST"),
        port=9200,
        ...
    )
]

tools=[
    ToolDescription(
        name="opensearch",
        ranking_model="splade",
        encode_model="opensearch-project/opensearch-neural-sparse-encoding-multilingual-v1",
        index_name="[index name]",
        host=os.getenv("OPENSEARCH_HOST"),
        port=9200,
        ...
    )
]

How to change datasets

TREC Robust 2004 (Fold 1/2)TREC Robust 2005NTCIR-1 AdHocNTCIR-2 AdHoc

topicset=TopicDescription(
    name="disks45/nocr/trec-robust-2004/fold1",
    # name="disks45/nocr/trec-robust-2004/fold2",
    type="ir_datasets",
    ...
)
...
corpus=CorpusDescription(
    name="Disk45",
    description="A document collection of about 528,000 English news documents. Sources include the Financial Times (1991-1994), the Federal Register (1994), the Foreign Broadcast Information Service (1996), and the Los Angeles Times (1989-1990).",
    ...
)

topicset=TopicDescription(
    name="aquaint/trec-robust-2005",
    type="ir_datasets",
    ...
)
...
corpus=CorpusDescription(
    name="Aquaint",
    description="A document collection of about 1M English newswire text. Sources include the Xinhua News Service (1996-2000), the New York Times News Service (1998-2000), and the Associated Press Worldstream News Service (1998-2000).",
    ...
)

Download ntcir1_adhoc.py and ntcir1-adhoc.yaml from geniie-backend to the scripts folder of geniie-lab. The experiment script imports the loader automatically for ntcir* dataset names.

topicset=TopicDescription(
    name="ntcir1-adhoc",
    type="ir_datasets",
    ...
)
...
corpus=CorpusDescription(
    name="NTCIR1",
    description="A collection of about 340,000 Japanese academic documents. Sources include author abstracts of academic conference papers hosted by 65 Japanese academic societies (1988-1997).",
    ...
)

Download ntcir2_adhoc.py and ntcir2-adhoc.yaml from geniie-backend to the scripts folder of geniie-lab. The experiment script imports the loader automatically for ntcir* dataset names.

topicset=TopicDescription(
    name="ntcir2-adhoc",
    type="ir_datasets",
    ...
)
...
corpus=CorpusDescription(
    name="NTCIR2",
    description="A collection of about 736,000 Japanese academic documents. Sources include author abstracts of academic conference papers hosted by Japanese academic societies (1997-1999) and extended summaries of grant reports (1986-1997).",
    ...
)

Advanced instructions

Tips for hacking geniie-lab code base to achieve advanced method of generating instructions.

You might need to re-run python -m pip install -e . to activate the change.

Instruction templateDescription of structured output

See https://github.com/geniie-lab/geniie-lab/blob/dev/geniie_lab/dataclasses/instruction.py
Example of template for query formulation

    instruction = f"""
        **Instruction**:
        {self.instruction}
        ============================
        **Task Description**: {self.task.description}
        **Corpus Description**: {self.corpus.description}
        **Search Tool Description**: {self.tool.description}
        **Topic Description**: {self.topic}
    """

See https://github.com/geniie-lab/geniie-lab/blob/dev/geniie_lab/response.py
Example of descriptions in Query output

class Query(BaseModel):
    """A model for submitting a query to a search tool."""
    query: str = Field(
        ...,
        title="query",
        description="The query string submitted to the search tool."
    )
    start: int = Field(
        0,
        title="start",
        description=(
            "The starting index of the search results. Defaults to 0. "
        )
    )
    size: int = Field(
        10,
        title="size",
        description=(
            "The number of documents per search result page. Defaults to 10. "
        )
    )
    reason: str = Field(
        ...,
        title="reason",
        description="A brief explanation of the intent behind your query."
    )

How to measure the performance

When an experiment finishes successfully, you will see the performance statistics like below: a per-topic summary with the progression of cumulative unique relevant documents over iterations, followed by a per-iteration aggregate showing where improvements level off.

Topic       Unique rel docs    Recall   Total tokens   Total time (s)  Progression
302                      14     0.215        1767046            211.3  6→10→14→14→14
303                       2     0.200         311770             62.6  2→2→2→2→2
309                       0     0.000         388054             84.9  0→0→0→0→0
316                       2     0.059         201905             53.7  2→2→2→2→2
317                       8     0.571        1119621            167.1  7→8→8→8→8
...

Iteration  Topics  New rel docs  Cum rel docs  Cum recall      Tokens
1               5          3.40          3.40       0.170       31775
2               5          1.00          4.40       0.197       92826
3               5          0.80          5.20       0.209      161501
4               5          0.00          5.20       0.209      208060
5               5          0.00          5.20       0.209      263517

Organiser baseline

For reference, we provide baseline performance for all six official configurations, produced with the default instructions in run_session_experiment_ntcir19.py (gpt-oss-120b via Amazon Bedrock, five iterations). Use these as a starting point to beat.

Language	Dataset	IR Model	Baseline
English	TREC Robust 2004 (fold1)	BM25	results
English	TREC Robust 2004 (fold1)	Sparse Encoder	results
English	TREC Robust 2004 (fold2)	BM25	results
English	TREC Robust 2004 (fold2)	Sparse Encoder	results
Japanese	NTCIR-1 AdHoc	BM25	results
Japanese	NTCIR-1 AdHoc	Sparse Encoder	results

These baselines use default instructions with no optimisation — the goal of the task is to design instructions that identify more unique relevant documents (higher recall) using fewer tokens.

Need help?

Contact us via Discord server or email agenticinstruction-org at googlegroups dot com.