FAQ

How did you index the document collection?

• We used an out-of-the-box version of OpenSearch (ver. 3.5.0) to index documents.
• For sparse encoding, we used opensearch-project/opensearch-neural-sparse-encoding-multilingual-v1 for English and Japanese.
• See indexing and model hosting for replicating our environment.

Can I use a locally deployed gpt-oss-120b?

Yes. geniie-lab supports ollama and vllm for local models. See Model Description of Common settings

How can I resume my experiment from a particular topic?

If you have a total of 50 topics, and your experiment stopped at Nth topic, then

topic_ids="N:50"

allows you to resume the experiment from Nth topic.

What should I do when experiments failed on the same topic multiple times?

• In most cases, submitting the same instruction (prompts) will eventually get through to generate valid responses, without changing any parameters.
• Sometimes, you might need to try several times.
• However, if the problem persists, you can relax the temperature and top_p parameters only for those problematic topics, to maximise the reproducibility of experimental results.

models=[
    ModelDescription(
        ...,
        temperature=0.7,
        top_p=0.95
    )
]

How can I see the exact transaction between the LLM and geniie-lab?

Set full_log=True

How much will Amazon Bedrock usage cost?

• gpt-oss-120b on Amazon Bedrock is priced at $0.15 per million input tokens and $0.60 per million output tokens (check the current prices at Amazon Bedrock pricing).
• The pilot experiment in Getting Started costs well under $0.01.
• Measured on our baseline runs with default instructions (TREC Robust 2004 fold1, 50 topics, five iterations): a full session uses roughly 0.7-0.9M tokens per topic, so one 50-topic experiment is 34-44M tokens, or about $7-9.
• One run (BM25 + Sparse Encoder session files) is therefore roughly $15-20.

Not finding answers?

Contact us via Discord server or email agenticinstruction-org at googlegroups dot com.