LLM research · reproduction · domain adaptation
TISER Temporal Reasoning
A reproduction of ACL 2025 TISER on Qwen2.5-7B, extended with a context-memory conflict probe and a professional-tennis temporal reasoning adaptation study.
Problem
The project asks two questions beyond simply running an LLM benchmark: can the structured temporal-reasoning target from TISER be faithfully reproduced, and does that reasoning behaviour transfer when the model is pushed into conflicting context or a new domain?
What I built
The repository fine-tunes Qwen2.5-7B-Instruct with LoRA on the released TISER training data, evaluates exact match and token F1 on the full benchmark, then adds two experimental branches: a context-memory conflict probe and a tennis adaptation pipeline with audited synthetic examples and continued fine-tuning.
Macro F1 across the five in-domain splits; macro EM 0.878 over 22,014 test examples.
Engineering decisions
Where the project becomes more than a demo.
The details that matter are usually in the constraints: who is allowed to decide, what gets measured, what is kept separate, and how failure is represented.
Reproduce before extending.
The baseline uses the released prompt/output format and full training/evaluation path before interpreting extension results.
Keep evaluation split-aware.
Five in-domain datasets define the reported macro; the OOD ToT-semantic split is reported separately rather than blended into a flattering aggregate.
Test failure modes, not just scores.
The context-memory study measures whether the model follows edited evidence over memorized facts and audits whether the reflection actually identifies the contradiction.
Treat domain adaptation as an experiment.
Tennis examples are audited, converted into the same structured format and compared across base, transferred and continued-adaptation conditions.
Evidence
What the repository actually supports.
five in-domain splits on the full TISER test set
22,014 total evaluated examples
7B TISER continued adaptation; F1 0.856
rows where the reflection explicitly names the context-memory contradiction
Limits
What I am not claiming.
- The baseline reproduction does not exactly match every number in the paper; the repository reports the reproduced numbers rather than hiding the gap.
- The OOD ToT-semantic split is substantially harder and is excluded from the in-domain macro by design.
- The conflict study shows that a structured reflection can improve behaviour without reliably verbalising the contradiction.
Signal