LLM4Subjects

What I believed in 2024

This was the first real training project of my NLP master's, in the autumn of 2024. I believed the strongest available model wins and the job is to fine-tune it well, so I built a classifier: multilingual BERT with a dense head over the 14,607 subject labels that appear in the training split, fifteen epochs on an A100. It was not competitive, and the compute it wanted was wildly out of proportion to what the task actually asks for.

Two years later I came back and rebuilt it, starting from the data instead of the model. The first thing I found was that my old numbers were not wrong so much as meaningless: 142 records in the training set were also in the gold test split, and 8,966 titles had been pulled from the wrong track. Re-measured on a clean split, the 2024 classifier scores 0.0667 micro recall@10. The rebuilt pipeline scores 0.6044 on the same dev split.

The damaging part was not the headline. It was the band breakdown. That classifier scores exactly 0.0000 on the tail and on unseen labels, at every cutoff out to 50, and it emits 78 distinct codes out of 14,607 anywhere in 5,354 × 50 prediction slots. It had quietly learned to answer every record out of the same few dozen common headings.

That is not a training failure, it is a shape failure. 19.1% of the gold labels in the test split never appear in training at all, and a closed-vocabulary head has no output column for them. No amount of fine-tuning creates one. So the rebuild treats the problem as retrieval over a vocabulary rather than classification into a label set, and every decision after that follows from wanting to reach labels the training data never showed me.

Where the labels actually live

Before measuring anything I split the vocabulary by how often a label appears in the training records and froze that split in a file. Every number in the project reads the bands from there and never recomputes them, so a row measured in the first week is comparable to one measured in the last.

The shape is the whole problem. Sixty-five headings carry 16% of the gold assignments in the test split. Nearly thirteen thousand headings appear between one and nine times each and carry 30.7%. For the neighbour harvest I started from, a single averaged recall number hides a factor of six between the top band and the tail: 0.69 against 0.11. My 2024 classifier hid something worse behind its own average — 0.4311 on the head and 0.0000 on the tail outright.

Three retrievers that are wrong about different things

The winning system on this task harvested subjects from a document's nearest neighbours, so I started there rather than treating it as a warm-up. Encode 8,000 training documents, encode the record, take the subjects of the twenty nearest documents. It is fast and it is good at exactly what you would expect: 0.6923 on the head band.

It also scores 0.0000 on labels that appear in no indexed document, at every cutoff and at every index size. That is not a tuning result, it is arithmetic — a heading you cannot find on a neighbour cannot be copied off one. On dev that band holds 18.9% of the distinct gold labels, and 8.5% of the assignments recall is actually counted over.

So I built two retrievers that never look at a document at all. The first embeds all 79,427 vocabulary entries as text and scores them directly against the record; the second is BM25 over the labels' own surface strings, 150,984 of them once synonyms are counted. Both are worse than the neighbour harvest on aggregate, and both reach the band it cannot.

Their band profiles are what made the rest of the project obvious. kNN falls monotonically as labels get rarer. The label tower rises monotonically — it is best exactly where kNN is worst. BM25 is flat across all four bands, because whether a heading's name turns up in a title has nothing to do with how often that heading was assigned.

Fusing all three, and what it costs

Three retrievers that fail in three directions is an argument for combining them rather than choosing between them. The obstacle is that their outputs are not in the same units: a cosine similarity, a BM25 term score and a count of neighbour votes cannot be averaged without inventing a conversion rate between them. Reciprocal rank fusion sidesteps that by throwing the scores away at the boundary and keeping only each retriever's ordering.

I tuned the three weights on dev with a 256-combination sweep, scored from a single retrieval pass so the sweep costs one run rather than 256. The constant k came out at the conventional 60, which I record as "the default was not beaten" rather than as a tuned parameter; the weights themselves are worth +0.0259 over equal weighting.

Every pair, and then all three

I ran every combination from one retrieval pass. Every pair beats both of its members, and all three beat every pair, so nothing in the set is dead weight — the third retriever is worth +0.0530 on top of the best pair and +0.1865 on top of the weakest one.

The honest version of the headline is narrower than it looks. Fusion's +0.0942 over the best single retriever is measured with weights tuned on the same dev split the row is reported on, and the single retrievers are untuned by construction because they have nothing to fuse. The untuned equal-weight figure, +0.0682, is the lower bound I actually trust.

It also costs something specific. The weight sweep selects on micro recall@10, an aggregate that is 59.5% head-and-torso, so it was never asked to protect unseen labels — and it did not. The label tower alone reaches 0.2623 there; fused, the band drops to 0.1871. The clearest illustration is the kNN + BM25 pair, which scores 0.0000 on a band where BM25 alone scores 0.1925: kNN is structurally empty there, and giving it weight 1.5 pushes every unseen candidate the other retrievers found out of the top ten. In the committed three-retriever configuration those labels are demoted rather than lost — at the hundred-candidate ceiling the band still holds 0.4288, against the label tower's own 0.4333 — and 0.1871 is the number the reranking stage was later built to beat.

Every label is German; most of the documents are not

Every entry in the GND vocabulary has a German name, and 58.7% of the gold assignments in this dataset belong to English documents. That is the largest single mismatch in the pipeline, and it is why BM25 over label names was so much better on German records than English ones — German ran 2.07× English at the hundred-candidate mark before anything was done about it.

So I machine-translated all 79,224 distinct German label strings once, with Helsinki-NLP/opus-mt-de-en, in 122 seconds, and rendered every label in both languages. 23.8% come back identical to their source — proper nouns, formulae, loanwords — and render in one language rather than two.

The verdict splits. BM25 gains +0.0223 micro recall@10 and its German-to-English ratio falls from 2.07× to 1.26×, the largest single language correction in the project. The dense label tower loses 0.0075: every German label vector moved, and German gave up nearly four times as much recall as English picked up. Fused, the net is positive, so bilingual label text stays on.

It also moves the baseline. Every figure from the encoder screen onward is measured against bilingual label text, so the like-for-like fused reference is 0.4149 rather than the German-only 0.3964 in the ablation table above. The two numbers are the same configuration reading different label text, and mixing them would credit the encoders with a gain the translation had already paid for.

Which encoder deserved the GPU

Everything above runs on one encoder, and swapping it is a config change that costs no training at all, so I screened four multilingual models before spending any GPU time. Every candidate had to predate 2025-01-31, the close of the shared task's evaluation window, and every one is pinned to a specific hub revision — both E5 checkpoints had commits landed on them in April 2026, and a config naming a model without a revision quietly resolves to a 2026 state of a 2023 model while every date in the claim stays true.

The screen runs on a quarter of the training split: 8,000 documents sampled stratified by record type and language, with every cell above 200 documents inside 0.002 of its share of the full corpus and every smaller cell given at least one document, because an index missing a language outright would be a different experiment rather than a smaller one. Everything else is held fixed, and the BM25 column is the control — it reads no vectors, so it is identical to four decimals across all four rows.

Then I grew the index to the whole 32,043-document split with nothing else changed, and the ranking did not survive. The top two swapped: one adjacent transposition of four, Spearman ρ 0.800. A cheap screen generalises for shortlist membership, not for order.

Two models had already been carried forward rather than one, for a reason that had nothing to do with the aggregate: bge-m3 had the strongest label tower of the four and took the unseen-label band even while losing the screen. The flip is what turned that hedge from caution into the right call. The pair sat 0.0022 apart at the screen and 0.0039 apart at the full index, against a table spread of 0.0411, so picking one on the screen alone would have been a coin flip dressed as a decision.

The other result in that table is that scale within a family did not carry. multilingual-e5-large loses to its own base model at both index sizes, and the ablation says why: it has the better document-to-document tower and the worse document-to-label one. The two towers are not one capability, and the retriever I needed most was the one it was worse at.

More documents, or more GPU

The last stage grows the index again — from the tib-core training split to the all-subjects split, 70,579 documents after dropping nine records that leak into dev and merging 45 duplicated ids — and contrastively fine-tunes both shortlisted encoders with LoRA. Running each encoder off the shelf and fine-tuned at the same index is what separates the corpus effect from the training effect instead of reporting their sum.

The answer was lopsided. Doubling the corpus buys under a hundredth of a point; the GPU buys roughly seven times that. Index size is spent, and any budget left belongs in training or in the stages after retrieval, not in more documents.

The training itself ran under a constraint worth admitting to: a vLLM process held 74.7 GB of the shared A100's 80 GB, so the run that finished used batch size 8 with two mined negatives instead of the defaults of 32 and 4, after three OOM crashes cost about an hour. In-batch negatives are most of the contrastive signal, so batch 8 is the one place this is knowingly under-powered and the +0.05 / +0.07 should be read as a floor.

gte-multilingual-base starts 0.0035 behind and finishes 0.0135 ahead, gaining half again as much from training on 2.9M trainable parameters against 7.1M, in 87 GPU-minutes against 197. The cheaper model was also the more trainable one, which is not a result I would have predicted in 2024.

The test split, opened once

The configuration was frozen and committed one commit before the gold split was read, and the split was read exactly once: fine-tuned gte-multilingual-base over the 70,579-document index, a hundred fused candidates reordered by an off-the-shelf cross-encoder and cut to fifty. 4,910 records and 11,798 gold assignments, two and a half hours on a laptop. The organisers' own scorer and my evaluator agree to 1.1e-16.

It lands above every published row on recall and in the middle of the field on precision. I want to be precise about what that does and does not license: this is a reconstruction scored after the fact against numbers published by teams who competed in January 2025, not an entry that competed against them. Every model it loads predates the evaluation window's close, so the comparison is at least not flattered by a year of model progress — but it had their published scores to aim at, and they did not have mine.

The number moves more with the average than with the method

One set of predictions, scored three ways: 0.5910 averaging over records, 0.7156 giving one vote to each record-type × language cell, and 0.6299 from the organisers' script. The gap between the first two is 0.1245. The gap between the best and worst published leaderboard row is 0.08.

The reason is cell sizes. Nine of twenty cells carry more than half of the macro figure between them, and seven of those nine hold nine records or fewer — a cell of two conference papers in Spanish weighs more than 1,465 German books. The organisers' script also cannot read eleven of the twenty cells at all: its reader seeds a dictionary with de and en and then subscripts it by directory name, so the 28 records in French, Spanish, Czech, Turkish, Dutch, Japanese and blank-language cells raise a KeyError inside their own code.

I report both aggregations everywhere and select models on the micro one, because a benchmark figure that moves further with the choice of average than with the choice of method is not a figure to tune against.

The long tail, and the bound underneath it

The tail is where the whole project happens. It holds 30.7% of the gold assignments, it is where the four encoders actually differ, and it is the band an averaged score hides best.

What buys the tail is candidates, not cleverness. Growing the index fourfold lifted tail recall by +0.12 to +0.14 for every encoder while moving the head of the top two by +0.008 and −0.012 — the head was already near its ceiling whatever the pipeline did. Fusion is the other lever: on the tail the fused row reaches 0.2262 against the label tower's 0.1628 and kNN's 0.1089.

Underneath the tail is a bound that no model reaches. Of the 992 headings in the test split that no training record carries, 812 are carried by no document in any corpus the shared task releases. They hold 851 gold assignments, 7.2% of everything the split asks for. That figure comes from three committed files and no run at all — it is a property of the benchmark, and every system that ranks by document similarity scores 0.000 on those assignments by construction. This one scores 0.323, because it also reads the labels' own text.

The least comfortable table in the project

Fine-tuning is monotonic in label frequency. It gives the most where recall was already highest and takes from the band with no training signal to learn from — which is the exact opposite of the reason I chose retrieval over classification in the first place.

On aggregate the trade is worth taking: +0.067 fused against −0.023 on a band carrying 8.9% of assignments. But it means the retrieval design's own selling point now rests on the reranker and the adjudicator rather than on the tower — and the adjudication rows never ran, because the environment had no API credential, so on the test run it rests on the reranker alone. The version of this project that cared most about unseen labels would have stopped before the GPU.

The corpus tells a related story. Growing to 70,579 documents reached 173 of the 1,053 unseen dev headings — and all 173 land in the tail band, appearing in one to nine documents each. Reaching a label is not retrieving it: the +0.02 the zero-shot band gained is what 173 barely-present labels are worth.

Measured and not kept

Roughly half the work in the repository is things that did not make the final configuration. They are in it because a negative result measured on the same split as everything else is worth more than an untested idea.

• A 66-way subject-area prior, one logistic regression per classification group over frozen document vectors, applied after fusion. Worth +0.0053 on a like-for-like baseline, +0.0158 on the head and −0.0018 on unseen labels. It is a head-band instrument and the head was not the problem, so it is off.
• The larger cross-encoder, bge-reranker-v2-m3 at 568M. On its own 100-record sample it landed 0.076 R@10 below the fused candidates it was given, and ran at 6 pairs a second against the smaller model's 42. A full dev pass with it would have been 25 hours on the laptop.
• Fine-tuning the cross-encoder. The untrained model in mixing mode already takes 16% of the headroom left in the candidate set for free, which was not enough of a case to spend a fourth GPU run.
• A graph refiner over the label hierarchy, which I had assumed existed. The release contains zero skos:broader triples, and the edges I could actually build covered 130 of 14,607 labels — 65 disconnected two-node components.
• LLM adjudication of the least-confident fifth of records. Built, validated against a closed candidate list so the model can never author a code, and never measured: the environment has no API credential. Dev bounds the best case at +0.046 recall@10, against the +0.034 the free cross-encoder already takes.
• Abstention. The submission format is fifty fixed slots with no way to decline a record, so the official metric cannot reward it. I measured it anyway: declining the least-confident half buys +0.0424 P@5 and leaves 46.6% of the gold assignments unanswered. The curve is smooth with no elbow, so there is no discovered operating point — only a review-capacity policy.

What I took from the rebuild

• Fuse mechanisms that are right about different inputs instead of ranking them and keeping the winner. The best single retriever here scores 0.3023 and the three together score 0.3964, and the ones that lose on aggregate are the only ones that reach the fifth of the split's gold labels no training record carries.
• Measure by subgroup before trusting an average. My 2024 classifier's headline was bad; re-run on a clean split, its band breakdown was disqualifying — and only one of those two numbers told me what to build next.
• A cheap screen is valid for membership, not for order. Screening on a quarter of the index picked the right pair of encoders and the wrong winner of the two.
• Spend the config change before the GPU. Swapping the encoder was worth +0.0575 at the 8,000-document screen and +0.0337 once the index was full; the LoRA run was worth +0.0668. One of those cost 87 GPU-minutes and the other cost a line in a YAML file.
• Make the discipline executable rather than intended. The test split opens once and the code refuses a second read without a justification; the evaluation plan is digested and committed before the split is touched; the frequency bands are read from a file and never recomputed. I wrote those rules after finding contamination in my own 2024 numbers, and a rule that only lives in a README would not have survived the week.

Stack