Danish stemming - some words are wrong

[KeLeKaPe]

I am encountering a lot of issues with the danish stemming.
Here is some examples of some wrong stem results:

• "bøger" -> "bøg": Incorrect, as "bøger" means "books," while "bøg" means "beech tree."
• "talen" -> "tal": Incorrect, as "talen" means "the speech," while "tal" means "number."
• "moden" -> "mod": Incorrect, as "moden" means "mature," while "mod" means "courage."
• "modner" -> "modn": Incorrect, as "modner" means "ripens," while "modn" is not a valid word.
• "slangen" -> "slang": Incorrect, as "slangen" means "the snake," while "slang" means "casual language."
• "slanger" -> "slang": Incorrect, as "slanger" means "snakes," while "slang" means "casual language."
• "vinder" -> "vind": Incorrect, as "vinder" means "winner" or "wins," while "vind" means "wind."
• "ruten" -> "rut": Incorrect, as "ruten" means "the route," while "rut" is not a valid word.
• "ruter" -> "rut": Incorrect, as "ruter" means "routes," while "rut" is not a valid word.
• "haven" -> "hav": Incorrect, as "haven" means "the garden," while "hav" means "ocean."
• "haver" -> "hav": Incorrect, as "haver" means "gardens," while "hav" means "ocean."

These are some of the examples I could find, there might be more.

[ojwb]

Thanks for raising this.

Note that we aren't aiming to reduce words to their linguistic root (if that's what you're after, snowball probably isn't the solution to your problem).

Snowball aims to improve recall in text search. The stems produced aren't necessarily words - conceptually the stemmer maps a word to a stem, where the only thing that matters about the stems are that words that are "the same" map tend to map to the same stem, and words which aren't don't. The latter part is actually more important as conflating unrelated words (like most of your cases above) is problematic as it results in false matches to searches whereas failing to conflate is at least no worse than not using a stemmer at all. In practice it's easiest to design an algorithm where the stems either are or look a lot like the linguistic root in most cases, but it's explicitly not a design goal.

"modner" -> "modn": Incorrect, as "modner" means "ripens," while "modn" is not a valid word.
"ruten" -> "rut": Incorrect, as "ruten" means "the route," while "rut" is not a valid word.
"ruten" -> "rut": Incorrect, as "ruten" means "the route," while "rut" is not a valid word.

So these are actually OK (I don't know Danish, but other forms of the first case seem to include modne and modnet which also both stem to modn, while other forms of the second case seem to include rute and ruterne which also both stem to rut).

I mention this not to split hairs, but because it helps understand what the problem here actually is.

Anyway, I wonder if the best way to solve this might be to change handling of -er, -en, and probably many of the other suffixes that start with e which we currently remove completely, to be instead replaced with -e. So e.g. bøger would then stem to bøge instead.

@KeLeKaPe Are you a Danish speaker? If so, could you suggest which of these suffixes that change should probably be applied to:

            'hed' 'ethed' 'ered' 'e' 'erede' 'ende' 'erende' 'ene' 'erne' 'ere'
            'en' 'heden' 'eren' 'er' 'heder' 'erer' 'heds' 'es' 'endes'
            'erendes' 'enes' 'ernes' 'eres' 'ens' 'hedens' 'erens' 'ers' 'ets'
            'erets' 'et' 'eret'

The longest matching suffix from this list gets removed (but only if it occurs in region R1, which starts after the first consonant which follows a vowel, or at least 3 characters from the start so you don't need to worry about the effect on very short words).

We're wanting to change this for all the cases like modn-er, modn-e, modn-et, rut-en, rut-er, rut-e, rut-erne, etc but ideally without affecting any suffixes which should be completely removed. If there's a suffix on the list where the correct answer depends on the word we may need to come up with a rule to decide which way to handle it, but probably best to just note if you see any such issues rather than worry about how to solve them at this point.

I can then amend the stemmer code and run a comparison of the output with and without the change (we have a script which analyses changes in how the stemmer groups words from a test vocabulary).

[ojwb]

I had more of a poke.

The natural stem of bøger looks to be bog, but any simple rule to change ø to o when we remove suffix -er seems like it's not going to work - e.g. søger is the present tense of the verb "to search" with infinitive søge (https://en.wiktionary.org/wiki/s%C3%B8ge#Danish). Currently it stems to søg which seems sensible. Stemming to sog seems not to create a collision, but the gerund is søgen and bøgen is the singular definite of bøg which should not be stemmed to bog.

There also seems to be an inherent ambiguity around bog in Danish, since it means both "book" and "beech nut" (https://en.wiktionary.org/wiki/bog#Etymology_2_2) (but the declension for the second meaning doesn't ever involve changing o to ø so the algorithm is exacerbating this ambiguity).

That also seems to rule out my initial thought to leave an e in place for some cases.

talen is a form of the noun and verb tale (https://en.wiktionary.org/wiki/tale#Danish). The imperative of the verb is actually tal so again there's an inherent ambiguity in Danish. It seems tale would be a natural stem; past tense forms talt, talte and taltes are challenging - currently these all stem to talt which is probably as good as we can do since these are also past tense forms of tælle "to count" (https://en.wiktionary.org/wiki/t%C3%A6lle#Danish).

Looking at moden and modner, the only form of mod seems to be modet so at least there's no obvious overlap of forms here. Ideally modner would map to the same stem as moden, but failing to conflate these is much less problematic than conflating words with unrelated meanings.

It seems slangen means both "snake" and "slang" (https://en.wiktionary.org/wiki/slangen#Danish) and also slang both means "slang" and the imperative form of "to snake" (https://en.wiktionary.org/wiki/slang#Danish), though I'd imagine that's not a commonly used imperative. So another inherent ambiguity in the Danish language here.

I didn't look into the rest, but I think to make useful progress here we're really going to need input from a native Danish speaker. @KeLeKaPe @tmikaeld

These are some of the examples I could find, there might be more.

It would be useful to have a fuller set of examples. If there aren't many more, or the list above is actually complete, then it would be feasible to add exceptions to handle these, though for the cases of inherent ambiguities in Danish we're limited in what we can do - we can either leave the stemming as it is, or assume one meaning, or not conflate the ambiguous forms with other forms.

If there are hundreds or thousands of cases we need a different approach, but if there are that many it seems likely they'll mostly divide into a small number of patterns.