HomePhorge

Stem fulltext tokens before filtering them for stopwords

Description

Stem fulltext tokens before filtering them for stopwords

Summary:
Fixes T12596. A query for a token (like "having") which stems to a stopword (like "have") currently survives filtering. Stem it first so it gets caught.

Also, for InnoDB, a custom stopword table can be configured. If it is, read that instead of the default stopword list (I configured it locally, but the default list is reasonable so we never formally recommended installs configure it).

Test Plan:
Queried for words that stem to stopwords, saw them filtered:

{F4915843}

Queried for the original problem query and saw "having" caught with "have" in the stopword list:

{F4915844}

Fiddled with local InnoDB stopword table config and saw the stopword list get loaded correctly.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12596

Differential Revision: https://secure.phabricator.com/D17728

Details

Provenance
epriestleyAuthored on Apr 19 2017, 8:56 AM
themackabuPushed on Mar 25 2025, 8:07 PM
Parents
rPdf7f56d8e3b7: Minor CSS tweaks Conpherence
Branches
Unknown
Tags
Unknown

Event Timeline