# whoosh-ng : Full Technical Documentation



## DOCUMENT: Analysis

# Analysis API

Classes and functions for turning text into indexable "tokens" (usually words).
Analysis is the first step in the indexing pipeline: an analyzer tokenizes text
and applies zero or more filters to the resulting token stream.

## Overview

Three general categories of objects make up the analysis pipeline:

- **Tokenizers** split text into individual tokens (words, n-grams, identifiers).
  Every tokenizer is callable: `tokenizer(text) -> iterator of Token objects`.
- **Filters** transform one token stream into another. Common operations include
  lowercasing, stop-word removal, stemming, and synonym expansion. Every filter
  is callable: `filter(token_generator) -> token_generator`.
- **Analyzers** compose a tokenizer and zero or more filters into a single unit.
  Every analyzer is callable and can be used directly as a field's `analyzer`
  argument.

Tokenizers and filters are combined using the `|` operator:

```python
my_analyzer = RegexTokenizer() | LowercaseFilter() | StopFilter()
```

The first item must be a tokenizer; subsequent items must be filters.

## Composition

### Composable

```python
class whoosh.analysis.Composable
```

Base class for tokenizers and filters, providing `|` composition.

**Attributes:**
- `is_morph (bool)`: Whether this object performs morphological transformation
  (e.g. stemming). Defaults to `False`.

**Methods:**

#### `__or__(self, other)`

Combines this object with `other` using `CompositeAnalyzer`.

```python
analyzer = RegexTokenizer() | LowercaseFilter() | StopFilter()
```

### CompositeAnalyzer

```python
class whoosh.analysis.CompositeAnalyzer
```

Composed analyzer created by chaining a tokenizer and filters with `|`.

**Example:**
```python
from whoosh.analysis import RegexTokenizer, LowercaseFilter, StopFilter

analyzer = RegexTokenizer() | LowercaseFilter() | StopFilter()
tokens = list(analyzer("Hello world, this is a test"))
```

## Token

```python
class whoosh.analysis.Token
```

Represents a single token (usually a word) extracted from source text.
Tokenizers yield the **same** `Token` object repeatedly (for performance), so
consumers must not hold references between iterations.

**Slots:**

| Attribute | Type | Description |
|-----------|------|-------------|
| `text` | `str` | The text of this token |
| `pos` | `int` | Token position (if `positions=True`) |
| `startchar` | `int` | Start character offset (if `chars=True`) |
| `endchar` | `int` | End character offset (if `chars=True`) |
| `original` | `str` | Original text before filters (if `keeporiginal=True`) |
| `positions` | `bool` | Whether position info was requested |
| `chars` | `bool` | Whether character offsets were requested |
| `stopped` | `bool` | Set by `StopFilter` |
| `boost` | `float` | Token boost factor (default `1.0`) |
| `removestops` | `bool` | Whether stop words should be removed |
| `mode` | `str` | `'index'` or `'query'` |
| `boosts` | `dict` | Per-position boost values (if requested) |
| `tokenize` | `bool` | Whether tokenization should proceed |
| `matched` | `bool` | Used during highlighting |
| `fieldname` | `str` | Field name for this token |

**Methods:**

#### `copy()`

Returns a new `Token` with the same attribute values. Use this if you need to
retain a token between iterations.

```python
def remove_duplicates(stream):
    last = None
    for t in stream:
        if last != t.text:
            yield t
        last = t.text
```

## Utility Functions

### entoken

```python
whoosh.analysis.entoken(
    textstream,
    positions=False,
    chars=False,
    start_pos=0,
    start_char=0,
    **kwargs
) -> Iterator[Token]
```

Converts a sequence of strings into a stream of `Token` objects.

### unstopped

```python
whoosh.analysis.unstopped(tokenstream) -> Iterator[Token]
```

Removes tokens where `token.stopped` is `True`.

## Analyzers

### Analyzer (Base)

```python
class whoosh.analysis.Analyzer
```

Abstract base class for all analyzers. Subclasses implement `__call__`.

### CompositeAnalyzer

Created automatically when you use `|` to compose tokenizers and filters.

### Predefined Analyzers

#### IDAnalyzer

```python
whoosh.analysis.IDAnalyzer(lowercase=False) -> Analyzer
```

Yields the entire input as a single token. Deprecated; use `IDTokenizer` directly.

- `lowercase (bool)`: If True, add a `LowercaseFilter`.

#### KeywordAnalyzer

```python
whoosh.analysis.KeywordAnalyzer(
    lowercase=False,
    commas=False
) -> Analyzer
```

Splits on whitespace or commas. Suitable for field values that are lists of
keywords.

- `lowercase (bool)`: Lowercase each token.
- `commas (bool)`: Split on commas instead of whitespace.

**Example:**
```python
from whoosh.analysis import KeywordAnalyzer

an = KeywordAnalyzer(lowercase=True, commas=True)
list(an("Hello, WORLD, test"))
# => ["hello", "world", "test"]
```

#### RegexAnalyzer

```python
whoosh.analysis.RegexAnalyzer(
    expression=r"\w+(\.?\w+)*",
    gaps=False
) -> Analyzer
```

Deprecated; use `RegexTokenizer` directly.

#### SimpleAnalyzer

```python
whoosh.analysis.SimpleAnalyzer(
    expression=default_pattern,
    gaps=False
) -> Analyzer
```

Composes `RegexTokenizer` with `LowercaseFilter`.

- `expression`: Regex pattern for tokens.
- `gaps`: If True, split on the expression instead of matching it.

**Example:**
```python
an = SimpleAnalyzer()
list(an("Hello there, this is a TEST"))
# => ["hello", "there", "this", "is", "a", "test"]
```

#### StandardAnalyzer

```python
whoosh.analysis.StandardAnalyzer(
    expression=default_pattern,
    stoplist=STOP_WORDS,
    minsize=2,
    maxsize=None,
    gaps=False
) -> Analyzer
```

Composes `RegexTokenizer`, `LowercaseFilter`, and optional `StopFilter`.

- `expression`: Regex pattern for tokens.
- `stoplist`: Words to remove (set to `None` to disable).
- `minsize`: Minimum token length (default `2`).
- `maxsize`: Maximum token length (default `None`, no limit).
- `gaps`: If True, split on the expression instead of matching it.

**Example:**
```python
an = StandardAnalyzer()
list(an("Testing is testing and testing"))
# => ["testing", "testing", "testing"]
```

#### StemmingAnalyzer

```python
whoosh.analysis.StemmingAnalyzer(
    expression=default_pattern,
    stoplist=STOP_WORDS,
    minsize=2,
    maxsize=None,
    gaps=False,
    stemfn=stem,
    ignore=None,
    cachesize=50000
) -> Analyzer
```

Composes `RegexTokenizer`, `LowercaseFilter`, optional `StopFilter`, and
`StemFilter`.

- `expression`: Regex pattern for tokens.
- `stoplist`: Words to remove (set to `None` to disable).
- `minsize`: Minimum token length (default `2`).
- `maxsize`: Maximum token length.
- `gaps`: If True, split on the expression instead of matching it.
- `stemfn`: Stemming function (default: Porter stemmer for English).
- `ignore`: Words to not stem (set).
- `cachesize`: Stem cache size (default `50000`). Use `-1` for unbounded,
  `None` for no cache.

**Example:**
```python
an = StemmingAnalyzer()
list(an("Testing is testing and testing"))
# => ["test", "test", "test"]
```

#### FancyAnalyzer

```python
whoosh.analysis.FancyAnalyzer(
    expression=r"\s+",
    stoplist=STOP_WORDS,
    minsize=2,
    gaps=True,
    splitwords=True,
    splitnums=True,
    mergewords=False,
    mergenums=False
) -> Analyzer
```

Composes `RegexTokenizer`, `IntraWordFilter`, `LowercaseFilter`, and `StopFilter`.
Splits on whitespace and breaks compound words into subwords.

**Example:**
```python
an = FancyAnalyzer()
list(an("Should I call getInt or get_real?"))
# => ["should", "call", "get", "int", "get", "real"]
```

#### LanguageAnalyzer

```python
whoosh.analysis.LanguageAnalyzer(
    lang,
    expression=default_pattern,
    gaps=False,
    cachesize=50000
) -> Analyzer
```

Configures a language-specific analyzer with `LowercaseFilter`, `StopFilter`,
and `StemFilter`.

- `lang`: Language code (e.g., `"en"`, `"es"`, `"fr"`).
- `expression`: Regex pattern for tokens.
- `gaps`: If True, split on the expression instead of matching it.
- `cachesize`: Stem cache size.

Available languages: `ar`, `da`, `nl`, `en`, `fi`, `fr`, `de`, `hu`, `it`,
`no`, `pt`, `ro`, `ru`, `es`, `sv`, `tr`.

See `whoosh.lang` for `has_stemmer()` and `has_stopwords()` helper functions.

## Tokenizers

All tokenizers inherit from `Tokenizer`.

### Tokenizer

```python
class whoosh.analysis.Tokenizer
```

Base class for tokenizers. Each tokenizer is callable and yields `Token`
objects.

### RegexTokenizer

```python
class whoosh.analysis.RegexTokenizer(
    expression=default_pattern,
    gaps=False
)
```

Uses a regular expression to extract tokens from text. Each match of the
expression equals one token; group 0 (the entire match) is used as the text.

- `expression`: Compiled regex or pattern string.
- `gaps`: If True, split on the expression rather than matching it.

**Example:**
```python
from whoosh.analysis import RegexTokenizer

rext = RegexTokenizer()
list(rext("hi there 3.141 big-time under_score"))
# => ["hi", "there", "3.141", "big", "time", "under_score"]
```

### IDTokenizer

```python
class whoosh.analysis.IDTokenizer
```

Yields the entire input string as a single token. Used for indexed but
untokenized fields (e.g., document paths).

### CharsetTokenizer

```python
class whoosh.analysis.CharsetTokenizer(charmap)
```

Tokenizes and translates text according to a character mapping dictionary.
Characters that map to `None` are treated as token break characters.

- `charmap`: Mapping from integer character codes to unicode characters
  (as used by `unicode.translate()`).

### PathTokenizer

```python
class whoosh.analysis.PathTokenizer(expression="[^/]+")
```

Tokenizes path strings into hierarchical prefixes. Given `"/a/b/c"`, yields
`["/a", "/a/b", "/a/b/c"]`.

### NgramTokenizer

```python
class whoosh.analysis.NgramTokenizer(minsize, maxsize=None)
```

Splits input text into N-grams instead of words. Unlike `RegexTokenizer`, this
tokenizer does not use a regex, so grams may include whitespace and punctuation.

- `minsize`: Minimum N-gram size.
- `maxsize`: Maximum N-gram size (defaults to `minsize`).

**Example:**
```python
from whoosh.analysis import NgramTokenizer

ngt = NgramTokenizer(4)
list(ngt("hi there"))
# => ["hi t", "i th", " the", "ther", "here"]
```

### CachedRegexTokenizer

```python
class whoosh.analysis.CachedRegexTokenizer(
    expression=default_pattern,
    gaps=False,
    maxsize=8192
)
```

A `RegexTokenizer` wrapper that caches tokenization results for repeated
strings, trading memory for speed.

- `expression`: Regex pattern.
- `gaps`: If True, split on the expression.
- `maxsize`: Maximum cache size (LRU eviction when exceeded).

### SpaceSeparatedTokenizer

```python
whoosh.analysis.SpaceSeparatedTokenizer() -> RegexTokenizer
```

Returns a `RegexTokenizer` that splits on whitespace.

### CommaSeparatedTokenizer

```python
whoosh.analysis.CommaSeparatedTokenizer() -> CompositeAnalyzer
```

Returns a composed analyzer that splits on commas and strips whitespace.

## Filters

All filters inherit from `Filter`.

### Filter

```python
class whoosh.analysis.Filter
```

Base class for filters. Subclasses implement `__call__(self, tokens)` which
takes a token generator and returns a token generator.

- `is_morph (bool)`: Set to `True` for morphological filters (e.g., stemming).
  This allows the filter to be bypassed during query analysis if desired.

### STOP_WORDS

```python
whoosh.analysis.STOP_WORDS
```

A frozenset of common English stop words: `"a"`, `"an"`, `"and"`, `"the"`, etc.
Used as the default stoplist for `StopFilter` and `StandardAnalyzer`.

### url_pattern

```python
whoosh.analysis.url_pattern
```

A compiled regex useful for URL filtering.

### LowercaseFilter

```python
class whoosh.analysis.LowercaseFilter
```

Lowercases token text using `unicode.lower()`.

**Example:**
```python
rext = RegexTokenizer() | LowercaseFilter()
list(rext("This is a TEST"))
# => ["this", "is", "a", "test"]
```

### StopFilter

```python
class whoosh.analysis.StopFilter(
    stoplist=STOP_WORDS,
    minsize=2,
    maxsize=None,
    renumber=True,
    lang=None
)
```

Marks and optionally removes stop words from the token stream.

- `stoplist`: Set of words to filter out (defaults to `STOP_WORDS`).
- `minsize`: Minimum token length; shorter tokens are removed (default `2`).
- `maxsize`: Maximum token length; longer tokens are removed (default `None`).
- `renumber`: Renumber positions to account for removed tokens (default `True`).
- `lang`: If set, loads stop words for the given language code.

**Example:**
```python
from whoosh.analysis import RegexTokenizer, StopFilter

stopper = RegexTokenizer() | StopFilter()
list(stopper("this is a test"))
# => ["test"]
```

### StripFilter

```python
class whoosh.analysis.StripFilter
```

Calls `unicode.strip()` on each token's text.

### CharsetFilter

```python
class whoosh.analysis.CharsetFilter(charmap)
```

Translates token text using `unicode.translate()` with the given character map.
Useful for case folding and accent folding.

- `charmap`: Dictionary mapping character ordinals to unicode characters.

**Example:**
```python
from whoosh.support.charset import accent_map

rext = RegexTokenizer() | CharsetFilter(accent_map)
list(rext("café"))
# => ["cafe"]
```

### DelimitedAttributeFilter

```python
class whoosh.analysis.DelimitedAttributeFilter(
    delimiter="^",
    attribute="boost",
    default=1.0,
    type=float
)
```

Looks for delimiter characters in token text and extracts data after the
delimiter into a named token attribute.

- `delimiter`: Separator character (default `"^"`).
- `attribute`: Attribute name on the token (default `"boost"`).
- `default`: Default value if no delimiter is found (default `1.0`).
- `type`: Type to cast the extracted value (default `float`).

**Example:**
```python
from whoosh.analysis import RegexTokenizer, DelimitedAttributeFilter

daf = DelimitedAttributeFilter()
an = RegexTokenizer(r"\S+") | daf
for t in an(u"image 3.14^2 render"):
    print(t.text, t.boost)
# image 1.0
# 3.14 2.0
# render 1.0
```

### SubstitutionFilter

```python
class whoosh.analysis.SubstitutionFilter(pattern, replacement)
```

Performs regex substitution on token text using `re.sub()`.

- `pattern`: Pattern string or compiled regex.
- `replacement`: Replacement text.

**Example:**
```python
from whoosh.analysis import RegexTokenizer, SubstitutionFilter

# Remove hyphens
ana = RegexTokenizer(r"\S+") | SubstitutionFilter("-", "")
```

### MultiFilter

```python
class whoosh.analysis.MultiFilter(**kwargs)
```

Selects between two or more sub-filters based on the `mode` attribute of the
token stream. Useful for using different filters during indexing vs. querying.

- Keyword arguments map mode names to filter instances.

**Example:**
```python
from whoosh.analysis import MultiFilter, IntraWordFilter

iwf_index = IntraWordFilter(mergewords=True, mergenums=True)
iwf_query = IntraWordFilter(mergewords=False, mergenums=False)
mf = MultiFilter(index=iwf_index, query=iwf_query)
```

### TeeFilter

```python
class whoosh.analysis.TeeFilter(*filters)
```

Interleaves the results of two or more filter chains. Requires at least two
filters. Note: this filter is slow because it creates token copies.

**Example:**
```python
# Lowercase in one branch, reverse in another
f1 = LowercaseFilter()
f2 = ReverseTextFilter()
ana = RegexTokenizer(r"\S+") | TeeFilter(f1, f2)
```

### ReverseTextFilter

```python
class whoosh.analysis.ReverseTextFilter
```

Reverses the text of each token.

**Example:**
```python
an = RegexTokenizer() | ReverseTextFilter()
list(an("hello there"))
# => ["olleh", "ereht"]
```

### PassFilter

```python
class whoosh.analysis.PassFilter
```

Identity filter; passes tokens through unchanged.

### LoggingFilter

```python
class whoosh.analysis.LoggingFilter(logger=None)
```

Prints debug log entries for every token that passes through.

- `logger`: Logger instance (defaults to `whoosh.analysis` logger).

## Intraword Filters

### IntraWordFilter

```python
class whoosh.analysis.IntraWordFilter(
    delims="-_'\"()!@#$%^&*[]{}<>\\|;:,./?`~+=",
    splitwords=True,
    splitnums=True,
    mergewords=False,
    mergenums=False
)
```

Splits words into subwords and performs optional merging. Based on
WordDelimiterFilter in Solr.

- `delims`: String of delimiter characters.
- `splitwords`: Split at case transitions (e.g., `PowerShot` → `Power`, `Shot`).
- `splitnums`: Split at letter-number transitions (e.g., `SD500` → `SD`, `500`).
- `mergewords`: Merge consecutive alphabetic subwords.
- `mergenums`: Merge consecutive numeric subwords.

### CompoundWordFilter

```python
class whoosh.analysis.CompoundWordFilter(wordset, keep_compound=True)
```

Breaks compound tokens into their constituent parts if they match words in the
given wordset. Useful for agglutinative languages and trademarks.

- `wordset`: A set (or any `__contains__` object) of known words.
- `keep_compound`: If True, keep the original compound token in the stream.

### BiWordFilter

```python
class whoosh.analysis.BiWordFilter(sep="-")
```

Merges adjacent tokens into bigram tokens. Useful for pseudo-phrase searching.

- `sep`: Separator string for bigrams.

### ShingleFilter

```python
class whoosh.analysis.ShingleFilter(size=2, sep="-")
```

Merges N adjacent tokens into multi-word tokens (shingles).

- `size`: Number of tokens to combine.
- `sep`: Separator string.

**Note:** For `size=2`, `BiWordFilter` is faster.

## Morphological Filters

### StemFilter

```python
class whoosh.analysis.StemFilter(
    stemfn=stem,
    lang=None,
    ignore=None,
    cachesize=50000
)
```

Stems tokens using the Porter stemming algorithm (or a language-specific
stemmer if `lang` is specified).

- `stemfn`: Stemming function (default: Porter stemmer).
- `lang`: Language code to override `stemfn` with a Snowball stemmer.
- `ignore`: Set of words to not stem (defaults to stemming all words).
- `cachesize`: Cache size for stemmed words. Use `-1` for unbounded,
  `None` for no cache.

**Example:**
```python
from whoosh.analysis import RegexTokenizer, StemFilter

stemmer = RegexTokenizer() | StemFilter()
list(stemmer("fundamentally willows"))
# => ["fundament", "willow"]
```

### PyStemmerFilter

```python
class whoosh.analysis.PyStemmerFilter(
    lang="english",
    ignore=None,
    cachesize=10000
)
```

Subclass of `StemFilter` that uses the third-party `py-stemmer` library.
Requires the py-stemmer package to be installed.

**Methods:**
- `algorithms()`: Returns available stemming algorithms from py-stemmer.

### DoubleMetaphoneFilter

```python
class whoosh.analysis.DoubleMetaphoneFilter(
    primary_boost=1.0,
    secondary_boost=0.5,
    combine=False
)
```

Encodes tokens using Lawrence Philips's Double Metaphone algorithm. Useful
for phonetic matching of names and places.

- `primary_boost`: Boost factor for the primary code token.
- `secondary_boost`: Boost factor for the secondary code token.
- `combine`: If True, keep the original token alongside the encoded tokens.

## N-gram Filters and Analyzers

### NgramFilter

```python
class whoosh.analysis.NgramFilter(minsize, maxsize=None, at=None)
```

Splits token text into N-grams of varying sizes.

- `minsize`: Minimum N-gram size.
- `maxsize`: Maximum N-gram size (defaults to `minsize`).
- `at`: `'start'` for prefix grams, `'end'` for suffix grams, or `None`
  for all position grams.

### NgramAnalyzer

```python
whoosh.analysis.NgramAnalyzer(minsize, maxsize=None) -> Analyzer
```

Composes `NgramTokenizer` with `LowercaseFilter`.

### NgramWordAnalyzer

```python
whoosh.analysis.NgramWordAnalyzer(
    minsize,
    maxsize=None,
    tokenizer=None,
    at=None
) -> Analyzer
```

Composes `RegexTokenizer`, `LowercaseFilter`, and `NgramFilter`. Use this
when you want sub-word n-grams (without whitespace) rather than raw
character n-grams.


## DOCUMENT: Automata

# Automata API

Module for constructing and manipulating finite state automata (FSAs),
including NFAs, DFAs, finite state transducers (FSTs), Levenshtein
automata, and regular expression automata. Used internally for spelling
correction, fuzzy term queries, and term dictionary operations.

The automata module is a refactored package with submodules. All classes
and functions are importable directly from `whoosh.automata`.

## Module Functions

### `parse_glob`

```python
whoosh.automata.parse_glob(pattern, _glob_multi="*", _glob_single="?", _glob_range1="[", _glob_range2="]") -> NFA
```

Parses a glob-style pattern string and returns an NFA that matches strings
matching the pattern.

**Parameters:**
- `pattern`: Glob pattern string (`*` matches any sequence, `?` matches any
  single character).
- `_glob_multi`, `_glob_single`: Override the wildcard characters.
- `_glob_range1`, `_glob_range2`: Override the range syntax brackets.

### `glob_automaton`

```python
whoosh.automata.glob_automaton(pattern) -> NFA
```

Convenience function that parses a glob pattern and returns an NFA.

## FSA (Finite State Automaton) Classes

### `FSA`

```python
class whoosh.automata.FSA(initial)
```

Base class for finite state automata.

**Constructor:**
- `initial`: The initial state.

**Attributes:**
- `initial`: Initial state.
- `transitions`: Dict mapping source states to dicts mapping labels to
  target states.
- `final_states`: Set of accepting (final) states.

**Methods:**
- `__eq__(other)`: Compares initial state, final states, and transitions.
- `all_states()`: Returns a set of all states reachable in the automaton.
- `all_labels()`: Returns a set of all transition labels.
- `get_labels(src)`: Yields all labels leaving state `src`.
- `generate_all(state=None, sofar="")`: Yields all strings accepted by the
  automaton.
- `move(state, label)`: Returns the state reached by following `label` from
  `state`, or `None`.
- `moves(state, labels)`: Yields `(label, next_state)` pairs.
- `next(state)`: Yields target states reachable from `state` via any label.
- `is_final(state)`: Returns `True` if `state` is a final state.
- `start()`: Returns the initial state.
- `has_path_to(target)`: Returns `True` if there is a path to `target`.

### `Marker`

```python
class whoosh.automata.Marker(name)
```

Marker object used as a special transition label in NFAs (e.g., `ANY`,
`EPSILON`).

### `EPSILON`

```python
whoosh.automata.EPSILON = Marker("EPSILON")
```

Special marker representing an epsilon transition (no input consumed).

### `ANY`

```python
whoosh.automata.ANY = Marker("ANY")
```

Special marker representing a transition that matches any input character.

### `NFA`

```python
class whoosh.automata.NFA(initial)
```

Nondeterministic Finite Automaton. Extends `FSA` with epsilon transitions
and NFA-specific construction methods.

**Methods:**
- `add_transition(src, label, dst)`: Adds a transition from `src` to `dst`
  consuming `label`.
- `add_final_state(state, final=True)`: Marks `state` as a final/accepting
  state.
- `epsilon_closure(state)`: Returns the set of states reachable from `state`
  via epsilon transitions.
- `to_dfa()`: Converts this NFA to an equivalent DFA and returns it.

### `DFA`

```python
class whoosh.automata.DFA(initial)
```

Deterministic Finite Automaton. Extends `FSA` with DFA-specific operations.

**Methods:**
- `next_valid_string(string)`: Finds the lexicographically smallest string
  accepted by the DFA that is greater than or equal to `string`.
- `to_dfa()`: Returns self (already a DFA).

### `renumber_dfa`

```python
whoosh.automata.renumber_dfa(dfa, base=0) -> DFA
```

Renumerates the states of a DFA to integers starting at `base`.

### `u_to_utf8`

```python
whoosh.automata.u_to_utf8(dfa, base=0) -> DFA
```

Converts a Unicode DFA to a UTF-8 DFA.

### `find_all_matches`

```python
whoosh.automata.find_all_matches(dfa, lookup_func, first=unull)
```

Yields all strings accepted by the DFA, using `lookup_func` to determine
which strings exist in the dictionary.

**Parameters:**
- `dfa`: A deterministic finite automaton.
- `lookup_func`: Function called with each candidate string; returns the
  string if found in the dictionary.
- `first`: First string to start matching from (default `chr(0)`).

### `reverse_nfa`

```python
whoosh.automata.reverse_nfa(n) -> NFA
```

Returns the reverse of an NFA (reversed transitions, swapped initial
and final states).

### `product`

```python
whoosh.automata.product(dfa1, op, dfa2) -> DFA
```

Computes the product of two DFAs using a binary operation.

**Parameters:**
- `dfa1`, `dfa2`: Input DFAs.
- `op`: A function `(set1, set2) -> set` computing the output final states
  from the two input final state sets.

### `intersection`

```python
whoosh.automata.intersection(dfa1, dfa2) -> DFA
```

Returns the intersection of two DFAs.

### `union`

```python
whoosh.automata.union(dfa1, dfa2) -> DFA
```

Returns the union of two DFAs.

### `epsilon_nfa`

```python
whoosh.automata.epsilon_nfa() -> NFA
```

Returns an NFA that accepts only the empty string.

### `dot_nfa`

```python
whoosh.automata.dot_nfa() -> NFA
```

Returns an NFA that accepts any single character.

### `basic_nfa`

```python
whoosh.automata.basic_nfa(label) -> NFA
```

Returns an NFA that accepts exactly the string `label`.

### `charset_nfa`

```python
whoosh.automata.charset_nfa(labels) -> NFA
```

Returns an NFA that accepts any single character in `labels`.

### `string_nfa`

```python
whoosh.automata.string_nfa(string) -> NFA
```

Returns an NFA that accepts exactly `string`.

### `choice_nfa`

```python
whoosh.automata.choice_nfa(n1, n2) -> NFA
```

Returns an NFA that accepts strings accepted by either `n1` or `n2`.

### `concat_nfa`

```python
whoosh.automata.concat_nfa(n1, n2) -> NFA
```

Returns an NFA that accepts the concatenation of `n1` and `n2`.

### `star_nfa`

```python
whoosh.automata.star_nfa(n) -> NFA
```

Returns an NFA that accepts zero or more repetitions of `n`.

### `plus_nfa`

```python
whoosh.automata.plus_nfa(n) -> NFA
```

Returns an NFA that accepts one or more repetitions of `n`.

### `optional_nfa`

```python
whoosh.automata.optional_nfa(n) -> NFA
```

Returns an NFA that accepts zero or one occurrence of `n`.

### `strings_dfa`

```python
whoosh.automata.strings_dfa(strings) -> DFA
```

Constructs a minimal DFA that accepts exactly the given strings.

### `add_suffix`

```python
whoosh.automata.add_suffix(dfa, nodes, last, downto, seen)
```

Internal function for adding suffixes to a trie during DFA construction.

## Levenshtein Automata

### `levenshtein_automaton`

```python
whoosh.automata.levenshtein_automaton(term, k, prefix=0) -> NFA
```

Constructs an NFA that matches all strings within edit distance `k` of
`term`. This is the core function for fuzzy term queries and spelling
suggestions.

**Parameters:**
- `term`: The reference string to compute edit distance from.
- `k`: Maximum edit distance (number of insertions, deletions, or
  substitutions).
- `prefix`: If positive, require matched strings to share this length of
  prefix with `term` (speeds up matching significantly).

**Returns:** An NFA that can be converted to a DFA via `.to_dfa()`.

```python
from whoosh.automata import levenshtein_automaton

nfa = levenshtein_automaton("hello", k=1, prefix=0)
dfa = nfa.to_dfa()
```

## RegEx

### `parse`

```python
whoosh.automata.parse(pattern) -> NFA
```

Parses a regular expression pattern string and returns an NFA.

**Parameters:**
- `pattern`: A regex pattern string (Python `re`-style syntax).

### `RegexBuilder`

```python
class whoosh.automata.RegexBuilder(pattern)
```

Helper class for building NFAs from regex patterns.

## FST (Finite State Transducer) Classes

### `Values`

```python
class whoosh.automata.Values
```

Abstract base class for value types stored in FST arcs.

### `IntValues`

```python
class whoosh.automata.IntValues
```

Stores integer values in FST arcs.

### `SequenceValues`

```python
class whoosh.automata.SequenceValues
```

Base class for value types that store sequences of values.

### `BytesValues`

```python
class whoosh.automata.BytesValues
```

Stores byte string values in FST arcs.

### `ArrayValues`

```python
class whoosh.automata.ArrayValues
```

Stores arrays of values in FST arcs.

### `IntListValues`

```python
class whoosh.automata.IntListValues
```

Stores lists of integers in FST arcs.

### `Node`

```python
class whoosh.automata.Node
```

Base class for nodes in an FST.

### `ComboNode`

```python
class whoosh.automata.ComboNode
```

Base class for nodes that combine multiple sub-nodes (intersection, union).

### `UnionNode`

```python
class whoosh.automata.UnionNode
```

A node that represents the union of multiple sub-nodes.

### `IntersectionNode`

```python
class whoosh.automata.IntersectionNode
```

A node that represents the intersection of multiple sub-nodes.

### `BaseCursor`

```python
class whoosh.automata.BaseCursor
```

Base class for cursors that iterate over FST contents.

### `Cursor`

```python
class whoosh.automata.Cursor
```

Concrete cursor for iterating over an FST, supporting `next()`, `find()`,
`text()`, and other navigation methods.

### `UncompiledNode`

```python
class whoosh.automata.UncompiledNode
```

Represents an FST node that has not yet been compiled into a binary
representation. Used during FST construction.

### `Arc`

```python
class whoosh.automata.Arc
```

Represents a single arc in an FST, with a label, target node, and associated
value.

### `GraphWriter`

```python
class whoosh.automata.GraphWriter
```

Writes an FST to a binary file on disk or to an in-memory buffer.

### `BaseGraphReader`

```python
class whoosh.automata.BaseGraphReader
```

Base class for reading FSTs from disk.

### `GraphReader`

```python
class whoosh.automata.GraphReader
```

Concrete reader for FSTs stored on disk. Supports `find()`, `next()`, and
`text()` for navigating the graph.

### `to_labels`

```python
whoosh.automata.to_labels(key)
```

Converts a key (string, int, etc.) into a list of FST arc labels.

### `within`

```python
whoosh.automata.within(graph, text, k=1, prefix=0, address=None)
```

Uses a pre-built FST and a Levenshtein automaton to find all keys in the
graph within edit distance `k` of `text`.

**Parameters:**
- `graph`: A `GraphReader` instance.
- `text`: The search term.
- `k`: Maximum edit distance.
- `prefix`: Required shared prefix length.
- `address`: Optional starting address in the graph.

### `dump_graph`

```python
whoosh.automata.dump_graph(graph, address=None, tab=0, out=None)
```

Debug utility that prints the structure of an FST to stdout or a file.

### `FileVersionError`

```python
class whoosh.automata.FileVersionError
```

Raised when reading an FST file with an incompatible version.

### `InactiveCursor`

```python
class whoosh.automata.InactiveCursor
```

Raised when operating on a cursor that is not at a valid position.


## DOCUMENT: Backends

# Backends API

Storage backend abstractions.

## Backend (ABC)

```python
class whoosh.backends.abc.Backend
```

Abstract base class for storage backends.

### Methods

#### `create()`

Create a new segment.

#### `open()`

Open an existing segment.

#### `close()`

Close the backend.

#### `commit()`

Commit changes.

#### `startup()`

Called on backend startup.

#### `shutdown()`

Called on backend shutdown.

---

## FileBackend

```python
class whoosh.backends.file.FileBackend
```

Default backend storing index as files.

```python
from whoosh.backends.file import FileBackend
from whoosh.store.filestore import FileStorage

storage = FileStorage("indexdir")
backend = FileBackend(storage=storage)
```

---

## SQLiteBackend

```python
class whoosh.backends.sqlite.SQLiteBackend
```

Stores index in SQLite database.

```python
from whoosh.backends.sqlite import SQLiteBackend
from whoosh.store.sqlite import SQLiteStorage

storage = SQLiteStorage("index.db")
backend = SQLiteBackend(storage=storage)
```

---

## MemoryBackend

```python
class whoosh.backends.memory.MemoryBackend
```

In-memory backend for testing.

```python
from whoosh.backends.memory import MemoryBackend

backend = MemoryBackend()
```

---

## BackendRegistry

```python
class whoosh.registry.BackendRegistry
```

Register backends:

```python
BackendRegistry.register("my_backend", MyBackendClass, "my_package")
backend = BackendRegistry.get("my_backend")
```


## DOCUMENT: Classify

# Classify API

Classes and functions for classifying and extracting information from
documents. This module provides query expansion models, similarity
functions (shingling, simhash), and clustering algorithms.

## Expansion Models

### `ExpansionModel`

```python
class whoosh.classify.ExpansionModel(doc_count, field_length)
```

Abstract base class for query expansion models. Subclass to implement custom
expansion scoring.

**Constructor:**
- `doc_count`: Total number of documents in the collection.
- `field_length`: Total length of the field across all documents.

**Computed Attributes:**
- `N`: Document count.
- `collection_total`: Total field length.
- `mean_length`: Average field length (`collection_total / N`).

**Methods:**
- `normalizer(maxweight, top_total)`: Returns a normalization factor.
- `score(weight_in_top, weight_in_collection, top_total)`: Returns the
  expansion score for a term.

### `Bo1Model`

```python
class whoosh.classify.Bo1Model(doc_count, field_length)
```

Bayesian One-Poisson expansion model. One of the standard query expansion
models.

### `Bo2Model`

```python
class whoosh.classify.Bo2Model(doc_count, field_length)
```

Bayesian Two-Poisson expansion model. Another standard query expansion model.

### `KLModel`

```python
class whoosh.classify.KLModel(doc_count, field_length)
```

Kullback-Leibler divergence-based expansion model.

## Expander

### `Expander`

```python
class whoosh.classify.Expander(
    ixreader,
    fieldname,
    model=Bo1Model
)
```

Uses an `ExpansionModel` to expand the set of query terms based on the top N
result documents.

**Constructor:**
- `ixreader`: An `IndexReader` object.
- `fieldname`: The name of the field to expand terms from.
- `model`: An `ExpansionModel` class or instance. Defaults to `Bo1Model`.

**Methods:**

#### `add(vector)`

Adds forward-index information about one of the "top N" documents.

- `vector`: A series of `(text, weight)` tuples, such as is returned by
  `Reader.vector_as("weight", docnum, fieldname)`.

#### `add_document(docnum)`

Adds a document's term vector to the expander. If the field has a term vector,
uses it; otherwise falls back to stored field text.

#### `add_text(string)`

Adds a text string by indexing it with the field's analyzer.

#### `expanded_terms(number, normalize=True)`

Returns the N most important terms in the vectors added so far, ranked by
the expansion model's score.

- `number`: Number of terms to return.
- `normalize`: Whether to normalize weights.
- Returns: List of `(term, weight)` tuples, sorted by weight descending.

```python
from whoosh.classify import Expander, Bo1Model

expander = Expander(ix.reader(), "content")
for docnum in results.ids()[:10]:
    expander.add_document(docnum)

for word, weight in expander.expanded_terms(5):
    print(word, weight)
```

## Similarity Functions

### `shingles`

```python
whoosh.classify.shingles(input, size=2) -> iterable
```

Generates `(shingle, frequency)` pairs from a string by sliding a window of
the given size over the input.

**Parameters:**
- `input`: The input string.
- `size`: The shingle size (default `2`).

```python
from whoosh.classify import shingles

for shingle, freq in shingles("hello world", size=2):
    print(shingle, freq)
```

### `simhash`

```python
whoosh.classify.simhash(features, hashbits=32) -> int
```

Computes a simhash (perceptual hash) from a sequence of weighted features.
Simhashes that are similar produce similar hash values, allowing fast
near-duplicate detection via Hamming distance.

**Parameters:**
- `features`: Iterable of `(feature, weight)` tuples.
- `hashbits`: Number of bits in the hash (default `32`).
- Returns: An integer hash value.

```python
from whoosh.classify import shingles, simhash

h1 = simhash(shingles(text1))
h2 = simhash(shingles(text2))
from whoosh.classify import hamming_distance
dist = hamming_distance(h1, h2)
```

### `hamming_distance`

```python
whoosh.classify.hamming_distance(first_hash, other_hash, hashbits=32) -> int
```

Computes the Hamming distance between two hash values. A small distance
indicates high similarity.

**Parameters:**
- `first_hash`: First hash integer.
- `other_hash`: Second hash integer.
- `hashbits`: Number of bits in the hashes (default `32`).

## Clustering

### `kmeans`

```python
whoosh.classify.kmeans(
    data,
    k,
    t=0.0001,
    distfun=None,
    maxiter=50,
    centers=None
) -> (labels, centroids)
```

One-dimensional K-means clustering. Assigns each data point to the nearest
of `k` centroids and returns cluster labels and final centroids.

**Parameters:**
- `data`: List of data points (numeric values).
- `k`: Number of clusters.
- `t`: Tolerance; stops if centroid changes are below this value.
- `distfun`: Optional distance function (unused if `None`).
- `maxiter`: Maximum iterations (default `50`).
- `centers`: Optional list of initial centroids. If `None`, selects `k`
  random points from `data`.

**Returns:** A tuple `(labels, centroids)` where `labels` is a list of
cluster assignments per data point and `centroids` is the list of final
centroid positions.

### `two_pass_variance`

```python
whoosh.classify.two_pass_variance(data) -> float
```

Computes the sample variance of a data list using the two-pass algorithm
(first pass computes the mean, second pass accumulates squared deviations).

### `weighted_incremental_variance`

```python
whoosh.classify.weighted_incremental_variance(data_weight_pairs) -> float
```

Computes the weighted variance incrementally from a sequence of
`(value, weight)` pairs.

### `swin`

```python
whoosh.classify.swin(data, size) -> list
```

Sliding window clustering that groups data points where the range (max - min)
within a window of `size` is below a threshold. Uses variance for ranking.

**Parameters:**
- `data`: Sorted list of data points.
- `size`: Maximum window range (max - min) for clustering.

**Returns:** A list of `(left, right, count, variance)` tuples representing
clusters, sorted by count descending then by variance ascending.


## DOCUMENT: Codecs

# Codecs API

Classes and interfaces for how Whoosh writes and reads the inverted index,
postings, and per-document values. The codecs module is a refactored package
exposing the same public API as the former monolithic module.

## Module Functions

### `default_codec`

```python
whoosh.codec.default_codec(*args, **kwargs) -> Codec
```

Returns the default codec used by the index. Currently returns a
`W3Codec` instance.

```python
from whoosh.codec import default_codec
codec = default_codec()
```

## Exceptions

### `OutOfOrderError`

```python
whoosh.codec.OutOfOrderError
```

Raised when documents are added to a field out of order. Fields must
receive documents in ascending docnum order.

## Base Classes

### `Codec`

```python
class whoosh.codec.Codec
```

Abstract base class for index codecs. Subclasses implement methods for
writing and reading the index format.

**Class Attributes:**
- `length_stats (bool)`: If `True`, the codec stores per-document field
  length statistics. Default `True`.

**Methods:**

#### `per_document_writer(storage, segment)`

Abstract. Returns a `PerDocumentWriter` for writing per-document values
(columns, term vectors) to the given segment.

#### `field_writer(storage, segment)`

Abstract. Returns a `FieldWriter` for writing postings to the given segment.

#### `postings_writer(dbfile, byteids=False)`

Abstract. Returns a `PostingsWriter` for writing posting lists to `dbfile`.

#### `postings_reader(dbfile, terminfo, format_, term=None, scorer=None)`

Abstract. Returns a `Matcher` for reading postings from `dbfile`.

#### `automata(storage, segment)`

Returns an `Automata` instance for spelling correction using automata-based
edit distance. Default returns a base `Automata()` object.

#### `terms_reader(storage, segment)`

Abstract. Returns a `TermsReader` for reading the term dictionary and
postings of the given segment.

#### `per_document_reader(storage, segment)`

Abstract. Returns a `PerDocumentReader` for reading per-document values
from the given segment.

#### `new_segment(storage, indexname)`

Abstract. Creates and returns a new `Segment` object for the given storage
and index name.

### `WrappingCodec`

```python
class whoosh.codec.WrappingCodec(child)
```

A `Codec` that delegates all operations to a child codec. Useful for
creating codec wrappers that modify or intercept specific operations.

**Constructor:**
- `child`: The underlying `Codec` instance to wrap.

All methods delegate to the child codec:
`per_document_writer()`, `field_writer()`, `postings_writer()`,
`postings_reader()`, `automata()`, `terms_reader()`, `per_document_reader()`,
`new_segment()`.

## Writer Classes

### `PerDocumentWriter`

```python
class whoosh.codec.PerDocumentWriter
```

Abstract base class for writing per-document values (columns, term vectors).

**Methods:**

#### `start_doc(docnum)`

Abstract. Called when starting to write a new document.

#### `add_field(fieldname, fieldobj, value, length)`

Abstract. Adds a field value to the current document.

#### `add_column_value(fieldname, columnobj, value)`

Abstract. Adds a column value. Raises `NotImplementedError` if the codec
doesn't support columns.

#### `add_vector_items(fieldname, fieldobj, items)`

Abstract. Adds term vector items.

#### `add_vector_matcher(fieldname, fieldobj, vmatcher)`

Convenience method that reads items from a `Matcher` and calls
`add_vector_items()`.

#### `finish_doc()`

Called when finishing a document. Default does nothing.

#### `close()`

Called when done writing. Default does nothing.

### `FieldWriter`

```python
class whoosh.codec.FieldWriter
```

Abstract base class for writing postings (inverted index) data.

**Methods:**

#### `add_postings(schema, lengths, items)`

Translates a generator of `(fieldname, btext, docnum, weight, vbytes)`
tuples into calls to `start_field()`, `start_term()`, `add()`,
`finish_term()`, and `finish_field()`.

**Parameters:**
- `schema`: The `Schema` object.
- `lengths`: Optional `FieldLengthTable` for document field lengths.
- `items`: Iterable of posting tuples.

#### `start_field(fieldname, fieldobj)`

Abstract. Called when starting a new field.

#### `start_term(text)`

Abstract. Called when starting a new term within a field.

#### `add(docnum, weight, vbytes, length=None)`

Abstract. Adds a posting to the current term.

#### `add_spell_word(fieldname, text)`

Called to add a word to the spelling index. Default does nothing.

#### `finish_term()`

Abstract. Called when finishing a term.

#### `finish_field()`

Called when finishing a field. Default does nothing.

#### `close()`

Called when done writing. Default does nothing.

### `PostingsWriter`

```python
class whoosh.codec.PostingsWriter
```

Abstract base class for writing posting lists (the inverted index).

**Methods:**

#### `start_postings(format_, terminfo)`

Abstract. Starts writing postings for a new term.

#### `add_posting(id_, weight, vbytes, length=None)`

Abstract. Adds a posting to the current term.

#### `finish_postings(allow_compact=True)`

Called when finished writing postings. Default does nothing.

#### `written()`

Abstract. Returns `True` if this writer has already written to disk.

## Reader Classes

### `FieldCursor`

```python
class whoosh.codec.FieldCursor
```

Abstract base class for iterating over terms in a field.

**Methods:**
- `first()`: Move to the first term.
- `find(string)`: Find a term matching or closest to `string`.
- `next()`: Move to the next term.
- `term()`: Returns the current term's text.

### `EmptyCursor`

```python
class whoosh.codec.EmptyCursor
```

A `FieldCursor` representing an empty field. All methods return `None` or
`False`.

### `TermsReader`

```python
class whoosh.codec.TermsReader
```

Abstract base class for reading the term dictionary and postings of a
segment.

**Methods:**
- `__contains__(term)`: Returns `True` if the term exists.
- `cursor(fieldname, fieldobj)`: Returns a `FieldCursor`.
- `terms()`: Yields `(fieldname, text)` tuples for all terms.
- `terms_from(fieldname, prefix)`: Yields terms from `fieldname` starting
  with `prefix`.
- `items()`: Yields `((fieldname, text), TermInfo)` tuples.
- `items_from(fieldname, prefix)`: Like `items()` but filtered by prefix.
- `term_info(fieldname, text)`: Returns a `TermInfo` for the term.
- `frequency(fieldname, text)`: Returns the total frequency.
- `doc_frequency(fieldname, text)`: Returns the document frequency.
- `matcher(fieldname, text, format_, scorer=None)`: Returns a `Matcher`.
- `indexed_field_names()`: Yields names of indexed fields.
- `close()`: Close the reader.

### `PerDocumentReader`

```python
class whoosh.codec.PerDocumentReader
```

Abstract base class for reading per-document values (columns, term vectors,
stored fields).

**Methods:**
- `close()`: Close the reader.
- `doc_count()`: Returns number of non-deleted documents.
- `doc_count_all()`: Returns total document count (including deleted).
- `has_deletions()`: Returns `True` if any documents are deleted.
- `is_deleted(docnum)`: Returns `True` if docnum is deleted.
- `deleted_docs()`: Yields docnums of deleted documents.
- `all_doc_ids()`: Yields docnums of all non-deleted documents.
- `supports_columns()`: Returns `True` if column storage is supported.
- `has_column(fieldname)`: Returns `True` if field has a column.
- `list_columns()`: Yields names of available columns.
- `column_reader(fieldname, column)`: Returns a column reader.
- `doc_field_length(docnum, fieldname)`: Returns field length for docnum.
- `field_length(fieldname)`: Returns total field length.
- `min_field_length(fieldname)`: Returns minimum field length.
- `max_field_length(fieldname)`: Returns maximum field length.
- `has_vector(docnum, fieldname)`: Returns `True` if docnum has a vector.
- `vector(docnum, fieldname, format_)`: Returns a `Matcher` for the vector.
- `stored_fields(docnum)`: Returns dict of stored field values.
- `all_stored_field()`: Yields stored fields for all documents.

### `MultiPerDocumentReader`

```python
class whoosh.codec.MultiPerDocumentReader(readers, offset=0)
```

Combines multiple `PerDocumentReader` instances into one for multi-segment
indices.

**Constructor:**
- `readers`: List of `PerDocumentReader` instances.
- `offset`: Base document offset (usually `0`).

## Automata

### `Automata`

```python
class whoosh.codec.Automata
```

Provides static methods for automata-based term matching, used by the
spelling corrector.

**Static Methods:**

#### `levenshtein_dfa(uterm, maxdist, prefix=0)`

Returns a deterministic finite automaton (DFA) that matches all edit-distance
variants of `uterm` within `maxdist` edits, optionally requiring a minimum
shared prefix of length `prefix`.

#### `find_matches(dfa, cur)`

Given a DFA and a `FieldCursor`, yields all matching terms.

**Methods:**

#### `terms_within(fieldcur, uterm, maxdist, prefix=0)`

Returns an iterator of matching terms within the given edit distance of
`uterm`.

## Segment

### `Segment`

```python
class whoosh.codec.Segment
```

Represents a segment of the index. Instances are pickled into the TOC file
to describe on-disk files.

**Class Attributes:**
- `COMPOUND_EXT = ".seg"`: Extension for compound segment files.

**Instance Attributes:**
- `indexname`: Base name of the segment.
- `segid`: Random unique ID string.
- `compound (bool)`: Whether this segment uses compound file format.

**Methods:**
- `make_filename(ext)`: Returns `f"{segment_id()}{ext}"`.
- `list_files(storage)`: Lists all files belonging to this segment.
- `create_file(storage, ext, **kwargs)`: Creates a new file for this segment.
- `open_file(storage, ext, **kwargs)`: Opens a file for this segment.
- `create_compound_file(storage)`: Combines all segment files into a
  compound `.seg` file.
- `open_compound_file(storage)`: Opens the compound segment file.
- `doc_count_all()`: Abstract. Returns total document count.
- `doc_count()`: Returns non-deleted document count.
- `set_doc_count(doccount)`: Sets the document count.
- `has_deletions()`: Returns `True` if any documents are deleted.
- `deleted_count()`: Abstract. Returns number of deleted documents.
- `deleted_docs()`: Abstract. Yields docnums of deleted documents.
- `delete_document(docnum, delete=True)`: Abstract. Deletes/undeletes a
  document.
- `is_deleted(docnum)`: Abstract. Returns `True` if docnum is deleted.
- `should_assemble()`: Returns `True` by default. Override to control
  compound file behavior.
- `validate(storage)`: Checks on-disk integrity of this segment.
- `segment_id()`: Returns the unique segment identifier string.
- `is_compound()`: Returns `True` if this segment uses compound file format.

### `WrappingSegment`

```python
class whoosh.codec.WrappingSegment(child)
```

A `Segment` that delegates all operations to a child segment.

**Constructor:**
- `child`: The underlying `Segment` instance to wrap.

## W3 Codec (Default)

The `W3` codec ("Whoosh 3") is the default index format, storing postings in
compressed blocks for efficient reading and skipping.

### `W3Codec`

```python
class whoosh.codec.whoosh3.W3Codec(blocklimit=128, compression=3, inlinelimit=1)
```

The default codec. Uses compressed blocks and term inlining for efficient
storage and fast lookups.

**Constructor:**
- `blocklimit`: Number of postings per block (default `128`).
- `compression`: zlib compression level (default `3`, `0` = no compression).
- `inlinelimit`: Maximum number of postings to inline directly in the term
  info (default `1`).

**File Extensions:**
- `.trm`: Term dictionary
- `.pst`: Postings
- `.vps`: Vector postings
- `.col`: Per-document value columns

### `W3PerDocWriter`

Writer for per-document values using the W3 format. Handles columns,
stored fields, term vectors, and field lengths.

### `W3FieldWriter`

Writer for the inverted term index using the W3 format. Uses a
`OrderedHashWriter` for the term dictionary and posts to a postings file.

### `W3LeafMatcher`

```python
class whoosh.codec.whoosh3.W3LeafMatcher(postfile, startoffset, length, format_, term=None, byteids=None, scorer=None)
```

Reads on-disk postings from the postings file and presents the
`Matcher` interface. Supports block-level skipping and lazy block loading.

**Optimization methods:**
- `block_min_id()`: Returns the first doc ID in the current block.
- `block_max_id()`: Returns the last doc ID in the current block.
- `block_min_length()`: Returns the minimum field length in the current block.
- `block_max_length()`: Returns the maximum field length in the current block.
- `block_max_weight()`: Returns the maximum weight in the current block.
- `skip_to_quality(minquality)`: Skips blocks exceeding a quality threshold.

### `W3TermsReader`

Reader for the term dictionary using the W3 format. Uses an
`OrderedHashReader` for fast lookups.

### `W3TermInfo`

```python
class whoosh.codec.whoosh3.W3TermInfo
```

Stores term statistics and posting location information. Supports inlining
small posting sets directly in the term dictionary for fast lookups.

**Flags:**
- `_FLAG_OFFSET` (0): Postings stored at an offset in the postings file.
- `_FLAG_INLINE_PICKLE` (1): Postings inlined as a pickled tuple.
- `_FLAG_INLINE_COMPACT` (2): Single posting compactly inlined.
- `_FLAG_INLINE_COMPACT_SHORT` (3): Multiple postings compactly inlined.

**Methods:**
- `add_block(block)`: Merges block statistics into this term info.
- `set_extent(offset, length)`: Sets offset and length of postings in file.
- `extent()`: Returns `(offset, length)`.
- `set_inlined(ids, weights, values)`: Sets inlined posting data.
- `set_compact_inline(id_, weight, value)`: Sets single inlined posting.
- `set_compact_short_inline(ids, weights, values)`: Sets multiple compact
  inlined postings.
- `is_inlined()`: Returns `True` if postings are inlined.
- `inlined_postings()`: Returns `(ids, weights, values)` tuples for inlined
  postings.
- `to_bytes()` / `from_bytes()`: Serialize/deserialize.

### `W3Segment`

```python
class whoosh.codec.whoosh3.W3Segment(codec, indexname, doccount=0, segid=None, deleted=None)
```

Segment class for the W3 codec. Stores a reference to the codec, document
count, and deleted document set.

## Plain Text Codec (Debugging)

### `PlainTextCodec`

```python
class whoosh.codec.plaintext.PlainTextCodec
```

A codec that stores the index as human-readable plain text. Intended for
debugging and manual inspection, not for production use.

**Class Attributes:**
- `length_stats = False`

**File extensions:**
- `.dcs`: Document (stored fields, columns, vectors)
- `.trm`: Term dictionary (plain text)

### `PlainPerDocWriter`

Plain text writer for per-document values.

### `PlainPerDocReader`

Plain text reader for per-document values.

### `PlainFieldWriter`

Plain text writer for the inverted index.

### `PlainTermsReader`

Plain text reader for the term dictionary.

### `PlainSegment`

```python
class whoosh.codec.plaintext.PlainSegment(indexname)
```

Segment class for the plain text codec. Does not support compound files
(`should_assume()` returns `False`).

## Memory Codec

### `MemoryCodec`

```python
class whoosh.codec.memory.MemoryCodec
```

An in-memory-only codec for testing. Stores all data in Python objects
rather than on disk.

**Class Attributes:**
- `storage`: A `RamStorage` instance.
- `segment`: A `MemSegment` instance.

**Methods:**
- `writer(schema)`: Returns a `MemWriter`.
- `reader(schema)`: Returns a `SegmentReader`.

### `MemWriter`

```python
class whoosh.codec.memory.MemWriter
```

A `SegmentWriter` subclass that commits immediately without merging.

### `MemPerDocWriter`

In-memory writer for per-document values.

### `MemPerDocReader`

In-memory reader for per-document values.

### `MemFieldWriter`

In-memory writer for the inverted index.

### `MemTermsReader`

In-memory reader for the term dictionary.

### `MemSegment`

```python
class whoosh.codec.memory.MemSegment(codec, indexname)
```

In-memory segment storing all data in Python dictionaries (inverted index,
stored fields, lengths, vectors, term infos). Uses a `Lock` for thread-safe
access.


## DOCUMENT: Collectors

# Collectors API

Classes and functions for gathering search results. Collectors are used
internally by `Searcher.search()` to collect matching documents and build
`Results` objects. The collectors module is a refactored package exposing the
same public API as the former monolithic module.

## Overview

A `Collector` iterates over matching documents in an index, collects
information about them, and produces a `Results` object. The base `Collector`
class defines the interface; specialized subclasses implement different
collection strategies (top-N, unlimited, sorting, filtering, faceting, etc.).

## Core Classes

### `Collector`

```python
class whoosh.collectors.Collector
```

Abstract base class for all collectors. Subclasses must implement `collect()`
and `results()`.

**Methods:**

#### `prepare(top_searcher, q, context)`

Called before a search begins. Sets up `self.top_searcher`, `self.q`,
`self.context`, `self.starttime`, and `self.docset`.

#### `run()`

Iterates over sub-searchers, calling `set_subsearcher()` and
`collect_matches()` for each, then calls `finish()`.

#### `set_subsearcher(subsearcher, offset)`

Called when moving to a new sub-searcher. Sets `self.subsearcher`,
`self.offset`, and `self.matcher`.

#### `collect(sub_docnum)`

Called for every matched document. Must add the document to results and
return a sort key. Subclasses must implement this.

- `sub_docnum`: Segment-relative document number. Add `self.offset` to get
  the top-level document number.

#### `sort_key(sub_docnum)`

Returns a sort key for the current match without the side effect of adding
the document to results. Subclasses must implement this.

#### `collect_matches()`

Calls `matches()` and then `collect()` for each matched document.

#### `matches()`

Yields segment-relative document numbers for matches in the current
sub-searcher.

#### `count()`

Returns the total number of matching documents.

#### `all_ids()`

Returns a sequence of docnums matched in this collector.

#### `computes_count()`

Returns `True` if the collector naturally computes the exact count of
matching documents.

#### `finish()`

Called after the search completes. Sets `self.runtime`.

#### `remove(global_docnum)`

Removes a document from the collector using its global docnum.

#### `results()`

Returns a `Results` object. Subclasses must implement this.

### `ilen`

```python
whoosh.collectors.ilen(iterator) -> int
```

Counts the number of items in an iterator without loading it all into memory.

## Scored Collectors

### `ScoredCollector`

```python
class whoosh.collectors.ScoredCollector(replace=10)
```

Base class for collectors that sort by document score.

**Constructor:**
- `replace`: Number of matches between attempts to replace the matcher with
  a more efficient version.

### `TopCollector`

```python
class whoosh.collectors.TopCollector(
    limit=10,
    usequality=True,
    **kwargs
)
```

A collector that returns only the top N scored results.

**Constructor:**
- `limit`: Maximum number of results to return.
- `usequality`: Whether to use block-quality optimizations for faster
  search. Can be set to `False` for debugging.

**Notes:**
- When `usequality=True`, `computes_count()` returns `False` and
  `all_ids()` requires re-searching.
- Uses a min-heap to efficiently track the top N documents.

### `UnlimitedCollector`

```python
class whoosh.collectors.UnlimitedCollector(reverse=False)
```

A collector that returns **all** scored results. Sorts by score (descending
by default).

**Constructor:**
- `reverse`: If `True`, sort results in ascending order (lowest scores first).

### `UnsortedCollector`

```python
class whoosh.collectors.UnsortedCollector
```

A collector that returns results in document order (no sorting). Used when
the search weighting is `None`.

## Wrapping Collectors

### `WrappingCollector`

```python
class whoosh.collectors.WrappingCollector(child)
```

Base class for collectors that wrap other collectors. Delegates most
operations to the child collector while adding additional behavior.

**Constructor:**
- `child`: The collector to wrap.

**Methods** (all delegated to child):
`top_searcher`, `context`, `prepare`, `set_subsearcher`, `all_ids`,
`count`, `collect_matches`, `sort_key`, `collect`, `remove`, `matches`,
`finish`, `results()`

### `SortingCollector`

```python
class whoosh.collectors.SortingCollector(
    sortedby,
    limit=10,
    reverse=False
)
```

A collector that returns results sorted by a `FacetType` object.

**Constructor:**
- `sortedby`: A `FacetType` or field name to sort by.
- `limit`: Maximum number of results (0 for no limit).
- `reverse`: If `True`, reverse the overall sort order.

### `FilterCollector`

```python
class whoosh.collectors.FilterCollector(
    child,
    allow=None,
    restrict=None
)
```

A collector that allows and/or restricts certain document numbers in
results.

A document is discarded if:
- `allow` is set and the docnum is not in the allowed set, or
- `restrict` is set and the docnum is in the restricted set.

**Constructor:**
- `child`: The collector to wrap.
- `allow`: A query, `Results` object, or set-like of allowed docnums.
  `None` means everything is allowed.
- `restrict`: A query, `Results` object, or set-like of disallowed docnums.
  `None` means nothing is disallowed.

**Attributes:**
- `filtered_count`: Number of documents filtered out.

### `FacetCollector`

```python
class whoosh.collectors.FacetCollector(child, groupedby, maptype=None)
```

A collector that creates groups of documents based on facet objects. Used
when `groupedby` is specified in `Searcher.search()`.

**Constructor:**
- `child`: The collector to wrap.
- `groupedby`: A field name, `FacetType`, dict, or `Facets` object.
- `maptype`: Default `FacetMap` class for facets that don't specify one.

**Attributes:**
- `facetmaps`: Dictionary of facet name to `FacetMap` objects.

### `CollapseCollector`

```python
class whoosh.collectors.CollapseCollector(
    child,
    keyfacet,
    limit=1,
    order=None
)
```

A collector that eliminates all but the top N results sharing the same facet
key. Useful for "dedup" or grouped result views.

**Constructor:**
- `child`: The collector to wrap.
- `keyfacet`: A `FacetType` to collapse on. All but the top N documents
  sharing a key are eliminated.
- `limit`: Maximum documents to keep per key (default `1`).
- `order`: Optional `FacetType` to determine which documents are "top" within
  each group. Defaults to the results order (e.g., highest score).

**Attributes:**
- `collapsed_counts`: Dictionary mapping keys to the number of documents
  eliminated.

### `TimeLimitCollector`

```python
class whoosh.collectors.TimeLimitCollector(
    child,
    timelimit,
    greedy=False,
    use_alarm=True
)
```

A collector that raises a `TimeLimit` exception if the search exceeds a
time limit. Partial results are still available via `results()`.

**Constructor:**
- `child`: The collector to wrap.
- `timelimit`: Maximum search time in seconds.
- `greedy`: If `True`, finish adding the current hit before raising.
- `use_alarm`: If `True` (default), use `signal.SIGALRM` on Unix for
  immediate interruption. On Windows, time is only checked between
  documents.

```python
from whoosh.searching import TimeLimit

uc = collectors.UnlimitedCollector()
tlc = TimeLimitCollector(uc, timelimit=5.8)
try:
    searcher.search_with_collector(myquery, tlc)
except TimeLimit:
    print("Search timed out!")
# Still get partial results:
print(tlc.results())
```

### `TermsCollector`

```python
class whoosh.collectors.TermsCollector(child, settype=set)
```

A collector that records which terms appeared in which matched documents.
Used when `terms=True` in `Searcher.search()`.

**Constructor:**
- `child`: The collector to wrap.
- `settype`: Set type to use for docnum collections (default `set`).

**Attributes:**
- `termdocs`: Dict mapping `(fieldname, text)` tuples to arrays of docnums.
- `docterms`: Dict mapping docnums to lists of `(fieldname, text)` tuples.

## Exceptions

### `TimeLimit`

```python
from whoosh.searching import TimeLimit
```

Raised by `TimeLimitCollector` when the search exceeds the time limit.
Partial results are still available from the collector.


## DOCUMENT: Columns

# Columns API

Classes for storing per-document values (column-oriented storage) used for
fast sorting, faceting, and filtering. Columns are the mechanism by which
Whoosh stores field values alongside the inverted index, in a column-oriented
layout for efficient range access.

The default column type for most fields is `VarBytesColumn`, although numeric
and date fields use `NumericColumn`. Expert users may use other column types
that may be faster or more storage-efficient based on the field contents.

A `Column` object stores configuration information and provides two important
methods: `writer()` to return a `ColumnWriter` and `reader()` to return a
`ColumnReader`.

## Module Functions

### `bytes_column`

```python
whoosh.columns.bytes_column
```

A default `VarBytesColumn` instance used as the column type for string fields.

### `numeric_column`

```python
whoosh.columns.numeric_column
```

A default `NumericColumn` instance used as the column type for numeric fields.

## Base Classes

### `Column`

```python
class whoosh.columns.Column
```

Base class for all column types.

**Class Attributes:**
- `reversible (bool)`: Whether values can be reversed for descending sort.
  Default `False`.

**Methods:**
- `writer(dbfile)`: Returns a `ColumnWriter` for this column type.
- `reader(dbfile, basepos, length, doccount)`: Returns a `ColumnReader` for
  this column type.
- `default_value(reverse=False)`: Returns the default value for documents
  without a column value at index time.
- `stores_lists()`: Returns `True` if the column stores a list of values per
  document instead of a single value.

### `ColumnWriter`

```python
class whoosh.columns.ColumnWriter(dbfile)
```

Base class for writing column values to disk.

**Constructor:**
- `dbfile`: The `StructFile` to write to.

**Methods:**
- `fill(docnum)`: Fills any gap in docnums up to `docnum` with default values.
- `add(docnum, value)`: Adds a value for the given docnum.
- `finish(docnum)`: Called when done writing. Default does nothing.

### `ColumnReader`

```python
class whoosh.columns.ColumnReader(dbfile, basepos, length, doccount)
```

Base class for reading column values from disk.

**Constructor:**
- `dbfile`: The `StructFile` to read from.
- `basepos`: The offset within the file at which the column starts.
- `length`: The length in bytes the column occupies in the file.
- `doccount`: The number of rows (documents) in the column.

**Methods:**
- `__getitem__(docnum)`: Returns the value for the given docnum.
- `sort_key(docnum)`: Returns the value for sorting (defaults to
  `__getitem__`).
- `__iter__()`: Yields values for all documents.
- `load()`: Returns a list of all values.
- `set_reverse()`: Prepares the reader for reverse iteration.

## Concrete Column Types

### `VarBytesColumn`

```python
class whoosh.columns.VarBytesColumn(
    allow_offsets=True,
    write_offsets_cutoff=2**15
)
```

Stores variable-length byte strings. The default value for documents without
a value is `b''` (empty bytes).

**Constructor:**
- `allow_offsets`: Whether to write offsets for faster lookup when there are
  many rows. Default `True`.
- `write_offsets_cutoff`: Write offsets when there are more than this many
  rows (default `2**15`).

### `FixedBytesColumn`

```python
class whoosh.columns.FixedBytesColumn(blocksize, default=emptybytes)
```

Stores fixed-length byte strings, saving space by not storing the length of
each value.

**Constructor:**
- `blocksize`: Fixed size of each value in bytes.
- `default`: Default value for documents without a value.

### `RefBytesColumn`

```python
class whoosh.columns.RefBytesColumn(
    cachesize=1000,
    stable=True,
    default=emptybytes
)
```

Stores references to unique values rather than the values themselves, saving
space when the field has few unique values. Uses a `DocIdSet` to track which
documents contain each value.

**Constructor:**
- `cachesize`: Size of the LRU cache for value lookups (default `1000`).
- `stable`: Whether to use a stable sort of references (default `True`).
- `default`: Default value for missing documents.

### `NumericColumn`

```python
class whoosh.columns.NumericColumn(
    typecode,
    default=None,
    nullable=False
)
```

Stores numbers (int, float, datetime) encoded as binary values. Extends
`FixedBytesColumn`.

**Constructor:**
- `typecode`: A `struct` typecode string (e.g., `"I"` for unsigned int,
  `"q"` for long, `"d"` for float).
- `default`: Default numeric value (None for the type's zero value).
- `nullable`: Whether `None` values are allowed.

### `BitColumn`

```python
class whoosh.columns.BitColumn
```

Stores boolean values as a bitmap. Each value is either `True` (1) or
`False` (0). Uses a `BitSet` internally.

### `CompressedBytesColumn`

```python
class whoosh.columns.CompressedBytesColumn(default=emptybytes)
```

Wraps a `VarBytesColumn` with zlib compression for the value bytes.

### `CompressedBlockColumn`

```python
class whoosh.columns.CompressedBlockColumn
```

Stores values with block-level zlib compression. More efficient for large
columns.

### `StructColumn`

```python
class whoosh.columns.StructColumn(struct, name)
```

Wraps a `FixedBytesColumn` to store structured binary data (e.g., tuples
encoded with `struct`).

**Constructor:**
- `struct`: A `struct.Struct` object defining the format.
- `name`: Field name for error messages.

### `EmptyColumnReader`

```python
class whoosh.columns.EmptyColumnReader(default, doccount)
```

A `ColumnReader` that returns a constant default value for every document.
Used when a field has no column.

### `MultiColumnReader`

```python
class whoosh.columns.MultiColumnReader(readers)
```

Combines multiple `ColumnReader` instances into one for multi-segment indices.

**Constructor:**
- `readers`: List of `ColumnReader` instances (one per segment).

### `TranslatingColumnReader`

```python
class whoosh.columns.TranslatingColumnReader(child, translator)
```

Wraps a `ColumnReader` to apply a translation function to the values.

**Constructor:**
- `child`: The underlying `ColumnReader`.
- `translator`: Function that maps sort keys to human-readable values.

### `WrappedColumn`

```python
class whoosh.columns.WrappedColumn(child)
```

Base class for column wrappers that adapt another column type.

### `WrappedColumnWriter`

```python
class whoosh.columns.WrappedColumnWriter(child)
```

Base class for column writer wrappers.

### `WrappedColumnReader`

```python
class whoosh.columns.WrappedColumnReader(child)
```

Base class for column reader wrappers.

### `ClampedNumericColumn`

```python
class whoosh.columns.ClampedNumericColumn(child, clampfn)
```

Wraps a `NumericColumn` to clamp values to a valid range before sorting.

**Constructor:**
- `child`: The wrapped `NumericColumn`.
- `clampfn`: Function that clamps a value to the valid range.

### `PickleColumn`

```python
class whoosh.columns.PickleColumn(child, ...)
```

Wraps another column to store pickled Python objects.

### `ListColumn`

```python
class whoosh.columns.ListColumn(child)
```

Base class for columns that store multiple values per document.

### `ListColumnReader`

```python
class whoosh.columns.ListColumnReader(child)
```

Reader for list-valued columns.

### `VarBytesListColumn`

```python
class whoosh.columns.VarBytesListColumn
```

A `ListColumn` variant of `VarBytesColumn` that stores lists of byte strings.

### `FixedBytesListColumn`

```python
class whoosh.columns.FixedBytesListColumn(blocksize)
```

A `ListColumn` variant of `FixedBytesColumn` that stores lists of fixed-size
byte strings.


## DOCUMENT: Core

# Core API

The core module provides the main `Index` class and related functions for managing indexes.

## Functions

### create_in

```python
whoosh.index.create_in(
    dirname: str,
    schema: Schema,
    indexname: str = "MAIN",
    create: bool = True,
    **kwargs
) -> FileIndex
```

Create a new index in the given directory.

**Args:**
- `dirname (str)`: Path to the directory where the index will be stored.
- `schema (Schema)`: The `Schema` object defining the index fields.
- `indexname (str)`: Name of the index. Allows multiple indexes in the same directory.
- `create (bool)`: If True, create the index even if it already exists (clears existing).

**Returns:**
- `FileIndex`: A new index object.

**Example:**
```python
from whoosh.index import create_in
from whoosh.fields import Schema, TEXT

schema = Schema(title=TEXT(stored=True), content=TEXT)
index = create_in("indexdir", schema)
```

---

### open_dir

```python
whoosh.index.open_dir(
    dirname: str,
    indexname: str = "MAIN",
    readonly: bool = False,
    **kwargs
) -> FileIndex
```

Open an existing index.

**Args:**
- `dirname (str)`: Path to the index directory.
- `indexname (str)`: Name of the index to open.
- `readonly (bool)`: If True, open in read-only mode.

**Returns:**
- `FileIndex`: An index object.

**Example:**
```python
from whoosh.index import open_dir
index = open_dir("indexdir")
```

---

### exists_in

```python
whoosh.index.exists_in(
    dirname: str,
    indexname: str = "MAIN",
    **kwargs
) -> bool
```

Check if a valid index exists in the given directory.

**Returns:**
- `bool`: True if the index exists.

---

### create_index

```python
whoosh.index.create_index(
    schema: Schema,
    storage: Storage,
    indexname: str = "MAIN",
    create: bool = True,
    **kwargs
) -> Index
```

Low-level index creation. Use `create_in` instead unless you need custom storage.

---

### open_index

```python
whoosh.index.open_index(
    storage: Storage,
    indexname: str = "MAIN",
    readonly: bool = False,
    **kwargs
) -> Index
```

Low-level index opening. Use `open_dir` instead unless you need custom storage.

## Classes

### Index (Base Class)

```python
class whoosh.index.Index
```

Abstract base class for index objects. Provides common methods for reading and writing.

**Methods:**

#### `writer()`

```python
writer = ix.writer(
    timeout: float = 0.0,
    delay: float = 0.1,
    limitmb: int = 128,
    **kwargs
) -> IndexWriter
```

Return a writer for this index.

**Args:**
- `timeout (float)`: Max seconds to wait for write lock.
- `delay (float)`: Seconds between lock retries.
- `limitmb (int)`: Maximum size of posting pool runs.

**Returns:**
- `IndexWriter`: A writer object.

**Example:**
```python
writer = ix.writer()
writer.add_document(title="Hello", content="World")
writer.commit()
```

---

#### `searcher()`

```python
searcher = ix.searcher(
    weighting: WeightingModel = None,
    **kwargs
) -> Searcher
```

Return a searcher for the current index state.

**Returns:**
- `Searcher`: A searcher object.

**Example:**
```python
with ix.searcher() as searcher:
    results = searcher.search("query")
```

---

#### `reader()`

```python
reader = ix.reader() -> IndexReader
```

Return a reader for the current index state.

---

#### `commit()`

```python
ix.commit(mergetype=None, optimize=None, merge=None)
```

Convenience method: create a writer, call commit, and close.

---

#### `optimize()`

```python
ix.optimize()
```

Merge all segments into a single segment.

---

#### `add_field()`

```python
ix.add_field(fieldname: str, fieldtype, **kwargs)
```

Add a field to the index schema.

---

#### `remove_field()`

```python
ix.remove_field(fieldname: str, **kwargs)
```

Remove a field from the index schema.

---

#### `doc_count()`

```python
count = ix.doc_count() -> int
```

Return the number of documents in the index.

---

#### `doc_count_all()`

```python
count = ix.doc_count_all() -> int
```

Return the total number of documents (including deleted).

---

#### `lock()`

```python
lock = ix.lock(name: str) -> Lock
```

Acquire a named lock on the index.

## FileIndex

The concrete implementation returned by `create_in` and `open_dir`.

All `Index` methods are available. Additional methods:

### `_read_toc()`

Read the table of contents.

### `_write_toc()`

Write the table of contents.

## Exceptions

### LockError

Raised when the index is locked by another writer.

```python
from whoosh.index import LockError

try:
    writer = ix.writer(timeout=5.0)
except LockError:
    print("Index is locked, try again later")
```

### IndexMissingError

Raised when trying to open a non-existent index.

## Constants

### IndexVersion

Current index format version.

---

# Index API

## Index

```python
class whoosh.index.Index
```

Base index class providing reading and writing access.

### Methods

- `writer(**kwargs)` -> `IndexWriter`
- `searcher(**kwargs)` -> `Searcher`
- `reader()` -> `IndexReader`
- `commit(mergetype=None, optimize=None, merge=None)`
- `optimize()`
- `add_field(fieldname, fieldtype, **kwargs)`
- `remove_field(fieldname, **kwargs)`
- `doc_count() -> int`
- `doc_count_all() -> int`
- `lock(name) -> Lock`

## IndexingError

```python
class whoosh.writing.IndexingError(Exception)
```

Raised when an indexing operation fails.

## Exceptions

```python
class whoosh.index.LockError(Exception)
class whoosh.index.IndexMissingError(Exception)
```


## DOCUMENT: Events

# Event Bus & Hooks API

Loose coupling through events and lightweight hooks.

## Event Bus

```python
class whoosh.event_bus.EventBus
```

Publish/subscribe event system supporting both synchronous and asynchronous
listeners. The module-level singleton `event_bus` is available for use:

```python
from whoosh.event_bus import event_bus
```

### Methods

#### `subscribe()`

```python
@event_bus.subscribe(DocumentIndexed)
async def handler(event: DocumentIndexed):
    print(f"Indexed document: {event.document_id}")

# Or without decorator
event_bus.subscribe(DocumentIndexed)(handler)
```

Register a handler function (synchronous or async) for a specific event type.
The decorator takes the event class as an argument. Returns the handler
unchanged so it can be used normally.

---

#### `publish()`

```python
from whoosh.event_bus import event_bus, DocumentIndexed

event_bus.publish(DocumentIndexed(document_id="doc123"))
```

Publish an event to all subscribers. If listeners are async coroutines and no
event loop is running, they are executed via `asyncio.run()`. If an event
loop is running, tasks are scheduled on it. Exceptions in listeners are
swallowed.

---

#### `clear()`

```python
event_bus.clear()
```

Remove all subscribers.

---

## Events

Events are immutable dataclasses.

### `Event`

```python
@dataclass(frozen=True)
class Event:
    pass
```

Base class for all events.

---

### `DocumentIndexed`

```python
@dataclass(frozen=True)
class DocumentIndexed(Event):
    document_id: str
```

Published when a document is indexed. Contains the document ID.

---

### `SearchExecuted`

```python
@dataclass(frozen=True)
class SearchExecuted(Event):
    query: str
```

Published after a search is executed. Contains the query string.

---

## Hooks

Hook system for cross-cutting concerns. Hooks are registered globally using a
module-level registry.

### `hookimpl`

```python
from whoosh.hooks import hookimpl

@hookimpl
def before_search(context):
    context.query = optimize_query(context.query)
    return context
```

Decorator that marks a function as a hook implementation. Returns a `HookImpl`
wrapper.

### `register_hook()`

```python
from whoosh.hooks import register_hook, hookimpl

@hookimpl
def before_search(context):
    ...

register_hook("before_search", before_search)
```

Register a `HookImpl` under a named hook. Multiple hooks can be registered
per name; they are called in registration order.

### `call_hook()`

```python
from whoosh.hooks import call_hook

results = await call_hook("before_search", context)
```

Async function that calls all hooks registered under the given name. Returns
a list of results from each hook's execution. Exceptions in individual hooks
are logged but do not stop execution.

---
## Example: Event Bus

```python
from whoosh.event_bus import event_bus, DocumentIndexed, SearchExecuted

@event_bus.subscribe(DocumentIndexed)
async def on_document_indexed(event: DocumentIndexed):
    print(f"Document indexed: {event.document_id}")

@event_bus.subscribe(SearchExecuted)
async def on_search_executed(event: SearchExecuted):
    print(f"Search executed: {event.query}")

# Publish events
event_bus.publish(DocumentIndexed(document_id="doc123"))
event_bus.publish(SearchExecuted(query="hello world"))
```

## Example: Hooks

```python
from whoosh.hooks import hookimpl, register_hook, call_hook
import asyncio

@hookimpl
def before_search(context):
    print(f"Searching for: {context['query']}")
    return context

register_hook("before_search", before_search)

# Call hooks
context = {"query": "hello"}
results = asyncio.run(call_hook("before_search", context))
```


## DOCUMENT: Fields

# Fields API

Define the structure of your index with field types.

## Schema

```python
class whoosh.fields.Schema
```

The `Schema` class defines the fields available in an index.

### Constructor

```python
schema = Schema(
    title=TEXT(stored=True),
    content=TEXT,
    path=ID(stored=True, unique=True),
    tags=KEYWORD(lowercase=True),
    rating=NUMERIC(float, stored=True),
    published=DATETIME(stored=True),
    active=BOOLEAN
)
```

### Methods

#### `add()`

```python
schema.add(
    fieldname: str,
    fieldtype,
    glob: bool = False,
    **kwargs
)
```

Add a field to the schema. If `glob=True`, the fieldname is treated as a glob pattern.

#### `remove()`

```python
schema.remove(fieldname: str, **kwargs)
```

Remove a field from the schema.

#### `items()`

```python
for name, field in schema.items():
    print(name, field)
```

Return a list of (fieldname, field object) pairs.

#### `names()`

```python
names = schema.names()
```

Return a list of field names.

## FieldType Base Class

```python
class whoosh.fields.FieldType
```

Base class for all field types.

### Attributes

| Attribute | Type | Description |
|-----------|------|-------------|
| `format` | `Format` | Defines how the field is indexed |
| `vector` | `Format` or None | Optional per-document vector format |
| `scorable` | `bool` | Whether field length is stored (for BM25F) |
| `stored` | `bool` | Whether field value is stored in index |
| `unique` | `bool` | Whether field uniquely identifies documents |

### Methods

#### `index()`

Convert a value into indexed items.

#### `indexable()`

Check if the value can be indexed.

#### `spelling_fieldname()`

Return the field name used for spelling data.

#### `spellable_words()`

Generate spellable words from a value.

## Built-in Field Types

### TEXT

```python
whoosh.fields.TEXT(
    stored: bool = False,
    unique: bool = False,
    phrase: bool = True,
    analyzer: Analyzer = None,
    field_boost: float = 1.0,
    **kwargs
)
```

Full-text field with tokenization and optional phrase search.

**Example:**
```python
title = TEXT(stored=True)
body = TEXT(analyzer=StemmingAnalyzer(), phrase=False)
```

---

### ID

```python
whoosh.fields.ID(
    stored: bool = False,
    unique: bool = False,
    field_boost: float = 1.0,
    **kwargs
)
```

Untokenized identifier field. Stores the entire value as a single term.

**Example:**
```python
path = ID(stored=True, unique=True)
slug = ID(stored=True)
```

---

### KEYWORD

```python
whoosh.fields.KEYWORD(
    stored: bool = False,
    lowercase: bool = False,
    commas: bool = False,
    scorable: bool = False,
    field_boost: float = 1.0,
    **kwargs
)
```

Space or comma-separated keywords. Phrase search is not supported.

**Example:**
```python
tags = KEYWORD(lowercase=True, commas=True, stored=True)
```

---

### STORED

```python
whoosh.fields.STORED(
    stored: bool = True,
    unique: bool = False,
    **kwargs
)
```

Stored-only field. Not indexed or searchable.

**Example:**
```python
icon = STORED()
description = STORED()
```

---

### NUMERIC

```python
whoosh.fields.NUMERIC(
    numtype: type = int,
    stored: bool = False,
    unique: bool = False,
    field_boost: float = 1.0,
    **kwargs
)
```

Numeric field for integers or floats.

**Example:**
```python
rating = NUMERIC(float, stored=True)
count = NUMERIC(int)
price = NUMERIC(float, stored=True, sortable=True)
```

---

### DATETIME

```python
whoosh.fields.DATETIME(
    stored: bool = False,
    unique: bool = False,
    field_boost: float = 1.0,
    **kwargs
)
```

Date/time field. Stores `datetime` objects.

**Example:**
```python
published = DATETIME(stored=True)
updated = DATETIME()
```

---

### BOOLEAN

```python
whoosh.fields.BOOLEAN(
    stored: bool = False,
    unique: bool = False,
    field_boost: float = 1.0,
    **kwargs
)
```

Boolean field. Searchable with `yes`, `no`, `true`, `false`, `1`, `0`, `t`, `f`.

**Example:**
```python
published = BOOLEAN(stored=True)
```

---

### NGRAM

```python
whoosh.fields.NGRAM(
    minsize: int = 2,
    maxsize: int = 5,
    stored: bool = False,
    field_boost: float = 1.0,
    **kwargs
)
```

Character n-gram field.

---

### NGRAMWORDS

```python
whoosh.fields.NGRAMWORDS(
    minsize: int = 2,
    maxsize: int = 5,
    stored: bool = False,
    field_boost: float = 1.0,
    **kwargs
)
```

Word-level n-gram field.

---

### VectorField

```python
whoosh.fields.VectorField(
    dimensions: int,
    metric: str = "cosine",
    provider: str = "numpy",
    stored: bool = False,
    **kwargs
)
```

Field for storing and searching vector embeddings.

**Args:**
- `dimensions (int)`: Embedding dimension (e.g., 384 for all-MiniLM-L6-v2).
- `metric (str)`: Similarity metric: `"cosine"`, `"euclidean"`, `"dot"`.
- `provider (str)`: Vector provider name from registry.

**Example:**
```python
embedding = VectorField(dimensions=384, metric="cosine", stored=True)
```

## SchemaBuilder

Fluent API for building schemas:

```python
from whoosh.fields import SchemaBuilder

schema = (
    SchemaBuilder()
    .field("title", TEXT(stored=True))
    .field("path", ID(stored=True, unique=True))
    .field("content", TEXT)
    .field("tags", KEYWORD(lowercase=True))
    .field("published", DATETIME(stored=True))
    .build()
)
```

## Constants

- `whoosh.fields.STORED`: Stored-only field type
- `whoosh.fields.TEXT`: Full-text field
- `whoosh.fields.ID`: Identifier field
- `whoosh.fields.KEYWORD`: Keyword field
- `whoosh.fields.NUMERIC`: Numeric field
- `whoosh.fields.DATETIME`: Date/time field
- `whoosh.fields.BOOLEAN`: Boolean field


## DOCUMENT: Filedb Storage

# File DB / Storage API

Classes for storing and retrieving index data on disk or in memory. The
`Storage` class is the main entry point for persisting an index.

## Storage Classes

### `Storage`

```python
class whoosh.filedb.filestore.Storage(path=None)
```

Abstract base class for storage backends. A `Storage` manages a filesystem-
or memory-based location where index files can be created, read, and
manipulated.

**Constructor:**
- `path`: Optional path string. Subclasses may use this to set the storage
  location.

**Methods:**

#### `create_file(name, **kwargs)`

Creates and returns a file object for writing.

#### `open_file(name, **kwargs)`

Opens and returns a file object for reading.

#### `list()`

Returns a list of all filenames in this storage.

#### `exists(name)`

Returns `True` if a file/named item exists in the storage.

#### `file_exists(name)`

Alias for `exists()`.

#### `file_length(name)`

Returns the length of file `name` in bytes.

#### `rename(src, dst)`

Renames a file from `src` to `dst`.

#### `delete_file(name)`

Deletes file `name` from storage.

#### `destroy()`

Deletes all files and the storage itself.

#### `temp_storage()`

Creates and returns a temporary isolated `Storage` for scratch space.

#### `supports_mmap`

Returns `True` if this storage supports memory-mapped file access.

**Properties:**
- `schema`: The `Schema` for this storage (if it holds an index).
- `lock`: The lock object used for this storage.

### `FileStorage`

```python
class whoosh.filedb.filestore.FileStorage(
    path,
    cachesize_limit=40,
    supports_mmap=None,
    **kwargs
)
```

A `Storage` subclass that uses the operating system's filesystem.

**Constructor:**
- `path`: A `Path` (or string path) to the directory where files are stored.
- `cachesize_limit`: Maximum number of open file handles to cache.
- `supports_mmap`: If `None`, auto-detected; otherwise force enable/disable.

**Methods:** All `Storage` methods plus:
- `create_index(schema, indexname="index", ...)`: Creates and returns a new
  `Index` object.
- `open_index(indexname="index", ...)`: Opens an existing `Index`.
- `lock(name)`: Returns a lock object for the given lock name.

### `RamStorage`

```python
class whoosh.filedb.filestore.RamStorage(cachesize_limit=10)
```

A `Storage` subclass that keeps all files in memory as bytes. Useful for
testing and small indexes.

**Constructor:**
- `cachesize_limit`: Maximum number of files to cache as decoded objects.

**Methods:** All `Storage` methods plus:
- `create_index(schema, ...)`: Creates an in-memory `Index`.
- `save_to_file(filename, ...)`: Saves the entire storage to a file.
- `load_from_file(filename, ...)`: Loads storage contents from a file.

### `OverlayStorage`

```python
class whoosh.filedb.filestore.OverlayStorage(base, overlay)
```

A `Storage` wrapper that presents two storage layers: a base and an overlay.
Files in the overlay take precedence over the base.

**Constructor:**
- `base`: The base `Storage` (e.g., read-only original).
- `overlay`: The overlay `Storage` (e.g., writable copy).

## Storage Exceptions

### `StorageError`

```python
class whoosh.filedb.filestore.StorageError
```

Base exception for storage-related errors.

### `ReadOnlyError`

```python
class whoosh.filedb.filestore.ReadOnlyError(StorageError)
```

Raised when attempting to write to a read-only storage.

## File Tables

### `HashWriter`

```python
class whoosh.filedb.filetables.HashWriter(dbfile, keycoder=None, keydecoder=None, data_encoder=None, data_decoder=None, **kwargs)
```

Writes key-value pairs to a file, with optional indexing by key.

**Constructor:**
- `dbfile`: The `StructFile` to write to.
- `keycoder`: Function to encode keys for storage.
- `keydecoder`: Function to decode keys from storage.
- `data_encoder`: Function to encode values.
- `data_decoder`: Function to decode values.

### `HashReader`

```python
class whoosh.filedb.filetables.HashReader(dbfile, length, keycoder=None, keydecoder=None, data_decoder=None, **kwargs)
```

Reads key-value pairs from a file written by `HashWriter`.

**Constructor:**
- `dbfile`: The `StructFile` to read from.
- `length`: Length of the data section.
- `keycoder`/`keydecoder`/`data_decoder`: Same as `HashWriter`.

**Methods:**
- `__getitem__(key)`: Returns the value for `key`.
- `keys()`: Yields all keys.
- `values()`: Yields all values.
- `items()`: Yields `(key, value)` pairs.
- `keys_from(prefixbytes)`: Yields keys starting at `prefixbytes`.
- `items_from(prefixbytes)`: Yields `(key, value)` pairs starting at prefix.
- `closest_key_pos(key)`: Returns the position of the closest matching key.
- `range_for_key(key)`: Returns `(startpos, endpos)` for a key range.

### `OrderedHashWriter`

```python
class whoosh.filedb.filetables.OrderedHashWriter(HashWriter)
```

A `HashWriter` that maintains keys in sorted order.

### `OrderedHashReader`

```python
class whoosh.filedb.filetables.OrderedHashReader(HashReader)`

A `HashReader` for reading data written by `OrderedHashWriter`. Preserves
key ordering for efficient prefix iteration.

### `FieldedOrderedHashWriter`

```python
class whoosh.filedb.filetables.FieldedOrderedHashWriter(HashWriter)
```

An `OrderedHashWriter` that stores an extra "fieldmap" in the extras dict,
mapping field names to numeric IDs.

### `FieldedOrderedHashReader`

```python
class whoosh.filedb.filetables.FieldedOrderedHashReader(HashReader)
```

Reader for data written by `FieldedOrderedHashWriter`.

## Struct File

### `StructFile`

```python
class whoosh.filedb.structfile.StructFile(name, source, cachesize_limit=40)
```

Wraps a file object and adds methods for reading/writing packed binary
values, arrays, varints, and pickle objects.

**Methods include:**
- `read_int()`, `write_int(n)`: Read/write a 4-byte signed integer.
- `read_long()`, `write_long(n)`: Read/write a 8-byte signed integer.
- `read_uint()`, `write_uint(n)`: Read/write unsigned int.
- `read_ulong()`, `write_ulong(n)`: Read/write unsigned long.
- `read_float()`, `write_float(n)`: Read/write a float.
- `read_ushort()`, `write_ushort(n)`: Read/write unsigned short.
- `read_byte()`, `write_byte(b)`: Read/write a single byte.
- `write_array(arr)`: Write an array of values.
- `get_array(offset, typecode, length)`: Read an array from offset.
- `write_pickle(obj)`: Pickle and write an object.
- `read_pickle()`: Read and unpickle an object.
- `get(offset, length)`: Read `length` bytes from `offset`.
- `get_int()`, `get_uint()`, `get_long()`, `get_float()`, `get_byte()`:
  Read a single value from the given offset.

### `BufferFile`

```python
class whoosh.filedb.structfile.BufferFile
```

A `StructFile` that wraps an in-memory byte buffer.

### `ChecksumFile`

```python
class whoosh.filedb.structfile.ChecksumFile(dbfile)
```

A `StructFile` wrapper that computes a checksum as data is written, for
integrity verification.

## Compound Storage

### `CompoundStorage`

```python
class whoosh.filedb.compound.CompoundStorage(dbfile, use_mmap=True)
```

Treats a single file as a container for multiple sub-files. Used for compound
segment files.

**Methods:**
- `create_file(name)`: Create a sub-file within the compound file.
- `open_file(name)`: Open a sub-file for reading.
- `list()`: List all sub-file names.
- `close()`: Close the compound storage.

### `SubFile`

```python
class whoosh.filedb.compound.SubFile
```

A file-like object representing a sub-file within a `CompoundStorage`.

### `CompoundWriter`

```python
class whoosh.filedb.compound.CompoundWriter(storage)
```

Writes a compound file by assembling multiple files from a storage.

**Methods:**
- `create_file(name)`: Reserve a filename in the compound file.
- `save_as_files(dest_storage, fn_generator)`: Assemble the compound file
  from source files into the destination storage.

## Storage Utility Functions

### `copy_storage`

```python
whoosh.filedb.filestore.copy_storage(sourcestore, deststore)
```

Copies all files from one storage to another.

### `copy_to_ram`

```python
whoosh.filedb.filestore.copy_to_ram(storage)
```

Reads all files from a storage into a `RamStorage` and returns it.


## DOCUMENT: Formats

# Formats API

Classes that control how posting information (frequencies, positions,
character offsets, and weights) is encoded and stored for each field in the
index. The `Format` object is a factory and encoder/decoder for the
value strings stored alongside each posting.

## Module Functions

### `tokens`

```python
whoosh.formats.tokens(value, analyzer, kwargs)
```

Takes a text `value` and an `analyzer`, runs the analyzer on the value, and
returns the resulting token generator (wrapped with `unstopped()` to ignore
`STOP` tokens). Used internally by `Format.word_values()`.

## Format Classes

All format classes accept a `field_boost` parameter (default `1.0`) that
scales the score of all queries matching terms in that field.

### `Format`

```python
class whoosh.formats.Format(field_boost=1.0, **options)
```

Abstract base class for all posting formats. Format objects are
field-level objects: one is created per `Field` and shared across all
postings for that field.

**Attributes:**
- `posting_size (int)`: Fixed byte size of encoded postings, or `None`/`-1`
  if variable-size.
- `textual (bool)`: Whether this format expects string tokens (vs. bytes).
  Default `True`.

**Methods:**

#### `word_values(value, analyzer, **kwargs)`

Abstract. Takes a text value, runs it through the analyzer, and yields
`(tokentext, frequency, weight, valuestring)` tuples.

#### `encode(value)`

Abstract. Encodes raw posting data into the value string bytes.

#### `decode_frequency(valuestring)`

Abstract. Decodes the frequency (term count in document) from the value
string.

#### `decode_weight(valuestring)`

Abstract. Decodes the weight (total boost contribution) from the value string.

#### `combine(valuestrings)`

Abstract. Combines multiple value strings (from overlapping segments) into
a single value string.

#### `supports(name)`

Returns `True` if this format supports interpreting its postings as `name`
(e.g., `"frequency"`, `"positions"`, `"characters"`, `"position_boosts"`,
`"character_boosts"`). Equivalent to `hasattr(self, "decode_" + name)`.

#### `decoder(name)`

Returns the `decode_<name>` method for the given attribute name.

#### `decode_as(astype, valuestring)`

Calls the appropriate `decode_<astype>` method on `valuestring` and returns
the result.

#### `fixed_value_size()`

Returns `self.posting_size` if positive, otherwise `None`.

#### `__eq__(other)`

Returns `True` if `other` is the same class with equal `__dict__`.

### `Existence`

```python
class whoosh.formats.Existence(field_boost=1.0, **options)
```

Indexes only whether a term occurred in a document—not its frequency or
positions. Useful for non-scorable fields like paths.

- `posting_size = 0`
- Supports: `frequency` (always 1), `weight` (always `field_boost`)
- `encode()` returns empty bytes

### `Frequency`

```python
class whoosh.formats.Frequency(field_boost=1.0, boost_as_freq=False, **options)
```

Stores term frequency information (term count per document) for each posting.

- `posting_size = _INT_SIZE` (4 bytes)
- Supports: `frequency`, `weight`
- `encode()` encodes the count as a packed unsigned int
- `boost_as_freq`: If `True`, boosts are interpreted as frequency boosts

```python
from whoosh.formats import Frequency
fmt = Frequency(field_boost=1.0)
```

### `Positions`

```python
class whoosh.formats.Positions(field_boost=1.0, **options)
```

Stores position information (term offsets within the document) in each
posting, enabling phrase queries and "near" queries.

- Supports: `frequency`, `weight`, `positions`, `position_boosts`
- `encode(poslist)` encodes positions using variable-length delta encoding
- Positions are stored as delta-encoded variable-length integers

```python
from whoosh.formats import Positions
fmt = Positions()
```

### `Characters`

```python
class whoosh.formats.Characters(field_boost=1.0, **options)
```

Extends `Positions` to also store character start and end offsets for each
term occurrence, enabling character-precise highlighting.

- Supports: `frequency`, `weight`, `positions`, `position_boosts`,
  `characters`
- `encode()` encodes (position, startchar, endchar) triples with delta
  encoding

### `PositionBoosts`

```python
class whoosh.formats.PositionBoosts(field_boost=1.0, **options)
```

Extends `Positions` to store per-position boost values in addition to
positions.

- Supports: `frequency`, `weight`, `positions`, `position_boosts`
- `encode()` encodes `(position, boost)` pairs

### `CharacterBoosts`

```python
class whoosh.formats.CharacterBoosts(field_boost=1.0, **options)
```

Extends `Characters` to store per-position boost values along with
character offsets.

- Supports: `frequency`, `weight`, `positions`, `position_boosts`,
  `characters`, `character_boosts`
- `encode()` encodes `(position, startchar, endchar, boost)` tuples


## DOCUMENT: Highlight

# Highlight API

Classes and functions for highlighting matches in search result fragments.
The highlight module is a refactored package exposing the same public API as
the former monolithic module.

## Overview

The highlighting system has four components:

- **Fragmenters** split text into fragments.
- **Fragment Scorers** score fragments to determine which to display.
- **Formatters** render fragments as output (HTML, plain text, etc.).
- **Highlighter** ties these together and is used by `Searcher.highlights()`.

## Module-level Functions

### `highlight`

```python
whoosh.highlight.highlight(
    text: str,
    terms: list[str],
    analyzer,
    fragmenter,
    formatter,
    top: int = 3,
    scorer=None,
    minscore: int = 1,
    order=SCORE,
    mode: str = "query"
) -> str
```

Highlights the matched terms in `text` and returns a formatted string.

- `text`: The text to highlight.
- `terms`: A list of matched terms (strings).
- `analyzer`: The analyzer for the field.
- `fragmenter`: A `Fragmenter` instance or class.
- `formatter`: A `Formatter` instance or class.
- `top`: Maximum number of fragments to return.
- `scorer`: Optional fragment scorer (defaults to `BasicFragmentScorer`).
- `minscore`: Minimum score for a fragment to be included.
- `order`: Sort order for fragments (`FIRST`, `SCORE`, `LONGER`, `SHORTER`).
- `mode`: Analysis mode, typically `"query"` or `"index"`.

### `mkfrag`

```python
whoosh.highlight.mkfrag(
    text: str,
    tokens,
    startchar=None,
    endchar=None,
    charsbefore: int = 0,
    charsafter: int = 0
) -> Fragment
```

Returns a `Fragment` object based on `Token` objects in `tokens`.

### `get_text`

```python
whoosh.highlight.get_text(
    original: str,
    token,
    replace: bool
) -> str
```

Returns the text to use for a match when formatting. If `replace` is `False`,
returns the original text between `token.startchar` and `token.endchar`. If
`True`, returns `token.text`.

### `set_matched_filter`

```python
whoosh.highlight.set_matched_filter(
    tokens,
    termset: frozenset
) -> Iterator[Token]
```

Marks tokens as matched if their `text` attribute is in `termset`. Used for
phrase-agnostic highlighting.

### `set_matched_filter_phrases`

```python
whoosh.highlight.set_matched_filter_phrases(
    tokens,
    text: str,
    terms,
    phrases
) -> Iterator[Token]
```

Marks tokens as matched using phrase-aware logic. Highlights only tokens that
are part of matched phrases.

### `top_fragments`

```python
whoosh.highlight.top_fragments(
    fragments,
    count: int,
    scorer,
    order,
    minscore: int = 1
) -> list[Fragment]
```

Returns the best `count` fragments sorted by `order`, filtered by `minscore`.

## Constants

### `DEFAULT_CHARLIMIT`

```python
whoosh.highlight.DEFAULT_CHARLIMIT = 2**15
```

Default character limit for fragments.

### Sort Order Constants

```python
whoosh.highlight.FIRST   # Sort passages from earlier in the document first
whoosh.highlight.SCORE   # Sort higher scored passages first
whoosh.highlight.LONGER  # Sort longer passages first
whoosh.highlight.SHORTER # Sort shorter passages first
```

## Formatters

### `Formatter`

```python
class whoosh.highlight.Formatter
```

Base class for formatters. Subclasses implement `format_token()` to define
how matched tokens are rendered.

**Methods:**

- `format_token(text, token, replace=False)`: Returns formatted text for a
  matched token.
- `format_fragment(fragment, replace=False)`: Returns formatted text for a
  `Fragment`.
- `format(fragments, replace=False)`: Returns formatted text for a list of
  fragments, joined by `between`.

**Attributes:**
- `between`: String inserted between formatted fragments (default `"..."`).

### `NullFormatter`

```python
class whoosh.highlight.NullFormatter(Formatter)
```

A formatter that does not modify the string. Returns fragments unformatted.

### `UppercaseFormatter`

```python
class whoosh.highlight.UppercaseFormatter(between="...")
```

Formats matched terms in uppercase.

### `HtmlFormatter`

```python
class whoosh.highlight.HtmlFormatter(
    tagname="strong",
    between="...",
    classname="match",
    termclass="term",
    maxclasses=5,
    attrquote='"'
)
```

Wraps matched terms in HTML tags with CSS class names. Two classes are
applied to each match: `classname` (same for all matches) and `termclass`
(different for each term, e.g. `term0`, `term1`).

- `tagname`: The HTML tag to wrap matches (default `"strong"`).
- `between`: Text inserted between fragments.
- `classname`: CSS class applied to all matched term tags.
- `termclass`: CSS class prefix for per-term classes.
- `maxclasses`: Maximum number of distinct per-term class numbers.
- `attrquote`: Quote character for attribute values.

**Methods:**
- `clean()`: Clears the internal term-to-classname mapping dictionary.

### `GenshiFormatter`

```python
class whoosh.highlight.GenshiFormatter(qname="strong", between="...")
```

Formats matched terms as Genshi event streams (requires the Genshi library).

## Fragmenters

### `Fragmenter`

```python
class whoosh.highlight.Fragmenter
```

Base class for fragmenters. Subclasses implement `fragment_tokens()` and/or
`fragment_matches()`.

**Methods:**
- `must_retokenize()`: Returns `True` if this fragmenter needs to re-tokenize
  the text (calls `fragment_tokens` with all tokens). Returns `False` if it can
  work from matched token positions alone (calls `fragment_matches`).

### `WholeFragmenter`

```python
class whoosh.highlight.WholeFragmenter(charlimit=DEFAULT_CHARLIMIT)
```

Does not fragment text. Returns the entire text as one fragment. Useful for
highlighting short fields.

```python
results.fragmenter = WholeFragmenter()
```

### `SentenceFragmenter`

```python
class whoosh.highlight.SentenceFragmenter(
    maxchars: int = 200,
    sentencechars=".!?",
    charlimit=DEFAULT_CHARLIMIT
)
```

Breaks text at sentence-ending punctuation (`.`, `!`, `?`).

- `maxchars`: Maximum characters per fragment.
- `sentencechars`: Characters that indicate sentence boundaries.
- `charlimit`: Maximum character position to process.

**Note:** Should be used with an analyzer that does not remove stop words.

### `ContextFragmenter`

```python
class whoosh.highlight.ContextFragmenter(
    maxchars: int = 200,
    surround: int = 20,
    charlimit=DEFAULT_CHARLIMIT
)
```

The default fragmenter. Finds matched terms and includes `surround` characters
of context before and after each match.

- `maxchars`: Maximum characters per fragment.
- `surround`: Number of context characters to include around matches.
- `charlimit`: Maximum character position to process.

### `PinpointFragmenter`

```python
class whoosh.highlight.PinpointFragmenter(
    maxchars: int = 200,
    surround: int = 20,
    autotrim: bool = False,
    charlimit=DEFAULT_CHARLIMIT
)
```

A non-retokenizing fragmenter that builds fragments from character positions of
matched terms. Faster than `ContextFragmenter` because it doesn't need to
re-tokenize text.

- `maxchars`: Maximum characters per fragment.
- `surround`: Number of context characters around matches.
- `autotrim`: If `True`, trims fragments to the nearest spaces.
- `charlimit`: Maximum character position to process.

### `NullFragmeter`

Alias for `WholeFragmenter`.

### `Fragment`

```python
class whoosh.highlight.Fragment(
    text: str,
    matches,
    startchar: int = 0,
    endchar: int = -1
)
```

Represents a fragment (excerpt) from a hit document. Stores the start and end
character offsets and the list of matched term objects.

**Attributes:**
- `text`: The original source text.
- `matches`: List of objects with `startchar` and `endchar` attributes.
- `startchar`: Start index of the fragment.
- `endchar`: End index of the fragment.
- `matched_terms`: Set of text values of matched terms.

**Methods:**
- `overlaps(fragment)`: Returns `True` if this fragment overlaps the given one.
- `overlapped_length(fragment)`: Returns the combined length of overlapping
  fragments.

### `FragmentScorer`

```python
class whoosh.highlight.FragmentScorer
```

Base class for fragment scoring objects. Subclasses implement `__call__()`
to score a `Fragment`.

### `BasicFragmentScorer`

```python
class whoosh.highlight.BasicFragmentScorer
```

Scores fragments by summing the boosts of matched terms, then multiplying by
the number of distinct matched terms (favors diversity).

## Highlighter

### `Highlighter`

```python
class whoosh.highlight.Highlighter(
    fragmenter=None,
    scorer=None,
    formatter=None,
    always_retokenize: bool = False,
    order=SCORE
)
```

Main highlighter object used by `Searcher.highlights()`.

- `fragmenter`: Fragmenter instance (defaults to `ContextFragmenter`).
- `scorer`: Fragment scorer (defaults to `BasicFragmentScorer`).
- `formatter`: Formatter instance (defaults to `HtmlFormatter(tagname="b")`).
- `always_retokenize`: If `True`, always re-tokenize text instead of using
  character offsets from postings.
- `order`: Sort order for fragments.

**Methods:**
- `highlight_hit(hitobj, fieldname, top=3, minscore=1, strict_phrase=False)`:
  Returns the highlighted string for a single hit in a given field.
- `can_load_chars(results, fieldname)`: Returns `True` if the field supports
  "pinpoint" highlighting using stored character offsets.


## DOCUMENT: Idsets

# Idsets API

Specialized set implementations for storing sorted lists of positive
integers (document IDs). These are more memory-efficient than the built-in
`set` for certain use cases, though they are slower for most operations since
they are pure Python.

## Overview

The `DocIdSet` class is the abstract base class. Concrete implementations
include `BitSet`, `OnDiskBitSet`, `SortedIntSet`, `RoaringIdSet`, and
`MultiIdSet`. The `AutoIdSet` function selects the best implementation
based on the contents.

## Module Functions

### `autoset`

```python
whoosh.idsets.autoset
```

A factory that creates an appropriate `DocIdSet` subclass based on the
contents of a given iterable. If all integers in the set are below 10,000,
returns a `BitSet`; otherwise returns a `SortedIntSet`.

## `DocIdSet`

```python
class whoosh.idsets.DocIdSet
```

Abstract base class for set implementations specialized toward storing sorted
lists of positive integers.

**Inheritance:** Inherits from `set`-like interface.

**Methods:**
- `__eq__(other)`: Compares two `DocIdSet` instances by iterating.
- `__len__()`: Returns the number of elements. Override in subclasses.
- `__iter__()`: Yields elements in sorted order. Override in subclasses.
- `__contains__(i)`: Returns `True` if `i` is in the set.
- `__or__(other)`: Returns `self.union(other)`.
- `__and__(other)`: Returns `self.intersection(other)`.
- `__sub__(other)`: Returns `self.difference(other)`.
- `copy()`: Returns a copy of this set.
- `add(n)`: Adds `n` to the set.
- `discard(n)`: Removes `n` from the set (no error if absent).
- `update(other)`: Adds all elements from `other`.
- `intersection_update(other)`: Removes elements not in `other`.
- `difference_update(other)`: Removes all elements in `other`.
- `invert_update(size)`: In-place inversion over the range `[0, size)`.
- `intersection(other)`: Returns a new set with elements in both.
- `union(other)`: Returns a new set with elements from both.
- `difference(other)`: Returns a new set with elements in self but not other.
- `invert(size)`: Returns a new set that is the inversion over `[0, size)`.
- `isdisjoint(other)`: Returns `True` if no elements are shared.
- `before(i)`: Returns the previous integer in the set before `i`, or `None`.
- `after(i)`: Returns the next integer in the set after `i`, or `None`.
- `first()`: Returns the first (lowest) integer.
- `last()`: Returns the last (highest) integer.

## `BaseBitSet`

```python
class whoosh.idsets.BaseBitSet(DocIdSet)
```

Base class for bitmap-backed `DocIdSet` implementations. Uses a bytes-based
bitmap where each bit represents membership of an integer.

**Abstract Methods to Override:**
- `byte_count()`: Returns the number of bytes in the bitmap.
- `_get_byte(i)`: Returns the byte at index `i`.
- `_iter_bytes()`: Yields all bytes in the bitmap.

**Inherited Methods:** All `DocIdSet` methods with efficient bitmap
implementations of `__len__`, `__iter__`, `__contains__`, `first`, and
`last`.

## `OnDiskBitSet`

```python
class whoosh.idsets.OnDiskBitSet(file, doc_count)
```

A `BaseBitSet` that reads the bitmap from a file on disk, using `mmap` for
memory efficiency.

**Constructor:**
- `file`: A file-like object (opened in binary mode) containing the bitmap.
- `doc_count`: Total number of documents (bits) represented.

```python
from whoosh.idsets import OnDiskBitSet

with open("deletions.dat", "rb") as f:
    bs = OnDiskBitSet(f, doc_count=10000)
    if 42 in bs:
        print("Document 42 is deleted")
```

## `BitSet`

```python
class whoosh.idsets.BitSet
```

A `BaseBitSet` that stores the bitmap in memory as a `bytearray`. Fast for
membership tests and set operations on small ranges of integers.

**Constructor:**
- Optional initial iterable of integers.

```python
from whoosh.idsets import BitSet

bs = BitSet([0, 5, 10, 15])
print(5 in bs)  # True
print(bs.first())  # 0
print(len(bs))   # 4
```

**Methods:**
- `from_blob(data)`: Create a `BitSet` from raw bytes.
- `tostring()`: Returns the bitmap as a `bytes` string.
- `set_reverse()`: Prepares the set for reverse iteration.

## `SortedIntSet`

```python
class whoosh.idsets.SortedIntSet
```

A `DocIdSet` that stores integers as a sorted list of Python `int` objects.
More memory-efficient than `BitSet` for sparse sets but slower for membership
tests.

**Constructor:**
- Optional initial iterable of integers.

```python
from whoosh.idsets import SortedIntSet

sis = SortedIntSet([100, 500, 999])
print(500 in sis)  # True
print(sis.after(200))  # 500
```

## `ReverseIdSet`

```python
class whoosh.idsets.ReverseIdSet(child)
```

Wraps another `DocIdSet` to reverse the interpretation of integers. Instead
of representing membership directly, the set represents the *complement* of
the inner set. Useful for representing deleted documents.

**Constructor:**
- `child`: The `DocIdSet` to reverse.

**Example:** If `child` represents documents `{3, 7, 9}`, then
`ReverseIdSet(child)` represents all documents *except* `{3, 7, 9}`.

## `RoaringIdSet`

```python
class whoosh.idsets.RoaringIdSet
```

A `DocIdSet` that partitions integers into 16-bit buckets and uses `BitSet`
within each bucket. More memory-efficient than a single flat `BitSet` for
large, sparse sets of integers.

**Constructor:**
- Optional initial iterable of integers.

**Methods:**
- `from_bytes(data)`: Deserialize from bytes.
- `to_bytes()`: Serialize to bytes.
- `to_bytes_list()`: Returns a list of `(bucket, bytes)` pairs.

## `MultiIdSet`

```python
class whoosh.idsets.MultiIdSet(readers, offsets=None)
```

Combines multiple `DocIdSet` instances into one, handling document ID offsets
automatically. Used for combining deletions across multiple segments.

**Constructor:**
- `readers`: List of `DocIdSet` instances (one per segment).
- `offsets`: Optional list of base docnum offsets for each reader. If
  omitted, offsets are computed automatically.

**Methods:**
- `__contains__(i)`: Checks the appropriate sub-set based on offsets.
- `__iter__()`: Iterates over all integers in all sub-sets.
- `__len__()`: Returns the total count across all sub-sets.


## DOCUMENT: Lang

# Language Support API

Language detection helpers, stemmer selection, stop-word lists, and
language-specific modules (Snowball stemmers, ISRI stemmer, Soundex,
Double Metaphone, etc.).

## Module Overview

The `whoosh.lang` package provides functions for detecting and selecting
language-specific resources (stemmers, stop words) and submodules containing
stemmers for various languages.

## Supported Languages

```python
whoosh.lang.languages = ("ar", "da", "nl", "en", "fi", "fr", "de", "hu",
                         "it", "no", "pt", "ro", "ru", "es", "sv", "tr")
```

Two-letter ISO 639-1 language codes for which stemmers or stop-word lists
are available.

## Language Aliases

```python
whoosh.lang.aliases = { ... }
```

A dictionary mapping alternate language identifiers to their canonical
two-letter codes. Includes ISO 639-3 three-letter codes, English names,
and native-language names.

## Exceptions

### `NoStemmer`

```python
class whoosh.lang.NoStemmer
```

Raised by `stemmer_for_language()` when no stemmer is available for the
given language.

### `NoStopWords`

```python
class whoosh.lang.NoStopWords
```

Raised by `stopwords_for_language()` when no stop-word list is available for
the given language.

## Language Functions

### `two_letter_code`

```python
whoosh.lang.two_letter_code(name) -> str or None
```

Converts a language identifier to its canonical two-letter code. Accepts
two-letter codes, ISO 639-3 codes, English names, and native-language names.

```python
from whoosh.lang import two_letter_code

code = two_letter_code("french")   # 'fr'
code = two_letter_code("deutsch")  # 'de'
code = two_letter_code("español")  # 'es'
```

### `has_stemmer`

```python
whoosh.lang.has_stemmer(lang) -> bool
```

Returns `True` if a stemmer is available for the given language.

### `has_stopwords`

```python
whoosh.lang.has_stopwords(lang) -> bool
```

Returns `True` if a stop-word list is available for the given language.

### `stemmer_for_language`

```python
whoosh.lang.stemmer_for_language(lang) -> callable
```

Returns a stemmer function for the given language. Raises `NoStemmer` if
no stemmer is available.

**Supported languages and stemmers:**
- `"en"` / `"en_porter"`: Original Porter stemmer (`whoosh.lang.porter`)
- `"ar"`: ISRI Arabic stemmer (`whoosh.lang.isri`)
- `"da"`: Danish Snowball stemmer
- `"nl"`: Dutch Snowball stemmer
- `"en"`: English Snowball stemmer
- `"fi"`: Finnish Snowball stemmer
- `"fr"`: French Snowball stemmer
- `"de"`: German Snowball stemmer
- `"hu"`: Hungarian Snowball stemmer
- `"it"`: Italian Snowball stemmer
- `"no"`: Norwegian Snowball stemmer
- `"pt"`: Portuguese Snowball stemmer
- `"ro"`: (no stemmer currently)
- `"ru"`: Russian Snowball stemmer
- `"es"`: Spanish Snowball stemmer
- `"sv"`: Swedish Snowball stemmer
- `"tr"`: (no stemmer currently)

```python
from whoosh.lang import stemmer_for_language

stem = stemmer_for_language("en")
print(stem("running"))  # 'run'
```

### `stopwords_for_language`

```python
whoosh.lang.stopwords_for_language(lang) -> list
```

Returns the stop-word list for the given language. Raises `NoStopWords` if
no stop-word list is available.

```python
from whoosh.lang import stopwords_for_language

stops = stopwords_for_language("en")
```

## Snowball Stemmers

The `whoosh.lang.snowball` subpackage contains stemmers implementing the
Snowball stemming algorithms for various languages.

### Available Stemmers

| Module | Class | Language |
|--------|-------|----------|
| `snowball.english` | `EnglishStemmer` | English |
| `snowball.dutch` | `DutchStemmer` | Dutch |
| `snowball.finnish` | `FinnishStemmer` | Finnish |
| `snowball.french` | `FrenchStemmer` | French |
| `snowball.german` | `GermanStemmer` | German |
| `snowball.hungarian` | `HungarianStemmer` | Hungarian |
| `snowball.italian` | `ItalianStemmer` | Italian |
| `snowball.norwegian` | `NorwegianStemmer` | Norwegian |
| `snowball.portugese` | `PortugueseStemmer` | Portuguese |
| `snowball.russian` | `RussianStemmer` | Russian |
| `snowball.romanian` | `RomanianStemmer` | Romanian |
| `snowball.spanish` | `SpanishStemmer` | Spanish |
| `snowball.swedish` | `SwedishStemmer` | Swedish |
| `snowball.danish` | `DanishStemmer` | Danish |

### Base Classes

```python
class whoosh.lang.snowball.bases._ScandinavianStemmer
class whoosh.lang.snowball.bases._StandardStemmer
```

Internal base classes for Snowball stemmers. User code should use the
language-specific stemmer classes directly.

### `classes`

```python
whoosh.lang.snowball.classes = {"da": DanishStemmer, "nl": DutchStemmer, ...}
```

Dictionary mapping two-letter language codes to Snowball stemmer classes.

## Porter Stemmer

### `whoosh.lang.porter`

The original Porter stemming algorithm, faster but less accurate than
Snowball English stemmer.

#### `stem`

```python
whoosh.lang.porter.stem(w) -> str
```

Stems a single English word using the Porter algorithm.

## ISRI Stemmer

### `whoosh.lang.isri.ISRIStemmer`

```python
class whoosh.lang.isri.ISRIStemmer
```

Arabic stemmer based on the Information Science Research Institute (ISRI)
algorithm. Does not use a root dictionary.

#### `stem`

```python
def ISRIStemmer.stem(word) -> str
```

Stems an Arabic word.

## Double Metaphone

### `whoosh.lang.dmetaphone.double_metaphone`

```python
whoosh.lang.dmetaphone.double_metaphone(text) -> tuple
```

Returns a tuple of `(primary, secondary)` metaphone codes for the given
text, using the Double Metaphone algorithm.

## Soundex

### `whoosh.lang.phonetic`

Soundex implementations for phonetic matching.

#### `soundex_en`

```python
whoosh.lang.phonetic.soundex_en(word) -> str
```

English Soundex encoding.

#### `soundex_esp`

```python
whoosh.lang.phonetic.soundex_esp(word) -> str
```

Spanish Soundex encoding.

#### `soundex_ar`

```python
whoosh.lang.phonetic.soundex_ar(word) -> str
```

Arabic Soundex encoding.

## WordNet Thesaurus

### `whoosh.lang.wordnet.Thesaurus`

```python
class whoosh.lang.wordnet.Thesaurus
```

Provides synonym expansion based on WordNet-style data.

**Methods:**
- `synonyms(word)`: Returns the set of synonyms for `word`.
- `__contains__(word)`: Returns `True` if `word` is in the thesaurus.

### Functions

```python
whoosh.lang.wordnet.parse_file(f) -> dict
whoosh.lang.wordnet.make_index(storage, indexname, word2nums, num2words)
whoosh.lang.wordnet.synonyms(word2nums, num2words, word) -> set
```

## Lovins Stemmer

### `whoosh.lang.lovins`

A suffix-stripping stemmer by Lovins. Functions include:
- `stem(word)`: Main stemming function.
- `remove_ending(word)`: Removes suffixes.
- `fix_ending(word)`: Fixes the word ending after stemming.

## Paice-Husk Stemmer

### `whoosh.lang.paicehusk.PaiceHuskStemmer`

```python
class whoosh.lang.paicehusk.PaiceHuskStemmer(rules)
```

A rule-based stemmer using Paice-Husk rules.

#### `stem`

```python
def PaiceHuskStemmer.stem(word) -> str
```

Stems a word using the Paice-Husk algorithm.

**Usage note:** The module also exposes a pre-configured stemmer:
```python
whoosh.lang.paicehusk.stem = PaiceHuskStemmer(defaultrules).stem
```


## DOCUMENT: Matching

# Matching API

Classes and functions for iterating over and combining result sets during
searching. The matching module is a refactored package exposing the same
public API as the former monolithic module.

## Overview

When you search an index, Whoosh creates `Matcher` objects representing the
postings (document IDs and scores) produced by query objects. Matchers can
be combined (e.g., union, intersection) to build compound queries. The
matching module provides the core `Matcher` class hierarchy, utility
functions, and concrete implementations for various query types.

## Core Matcher Classes

### `Matcher`

```python
class whoosh.matching.Matcher
```

Abstract base class for all matchers. Concrete subclasses implement
`__init__()` and the `_set()` and `_maybe_values()` methods.

**Methods:**

#### `init = property(is_active)`

Property that returns whether the matcher is "active" (at top of segment
postings, not exhausted).

#### `init(view, docnum, score)`

Called when the matcher is initialized.

#### `set(matcher)`

Replaces this matcher with another one.

#### `copy()`

Returns a copy of this matcher.

#### `all_ids()`

Returns a list of docnums matched by this matcher.

#### `matches(matcher)`

Returns `True` if any of the current matches in `self` also match in
`matcher`.

#### `skip_to(docid)`

Advances the matcher to the first match at or after `docid`.

#### `skip_to_intersect(matcher)`

Moves this matcher to the earliest matching docnum that is also matched in
`matcher`.

#### `next()`

Advances the matcher to the next match.

#### `next_in_segment()`

Advances to the next match in the current segment.

#### `next_segment(matcher)`

Advances to the next segment in the context of `matcher`.

#### `is_active(in_segment=False)`

Returns `True` if this matcher has more matches to process.

#### `all_matching_segments()`

Generates `(segment_num, matcher)` pairs for all matching segments.

#### `doc()`

Returns the current document number of this matcher. May advance to next
document if not already on one.

#### `docnum()`

Returns the current docnum (segment-relative) of the matcher.

#### `score()`

Returns the current match's score.

#### `value()`

Returns the current match's value (e.g., the decoded stored value of the
term).

#### `supports()`

Returns `True` if `value()` is supported.

#### `value_matches()`

Returns the value at the current match.

#### `all_values()`

Returns a list of all values in this matcher.

#### `supports_lee()`

Returns `True` if the matcher uses lazy evaluation.

#### `lee`

Returns the current "lazy evaluation extension" value (for term vectors).

#### `spans()`

If the postings include positions, returns a list of `Position` objects for
the current match.

#### `spans()`

Returns the spans (positions) of the match in the current document.

#### `next_type()`

Returns the type of the next match.

#### `copy()`

Returns a shallow copy of this matcher.

### `Child`

```python
class whoosh.matching.Child
```

Mixin class for matchers that wrap other matchers.

### `FilterMixin`

```python
class whoosh.matching.FilterMixin
```

Mixin for matchers used as filters (boolean scoring, no relevance).

### `Custom`

```python
class whoosh.matching.Custom
```

Mixin for matchers that return a custom score from `score()` rather than 1.

### `Constant`

```python
class whoosh.matching.Constant
```

Mixin for matchers whose score is always the same value.

### `Coord`

```python
class whoosh.matching.Coord
```

Mixin for matchers that compute coordination factor (for phrase and other
queries that benefit from it).

## Concrete Matcher Classes

### `ListUnion`

```python
class whoosh.matching.ListUnion(matcher, items, maptype=None)
```

Base class for matchers that combine multiple matchers with a list of keys.

#### `filter`

```python
class whoosh.matching.filter
```

Decorator for creating filter matchers (boolean matchers with no relevance).

### `Union`

```python
class whoosh.matching.Union(matcher, items)
```

Base class for the `OR` operator.

### `Intersection`

```python
class whoosh.matching.Intersection(matcher, items)
```

The `AND` operator. A document matches only if it appears in all the child
matchers.

#### `IntersectionFilter`

```python
class whoosh.matching.IntersectionFilter(matcher, items)
```

A filter (no scoring) version of intersection.

### `And`

```python
class whoosh.matching.And(matcher, items)
```

Alias for `Intersection`.

### `Or`

```python
class whoosh.matching.Or(matcher, items)
```

Alias for `Union`.

### `Not`

```python
class whoosh.matching.Not(matcher, a, b)
```

The `NOT` operator. Matches all documents in `a` that are not in `b`.

### `Require`

```python
class whoosh.matching.Require(matcher, a, b)
```

Matches documents in `a` only if they also appear in `b`, but does not add
`b`'s score.

### `AndNot`

```python
class whoosh.matching.AndNot(matcher, a, b)
```

Matches documents in `a` that are not in `b`.

#### `AndMaybe`

```python
class whoosh.matching.AndMaybe(matcher, a, b)
```

Matches documents in `a`, adding `b`'s score if present.

### `BinaryUnion`

```python
class whoosh.matching.BinaryUnion(items)
```

Efficient intersection of exactly two matchers.

#### `BinaryUnion2`

```python
class whoosh.matching.BinaryUnion2
```

Optimized binary union for two items.

### `TreeMatcher`

```python
class whoosh.matching.TreeMatcher
```

A matcher that wraps a `Tree` object for combining results.

### `NestedParent`

```python
class whoosh.matching.NestedParent(parent, child, bools=False)
```

Matches parent documents that have at least one child document matched by
the child matcher. Used for nested document queries.

### `NestedChildren`

```python
class who which.matching.NestedChildren(parentmatch, child)
```

Matches child documents for a given parent document.

### `LengthMatcher`

```python
class whoosh.matching.LengthMatcher(child, q, polarity=False)
```

Matches documents based on field length (used by `Every` query).

### `Filter`

```python
class whoosh.matching.Filter(matcher)
```

Converts any matcher into a filter (no scoring).

### `AlwaysFilter`

```python
class whoosh.matching.AlwaysFilter
```

A filter that matches all documents.

### `NeverFilter`

```python
class whoosh.matching.NeverFilter
```

A filter that matches no documents.

### `PseudoMatcher`

```python
class whoosh.matching.PseudoMatcher
```

Base class for pseudo-matchers used in span queries.

## Matching Utilities

### `current_spans`

```python
whoosh.matching.current_spans(matcher) -> list
```

Returns a list of `Span` objects for the current match in `matcher`, or an
empty list if the matcher doesn't support positions.

### `disjunction_score`

```python
whoosh.matching.disjunction_score(matcher) -> float
```

Returns the sum of `matcher.score()` and the scores of all child matchers of
type `Union`.

### `intersection_score`

```python
whoosh.matching.intersection_score(matcher) -> float
```

Returns the sum of `matcher.score()` and all child matchers of type
`Intersection`.

### `child_count`

```python
whoosh.matching.child_count(matcher) -> int
```

Returns the number of child matchers in `matcher`.

### `has_quality`

```python
whoosh.matching.has_quality(matcher) -> bool
```

Returns `True` if `matcher` has a `query` attribute (i.e., is a
`QueryMatcher`-derived object, or a combination of such matchers).

### `has_untranslated`

```python
whoosh.matching.has_untranslated(matcher) -> bool
```

Returns `True` if the matcher has an `untranslated` attribute (set by
certain wrapper matchers like `TimeLimited`).

### `wrap`

```python
whoosh.matching.wrap(matcher)
```

Returns `matcher` if it has a `.copy()` method, otherwise wraps it in an
`AutoMatcher`.

### `wrap2`

```python
whoosh.matching.wrap2(a, b, m)
```

Returns either a `BinaryUnion2` or an `AutoMatcher` depending on whether `a`
and `b` are list-compatible.

### `unified`

```python
whoosh.matching.unified(matcher)
```

Returns `matcher` if it has an `untranslated` attribute, otherwise returns
`None`.

### `deletion`

```python
whoosh.matching.deletion(matcher)
```

If `matcher` has a `parent` attribute, returns the parent, otherwise returns
`None`.

### `AutoMatcher`

```python
class whoosh.matching.AutoMatcher(m, **kwargs)
```

A general-purpose matcher that wraps arbitrary objects and adds default
behavior for scoring, docnums, and other features. Created by `wrap()`.

### `MatchingTimeLimit`

```python
class whoosh.matching.MatchingTimeLimit
```

A lightweight exception raised when a query matcher exceeds a time limit.

### `TimeLimited`

```python
class whoosh.matching.TimeLimited(child, maxsteps=100, timeout=None, currenttime=None)
```

Wrapper that wraps a `Matcher` to enforce a time limit. Raises
`MatchingTimeLimit` if the time limit is exceeded.

**Parameters:**
- `child`: The matcher to wrap.
- `maxsteps`: Check time every N documents (default `100`).
- `timeout`: Maximum time in seconds (default `None`, no limit).
- `currenttime`: Optional function to use for getting the current time.

### `TermMatcher`

```python
class whoosh.matching.TermMatcher(postings, text, qname, scorer=None, boost=1.0)
```

Matches documents containing a specific term.

**Constructor:**
- `postings`: A `Postings` object from the index reader.
- `text`: The term text.
- `qname`: The query name for this term.
- `scorer`: Optional `Scorer` object.
- `boost`: Boost factor for this term's score.

### `MultiScorer`

```python
class whoosh.matching.MultiScorer(numgroups, start_i=0)
```

A `Scorer` that combines the scores from multiple scorers into one, weighted
across groups of segments.

### `RangeMatcher`

```python
class whoosh.matching.RangeMatcher(start_matcher, end_matcher, query)
```

Matches documents within a range of term values.

### `RegexMatcher`

```python
class whoosh.matching.RegexMatcher(regex, qname, boost=1.0)
```

Matches documents whose terms match a compiled regex.

### `SpanMatcher`

```python
class whoosh.matching.SpanMatcher(matcher, order=0, end=0)
```

Matches spans (positions) within documents.

### `SpanOverlap`

```python
class whoosh.matching.SpanOverlap(l, r)
```

Matches overlapping spans from two matchers.

### `SpanNear`

```python
class whoosh.matching.SpanNear(l, r, slop=1, ordered=True)
```

Matches spans that are near each other within a document.

### `SpanCondition`

```python
class whoosh.matching.SpanCondition(l, r)
```

Matches a condition on spans.

### `SpanBefore`

```python
class whoosh.matching.SpanBefore(l, r, end=0)
```

Matches spans before a given position.

### `SpanAfter`

```python
class whoosh.matching.SpanAfter(l, r, end=0)
```

Matches spans after a given position.

### `SpanOutside`

```python
class whoosh.matching.SpanOutside(l, r, end=0)
```

Matches spans outside a given range.

### `SpanFirst`

```python
class whoosh.matching.SpanFirst(l, start=0, end=1)
```

Matches spans at the beginning of a document.

### `SpanNot`

```python
class whoosh.matching.SpanNot(l, r)
```

Matches spans in `l` that are not in `r`.

### `SpanOr`

```python
class whoosh.matching.SpanOr(items)
```

Logical OR for span matchers.

### `SpanAnd`

```python
class whoosh.matching.SpanAnd(l, r)
```

Logical AND for span matchers.


## DOCUMENT: Middleware

# Middleware API

Reference for the middleware pipeline.

## MiddlewareContext

```python
class whoosh.middleware.context.MiddlewareContext
```

Context object passed through all middleware hooks.

### Attributes

| Attribute | Type | Description |
|-----------|------|-------------|
| `operation` | `str` | Operation type: `index`, `search`, `delete`, `commit` |
| `index` | `Any` | The index object |
| `backend` | `Any` | The backend storage object |
| `writer` | `Any` | The index writer |
| `searcher` | `Any` | The searcher |
| `document` | `dict \| None` | Document being indexed/deleted |
| `query` | `str` | Search query string |
| `collector` | `Any` | Collector instance |
| `results` | `Any` | Search results |
| `labels` | `dict` | Middleware identification labels |
| `metadata` | `dict` | Arbitrary middleware communication data |

### Methods

#### `copy()`

```python
ctx_copy = context.copy()
```

Create a shallow copy.

---

## Middleware

```python
class whoosh.middleware.base.Middleware
```

Base class for all middleware.

### Methods

#### `startup()`

```python
def startup(self, context: MiddlewareContext) -> None:
    """Called once on initialization."""
```

#### `shutdown()`

```python
def shutdown(self, context: MiddlewareContext) -> None:
    """Called once on teardown."""
```

#### `before_index()`

```python
def before_index(self, context: MiddlewareContext) -> MiddlewareContext:
    """Called before indexing a document."""
```

#### `after_index()`

```python
def after_index(self, context: MiddlewareContext) -> MiddlewareContext:
    """Called after indexing a document."""
```

#### `before_delete()`

```python
def before_delete(self, context: MiddlewareContext) -> MiddlewareContext:
    """Called before deleting a document."""
```

#### `after_delete()`

```python
def after_delete(self, context: MiddlewareContext) -> MiddlewareContext:
    """Called after deleting a document."""
```

#### `before_search()`

```python
def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
    """Called before executing a search."""
```

#### `after_search()`

```python
def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
    """Called after search results are returned."""
```

#### `on_error()`

```python
def on_error(self, context: MiddlewareContext, exc: Exception) -> None:
    """Called on exception. Re-raise by default."""
```

#### `on_commit()`

```python
def on_commit(self, context: MiddlewareContext) -> None:
    """Called after commit."""
```

---

## MiddlewareChain

```python
class whoosh.middleware.chain.MiddlewareChain
```

Orchestrates ordered middleware execution.

### Methods

#### `add()`

```python
chain.add(middleware: Middleware)
```

Add a middleware.

---

#### `extend()`

```python
chain.extend(middlewares: list[Middleware])
```

Add multiple middlewares.

---

#### `run_before()`

```python
context = chain.run_before(
    hook_name: str,
    context: MiddlewareContext,
    fail_open: bool = False
)
```

Run before hooks in order.

---

#### `run_after()`

```python
context = chain.run_after(
    hook_name: str,
    context: MiddlewareContext,
    fail_open: bool = False
)
```

Run after hooks in reverse.

---

#### `run_on_error()`

```python
chain.run_on_error(context, exc, fail_open=False)
```

Call on_error hooks.

---

## Built-in Middleware Classes

### MetricsMiddleware

```python
class whoosh.middleware.base.MetricsMiddleware
```

Tracks indexing/search counts.

#### Methods

##### `after_index()`

Increment documents indexed count.

##### `after_search()`

Increment searches executed count.

##### `get_metrics()`

```python
metrics = metrics_mw.get_metrics() -> dict
```

Return collected metrics.

---

### CacheMiddleware

```python
class whoosh.middleware.base.CacheMiddleware
```

In-memory search cache.

#### Methods

##### `before_search()`

Check cache for query.

##### `after_search()`

Store results in cache.

##### `get_cached()`

```python
cached = cache_mw.get_cached(query: str) -> Any
```

##### `set_cached()`

```python
cache_mw.set_cached(query: str, results: Any)
```

---

### CompressionMiddleware

```python
class whoosh.middleware.base.CompressionMiddleware
```

Mark documents for compression at backend level.

---

### EncryptionMiddleware

```python
class whoosh.middleware.base.EncryptionMiddleware
```

Mark documents for encryption at backend level.

---

## Exceptions

### StopOperation

```python
class whoosh.middleware.exceptions.StopOperation(Exception)
```

Raise to abort an operation.

---

## Integration Helpers

### `apply_middleware_to_writer()`

```python
def apply_middleware_to_writer(
    writer: IndexWriter,
    middleware: list[Middleware] = None
) -> MiddlewareWriter
```

---

### `apply_middleware_to_searcher()`

```python
def apply_middleware_to_searcher(
    searcher: Searcher,
    middleware: list[Middleware] = None
) -> MiddlewareSearcher
```


## DOCUMENT: Modern

# Modern API

The `whoosh_modern` package provides the modern, fully-typed surface of
Whoosh-NG. It includes data sources, schema discovery, validation,
middleware, profiling, autocomplete, vector search integration, and an
optimized batch writer.

## Data Sources

```python
from whoosh_modern.data_sources import (
    DataSource,
    SQLSource,
    RESTSource,
    FastCSVSource,
    JSONSource,
    GraphQLSource,
    PydanticSource,
    PandasSource,
    PolarsSource,
    ParquetSource,
    PeeweeSource,
    TortoiseSource,
    SQLAlchemySource,
    ObservableDataSource,
    DataSourceConfig,
)
```

### DataSource Protocol

The `DataSource` protocol defines the interface for all data source
implementations:

```python
class DataSource(Protocol):
    @property
    def name(self) -> str
    def discover_schema(self) -> Schema
    def iter_documents(self) -> Iterator[Document]
    def stream_batches(self, batch_size=1000) -> Iterator[list[dict]]
    def health_check(self) -> bool
```

Additional capability protocols are available:
- `IncrementalDataSource` — supports `iter_changes(since)` for incremental sync
- `AsyncDataSource` — supports `aiter_documents()` for async iteration
- `RefreshableDataSource` — supports `refresh()`
- `CountableDataSource` — supports `document_count()`
- `MetadataDataSource` — supports `metadata()`
- `ObservableDataSource` — supports `add_observer()`/`remove_observer()`

### SQLSource

```python
from whoosh_modern.data_sources import SQLSource

source = SQLSource(
    connection=conn,
    query="SELECT * FROM articles WHERE status='published'",
    incremental_field="updated_at",
    id_field="id",
    pool_size=5,
)
schema = source.discover_schema()
docs = list(source.iter_documents())
batches = list(source.stream_batches(batch_size=1000))
count = source.document_count()
meta = source.metadata()
```

### RESTSource

```python
from whoosh_modern.data_sources import RESTSource

source = RESTSource(
    url="https://api.example.com/v2/products",
    method="GET",
    pagination="page",  # or "offset", "cursor"
    page_size=50,
    headers={"Authorization": "Bearer token"},
    auth={"type": "bearer", "token": "..."},
    document_path="results",  # Extract from nested response
)
schema = source.discover_schema()
docs = list(source.iter_documents())
```

### DataSourceConfig

Declarative configuration with a factory:

```python
from whoosh_modern.data_sources import DataSourceConfig

config = DataSourceConfig(
    type="sql",
    connection=conn,
    query="SELECT * FROM articles",
    sequential_field="updated_at",
)
source = config.create()  # Returns a configured SQLSource
```

Supported types: `sql`, `sqlalchemy`, `rest`, `csv`, `json`, `graphql`,
`pydantic`, `pandas`, `polars`, `parquet`, `peewee`, `tortoise`.

## Schema Discovery

```python
from whoosh_modern.schema_discovery import SchemaDiscovery

# From column metadata (list of (name, sql_type) tuples)
columns = [("id", "INTEGER"), ("title", "TEXT"), ("published", "TIMESTAMP")]
schema = SchemaDiscovery.from_result_set(columns)

# From sample documents (auto-detects types)
schema = SchemaDiscovery.from_sample(docs)

# Optimized variant (drops non-searchable TEXT, infers IDs/booleans)
schema = SchemaDiscovery.from_sample_optimized(docs, searchable_text=["title", "content"])

# Detect ID field from schema
id_field = SchemaDiscovery.detect_id_field(dict(schema))
```

### SQL Type Mapping

`SchemaDiscovery` includes a built-in SQL type map: `VARCHAR`→`TEXT`,
`INTEGER`→`NUMERIC`, `BOOLEAN`→`BOOLEAN`, `TIMESTAMP`→`DATETIME`, `JSON`→
`KEYWORD`, `UUID`→`ID`, etc.

## FacetManager

```python
from whoosh_modern.facets import FacetManager, TermsFacet, RangeFacet, DateRangeFacet

manager = FacetManager(schema)
# Or with manual config:
manager = FacetManager(schema, config={"price": {"type": "range", "buckets": [...]}})

facets = manager.get_facets()          # Auto-discovered + manual facets
config = manager.get_facet_config("category")
stats = manager.get_facet_stats()
manager.set_manual_override("price", {"type": "range", "buckets": ["0-100", "100-500"]})
```

Auto-discovery rules:
- `TEXT`, `KEYWORD`, `BOOLEAN`, `ID` → `TermsFacet`
- `NUMERIC` → `RangeFacet`
- `DATETIME` → `DateRangeFacet`

## Validation Framework

```python
from whoosh_modern.validation import ValidationFramework, ValidationResult

validator = ValidationFramework()
results = validator.validate(source)

for result in results:
    print(f"Level {result.level}: passed={result.passed}")
    for warning in result.warnings:
        print(f"  Warning: {warning}")
    for error in result.errors:
        print(f"  Error: {error}")
```

Four validation levels:

| Level | Method | Purpose |
|-------|--------|---------|
| 1 | `validate_structural()` | DataSource availability, schema detection |
| 2 | `validate_search()` | Indexable fields, term vectors, searchable analyzers |
| 3 | `validate_performance()` | Performance warnings (e.g., TEXT fields on large datasets) |
| 4 | `validate_runtime()` | Sample iteration, type conformance |

## Middleware Pipeline

```python
from whoosh_modern.middleware import (
    Middleware,
    MiddlewarePipeline,
    RetryMiddleware,
    LoggingMiddleware,
    CacheMiddleware,
)

pipeline = MiddlewarePipeline(
    RetryMiddleware(attempts=3, backoff="exponential", jitter=True),
    LoggingMiddleware(level=logging.INFO),
    CacheMiddleware(maxsize=128),
)

def my_operation():
    return searcher.search(query)

result = pipeline.execute(my_operation)
```

### Middleware Types

- **`Middleware`**: The core base class (`whoosh.middleware.base.Middleware`, re-exported from `whoosh_modern.middleware`). Subclass it and implement the lifecycle hooks (`before_index`, `after_index`, `before_search`, `after_search`, `on_error`, `on_commit`). `RetryMiddleware`, `LoggingMiddleware`, and `CacheMiddleware` additionally keep a `wrap(operation)` helper for decorating callables.
- **`RetryMiddleware`**: Retries failed operations with exponential or linear
  backoff, with optional jitter.
- **`LoggingMiddleware`**: Logs execution time and errors.
- **`CacheMiddleware`**: Caches results keyed by operation name and arguments.
  Exposes `stats` property and `clear()` method.

## SearchView

```python
from whoosh_modern.views import SearchView

view = SearchView(
    name="articles",
    source=source,
    fields={"title": fields.TEXT(stored=True)},  # Field type overrides
    facets={"category": {"type": "terms", "limit": 50}},
    incremental_field="updated_at",
    strict=False,  # Raise on validation failures
    middleware=[LoggingMiddleware()],
    schema_version="1.0",
)

ix = view.build("indexdir")       # Create/populate index
count = view.reindex()            # Full reindex
count = view.refresh()            # Incremental refresh
results = view.validate()         # Run validation
view.evolve_schema({"new_field": fields.TEXT})  # Add fields without reindexing
```

## Optimized Writer

```python
from whoosh_modern.writer import ModernIndex

# Create or open an optimized index
index = ModernIndex.create("indexdir", schema=my_schema)
# Or open existing:
# index = ModernIndex.open("indexdir")

# Optimized writer for batch processing millions of docs
with index.writer(batch_size=5000, limitmb=512, multisegment=True) as writer:
    for batch in source.stream_batches(batch_size=5000):
        writer.add_batch(batch)
        # Or: writer.add_batches(source.stream_batches(batch_size=5000))

print(f"Documents indexed: {writer.doc_count}")

# Access searcher
with index.searcher() as searcher:
    results = searcher.search(query)
```

### Key Optimizations

- **Multisegment mode**: No merging during indexing (set with
  `multisegment=True`)
- **Reduced Python overhead**: Batch-oriented add API
- **Configurable memory limits**: `limitmb` parameter controls buffering

## Analysis Extensions

```python
from whoosh_modern.analysis import StemmingAnalyzer, get_stemmer, register_stemmer
```

### StemmingAnalyzer

Enhanced analyzer with pluggable stemmer backends:

```python
# Auto-detect best available stemmer
analyzer = StemmingAnalyzer(stemmer="auto")

# Explicit internal stemmer
analyzer = StemmingAnalyzer(stemmer="internal")

# PyStemmer (requires pip install whoosh-ng[fast-stemming])
analyzer = StemmingAnalyzer(stemmer="pystemmer")

# Custom stemmer provider
analyzer = StemmingAnalyzer(stemmer=my_custom_stemmer)

# Full parameters
analyzer = StemmingAnalyzer(
    expression=r"\S+",
    stoplist=None,
    minsize=2,
    maxsize=None,
    gaps=False,
    stemmer="auto",
    ignore=None,
    cachesize=50000,
)
```

### Stemmer Providers

```python
from whoosh_modern.analysis import get_stemmer, list_available_backends

# Get a stemmer provider
stemmer = get_stemmer("auto", "english")
stemmed = stemmer.stem("running")  # "run"

# List available backends
backends = list_available_backends()
# {"internal": "available", "pystemmer": "not installed"}

# Register a custom stemmer
@register_stemmer("my_stemmer")
class MyStemmer:
    def stem(self, word: str) -> str:
        return word.lower()

# Priority: PyStemmer > Internal (whoosh.lang)
```

### StemmerProvider Protocol

```python
class StemmerProvider(Protocol):
    def stem(self, word: str) -> str
    @property
    def name(self) -> str
    @property
    def language(self) -> str
```

Available providers:
- `InternalStemmerProvider` — wraps Whoosh's built-in `whoosh.lang.porter.stem`
- `PyStemmerProvider` — wraps the PyStemmer library (fastest)
- `IdentityStemmerProvider` — no-op stemmer for testing

## Autocomplete

```python
from whoosh_modern.autocomplete import create_autocomplete

# Create an autocomplete provider
provider = create_autocomplete("inverted")

# Add phrases
provider.add(["hello world", "hello there", "goodbye world"])

# Search
hits = provider.search("hello", limit=10)
for hit in hits:
    print(hit.text, hit.score)
```

### Classes

- **`AutocompleteHit`**: Simple data class with `text` and `score` attributes.
- **`AutocompleteProvider` (Plugin)**: Abstract base for autocomplete
  implementations. Implements `add()` and `search()`.
- **`InvertedIndexAutocomplete`**: Default provider using prefix matching with
  a scoring function favoring exact prefix matches.

## Exceptions

```python
from whoosh_modern.exceptions import (
    DataSourceError,
    DataSourceNotFoundError,
    DocumentIterationError,
    SchemaDiscoveryError,
    ValidationError,
)
```

All exceptions inherit from `DataSourceError`, which carries optional
`source` and `field` context attributes.

## Storage Providers

```python
from whoosh_modern.storage import (
    FileStorage,
    AsyncFileStorage,
    S3Storage,
    HybridStorage,
    AsyncHybridStorage,
)
```

### FileStorage

Local filesystem storage. `FileStorage` is an alias of `FileStorageProvider`. Keys are
relative paths under ``root``.

```python
from whoosh_modern.storage import FileStorage

storage = FileStorage("indexdir")
storage.write("segment_1.dat", b"data")
assert storage.read("segment_1.dat") == b"data"
assert storage.exists("segment_1.dat") is True
storage.delete("segment_1.dat")
keys = storage.list_keys()
```

### AsyncFileStorage

Async variant of ``FileStorage``. All operations run on a worker thread
via ``asyncio.to_thread``.

```python
import asyncio
from whoosh_modern.storage import AsyncFileStorage

storage = AsyncFileStorage("indexdir")

async def main() -> None:
    await storage.awrite("segment_1.dat", b"data")
    data = await storage.aread("segment_1.dat")
    await storage.adelete("segment_1.dat")

asyncio.run(main())
```

## See Also

- [Storage Providers Guide](../modern/storage-providers.md) — Storage backend integration and benchmarks
- [Provider Integration Guide](../modern/provider-integration.md) — Complete pipeline guide for all providers
- [Middleware Guide](../modern/middleware-pipeline.md) — Pipeline hooks and provider adapters
- [Stemming Guide](../modern/stemming-providers.md) — Stemmer provider integration
- [Vector Search Guide](../modern/vector.md) — Vector provider integration
- [Autocomplete Guide](../modern/autocomplete-providers.md) — Autocomplete provider integration
### S3Storage

S3-compatible blob storage. ``boto3`` is required only when this provider
is used; it is imported lazily so the rest of Whoosh-NG does not depend on
it. A ``client`` may be injected for testing.

```python
from whoosh_modern.storage import S3Storage

storage = S3Storage(bucket="my-index-bucket", prefix="segments")
storage.write("segment_1.dat", b"data")
data = storage.read("segment_1.dat")
keys = storage.list_keys()
```

### HybridStorage

Compose a local cache with a remote backend for cloud-native indexes.
The remote is the source of truth; the local cache is a write-through
performance layer.

```python
from whoosh_modern.storage import HybridStorage, S3Storage

remote = S3Storage(bucket="my-index-bucket", prefix="segments")
storage = HybridStorage(local_cache="./cache", remote=remote)

storage.write("segment_1.dat", b"data")
data = storage.read("segment_1.dat")  # served from cache after first read
storage.invalidate("segment_1.dat")   # force refresh from remote
storage.prefetch(["segment_2.dat"])   # warm cache proactively
```

Read path:

1. local cache hit → return immediately
2. cache miss → read from remote, write-through into cache, return

Write path:

- ``remote.write(key, data)`` (source of truth)
- on success → ``local_cache.write(key, data)``
- on failure → raise before polluting cache

### AsyncHybridStorage

Async variant of ``HybridStorage``. Remote operations are executed on a
worker thread via ``asyncio.to_thread`` so the event loop is never blocked.

```python
import asyncio
from whoosh_modern.storage import AsyncHybridStorage, S3Storage

remote = S3Storage(bucket="my-index-bucket", prefix="segments")
storage = AsyncHybridStorage(local_cache="./cache", remote=remote)

async def main() -> None:
    await storage.awrite("segment_1.dat", b"data")
    data = await storage.aread("segment_1.dat")
    await storage.adelete("segment_1.dat")

asyncio.run(main())
```


## DOCUMENT: Overview

# API Overview

This section provides a comprehensive reference of the Whoosh-NG public API.

## Modules

| Module | Description |
|--------|-------------|
| `whoosh.index` | High-level index creation, opening, and management |
| `whoosh.fields` | Schema and field type definitions |
| `whoosh.writing` | Writer classes and merge policies |
| `whoosh.searching` | Searcher, Results, and collectors |
| `whoosh.query` | Query classes and parsers |
| `whoosh.qparser` | Query parser implementation |
| `whoosh.analysis` | Tokenizers, filters, and analyzers |
| `whoosh.highlight` | Search result highlighting |
| `whoosh.spelling` | Spelling correction |
| `whoosh.sorting` | Facets and sorting |
| `whoosh.event_bus` | Event system |
| `whoosh.hooks` | Hook system |
| `whoosh.middleware` | Middleware pipeline |
| `whoosh.plugins` | Plugin system and registry |
| `whoosh.backends` | Storage backends |
| `whoosh.vector` | Vector search providers |
| `whoosh_modern.autocomplete` | Autocomplete providers |
| `whoosh_fastapi` | FastAPI integration |

## Quick Reference

### Index Lifecycle

```python
from whoosh.index import create_in, open_dir, exists_in

# Create
ix = create_in("indexdir", schema)

# Open
ix = open_dir("indexdir")

# Check
if exists_in("indexdir"):
    ix = open_dir("indexdir")
```

### Writing

```python
with ix.writer() as writer:
    writer.add_document(field1=value1, field2=value2)
    writer.commit()
```

### Reading

```python
from whoosh.qparser import QueryParser

with ix.searcher() as searcher:
    qp = QueryParser("content", ix.schema)
    q = qp.parse("query")
    results = searcher.search(q)
```

### Schema

```python
from whoosh.fields import Schema, TEXT, ID, NUMERIC

schema = Schema(
    title=TEXT(stored=True),
    path=ID(stored=True, unique=True),
    count=NUMERIC(int, stored=True)
)
```


## DOCUMENT: Plugins

# Plugins & Registry

Extend Whoosh-NG through the plugin system and registries.

## Plugin System

### BasePlugin

```python
class whoosh.plugins.base.BasePlugin
```

All plugins inherit from this class.

#### Attributes

- `name (str)`: Plugin name.
- `version (str)`: Plugin version.
- `dependencies (list[str])`: Required plugins.

#### Methods

##### `setup()`

```python
def setup(self, registry) -> None:
    """Called when the plugin is enabled."""
```

##### `teardown()`

```python
def teardown(self, registry) -> None:
    """Called when the plugin is disabled."""
```

##### `middleware()`

```python
def middleware(self) -> list[Middleware]:
    """Return middleware instances."""
```

---

### PluginManager

```python
class whoosh.plugins.manager.PluginManager
```

Manages plugin lifecycle.

#### Methods

##### `load_plugins()`

```python
PluginManager.load_plugins()
```

Auto-discover plugins from entry points.

---

##### `register()`

```python
PluginManager.register(name: str, plugin: BasePlugin)
```

Register a plugin manually.

---

##### `enable()`

```python
PluginManager.enable(name: str)
```

Enable a registered plugin.

---

##### `disable()`

```python
PluginManager.disable(name: str)
```

Disable a plugin.

---

##### `get()`

```python
plugin = PluginManager.get(name: str)
```

Get a plugin instance.

---

##### `list_plugins()`

```python
plugins = PluginManager.list_plugins()
```

List all registered plugins.

---

##### `get_middleware_chain()`

```python
chain = PluginManager.get_middleware_chain()
```

Get the combined middleware chain from all plugins.

---

## Registry System

Registries provide centralized object management.

### Registry Base

```python
class whoosh.registry.base.Registry
```

Generic registry.

#### Methods

##### `register()`

```python
Registry.register(
    key: str,
    value: Any,
    owner: str = None
)
```

Register a value.

---

##### `un


## DOCUMENT: Query

# Query API

Build and execute queries programmatically.

## QueryParser

```python
class whoosh.qparser.QueryParser(
    fieldname: str,
    schema: Schema,
    group=AndGroup,
    **kwargs
)
```

Convert a query string into a Query object.

### Methods

#### `parse()`

```python
query = qp.parse(querystring)
```

Parse a query string.

---

#### `tokenize()`

```python
tokens = qp.tokenize(querystring)
```

Tokenize a query string without parsing.

---

### MultifieldParser

```python
class whoosh.qparser.MultifieldParser(
    fieldnames: list,
    schema: Schema,
    fieldboosts: dict = None,
    group=OrGroup,
    **kwargs
)
```

Search multiple fields with different boosts.

**Example:**
```python
from whoosh.qparser import MultifieldParser

qp = MultifieldParser(
    ["title", "content"],
    schema,
    fieldboosts={"title": 2.0}
)
```

## Query Classes

All queries inherit from `Query`:

```python
class whoosh.query.Query
```

### Methods

#### `matcher()`

```python
matcher = query.matcher(searcher, context=None)
```

Return a matcher for executing the query.

#### `__and__()`, `__or__()`, `__invert__()`

Combine queries with `&`, `|`, `-`.

### Leaf Queries

#### Term

```python
Term(fieldname: str, text: str, boost: float = 1.0)
```

Match a specific term.

---

#### Phrase

```python
Phrase(fieldname: str, words: list, boost: float = 1.0, slop: int = 1)
```

Match a phrase.

---

#### Prefix

```python
Prefix(fieldname: str, text: str, boost: float = 1.0)
```

Match terms starting with `text`.

---

#### Wildcard

```python
Wildcard(fieldname: str, text: str, boost: float = 1.0)
```

Match terms with `?` and `*` wildcards.

---

#### FuzzyTerm

```python
FuzzyTerm(
    fieldname: str,
    text: str,
    maxdist: int = 2,
    prefix: int = 0,
    boost: float = 1.0
)
```

Fuzzy match with edit distance.

---

#### Range

```python
NumericRange(
    fieldname: str,
    start: Any,
    end: Any,
    startexact: bool = False,
    endexact: bool = False,
    boost: float = 1.0
)
```

Numeric range query.

```python
DateRange(
    fieldname: str,
    start: datetime,
    end: datetime,
    startexact: bool = False,
    endexact: bool = False,
    boost: float = 1.0
)
```

Date range query.

---

#### Every

```python
Every(fieldname: str, boost: float = 1.0)
```

Match every document with any term in this field.

### Boolean Queries

#### And

```python
And(children: list, boost: float = 1.0)
```

All children must match.

---

#### Or

```python
Or(children: list, boost: float = 1.0)
```

Any child must match.

---

#### Not

```python
Not(query, exclude)
```

Match docs matching query but not exclude.

---

#### DisjunctionMax

```python
DisjunctionMax(
    children: list,
    tiebreak: float = 0.0,
    boost: float = 1.0
)
```

OR-like with scoring tiebreaker.

### Special Queries

#### Require

```python
Require(match, requires)
```

Match must have `match`, and at least one of `requires`.

---

#### AndMaybe

```python
AndMaybe(must, should)
```

Must match `must`, optionally boosting with `should`.

---

#### Boost

```python
Boost(q, factor)
```

Multiply score by factor.

---

#### ConstantScore

```python
ConstantScore(q, score=1.0)
```

Assign constant score.

## Query Operators

```python
q1 & q2      # And
q1 | q2      # Or
~q1          # Not
q1 ^ q2      # DisjunctionMax
```

## Plugins

```python
from whoosh.qparser import QueryParserPlugin

class RangePlugin(QueryParserPlugin):
    def __init__(self):
        pass

    def evaluate(self, env, signode):
        # Return a query node
        return query.Range(signode.fieldname, ...)
```

## Exceptions

```python
class whoosh.qparser.QueryParserError(Exception)
```

Raised on parse errors.


## DOCUMENT: Reading

# Reading API

Classes and functions for reading from an index. The reading module is a
refactored package exposing the same public API as the former monolithic
module.

## Overview

The reading module provides classes for accessing documents, terms, and
postings in an index. The main entry points are `IndexReader` objects obtained
from a searcher. These readers allow you to enumerate terms, access stored
fields, iterate postings, and get term frequencies.

## Core Classes

### `IndexReader`

```python
class whoosh.reading.IndexReader
```

Abstract base class for reading index data. Concrete subclasses include
`SegmentReader` and `MultiReader` (which wraps multiple segment readers).

### `MultiReader`

```python
class whoosh.reading.MultiReader(readers, base=None)
```

Combines multiple `IndexReader` instances into one. All docnums are treated
as relative to the combined index.

**Constructor:**
- `readers`: A list of `IndexReader` instances.
- `base`: Optional list of cumulative document count offsets for each reader.

**Methods:**

#### `doc_frequency(fieldname, text)`

Returns the total number of documents that have the given term in the given
field across all sub-readers.

#### `documents()`

Yields dictionaries of stored fields for each document in the index.

#### `stored_fields(docnum)`

Returns a dictionary of stored field values for the given document number
(index-wide docnum).

```python
r = my_index.reader()
print(r.stored_fields(20))
```

#### `all_stored_fields()`

Yields a `(docnum, stored_fields)` tuple for each document in the index.

#### `terms(fieldname)`

Yields `(fieldname, text)` tuples for every term in the given field.

#### `terms_from(segmentreader,fieldnameprefix)`

Low-level method for multi-reader.

#### `has_termvector(docnum, fieldname)`

Returns `True` if the document has a term vector for the given field.

#### `term_vector(docnum, fieldname)`

Returns a `TermVector` for the given document and field.

#### `is_deleted(docnum)`

Returns `True` if the given document (index-wide docnum) is deleted.

#### `all_doc_ids()`

Returns a sorted array of non-deleted document IDs.

#### `min_spam(fieldname)`

Returns the minimum spam value for the given field.

#### `set_spam(fieldname)`

Returns the set spam value for the given field.

#### `has_exact_length(docnum)`

Returns `True` if the exact length is known for `docnum`.

#### `doc_field_length(docnum, fieldname=None, default=1)`

Returns the length of the given field in the given document.

```python
r = my_index.reader()
length = r.doc_field_length(20, "content")
```

#### `max_field_length(fieldname)`

Returns the maximum length of the given field across all documents.

#### `iter_fieldname`

Low-level method for multi-reader.

#### `lexicon(fieldname)`

Returns an array of all unique terms in the given field, sorted.

#### `expanded_lexicon(fieldname)`

Low-level method that yields terms without the overhead of building an array.

#### `term_info(fieldname, text)`

Returns a `TermInfo` object for the given term, or `None` if the term does
not appear in the index.

#### `terminfos(fieldname)`

Yields `(text, TermInfo)` pairs for the given field.

#### `postings(fieldname, text, stype=None)`

Returns a `Matcher` for the postings list of the given term.

```python
r = my_index.reader()
m = r.postings("content", "whoosh")
for docnum, score in m:
    print("doc %d has term" % docnum)
```

#### `_all_postings(fieldname)`

Low-level. Yields `(text, matcher)` pairs for all terms in a field.

#### `_posting_fragments()`

Low-level.

#### `has_vector(docnum, fieldname)`

Returns `True` if the given field has a term vector in the given document.

#### `vectors(docnum)`

Yields `(fieldname, TermVector)` pairs for all term vectors in the document.

#### `all_items(fieldname)`

Yields `(term, weight, docfreq)` tuples for every term in the given field.

#### `frequency(fieldname, text)`

Returns the total frequency of the term across all documents.

#### `idf(term)`

Returns an iterator of `(docnum, idf)` pairs for the given term.

#### `spelling`

Returns a `SpellingAnalyzer` for the given field.

#### `doc_term(slicenum, fieldname, word)`

Returns `(df, weight)` for `word` in `fieldname` in segment `slicenum`.

#### `doc_diff(slicenum, fieldname, text, num)`

Returns `(df, weight)` for `word` in `fieldname` in segment `slicenum`.

### `SegmentReader`

```python
class whoosh.reading.SegmentReader(segment, schema, storage, base=True)
```

Reader for a single segment of the index.

**Constructor:**
- `segment`: The `Segment` object.
- `schema`: The `Schema` object.
- `storage`: The `Storage` instance.
- `base`: Base document number offset (usually `True`, meaning compute it).

### `MultiID3Reader`

```python
class whoosh.reading.MultiID3Reader(readers, base)
```

Combines multiple readers that have ID3 codec.

### `TermInfo`

```python
class whoosh.reading.TermInfo(
    df=0,
    weight=0,
    minlength=0,
    maxlen=0,
    maxnum=0,
    numdocs=0,
    scorable=True
)
```

Information about a term in the index.

**Attributes:**
- `df`: Document frequency (number of documents containing the term).
- `weight`: Total term frequency across all documents.
- `minlength`: Minimum document length where the term appears.
- `maxlength`: Maximum document length where the term appears. This is `0`
  if lengths are not stored.
- `maxnum`: Maximum number of occurrences per document.
- `numdocs`: Number of documents where the term has a non-zero contribution
  to the score.
- `scorable`: Whether this term is scorable.

## Term Vector

### `TermVector`

```python
class whoosh.reading.TermVector(docnum, fieldname, format_, terms, store_term_vector)
```

Represents the term vector for a single document/field pair.

**Methods:**

#### `tokens(text=None)`

Yields `(t, w, v, p)` tuples for terms in this field.

- `t`: The term string.
- `w`: The term weight (frequency in this document).
- `v`: The list of positions where the term occurs. (`None` if positions
  are not stored.)
- `p`: The list of characters where the term occurs. (`None` if character
  vectors are not stored.)

#### `items(text=None)`

Like `tokens()` but includes term strings in the result.

```python
tv = my_index.reader().term_vector(0, "content")
for token, frequency, positions, chars in tv.tokens():
    print(token, frequency, positions)
```

**Parameters:**
- `text`: Optional `Bytes` object. If given, only yield terms starting with
  this text (used for multi-byte tokenization).

## Reader Utilities

### `get_storage`

```python
whoosh.reading.get_storage(searcher) -> Storage
```

Returns the storage object associated with the searcher.

### `get_index_schema`

```python
whoosh.reading.get_index_schema(searcher) -> Schema
```

Returns the schema object associated with the searcher.

### `load_termdocs`

```python
whoosh.reading.load_termdocs(reader, fieldname, text) -> list
```

Returns a list of document numbers that have the given term.

### `read_pattern`

```python
whoosh.reading.read_pattern(reader, fieldname, expression) -> list
```

Returns sorted term list from `reader.lexicon(fieldname)` filtered to those
matching `expression`.

### `read_terminfo`

```python
whoosh.reading.read_terminfo(reader, fieldname, text) -> TermInfo or None
```

Returns a `TermInfo` for the given term, or `None` if not found.


## DOCUMENT: Reference

# API Reference

The Whoosh-NG API reference is auto-generated from source code using
[pydoctor](https://pydoctor.readthedocs.io/), which parses Python modules
and generates HTML documentation from docstrings.

:::note
If the embedded documentation does not display, the API docs may not have
been generated yet in this deployment. [View on GitHub](https://github.com/dorel14/whoosh-ng/tree/master/website/static/api_docs)
for the full API documentation, or check the
[API modules list](#api-modules) below.
:::

## API Modules

### Core API

| Module | Description |
|--------|-------------|
| `whoosh.index` | High-level index creation, opening, and management |
| `whoosh.fields` | Schema and field type definitions |
| `whoosh.writing` | Writer classes and merge policies |
| `whoosh.searching` | Searcher, Results, and collectors |
| `whoosh.query` | Query classes and parsers |
| `whoosh.qparser` | Query parser implementation |
| `whoosh.analysis` | Tokenizers, filters, and analyzers |
| `whoosh.highlight` | Search result highlighting |
| `whoosh.spelling` | Spelling correction |
| `whoosh.sorting` | Facets and sorting |
| `whoosh.event_bus` | Event system |
| `whoosh.hooks` | Hook system |
| `whoosh.middleware` | Middleware pipeline |
| `whoosh.plugins` | Plugin system and registry |
| `whoosh.backends` | Storage backends |

### Modern API

| Module | Description |
|--------|-------------|
| `whoosh_modern.data_sources` | Data source protocol and implementations |
| `whoosh_modern.views` | SearchView unified interface |
| `whoosh_modern.middleware` | Retry, cache, logging middleware |
| `whoosh_modern.facets` | FacetManager for auto-discovery |
| `whoosh_modern.validation` | 4-level validation framework |
| `whoosh_modern.indexing` | BatchIndexWriter, AnalyzerCache |
| `whoosh_modern.linguistics` | Linguistic engine (stemmers, synonyms) |
| `whoosh_modern.storage` | Storage providers (HybridStorage, etc.) |
| `whoosh_modern.vector` | NumpyProvider for vector similarity |
| `whoosh_modern.autocomplete` | Autocomplete provider plugins |
| `whoosh_fastapi` | FastAPI REST API endpoints |
| `whoosh_admin` | Admin UI dashboard |

:::info
For the full interactive API documentation, run:
```bash
pip install pydoctor
python scripts/generate_api_docs.py
```
Then open `website/static/api_docs/index.html` in your browser.
:::


## DOCUMENT: Searching

# Searching API

Execute queries and retrieve results.

## Searcher

```python
class whoosh.searching.Searcher
```

The Searcher is the primary interface for reading from the index.

### Methods

#### `search()`

```python
results = searcher.search(query, limit=10, **kwargs)
```

Execute a query and return Results.

**Args:**
- `query`: The query to run.
- `limit (int)`: Maximum number of results. Use `None` for all results.

**Returns:**
- `Results`: A Results object.

---

#### `search_page()`

```python
results = searcher.search_page(query, pagenum, pagelen=10)
```

Get a page of results.

---

#### `search_with_collector()`

```python
searcher.search_with_collector(query, collector)
```

Advanced search with custom collector.

---

#### `find()`

```python
results = searcher.find("field", "text")
```

Convenience method to search a single field.

---

#### `documents()`

```python
docs = list(searcher.documents(fieldname=value))
```

Get stored documents matching a term.

---

#### `document()`

```python
doc = searcher.document(fieldname=value)
```

Get a single stored document.

---

#### `lexicon()`

```python
terms = list(searcher.lexicon("fieldname"))
```

List all terms in a field.

---

#### `all_stored_fields()`

```python
for fields in searcher.all_stored_fields():
    print(fields)
```

Iterate over all stored fields.

---

#### `all_features()`

```python
with searcher.all_features() as features:
    facets = features.facet(facet)
```

Get facet counts across all documents.

## Results

```python
class whoosh.searching.Results
```

List-like container for matched documents.

### Methods

#### `__len__()`

```python
total = len(results)
```

Total matching documents (may recount).

#### `scored_length()`

```python
scored = results.scored_length()
```

Number of scored/sorted documents in this results object.

#### `__getitem__()`

```python
hit = results[0]
hits = results[0:10]
```

Get a hit by index or slice.

#### `has_matched_terms()`

```python
if results.has_matched_terms():
    print(results.matched_terms())
```

Check if matched terms were collected.

#### `iter_matched_terms()`

Iterate over (docnum, term) pairs.

#### `upgrade()`

Move docs from another Results to top.

#### `extend()`

Append docs from another Results.

#### `upgrade_and_extend()`

Upgrade docs and append rest.

#### `filtered_count`

Number of documents filtered out.

#### `collapsed_counts`

Dict of collapse keys to filtered counts.

## Hit

```python
class whoosh.searching.Hit
```

A single matched document.

### Attributes

- `hit["fieldname"]`: Stored field value
- `hit.score`: Relevance score
- `hit.docnum`: Internal document number

### Methods

#### `highlights()`

```python
snippets = hit.highlights("content", top=3)
```

Get highlighted snippets.

#### `matched_terms()`

```python
terms = hit.matched_terms()
```

Get terms that matched (if `terms=True`).

## Highlight

```python
from whoosh.highlight import highlight, Fragment

snippets = hit.highlights(
    "content",
    top=3,
    fragmenter=None,
    formatter=None
)
```

## Collectors

```python
from whoosh.collectors import Collector, FacetCollector, TimeLimitCollector
```

## Sorting and Facets

```python
from whoosh import sorting

facet = sorting.FieldFacet("category")
results = searcher.search(query, sortedby="date")
```


## DOCUMENT: Sorting

# Sorting API

Classes and functions for faceting and sorting search results. The sorting
module is a refactored package exposing the same public API as the former
monolithic module.

## Overview

Sorting and faceting use `FacetType` objects to compute sort keys for documents.
A `FacetType` creates a `Categorizer` that computes a key for each document.
The key is used for sorting and grouping. `FacetMap` objects hold the
results of grouping documents by a facet.

## Facet Types

### `FacetType`

```python
class whoosh.sorting.FacetType
```

Base class for "facets" — aspects that can be sorted and/or faceted.

**Attributes:**
- `maptype`: Default `FacetMap` class to use for this facet.

**Methods:**

#### `categorizer(global_searcher)`

Returns a `Categorizer` corresponding to this facet.

- `global_searcher`: A parent searcher for global document ID references.

#### `map(default=None)`

Returns a `FacetMap` instance for holding facet results.

#### `default_name()`

Returns the default name for this facet (default `"facet"`).

### `Categorizer`

```python
class whoosh.sorting.Categorizer
```

Base class for objects that compute a key value for a document for sorting and
faceting. Created by `FacetType` objects via `categorizer()`.

**Attributes:**
- `allow_overlap (bool)`: If `True`, use `keys_for()` to allow overlapping
  groups. Default `False`.
- `needs_current (bool)`: If `True`, the categorizer needs the matcher to be
  in a valid state when `key_for()` is called. Default `False`.

**Methods:**

#### `set_searcher(segment_searcher, docoffset)`

Called when the collector moves to a new segment. Sets up segment-specific
data.

- `segment_searcher`: The atomic sub-searcher for the current segment.
- `docoffset`: Offset of the segment's docnums relative to the full index.

#### `key_for(matcher, segment_docnum)`

Returns a sort key for the current match.

- `matcher`: A `Matcher` object. If `needs_current` is `False`, do not use
  this object as it may be inconsistent.
- `segment_docnum`: Segment-relative document number.

#### `keys_for(matcher, segment_docnum)`

Yields multiple keys for the current match. Called instead of `key_for()`
when `allow_overlap` is `True`.

#### `key_to_name(key)`

Translates the sort key into a human-readable representation for facet
group names (e.g., converts an integer date sort key to a `datetime`).

### `FieldFacet`

```python
class whoosh.sorting.FieldFacet(
    fieldname,
    reverse=False,
    allow_overlap=False,
    maptype=None
)
```

Sorts/facets by the contents of a field.

**Constructor:**
- `fieldname`: Name of the field to sort/facet on.
- `reverse`: If `True`, reverse the sort order.
- `allow_overlap`: If `True`, allow documents to appear in multiple groups
  when they have multiple terms in the field.
- `maptype`: `FacetMap` class for holding results.

```python
paths = FieldFacet("path", reverse=True)
tags = FieldFacet("tag")
results = searcher.search(myquery, sortedby=paths, groupedby=tags)
```

### `ColumnCategorizer`

Categorizer that reads values from a column for sorting. Used when a field
has a column type.

### `ReversedColumnCategorizer`

Categorizer that reverses column values for fields that are not naturally
reversible.

### `OverlappingCategorizer`

```python
class whoosh.sorting.OverlappingCategorizer
```

Categorizer used when `allow_overlap=True`. A single document can belong to
multiple facet groups.

### `PostingCategorizer`

```python
class whoosh.sorting.PostingCategorizer
```

Categorizer for fields without column values. Builds an array caching the
order of all documents. Used as a fallback; prefer setting
`sortable=True` on fields.

### `QueryFacet`

```python
class whoosh.sorting.QueryFacet(
    querydict: dict,
    other=None,
    allow_overlap=False,
    maptype=None
)
```

Sorts/facets based on the results of a series of queries.

**Constructor:**
- `querydict`: Dictionary mapping keys to `Query` objects.
- `other`: Key to use for documents matching no queries.

### `RangeFacet`

```python
class whoosh.sorting.RangeFacet(
    fieldname,
    start,
    end,
    gap,
    hardend=False,
    maptype=None
)
```

Sorts/facets based on numeric ranges. Ranges are inclusive at the start and
exclusive at the end.

```python
prices = RangeFacet("price", 0, 1000, 100)
results = searcher.search(myquery, groupedby=prices)
```

- `fieldname`: The numeric field to facet on.
- `start`: Start of the entire range.
- `end`: End of the entire range.
- `gap`: Size of each bucket (can be a sequence for progressive gaps).
- `hardend`: If `True`, clamp the last bucket to `end`.

### `DateRangeFacet`

```python
class whoosh.sorting.DateRangeFacet(
    fieldname,
    startdate,
    enddate,
    gap,
    hardend=False,
    maptype=None
)
```

Sorts/facets based on date ranges. Extends `RangeFacet` but uses
`datetime` objects for start/end and `timedelta`/`relativedelta` for gaps.
Generates `DateRange` queries instead of `TermRange` queries.

```python
from datetime import datetime
from whoosh.support.relativedelta import relativedelta

startdate = datetime(1920, 1, 1)
enddate = datetime.now()
gap = relativedelta(years=5)
bdays = DateRangeFacet("birthday", startdate, enddate, gap)
```

### `ScoreFacet`

```python
class whoosh.sorting.ScoreFacet
```

Uses a document's relevance score as a sorting criterion.

```python
tag_score = MultiFacet(["tag", ScoreFacet()])
results = searcher.search(myquery, sortedby=tag_score)
```

### `FunctionFacet`

```python
class whoosh.sorting.FunctionFacet(fn)
```

Lets you pass an arbitrary function that computes the sort key. The function
is called with `(searcher, docid)` where `docid` is an absolute index
document number.

```python
fn = lambda s, docid: s.doc_field_length(docid, "content")
lengths = FunctionFacet(fn)
```

### `TranslateFacet`

```python
class whoosh.sorting.TranslateFacet(fn, *facets)
```

Applies a custom function to the key generated by one or more wrapped facets.
Useful for custom collation, such as Unicode Collation Algorithm (UCA) sorting.

```python
from pyuca import Collator

c = Collator("allkeys.txt")
facet = FieldFacet("name")
facet = TranslateFacet(c.sort_key, facet)
results = searcher.search(myquery, sortedby=facet)
```

**Constructor:**
- `fn`: Function applied to the computed key values.
- `*facets`: One or more `FacetType` objects whose keys are passed to `fn`.

### `StoredFieldFacet`

```python
class whoosh.sorting.StoredFieldFacet(
    fieldname,
    allow_overlap=False,
    split_fn=None,
    maptype=None
)
```

Sorts/groups using the value in an unindexed, stored field (e.g., `STORED`).
Usually slower than using an indexed field.

**Constructor:**
- `fieldname`: Name of the stored field.
- `allow_overlap`: If `True`, when grouping, allow documents to appear in
  multiple groups when they have multiple values (split by `split_fn` or
  `string.split()`).
- `split_fn`: Custom function to split a stored field value into multiple
  facet values (only used when `allow_overlap=True`).

### `MultiFacet`

```python
class whoosh.sorting.MultiFacet(items=None, maptype=None)
```

Sorts/facets by the combination of multiple sub-facets.

```python
facet = MultiFacet([FieldFacet("tag"), FieldFacet("path")])
results = searcher.search(myquery, sortedby=facet)
```

Strings in the items list are treated as field names:

```python
facet = MultiFacet(["tag", "path"])
```

**Methods:**
- `from_sortedby(sortedby)`: Class method that creates a `MultiFacet` from
  a field name, facet, or list thereof.
- `add_field(fieldname, reverse=False)`: Add a `FieldFacet`.
- `add_query(querydict, other=None, allow_overlap=False)`: Add a `QueryFacet`.
- `add_score()`: Add a `ScoreFacet`.
- `add_facet(facet)`: Add an arbitrary `FacetType`.

### `Facets`

```python
class whoosh.sorting.Facets(x=None)
```

Maps facet names to `FacetType` objects for creating multiple independent
groupings of documents.

```python
facets = Facets()
facets.add_field("tag")
facets.add_facet("price", RangeFacet("price", 0, 1000, 100))
results = searcher.search(myquery, groupedby=facets)

tag_groups = results.groups("tag")
price_groups = results.groups("price")
```

**Class Methods:**
- `from_groupedby(groupedby)`: Creates a `Facets` object from a field name,
  `FacetType`, dict, list, or another `Facets` object.

**Methods:**
- `names()`: Returns an iterator of facet names.
- `items()`: Returns a list of `(name, facet)` tuples.
- `add_field(fieldname, **kwargs)`: Adds a `FieldFacet`.
- `add_query(name, querydict, **kwargs)`: Adds a `QueryFacet`.
- `add_facet(name, facet)`: Adds a `FacetType` under the given name.
- `add_facets(facets, replace=True)`: Adds the contents of a `Facets` or
  `dict` to this object.

## Facet Maps

### `FacetMap`

```python
class whoosh.sorting.FacetMap
```

Base class for objects holding the results of grouping search results by a
facet. Use `as_dict()` to access results.

```python
myfacet = FieldFacet("size", maptype=OrderedList)
myfacet = FieldFacet("size", maptype=Count)
```

**Methods:**
- `add(groupname, docid, sortkey)`: Adds a document to the facet results.
- `as_dict()`: Returns a dictionary mapping group names to values.

### `OrderedList`

```python
class whoosh.sorting.OrderedList
```

Stores a list of document numbers for each group, in sorted order.

### `UnorderedList`

```python
class whoosh.sorting.UnorderedList
```

Stores a list of document numbers for each group in arbitrary order. Slightly
faster and more memory-efficient than `OrderedList` when ordering doesn't
matter.

### `Count`

```python
class whoosh.sorting.Count
```

Stores the count of documents in each group.

### `Best`

```python
class whoosh.sorting.Best
```

Stores the "best" (highest sort key) document in each group.

## Sorting Utilities

### `add_sortable`

```python
whoosh.sorting.add_sortable(
    writer,
    fieldname,
    facet,
    column=None
)
```

Adds a per-document value column to an existing field, making it sortable.
Useful for retrofitting fields that were created without `sortable=True`.

**Example:**
```python
from whoosh import index, sorting

ix = index.open_dir("indexdir")
with ix.writer() as w:
    facet = sorting.FieldFacet("price")
    sorting.add_sortable(w, "price", facet)
```

**Parameters:**
- `writer`: An `IndexWriter` object.
- `fieldname`: Name of the field to add sortable values to.
- `facet`: A `FacetType` object to generate per-document values.
- `column`: Optional `ColumnType` to store the values. If omitted, uses the
  field's default column type.


## DOCUMENT: Spelling

# Spelling API

Functions and classes for correcting typos in user queries using edit-distance
(Damerau-Levenshtein) matching against the terms in the index.

## Corrector Objects

### `Corrector`

```python
class whoosh.spelling.Corrector
```

Base class for spelling correction objects. Concrete subclasses implement the
`_suggestions()` method.

**Methods:**

#### `suggest(text, limit=5, maxdist=2, prefix=0)`

Returns a list of suggested corrections for `text`, ranked by edit distance
then by frequency.

- `text`: The text to check. Will **not** be added to suggestions even if it
  appears in the index.
- `limit`: Maximum number of suggestions to return.
- `maxdist`: Maximum edit distance to look at (values > 2 are inefficient).
- `prefix`: Require suggestions to share this length of prefix with `text`.
  Increasing to even `1` dramatically speeds up suggestions.

#### `_suggestions(text, maxdist, prefix)`

Low-level method yielding `(score, suggestion)` tuples. Subclasses must
implement this.

### `ReaderCorrector`

```python
class whoosh.spelling.ReaderCorrector(reader, fieldname, fieldobj)
```

Suggests corrections based on terms in a specific field of an `IndexReader`.

**Ranks suggestions by edit distance, then by highest to lowest frequency.**

**Constructor:**
- `reader`: An `IndexReader` object.
- `fieldname`: The name of the field to get suggestions from.
- `fieldobj`: The `FieldType` for the field.

### `ListCorrector`

```python
class whoosh.spelling.ListCorrector(wordlist)
```

Suggests corrections based on a sorted list of strings.

**Constructor:**
- `wordlist`: A sorted list of words to match against.

### `MultiCorrector`

```python
class whoosh.spelling.MultiCorrector(correctors, op)
```

Merges suggestions from a list of sub-correctors.

**Constructor:**
- `correctors`: List of `Corrector` objects.
- `op`: A function (e.g., `max` or `operator.add`) to combine scores from
  multiple correctors for the same suggestion.

## Query Correction

### `Correction`

```python
class whoosh.spelling.Correction(q, qstring, corr_q, tokens)
```

Represents the corrected version of a user query string.

**Attributes:**
- `query`: The corrected `Query` object.
- `string`: The corrected user query string.
- `original_query`: The original `Query` object.
- `original_string`: The original user query string.
- `tokens`: List of token objects representing corrected words.

**Methods:**

#### `format_string(formatter)`

Highlights corrected words in the original query string using the given
`Formatter`.

```python
from whoosh import highlight

correction = searcher.correct_query(q, qstring)
hf = highlight.HtmlFormatter(classname="change")
html = correction.format_string(hf)
```

- `formatter`: A `Formatter` instance (or class, which will be instantiated).
- Returns: Formatted string, typically with corrections emphasized.

### `QueryCorrector`

```python
class whoosh.spelling.QueryCorrector(fieldname)
```

Base class for objects that correct words in a user query.

**Constructor:**
- `fieldname`: The default field name for corrections.

**Methods:**

#### `correct_query(q, qstring)`

Returns a `Correction` object representing the corrected form of the given
query.

- `q`: The original `Query` tree to be corrected.
- `qstring`: The original user query string (may be `None`).
- Returns: A `Correction` object.

#### `field()`

Returns the field name this corrector operates on.

### `SimpleQueryCorrector`

```python
class whoosh.spelling.SimpleQueryCorrector(
    correctors: dict,
    terms: list,
    aliases=None,
    prefix: int = 0,
    maxdist: int = 2
)
```

A simple query corrector based on a mapping of field names to `Corrector`
objects, and a list of `(fieldname, text)` tuples to correct.

**Constructor:**
- `correctors`: Dictionary mapping field names to `Corrector` objects.
- `terms`: Sequence of `(fieldname, text)` tuples representing terms to be
  corrected.
- `aliases`: Dictionary mapping field names in the query to field names for
  spelling suggestions.
- `prefix`: Suggested replacement words must share this number of initial
  characters. Default `0`.
- `maxdist`: Maximum edit distance for suggestions. Values > 2 may be slow.


## DOCUMENT: Writing

# Writing API

Write, update, and delete documents using the `IndexWriter` interface.

## IndexWriter

```python
class whoosh.writing.IndexWriter
```

Base class for writing documents.

### Context Manager

```python
with ix.writer() as writer:
    writer.add_document(title="Hello", content="World")
    # commit() called automatically
```

### Methods

#### `add_document()`

```python
writer.add_document(**fields)
```

Add a document to the index.

**Special kwargs:**
- `_stored_<fieldname>`: Alternate stored value
- `_<fieldname>_boost`: Field-specific boost
- `_boost`: Document-wide boost

---

#### `update_document()`

```python
writer.update_document(**fields)
```

Update/replace a document. Uses `unique` fields to find existing documents.

---

#### `delete_document()`

```python
writer.delete_document(docnum: int, delete: bool = True)
```

Delete by document number.

---

#### `delete_by_term()`

```python
writer.delete_by_term(fieldname: str, text: str) -> int
```

Delete all documents with term in field.

**Returns:**
- `int`: Number of documents deleted.

---

#### `delete_by_query()`

```python
writer.delete_by_query(q: Query, searcher=None) -> int
```

Delete documents matching query.

---

#### `commit()`

```python
writer.commit(
    mergetype=None,
    optimize=False,
    merge=True
)
```

Commit changes to disk.

**Args:**
- `mergetype`: Custom merge function
- `optimize`: Merge all segments into one
- `merge`: If False, don't merge existing segments

---

#### `cancel()`

```python
writer.cancel()
```

Cancel pending changes and release lock.

---

#### `add_field()`

```python
writer.add_field(fieldname: str, fieldtype, **kwargs)
```

Add a field to schema (before adding documents).

---

#### `remove_field()`

```python
writer.remove_field(fieldname: str)
```

Remove a field from schema.

---

#### `searcher()`

```python
searcher = writer.searcher(**kwargs)
```

Return a searcher (for reading during write session).

---

#### `reader()`

```python
reader = writer.reader(**kwargs)
```

Return a reader for the current state.

---

#### `group()`

```python
with writer.group():
    writer.add_document(kind="class", name="MyClass")
    writer.add_document(kind="method", name="my_method")
```

Context manager for grouping documents into one segment.

## SegmentWriter

Concrete implementation of `IndexWriter`.

### Constructor

```python
SegmentWriter(
    ix,
    poolclass=None,
    timeout=0.0,
    delay=0.1,
    _lk=True,
    limitmb=128,
    docbase=0,
    codec=None,
    compound=True,
    **kwargs
)
```

## AsyncWriter

Threaded writer that automatically retries on lock contention.

```python
from whoosh.writing import AsyncWriter

writer = AsyncWriter(
    index,
    delay=0.25,
    writerargs={}
)
```

## BufferedWriter

Buffers documents in memory and commits periodically.

```python
from whoosh.writing import BufferedWriter

writer = BufferedWriter(
    index,
    period=60,       # Max seconds between commits
    limit=100,       # Max documents per commit
    writerargs={}    # Extra args for writer
)
```

The `BufferedWriter` also provides `reader()` and `searcher()` methods for quasi-real-time search.

## Merge Policies

```python
from whoosh.writing import NO_MERGE, MERGE_SMALL, OPTIMIZE, CLEAR

writer.commit(mergetype=NO_MERGE)     # No merging
writer.commit(mergetype=MERGE_SMALL) # Merge small segments
writer.commit(mergetype=OPTIMIZE)    # Merge all into one
writer.commit(mergetype=CLEAR)       # Delete all existing segments
```

## PostingPool

Internal pool for sorting postings. Typically not used directly.

```python
class whoosh.writing.PostingPool
```

## Exceptions

### IndexingError

```python
class whoosh.writing.IndexingError(Exception)
```

Raised when an indexing operation fails.


## DOCUMENT: Analysis

:::info
Following the rename of `whoosh-reloaded` to `whoosh-ng`, new Whoosh-NG specific modules are typically found under `whoosh_modern`.
Core Whoosh components (like `whoosh.analysis`, `whoosh.index`) remain accessible directly under the `whoosh` namespace for backward compatibility.
:::

# About analyzers

## Overview

An analyzer is a function or callable class (a class with a `__call__` method)
that takes a unicode string and returns a generator of tokens. Usually a
"token" is a word, for example the string "Mary had a little lamb" might yield
the tokens "Mary", "had", "a", "little", and "lamb". However, tokens do not
necessarily correspond to words. For example, you might tokenize Chinese text
into individual characters or bi-grams. Tokens are the units of indexing, that
is, they are what you are able to look up in the index.

An analyzer is basically just a wrapper for a tokenizer and zero or more
filters. The analyzer's `__call__` method will pass its parameters to a
tokenizer, and the tokenizer will usually be wrapped in a few filters.

A tokenizer is a callable that takes a unicode string and yields a series of
`analysis.Token` objects.

For example, the provided `whoosh.analysis.RegexTokenizer` class implements a
customizable, regular-expression-based tokenizer that extracts words and
ignores whitespace and punctuation:

```python
from whoosh.analysis import RegexTokenizer

tokenizer = RegexTokenizer()
for token in tokenizer("Hello there my friend!"):
    print(repr(token.text))
# u'Hello'
# u'there'
# u'my'
# u'friend'
```

A filter is a callable that takes a generator of Tokens (either a tokenizer or
another filter) and in turn yields a series of Tokens.

For example, the provided `whoosh.analysis.LowercaseFilter()` filters tokens by
converting their text to lowercase. The implementation is very simple:

```python
def LowercaseFilter(tokens):
    """Uses lower() to lowercase token text."""
    for t in tokens:
        t.text = t.text.lower()
        yield t
```

You can wrap the filter around a tokenizer to see it in operation:

```python
from whoosh.analysis import LowercaseFilter, RegexTokenizer

tokenizer = RegexTokenizer()
for token in LowercaseFilter(tokenizer("These ARE the things I want!")):
    print(repr(token.text))
# u'these'
# u'are'
# u'the'
# u'things'
# u'i'
# u'want'
```

An analyzer is just a means of combining a tokenizer and some filters into a
single package.

You can implement an analyzer as a custom class or function, or compose
tokenizers and filters together using the `|` character:

```python
my_analyzer = RegexTokenizer() | LowercaseFilter() | StopFilter()
```

The first item must be a tokenizer and the rest must be filters (you can't put
a filter first or a tokenizer after the first item). Note that this only works
if at least the tokenizer is a subclass of `whoosh.analysis.Composable`, as all
the tokenizers and filters that ship with Whoosh are.

## Using analyzers

When you create a field in a schema, you can specify your analyzer as a keyword
argument to the field object:

```python
schema = Schema(content=TEXT(analyzer=StemmingAnalyzer()))
```

## Advanced analysis

### Token objects

The `Token` class has no methods. It is merely a place to record certain
attributes. A `Token` object actually has two kinds of attributes: *settings*
that record what kind of information the `Token` object does or should contain,
and *information* about the current token.

#### Token setting attributes

A `Token` object should always have the following attributes. A tokenizer or
filter can check these attributes to see what kind of information is available
and/or what kind of information they should be setting on the `Token` object.
Filters **should not** change the values of these attributes.

| Type | Attribute name | Description | Default |
|------|----------------|-------------|---------|
| str | mode | The mode in which the analyzer is being called, e.g. `'index'` during indexing or `'query'` during query parsing | `''` |
| bool | positions | Whether term positions are recorded in the token | `False` |
| bool | chars | Whether term start and end character indices are recorded in the token | `False` |
| bool | boosts | Whether per-term boosts are recorded in the token | `False` |
| bool | removestops | Whether stop-words should be removed from the token stream | `True` |

#### Token information attributes

A `Token` object may have any of the following attributes. The `text` attribute
should always be present. The `original` attribute may be set by a tokenizer.
All other attributes should only be accessed or set based on the values of the
"settings" attributes above.

| Type | Name | Description |
|------|------|-------------|
| unicode | text | The text of the token (this should always be present) |
| unicode | original | The original (pre-filtered) text of the token |
| int | pos | The position of the token in the stream, starting at 0 (only set if positions is True) |
| int | startchar | The character index of the start of the token in the original string (only set if chars is True) |
| int | endchar | The character index of the end of the token in the original string (only set if chars is True) |
| float | boost | The boost for this token (only set if boosts is True) |
| bool | stopped | Whether this token is a "stop" word (only set if removestops is False) |

### Performing different analysis for indexing and query parsing

Whoosh sets the `mode` setting attribute to indicate whether the analyzer is
being called by the indexer (`mode='index'`) or the query parser
(`mode='query'`). This is useful if there's a transformation that you only want
to apply at indexing or query parsing:

```python
class MyFilter(Filter):
    def __call__(self, tokens):
        for t in tokens:
            if t.mode == 'query':
                ...
            else:
                ...
```

The `whoosh.analysis.MultiFilter` filter class lets you specify different
filters to use based on the mode setting:

```python
intraword = MultiFilter(
    index=IntraWordFilter(mergewords=True, mergenums=True),
    query=IntraWordFilter(mergewords=False, mergenums=False),
)
```

### Stop words

"Stop" words are words that are so common it's often counter-productive to
index them, such as "and", "or", "if", etc. The provided `analysis.StopFilter`
lets you filter out stop words, and includes a default list of common stop
words.

```python
from whoosh.analysis import StopFilter

stopper = StopFilter()
for token in stopper(LowercaseFilter(tokenizer("These ARE the things I want!"))):
    print(repr(token.text))
# u'these'
# u'things'
# u'want'
```

#### Renumbering term positions

Remember that analyzers are sometimes asked to record the position of each
token in the token stream. So what happens to the `pos` attribute of the
tokens if `StopFilter` removes the words `had` and `a` from the stream? Should
it renumber the positions to pretend the "stopped" words never existed? Or
should it preserve the original positions of the words?

It turns out that different situations call for different solutions, so the
provided `StopFilter` class supports both of the above behaviors. Renumbering
is the default, since that is usually the most useful and is necessary to
support phrase searching. However, you can set a parameter in StopFilter's
constructor to tell it not to renumber positions:

```python
stopper = StopFilter(renumber=False)
```

#### Removing or leaving stop words

The point of using `StopFilter` is to remove stop words, right? Well, there are
actually some situations where you might want to mark tokens as "stopped" but
not remove them from the token stream.

The `removestops` parameter passed to the analyzer's `__call__` method (and
copied to the `Token` object as an attribute) specifies whether stop words
should be removed from the stream or left in.

```python
from whoosh.analysis import StandardAnalyzer

analyzer = StandardAnalyzer()
print([(t.text, t.stopped) for t in analyzer("This is a test")])
# [(u'test', False)]

print([(t.text, t.stopped) for t in analyzer("This is a test", removestops=False)])
# [(u'this', True), (u'is', True), (u'a', True), (u'test', False)]
```

The `analysis.unstopped()` filter function takes a token generator and yields
only the tokens whose `stopped` attribute is `False`.

> Even if you leave stopped words in the stream in an analyzer you use for
> indexing, the indexer will ignore any tokens where the `stopped` attribute is
> `True`.

### Implementation notes

Because object creation is slow in Python, the stock tokenizers do not create a
new `analysis.Token` object for each token. Instead, they create one `Token`
object and yield it over and over. This is a nice performance shortcut but can
lead to strange behavior if your code tries to remember tokens between loops of
the generator.

```python
# WRONG: the generator reuses the same Token object
print(list(tokenizer("Hello there my friend")))
# [Token(u"friend"), Token(u"friend"), Token(u"friend"), Token(u"friend")]

# RIGHT: save the attributes, not the token object
print([t.text for t in tokenizer("Hello there my friend")])
# [u'Hello', u'there', u'my', u'friend']
```

If you implement your own tokenizer, filter, or analyzer as a class, you should
implement an `__eq__` method. This is important to allow comparison of `Schema`
objects.

## See also

- [Stemming & Stop Words](/core/stemming) — Practical stemming and stop-word guides
- [N-grams](/core/ngrams) — Substring and prefix matching with N-gram analyzers
- [API: analysis](../api/analysis) — Full reference for analyzers, tokenizers, and filters


## DOCUMENT: Backends

# Backends

Whoosh-NG supports pluggable storage backends through the Provider Architecture. The default backend stores data as files on disk, but you can use SQLite, PostgreSQL, S3, and more.

## Built-in Backends

| Backend | Class | Description |
|---------|-------|-------------|
| File (default) | `FileBackend` | Stores index as files on disk |
| SQLite | `SQLiteBackend` | Stores index in SQLite database |
| Memory | `MemoryBackend` | In-memory backend (testing only) |

## File Backend (Default)

```python
from whoosh.index import create_in

# Uses FileBackend by default
ix = create_in("indexdir", schema)
```

### Configuration

```python
from whoosh.backends.file import FileBackend

backend = FileBackend(
    storage=FileStorage("indexdir"),
    compound=True  # Use compound files
)
```

## SQLite Backend

```python
from whoosh.backends.sqlite import SQLiteBackend
from whoosh.store.sqlite import SQLiteStorage

storage = SQLiteStorage("index.db")
backend = SQLiteBackend(storage=storage)
```

### Advantages

- Single file index
- Better for transactional workloads
- Easier backups
- Supports concurrent reads

### Disadvantages

- Slower for large indexes
- Limited by SQLite performance

## Memory Backend

```python
from whoosh.backends.memory import MemoryBackend

backend = MemoryBackend()
# Useful for testing
```

## Custom Backend

Create a custom backend by subclassing `Backend`:

```python
from whoosh.backends.abc import Backend

class MyBackend(Backend):
    def create(self):
        """Create a new segment."""
        pass

    def open(self):
        """Open existing segment."""
        pass

    def close(self):
        """Close the backend."""
        pass

    def commit(self):
        """Commit changes."""
        pass
```

## Registering a Backend

```python
from whoosh.registry import BackendRegistry

BackendRegistry.register("my_backend", MyBackend, "my_package")
```

## Backend Selection

Choose a backend based on your use case:

| Use Case | Recommended Backend |
|----------|---------------------|
| Small to medium indexes | File (default) |
| Single-file deployment | SQLite |
| Testing | Memory |
| Distributed systems | Object storage (S3, MinIO) |
| High concurrency | SQLite or custom |

## Best Practices

1. **File backend for production**: Most battle-tested
2. **SQLite for single-file**: Easier deployment
3. **Memory for tests**: Fast, no cleanup needed
4. **Compound files**: Enable for reduced file count
5. **Backup strategy**: File backend = copy directory; SQLite = copy file


## DOCUMENT: Batch

# Tips for speeding up batch indexing

## Overview

Indexing documents tends to fall into two general patterns: adding documents
one at a time as they are created (as in a web application), and adding a bunch
of documents at once (batch indexing).

The following settings and alternate workflows can make batch indexing faster.

## StemmingAnalyzer cache

The stemming analyzer by default uses a least-recently-used (LRU) cache to
limit the amount of memory it uses, to prevent the cache from growing very
large if the analyzer is reused for a long period of time. However, the LRU
cache can slow down indexing by almost 200% compared to a stemming analyzer
with an "unbounded" cache.

When you're indexing in large batches with a one-shot instance of the analyzer,
consider using an unbounded cache:

> **Note**: For new implementations or complex multilingual scenarios, consider
> using the `CachedStemmingAnalyzer` (from `whoosh_modern.analysis.cached_stemming_analyzer`)
> which offers integrated LRU caching and flexible configuration.

```python
w = myindex.writer()
# Get the analyzer object from a text field
stem_ana = w.schema["content"].analyzer
# Set the cachesize to -1 to indicate unbounded caching
stem_ana.cachesize = -1
# Reset the analyzer to pick up the changed attribute
stem_ana.clear()

# Use the writer to index documents...
```

## The `limitmb` parameter

The `limitmb` parameter to `whoosh.index.Index.writer()` controls the
*maximum* memory (in megabytes) the writer will use for the indexing pool. The
higher the number, the faster indexing will be.

The default value of `128` is actually somewhat low, considering many people
have multiple gigabytes of RAM these days. Setting it higher can speed up
indexing considerably:

```python
from whoosh import index

ix = index.open_dir("indexdir")
writer = ix.writer(limitmb=256)
```

> The actual memory used will be higher than this value because of interpreter
> overhead (up to twice as much!). It is very useful as a tuning parameter, but
> not for trying to exactly control the memory usage of Whoosh.

## The `procs` parameter

The `procs` parameter to `whoosh.index.Index.writer()` controls the number of
processors the writer will use for indexing (via the `multiprocessing` module):

```python
from whoosh import index

ix = index.open_dir("indexdir")
writer = ix.writer(procs=4)
```

When you use multiprocessing, the `limitmb` parameter controls the amount of
memory used by *each process*, so the actual memory used will be
`limitmb * procs`:

```python
# Each process will use a limit of 128, for a total of 512
writer = ix.writer(procs=4, limitmb=128)
```

## The `multisegment` parameter

The `procs` parameter causes the default writer to use multiple processors to
do much of the indexing, but then still uses a single process to merge the pool
of each sub-writer into a single segment.

You can get much better indexing speed by also using the `multisegment=True`
keyword argument, which instead of merging the results of each sub-writer,
simply has them each just write out a new segment:

```python
from whoosh import index

ix = index.open_dir("indexdir")
writer = ix.writer(procs=4, multisegment=True)
```

The drawback is that instead of creating a single new segment, this option
creates a number of new segments **at least** equal to the number of processors
you use. For example, if you use `procs=4`, the writer will create four new
segments.

So, while `multisegment=True` is much faster than a normal writer, you should
only use it for large batch indexing jobs (or perhaps only for indexing from
scratch). It should not be the only method you use for indexing, because
otherwise the number of segments will tend to increase forever!

## See also

- [Indexing](/core/indexing) — Writer options and merge policies
- [API: writing](../api/writing) — `Index.writer()` parameters


## DOCUMENT: Changelog

# Changelog

Release notes for Whoosh-NG, auto-generated from GitHub releases and commit messages.

## v5.3.1 (2026-08-13)
**Tag**: `v5.3.1`

## v5.3.1 (2026-08-13)

_This release is published under the MIT License._

### Bug Fixes

- **linguistics**: Rendre l'import langdetect robuste pour pyright ([`54ab70e`](https://github.com/dorel14/whoosh-ng/commit/54ab70e9247c3ecfdebef592f1107dc794921194))

- **tooling**: Corriger le type:ignore de langdetect pour mypy/pyright ([`e37be1c`](https://github.com/dorel14/whoosh-ng/commit/e37be1ce62ff958011983f8ed3efded56428fda8))

### Documentation

- Update changelog and API docs [skip ci] ([`d884547`](https://github.com/dorel14/whoosh-ng/commit/d884547c17bcaea800e7e2277bcdfc25753ad446))

---

**Detailed Changes**: [v5.3.0...v5.3.1](https://github.com/dorel14/whoosh-ng/compare/v5.3.0...v5.3.1)

### Commits

### Chores

- merge master into dev [skip ci]
- apply pre-commit fixes
- v5.3.1 [skip ci]

### Other

- .
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge pull request #455 from dorel14/dev

### Bug Fixes

- corriger le type:ignore de langdetect pour mypy/pyright
- rendre l'import langdetect robuste pour pyright

### Documentation

- update changelog and API docs [skip ci]


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v5.3.1)

## v5.3.0 (2026-08-13)
**Tag**: `v5.3.0`

## v5.3.0 (2026-08-13)

_This release is published under the MIT License._

### Bug Fixes

- **linguistics**: Corriger les défauts signalés par revue et docs ([`ba05a8d`](https://github.com/dorel14/whoosh-ng/commit/ba05a8d5cbdba8d71f39d8e24920d55f77a8eb32))

### Documentation

- Auto-update llms context files [skip ci] ([`4585b5e`](https://github.com/dorel14/whoosh-ng/commit/4585b5eb665c0329c70da0efe5225b859fc14573))

- Auto-update llms context files [skip ci] ([`c897f1f`](https://github.com/dorel14/whoosh-ng/commit/c897f1f0e6c6bbebee46a7672f5ca6d3d6abebf4))

- Auto-update llms context files [skip ci] ([`a65b926`](https://github.com/dorel14/whoosh-ng/commit/a65b926027472d65244c76f250833af27c4bb8e5))

- Auto-update llms context files [skip ci] ([`4911cf5`](https://github.com/dorel14/whoosh-ng/commit/4911cf57abc555eadfa47cf962943b321fb2457a))

- Auto-update llms context files [skip ci] ([`8605b34`](https://github.com/dorel14/whoosh-ng/commit/8605b344f85f3e40dec93c9f9c7d04e7c10fd9f6))

- Auto-update llms context files [skip ci] ([`8c88da5`](https://github.com/dorel14/whoosh-ng/commit/8c88da5d29097f8a4ee6a9c47ffcc5afdcf2ff55))

- Update changelog and API docs [skip ci] ([`ae57714`](https://github.com/dorel14/whoosh-ng/commit/ae57714558186480f0f15e4aa21e093c61c166bd))

- **config**: Corriger l'exemple ConfigEngine.build() ([#422](https://github.com/dorel14/whoosh-ng/pull/422), [`f09d6da`](https://github.com/dorel14/whoosh-ng/commit/f09d6dac72482868275ad581046909a5b4169885))

- **documentation**: Mettre à jour la documentation pour le renommage en whoosh-ng ([`ec35fbd`](https://github.com/dorel14/whoosh-ng/commit/ec35fbd5182dc0c01e5888d7285f62be8c21a262))

- **linguistics**: Documenter la détection automatique de langue et l'indexation multilingue ([`f86a17b`](https://github.com/dorel14/whoosh-ng/commit/f86a17bd7820a2e5a3f99fc8a8516c4b9b460a18))

- **website**: Restructurer la documentation et déplacer les fonctionnalités classiques vers Core ([`54fb1f6`](https://github.com/dorel14/whoosh-ng/commit/54fb1f61f81b441febda3a0ecfce3fef8edb400f))

### Features

- **analysis**: Add EPIC 5.6 Modern N-Gram Framework Sprint NGRAM-1 ([#440](https://github.com/dorel14/whoosh-ng/pull/440), [`c634718`](https://github.com/dorel14/whoosh-ng/commit/c6347184bb2b6d307da5008c1edf4850eab88d0b))

- **api**: Exposer les nouvelles fonctionnalités pour la version 5.1.0 ([`e306236`](https://github.com/dorel14/whoosh-ng/commit/e306236a71fdcd5f018d9af10cf923398ed852c1))

- **linguistics**: Add language auto-detection for multilingual indexing ([#434](https://github.com/dorel14/whoosh-ng/pull/434), [`a91d6bf`](https://github.com/dorel14/whoosh-ng/commit/a91d6bf59b8faae98fe80aa4d6c5ac4e0813ec0e))

- **linguistics**: Add language auto-detection in SearchApplication ([#435](https://github.com/dorel14/whoosh-ng/pull/435), [`99f4290`](https://github.com/dorel14/whoosh-ng/commit/99f42904e302191de630cfdefe8eec6f5a8b50ca))

- **linguistics**: Add Wiktionary dictionary indexer (#429 #432) ([`726542d`](https://github.com/dorel14/whoosh-ng/commit/726542dd4bf8e79d35dadb36e126f1973207a8b4))

- **linguistics**: Add Wiktionary synonym provider (#425 #426 #427) ([`2961b93`](https://github.com/dorel14/whoosh-ng/commit/2961b93726cf1792faf89b293f2439c5c92e7ee6))

- **linguistics**: Integrate Wiktionary indexer with SearchApplication ([#433](https://github.com/dorel14/whoosh-ng/pull/433), [`f201f5c`](https://github.com/dorel14/whoosh-ng/commit/f201f5c37608df75dc79be5b2526fb71fa45ed36))

- **vector**: Add HnswlibProvider for EPIC 9 Sprint VEC-1 ([#406](https://github.com/dorel14/whoosh-ng/pull/406), [`cd590d4`](https://github.com/dorel14/whoosh-ng/commit/cd590d4601e5c238385be6fe693ee22ebe4a0912))

- **vector,embeddings,config**: Add HnswlibProvider, EmbeddingProvider, ConfigEngine extensions (#406 #407) ([`03a15aa`](https://github.com/dorel14/whoosh-ng/commit/03a15aabcb32822a71635a17b99e9c9545e07962))

---

**Detailed Changes**: [v5.2.0...v5.3.0](https://github.com/dorel14/whoosh-ng/compare/v5.2.0...v5.3.0)

### Commits

### Chores

- merge master into dev [skip ci]
- améliorer la qualité du code et nettoyer la configuration
- v5.3.0 [skip ci]

### Documentation

- update changelog and API docs [skip ci]
- corriger l'exemple ConfigEngine.build() (#422)
- auto-update llms context files [skip ci]
- restructurer la documentation et déplacer les fonctionnalités classiques vers Core
- auto-update llms context files [skip ci]
- documenter la détection automatique de langue et l'indexation multilingue
- auto-update llms context files [skip ci]
- auto-update llms context files [skip ci]
- mettre à jour la documentation pour le renommage en whoosh-ng
- auto-update llms context files [skip ci]
- auto-update llms context files [skip ci]

### Features

- add Wiktionary synonym provider (#425 #426 #427)
- add Wiktionary dictionary indexer (#429 #432)
- integrate Wiktionary indexer with SearchApplication (#433)
- add language auto-detection for multilingual indexing (#434)
- add language auto-detection in SearchApplication (#435)
- add HnswlibProvider for EPIC 9 Sprint VEC-1 (#406)
- add HnswlibProvider, EmbeddingProvider, ConfigEngine extensions (#406 #407)
- add EPIC 5.6 Modern N-Gram Framework Sprint NGRAM-1 (#440)
- exposer les nouvelles fonctionnalités pour la version 5.1.0

### Other

- eat(linguistics): complete EPIC 6.7 Language Auto-Detection & Multilingual Indexing (#430)
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge pull request #454 from dorel14/dev

### Code Refactoring

- améliorer le typage et nettoyer les imports

### Bug Fixes

- corriger les défauts signalés par revue et docs


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v5.3.0)

## v5.2.0 (2026-08-12)
**Tag**: `v5.2.0`

## v5.2.0 (2026-08-12)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **config**: Corriger le passage de booléen à StandardAnalyzer.stoplist ([#422](https://github.com/dorel14/whoosh-ng/pull/422), [`7c3649c`](https://github.com/dorel14/whoosh-ng/commit/7c3649c268d0551793c52f5855b6f1b7cab06a01))

- **config**: Corriger les bugs et améliorer les tests et la documentation ([#422](https://github.com/dorel14/whoosh-ng/pull/422), [`377a531`](https://github.com/dorel14/whoosh-ng/commit/377a531a9e20c7534bb8c54803fd91910b8e2323))

- **config**: Corriger pyright, unused imports et StandardAnalyzer.stoplist ([#422](https://github.com/dorel14/whoosh-ng/pull/422), [`10192bb`](https://github.com/dorel14/whoosh-ng/commit/10192bbd8c7824c3ccd89807c5bc6699c52a29b7))

### Documentation

- Auto-update llms context files [skip ci] ([`d188581`](https://github.com/dorel14/whoosh-ng/commit/d188581f5ca7fd71182ac26f71d06e0d045c2f0f))

- Auto-update llms context files [skip ci] ([`599c979`](https://github.com/dorel14/whoosh-ng/commit/599c979675149203172e73efa587578ffa4dab4f))

- Auto-update llms context files [skip ci] ([`7405bd0`](https://github.com/dorel14/whoosh-ng/commit/7405bd03e8445f547830b461de274d1f73a84cb9))

- Update changelog and API docs [skip ci] ([`9bc3cfd`](https://github.com/dorel14/whoosh-ng/commit/9bc3cfd033c2999aa6dd73ff15bc992c9883dccf))

- Update changelog and API docs [skip ci] ([`2920f65`](https://github.com/dorel14/whoosh-ng/commit/2920f653921efae2075911bd0f9c9a3de43c6808))

- Update changelog and API docs [skip ci] ([`dc32e6f`](https://github.com/dorel14/whoosh-ng/commit/dc32e6fb7b16659c2757ce86ed38c6ee43b8b9df))

### Features

- **config**: Add ConfigEngine sub-engines for Sprint CFG-1 ([#422](https://github.com/dorel14/whoosh-ng/pull/422), [`47c59b4`](https://github.com/dorel14/whoosh-ng/commit/47c59b4d0d66f2105de4de3e0dee5501133ff96e))

- **config**: Add ConfigEngine.build() orchestration for Sprint CFG-1 ([#422](https://github.com/dorel14/whoosh-ng/pull/422), [`1329b42`](https://github.com/dorel14/whoosh-ng/commit/1329b4296a35e2b115580bcad95ac83a394e8911))

- **config**: Ajouter la mise en cache et valider les sources de données ([`add4bb8`](https://github.com/dorel14/whoosh-ng/commit/add4bb8cc4b8d7fcaf4ed06b5c20b93bd3fa312c))

---

**Detailed Changes**: [v5.1.0...v5.2.0](https://github.com/dorel14/whoosh-ng/compare/v5.1.0...v5.2.0)

### Commits

### Chores

- merge master into dev [skip ci]
- synchronize version from pyproject.toml [skip ci]
- merge master into dev [skip ci]
- merge master into dev [skip ci]
- merge master into dev [skip ci]
- synchronize version from pyproject.toml [skip ci]
- apply pre-commit fixes
- v5.2.0 [skip ci]

### Other

- Delete LICENSE_OLD.txt
- Add MIT License to the project
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge pull request #424 from dorel14/dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- style(codebase): uniformiser le formatage du code et améliorer la lisibilité
- .
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- .
- Merge pull request #428 from dorel14/dev

### CI/CD

- restructurer la génération du changelog et simplifier Pages
- documenter le jeton GH_TOKEN et ajouter pydoctor
- documenter le jeton GH_TOKEN et ajouter pydoctor (#422)
- ajouter pydoctor, documenter GH_TOKEN et ignorer les benchmarks JSON (#422)
- factoriser should_release et rendre l'i18n dynamique (#422)

### Documentation

- auto-update llms context files [skip ci]
- update changelog and API docs [skip ci]
- update changelog and API docs [skip ci]
- update changelog and API docs [skip ci]
- auto-update llms context files [skip ci]
- auto-update llms context files [skip ci]

### Code Refactoring

- restructurer la chaîne de déploiement et supprimer les workflows obsolètes
- supprimer le benchmark stemmer français et moderniser les chemins

### Features

- add ConfigEngine sub-engines for Sprint CFG-1 (#422)
- add ConfigEngine.build() orchestration for Sprint CFG-1 (#422)
- ajouter la mise en cache et valider les sources de données

### Bug Fixes

- corriger le passage de booléen à StandardAnalyzer.stoplist (#422)
- corriger les bugs et améliorer les tests et la documentation (#422)
- corriger pyright, unused imports et StandardAnalyzer.stoplist (#422)


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v5.2.0)

## v5.1.0 (2026-08-11)
**Tag**: `v5.1.0`

## v5.1.0 (2026-08-11)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **config,linguistics**: Resolve mypy/pyright issues in config loader and yaml provider ([`3f59f96`](https://github.com/dorel14/whoosh-ng/commit/3f59f9607398547225b090a2eddc0fc7e1f62efd))

- **fastapi**: Correct WebSocket autocomplete test assertion ([`b9b1d57`](https://github.com/dorel14/whoosh-ng/commit/b9b1d57524b6f121bff60833c6b5563264bc4152))

- **storage**: Validate SnapshotStorage key before remote read ([`3ec0b43`](https://github.com/dorel14/whoosh-ng/commit/3ec0b43120e05578a4191d25e0d66b10ab06b914))

- **tests**: Import CoreStorageAdapter in test_storage_providers.py ([`18484b0`](https://github.com/dorel14/whoosh-ng/commit/18484b00c2e3f66e1baab76349d28694a3fde45a))

### Documentation

- Add Configuration Engine docs, update CHANGELOG, FastAPI WebSocket, CoreStorageAdapter, and SnapshotStorage fix ([`5e8ec4a`](https://github.com/dorel14/whoosh-ng/commit/5e8ec4ae49aaa64d66c554751f7cd67d9ca37f31))

- Auto-update llms context files ([`916bb34`](https://github.com/dorel14/whoosh-ng/commit/916bb34e16c4b5adf7cf6cc7a1449864db60873a))

- Auto-update llms context files ([`5b6b9a6`](https://github.com/dorel14/whoosh-ng/commit/5b6b9a63346559febe51c16a23bbe1602188a2c2))

- Auto-update llms context files ([`25f4477`](https://github.com/dorel14/whoosh-ng/commit/25f4477091e5506c58bfe6787e2fa7a6fcc22943))

- Auto-update llms context files [skip ci] ([`127bc4a`](https://github.com/dorel14/whoosh-ng/commit/127bc4aa3277c109f73d932e878be90807c4aafc))

- **config**: Clarify list merge behavior, refactor PyYAML import, document pre-push hook rationale ([`3d122d5`](https://github.com/dorel14/whoosh-ng/commit/3d122d58b85d10b85bf079ad9d3d15acdeab09e2))

- **config**: Clarify list merge rationale and make unsupported format error dynamic ([`25076b6`](https://github.com/dorel14/whoosh-ng/commit/25076b61ab91df44cd81327bf827732325fc1aab))

- **fastapi,config**: Clarify optional FastAPI dependency and warn on list merge behavior ([`e917567`](https://github.com/dorel14/whoosh-ng/commit/e917567509d1b1d84bb27b7ceb75b1b8c15941a6))

### Features

- Add PyYAML extra, make WebSocket limit configurable, validate ConfigEngine priority ([`5fb4a38`](https://github.com/dorel14/whoosh-ng/commit/5fb4a382ccda5e6408ec93ce1e6e2efa8fd583d7))

- **config**: Implement Configuration Engine core with Pydantic models, YAML/JSON loader, and hierarchical merging ([`de7bbca`](https://github.com/dorel14/whoosh-ng/commit/de7bbcadd384391adef090f1944865490b93d8fe))

- **fastapi**: Make WebSocket limit configurable and run autocomplete off the event loop ([`c538ea6`](https://github.com/dorel14/whoosh-ng/commit/c538ea6af2195bd292035c5de3e520577f090bc4))

- **storage**: Add CoreStorageAdapter wrapping core FileStorage for SyncStorageProvider ([`52ab7e6`](https://github.com/dorel14/whoosh-ng/commit/52ab7e614de1f92a7ac8d7afbf95bbc52aaba0e2))

- **website**: Ajouter les fichiers de configuration du site statique ([`103fcff`](https://github.com/dorel14/whoosh-ng/commit/103fcff73199a4e8e126f7f887d0f31bcd003999))

---

**Detailed Changes**: [v5.0.0...v5.1.0](https://github.com/dorel14/whoosh-ng/compare/v5.0.0...v5.1.0)

### Commits

### Code Refactoring

- Sprint D cleanup + P1-1/P3 dedup fixes
- improve WebSocket error handling and document list merge behavior

### Features

- add CoreStorageAdapter wrapping core FileStorage for SyncStorageProvider
- implement Configuration Engine core with Pydantic models, YAML/JSON loader, and hierarchical merging
- ajouter les fichiers de configuration du site statique
- add PyYAML extra, make WebSocket limit configurable, validate ConfigEngine priority
- make WebSocket limit configurable and run autocomplete off the event loop

### Bug Fixes

- correct WebSocket autocomplete test assertion
- validate SnapshotStorage key before remote read
- import CoreStorageAdapter in test_storage_providers.py
- resolve mypy/pyright issues in config loader and yaml provider

### Other

- .
- Rename LICENSE.txt to LICENSE_OLD.txt
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge pull request #418 from dorel14/dev

### Chores

- merge master into dev [skip ci]
- synchronize version from pyproject.toml [skip ci]
- apply pre-commit fixes
- apply pre-commit fixes
- v5.1.0 [skip ci]

### Documentation

- add Configuration Engine docs, update CHANGELOG, FastAPI WebSocket, CoreStorageAdapter, and SnapshotStorage fix
- auto-update llms context files
- auto-update llms context files
- clarify list merge behavior, refactor PyYAML import, document pre-push hook rationale
- auto-update llms context files
- clarify list merge rationale and make unsupported format error dynamic
- clarify optional FastAPI dependency and warn on list merge behavior
- auto-update llms context files [skip ci]

### CI/CD

- regenerate LLM context files only on documentation changes


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v5.1.0)

## v5.0.0 (2026-08-11)
**Tag**: `v5.0.0`

## v5.0.0 (2026-08-11)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **ci**: Ensure git push runs even when pre-commit commit is a no-op ([`b4fa710`](https://github.com/dorel14/whoosh-ng/commit/b4fa710bfb979a764373b1b9dc67e60f0a84ff0e))

- **ci**: Restore || true suppression for git pull --rebase in test workflow ([`6b8ae46`](https://github.com/dorel14/whoosh-ng/commit/6b8ae46f08908aa132b55261643589993de11ba9))

- **indexing**: Avoid NameError when ix.writer() fails in ParallelIndexBuilder ([`b629fed`](https://github.com/dorel14/whoosh-ng/commit/b629feda7398ece400636ef1ab7bf1df742c4255))

- **linguistics**: Restore constructor-style calls for language analyzers ([`8307731`](https://github.com/dorel14/whoosh-ng/commit/8307731b5958654307f6014743fe1e39700ba547))

- **s3**: Corriger l'ordre de validation des chemins dans SnapshotStorage ([`4fdf384`](https://github.com/dorel14/whoosh-ng/commit/4fdf384158249d66937f16976307db01f232ddf0))

- **storage**: Sanitize S3 keys in SnapshotStorage.read to prevent path traversal ([`b57b00d`](https://github.com/dorel14/whoosh-ng/commit/b57b00d84e2adc8d73e83a93bdc9813eeeff69bf))

### Documentation

- Auto-update llms context files ([`6318ddd`](https://github.com/dorel14/whoosh-ng/commit/6318dddda1db67f1d9ffc1492038340eafd2b62e))

- Auto-update llms context files ([`eb618ab`](https://github.com/dorel14/whoosh-ng/commit/eb618ab65aceb212b417a09d5c2ca74e3296bd0d))

### Features

- **workflows**: Amélioration des workflows CI et ajout de la section LLM Context ([`359e19a`](https://github.com/dorel14/whoosh-ng/commit/359e19a79002a1b67ea8faeef868a0cbe877612b))

---

**Detailed Changes**: [v4.3.0...v5.0.0](https://github.com/dorel14/whoosh-ng/compare/v4.3.0...v5.0.0)

### Commits

### Other

- Merge pull request #15 from dorel14/master
- Merge branch 'master' into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'master' into feat/last_updates
- Merge pull request #417 from dorel14/feat/last_updates

### Documentation

- auto-update llms context files
- rename sprint-c/d docs, add provider-integration guide, sync website docs
- auto-update llms context files
- auto-update llms context files
- auto-update llms context files
- auto-update llms context files

### Chores

- synchronize version from pyproject.toml [skip ci]
- synchronize version from pyproject.toml [skip ci]
- apply pre-commit fixes
- synchronize version from pyproject.toml [skip ci]
- v5.0.0 [skip ci]

### Code Refactoring

- modernisation architecturale et unification sur les composants core
- améliorations de la sécurité S3 et du typage statique

### CI/CD

- ajout de la détection des doublons de code

### Bug Fixes

- sanitize S3 keys in SnapshotStorage.read to prevent path traversal
- restore constructor-style calls for language analyzers
- corriger l'ordre de validation des chemins dans SnapshotStorage
- ensure git push runs even when pre-commit commit is a no-op
- restore || true suppression for git pull --rebase in test workflow
- avoid NameError when ix.writer() fails in ParallelIndexBuilder

### Features

- amélioration des workflows CI et ajout de la section LLM Context


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v5.0.0)

## v4.3.0 (2026-08-09)
**Tag**: `v4.3.0`

## v4.3.0 (2026-08-09)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- Add actions: read permission to workflow_run triggers ([`d1cb49c`](https://github.com/dorel14/whoosh-ng/commit/d1cb49c7d5d84b4d790090d8b24250bb0bf5791b))

- Normalize line endings to LF in workflow files ([`d68513b`](https://github.com/dorel14/whoosh-ng/commit/d68513b84b97115f1708412e731c6bfcc7a2b6b1))

- Use repository_dispatch instead of workflow_run for Pages trigger ([`c601ccb`](https://github.com/dorel14/whoosh-ng/commit/c601ccbc6469b4048083ab71c334f937422b2b2f))

- **pages**: Ensure Pages deploys on all master pushes + changelog sidebar fix ([`d5d297f`](https://github.com/dorel14/whoosh-ng/commit/d5d297f17ef941e5575f7f0ac7f5c99d0c188f76))

### Features

- **website**: Ajouter des sidebars dédiées et refondre la page de référence API ([`a69d3c9`](https://github.com/dorel14/whoosh-ng/commit/a69d3c9c9093fa78c246010f79d751ad36c8b9b0))

- **website**: Ajouter des sidebars dédiées par section de documentation ([`ffaf1a9`](https://github.com/dorel14/whoosh-ng/commit/ffaf1a969c53c80fa2d968f5f733b40ae9e2cc00))

---

**Detailed Changes**: [v4.2.3...v4.3.0](https://github.com/dorel14/whoosh-ng/compare/v4.2.3...v4.3.0)

### Commits

### Chores

- synchronize version from pyproject.toml [skip ci]
- synchronize version from pyproject.toml [skip ci]
- apply pre-commit fixes
- synchronize version from pyproject.toml [skip ci]
- v4.3.0 [skip ci]

### CI/CD

- restructurer la chaîne de déploiement CI/CD et optimiser les workflows
- ajouter des garde-fous et corriger le script de synchronisation de version
- améliorer la logique de déclenchement du déploiement GitHub Pages
- ajouter un déclenchement push pour les modifications de documentation

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng

### Features

- ajouter des sidebars dédiées par section de documentation
- ajouter des sidebars dédiées et refondre la page de référence API

### Code Refactoring

- simplifier la configuration des sidebars

### Bug Fixes

- normalize line endings to LF in workflow files
- ensure Pages deploys on all master pushes + changelog sidebar fix
- add actions: read permission to workflow_run triggers
- use repository_dispatch instead of workflow_run for Pages trigger


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v4.3.0)

## v4.2.3 (2026-08-08)
**Tag**: `v4.2.3`

## v4.2.3 (2026-08-08)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **ci**: Add concurrency groups to prevent workflow race conditions ([`76e38f0`](https://github.com/dorel14/whoosh-ng/commit/76e38f03045d19b8d8032f066432af4e9d17d133))

---

**Detailed Changes**: [v4.2.2...v4.2.3](https://github.com/dorel14/whoosh-ng/compare/v4.2.2...v4.2.3)

### Commits

### Chores

- synchronize version from pyproject.toml [skip ci]
- sync version to 4.2.1 [skip ci]
- v4.2.3 [skip ci]

### Other

- Merge: resolve sync-version conflicts (CI and local generated same changes)

### Bug Fixes

- add concurrency groups to prevent workflow race conditions


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v4.2.3)

## v4.2.2 (2026-08-08)
**Tag**: `v4.2.2`

## v4.2.2 (2026-08-08)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **ci**: Fix changelog workflow detached HEAD ([`db4b78e`](https://github.com/dorel14/whoosh-ng/commit/db4b78e6dc51a0e10fde48fd93eb2ce5d1dbecd0))

---

**Detailed Changes**: [v4.2.1...v4.2.2](https://github.com/dorel14/whoosh-ng/compare/v4.2.1...v4.2.2)

### Commits

### Chores

- synchronize version from pyproject.toml [skip ci]
- v4.2.2 [skip ci]

### Bug Fixes

- fix changelog workflow detached HEAD


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v4.2.2)

## v4.2.1 (2026-08-07)
**Tag**: `v4.2.1`

## v4.2.1 (2026-08-07)

_This release is published under the BSD-2-Clause License._

---

**Detailed Changes**: [v4.2.0...v4.2.1](https://github.com/dorel14/whoosh-ng/compare/v4.2.0...v4.2.1)

### Commits

### Bug Fixes

- fix sync-changelog.yml YAML syntax error

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng
- i18n(fr): translate nested.md, storage-providers.md, stemming.md, ngrams.md, glossary.md

### Chores

- synchronize version to 4.2.0 [skip ci]
- synchronize version from pyproject.toml [skip ci]
- v4.2.1 [skip ci]


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v4.2.1)

## v4.2.0 (2026-08-07)
**Tag**: `v4.2.0`

## v4.2.0 (2026-08-07)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **ci**: Fix changelog workflow typo and pages.yml checkout depth ([`d88b82b`](https://github.com/dorel14/whoosh-ng/commit/d88b82b68f7ea2ed28c4530cbda1ecad77308acf))

### Features

- **docs**: Enable dark theme by default and remove edit button ([`2c3d98b`](https://github.com/dorel14/whoosh-ng/commit/2c3d98b27087125073729505cba156d1c2578d66))

---

**Detailed Changes**: [v4.1.0...v4.2.0](https://github.com/dorel14/whoosh-ng/compare/v4.1.0...v4.2.0)

### Commits

### Chores

- synchronize version from pyproject.toml [skip ci]
- v4.2.0 [skip ci]

### Features

- enable dark theme by default and remove edit button

### Bug Fixes

- fix changelog workflow typo and pages.yml checkout depth


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v4.2.0)

## v4.1.0 (2026-08-07)
**Tag**: `v4.1.0`

## v4.1.0 (2026-08-07)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **ci**: Fix broken workflows and Docusaurus build errors after migration ([`0e86e6c`](https://github.com/dorel14/whoosh-ng/commit/0e86e6c51e3d41fec984f0f79a4457e9d5d0b1ef))

- **ci**: Fix sync_version.py NameError and missing imports ([`f809f56`](https://github.com/dorel14/whoosh-ng/commit/f809f563851b102c7220cec0371c560191bdb217))

### Features

- **docs**: Migrate Jekyll/Just the Docs to Docusaurus v3 ([`7e70791`](https://github.com/dorel14/whoosh-ng/commit/7e70791d23fb0f3097e3603ba0ff3fa5c8d822c2))

---

**Detailed Changes**: [v4.0.1...v4.1.0](https://github.com/dorel14/whoosh-ng/compare/v4.0.1...v4.1.0)

### Commits

### Chores

- synchronize version from pyproject.toml [skip ci]
- v4.1.0 [skip ci]

### Features

- migrate Jekyll/Just the Docs to Docusaurus v3

### Bug Fixes

- fix broken workflows and Docusaurus build errors after migration
- fix sync_version.py NameError and missing imports


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v4.1.0)

## v4.0.1 (2026-08-07)
**Tag**: `v4.0.1`

## v4.0.1 (2026-08-07)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **pages**: Pointer Bundler sur docs/Gemfile via BUNDLE_GEMFILE ([`0f3a8f1`](https://github.com/dorel14/whoosh-ng/commit/0f3a8f11087c5df910d1762f31ae1cee4017ed31))

---

**Detailed Changes**: [v4.0.0...v4.0.1](https://github.com/dorel14/whoosh-ng/compare/v4.0.0...v4.0.1)

### Commits

### Chores

- synchronize version from pyproject.toml [skip ci]
- v4.0.1 [skip ci]

### Bug Fixes

- pointer Bundler sur docs/Gemfile via BUNDLE_GEMFILE


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v4.0.1)

## v4.0.0 (2026-08-07)
**Tag**: `v4.0.0`

## v4.0.0 (2026-08-07)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **ci**: Corriger les erreurs de lint, mypy et pyright ([`4283355`](https://github.com/dorel14/whoosh-ng/commit/428335590e641870f6bc399b241b453745e4b9a3))

---

**Detailed Changes**: [v3.0.0...v4.0.0](https://github.com/dorel14/whoosh-ng/compare/v3.0.0...v4.0.0)

### Commits

### Features

- restructurer le système de middleware et étendre PluginManager
- ajouter le module linguistics avec analyseurs multilingues
- add asearch/awriter bridges and AsyncFileStorage
- enhance FastAPI models and Admin Studio modules
- add SearchApplication and FileStorage exports
- add S3Storage, HybridStorage, AsyncHybridStorage
- add SnapshotStorage, CachedObjectStorage alias, and Phase 3 roadmap
- publier la version 3.0.0 et ajouter la documentation LLM

### Documentation

- ajouter les guides Whoosh-NG 2.0 et ajuster la configuration de release
- add Gemfile.lock for reproducible Jekyll build and fix French index permalinks
- add S3 storage benchmarks and documentation
- auto-update llms context files

### Chores

- restructurer les workflows CI/CD et nettoyer le code
- apply pre-commit fixes
- v4.0.0 [skip ci]

### Other

- Merge branch 'master' into dev
- Merge pull request #14 from dorel14/dev

### Bug Fixes

- corriger les erreurs de lint, mypy et pyright


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v4.0.0)

## v3.0.0 (2026-08-06)
**Tag**: `v3.0.0`

## v3.0.0 (2026-08-06)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- Ajouter reportAssignmentType en warning dans pyrightconfig.json ([`3259c76`](https://github.com/dorel14/whoosh-ng/commit/3259c76660a4f99135a8cda1ac26baa3f642a873))

- Corriger docstring PathTokenizer et restaurer Literal pour engine ([`b90222e`](https://github.com/dorel14/whoosh-ng/commit/b90222e5962671dfb20f496d396f546f33c95747))

- Mapper engine pyarrow vers auto pour pandas et exclure tests/ de mypy ([`dadeb91`](https://github.com/dorel14/whoosh-ng/commit/dadeb91cffd9462d0c5c2aff77fd36f4cbac8f4a))

- Ne pas yield de token vide dans RegexTokenizer gaps=True ([`6385a24`](https://github.com/dorel14/whoosh-ng/commit/6385a24cb048777aef9cc5d2bad46f2e5d2e5dfa))

- Remove unreachable dead code after build() return in parallel_builder ([`c2e03c9`](https://github.com/dorel14/whoosh-ng/commit/c2e03c97bda6ac61d78c38c6ac3ad37df4f9345a))

- Rendre test_buffered_threads deterministe en couverture des valeurs ([`77a66f6`](https://github.com/dorel14/whoosh-ng/commit/77a66f6320f9ff7bfe8a1c787c52d1bb0e3cf0d6))

- Resolve mypy errors in CI (parquet, tortoise, hnswlib) ([`514f1f6`](https://github.com/dorel14/whoosh-ng/commit/514f1f62d55edd92633654e0096caf920a9ff3a6))

- Réintégrer assert segment_reader is not None après merge distant ([`c42b065`](https://github.com/dorel14/whoosh-ng/commit/c42b065355bb0004094223748a7c9b6eca810405))

- Résoudre les erreurs pyright dans pre-commit (Token typing, reportAttributeAccessIssue) ([`eff5adf`](https://github.com/dorel14/whoosh-ng/commit/eff5adf55cbbae6169bf2efacc76fddacc1891dd))

- Résoudre les erreurs pyright reportOptionalMemberAccess et reportAssignmentType ([`572529e`](https://github.com/dorel14/whoosh-ng/commit/572529e1fb7f0828f3729ed7920e219a71df139c))

- Supprimer cast redondant et ajouter reportAssignmentType en warning ([`0834923`](https://github.com/dorel14/whoosh-ng/commit/0834923f185a4fb62ded3e5020ea6d36b90bfc42))

- **analysis**: Corriger l'indentation du RegexTokenizer et ajuster les annotations de type ([`69e9ad8`](https://github.com/dorel14/whoosh-ng/commit/69e9ad815192ed9113acea4075e21eaf9e514f8d))

- **deps**: Retirer les modules obsolètes des exclusions mypy ([`f53e157`](https://github.com/dorel14/whoosh-ng/commit/f53e15729647ac7473066f193a33fcbfb2a3dce0))

- **indexing**: Close segment_ix in ParallelIndexBuilder to prevent fd leak ([`bad2928`](https://github.com/dorel14/whoosh-ng/commit/bad2928155d424e19de24e5b00822831695e7449))

- **indexing**: Corriger les fuites de handles et erreurs de nettoyage sous Windows ([`3cb85c7`](https://github.com/dorel14/whoosh-ng/commit/3cb85c7d9f80eccdbb5d2225d270d9f4c15b79bd))

- **indexing**: Merge parallel worker segments into main index ([`6f18248`](https://github.com/dorel14/whoosh-ng/commit/6f182488b18aedaceaf4c886b1b0d87eadc02a99))

- **indexing**: Merge worker segments into main index in ParallelIndexBuilder ([`27b0541`](https://github.com/dorel14/whoosh-ng/commit/27b054140d1a99228a804b75a1d4efd3e4f300ce))

- **mypy**: Restore ignore_missing_imports for pytest, peewee, httpx, re2, psutil ([`699ee4f`](https://github.com/dorel14/whoosh-ng/commit/699ee4f2fb3007012913d5970376f97fc99a0296))

- **profiling**: Add segment_write() and sibling step context managers to CommitProfilerV2 ([`599b650`](https://github.com/dorel14/whoosh-ng/commit/599b6507149332b3e0522b753e547bae718b468e))

- **profiling**: Implement SegmentProfiler to resolve NameError in benchmark.py ([`1b03b3a`](https://github.com/dorel14/whoosh-ng/commit/1b03b3ae7fbc3d03c7223ab4f7759eb500c9c8bf))

### Documentation

- Auto-update llms context files ([`5cca243`](https://github.com/dorel14/whoosh-ng/commit/5cca243cba55d49836e89793af3ac126815b862c))

- Auto-update llms context files ([`55825c1`](https://github.com/dorel14/whoosh-ng/commit/55825c15502f0b15041aec9ac15d7f9c4bc23e6c))

- Auto-update llms context files ([`c0d74a4`](https://github.com/dorel14/whoosh-ng/commit/c0d74a4cda58a44898a0b3f67fdc009fefa565fe))

- Restructurer la documentation et ajouter les pages API et guides ([`4e9b1e2`](https://github.com/dorel14/whoosh-ng/commit/4e9b1e2ce8a494c66f1725cec3b03e5025643e8d))

- **guides**: Ajouter le guide d'indexation moderne et mettre à jour les index de documentation ([`9fbc4ff`](https://github.com/dorel14/whoosh-ng/commit/9fbc4ff41dafc184219f0b4f1da3ee4f3e0ecb71))

### Features

- Ajouter FastCSVSource, indexation par lots, infrastructure de profiling et optimisations du cœur ([`fc63f9c`](https://github.com/dorel14/whoosh-ng/commit/fc63f9c890155fa9ce0b25c03e9db733eb7f0139))

- **analysis**: Ajouter le système de stemmers, FastCSVSource et l'infrastructure de profiling des performances ([`a077319`](https://github.com/dorel14/whoosh-ng/commit/a077319a8fc7435ef283b5c2b5402538c6550819))

- **core**: Ajouter CacheMiddleware et ObservableDataSource ([`508589a`](https://github.com/dorel14/whoosh-ng/commit/508589a8c86c7d1eadeb6ffd044a340fe84363be))

- **data-sources**: Ajouter les sources de données et le pooling de connexions ([`51805d4`](https://github.com/dorel14/whoosh-ng/commit/51805d4c08109b78704f6938deb4c135f57b0674))

- **data-sources**: Améliorer la robustesse et la validation des sources de données ([`35281b5`](https://github.com/dorel14/whoosh-ng/commit/35281b534865111e30d0d1195c492d72b8769e75))

- **profiling**: Ajouter les groupes d'options profiling et fast-stemming, stream_batches et restructurer les chemins d'import des sources de données ([`152e9c4`](https://github.com/dorel14/whoosh-ng/commit/152e9c44b68380a9d238489a29ebfd52e28f97a9))

---

**Detailed Changes**: [v2.0.0...v3.0.0](https://github.com/dorel14/whoosh-ng/compare/v2.0.0...v3.0.0)

### Commits

### Other

- Remove workflows permission from test.yml
- Revise README for version 2.0.0 updates
- Merge pull request #12 from dorel14/master
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Potential fix for pull request finding 'CodeQL / Workflow does not contain permissions'
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- .
- Update src/whoosh_modern/indexing/parallel_builder.py
- codec/base: restore missing out-of-order term check in add_postings
- Update src/whoosh_modern/profiling/segment_profiler.py
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge c42b065355bb0004094223748a7c9b6eca810405 into 9988770df03a2d255a21b74843772776dbbf998a
- Merge pull request #13 from dorel14/dev
- Update commit_parser_options in pyproject.toml
- Add files via upload

### Build System

- automatiser la synchronisation de la version entre les fichiers du projet

### Chores

- apply pre-commit fixes
- simplifier la configuration type-checking et retirer les dépendances inutilisées
- apply pre-commit fixes
- apply pre-commit fixes
- trigger pre-commit workflow
- v3.0.0 [skip ci]

### Features

- ajouter CacheMiddleware et ObservableDataSource
- ajouter les sources de données et le pooling de connexions
- ajouter FastCSVSource, indexation par lots, infrastructure de profiling et optimisations du cœur
- ajouter le système de stemmers, FastCSVSource et l'infrastructure de profiling des performances
- ajouter les groupes d'options profiling et fast-stemming, stream_batches et restructurer les chemins d'import des sources de données
- améliorer la robustesse et la validation des sources de données

### Code Refactoring

- simplifier les expressions multi-lignes et optimiser Token avec __slots__
- nettoyer les annotations de type et supprimer les dépendances inutilisées

### Documentation

- auto-update llms context files
- restructurer la documentation et ajouter les pages API et guides
- auto-update llms context files
- ajouter le guide d'indexation moderne et mettre à jour les index de documentation
- auto-update llms context files

### Bug Fixes

- corriger l'indentation du RegexTokenizer et ajuster les annotations de type
- resolve mypy errors in CI (parquet, tortoise, hnswlib)
- ne pas yield de token vide dans RegexTokenizer gaps=True
- résoudre les erreurs pyright dans pre-commit (Token typing, reportAttributeAccessIssue)
- supprimer cast redondant et ajouter reportAssignmentType en warning
- ajouter reportAssignmentType en warning dans pyrightconfig.json
- corriger docstring PathTokenizer et restaurer Literal pour engine
- mapper engine pyarrow vers auto pour pandas et exclure tests/ de mypy
- remove unreachable dead code after build() return in parallel_builder
- add segment_write() and sibling step context managers to CommitProfilerV2
- implement SegmentProfiler to resolve NameError in benchmark.py
- merge parallel worker segments into main index
- merge worker segments into main index in ParallelIndexBuilder
- rendre test_buffered_threads deterministe en couverture des valeurs
- retirer les modules obsolètes des exclusions mypy
- close segment_ix in ParallelIndexBuilder to prevent fd leak
- résoudre les erreurs pyright reportOptionalMemberAccess et reportAssignmentType
- réintégrer assert segment_reader is not None après merge distant
- restore ignore_missing_imports for pytest, peewee, httpx, re2, psutil
- corriger les fuites de handles et erreurs de nettoyage sous Windows


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v3.0.0)

## v2.0.0 (2026-07-31)
**Tag**: `v2.0.0`

## v2.0.0 (2026-07-31)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- Address review findings in CI/CD workflows and documentation ([`b1e0b89`](https://github.com/dorel14/whoosh-ng/commit/b1e0b89f10ad182ad0120d6d19bcac2f8a04470a))

### Documentation

- Auto-update llms context files ([`10e6c90`](https://github.com/dorel14/whoosh-ng/commit/10e6c90fb6e7ffd9ec9f1b3fa7f4e398a933077b))

- Auto-update llms context files ([`992e4d7`](https://github.com/dorel14/whoosh-ng/commit/992e4d780fd7829bd804d68b215e06f255e9cacf))

- Auto-update llms context files ([`5dfe9dc`](https://github.com/dorel14/whoosh-ng/commit/5dfe9dcfdd725d1ca12f3bba0c09a2f7d163aa88))

- Auto-update llms context files ([`aadce99`](https://github.com/dorel14/whoosh-ng/commit/aadce99460c30cd5950e44c927c07eb6fa5ffde8))

### Features

- **deps**: Add sqlalchemy and sqlmodel to models extra and configure mypy overrides ([`4486b5f`](https://github.com/dorel14/whoosh-ng/commit/4486b5feef2700fd8720a0dec1419985f9479951))

- **models**: Ajouter AutoIndexer et améliorer la génération de schémas ([`919131c`](https://github.com/dorel14/whoosh-ng/commit/919131ce7f475b8584e74d98d9cf41e04891b7c7))

- **models**: ✨ Introduce ModelIndex and SearchField for auto-mapping ([`6c202bc`](https://github.com/dorel14/whoosh-ng/commit/6c202bc6c4e20ad9dba7e4cd81deadfdaf3dcf2a))

- **whoosh_modern**: Add modern API with data sources, schema discovery, facets, validation, middleware, and SearchView ([`36bfbae`](https://github.com/dorel14/whoosh-ng/commit/36bfbaebd6acc18cda06aba83bebade7154c9972))

---

**Detailed Changes**: [v1.3.3...v2.0.0](https://github.com/dorel14/whoosh-ng/compare/v1.3.3...v2.0.0)

### Commits

### Documentation

- merge duplicate FastAPI example into fastapi-search.md
- auto-update llms context files
- auto-update llms context files
- auto-update llms context files
- auto-update llms context files

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng
- ..
- Merge pull request #10 from dorel14/master
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- style(benchmark): ajouter un saut de ligne final manquant dans reuters_modern.py
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Potential fix for pull request finding 'CodeQL / Workflow does not contain permissions'
- Potential fix for pull request finding 'CodeQL / Workflow does not contain permissions'
- Potential fix for pull request finding 'CodeQL / Workflow does not contain permissions'
- .
- .
- .
- Merge pull request #11 from dorel14/dev

### Features

- ✨ Introduce ModelIndex and SearchField for auto-mapping
- ajouter AutoIndexer et améliorer la génération de schémas
- add modern API with data sources, schema discovery, facets, validation, middleware, and SearchView
- add sqlalchemy and sqlmodel to models extra and configure mypy overrides

### Bug Fixes

- address review findings in CI/CD workflows and documentation

### Code Refactoring

- moderniser les annotations de type avec la syntaxe union PEP 604
- moderniser les annotations de type avec Coroutine et ajouter des ignores pyright
- ajouter des annotations de retour aux méthodes replace des matchers
- moderniser la vérification isinstance avec la syntaxe union PEP 604

### CI/CD

- ajouter des extras d'installation et simplifier la couverture

### Chores

- v2.0.0 [skip ci]


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v2.0.0)

## v1.3.3 (2026-07-26)
**Tag**: `v1.3.3`

## v1.3.3 (2026-07-26)

_This release is published under the BSD-2-Clause License._

---

**Detailed Changes**: [v1.3.2...v1.3.3](https://github.com/dorel14/whoosh-ng/compare/v1.3.2...v1.3.3)

### Commits

### Bug Fixes

- reorganize nav_order for coherent navigation (Guides 1-90, API 100-190, Examples 200-270)

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng

### Chores

- v1.3.3 [skip ci]


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.3.3)

## v1.3.2 (2026-07-26)
**Tag**: `v1.3.2`

## v1.3.2 (2026-07-26)

_This release is published under the BSD-2-Clause License._

---

**Detailed Changes**: [v1.3.1...v1.3.2](https://github.com/dorel14/whoosh-ng/compare/v1.3.1...v1.3.2)

### Commits

### Bug Fixes

- remove color_scheme from individual pages, use global config

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng

### Chores

- v1.3.2 [skip ci]


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.3.2)

## v1.3.1 (2026-07-26)
**Tag**: `v1.3.1`

## v1.3.1 (2026-07-26)

_This release is published under the BSD-2-Clause License._

---

**Detailed Changes**: [v1.3.0...v1.3.1](https://github.com/dorel14/whoosh-ng/compare/v1.3.0...v1.3.1)

### Commits

### Bug Fixes

- align _config.yml with taskiq-flow and clean README front matter

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng

### Chores

- v1.3.1 [skip ci]


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.3.1)

## v1.3.0 (2026-07-26)
**Tag**: `v1.3.0`

## v1.3.0 (2026-07-26)

_This release is published under the BSD-2-Clause License._

---

**Detailed Changes**: [v1.2.4...v1.3.0](https://github.com/dorel14/whoosh-ng/compare/v1.2.4...v1.3.0)

### Commits

### Features

- éviter les exécutions inutiles du workflow lors des commits de release

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng

### Chores

- v1.3.0 [skip ci]


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.3.0)

## v1.2.4 (2026-07-26)
**Tag**: `v1.2.4`

## v1.2.4 (2026-07-26)

_This release is published under the BSD-2-Clause License._

---

**Detailed Changes**: [v1.2.3...v1.2.4](https://github.com/dorel14/whoosh-ng/compare/v1.2.3...v1.2.4)

### Commits

### Bug Fixes

- remove invalid parent fields and align config with taskiq-flow

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng

### Chores

- v1.2.4


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.2.4)

## v1.2.3 (2026-07-26)
**Tag**: `v1.2.3`

## v1.2.3 (2026-07-26)

_This release is published under the BSD-2-Clause License._

---

**Detailed Changes**: [v1.2.2...v1.2.3](https://github.com/dorel14/whoosh-ng/compare/v1.2.2...v1.2.3)

### Commits

### Bug Fixes

- remove invalid parent fields from all pages

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng

### Chores

- v1.2.3


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.2.3)

## v1.2.2 (2026-07-26)
**Tag**: `v1.2.2`

## v1.2.2 (2026-07-26)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **docs**: Align _config.yml with taskiq-flow pattern ([`2124be0`](https://github.com/dorel14/whoosh-ng/commit/2124be0cf1fd4669c9ebdaa4ad485eabae22c279))

---

**Detailed Changes**: [v1.2.1...v1.2.2](https://github.com/dorel14/whoosh-ng/compare/v1.2.1...v1.2.2)

### Commits

### Code Refactoring

- supprimer la navigation statique codée en dur

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng

### Bug Fixes

- align _config.yml with taskiq-flow pattern

### Chores

- v1.2.2


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.2.2)

## v1.2.1 (2026-07-26)
**Tag**: `v1.2.1`

## v1.2.1 (2026-07-26)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **docs**: Restore front matter and add explicit nav config ([`74dc151`](https://github.com/dorel14/whoosh-ng/commit/74dc151104a7a89ffd1459a60e0e28488bf110ff))

### Documentation

- Fix Jekyll links with relative_url and clean deploy workflow ([`2f9afc2`](https://github.com/dorel14/whoosh-ng/commit/2f9afc20f5e41022117a636b0d223f92c660df3d))

---

**Detailed Changes**: [v1.2.0...v1.2.1](https://github.com/dorel14/whoosh-ng/compare/v1.2.0...v1.2.1)

### Commits

### Documentation

- fix Jekyll links with relative_url and clean deploy workflow

### Bug Fixes

- restore front matter and add explicit nav config

### Chores

- v1.2.1


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.2.1)

## v1.2.0 (2026-07-26)
**Tag**: `v1.2.0`

## v1.2.0 (2026-07-26)

_This release is published under the BSD-2-Clause License._

### Documentation

- Ajouter la documentation complète bilingue et le déploiement GitHub Pages ([`a091ced`](https://github.com/dorel14/whoosh-ng/commit/a091cede82bb5d722e5db0ff233c1dcbcecd35ea))

- Auto-update llms context files ([`96e4f2c`](https://github.com/dorel14/whoosh-ng/commit/96e4f2c251ccdde5e9fd7e9cb0ce6b2650ab64d9))

- Auto-update llms context files ([`6abefe4`](https://github.com/dorel14/whoosh-ng/commit/6abefe4b84d99e29c1d6ccb471dd7bf27ea4f31d))

### Features

- **docs**: Ajouter le support multilingue dans la configuration ([`967ed12`](https://github.com/dorel14/whoosh-ng/commit/967ed125e5fcd167e237e1c32b7feeb3ab6f9c4f))

---

**Detailed Changes**: [v1.1.0...v1.2.0](https://github.com/dorel14/whoosh-ng/compare/v1.1.0...v1.2.0)

### Commits

### CI/CD

- restructurer le workflow de release sémantique

### Other

- Merge pull request #7 from dorel14/dev
- revert: supprimer la documentation complète bilingue et restaurer l'état précédent
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge pull request #8 from dorel14/dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge pull request #9 from dorel14/dev

### Documentation

- ajouter la documentation complète bilingue et le déploiement GitHub Pages
- auto-update llms context files
- auto-update llms context files

### Features

- ajouter le support multilingue dans la configuration

### Chores

- v1.2.0


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.2.0)

## v1.1.0 (2026-07-26)
**Tag**: `v1.1.0`

## v1.1.0 (2026-07-26)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **bench**: 🐛 add type ignores for method overrides in `XappyModule` ([`822e957`](https://github.com/dorel14/whoosh-ng/commit/822e957c986f603e9fd1c2de2799fb60d3c122a9))

- **schema**: 🐛 corriger l'initialisation de l'objet dans `__new__` ([`5a764a7`](https://github.com/dorel14/whoosh-ng/commit/5a764a78751a6a2bb94118340784da074fdcf2c8))

- **stress**: 🐛 ensure correct handling of string encoding in `test_bigtable` ([`822e957`](https://github.com/dorel14/whoosh-ng/commit/822e957c986f603e9fd1c2de2799fb60d3c122a9))

### Build System

- **deps**: Supprimer les dépendances obsolètes de la section models ([`227a59f`](https://github.com/dorel14/whoosh-ng/commit/227a59fc195799aca02289c2cb5f3d60135ea063))

### Features

- **benchmark**: Refonte du système de benchmarks avec nouvelles spécifications ([`417efd1`](https://github.com/dorel14/whoosh-ng/commit/417efd1be034fe8be4377ad2716e7b15028ad929))

- **matching**: ✨ add type hints for `supports_block_quality` methods ([`822e957`](https://github.com/dorel14/whoosh-ng/commit/822e957c986f603e9fd1c2de2799fb60d3c122a9))

- **support**: ✨ add compatibility for Python 3 unicode handling ([`822e957`](https://github.com/dorel14/whoosh-ng/commit/822e957c986f603e9fd1c2de2799fb60d3c122a9))

- **writing**: ✨ Implement segment writing and merging policies ([`ae5fc62`](https://github.com/dorel14/whoosh-ng/commit/ae5fc62d2185ac349b56cbbf006bbb7fd4f92c80))

---

**Detailed Changes**: [v1.0.0...v1.1.0](https://github.com/dorel14/whoosh-ng/compare/v1.0.0...v1.1.0)

### Commits

### Chores

- ✏️ Mise à jour de la version dans le README
- prepare whoosh-ng 1.0.0
- bump version to 1.0.1
- 🔄 Update project dependencies
- apply pre-commit fixes
- apply pre-commit fixes
- v1.1.0

### Other

- Potential fix for code scanning alert no. 1: Workflow does not contain permissions
- Potential fix for code scanning alert no. 1: Workflow does not contain permissions
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge pull request #5 from dorel14/alert-autofix-1
- Merge branch 'master' into dev
- Potential fix for pull request finding 'CodeQL / Workflow does not contain permissions'
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge 3d0187143ec5a2a4e19f62d26a622d2c53efc2c7 into 68e67aaefb0d48f136d53576e00f10d68f77f15a
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge 4473223d510f0cde2d30ca638d284f23dc467b14 into 68e67aaefb0d48f136d53576e00f10d68f77f15a
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge pull request #6 from dorel14/dev

### Features

- ✨ Ajout de la validation de l'intégrité des segments
- ✨ Ajout d'un wrapper Asyncio pour la gestion asynchrone des écritures
- ✨ Isolation des espaces de stockage temporaires pour les écrivains concurrents
- ✨ Ajout de la normalisation des boosts pour les sous-requêtes
- ✨ Ajout d'un module de reporting pour les résultats de benchmark
- ✨ Ajout d'un backend LMDB et d'un support d'autocomplétion
- ✨ Implement segment writing and merging policies
- refonte du système de benchmarks avec nouvelles spécifications
- ✨ add type hints for `supports_block_quality` methods

### Bug Fixes

- 🐛 Ajout d'un type d'ignore pour l'appel de la requête
- 🐛 Amélioration des benchmarks avec un échauffement et ajustement des seuils d'alerte
- 🐛 corriger l'initialisation de l'objet dans `__new__`

### Code Refactoring

- improve _posting_size estimation, fix benchmark CLI, update mypy target to 3.12

### Build System

- supprimer les dépendances obsolètes de la section models


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.1.0)

## v1.0.0 (2026-07-12)
**Tag**: `v1.0.0`

## v1.0.0 (2026-07-12)

_This release is published under the BSD-2-Clause License._

- Initial Release


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.0.0)


## DOCUMENT: Core Concepts

# Core Concepts

Whoosh-NG is a pure-Python search engine library. This guide explains the main concepts you need to understand to use it effectively.

## Architecture

Whoosh-NG follows a layered architecture:

![Architecture diagram: Application → Whoosh-NG Core (Search Engine, Schema, Plugin Manager, Registry, Middleware, Event Bus, Hooks) → Plugins (FastAPI, Vector Search, Autocomplete, PostgreSQL, S3, Monitoring, Admin UI)](/assets/architecture.svg)

## Key Components

### Index

An `Index` is the top-level container for your searchable documents. It manages one or more segments on disk.

```python
from whoosh.index import create_in, open_dir

# Create a new index
ix = create_in("indexdir", schema)

# Open an existing index
ix = open_dir("indexdir")
```

### Schema

The `Schema` defines the fields that documents in your index can have. Each field has a type that determines how it is indexed and stored.

```python
from whoosh.fields import Schema, TEXT, ID, NUMERIC

schema = Schema(
    title=TEXT(stored=True),
    path=ID(stored=True, unique=True),
    content=TEXT,
    rating=NUMERIC(float, stored=True)
)
```

### Writer

An `IndexWriter` lets you add, update, and delete documents in the index.

```python
writer = ix.writer()
writer.add_document(title="Hello", content="World")
writer.commit()
```

### Searcher

A `Searcher` lets you query the index and retrieve results.

```python
with ix.searcher() as s:
    results = s.search("hello")
```

### Query Parser

The `QueryParser` converts a query string into a query object that the searcher can execute.

```python
from whoosh.qparser import QueryParser

qp = QueryParser("content", schema)
query = qp.parse("hello world")
```

## Modern Features

### Plugin System

Plugins extend Whoosh-NG without modifying the core. Plugins can:

- Register new vector providers
- Add FastAPI endpoints
- Provide custom analyzers
- Hook into the middleware pipeline

```python
from whoosh.plugins.manager import PluginManager

# Load plugins from entry points
PluginManager.load_plugins()

# Or register manually
PluginManager.register("my_plugin", MyPlugin())
```

### Middleware Pipeline

Middleware intercepts indexing and search operations:

```python
from whoosh.middleware import Middleware, MiddlewareContext

class LoggingMiddleware(Middleware):
    def before_search(self, context: MiddlewareContext):
        print(f"Searching: {context.query}")
        return context

    def after_search(self, context: MiddlewareContext):
        print(f"Found: {len(context.results) if context.results else 0} results")
        return context
```

### Vector Search

Vector fields enable semantic search using embeddings:

```python
from whoosh.fields import Schema, TEXT, VectorField

schema = Schema(
    content=TEXT,
    embedding=VectorField(dimensions=384)
)
```

### Event Bus

The event system allows loose coupling between components:

```python
from whoosh.event_bus import EventBus, DocumentIndexed

bus = EventBus()

@bus.subscribe
def on_document_indexed(event: DocumentIndexed):
    print(f"Document indexed: {event.docnum}")
```

## Data Flow

### Indexing Flow

1. Application calls `writer.add_document()`
2. Schema validates and analyzes fields
3. Middleware `before_index` hooks run
4. Document is written to segment
5. Middleware `after_index` hooks run
6. `DocumentIndexed` event is published
7. `commit()` merges segments and writes TOC

### Search Flow

1. Application calls `searcher.search(query)`
2. Query is parsed into query tree
3. Middleware `before_search` hooks run
4. Searcher executes query against segments
5. Results are scored and sorted
6. Middleware `after_search` hooks run
7. `SearchExecuted` event is published
8. Results are returned to application

## Design Principles

1. **Composability**: Components combine via `|` and `+` operators
2. **Zero-cost abstractions**: No middleware = no overhead
3. **Sync-first**: Core is synchronous; async is opt-in
4. **Plugin isolation**: Plugins cannot break the core
5. **Type safety**: Comprehensive type hints throughout


## DOCUMENT: Dates

# Dates and Numeric Ranges

This guide covers working with `DATETIME` and `NUMERIC` fields, including
range queries, range faceting, and date math.

## DATETIME Fields

`DATETIME` fields store Python `datetime` objects and can be queried with
range queries.

```python
from datetime import datetime
from whoosh import fields, index

schema = fields.Schema(
    title=fields.TEXT(stored=True),
    published_date=fields.DATETIME(stored=True, sortable=True),
)
```

### Indexing Dates

```python
ix = index.create_in("indexdir", schema)
with ix.writer() as w:
    w.add_document(
        title="Article 1",
        published_date=datetime(2024, 6, 15, 14, 30),
    )
```

### Date Range Queries

Use `Range` or `QueryParser` syntax:

```python
from whoosh.qparser import QueryParser
from whoosh.query import Range, Every

# Using QueryParser syntax
qp = QueryParser("published_date", schema=ix.schema)
q = qp.parse("[2024-01-01 TO 2024-12-31]")

# Using Range query directly
from datetime import datetime
q = Range(
    "published_date",
    datetime(2024, 1, 1),
    datetime(2024, 12, 31),
)

with ix.searcher() as searcher:
    results = searcher.search(q)
```

### Sorting by Date

```python
from whoosh.sorting import FieldFacet

# Sort by date, most recent first
results = searcher.search(
    query,
    sortedby=FieldFacet("published_date", reverse=True),
)
```

## NUMERIC Fields

`NUMERIC` fields store integers and floating-point numbers.

```python
schema = fields.Schema(
    title=fields.TEXT(stored=True),
    price=fields.NUMERIC(int, stored=True, sortable=True),
    rating=fields.NUMERIC(float, stored=True),
)
```

### Numeric Range Queries

```python
from whoosh.query import NumericRange

q = NumericRange("price", 100, 500)

# Or with QueryParser
qp = QueryParser("price", schema=ix.schema)
q = qp.parse("[100 TO 500]")
```

### Numeric Faceting

Group results into numeric ranges using `RangeFacet`:

```python
from whoosh.sorting import RangeFacet

price_ranges = RangeFacet("price", 0, 1000, 100)
results = searcher.search(query, groupedby=price_ranges)

for groupname, docnums in results.groups("price").items():
    print(f"Price ${groupname}: {len(docnums)} results")
```

## Date Faceting

Group results by date intervals using `DateRangeFacet`:

```python
from datetime import datetime
from whoosh.sorting import DateRangeFacet

start = datetime(2020, 1, 1)
end = datetime(2026, 1, 1)
date_facet = DateRangeFacet(
    "published_date",
    start,
    end,
    relativedelta(years=1),  # Requires: from dateutil.relativedelta import relativedelta
)
results = searcher.search(query, groupedby=date_facet)

for year_range, docnums in results.groups("published_date").items():
    print(f"Year {year_range}: {len(docnums)} results")
```

## Sorting and Filtering by Numbers

### Sorting

```python
from whoosh.sorting import FieldFacet

# Sort by price ascending
results = searcher.search(query, sortedby=FieldFacet("price"))
```

### Filtering

```python
from whoosh.query import NumericRange

# Only results with price >= 50 and price < 200
filter_q = NumericRange("price", 50, 200)
results = searcher.search(query, filter=filter_q)
```

## Making Date/Numeric Fields Sortable

When defining a schema, set `sortable=True` on `NUMERIC` or `DATETIME` fields
to enable sorting by that field:

```python
schema = fields.Schema(
    title=fields.TEXT(stored=True),
    price=fields.NUMERIC(int, sortable=True),
    date=fields.DATETIME(sortable=True),
)
```

If you forgot to set `sortable=True`, you can add it after indexing:

```python
from whoosh import index, sorting

ix = index.open_dir("indexdir")
with ix.writer() as w:
    sorting.add_sortable(w, "price", sorting.FieldFacet("price"))
```


## DOCUMENT: Fieldcaches

# Field caches

The default (`filedb`) backend uses *field caches* in certain circumstances.
The field cache basically pre-computes the order of documents in the index to
speed up sorting and faceting.

Generating field caches can take time the first time you sort/facet on a large
index. The field cache is kept in memory (and by default written to disk when
it is generated) so subsequent sorted/faceted searches should be faster.

The default caching policy never expires field caches, so reused searchers
and/or sorting a lot of different fields could use up quite a bit of memory
with large indexes.

## Customizing cache behaviour

(By default, Whoosh saves field caches to disk. To prevent a reader or
searcher from writing out field caches, do this before you start using it:)

```python
searcher.set_caching_policy(save=False)
```

By default, if caches are written to disk they are saved in the index
directory. To tell a reader or searcher to save cache files to a different
location, create a storage object and pass it to the `storage` keyword
argument:

```python
from whoosh.filedb.filestore import FileStorage

mystorage = FileStorage("path/to/cachedir")
reader.set_caching_policy(storage=mystorage)
```

## Creating a custom caching policy

Expert users who want to implement a custom caching policy (for example, to add
cache expiration) should subclass `whoosh.filedb.fieldcache.FieldCachingPolicy`.
Then you can pass an instance of your policy object to the `set_caching_policy`
method:

```python
searcher.set_caching_policy(MyPolicy())
```

## See also

- [Sorting](/core/sorting) — Facets and sort keys
- [API: sorting](../api/sorting) — Sorting and faceting reference


## DOCUMENT: Glossary

# Glossary

A glossary of key terms used in Whoosh.

## Analysis

The process of converting text into tokens (individual units like words or
terms) for indexing. Involves tokenization, normalization (lowercasing,
stemming), and filtering (stop word removal, etc.).

## Analyzer

A chain of `Tokenizer` and `Filter` objects that processes text into
tokens. Examples include `RegexTokenizer`, `NgramTokenizer`, `LowercaseFilter`,
`StopFilter`, and `StemmerFilter`.

## Compound File

A file format that combines multiple index segment files into a single
`.seg` file. This can improve performance on some filesystems by reducing
file handle usage. Configured via the codec's `should_assemble` setting.

## Document

A single record in the index, similar to a row in a database. A document
contains fields (analogous to columns).

## Field

A named attribute of a document. Fields have a type (defined by `FieldType`)
that determines how the field's value is indexed and stored.

## Field Type

The class (e.g., `TEXT`, `ID`, `NUMERIC`, `DATETIME`, `BOOLEAN`) that
defines how a field's value is tokenized, stored, indexed, and made
sortable/facetable.

## Filter

An `Analyzer` component that processes, transforms, or filters tokens
after tokenization. Examples: `LowercaseFilter`, `StopFilter`,
`StemmerFilter`.

## Format

A `Format` object controls how posting information (term frequency, positions,
character offsets) is encoded for each field in the inverted index.
Examples: `Existence`, `Frequency`, `Positions`, `Characters`.

## Fragmentation

The process of selecting text spans around matched terms for highlighting.

## Highlighter

The `whoosh.highlight` module, which provides formatters, fragmenters, and
scorers for highlighting search terms in documents.

## Index

The collection of segment files that store the inverted index, document
data, and metadata (the table of contents, or TOC).

## IndexWriter

The `IndexWriter` class is used to create and modify the index. It buffers
document additions and deletions and commits them to disk.

## Inverted Index

The core data structure of a search engine: for each unique term, it stores a
list of documents (and positions) where that term appears.

## Matcher

An object that iterates over matching documents in the postings list for a
query. Matchers can be combined (union, intersection, etc.) for compound
queries.

## Posting

A single entry in the inverted index: a (document ID, term frequency, value)
tuple for a given term.

## Schema

Defines the fields, their types, and indexing options for an index. A schema
is passed to `Storage.create_index()`.

## Scorer

An object that computes a relevance score for a document given a query and
term weights. Different weighting models (BM25, TF-IDF, etc.) use different
scorers.

## Segment

A self-contained portion of the inverted index. An index may consist of
multiple segments. Segments are merged periodically (during optimize or
merge operations) to improve performance.

## Sort Key

A value computed per-document (via a `FacetType` and its `Categorizer`)
used to order results during sorting and faceting.

## Stemming

The process of reducing words to their root form (e.g., "running" → "run",
"cats" → "cat") to improve recall by matching inflected forms.

## Stop Words

High-frequency, low-information words (e.g., "the", "a", "and") that are
typically filtered out during indexing.

## Term

A unique (field name, token text) pair in the inverted index.

## Term Vector

Optional per-document data structure storing the terms (and optionally
positions and character offsets) that appear in a document's field, enabling
features like highlighting and pseudo-relevance feedback.

## Tokenizer

An `Analyzer` component that splits input text into tokens. Examples:
`RegexTokenizer`, `PathTokenizer`, `NgramTokenizer`.

## Whoosh Query

Whoosh's own query syntax, parsed by `QueryParser`. Supports fielded
search, phrase queries, wildcards, ranges, and more.


## DOCUMENT: Highlight

# Highlighting search result excerpts

## Overview

The highlighting system works as a pipeline, with four component types.

- **Fragmenters** chop up the original text into *fragments*, based on the
  locations of matched terms in the text.
- **Scorers** assign a score to each fragment, allowing the system to rank the
  best fragments by whatever criterion.
- **Order functions** control in what order the top-scoring fragments are
  presented to the user. For example, you can show the fragments in the order
  they appear in the document (`FIRST`) or show higher-scoring fragments first
  (`SCORE`).
- **Formatters** turn the fragment objects into human-readable output, such as
  an HTML string.

## Requirements

Highlighting requires that you have the text of the indexed document available.
You can keep the text in a stored field, or if the original text is available in
a file, database column, etc, just reload it on the fly. Note that you might
need to process the text to remove e.g. HTML tags, wiki markup, etc.

## How to

Get search results and use the `highlights()` method on the
`whoosh.searching.Hit` object to get highlighted snippets:

```python
results = mysearcher.search(myquery)
for hit in results:
    print(hit["title"])
    # Assume "content" field is stored
    print(hit.highlights("content"))
```

If the field is not stored, you need to retrieve the text of the field some
other way, then supply it with the `text` argument:

```python
results = mysearcher.search(myquery)
for hit in results:
    print(hit["title"])
    # Assume the "path" stored field contains a path to the original file
    with open(hit["path"]) as fileobj:
        filecontents = fileobj.read()
    print(hit.highlights("content", text=filecontents))
```

## The character limit

By default, Whoosh only pulls fragments from the first 32K characters of the
text. This prevents very long texts from bogging down the highlighting process
too much. You can change the character limit on the results object:

```python
results = mysearcher.search(myquery)
results.fragmenter.charlimit = 100000
```

To turn off the character limit:

```python
results.fragmenter.charlimit = None
```

If you instantiate a custom fragmenter, you can set the character limit directly:

```python
sf = highlight.SentenceFragmenter(charlimit=100000)
results.fragmenter = sf
```

## Customizing the highlights

### Number of fragments

Use the `top` keyword argument to control the number of fragments returned:

```python
# Show a maximum of 5 fragments from the document
print(hit.highlights("content", top=5))
```

### Fragment size

The default fragmenter has a `maxchars` attribute (default 200) controlling the
maximum length of a fragment, and a `surround` attribute (default 20)
controlling the maximum number of characters of context to add at the beginning
and end of a fragment:

```python
# Allow larger fragments
results.fragmenter.maxchars = 300
# Show more context before and after
results.fragmenter.surround = 50
```

### Fragmenter

A fragmenter controls how to extract excerpts from the original text. The
`highlight` module has the following pre-made fragmenters:

- `whoosh.highlight.ContextFragmenter` (the default) — a "smart" fragmenter
  that finds matched terms and pulls in surround text. Only yields fragments
  that contain matched terms.
- `whoosh.highlight.SentenceFragmenter` — tries to break the text into
  fragments based on sentence punctuation.
- `whoosh.highlight.WholeFragmenter` — returns the entire text as one
  "fragment". Useful for short bits of text.

```python
my_cf = highlight.ContextFragmenter(maxchars=100, surround=30)
results.fragmenter = my_cf
```

### Scorer

A scorer is a callable that takes a `whoosh.highlight.Fragment` object and
returns a sortable value (where higher values represent better fragments). The
default scorer adds up the number of matched terms in the fragment, and adds a
"bonus" for the number of *different* matched terms.

```python
def StandardDeviationScorer(fragment):
    """Gives higher scores to fragments where the matched terms are close together."""
    return 0 - stddev([t.pos for t in fragment.matched])

results.scorer = StandardDeviationScorer
```

### Order

The order is a function that takes a fragment and returns a sortable value used
to sort the highest-scoring fragments before presenting them to the user.

- `FIRST` (the default) — show fragments in document order.
- `SCORE` — show highest scoring fragments first.
- `LONGER` / `SHORTER` — longer/shorter fragments first (less generally useful).

```python
results.order = highlight.SCORE
```

### Formatter

A formatter controls how the highest scoring fragments are turned into a
formatted bit of text. The `highlight` module contains:

- `whoosh.highlight.HtmlFormatter` — outputs HTML with a class attribute around
  matched terms.
- `whoosh.highlight.UppercaseFormatter` — converts matched terms to UPPERCASE.

The easiest way to create a custom formatter is to subclass `highlight.Formatter`
and override `format_token`:

```python
class BracketFormatter(highlight.Formatter):
    """Puts square brackets around the matched terms."""

    def format_token(self, text, token, replace=False):
        tokentext = highlight.get_text(text, token, replace)
        return "[%s]" % tokentext

brf = BracketFormatter()
results.formatter = brf
```

## Highlighter object

Rather than setting attributes on the results object, you can create a reusable
`whoosh.highlight.Highlighter` object:

```python
hi = highlight.Highlighter(fragmenter=my_cf, scorer=sds)
for hit in results:
    print(hit["title"])
    print(hi.highlight_hit(hit))
```

## Speeding up highlighting

Recording which terms matched in which documents during the search may make
highlighting faster:

```python
# Record per-document term matches
results = searcher.search(myquery, terms=True)
```

### PinpointFragmenter

Instead of re-tokenizing the document text, Whoosh can look up the character
positions of the matched terms in the index. To use
`whoosh.highlight.PinpointFragmenter` and avoid re-tokenizing:

1. Index the field with character information (requires re-indexing):

   ```python
   schema = fields.Schema(content=fields.TEXT(stored=True, chars=True))
   ```

2. Record per-document term matches:

   ```python
   results = searcher.search(myquery, terms=True)
   ```

3. Set the `PinpointFragmenter` as the fragmenter:

   ```python
   results.fragmenter = highlight.PinpointFragmenter()
   ```

Use the `autotrim` option to strip whitespace before the first space and after
the last space in the fragments:

```python
results.fragmenter = highlight.PinpointFragmenter(autotrim=True)
```

## Using the low-level API

```python
from whoosh.highlight import highlight

excerpts = highlight(
    text, terms, analyzer, fragmenter, formatter, top=3,
    scorer=BasicFragmentScorer, minscore=1, order=FIRST,
)
```

| Argument | Description |
|----------|-------------|
| `text` | The original text of the document. |
| `terms` | A sequence or set containing the query words to match. |
| `analyzer` | The analyzer to use to break the document text into tokens. |
| `fragmenter` | A `Fragmenter` object. |
| `formatter` | A `Formatter` object. |
| `top` | The number of fragments to include in the output. |
| `scorer` | A `FragmentScorer` object. |
| `minscore` | The minimum score a fragment must have to be included. |
| `order` | An ordering function for the "top" fragments. |

## See also

- [Searching](/core/searching) — The `search()` method and `Hit` objects
- [API: highlight](../api/highlight) — Full `whoosh.highlight` reference


## DOCUMENT: Indexing

:::info
Following the rename of `whoosh-reloaded` to `whoosh-ng`, new Whoosh-NG specific modules are typically found under `whoosh_modern`.
Core Whoosh components (like `whoosh.analysis`, `whoosh.index`) remain accessible directly under the `whoosh` namespace for backward compatibility.
:::

# Indexing

This guide covers adding, updating, and deleting documents in your Whoosh-NG index.

## Opening a Writer

```python
from whoosh import index

ix = index.open_dir("indexdir")

# Basic writer
writer = ix.writer()

# Writer with custom options
writer = ix.writer(
    timeout=10.0,      # Lock acquisition timeout (seconds)
    delay=0.1,         # Delay between lock retries (seconds)
    limitmb=128,       # Posting pool run size (MiB)
    compound=True      # Use compound files
)
```

## Adding Documents

```python
with ix.writer() as writer:
    writer.add_document(
        title="First document",
        content="Hello world",
        path="/doc1",
        tags=["python", "search"]
    )
    writer.add_document(
        title="Second document",
        content="Goodbye world",
        path="/doc2",
        tags=["python", "tutorial"]
    )
    # commit() is called automatically on exit
```

### Multi-value Fields

Pass lists to add multiple values for multi-valued fields:

```python
writer.add_document(
    title="Document with multiple tags",
    content="Content here",
    tags=["python", "whoosh", "search", "tutorial"]
)
```

### Stored vs Indexed Values

For fields that are both indexed and stored, you can store a different value:

```python
writer.add_document(
    title="Title to be indexed",
    _stored_title="Display title to show in results"
)
```

> **Note**: The underscore prefix (`_stored_<field>`, `_<field>_boost`) is a
> Whoosh convention for per-document overrides. It lets you store a different
> value for display (`_stored_title`) without changing what is indexed, or
> boost a specific field for a single document (`_title_boost`) without
> affecting the schema-level boost.

### Field Boosts

Boost individual fields at document level:

```python
writer.add_document(
    title="Important title",
    _title_boost=2.0,   # Double weight for title terms
    content="Body content"
)
```

## Updating Documents

Use `update_document` to replace documents with matching unique fields:

```python
schema = Schema(path=ID(unique=True, stored=True), content=TEXT)
ix = index.create_in("indexdir", schema)

with ix.writer() as writer:
    writer.add_document(path="/doc1", content="Original content")
    writer.commit()

with ix.writer() as writer:
    # Replaces any document with path="/doc1"
    writer.update_document(path="/doc1", content="Updated content")
    writer.commit()
```

## Deleting Documents

```python
# Delete by document number
writer.delete_document(docnum=42)

# Delete by term in a field
writer.delete_by_term("path", "/doc1")

# Delete by query
from whoosh.query import Term
q = Term("tags", "deprecated")
writer.delete_by_query(q)

writer.commit()
```

## Commit and Merge Policies

### Basic Commit

```python
writer.commit()
```

### Optimize (Merge All)

```python
writer.commit(optimize=True)
```

### No Merge

```python
writer.commit(merge=False)
```

### Custom Merge Policy

```python
from whoosh.writing import NO_MERGE, MERGE_SMALL, OPTIMIZE

writer.commit(mergetype=NO_MERGE)
writer.commit(mergetype=MERGE_SMALL)
writer.commit(mergetype=OPTIMIZE)

# Custom function
def my_merge(writer, segments):
    # Custom merge logic
    return segments

writer.commit(mergetype=my_merge)
```

## BufferedWriter

For high-throughput scenarios where documents arrive one at a time:

```python
from whoosh.writing import BufferedWriter

# Buffers documents and commits periodically
buffered = BufferedWriter(
    ix,
    period=60,    # Max seconds between commits
    limit=100,    # Max documents per commit
    writerargs={} # Extra args for underlying writer
)

with buffered:
    buffered.add_document(title="Doc 1", content="Content")
    buffered.add_document(title="Doc 2", content="More")
# commit() called automatically on close
```

## AsyncWriter

For web applications where multiple processes may write:

```python
from whoosh.writing import AsyncWriter

# Automatically retries on lock contention
async_writer = AsyncWriter(ix, delay=0.25)

async_writer.add_document(title="Async doc", content="Content")
async_writer.commit()
```

## Middleware Integration

```python
from whoosh.middleware import MiddlewareChain, MetricsMiddleware, CacheMiddleware
from whoosh.middleware.integration import apply_middleware_to_writer

chain = MiddlewareChain([
    MetricsMiddleware(),
    CacheMiddleware()
])

with apply_middleware_to_writer(ix.writer(), chain.middlewares) as writer:
    writer.add_document(title="Tracked", content="Content")
```

## Best Practices

1. **Use context managers**: `with ix.writer() as w:` ensures proper cleanup
2. **Batch commits**: Group many documents per commit for better performance
3. **Choose merge policy wisely**: `MERGE_SMALL` is usually fine; use `NO_MERGE` for bulk loads followed by `OPTIMIZE`
4. **Handle locks**: Use `BufferedWriter` or `AsyncWriter` in multi-process environments
5. **Don't forget to close**: Always call `commit()` or `cancel()` to release the write lock


## DOCUMENT: Installation

# Installation

## Requirements

- Python 3.10+
- No mandatory dependencies (pure Python)
- Optional extras for advanced features

## pip install

```bash
pip install whoosh-ng
```

## Extras

| Extra | Description |
|-------|-------------|
| `vector` | NumPy-based vector providers |
| `autocomplete` | Autocomplete plugin |
| `api` | FastAPI plugin |
| `metrics` | Prometheus metrics integration |
| `all` | Install everything |

```bash
pip install whoosh-ng[all]
```

## Development install

```bash
git clone https://github.com/your-org/whoosh-NG.git
cd whoosh-NG
uv sync --extra dev
```

## Verification

```bash
uv run pytest tests/ -q
uv run ruff check src/ tests/
uv run ruff format --check .
uv run mypy src/whoosh
```

## Next Steps

- [Quick Start](/core/quickstart)
- [Core Concepts](/core/core-concepts)


## DOCUMENT: Intro

# Introduction to Whoosh

## About Whoosh

Whoosh was created by Matt Chaput. It started as a quick and dirty search
server for the online documentation of the Houdini 3D animation software
package. Side Effects Software generously allowed Matt to open source the code
in case it might be useful to anyone else who needs a very flexible or
pure-Python search engine (or both!).

- Whoosh is fast, but uses only pure Python, so it will run anywhere Python
  runs, without requiring a compiler.
- By default, Whoosh uses the [Okapi BM25F](https://en.wikipedia.org/wiki/Okapi_BM25)
  ranking function, but like most things the ranking function can be easily
  customized.
- Whoosh creates fairly small indexes compared to many other search libraries.
- All indexed text in Whoosh must be **unicode**.
- Whoosh lets you store arbitrary Python objects with indexed documents.

## What is Whoosh?

Whoosh is a fast, pure Python search engine library.

The primary design impetus of Whoosh is that it is pure Python. You should be
able to use Whoosh anywhere you can use Python, no compiler or Java required.

Like one of its ancestors, Lucene, Whoosh is not really a search engine, it's a
programmer library for creating a search engine.

Practically no important behavior of Whoosh is hard-coded. Indexing of text, the
level of information stored for each term in each field, parsing of search
queries, the types of queries allowed, scoring algorithms, etc. are all
customizable, replaceable, and extensible.

## What can Whoosh do for you?

Whoosh lets you index free-form or structured text and then quickly find
matching documents based on simple or complex search criteria.

## Whoosh-NG

Whoosh-NG is the maintained evolution of Whoosh. It preserves the pure-Python
core described above while adding optional, opt-in extensions (vector search,
a plugin system, a middleware pipeline, linguistics, and pluggable storage).
Classic features documented in this section remain backwards-compatible with
Whoosh 1.x/2.x.

## Getting help with Whoosh

You can view outstanding issues on the
[Whoosh-NG GitHub page](https://github.com/dorel14/whoosh-ng) and get help by
opening an issue or discussion there.


## DOCUMENT: Keywords

# Query expansion and keyword extraction

## Overview

Whoosh provides methods for computing the "key terms" of a set of documents.
For these methods, "key terms" basically means terms that are frequent in the
given documents, but relatively infrequent in the indexed collection as a whole.

Because this is a purely statistical operation, not a natural language
processing or AI function, the quality of the results will vary based on the
content, the size of the document collection, and the number of documents for
which you extract keywords.

These methods can be useful for providing the following features to users:

- **Search term expansion.** Extract key terms for the top N results from a
  query and suggest them to the user as additional/alternate query terms.
- **Tag suggestion.** Extracting the key terms for a single document may yield
  useful suggestions for tagging the document.
- **"More like this".** Extract key terms for the top ten or so results from a
  query (and removing the original query terms), and use those key words as the
  basis for another query that may find more documents using terms the user
  didn't think of.

## Usage

### More like this

Get more documents like a certain search hit. *This requires that the field you
want to match on is vectored or stored, or that you have access to the original
text.*

```python
results = mysearcher.search(myquery)
first_hit = results[0]
more_results = first_hit.more_like_this("content")
```

### Key terms from top N results

*This requires that the field is either vectored or stored.*

```python
# Extract five key terms from the "content" field of the top ten documents
keywords = [keyword for keyword, score
            in results.key_terms("content", docs=10, numterms=5)]
```

### Key terms from an arbitrary set of documents

*This requires that the field is either vectored or stored.*

```python
with email_index.searcher() as s:
    docnums = s.document_numbers(emailto="matt@whoosh.ca")
    keywords = [keyword for keyword, score
                in s.key_terms(docnums, "body")]
```

### Key terms from arbitrary text not in the index

```python
with email_index.searcher() as s:
    keywords = [keyword for keyword, score
                in s.key_terms_from_text("body", mytext)]
```

## Expansion models

The `ExpansionModel` subclasses in the `whoosh.classify` module implement
different weighting functions for key words. These models are translated into
Python from original Java implementations in Terrier.

```python
from whoosh.classify import Bo1Model

results = mysearcher.search(myquery)
keywords = results.key_terms("content", docs=10, numterms=5, model=Bo1Model)
```

Available models include `Bo1Model`, `Bo2Model`, and `KLModel`.

## See also

- [Searching](/core/searching) — `Results`, `Hit`, and the `search()` method
- [API: searching](../api/searching) — `key_terms`, `more_like_this` reference


## DOCUMENT: Legacy Cleanup

# Legacy Code Cleanup Strategy

This guide explains how Whoosh-NG separates modern typed code from legacy code,
and how the legacy cleanup is progressing.

## Why a legacy boundary?

`whoosh-modern` is the new, fully typed surface of Whoosh-NG.
The original `whoosh` package still works at runtime, but it carries decades of
Python 2/3 compatibility patterns, dynamic metaprogramming, and untyped internals.
Trying to force strict types on all of it at once would block development.

The cleanup strategy is **incremental and opt-in**:

1. `src/whoosh_modern/` is typed and linted with `pyright` and `mypy` strict.
2. `src/whoosh/` is the legacy surface. It is split into:
   - **excluded modules** (documented in `pyrightconfig.json`) — code that is too
     dynamic or vendored for an economical type pass right now;
   - **cleanup candidates** — small, isolated files that are straightforward to
     annotate and verify.
3. Each sprint, a wave of candidates is typed, tested, and promoted out of the
   high-tolerance zone.

## Current pyright/mypy thresholds (Sprint 2)

| Checker | Scope | Threshold |
|---------|-------|-----------|
| `pyright` | `src/whoosh_modern/` | **0 errors** (strict) |
| `pyright` | legacy | **≤ 500 errors** (tolerant) |
| `mypy` | `src/` | **0 errors** (via overrides + `ignore_errors`) |

## Exclusion rationale (pyrightconfig.json)

The `exclude` list in `pyrightconfig.json` groups excluded files by theme:

- **Vendored / no stubs**: `pyparsing.py`, `relativedelta.py`
- **Migration shims**: `codec/whoosh2.py`, `codec/whoosh3.py`
- **Dynamic parsing**: `qparser/`, `query/`, `analysis/`, `automata/`
- **Large datastores**: `filedb/`, `reading/`, `writing/`
- **Heuristic / data-driven**: `lang/dmetaphone.py`, `lang/lovins.py`,
  `lang/phonetic.py`, `lang/wordnet.py`
- **Core dynamic objects**: `classify.py`, `index.py`, `locking.py`,
  `formats.py`, `middleware/`
- **Vendored low-level**: `support/bench.py`, `support/base85.py`,
  `support/bitstream.py`, `support/bitvector.py`, `support/charset.py`,
  `support/levenshtein.py`

## Sprint 2 cleanup plan

For Sprint 2, the focus is on small utility and support modules that have few
external dependencies and no heavy metaprogramming.

Candidate wave:

- `src/whoosh/util/varints.py`
- `src/whoosh/util/text.py`
- `src/whoosh/util/loading.py`
- `src/whoosh/support/bitstream.py`
- `src/whoosh/support/levenshtein.py`

For each file:

1. Remove the blanket `# type: ignore` (if present).
2. Add precise function signatures.
3. Run `pyright` and `mypy` to confirm **0 new errors**.
4. Move the file out of `pyrightconfig.json` excludes.
5. Add a regression test in `tests/test_legacy_cleanup.py`.

## Long-term goal

Eventually every file in `src/whoosh/` should be checkable by `mypy` and
`pyright` without blanket excludes. Until then, the exclude list is the
explicit ledger of debt, and each sprint chips away at it.


## DOCUMENT: Migration

# Migration Guide

This guide helps you migrate from Whoosh legacy or Whoosh-Reloaded 3.x to Whoosh-NG v3.0.0.
> **Next release**: v4.0.0.dev0 (in development) will add SchemaBuilder, enhanced middleware exception hierarchy, and more — see the [CHANGELOG](https://github.com/dorel14/whoosh-ng/blob/master/CHANGELOG.md) for details.

## From Whoosh 1.x/2.x (Legacy)

### Import Paths

| Legacy | Whoosh-NG |
|--------|-----------|
| `import whoosh` | `import whoosh` |
| `from whoosh.index import create_in` | `from whoosh.index import create_in` |
| `from whoosh.fields import Schema, TEXT` | `from whoosh.fields import Schema, TEXT` |
| `from whoosh.qparser import QueryParser` | `from whoosh.qparser import QueryParser` |

The core API is intentionally stable. Most existing code works unchanged.

### Spelling API

```python
# Legacy
from whoosh.spelling import SpellChecker
corrector = SpellChecker(ix.reader(), "content")

# Whoosh-NG
from whoosh.spelling import ReaderCorrector
corrector = ReaderCorrector(ix.searcher().reader(), "content", ix.schema["content"])
suggestions = corrector.suggest("helo", limit=5)
```

### Highlighting

```python
# Legacy API unchanged
results[0].highlights("content")
```

## From Whoosh-Reloaded 3.x

### No Breaking Changes

Whoosh-NG is a continuation of Whoosh-Reloaded. All existing code works as-is.

### Optional: Plugin Migration

If you used `whoosh_modern` directly:

```python
# Old
from whoosh_modern.vector.numpy_provider import NumpyProvider

# New (via registry)
from whoosh.vector import NumpyProvider
from whoosh.registry import VectorRegistry

VectorRegistry.register("numpy", NumpyProvider(), "my_app")
```

### Middleware (New in v4.0.0.dev0)

```python
# Optional migration: add middleware to existing code

from whoosh.middleware import Middleware, MiddlewareContext

class LoggingMiddleware(Middleware):
    def before_search(self, context):
        print(f"Query: {context.query}")
        return context

# Wrap existing writer/searcher
writer = apply_middleware_to_writer(ix.writer(), [LoggingMiddleware()])
```

### SchemaBuilder (New in v4.0.0.dev0)

```python
# Old
schema = Schema(title=TEXT(stored=True), content=TEXT)

# New (fluent API)
from whoosh.fields import SchemaBuilder

schema = (
    SchemaBuilder()
    .field("title", TEXT(stored=True))
    .field("content", TEXT)
    .build()
)
```

## Upgrade Checklist

1. **Update dependencies**:
   ```bash
   pip install --upgrade whoosh-ng
   ```

2. **Run tests**:
   ```bash
   uv run pytest tests/ -q
   ```

3. **Update optional deps** (if using plugins):
   ```bash
   pip install whoosh-ng[all]
   ```

4. **Review middleware**: Consider adding middleware for cross-cutting concerns

5. **Update config**: If using `whoosh.config`, review new options

## Deprecations

| Feature | Status | Replacement |
|---------|--------|-------------|
| `whoosh_modern.vector` | Deprecated | `whoosh.vector` |
| Raw `whoosh.store` | Deprecated | `whoosh.backends` |
| Direct `SegmentWriter` usage | Discouraged | Use `IndexWriter` |

## Breaking Changes

Whoosh-NG maintains backward compatibility. If you find a breaking change, please report it as an issue.

### Exception Hierarchy

New in v4.0.0.dev0: `MiddlewareError` and `StopOperation` in middleware:

```python
from whoosh.middleware.exceptions import MiddlewareError, StopOperation
```

## Getting Help

- [GitHub Issues](https://github.com/your-org/whoosh-NG/issues)
- [Documentation](/)
- [Migration Examples](https://github.com/dorel14/whoosh-ng/tree/master/docs/archive_jekyll/_en/examples)


## DOCUMENT: Nested

# Nested Documents

This guide covers indexing and searching hierarchical/nested document
structures (e.g., a parent document with multiple child documents) using
Whoosh's parent-child relationship features.

## Defining Nested Documents

You can index parent documents that contain child documents by using a
parent field and child fields:

```python
from whoosh import fields, index

schema = fields.Schema(
    type=fields.ID(sortable=True),
    title=fields.TEXT(stored=True),
    content=fields.TEXT,
    section_name=fields.ID,
    section_content=fields.TEXT,
)
```

The `type` field distinguishes parent documents from child documents.

## Indexing Nested Documents

Use `IndexWriter.add_all()` with a generator that yields parent and child
documents grouped together:

```python
writer = ix.writer()
writer.add_all([
    parent_doc,
    child_doc_1,
    child_doc_2,
    parent_doc_2,
    child_doc_3,
])
```

Parent documents have `type="parent"` and child documents have
`type="child"`.

## Searching Nested Documents

### Parent-Query Child-Search

Search within child documents and match their parents:

```python
from whoosh.query import Every, Term
from whoosh.sorting import NestedParent

# Match all parent documents
parents = NestedParent(Term("type", "parent"))
q = Every("section_content", "hello")
results = searcher.search(q, sortedby=parents)
```

### Child-Query Parent-Search

Search for parent documents whose children match:

```python
from whoosh.sorting import NestedChildren

# Match parent documents that have children matching the query
parent_results = searcher.search(child_query, groupedby=NestedChildren(parent_matcher, child_matcher))
```

## Parent-Child Relationships at Index Time

When writing documents, use the `parent` parameter to link children to
parents:

```python
writer.add_document(type="parent", title="Chapter 1", _key="chapter1")
writer.add_document(type="child", section_name="Section 1.1",
                    section_content="...", parent="chapter1")
writer.add_document(type="child", section_name="Section 1.2",
                    section_content="...", parent="chapter1")
```

## Accessing Nested Results

To retrieve child matches alongside parent results, use the `expand` method
on the results:

```python
results = searcher.search(parent_query)
expanded = results.expand_child("section")
```

## Nested Faceting

Combine parent-child relationships with faceting using `NestedParent` and
`NestedChildren` as facets:

```python
parent_facet = NestedParent(FieldFacet("type"))
results = searcher.search(query, groupedby=parent_facet)
```

## Performance Considerations

- Parent-child joins are more expensive than flat document searches
- Use `childperm` searcher option to limit the number of permutations
  examined
- Consider whether hierarchical structure is needed at query time, or
  whether documents can be flattened during indexing


## DOCUMENT: Ngrams

# N-grams

This guide covers N-gram tokenization and analysis for substring matching,
prefix queries, and autocomplete functionality.

## What Are N-grams?

An N-gram is a contiguous sequence of N characters (or tokens) from a string.
For example, the 2-grams of "hello" are: "he", "el", "ll", "lo".

N-gram analysis is useful for:
- Substring search (finding "ell" within "hello")
- Autocomplete / typeahead suggestions
- Fuzzy matching without edit distance computation

## NgramTokenizer

The `NgramTokenizer` splits text into character-level N-grams:

```python
from whoosh.analysis import NgramTokenizer
from whoosh import fields

tokenizer = NgramTokenizer(minsize=2, maxsize=4)

schema = fields.Schema(
    content=fields.TEXT(analyzer=tokenizer),
)
```

### NgramTokenizer Parameters

- `minsize`: Minimum N-gram length (default `2`)
- `maxsize`: Maximum N-gram length (default `4`)

With the example above, the text "hello" produces these 2-4-grams:
`he, hel, hell, el, ell, ello, l, ll, llo, l, lo, o`

## NgramFilter

The `NgramFilter` creates word-level N-grams from tokenized text:

```python
from whoosh.analysis import RegexTokenizer, NgramFilter

analyzer = RegexTokenizer() | NgramFilter(maxsize=2)
```

This produces word-level grams: for "hello world", it produces ("hello",)
and ("hello", "world").

## NgramWordAnalyzer

A convenience analyzer that combines `NgramTokenizer` with `LowercaseFilter`:

```python
from whoosh.analysis import NgramWordAnalyzer

analyzer = NgramWordAnalyzer(minsize=2, maxsize=4)

schema = fields.Schema(
    content=fields.TEXT(analyzer=analyzer),
)
```

## Use Cases

### Substring Search

With N-gram analysis, you can match substrings:

```python
from whoosh.qparser import QueryParser

# Index text with N-grams
# Searching for "ell" matches "hello" because "ell" is a substring
qp = QueryParser("content", schema=ix.schema)
q = qp.parse("ell")
results = searcher.search(q)
```

### Prefix Matching

Set `maxsize` equal to a large value to effectively create prefix N-grams:

```python
from whoosh.analysis import NgramWordAnalyzer

# Create N-grams where each word's prefixes become searchable tokens
# e.g., "hello" -> "h", "he", "hel", "hell", "hello"
analyzer = NgramWordAnalyzer(minsize=1, maxsize=10)
```

### Autocomplete

N-gram indexes are commonly used for autocomplete/typeahead. For more
advanced autocomplete with edge n-grams, consider:

```python
from whoosh.analysis import RegexTokenizer, NgramFilter
from whoosh.query import Prefix

# Index with standard tokenization, then use Prefix queries for autocomplete
analyzer = RegexTokenizer()
schema = fields.Schema(
    title=fields.TEXT(stored=True, analyzer=analyzer),
    content=fields.TEXT(analyzer=analyzer),
)

# For autocomplete, query with Prefix
from whoosh.qparser import QueryParser
qp = QueryParser("title", schema=ix.schema)
q = Prefix("title", "hel")  # Find documents where title starts with "hel"
```

## Comparison with Edge N-grams

Some search engines support "edge n-grams" (only generating N-grams from the
beginning of words). This is more space-efficient for autocomplete:

- Full N-grams: "hello" → "he", "el", "ll", "lo", "hel", "ell", ...
- Edge N-grams: "hello" → "h", "he", "hel", "hell", "hello"

Whoosh's `NgramTokenizer` generates full (bidirectional) N-grams. For
edge-ngram-like behavior, use the `minsize` and `maxsize` parameters
strategically, or use `Prefix` queries against a standard tokenized field.

## Performance Considerations

- N-gram indexes are typically much larger than standard indexes
- Each original token produces multiple N-gram tokens, increasing index size
- Choose `minsize` and `maxsize` carefully to balance search quality against
  index size
- For autocomplete, consider using `Prefix` queries with a
  non-N-gram field for better performance


## DOCUMENT: Query

# Query Language

Whoosh-NG provides a powerful query language similar to Lucene's, as well as a programmatic query API.

## QueryParser

The `QueryParser` converts a query string into a query tree:

```python
from whoosh.qparser import QueryParser

# Parse a query for a specific field
qp = QueryParser("content", schema)
query = qp.parse("hello world")
```

## Query Syntax

### Basic Terms

```
hello                    # Single term
hello world              # Multiple terms (default AND)
hello OR world           # Explicit OR
"hello world"            # Phrase
```

### Field-Specific

```
title:python             # Search only in title field
title:"Python Tutorial"  # Phrase in specific field
```

### Boolean Operators

```
python AND whoosh
python OR whoosh
python AND NOT java
python AND (whoosh OR lucene)
```

### Prefix and Wildcard

```
pyth*                    # Prefix query
pyth?n                   # Single character wildcard
```

### Range Queries

```
date:[2020 TO 2025]
price:[10 TO 50]
rating:[4.0 TO *]        # Open-ended range
```

### Fuzzy Search

```
python~2                 # Edit distance <= 2
lucene~1                 # Approximate match
```

### Proximity Search

```
"hello world"~5          # Within 5 terms
```

### Boosting

```
python^2.0 whoosh        # Boost python by 2x
(title:python)^3 content:python  # Boost title matches
```

## Query Classes

You can build queries programmatically:

```python
from whoosh.query import *

# Simple term
q = Term("content", "python")

# Multiple terms (AND)
q = And([Term("content", "python"), Term("content", "whoosh")])

# Multiple terms (OR)
q = Or([Term("content", "python"), Term("content", "lucene")])

# Phrase
q = Phrase("content", ["hello", "world"])

# Range
q = NumericRange("price", 10, 50)
q = DateRange("date", datetime(2020,1,1), datetime(2025,1,1))

# Prefix
q = Prefix("content", "pyth")

# Wildcard
q = Wildcard("content", "pyth?n")

# Fuzzy
q = FuzzyTerm("content", "python", maxdist=2)

# Boost
q = Boost(Term("title", "python"), 2.0) & Term("content", "python")
```

## QueryParser Plugins

Extend query parsing with plugins:

```python
from whoosh.qparser import QueryParserPlugin

class MyPlugin(QueryParserPlugin):
    def __init__(self):
        pass

    def evaluate(self, env, signode):
        # Custom evaluation logic
        return Term("custom_field", signode.content)
```

## Multifield Search

Search multiple fields with different boosts:

```python
from whoosh.qparser import MultifieldParser

qp = MultifieldParser(
    ["title", "content", "tags"],
    schema,
    fieldboosts={"title": 2.0, "tags": 1.5}
)
q = qp.parse("python search")
```

## Default Operator

```python
from whoosh.qparser import QueryParser, OrGroup

# Default AND
qp = QueryParser("content", schema)

# Default OR
qp = QueryParser("content", schema, group=OrGroup)
```

## Escaping Special Characters

```
title\:python              # Literal colon
path\:\/\/example          # Escape special chars
```

## Regex Queries

```
content:/p[ya]thon/        # Regex match
```

## Advanced: Custom Queries

```python
from whoosh.query import Query

class CustomQuery(Query):
    def __init__(self, fieldname, text):
        self.fieldname = fieldname
        self.text = text

    def __repr__(self):
        return f"CustomQuery({self.fieldname!r}, {self.text!r})"

    def __hash__(self):
        return hash((self.fieldname, self.text))

    def __eq__(self, other):
        return (
            isinstance(other, CustomQuery)
            and self.fieldname == other.fieldname
            and self.text == other.text
        )

    def __ne__(self, other):
        return not self.__eq__(other)

    def matcher(self, searcher, context=None):
        # Return a custom matcher
        return CustomMatcher(searcher, self)
```


## DOCUMENT: Quickstart

# Quick Start

## Installation

```bash
pip install whoosh-ng
uv pip install whoosh-ng
```

## Basic Example

```python
from whoosh import index
from whoosh.fields import Schema, TEXT, ID

schema = Schema(id=ID(stored=True), content=TEXT())
ix = index.create_in("indexdir", schema)

with ix.writer() as w:
    w.add_document(id="1", content="hello world")
    w.add_document(id="2", content="goodbye world")

with ix.searcher() as s:
    results = s.search("world")
    for hit in results:
        print(hit["id"], hit.score)
```

## With Plugins

```bash
pip install whoosh-ng[vector,autocomplete,api]
```

```python
from whoosh.plugins.manager import PluginManager
from whoosh_modern.vector.plugin import VectorPlugin

PluginManager.load_plugins()
```

## Data Sources

```bash
pip install whoosh-ng
```

```python
import sqlite3
from whoosh_modern.data_sources import SQLSource
from whoosh_modern.views import SearchView

# Use existing benchmark data
conn = sqlite3.connect("benchmark/benchmark_data.db")
source = SQLSource(
    connection=conn,
    query="SELECT * FROM reuters_articles",
)

view = SearchView(name="reuters", source=source)
ix = view.build("indexdir")
```


## DOCUMENT: Recipes

# Whoosh recipes

A collection of small, practical code snippets for common tasks.

## General

### Get the stored fields for a document from the document number

```python
stored_fields = searcher.stored_fields(docnum)
```

## Analysis

### Eliminate words shorter/longer than N

Use a `StopFilter` and the `minsize` and `maxsize` keyword arguments. If you
just want to filter based on size and not common words, set the `stoplist` to
`None`:

```python
sf = analysis.StopFilter(stoplist=None, minsize=2, maxsize=40)
```

### Allow optional case-sensitive searches

Index both the original and lowercased versions of each word. If the user
searches for an all-lowercase word, it acts as a case-insensitive search, but
if they search for a word with any uppercase characters, it acts as a
case-sensitive search:

```python
class CaseSensitivizer(analysis.Filter):
    def __call__(self, tokens):
        for t in tokens:
            yield t
            if t.mode == "index":
                low = t.text.lower()
                if low != t.text:
                    t.text = low
                    yield t

ana = analysis.RegexTokenizer() | CaseSensitivizer()
print([t.text for t in ana("The new SuperTurbo 5000", mode="index")])
# ["The", "the", "new", "SuperTurbo", "superturbo", "5000"]
```

## Searching

### Find every document

```python
myquery = query.Every()
```

### iTunes-style search-as-you-type

Use `whoosh.analysis.NgramWordAnalyzer` as the analyzer for the field you want
to search as the user types. You can save space in the index by turning off
positions in the field using `phrase=False`:

```python
# For example, to search the "title" field as the user types
analyzer = analysis.NgramWordAnalyzer()
title_field = fields.TEXT(analyzer=analyzer, phrase=False)
schema = fields.Schema(title=title_field)
```

See the documentation for the `NgramWordAnalyzer` class for information on the
available options. Also see [N-grams](/core/ngrams).

## Shortcuts

### Look up documents by a field value

```python
# Single document (unique field value)
stored_fields = searcher.document(id="bacon")

# Multiple documents
for stored_fields in searcher.documents(tag="cake"):
    ...
```

## Sorting and scoring

See [Sorting](/core/sorting).

### Score results based on the position of the matched term

The following scoring function uses the position of the first occurrence of a
term in each document to calculate the score, so documents with the given term
earlier in the document will score higher:

```python
from whoosh import scoring

def pos_score_fn(searcher, fieldname, text, matcher):
    poses = matcher.value_as("positions")
    return 1.0 / (poses[0] + 1)

pos_weighting = scoring.FunctionWeighting(pos_score_fn)
with myindex.searcher(weighting=pos_weighting) as s:
    ...
```

## Results

### How many hits were there?

```python
# The number of scored hits
found = results.scored_length()

if results.has_exact_length():
    print("Scored", found, "of exactly", len(results), "documents")
else:
    low = results.estimated_min_length()
    high = results.estimated_length()
    print("Scored", found, "of between", low, "and", high, "documents")
```

### Which terms matched in each hit?

```python
# Use terms=True to record term matches for each hit
results = searcher.search(myquery, terms=True)

for hit in results:
    # Which terms matched in this hit?
    print("Matched:", hit.matched_terms())
    # Which terms from the query didn't match in this hit?
    print("Didn't match:", myquery.all_terms() - hit.matched_terms())
```

## Global information

### How many documents are in the index?

```python
# Including documents that are deleted but not yet optimized away
numdocs = searcher.doc_count_all()
# Not including deleted documents
numdocs = searcher.doc_count()
```

### What fields are in the index?

```python
return myindex.schema.names()
```

### Is term X in the index?

```python
return ("content", "wobble") in searcher
```

### How many times does term X occur in the index?

```python
# Number of times content:wobble appears in all documents
freq = searcher.frequency("content", "wobble")
# Number of documents containing content:wobble
docfreq = searcher.doc_frequency("content", "wobble")
```

### Is term X in document Y?

```python
# Without term vectors
postings = searcher.postings("content", "wobble")
postings.skip_to(500)
return postings.id() == 500

# If field has term vectors
vector = searcher.vector(500, "content")
vector.skip_to("wobble")
return vector.id() == "wobble"
```

## See also

- [Analysis](/core/analysis) — Analyzers, tokenizers, and filters
- [N-grams](/core/ngrams) — Search-as-you-type with N-gram analyzers
- [Searching](/core/searching) — The `search()` method and `Hit` objects


## DOCUMENT: Schema

# Schema Design

How to model documents with Whoosh-NG fields.

## Field types

| Type | Searchable | Stored |
|------|------------|--------|
| TEXT | Yes | Optional |
| ID | Yes | Optional |
| KEYWORD | Yes | Optional |
| STORED | No | Yes |
| NUMERIC | Yes | Optional |
| DATETIME | Yes | Optional |
| BOOLEAN | Yes | Optional |
| VectorField | Provider | Optional |

## Building a schema

```python
from whoosh.fields import Schema, TEXT, ID, KEYWORD, STORED, NUMERIC, BOOLEAN, VectorField

schema = Schema(
    title=TEXT(stored=True),
    slug=ID(stored=True, unique=True),
    content=TEXT,
    tags=KEYWORD(lowercase=True, commas=True),
    published=NUMERIC(int, stored=True),
    featured=BOOLEAN(stored=True),
    embedding=VectorField(dimensions=384, metric="cosine")
)
```

## Multi-value fields

Pass lists for multiple values.

```python
writer.add_document(
    title="Multi-tag post",
    tags=["whoosh", "python", "search"],
    content="..."
)
```

## Per-field boost

Boost fields at write time.

```python
writer.add_document(
    title="Breaking News",
    title_boost=3.0,
    content="..."
)
```

## SchemaBuilder

```python
from whoosh.fields import SchemaBuilder, TEXT, ID, NUMERIC

schema = (
    SchemaBuilder()
    .field("title", TEXT(stored=True))
    .field("path", ID(stored=True, unique=True))
    .field("rating", NUMERIC(float, stored=True))
    .build()
)
```

## Modifying fields

```python
writer.add_document(
    title="Multi-tag post",
    tags=["whoosh", "python", "search"],
    content="..."
)
```

## Search Models

Whoosh-NG can auto-map Python models (dataclasses, Pydantic, SQLAlchemy, SQLModel, msgspec) to a Whoosh `Schema` using `ModelIndex`.

### Level 1: Auto-mapping

```python
from dataclasses import dataclass
from whoosh_modern.models import ModelIndex

@dataclass
class Book:
    title: str
    count: int
    tag: str | None = None

idx = ModelIndex(Book)
schema = idx.schema
```

`ModelIndex` inspects type annotations and maps them to Whoosh fields:

| Python type | Whoosh field |
|-------------|--------------|
| `str` | `TEXT` |
| `int` / `float` | `NUMERIC` |
| `bool` | `BOOLEAN` |
| `datetime` / `date` | `DATETIME` |
| `Decimal` | `NUMERIC(int, decimal_places=2)` |
| `Enum` | `KEYWORD` |
| `bytes` | `KEYWORD` (hex-encoded) |
| `list[str]` | `KEYWORD` |
| `Optional[T]` | mapped type or `STORED` |

ID fields are auto-detected: explicit `SearchOptions(id=True)` > field named `id`/`ID`/`_id` > first `str` field.

### Level 2: Explicit options

Use `SearchField` to override defaults:

```python
from whoosh_modern.models import SearchField, SearchOptions

class Book:
    title: str = SearchField(fulltext=True, stored=True, analyzer="Simple")
    count: int = SearchField(sortable=True)
    tags: list[str] = SearchField(multi=True)
```

### Level 3: Annotated types

Use `Annotated` to attach metadata directly to annotations:

```python
from typing import Annotated
from whoosh_modern.models import SearchField

class Book:
    title: Annotated[str, SearchField(fulltext=True, stored=True)]
```

### Integrations

#### Dataclass

```python
from dataclasses import dataclass
from whoosh_modern.models import ModelIndex

@dataclass
class Article:
    title: str
    body: str
    published: datetime.datetime

idx = ModelIndex(Article)
```

#### Pydantic v2

```python
from pydantic import BaseModel
from whoosh_modern.models import register_model

class Article(BaseModel):
    title: str
    body: str
    published: datetime.datetime

    # Per-field search metadata via json_schema_extra
    model_config = {"json_schema_extra": {"search": {"fulltext": True}}}

idx = register_model(Article)
```

#### SQLAlchemy

```python
from sqlalchemy import Column, Integer, String, DateTime
from sqlalchemy.orm import DeclarativeBase
from whoosh_modern.models import register_model

class Base(DeclarativeBase):
    pass

class Article(Base):
    __tablename__ = "articles"
    id = Column(Integer, primary_key=True)
    title = Column(String, info={"search": {"fulltext": True, "stored": True}})
    published = Column(DateTime, info={"search": {"sortable": True}})

idx = register_model(Article)
```

#### SQLModel

```python
from sqlmodel import SQLModel, Field
from whoosh_modern.models import register_model

class Article(SQLModel, table=True):
    id: int = Field(primary_key=True)
    title: str = Field(sa_column_kwargs={"info": {"search": {"fulltext": True}}})
    published: datetime.datetime

idx = register_model(Article)
```

#### msgspec

```python
import msgspec
from whoosh_modern.models import register_model

class Article(msgspec.Struct):
    title: str = msgspec.field(metadata={"search": {"fulltext": True}})
    published: datetime.datetime

idx = register_model(Article)
```

### Converting instances

```python
doc = idx.to_whoosh_document(book_instance)
writer.add_document(**doc)
```

`to_whoosh_document` handles:
- dataclass: `dataclasses.fields()` iteration
- Pydantic/SQLModel: `model_fields` iteration
- SQLAlchemy: `__mapper__.columns` iteration
- Enum values converted to `.value`
- `bytes` converted to hex string

## Best practices

1. **Minimal**: Only index what you search
2. **STORED sparingly**: Increases index size
3. **Unique fields**: Use `unique=True` for identifiers
4. **Field boost**: Boost important fields at schema level
5. **TEXT options**: Disable `phrase` if you don't need phrase search
6. **ID field**: Let `ModelIndex` auto-detect or explicitly mark with `SearchOptions(id=True)`


## DOCUMENT: Searching

# Searching

This guide covers executing searches, working with results, scoring, sorting, and filtering.

## Basic Search

```python
from whoosh.qparser import QueryParser

qp = QueryParser("content", ix.schema)
query = qp.parse("hello world")

with ix.searcher() as searcher:
    results = searcher.search(query)
    for hit in results:
        print(hit["title"], hit.score)
```

## The Searcher

The `Searcher` is the main interface for reading the index. It is lightweight and supports context management:

```python
# Always use context manager when possible
with ix.searcher() as searcher:
    results = searcher.search(query)

# Or manage manually
searcher = ix.searcher()
try:
    results = searcher.search(query)
finally:
    searcher.close()
```

### Searcher Options

```python
searcher = ix.searcher(
    weighting=None,       # Custom weighting model
    childperm=None,       # Permutations for nested documents
    fromindex=None        # Source index for cached readers
)
```

## QueryParser

Convert query strings into query objects:

```python
from whoosh.qparser import QueryParser, OrGroup

# Default: AND between terms
qp = QueryParser("content", schema)
q = qp.parse("hello world")  # Equivalent to: content:hello AND content:world

# Change default operator
qp = QueryParser("content", schema, group=OrGroup)
q = qp.parse("hello world")  # Equivalent to: content:hello OR content:world
```

## Search Methods

### search()

```python
results = searcher.search(
    query,
    limit=10,           # Max results (None for all)
    sortedby=None,      # Sort key(s)
    reverse=False,      # Reverse sort order
    terms=False,        # Collect matched terms
    filter=None,        # Allow only these docnums
    mask=None,          # Exclude these docnums
    collapse=None,      # Collapse facet
    collapse_limit=1    # Max docs per collapse key
)
```

### search_page()

```python
# Get page 1, 10 results per page (default)
results = searcher.search_page(query, 1)

# Get page 3, 20 results per page
results = searcher.search_page(query, 3, pagelen=20)
```

### search_with_collector()

For advanced result collection:

```python
from whoosh.collectors import FacetCollector

collector = FacetCollector(facets=[sorting.FieldFacet("date")])
searcher.search_with_collector(query, collector)
```

## Results Object

`Results` acts like a list of matched documents:

```python
results = searcher.search(query)

# Slice support
first_five = results[0:5]

# Length (may trigger recount)
total = len(results)

# Scored length (usually what was actually returned)
scored = results.scored_length()

# Iteration
for hit in results:
    print(hit["title"], hit.score)
```

### Hit Object

```python
for hit in results:
    # Stored fields
    title = hit["title"]
    path = hit["path"]

    # Score
    print(hit.score)

    # Highlighting
    highlights = hit.highlights("content", top=3)

    # Matched terms (if terms=True was used)
    if results.has_matched_terms():
        print(hit.matched_terms())
```

## Scoring

The default scoring model is BM25F:

```python
from whoosh import scoring

with ix.searcher(weighting=scoring.BM25F()) as s:
    results = s.search(query)
```

### Custom Scoring

```python
class MyScorer(scoring.WeightingModel):
    def scorer(self, searcher, fieldname, text, qf=1):
        return MyCustomScorer(searcher, fieldname, text, qf)

with ix.searcher(weighting=MyScorer()) as s:
    results = s.search(query)
```

## Sorting

Sort by a field or facet:

```python
from whoosh import sorting

# Sort by a single field
results = searcher.search(query, sortedby="date")

# Reverse sort
results = searcher.search(query, sortedby="date", reverse=True)

# Multi-field sort
results = searcher.search(query, sortedby=[
    sorting.FieldFacet("category"),
    sorting.ScoreFacet()
])
```

## Faceting

Used for grouping results:

```python
from whoosh import sorting

facet = sorting.FieldFacet("category")
with searcher.all_features() as features:
    facets = features.facet(facet)
    for cat, count in facets.most_common():
        print(f"{cat}: {count}")
```

## Filtering and Masking

```python
# Only show documents matching a subquery
filter_q = Term("published", True)
results = searcher.search(query, filter=filter_q)

# Exclude documents
mask_q = Term("draft", True)
results = searcher.search(query, mask=mask_q)
```

## Collapsing

Remove duplicates or limit per-group:

```python
from whoosh import sorting

# Collapse by hostname, keep top 3 per host
results = searcher.search(
    query,
    collapse=sorting.FieldFacet("hostname"),
    collapse_limit=3
)

# Collapse ordering (keep highest rated per type)
results = searcher.search(
    query,
    sortedby=sorting.FieldFacet("price", reverse=True),
    collapse=sorting.FieldFacet("type"),
    collapse_order=sorting.FieldFacet("rating", reverse=True)
)
```

## Highlighting

Get highlighted snippets for query terms:

```python
results = searcher.search(query, terms=True)

for hit in results:
    print(hit.highlights("content", top=2))

# Custom fragmenter
from whoosh.highlight import highlight

fragments = hit.highlights(
    "content",
    top=3,
    fragmenter=...,
    formatter=...
)
```

## Time-Limited Searches

```python
from whoosh.collectors import TimeLimitCollector

with ix.searcher() as s:
    c = s.collector(limit=None)
    tlc = TimeLimitCollector(c, timelimit=5.0)
    try:
        s.search_with_collector(query, tlc)
    except TimeLimit:
        print("Search aborted: too slow")
    results = tlc.results()
```

## Combining Results

```python
# Run two queries
best_bet_results = s.search(best_bet_query, limit=5)
main_results = s.search(main_query, limit=10)

# Merge: duplicates go to top, then append rest
best_bet_results.upgrade_and_extend(main_results)
```


## DOCUMENT: Sorting

# Sorting

The `whoosh.sorting` module provides facets and sort-key computation for ordering and grouping search results.

## Quick start

```python
from whoosh import sorting

# Sort by a field
results = searcher.search(query, sortedby="date")

# Sort descending
results = searcher.search(query, sortedby=sorting.FieldFacet("price", reverse=True))
```

For the full API reference, see [Sorting API](/api/sorting).


## DOCUMENT: Spelling

# "Did you mean... ?" Correcting errors in user queries

## Overview

Whoosh can quickly suggest replacements for mis-typed words by returning a list
of words from the index (or a dictionary) that are close to the mis-typed word:

```python
with ix.searcher() as s:
    corrector = s.corrector("text")
    for mistyped_word in mistyped_words:
        print(corrector.suggest(mistyped_word, limit=3))
```

See the `whoosh.spelling.Corrector.suggest()` method documentation for
information on the arguments.

Currently the suggestion engine is more like a "typo corrector" than a real
"spell checker" since it doesn't do the kind of sophisticated phonetic matching
or semantic/contextual analysis a good spell checker might. However, it is
still very useful.

There are two main strategies for correcting words:

- Use the terms from an index field.
- Use words from a word list.

## Pulling suggestions from an indexed field

In Whoosh 2.7 and later, spelling suggestions are available on all fields.
However, if you have an analyzer that modifies the indexed words (such as
stemming), you can add `spelling=True` to a field to have it store separate
unmodified versions of the terms for spelling suggestions:

```python
ana = analysis.StemmingAnalyzer()
schema = fields.Schema(text=TEXT(analyzer=ana, spelling=True))
```

You can then use the `whoosh.searching.Searcher.corrector()` method to get a
corrector for a field:

```python
corrector = searcher.corrector("content")
```

The advantage of using the contents of an index field is that when you are
spell checking queries on that index, the suggestions are tailored to the
contents of the index. The disadvantage is that if the indexed documents
contain spelling errors, then the spelling suggestions will also be erroneous.

## Pulling suggestions from a word list

There are plenty of word lists available on the internet you can use to populate
the spelling dictionary. `word_list` can be a list of unicode strings, or a
file object with one word on each line.

```python
from whoosh.spelling import ListCorrector

# word_list must be a sorted list of unicode strings
corrector = ListCorrector(word_list)
```

## Merging two or more correctors

You can combine suggestions from two sources (for example, the contents of an
index field and a word list) using a `whoosh.spelling.MultiCorrector`:

```python
c1 = searcher.corrector("content")
c2 = spelling.ListCorrector(word_list)
corrector = MultiCorrector([c1, c2])
```

## Correcting user queries

You can spell-check a user query using the
`whoosh.searching.Searcher.correct_query()` method:

```python
from whoosh import qparser

# Parse the user query string
qp = qparser.QueryParser("content", myindex.schema)
q = qp.parse(qstring)

# Try correcting the query
with myindex.searcher() as s:
    corrected = s.correct_query(q, qstring)
    if corrected.query != q:
        print("Did you mean:", corrected.string)
```

The `correct_query` method returns an object with the following attributes:

- `query` — A corrected `whoosh.query.Query` tree. Compare it (`==`) with the
  original parsed query to check if the corrector changed anything.
- `string` — A corrected version of the user's query string.
- `tokens` — A list of corrected token objects representing the corrected terms.

You can use a `whoosh.highlight.Formatter` object to format the corrected query
string, for example the `HtmlFormatter` to format it as HTML:

```python
from whoosh import highlight

hf = highlight.HtmlFormatter()
corrected = s.correct_query(q, qstring, formatter=hf)
```

## See also

- [Highlighting](/core/highlight) — Format corrected query strings with a formatter
- [Query Language](/core/query) — Parsing user queries
- [API: spelling](../api/spelling) — Full `whoosh.spelling` reference


## DOCUMENT: Stemming

# Stemming and Stop Words

This guide covers using stemmers, stop-word filters, and language-specific
text analysis with Whoosh.

## Stemmers

A stemmer reduces words to their root form (e.g., "running" → "run",
"cats" → "cat"), so that different forms of the same word match in
searches.

### Using StemmerFilter

```python
from whoosh.analysis import RegexTokenizer, StemmerFilter
from whoosh.lang.porter import stem
from whoosh import fields

# English Porter stemmer
stem_analyzer = RegexTokenizer() | StemmerFilter(stemfn=stem)

schema = fields.Schema(
    title=fields.TEXT(stored=True),
    content=fields.TEXT(analyzer=stem_analyzer),
)
```

### Snowball Stemmers

Whoosh includes Snowball stemmers for multiple languages:

```python
from whoosh.analysis import StemmerFilter
from whoosh.lang.snowball import EnglishStemmer

stem_analyzer = RegexTokenizer() | StemmerFilter(stemfn=EnglishStemmer().stem)
```

### Language-Aware Stemmer Selection

```python
from whoosh.lang import stemmer_for_language, StemmerFilter
from whoosh.analysis import RegexTokenizer

stem = stemmer_for_language("en")
analyzer = RegexTokenizer() | StemmerFilter(stemfn=stem)

# Or use the analysis StemmingAnalyzer:
from whoosh.analysis import StemmingAnalyzer

analyzer = StemmingAnalyzer("en")
```

### Available Languages

```python
from whoosh.lang import languages, has_stemmer, has_stopwords

print(languages)  # ('ar', 'da', 'nl', 'en', 'fi', 'fr', ...)
print(has_stemmer("en"))  # True
print(has_stopwords("en"))  # True
```

## Stop Words

Stop words are common words (like "the", "a", "and") that are typically
filtered out during indexing since they appear in too many documents to be
useful for ranking.

### Using StopFilter

```python
from whoosh.analysis import RegexTokenizer, StopFilter
from whoosh.lang import stopwords_for_language

# English stop words
stop_words = set(stopwords_for_language("en"))
stop_analyzer = RegexTokenizer() | StopFilter(stoplist=stop_words)

schema = fields.Schema(
    content=fields.TEXT(analyzer=stop_analyzer),
)
```

### Combining Stemming and Stop Words

```python
from whoosh.analysis import StemmingAnalyzer

# StemmingAnalyzer automatically loads stemmer and stopwords for the language
analyzer = StemmingAnalyzer("en")

schema = fields.Schema(
    content=fields.TEXT(analyzer=analyzer),
)
```

### Custom Stop Words

```python
from whoosh.analysis import RegexTokenizer, StopFilter

# Custom stop words list
custom_stops = frozenset(["the", "a", "an", "foo", "bar"])
analyzer = RegexTokenizer() | StopFilter(stoplist=custom_stops)
```

## StemmingAnalyzer (Recommended)

The `StemmingAnalyzer` combines tokenizer, stemming, and stop word filtering:

```python
from whoosh.analysis import StemmingAnalyzer

# Automatically uses the correct stemmer and stop words for the language
analyzer = StemmingAnalyzer("en")

# You can override defaults
analyzer = StemmingAnalyzer("en",
                            use_stopwords=True,
                            use_stems=True)
```

### StemmingAnalyzer Options

- `lang`: Language code (e.g., `"en"`, `"fr"`, `"de"`)
- `use_stopwords`: Whether to load and apply stop words (default `True`)
- `use_stems`: Whether to apply stemming (default `True`)
- `args`: Arguments passed to the tokenizer
- `kwargs`: Keyword arguments for the stemmer or stopwords

## Language-Specific Considerations

### Arabic (ISRI Stemmer)

```python
from whoosh.analysis import StemmerFilter
from whoosh.lang.isri import ISRIStemmer

stem_analyzer = RegexTokenizer() | StemmerFilter(stemfn=ISRIStemmer().stem)
```

### Double Metaphone for Phonetic Matching

```python
from whoosh.analysis import RegexTokenizer, DoubleMetaphoneFilter

analyzer = RegexTokenizer() | DoubleMetaphoneFilter()
```

## Query-Side Stemming

The analyzer is applied at both index time and query time (via the query
parser), so stemming is automatically applied to search terms:

```python
from whoosh.qparser import QueryParser

# If the index uses stemming, queries are stemmed too
qp = QueryParser("content", schema=ix.schema)
q = qp.parse("running cats")  # Will match "run", "cat", etc.
```

## N-gram Analysis

For substring and prefix matching, use N-gram analyzers:

```python
from whoosh.analysis import NgramWordAnalyzer

analyzer = NgramWordAnalyzer(minsize=2, maxsize=4)
schema = fields.Schema(content=fields.TEXT(analyzer=analyzer))
```

See the [N-grams Guide](ngrams.md) for more details.

## Modern Stemmer Providers (Whoosh-NG 2.0)

Whoosh-NG 2.0 introduces a plugin-style stemmer provider system with auto-detection, PyStemmer support, and language-specific analyzers. For full details, see the [Stemmer Providers Guide](stemming-providers.md).


## DOCUMENT: Threads

# Concurrency, locking, and versioning

## Concurrency

The `FileIndex` object is "stateless" and should be share-able between threads.

A `Reader` object (which underlies the `Searcher` object) wraps open files and
often individual methods rely on consistent file cursor positions (e.g. they
do two `file.read()`s in a row, so if another thread moves the cursor between
the two read calls Bad Things would happen). You should use one Reader/Searcher
per thread in your code.

Readers/Searchers tend to cache information (such as field caches for
sorting), so if you can share one across multiple search requests, it's a big
performance win.

> Whoosh-NG also provides `AsyncWriter` and `BufferedWriter` in
> `whoosh.writing` (see [Indexing](/core/indexing)) as convenient wrappers for
> multi-process write scenarios.

## Locking

Only one thread/process can write to an index at a time. When you open a
writer, it locks the index. If you try to open a writer on the same index in
another thread/process, it will raise `whoosh.store.LockError`.

In a multi-threaded or multi-process environment your code needs to be aware
that opening a writer may raise this exception if a writer is already open.
Whoosh includes a couple of example implementations
(`whoosh.writing.AsyncWriter` and `whoosh.writing.BufferedWriter`) of ways to
work around the write lock.

While the writer is open and during the commit, **the index is still available
for reading**. Existing readers are unaffected and new readers can open the
current index normally.

### Lock files

Locking the index is accomplished by acquiring an exclusive file lock on the
`<indexname>_WRITELOCK` file in the index directory. The file is not deleted
after the file lock is released, so the fact that the file exists **does not**
mean the index is locked.

## Versioning

When you open a reader/searcher, the reader represents a view of the **current
version** of the index. If someone writes changes to the index, any readers
that are already open **will not** pick up the changes automatically. A reader
always sees the index as it existed when the reader was opened.

If you are re-using a Searcher across multiple search requests, you can check
whether the Searcher is a view of the latest version of the index using
`whoosh.searching.Searcher.up_to_date()`. If the searcher is not up to date,
you can get an up-to-date copy of the searcher using
`whoosh.searching.Searcher.refresh()`:

```python
# If 'searcher' is not up-to-date, replace it
searcher = searcher.refresh()
```

If the searcher has the latest version of the index, `refresh()` simply returns
it. Calling `Searcher.refresh()` is more efficient than closing the searcher
and opening a new one, since it will re-use any underlying readers and caches
that haven't changed.

## See also

- [Indexing](/core/indexing) — Writers, `AsyncWriter`, `BufferedWriter`
- [API: writing](../api/writing) — Writer concurrency helpers


## DOCUMENT: Translation Status

# Translation Completion Tracking

- [x] EN quickstart
- [x] EN guides
- [x] EN API pages
- [x] EN examples
- [x] FR quickstart
- [x] FR guides
- [x] FR API pages
- [x] FR examples


## DOCUMENT: Autocomplete

# Autocomplete with Whoosh-NG

Whoosh-NG provides autocomplete functionality through the `whoosh_modern.autocomplete` module.

## Install

```bash
pip install "whoosh-ng[autocomplete]"
```

## Schema with Keyword Field for Terms

```python
from whoosh import index
from whoosh.fields import Schema, TEXT, KEYWORD

schema = Schema(
    title=TEXT(stored=True),
    tags=KEYWORD(stored=True, commas=True),
)

ix = index.create_in("autocomplete_index", schema)
```

## Index Documents

```python
with ix.writer() as w:
    w.add_document(title="Python Programming", tags="python,programming,language")
    w.add_document(title="JavaScript Basics", tags="javascript,programming,web")
    w.add_document(title="Machine Learning", tags="ml,ai,data-science")
    w.add_document(title="Deep Learning", tags="ml,ai,neural-networks")
    w.commit()
```

## Basic Autocomplete

```python
from whoosh_modern.autocomplete import create_autocomplete

# Create an autocomplete provider (supports "inverted" provider type)
provider = create_autocomplete("inverted")

# Add phrases to index
provider.add(["python", "programming", "javascript", "machine learning", "deep learning"])

# Search for suggestions
hits = provider.search("py", limit=5)
for hit in hits:
    print(hit.text, hit.score)
# Output: python 1.5, programming 0.2
```

## Real-time Suggestion Endpoint

```python
from fastapi import FastAPI
from whoosh_modern.autocomplete import create_autocomplete

app = FastAPI()
provider = create_autocomplete("inverted")

# Populate provider with terms from your index
# (typically done during indexing)
provider.add(["python", "programming", "javascript", "machine learning"])

@app.get("/suggest")
async def suggest(q: str, limit: int = 5):
    hits = provider.search(q, limit=limit)
    return {"suggestions": [hit.text for hit in hits]}
```

## Key Points

- Install with `pip install whoosh-ng[autocomplete]`.
- Use `KEYWORD` fields to store multi-value tags/terms.
- Use `create_autocomplete("inverted")` to create a provider.
- The `InvertedIndexAutocomplete` provider supports prefix matching with scoring.
- Each result is an `AutocompleteHit` with `text` and `score` attributes.


## DOCUMENT: Basic Indexing

# Basic Indexing

Examples for indexing documents in Whoosh‑NG. Each section is a self-contained, **runnable** script.

> **Real-world scenario**: You're building a blog search engine. You have a CSV file
> of articles (`blog_posts.csv`) with `title`, `url`, `tags`, and `body` columns.

## 1. Define a Production Schema

```python
from whoosh import index
from whoosh.fields import Schema, TEXT, ID, KEYWORD, NUMERIC, DATETIME
from datetime import datetime

# Stored=True keeps the field value in the index so you can retrieve it
# in search results without querying an external DB.
schema = Schema(
    doc_id=ID(stored=True, unique=True),     # primary key
    title=TEXT(stored=True),                  # full-text searchable + retrievable
    url=ID(stored=True),                      # stored only, no full-text analysis
    tags=KEYWORD(stored=True, commas=True),   # multi-value: "python,search,guide"
    body=TEXT(stored=True, phrase=True),      # searchable text with phrase queries
    published_at=DATETIME(stored=True, sortable=True),
    word_count=NUMERIC(int, stored=True),
)
```

## 2. Build an Index from a CSV File

```python
import csv
import shutil
from whoosh import index

# Clean prior index (development only!)
shutil.rmtree("blog_index", ignore_errors=True)
ix = index.create_in("blog_index", schema)

# Simulate a CSV file with blog post data
# blog_posts.csv:
#   doc_id,title,url,tags,published_at,word_count,body
#   1,Building a Search Engine,/posts/1,python,search,2024-01-15,1200,"Learn how to build..."
#   2,Python Tips,/posts/2,python,tips,2024-02-20,800,"Ten tips for Python..."

with open("blog_posts.csv", newline="", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    with ix.writer() as writer:
        for row in reader:
            writer.add_document(
                doc_id=row["doc_id"],
                title=row["title"],
                url=row["url"],
                tags=row["tags"],
                published_at=datetime.fromisoformat(row["published_at"]),
                word_count=int(row["word_count"]),
                body=row["body"],
            )
        writer.commit()
```

## 3. Incremental Update — Re-index Modified Documents

```python
# Suppose your CMS tells you which posts were updated since the last sync
updated_posts = [
    {"doc_id": "1", "title": "Building a Search Engine (Updated)", "body": "Updated content..."},
    {"doc_id": "3", "title": "New Post", "body": "Fresh content..."},
]

with ix.writer() as writer:
    for post in updated_posts:
        writer.update_document(
            doc_id=post["doc_id"],
            title=post["title"],
            url=f"/posts/{post['doc_id']}",
            tags="python,search",
            published_at=datetime(2024, 6, 1),
            word_count=len(post["body"].split()),
            body=post["body"],
        )
    writer.commit()
```

## 4. Delete Documents by Term

```python
from whoosh.query import Term

# Remove a post by its unique doc_id
with ix.writer() as writer:
    writer.delete_by_term("doc_id", "3")
    writer.commit()
```

## 5. Bulk Insert for Large Datasets (10k+ Documents)

```python
from whoosh.writing import BufferedWriter

# Use BufferedWriter for high-throughput indexing.
# It buffers documents and commits in batches.
buffered = BufferedWriter(ix, period=60, limit=500)

try:
    for doc in large_dataset:  # your generator/list of dicts
        with buffered:
            buffered.add_document(
                doc_id=doc["doc_id"],
                title=doc["title"],
                url=doc["url"],
                tags=",".join(doc["tags"]),
                published_at=doc["published_at"],
                word_count=doc["word_count"],
                body=doc["body"],
            )
finally:
    buffered.close()
```

## 6. Run a Search on the Indexed Data

```python
from whoosh.qparser import QueryParser

ix = index.open_dir("blog_index")

with ix.searcher() as s:
    qp = QueryParser("body", ix.schema)
    q = qp.parse("search engine")

    results = s.search(q, limit=10)
    for hit in results:
        print(f"Title: {hit['title']}")
        print(f"URL:   {hit['url']}")
        print(f"Score: {hit.score:.3f}")
        print(f"Snippet: {hit.highlights('body')}")
        print("---")


## DOCUMENT: Data Sources

# Data Sources

Whoosh-NG provides a flexible data source layer for indexing documents from SQL databases, REST APIs, GraphQL APIs, file-based formats, and custom data providers.

## DataSource Protocol

All data sources implement the `DataSource` protocol, which defines the interface for querying, schema discovery, and metadata retrieval.

```python
from whoosh_modern.data_sources import DataSource

class DataSource(Protocol):
    @property
    def name(self) -> str: ...

    def discover_schema(self) -> Schema: ...
    def iter_documents(self) -> Iterator[Document]: ...
    def stream_batches(self, batch_size: int = 1000) -> Iterator[list[dict[str, Any]]]: ...
    def health_check(self) -> bool: ...
```

### Capability Protocols

| Protocol | Description |
|----------|-------------|
| `DataSource` | Base protocol: name, schema, iteration, metadata |
| `IncrementalDataSource` | Supports `iter_changes(since)` |
| `AsyncDataSource` | Async document streaming via `aiter_documents()` |
| `RefreshableDataSource` | `refresh()` support |
| `CountableDataSource` | `document_count()` |
| `MetadataDataSource` | `metadata()` |
| `ObservableDataSource` | Observer callbacks for document changes |

---

## SQLSource

`SQLSource` connects to SQL databases and yields documents from query results with automatic connection pooling.

### Basic Usage

```python
from whoosh_modern.data_sources.sql import SQLSource
import sqlite3

conn = sqlite3.connect("mydb.db")
source = SQLSource(
    connection=conn,
    query="SELECT * FROM products",
)

# Discover schema from result-set metadata
schema = source.discover_schema()

# Iterate documents
for doc in source.iter_documents():
    print(doc["title"], doc["price"])

# Get metadata
meta = source.metadata()
# {"type": "sql", "query": "SELECT * FROM products", ...}

# Get document count
count = source.document_count()
```

### Connection Pooling (SQLSource)

Connection pooling is supported via `pool_size` for long-running processes:

```python
from whoosh_modern.data_sources.sql import SQLSource

source = SQLSource(
    connection="sqlite:///mydb.db",  # URL or connection object
    query="SELECT * FROM products",
    pool_size=10,          # Max connections in pool
)
```

### GROUP BY Aggregation

```python
source = SQLSource(
    connection=conn,
    query="""
        SELECT category, COUNT(*) as doc_count,
               AVG(price) as avg_price
        FROM products
        GROUP BY category
    """,
)

# Each aggregated row becomes a document
for doc in source.iter_documents():
    print(doc["category"], doc["doc_count"], doc["avg_price"])
```

### JOINs with Column Aliases

`SQLSource` uses result-set metadata for schema discovery. Always use column aliases in JOINs.

```python
source = SQLSource(
    connection=conn,
    query="""
        SELECT
            p.id AS product_id,
            p.name AS product_name,
            c.name AS category_name
        FROM products p
        JOIN categories c ON p.category_id = c.id
    """,
)
```

### Incremental Sync

```python
from datetime import datetime

source = SQLSource(
    connection=conn,
    query="SELECT * FROM articles",
    incremental_field="updated_at",
    id_field="id",
)

# Get documents changed since a timestamp
for doc in source.iter_changes(since=datetime(2025, 1, 1)):
    print(doc["id"], doc["updated_at"])
```

### SQLAlchemySource

For SQLAlchemy users, use `SQLAlchemySource` which supports engine-based connection management:

```python
from whoosh_modern.data_sources.sqlalchemy_ds import SQLAlchemySource
from sqlalchemy import create_engine

engine = create_engine("postgresql://user:pass@localhost/mydb")
source = SQLAlchemySource(
    engine=engine,
    query="SELECT * FROM articles",
    incremental_field="updated_at",
    id_field="id",
)

schema = source.discover_schema()
```

### PeeweeSource

For Peewee ORM users:

```python
from whoosh_modern.data_sources.peewee_ds import PeeweeSource
from peewee import SqliteDatabase

db = SqliteDatabase("mydb.db")
source = PeeweeSource(
    database=db,
    model=MyArticleModel,
    fields=["id", "title", "content"],
)

schema = source.discover_schema()
```

### TortoiseSource

For Tortoise ORM users (async):

```python
from whoosh_modern.data_sources.tortoise_ds import TortoiseSource

source = TortoiseSource(
    model="myapp.models.Article",
    fields=["id", "title", "content"],
)

schema = source.discover_schema()
count = source.document_count()
```

---

## RESTSource

`RESTSource` fetches documents from REST APIs with pagination and authentication.

### Basic Usage

```python
from whoosh_modern.data_sources.rest import RESTSource

source = RESTSource(
    url="https://api.example.com/v2/products",
    method="GET",
    headers={"Authorization": "Bearer your_token"},
    pagination="page",
    page_size=50,
)

# Discover schema from first page
schema = source.discover_schema()

# Iterate all documents (handles pagination automatically)
for doc in source.iter_documents():
    print(doc["name"], doc["price"])

# Get document count
count = source.document_count()
```

### Pagination Strategies

| Strategy | Parameters | Behavior |
|----------|-----------|----------|
| `page` | `?page=N&size=M` | Fetches page N with M items per page |
| `offset` | `?offset=N&limit=M` | Fetches M items starting at N |
| `cursor` | `?cursor=XYZ&size=M` | Follows `next_cursor` in response |

```python
# Page-based pagination
source = RESTSource(
    url="https://api.example.com/articles",
    pagination="page",
    page_size=50,
)

# Offset-based pagination
source = RESTSource(
    url="https://api.example.com/records",
    pagination="offset",
    page_size=100,
)

# Cursor-based pagination
source = RESTSource(
    url="https://api.example.com/feed",
    pagination="cursor",
    page_size=100,
)
```

### Authentication

```python
# Bearer token via headers
source = RESTSource(
    url="https://api.example.com/data",
    headers={"Authorization": "Bearer your_token"},
)

# API key via headers
source = RESTSource(
    url="https://api.example.com/data",
    headers={"X-API-Key": "your_api_key"},
)

# Basic auth via headers
import base64
creds = base64.b64encode(b"user:pass").decode()
source = RESTSource(
    url="https://api.example.com/data",
    headers={"Authorization": f"Basic {creds}"},
)
```

### Document Path

For nested API responses, use `document_path` to extract documents:

```python
# API returns: {"data": {"results": [...]}}
source = RESTSource(
    url="https://api.example.com/api/v2/products",
    document_path="data.results",
    pagination="page",
)
```

---

## GraphQLSource

`GraphQLSource` fetches documents from a GraphQL API endpoint:

```python
from whoosh_modern.data_sources.graphql import GraphQLSource

source = GraphQLSource(
    url="https://api.example.com/graphql",
    query="""
        query GetProducts($limit: Int!, $offset: Int!) {
            products(limit: $limit, offset: $offset) {
                id
                name
                price
                description
            }
        }
    """,
    pagination="offset",
    page_size=100,
    headers={"Authorization": "Bearer your_token"},
)

schema = source.discover_schema()
for doc in source.iter_documents():
    print(doc["id"], doc["name"])
```

---

## File-Based Data Sources

Whoosh-NG supports indexing from various file formats with optimized readers.

### FastCSVSource

High-performance CSV reader with configurable encoding and delimiter:

```python
from whoosh_modern.data_sources.fast_csv import FastCSVSource

source = FastCSVSource(
    file_path="data/products.csv",
    id_field="id",
    incremental_field="updated_at",
    delimiter=",",
    encoding="utf-8",
)

schema = source.discover_schema()
count = source.document_count()
for doc in source.iter_documents():
    print(doc)
```

### JSONSource

Index from JSON files or JSON Lines (.jsonl) files:

```python
from whoosh_modern.data_sources.json import JSONSource

# JSON array file
source = JSONSource(file_path="data/products.json")

# JSON Lines file (one JSON object per line)
source = JSONSource(
    file_path="data/logs.jsonl",
    format="jsonl",
)

schema = source.discover_schema()
```

### ParquetSource

Index from Parquet files using pyarrow or pandas backend:

```python
from whoosh_modern.data_sources.parquet_ds import ParquetSource

source = ParquetSource(
    file_path="data/large_dataset.parquet",
    engine="pyarrow",  # or "pandas"
    batch_size=1000,
)

schema = source.discover_schema()
```

### PandasSource

Index directly from a pandas DataFrame:

```python
from whoosh_modern.data_sources.pandas_ds import PandasSource
import pandas as pd

df = pd.read_csv("data/products.csv")
source = PandasSource(dataframe=df)

schema = source.discover_schema()
```

### PolarsSource

Index from a Polars DataFrame (faster, lazy evaluation):

```python
from whoosh_modern.data_sources.polars_ds import PolarsSource
import polars as pl

df = pl.read_csv("data/products.csv")
source = PolarsSource(dataframe=df)

schema = source.discover_schema()
```

---

## DataSourceConfig

For programmatic configuration, use `DataSourceConfig` to define data source properties:

```python
from whoosh_modern.data_sources.config import DataSourceConfig

config = DataSourceConfig(
    type="sql",
    connection=conn,
    query="SELECT * FROM products",
    id_field="id",
    incremental_field="updated_at",
)

source = config.create()
schema = source.discover_schema()
```

### Config File Support

Data source configurations can be loaded from dictionaries:

```python
from whoosh_modern.data_sources.config import DataSourceConfig

# From dict
config = DataSourceConfig.from_dict({
    "type": "rest",
    "url": "https://api.example.com/v2/products",
    "pagination": "page",
    "page_size": 50,
})
source = config.create()
```

Supported `type` values: `sql`, `sqlalchemy`, `rest`, `csv`, `json`, `graphql`,
`pydantic`, `pandas`, `polars`, `parquet`, `peewee`, `tortoise`.

### Available Data Sources

| Class | Source Type | Dependencies |
|-------|------------|--------------|
| `SQLSource` | SQLite, PostgreSQL, MySQL | `sqlite3` (stdlib) |
| `SQLAlchemySource` | Any SQLAlchemy-supported DB | `sqlalchemy` |
| `RESTSource` | REST APIs | none (stdlib `urllib`) |
| `GraphQLSource` | GraphQL APIs | none (stdlib `urllib`) |
| `FastCSVSource` | CSV files | none |
| `JSONSource` | JSON/JSONL files | none |
| `ParquetSource` | Parquet files | `pyarrow` or `pandas` |
| `PandasSource` | pandas DataFrames | `pandas` |
| `PolarsSource` | Polars DataFrames | `polars` |
| `PeeweeSource` | Peewee ORM | `peewee` |
| `TortoiseSource` | Tortoise ORM | `tortoise-orm` |
| `PydanticSource` | Pydantic models | `pydantic` |


## DOCUMENT: Facets

# Facet Manager

`FacetManager` manages facet configuration for a Whoosh `Schema`. It auto-discovers facetable fields and supports manual overrides.

## Basic Usage

```python
from whoosh.fields import Schema, TEXT, NUMERIC, BOOLEAN
from whoosh_modern.facets import FacetManager, TermsFacet, RangeFacet

schema = Schema(
    title=TEXT(stored=True),
    category=TEXT(sortable=True),
    price=NUMERIC(),
    active=BOOLEAN(),
)

manager = FacetManager(schema)
```

## Auto-Discovery

FacetManager automatically identifies facetable fields:

| Whoosh Field Type | Facet Type |
|-------------------|------------|
| `TEXT`, `KEYWORD`, `BOOLEAN`, `ID` | `TermsFacet` |
| `NUMERIC` | `RangeFacet` |
| `DATETIME` | `DateRangeFacet` |

```python
# Auto-discovered facets
facets = manager.get_facets()
# {"title": TermsFacet(limit=100), "category": TermsFacet(limit=100),
#  "price": RangeFacet("price", 0, 1000, 100), "active": TermsFacet(limit=100)}
```

> `RangeFacet` and `DateRangeFacet` are re-exports of the core
> `whoosh.sorting.RangeFacet` / `whoosh.sorting.DateRangeFacet`, so the facet objects
> returned by `FacetManager` can be passed directly to `searcher.search(..., groupedby=...)`.

## Manual Override

```python
from whoosh_modern.facets import TermsFacet

manager.set_manual_override("category", {
    "type": "terms",
    "limit": 50,
})
manager.set_manual_override("price", {
    "type": "range",
    "buckets": ["0-10", "10-50", "50-100", "100+"],
})
```

## Inspection

```python
# Is a field facetable?
manager.is_facetable("category")   # True
manager.is_facetable("title")      # False

# Get config for a field
config = manager.get_facet_config("category")
# {"type": "terms", "limit": 50}

# Get all configs
all_configs = manager.get_all_facet_configs()

# Get statistics
stats = manager.get_facet_stats()
# {
#     "total_fields": 4,
#     "auto_discovered_facets": 2,
#     "manual_overrides": 1,
#     "total_facets_configured": 3,
#     "facet_fields": ["category", "price", ...]
# }
```


## DOCUMENT: Fastapi Search

# FastAPI Integration

A complete, runnable FastAPI service exposing Whoosh‑NG search via HTTP.

## 1. Install

```bash
pip install whoosh-ng[api] fastapi uvicorn
```

## 2. Create the index

```python
# setup_index.py
import json
from whoosh import index
from whoosh.fields import Schema, TEXT, ID

schema = Schema(
    id=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    content=TEXT,
)

ix = index.create_in("docs_index", schema)

with ix.writer() as w:
    for doc in json.load(open("documents.json")):
        w.add_document(
            id=doc["id"],
            title=doc["title"],
            content=doc["content"],
        )
    w.commit()
```

## 3. REST API Service

```python
# main.py
from fastapi import FastAPI, Query
from typing import Optional
from whoosh import index
from whoosh.qparser import QueryParser
from whoosh_fastapi import create_app

ix = index.open_dir("docs_index")

# Option A: Use the helper
app = create_app(ix, prefix="/api/v1")

# Option B: Manual endpoints
# app = FastAPI(title="Document Search API", version="1.0.0")
#
# @app.get("/api/v1/health")
# async def health():
#     return {"status": "ok"}
#
# @app.post("/api/v1/search")
# async def search(q: str = Query(...), limit: int = 10):
#     with ix.searcher() as s:
#         parser = QueryParser("content", ix.schema)
#         results = s.search(parser.parse(q), limit=limit)
#         return {"hits": [dict(h) for h in results], "total": len(results)}
#
# @app.get("/api/v1/documents/{doc_id}")
# async def get_doc(doc_id: str):
#     with ix.searcher() as s:
#         from whoosh.query import Term
#         results = s.search(Term("id", doc_id))
#         if results:
#             return dict(results[0])
#         return {"error": "not found"}

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)
```

## 4. Run the server

```bash
uvicorn main:app --reload --port 8000
```

## 5. Test the API

```bash
# Health check
curl http://localhost:8000/api/v1/health

# Search
curl -X POST http://localhost:8000/api/v1/search \
  -H "Content-Type: application/json" \
  -d '{"q": "python search"}'

# Get document by ID
curl http://localhost:8000/api/v1/documents/doc1
```

## 6. Bulk Indexing Endpoint

```python
# Add to main.py for dynamic indexing
from fastapi import FastAPI
from whoosh.writing import BufferedWriter

@app.post("/api/v1/index")
async def index_docs(docs: list[dict]):
    with BufferedWriter(ix, period=30, limit=50) as w:
        for doc in docs:
            w.add_document(**doc)
    return {"indexed": len(docs)}
```

## 7. Alternative: WhooshFastAPI Class

For more control, use the `WhooshFastAPI` class directly:

```python
from fastapi import FastAPI
from whoosh import index
from whoosh.fields import Schema, TEXT, ID
from whoosh.qparser import QueryParser
from whoosh_fastapi import WhooshFastAPI

app = FastAPI()

schema = Schema(title=TEXT(stored=True), content=TEXT)
ix = index.create_in("indexdir", schema)

api = WhooshFastAPI(ix)
api.register_search_endpoint("/search", "content")
api.register_index_endpoint("/documents", schema)
```

## Key points

- `create_app()` from `whoosh_fastapi` provides `/health`, `/search`, `/autocomplete`, and `/suggest` endpoints.
- All blocking calls run off the event loop via `run_sync`.
- Use `BufferedWriter` for high-throughput indexing via POST.
- `WhooshFastAPI` class offers per-endpoint registration for custom integrations.

## WebSocket autocomplete

Pass an ``AutocompleteProvider`` to ``create_app`` to enable the WebSocket
autocomplete endpoint. The client sends JSON messages with a ``q`` key and
receives ``{"suggestions": [...]}`` responses over a persistent connection.

```python
from whoosh_modern.autocomplete import EdgeNgramAutocomplete
from whoosh_fastapi import create_app

autocomplete = EdgeNgramAutocomplete(ix)
app = create_app(ix, prefix="/api/v1", autocomplete=autocomplete)
```

Client example using JavaScript:

```javascript
const ws = new WebSocket("ws://localhost:8000/api/v1/autocomplete/ws");
ws.onmessage = (event) => console.log(JSON.parse(event.data));
ws.send(JSON.stringify({ q: "pyth" }));
// {"suggestions": ["python", "pythagorean"]}

// Custom limit
ws.send(JSON.stringify({ q: "pyth", limit: 5 }));
// {"suggestions": ["python", "pythagorean", ...]} // up to 5 suggestions
```

When no autocomplete provider is configured, the endpoint returns an empty
suggestions list instead of raising.


## DOCUMENT: Middleware Pipeline

# Middleware Pipeline

The middleware pipeline wraps operations with cross-cutting concerns: retry, logging, etc.

## Architecture

```python
from whoosh_modern.middleware import Middleware, MiddlewarePipeline, RetryMiddleware, LoggingMiddleware

# Chain middlewares
pipeline = MiddlewarePipeline(
    RetryMiddleware(attempts=3, backoff="exponential"),
    LoggingMiddleware(),
)

# Execute an operation through the chain
result = pipeline.execute(my_operation)
```

## RetryMiddleware

```python
from whoosh_modern.middleware import RetryMiddleware

retry = RetryMiddleware(attempts=3, backoff="exponential")

def flaky_operation():
    # Will retry up to 3 times on exception
    return fetch_data()

wrapped_op = retry.wrap(flaky_operation)
result = wrapped_op()
```

Backoff strategies:
- `"exponential"`: 1s, 2s, 4s, 8s...
- `"linear"`: 1s, 2s, 3s, 4s...

## LoggingMiddleware

```python
from whoosh_modern.middleware import LoggingMiddleware
import logging

logger = logging.getLogger("benchmark")
logging_mw = LoggingMiddleware(logger=logger, level=logging.INFO)

tracked_op = logging_mw.wrap(lambda: fetch_data())
result = tracked_op()
# Logs: "Operation wrapped completed in 0.123s"
# On error: "Operation wrapped failed after 0.123s: <error>"
```

## CacheMiddleware

```python
from whoosh_modern.middleware import CacheMiddleware

cache = CacheMiddleware(maxsize=128)

cached_op = cache.wrap(expensive_query)
result1 = cached_op(args)  # cache miss
result2 = cached_op(args)  # cache hit

print(cache.stats)  # {"hits": 1, "misses": 1, "size": 1}
cache.clear()
```

## Custom Middleware

```python
from whoosh_modern.middleware import Middleware

class TimingMiddleware(Middleware):
    def __init__(self):
        self.timings = []

    def wrap(self, operation):
        def wrapped(*args, **kwargs):
            start = time.time()
            try:
                result = operation(*args, **kwargs)
                self.timings.append(time.time() - start)
                return result
            except Exception:
                raise
        return wrapped
```


## DOCUMENT: Middleware

# Middleware Examples

Practical examples for building and using Whoosh-NG middleware.

## 1. Logging Middleware

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class LoggingMiddleware(Middleware):
    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        print(f"[SEARCH] Query: {context.query}")
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if context.results is not None:
            print(f"[RESULTS] Found {len(context.results)} hits")
        return context
```

## 2. Metrics Middleware

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class MetricsMiddleware(Middleware):
    def __init__(self) -> None:
        self._metrics = {}

    def after_index(self, context: MiddlewareContext) -> MiddlewareContext:
        self._metrics["documents_indexed"] = self._metrics.get("documents_indexed", 0) + 1
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        self._metrics["searches_executed"] = self._metrics.get("searches_executed", 0) + 1
        return context

    def get_metrics(self) -> dict:
        return dict(self._metrics)
```

## 3. Cache Middleware

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class SearchCacheMiddleware(Middleware):
    def __init__(self) -> None:
        self._cache = {}

    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if context.query and str(context.query) in self._cache:
            context.metadata["_cached_result"] = self._cache[str(context.query)]
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if context.query and context.results is not None:
            self._cache[str(context.query)] = context.results
        return context
```

## 4. Applying Middleware to an Index

```python
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.integration import apply_middleware_to_searcher

# Create middleware chain
chain = MiddlewareChain([
    LoggingMiddleware(),
    MetricsMiddleware(),
])

# Apply to a searcher
with ix.searcher() as base_searcher:
    searcher = apply_middleware_to_searcher(base_searcher, chain.middlewares)
    results = searcher.search(query)
```

## 5. Middleware Lifecycle

```python
class LifecycleMiddleware(Middleware):
    def startup(self, context):
        print("Middleware initialized")

    def shutdown(self, context):
        print("Middleware shutting down")

    def on_error(self, context, exc):
        print(f"Error: {exc}")
        raise exc
```

## Key Hooks

| Hook | Phase | Context |
|------|-------|---------|
| `startup` | Init | Called once on middleware init |
| `shutdown` | Cleanup | Called once on teardown |
| `before_index` | Indexing | Before document added |
| `after_index` | Indexing | After document added |
| `before_delete` | Deletion | Before document deleted |
| `after_delete` | Deletion | After document deleted |
| `before_search` | Search | Before query executed |
| `after_search` | Search | After results returned |
| `on_error` | Error | When exception occurs |
| `on_commit` | Commit | After writer.commit() |


## DOCUMENT: Movie Search

# Movie Search Application

A complete, runnable example showing how to build a small **movie search** application with Whoosh‑NG: schema design, indexing from a JSON dataset, faceted search, highlighting, and filtering.

## 1. Schema

```python
from whoosh.fields import Schema, TEXT, ID, KEYWORD, NUMERIC

schema = Schema(
    id=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    director=TEXT(stored=True),
    genre=KEYWORD(stored=True, commas=True, scorable=True),
    year=NUMERIC(int, stored=True),
    synopsis=TEXT,
)
```

## 2. Index the dataset

```python
import json
import shutil
from whoosh import index
from whoosh.fields import Schema, TEXT, ID, KEYWORD, NUMERIC

schema = Schema(
    id=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    director=TEXT(stored=True),
    genre=KEYWORD(stored=True, commas=True, scorable=True),
    year=NUMERIC(int, stored=True),
    synopsis=TEXT,
)

# Clean and create new index
shutil.rmtree("movies", ignore_errors=True)
ix = index.create_in("movies", schema)

movies = json.load(open("movies.json"))  # list of dicts

with ix.writer() as w:
    for m in movies:
        w.add_document(
            id=str(m["id"]),
            title=m["title"],
            director=m["director"],
            genre=",".join(m["genres"]),
            year=m["year"],
            synopsis=m["synopsis"],
        )
    w.commit()
```

Example `movies.json`:

```json
[
  {
    "id": 1,
    "title": "Blade Runner",
    "director": "Ridley Scott",
    "genres": ["sci-fi", "thriller"],
    "year": 1982,
    "synopsis": "A replicant hunter questions humanity in a rain-soaked future."
  },
  {
    "id": 2,
    "title": "Inception",
    "director": "Christopher Nolan",
    "genres": ["sci-fi", "action"],
    "year": 2010,
    "synopsis": "A thief who steals corporate secrets through dream-sharing technology."
  }
]
```

## 3. Search with facets and highlighting

```python
from whoosh import index
from whoosh.qparser import MultifieldParser
from whoosh.sorting import FieldFacet

ix = index.open_dir("movies")

qp = MultifieldParser(["title", "synopsis", "director"], ix.schema)

with ix.searcher() as s:
    q = qp.parse("future")

    # Search with sorting by year descending, grouped by genre
    results = s.search(
        q,
        sortedby=FieldFacet("year", reverse=True),
        groupedby=FieldFacet("genre", allow_overlap=True),
        limit=20,
    )

    for hit in results:
        print(hit["title"], hit["year"], "|", round(hit.score, 2))
        print("  ", hit.highlights("synopsis"))

    # Show genre facets
    print("\nGenres:", results.groups("genre"))
```

## 4. Filtering

Find sci-fi movies after 1990:

```python
from whoosh import index
from whoosh.qparser import QueryParser
from whoosh.query import Term, And, NumericRange

ix = index.open_dir("movies")
qp = QueryParser("synopsis", ix.schema)

with ix.searcher() as s:
    user_q = qp.parse("dream")
    filters = And([
        Term("genre", "sci-fi"),
        NumericRange("year", 1990, None),
    ])
    results = s.search(user_q, filter=filters)
    for hit in results:
        print(hit["title"], hit["year"])
```

## 5. Key takeaways

- `KEYWORD(commas=True)` stores multi-value fields that can be faceted.
- `MultifieldParser` searches multiple fields with optional boosts.
- `FieldFacet` enables faceted grouping and sorting.
- `hit.highlights()` returns highlighted snippets ready for display.


## DOCUMENT: Plugin Dev

# Plugin Development

Complete guide to building, registering, and testing custom Whoosh-NG plugins.

## 1. Plugin Base Class

All plugins inherit from `whoosh.plugins.base.Plugin`:

```python
from whoosh.plugins.base import Plugin

class MyPlugin(Plugin):
    name = "my_plugin"
    version = "1.0.0"
    depends_on = []  # Other plugins this requires
    conflicts_with = []  # Plugins that conflict
    priority = 0  # Load order (higher = later)
    middleware = []  # List of middleware class names

    def register(self, manager):
        """Called when plugin is loaded. Register your handlers here."""
        manager.register("my_handler", MyHandler())

    def register_hooks(self):
        """Register hooks using hookimpl decorator."""
        from whoosh.hooks import hookimpl, register_hook

        @hookimpl
        def on_search(request, response):
            # Hook logic here
            pass

        register_hook("on_search", hookimpl(on_search))
```

## 2. Registering a Plugin

### Manual Registration

```python
from whoosh.plugins.manager import PluginManager

plugin = MyPlugin()
PluginManager.register(plugin)
```

### Auto-Discovery via Entry Points

In `pyproject.toml`:

```toml
[project]
name = "whoosh-ng-my-plugin"

[project.entry-points."whoosh_ng.plugins"]
my_plugin = "my_package.plugin:MyPlugin"
```

Auto-load all registered plugins:

```python
from whoosh.plugins.manager import PluginManager

PluginManager.load_plugins()  # Uses 'whoosh.plugins' group by default
```

## 3. Provider Plugin Example

A provider plugin registers a new implementation for a registry:

```python
from whoosh.plugins.base import Plugin
from whoosh.registry import VectorRegistry

class MyVectorProvider:
    def search(self, query_vector, k=10):
        # Your vector similarity logic
        return [{"doc_id": "1", "score": 0.95}]

class MyVectorPlugin(Plugin):
    name = "my_vector"
    version = "1.0.0"

    def register(self, manager):
        provider = MyVectorProvider()
        VectorRegistry.register("my_vector", provider, self.name)
```

## 4. Creating a Custom Field Type

```python
from whoosh.fields import FieldType, TEXT
from whoosh.formats import Postings

class TagField(FieldType):
    scorable = True
    stored = True
    indexed = True
    format = Postings()

    def __init__(self, stored=True, scorable=True):
        super().__init__(format=Postings(), analyzer=None,
                        scorable=scorable, stored=stored)
```

## 5. Testing Your Plugin

```python
import pytest
from whoosh.plugins.manager import PluginManager
from whoosh.registry.base import Registry

class TestMyPlugin:
    def test_register(self):
        plugin = MyPlugin()
        manager = PluginManager()
        plugin.register(manager)
        assert "my_handler" in manager._plugins

    def test_entry_point(self):
        """Test that entry point loading works."""
        manager = PluginManager()
        manager.register(MyPlugin())
        assert "my_plugin" in manager.list_enabled()

    def test_conflict_detection(self):
        plugin1 = MyPlugin()
        plugin2 = ConflictingPlugin()
        manager = PluginManager()
        manager.register(plugin1)
        assert manager.detect_conflicts("my_plugin", "conflicting_plugin")
```

## 6. Async Plugin Methods

Plugins support async methods:

```python
from whoosh.plugins.base import Plugin
from typing import Awaitable

class AsyncPlugin(Plugin):
    name = "async_plugin"
    version = "1.0.0"

    async def register(self, manager):
        # Async initialization
        await some_async_setup()

    def register_hooks(self):
        from whoosh.hooks import hookimpl

        @hookimpl
        async def on_search(request, response):
            # Async hook
            await log_search_async(request)
```

## 7. Plugin Manager API

```python
from whoosh.plugins.manager import PluginManager

manager = PluginManager()

# Register a plugin instance
manager.register(MyPlugin())

# Enable/disable
manager.enable("my_plugin")
manager.disable("my_plugin")

# Check status
manager.list_plugins()   # All registered
manager.list_enabled()   # Enabled plugins

# Get plugin
plugin = manager.get("my_plugin")

# Version checking
manager.validate_version("my_plugin", "1.0.0")
```

## 8. Built-in Plugins

Whoosh-NG includes several built-in plugins:

- `whoosh_modern.vector` - Vector similarity search (NumPy provider)
- `whoosh_modern.autocomplete` - Inverted index autocomplete
- `whoosh_fastapi` - FastAPI REST endpoints

Load them:

```python
from whoosh.plugins.manager import PluginManager
from whoosh_modern.vector.plugin import VectorPlugin
from whoosh_modern.autocomplete.plugin import AutocompletePlugin

PluginManager.load_plugins()  # Auto-loads entry points
# Or manually:
manager = PluginManager()
manager.register(VectorPlugin())
manager.register(AutocompletePlugin())
```


## DOCUMENT: Schema Discovery

# Schema Discovery

Schema discovery infers Whoosh `Schema` from data source results. It operates on **actual result metadata and sample documents**, not SQL syntax.

## How It Works

### From Result Sets

```python
from whoosh_modern.schema_discovery import SchemaDiscovery
import sqlite3

conn = sqlite3.connect("benchmark/benchmark_data.db")
cursor = conn.cursor()
cursor.execute("PRAGMA table_info(reuters_articles)")
columns = [(row[1], row[2]) for row in cursor.fetchall()]
# [("id", "INTEGER"), ("article_date", "TEXT"), ("headline", "TEXT"), ...]

schema = SchemaDiscovery.from_result_set(columns)
```

### From Sample Documents

```python
from whoosh_modern.data_sources.sql import SQLSource

source = SQLSource(
    connection=conn,
    query="SELECT * FROM reuters_articles LIMIT 10",
)

# Get a few documents
docs = list(source.iter_documents())[:10]

# Infer schema from document values
schema = SchemaDiscovery.from_sample(docs)
```

### Detect ID Field

```python
id_field = SchemaDiscovery.detect_id_field(dict(schema))
# Returns "id" if an ID field is found, otherwise None
```

### Optimized Schema Discovery

```python
# Infer schema with optimization rules applied
# - drops non-searchable TEXT fields
# - converts TEXT fields ending in "id" to ID
# - converts boolean-like TEXT fields to BOOLEAN
schema = SchemaDiscovery.from_sample_optimized(
    docs, searchable_text=["title", "content"]
)
```

## SQL Type Mapping

| SQL Type | Whoosh Field |
|----------|-------------|
| `VARCHAR`, `TEXT`, `CHAR`, `STRING_AGG` | `TEXT` |
| `INTEGER`, `BIGINT`, `SMALLINT`, `COUNT`, `SUM`, `AVG`, `MIN`, `MAX` | `NUMERIC` |
| `FLOAT`, `DOUBLE`, `DECIMAL` | `NUMERIC` |
| `BOOLEAN` | `BOOLEAN` |
| `DATE`, `TIMESTAMP` | `DATETIME` |
| `UUID` | `ID` |
| `JSON`, `ENUM` | `KEYWORD` |

## Duplicate Column Detection

`from_result_set` raises `SchemaDiscoveryError` on duplicate column names.

```python
columns = [
    ("id", "INTEGER"),
    ("headline", "TEXT"),
    ("id", "INTEGER"),  # Duplicate!
]

try:
    schema = SchemaDiscovery.from_result_set(columns)
except SchemaDiscoveryError as e:
    print(f"Duplicate column: {e.field}")
```

Use explicit SQL aliases to avoid duplicates:

```sql
SELECT
    p.id AS product_id,
    c.id AS category_id
FROM products p
JOIN categories c ON p.category_id = c.id
```


## DOCUMENT: Search Models

# Search Models

Examples for auto-mapping Python models to Whoosh schemas.

## Dataclass

```python
from dataclasses import dataclass
from whoosh.fields import Schema, TEXT, NUMERIC
from whoosh_modern.models import register_dataclass_model
import tempfile
import shutil

@dataclass
class Book:
    title: str
    year: int
    tags: list[str] | None = None

idx = register_dataclass_model(Book)
print(idx.schema)
```

## Pydantic

```python
from pydantic import BaseModel
from whoosh_modern.models import register_pydantic_model

class BookModel(BaseModel):
    title: str
    year: int
    tags: list[str] | None = None

idx = register_pydantic_model(BookModel)
schema = idx.schema
```

## SQLAlchemy

```python
from sqlalchemy import Column, Integer, String
from whoosh_modern.models import register_sqlalchemy_model

class BookSQL:
    __tablename__ = "book"
    title = Column(String, info={"search": {"fulltext": True, "stored": True}})
    year = Column(Integer, info={"search": {"sortable": True}})

idx = register_sqlalchemy_model(BookSQL)
schema = idx.schema
```

## SQLModel

```python
from sqlmodel import SQLModel, Field
from whoosh_modern.models import register_sqlmodel_model

class Book(SQLModel, table=True):
    id: int = Field(primary_key=True)
    title: str = Field(sa_column_kwargs={"info": {"search": {"fulltext": True}}})
    year: int

idx = register_sqlmodel_model(Book)
schema = idx.schema
```

## msgspec

```python
import msgspec
from whoosh_modern.models import register_msgspec_model

class Book(msgspec.Struct):
    title: str = msgspec.field(metadata={"search": {"fulltext": True}})
    year: int

idx = register_msgspec_model(Book)
schema = idx.schema
```

## Indexing documents

```python
from whoosh import index

tmp = tempfile.mkdtemp()
ix = index.create_in(tmp, schema)

with ix.writer() as w:
    book = Book(title="Whoosh Guide", year=2024, tags=["python", "search"])
    doc = idx.to_whoosh_document(book)
    w.add_document(**doc)
    w.commit()
```

## Auto-indexing with AutoIndexer

```python
from whoosh_modern.models import AutoIndexer

auto = AutoIndexer(ix, on_error="raise")
auto.register(Book)

# Index a single instance
book = Book(title="New Book", year=2024, tags=["python"])
auto.index(book)

# Remove by ID
auto.remove(book)

# Async versions
await auto.index_async(book)
await auto.remove_async(book)
```

For SQLAlchemy models, `AutoIndexer` automatically hooks into `after_insert`, `after_update`, and `after_delete` events.

## Cleanup

```python
shutil.rmtree(tmp)
```


## DOCUMENT: Search View

# SearchView

`SearchView` integrates a `DataSource` with Whoosh indexing. It discovers
schema, validates the source, builds the index, and supports incremental
refresh and schema evolution.

## Basic Usage

```python
from whoosh_modern.views import SearchView
from whoosh_modern.data_sources.sql import SQLSource
import sqlite3

conn = sqlite3.connect("data/articles.db")
source = SQLSource(
    connection=conn,
    query="SELECT * FROM articles",
    incremental_field="updated_at",
    id_field="id",
)

view = SearchView(
    name="articles",
    source=source,
)

# Build index (discovers schema, validates, populates)
ix = view.build("indexdir")
```

## Full Reindex vs Incremental Refresh

```python
# Full reindex (clears and rebuilds the entire index)
count = view.reindex()

# Incremental refresh (only changed documents since last sync)
count = view.refresh()
```

## Validation

```python
# Run validation before building
results = view.validate()
for result in results:
    print(f"Level {result.level}: {'PASS' if result.passed else 'FAIL'}")
    for error in result.errors:
        print(f"  ERROR: {error}")
    for warning in result.warnings:
        print(f"  WARNING: {warning}")
```

Validation levels:
1. **Structural** — DataSource availability, schema detection
2. **Search** — Indexable fields, term vectors, searchable analyzers
3. **Performance** — Performance warnings (e.g., TEXT fields on large datasets)
4. **Runtime** — Sample iteration, type conformance

## Field Overrides

Customize field types after schema discovery:

```python
from whoosh.fields import TEXT, NUMERIC, DATETIME

view = SearchView(
    name="custom",
    source=source,
    fields={
        "title": TEXT(stored=True, phrase=False),
        "price": NUMERIC(int, sortable=True),
        "published": DATETIME(sortable=True),
    },
)
```

## Facets

Configure facet settings:

```python
view = SearchView(
    name="faceted",
    source=source,
    facets={
        "category": {"type": "terms", "limit": 50},
        "price": {"type": "range", "buckets": ["0-100", "100-500", "500+"]},
    },
)
```

## Middleware

Attach middleware to the search pipeline:

```python
from whoosh_modern.views import SearchView
from whoosh_modern.middleware import LoggingMiddleware, RetryMiddleware

view = SearchView(
    name="with_middleware",
    source=source,
    middleware=[
        RetryMiddleware(attempts=3, backoff="exponential"),
        LoggingMiddleware(),
    ],
)
```

## Strict Mode

```python
view = SearchView(
    name="strict",
    source=source,
    strict=True,  # Raise ValidationError on any validation failure
)
```

## Schema Evolution

Add new fields to an existing index without a full reindex:

```python
view = SearchView(name="articles", source=source)
view.build("indexdir")

# Add a new field
view.evolve_schema({
    "new_field": TEXT(stored=True),
})
```

### Schema Version Checking

```python
view = SearchView(name="articles", source=source, schema_version="2.1")
view.build("indexdir")

# Check if the stored schema version matches
if not view.check_schema_version():
    print("Schema version mismatch, consider reindexing")
```


## DOCUMENT: Search

# Search Examples

Real, runnable search examples with Whoosh‑NG. Each section is a self-contained
script you can copy into a `.py` file and run.

> **Real-world scenario**: You built a book‑catalogue index (see
> `docs/_en/examples/basic-indexing.md`). Below are the search patterns
> you'll need for a production‑ready book search page.

## Prerequisites

The examples assume an index exists at `book_index/` with this schema:

```python
from whoosh import index
from whoosh.fields import Schema, TEXT, ID, NUMERIC, DATETIME

schema = Schema(
    isbn=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    author=TEXT(stored=True),
    content=TEXT,
    genre=KEYWORD(stored=True, commas=True),
    published_year=NUMERIC(int, stored=True, sortable=True),
    rating=NUMERIC(float, stored=True, sortable=True),
)
```

## 1. Basic Search — "Find books about Python"

```python
from whoosh import index
from whoosh.qparser import QueryParser

ix = index.open_dir("book_index")

with ix.searcher() as s:
    qp = QueryParser("content", ix.schema)
    q = qp.parse("python")

    results = s.search(q, limit=10)
    for hit in results:
        print(f"{hit['title']} by {hit['author']} (ISBN: {hit['isbn']}) — score={hit.score:.2f}")
```

## 2. Multi-field Search with Boosts

Search across `title`, `author`, and `content` simultaneously. Title matches
are boosted 3× so they rank higher:

```python
from whoosh.qparser import MultifieldParser

ix = index.open_dir("book_index")
qp = MultifieldParser(
    ["title", "author", "content"],
    ix.schema,
    fieldboosts={"title": 3.0, "author": 2.0, "content": 1.0},
)

q = qp.parse("clean code")

with ix.searcher() as s:
    results = s.search(q, limit=10)
    for hit in results:
        print(f"{hit['title']} — {hit['author']}")
```

## 3. Pagination — "Page 3 of search results"

```python
ix = index.open_dir("book_index")
qp = QueryParser("content", ix.schema)
q = qp.parse("machine learning")

with ix.searcher() as s:
    page = s.search_page(q, 3, pagelen=15)  # Page 3, 15 results per page

    print(f"Page {page.number} / {page.pagecount}  ({page.total} results total)")
    for hit in page:
        print(f"  {hit['title']}")
```

## 4. Sort and Filter — "High-rated sci-fi books after 2010"

```python
from whoosh.query import Term, And, NumericRange
from whoosh.sorting import FieldFacet, ScoreFacet

ix = index.open_dir("book_index")
qp = QueryParser("content", ix.schema)
q = qp.parse("space")

with ix.searcher() as s:
    # Filter: genre must be "sci-fi" AND year >= 2010
    filters = And([
        Term("genre", "sci-fi"),
        NumericRange("published_year", 2010, None),
    ])

    results = s.search(
        q,
        filter=filters,
        sortedby=FieldFacet("rating", reverse=True),  # highest-rated first
        limit=20,
    )
    for hit in results:
        print(f"{hit['title']} ({hit['published_year']}) — rating: {hit['rating']}")
```

## 5. Highlighting — "Show users where their query matched"

```python
ix = index.open_dir("book_index")
qp = QueryParser("content", ix.schema)
q = qp.parse("neural networks")

with ix.searcher() as s:
    results = s.search(q, limit=5)

    for hit in results:
        snippet = hit.highlights("content", top=2)  # show 2 best fragments
        print(f"{hit['title']}:")
        print(f"  {snippet}")
        print()
```

## 6. Date / Numeric Range Search — "Books published in 2023"

```python
from whoosh.query import NumericRange

ix = index.open_dir("book_index")

with ix.searcher() as s:
    q = NumericRange("published_year", 2023, 2023)
    results = s.search(q)
    print(f"{results.total} books published in 2023")
```

## 7. Prefix Search — "All books starting with 'Deep'"

```python
from whoosh.query import Prefix

ix = index.open_dir("book_index")

with ix.searcher() as s:
    q = Prefix("title", "Deep")  # titles starting with "Deep"
    results = s.search(q)
    for hit in results:
        print(hit["title"])
```

## 8. Faceted Search — "Group results by genre"

```python
from whoosh.sorting import FieldFacet

ix = index.open_dir("book_index")
qp = QueryParser("content", ix.schema)
q = qp.parse("programming")

with ix.searcher() as s:
    results = s.search(q, groupedby=FieldFacet("genre"))

    # Show top genres alongside results
    for genre, group in results.groups("genre").items():
        print(f"{genre}: {len(group)} hits")
```

## Key points

- `QueryParser` parses a string into a `Query` object.
- `MultifieldParser` searches multiple fields with optional per-field boosts.
- `search_page()` handles pagination automatically.
- `filter` restricts results without affecting relevance scores.
- `sortedby` sorts by field value or relevance score.
- `hit.highlights()` returns highlighted snippets ready for display.
- `groupedby` enables faceted result grouping.


## DOCUMENT: Validation

# Validation Framework

The validation framework runs checks against a data source before indexing. It provides 4 distinct validation levels with different failure modes.

## Validation Levels

| Level | Method | Purpose |
|-------|--------|---------|
| **Level 1** | `validate_structural(source)` | DataSource availability, schema detection |
| **Level 2** | `validate_search(schema)` | Indexable fields, analyzer compatibility |
| **Level 3** | `validate_performance(schema, source)` | Performance warnings (TEXT fields, etc.) |
| **Level 4** | `validate_runtime(source, sample_size)` | Sample iteration, type validation |

## Basic Usage

```python
from whoosh_modern.validation import ValidationFramework, ValidationResult
from whoosh_modern.data_sources.sql import SQLSource
import sqlite3

conn = sqlite3.connect("benchmark/benchmark_data.db")
source = SQLSource(connection=conn, query="SELECT * FROM reuters_articles")

validator = ValidationFramework()

# Run all 4 validation levels
results: list[ValidationResult] = validator.validate(source)

for result in results:
    level_name = f"Level {result.level}"
    status = "PASS" if result.passed else "FAIL"
    print(f"{level_name}: {status}")
    for error in result.errors:
        print(f"  ERROR: {error}")
    for warning in result.warnings:
        print(f"  WARN: {warning}")
```

## Individual Level Validation

```python
# Level 1: Structural
errors = validator.validate_structural(source)

# Level 2: Search
from whoosh.fields import Schema
schema = source.discover_schema()
errors = validator.validate_search(schema)

# Level 3: Performance
warnings = validator.validate_performance(schema, source)

# Level 4: Runtime
errors = validator.validate_runtime(source, sample_size=100)
```

## Validation Results

```python
@dataclass
class ValidationResult:
    level: int
    passed: bool
    warnings: list[str]
    errors: list[str]
```


## DOCUMENT: Vector Search

# Vector Search with Whoosh‑NG

This example shows how to enable **semantic/vector search** using the optional `vector` extra. We index document embeddings and perform k-nearest neighbour (k-NN) search.

## 1. Install Optional Dependencies

```bash
pip install "whoosh-ng[vector]" numpy
```

## 2. Schema with a Vector Field

```python
from whoosh.fields import Schema, TEXT, ID, VECTOR
from whoosh.vector import VectorField

schema = Schema(
    doc_id=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    content=TEXT,
    embedding=VECTOR(stored=True, dim=128),  # 128-dimensional embedding
)
```

## 3. Indexing Vectors

```python
import numpy as np
from whoosh import index

shutil.rmtree("vector_index", ignore_errors=True)
ix = index.create_in("vector_index", schema)

# Simulate embeddings (in practice, use a model like SentenceTransformer)
documents = [
    {"doc_id": "doc1", "title": "Python Basics", "content": "Learn Python programming fundamentals."},
    {"doc_id": "doc2", "title": "Advanced Python", "content": "Deep dive into Python decorators and metaclasses."},
    {"doc_id": "doc3", "title": "Data Science", "content": "Pandas and NumPy for data analysis."},
]

# Generate random embeddings for demo
np.random.seed(42)
embeddings = {d["doc_id"]: np.random.rand(128).astype(np.float32) for d in documents}

with ix.writer() as w:
    for doc in documents:
        w.add_document(
            doc_id=doc["doc_id"],
            title=doc["title"],
            content=doc["content"],
            embedding=embeddings[doc["doc_id"]].tobytes(),
        )
    w.commit()
```

## 4. Vector Search with NumpyProvider

```python
from whoosh_modern.vector import VectorField
from whoosh_modern.vector.numpy_provider import NumpyProvider
from whoosh_modern.vector.plugin import VectorPlugin
from whoosh.plugins.manager import PluginManager

# Register the vector plugin
VectorPlugin().register(PluginManager())

# Create provider and add vectors
provider = NumpyProvider()
for doc_id, vec in embeddings.items():
    provider.add([(doc_id, vec.tolist())])

# Search: find 2 most similar docs to a query vector
query_vec = embeddings["doc1"]  # use doc1's embedding as query
hits = provider.search(query_vec, k=2)

for hit in hits:
    print(f"doc_id={hit.doc_id}, score={hit.score:.3f}")
```

## 5. Using VectorField for Serialization

```python
from whoosh.vector import VectorField

vf = VectorField(dimension=128, name="embedding")

# Convert list to bytes for storage
values = [0.1, 0.2, 0.3, 0.4] + [0.0] * 124  # 128 values
raw = vf.vector_to_bytes(values)

# Restore from bytes
restored = vf.bytes_to_vector(raw)
print(restored == tuple(values))  # True
```

## 6. Key Takeaways

- Install with `pip install whoosh-ng[vector]` to get `whoosh_modern.vector`.
- `VECTOR` field stores raw bytes; use `VectorField` to convert to/from Python lists.
- `NumpyProvider` implements cosine similarity via dot product.
- Register the plugin via `VectorPlugin().register(manager)` or use `PluginManager.load_plugins()`.
- Use `filter_ids` in `provider.search()` to restrict to a subset of documents.


## DOCUMENT: Auto Indexing

# Auto-Indexing

Whoosh-NG provides utilities for automatic schema discovery and data-source driven indexing.

## Schema Discovery

The `SchemaDiscovery` utility inspects a data source and auto-generates a Whoosh schema:

```python
from whoosh_modern.discovery import SchemaDiscovery

discovery = SchemaDiscovery(source=data_source)
schema = discovery.discover()
```

See [SearchView](/examples/search-view) and [Data Sources](/examples/data-sources) for usage examples.


## DOCUMENT: Autocomplete Providers

# Autocomplete Providers

Module: `whoosh_modern.autocomplete`
Version: 2.0.0

The autocomplete module provides multiple provider strategies for query suggestion and type-ahead search. All providers implement a common interface so you can swap strategies at runtime. Providers are registered via the `AutocompleteRegistry` and loaded through entry points.

## Module Overview

```text
whoosh_modern.autocomplete
    ├── provider.py   # AutocompleteHit, AutocompleteProvider (Protocol)
    ├── ngram.py      # NGramProvider (character n-gram based)
    ├── edge_ngram.py # InvertedIndexAutocomplete (inverted index prefix matching)
    ├── fuzzy.py      # FuzzySuggestProvider (approximate matching via rapidfuzz)
    ├── factory.py    # create_autocomplete() factory
    └── plugin.py     # AutocompletePlugin (entry-point plugin)
```

## AutocompleteProvider (Base Class)

Located in `whoosh_modern.autocomplete.provider`:

```python
from whoosh_modern.autocomplete.provider import AutocompleteProvider, AutocompleteHit

class MyProvider(AutocompleteProvider):
    def add(self, phrases: Iterable[str]) -> None:
        """Add phrases to the provider's index."""
        ...

    def search(self, prefix: str, limit: int = 10) -> list[AutocompleteHit]:
        """Return autocomplete suggestions for the given prefix."""
        ...
```

### AutocompleteHit

A simple result object returned by providers:

```python
class AutocompleteHit:
    def __init__(self, text: str, score: float) -> None:
        self.text = text    # The matched phrase
        self.score = score  # Relevance score (higher = better)
```

## Built-in Providers

### InvertedIndexAutocomplete

Located in `whoosh_modern.autocomplete.edge_ngram`. Uses simple prefix matching against an in-memory list:

```python
from whoosh_modern.autocomplete.edge_ngram import InvertedIndexAutocomplete

provider = InvertedIndexAutocomplete()
provider.add(["python", "pyramid", "pytorch", "java", "javascript"])

hits = provider.search("py", limit=5)
for hit in hits:
    print(f"{hit.text} (score: {hit.score})")
# Output:
# python (score: 0.45)
# pyramid (score: 0.43)
# pytorch (score: 0.43)
```

**Scoring**: Exact prefix matches get a 1.5x bonus; base score is `1.0 / (len(phrase) + 1)`.

### NGramProvider

Located in `whoosh_modern.autocomplete.ngram`. Builds a character n-gram index for fuzzy substring matching:

```python
from whoosh_modern.autocomplete.ngram import NGramProvider

provider = NGramProvider(n=3)
provider.add(["python programming", "java development", "rust language"])

hits = provider.search("pyt", limit=5)
for hit in hits:
    print(f"{hit.text} (score: {hit.score})")
```

**Parameters:**

| Parameter | Type | Default | Description                          |
|-----------|------|---------|--------------------------------------|
| `n`       | `int` | `3`     | Size of character n-grams            |

**How it works**: N-grams are extracted from each phrase (lowercased). During search, n-grams from the prefix are matched against the index. Phrases with more matching n-gram occurrences receive higher scores.

### FuzzySuggestProvider

Located in `whoosh_modern.autocomplete.fuzzy`. Uses `rapidfuzz` for approximate string matching (typos, partial matches):

```python
from whoosh_modern.autocomplete.fuzzy import FuzzySuggestProvider

# Requires: pip install whoosh-ng[fuzzy]
provider = FuzzySuggestProvider(max_distance=2, score_cutoff=50.0)
provider.add(["python", "pyramid", "pytorch", "java", "javascript"])

hits = provider.search("pythn", limit=5)  # Typo in "python"
for hit in hits:
    print(f"{hit.text} (score: {hit.score})")
# Output: python (score: 0.95), ...
```

**Parameters:**

| Parameter       | Type  | Default  | Description                              |
|-----------------|-------|----------|------------------------------------------|
| `max_distance`  | `int` | `2`      | Maximum edit distance (unused by rapidfuzz directly, reserved for future use) |
| `score_cutoff`  | `float` | `50.0` | Minimum similarity score (0-100 scale)   |

**Note**: Requires `rapidfuzz` (`pip install whoosh-ng[fuzzy]`). Falls back to `ImportError` if not installed.

## Factory Function

Located in `whoosh_modern.autocomplete.factory`:

```python
from whoosh_modern.autocomplete import create_autocomplete

# Create any provider by name
provider = create_autocomplete("inverted")   # InvertedIndexAutocomplete
provider = create_autocomplete("ngram", n=3) # NGramProvider with custom n
provider = create_autocomplete("fuzzy", max_distance=2, score_cutoff=60.0)
```

**Available providers:**

| Name        | Class                    | Optional Dependency |
|-------------|--------------------------|---------------------|
| `"inverted"`| `InvertedIndexAutocomplete` | None              |
| `"ngram"`   | `NGramProvider`          | None               |
| `"fuzzy"`   | `FuzzySuggestProvider`   | `rapidfuzz`        |

## Registering with the AutocompleteRegistry

Providers are registered into `whoosh.registry.AutocompleteRegistry` (a `Registry` instance):

```python
from whoosh.registry import AutocompleteRegistry
from whoosh_modern.autocomplete import create_autocomplete

# Register a provider
provider = create_autocomplete("ngram", n=3)
AutocompleteRegistry.register("ngram-suggester", provider, owner="my_app")

# Retrieve it later
suggester = AutocompleteRegistry.get("ngram-suggester")

# List all registered providers
print(AutocompleteRegistry.list_keys())
```

## AutocompletePlugin (Entry Point)

Located in `whoosh_modern.autocomplete.plugin`, this is the built-in plugin registered via the `whoosh_ng.plugins` entry-point group:

```python
from whoosh_modern.autocomplete.plugin import AutocompletePlugin

# Automatically loaded by PluginManager.load_plugins()
# Registers "inverted" provider in AutocompleteRegistry
```

### Entry Point Declaration

In `pyproject.toml`:

```toml
[project.entry-points."whoosh_ng.plugins"]
whoosh_autocomplete = "whoosh_modern.autocomplete.plugin:AutocompletePlugin"
```

### Plugin Details

```python
class AutocompletePlugin(Plugin):
    name = "whoosh_autocomplete"
    version = "3.0.0"

    def register(self, manager):
        # Registers InvertedIndexAutocomplete as "inverted"
        AutocompleteRegistry.register(
            "inverted", create_autocomplete("inverted"), self.name
        )

    def register_hooks(self):
        # Registers an on_search hook (currently a no-op)
        from whoosh.hooks import hookimpl, register_hook
        register_hook("on_search", hookimpl(on_search))
```

## Usage Examples

### Basic Usage

```python
from whoosh_modern.autocomplete import create_autocomplete

# Create and populate a provider
provider = create_autocomplete("inverted")
provider.add([
    "python programming",
    "python tutorial",
    "java tutorial",
    "javascript framework",
])

# Search for suggestions
hits = provider.search("py", limit=3)
for hit in hits:
    print(f"{hit.text}: {hit.score:.3f}")
```

### Using Fuzzy Matching with Typo Tolerance

```python
from whoosh_modern.autocomplete import create_autocomplete

provider = create_autocomplete("fuzzy", score_cutoff=70.0)
provider.add(["python", "pytorch", "tensorflow", "keras"])

# Even with a typo, relevant suggestions are returned
hits = provider.search("pyton", limit=5)
for hit in hits:
    print(hit.text, hit.score)
```

### Using N-Gram Matching for Partial Words

```python
from whoosh_modern.autocomplete import create_autocomplete

# Use 3-grams for better substring matching
provider = create_autocomplete("ngram", n=3)
provider.add(["machine learning", "deep learning", "neural networks"])

# Finds phrases containing the n-grams of "machin"
hits = provider.search("machin", limit=5)
```

### Integration with Search

```python
from whoosh_modern.autocomplete import create_autocomplete

# Build the autocomplete provider
provider = create_autocomplete("inverted")
provider.add(["python", "java", "javascript", "go", "rust"])

# Use in a search endpoint
def suggest(prefix: str, limit: int = 5):
    hits = provider.search(prefix, limit=limit)
    return [{"text": h.text, "score": h.score} for h in hits]

# In your FastAPI/REST endpoint:
# GET /api/suggest?q=py&limit=5
# Response: [{"text": "python", "score": 0.45}, ...]
```

## Comparison of Providers

| Provider              | Matching       | Strengths                    | Weaknesses                | Dependency    |
|-----------------------|----------------|------------------------------|---------------------------|---------------|
| `inverted`            | Prefix         | Simple, fast, no deps        | No typo tolerance         | None          |
| `ngram`               | N-gram overlap | Substring matching, flexible | Slower than prefix        | None          |
| `fuzzy`               | Edit distance  | Typo tolerance, flexible     | Requires rapidfuzz        | `rapidfuzz`   |

## Installation

```bash
# Core autocomplete (inverted + n-gram)
pip install whoosh-ng

# With fuzzy matching
pip install whoosh-ng[fuzzy]

# Full modern analysis
pip install whoosh-ng[modern]
```

## See Also

- [Plugin System Guide](plugins-advanced.md) — Plugin registration and discovery
- [Middleware Guide](middleware-pipeline.md) — Middleware pipeline integration
- [Provider Integration Guide](provider-integration.md) — Complete pipeline guide for all providers
- [API: Modern](../api/modern.md) — Full API reference for autocomplete extensions


## DOCUMENT: Autocomplete

# Autocomplete

An optional edge-ngram style autocomplete layer for Whoosh-NG.

## Install

```bash
pip install whoosh-ng[autocomplete]
```

## Minimal index

```python
from whoosh.fields import Schema, TEXT, KEYWORD

schema = Schema(
    title=TEXT(stored=True),
    tags=KEYWORD(stored=True, commas=True),
)

with ix.writer() as writer:
    writer.add_document(title="Python Quickstart", tags="python,quickstart")
    writer.commit()
```

## Query autocomplete

```python
from whoosh_modern.autocomplete import create_autocomplete

# Create provider (inverted = prefix matching by default)
provider = create_autocomplete("inverted")

# Index terms (typically done at index time)
provider.add(["python", "quickstart", "programming", "pyramid"])

# Search for suggestions
hits = provider.search("py", limit=5)
for hit in hits:
    print(hit.text, hit.score)
# Output: python 0.9, pyramid 0.8
```

## Modern Autocomplete Providers (Whoosh-NG 2.0)

Whoosh-NG 2.0 introduces multiple autocomplete provider strategies: `InvertedIndexAutocomplete`, `NGramProvider`, and `FuzzySuggestProvider`. For full details on creating, registering, and switching providers, see the [Autocomplete Providers Guide](autocomplete-providers.md).

## Autocomplete Provider Integration in the Pipeline

Autocomplete providers operate in **two modes**: standalone (in-memory index of phrases) and registry-based (discovered via `AutocompleteRegistry`). The `AutocompletePlugin` registers the default `"inverted"` provider at startup.

### Architecture

```text
┌─────────────────────────────────────────────────────────────────┐
│  Registration (startup)                                         │
│                                                                 │
│  AutocompletePlugin.register(PluginManager)                     │
│    └── AutocompleteRegistry.register("inverted", provider)      │
│                                                                 │
│  The "inverted" provider is now available globally              │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│  Mode 1: Standalone (in-memory index)                           │
│                                                                 │
│  provider = create_autocomplete("inverted")                     │
│  provider.add(["python", "java", "javascript"])                 │
│  hits = provider.search("py", limit=5)                          │
│  └── [AutocompleteHit(text="python", score=0.9), ...]           │
│                                                                 │
│  No Whoosh index required. Pure in-memory lookup.               │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│  Mode 2: Registry-based (tied to a Whoosh index)                │
│                                                                 │
│  provider = AutocompleteRegistry.get("inverted")                │
│                                                                 │
│  # Populate from index terms                                    │
│  with ix.searcher() as s:                                       │
│      for term in s.reader().all_terms():                       │
│          provider.add_term(term, s.doc_freq(term))              │
│                                                                 │
│  # Query suggestions                                            │
│  hits = provider.suggest("py", maxdist=1, limit=5)              │
│  └── [AutocompleteHit(text="python", score=...), ...]           │
└─────────────────────────────────────────────────────────────────┘
```

### Full workflow: indexing + autocomplete

```python
from whoosh import index, fields
from whoosh_modern.autocomplete import create_autocomplete
from whoosh_modern.autocomplete.plugin import AutocompletePlugin
from whoosh.plugins.manager import PluginManager

# 1. Register autocomplete plugin at startup
manager = PluginManager()
AutocompletePlugin().register(manager)

# 2. Create index
schema = fields.Schema(
    title=fields.TEXT(stored=True),
    content=fields.TEXT,
)
ix = index.create_in("indexdir", schema)

# 3. Index documents
with ix.writer() as writer:
    writer.add_document(title="Python programming", content="Learn Python")
    writer.add_document(title="Java development", content="Learn Java")
    writer.add_document(title="JavaScript basics", content="Learn JS")
    writer.commit()

# 4. Build autocomplete provider from index terms
provider = create_autocomplete("inverted")

with ix.searcher() as searcher:
    reader = searcher.reader()
    for term in reader.all_terms():
        # Add term with its document frequency as score weight
        provider.add([term.decode("utf-8")])

# 5. Query autocomplete
hits = provider.search("py", limit=5)
for hit in hits:
    print(f"{hit.text} (score: {hit.score:.3f})")
# Output: python (score: 0.429)
```

### Provider strategies

| Provider | Strategy | Best for |
|----------|----------|----------|
| `InvertedIndexAutocomplete` | Prefix matching with scoring | Simple autocomplete, small vocabularies |
| `NGramProvider` | Character n-gram indexing | Substring matching, typo tolerance |
| `FuzzySuggestProvider` | Approximate matching via rapidfuzz | Typo-tolerant suggestions |

```python
# Prefix matching (default)
provider = create_autocomplete("inverted")
provider.add(["python", "pyramid", "pyodbc"])
hits = provider.search("py")
# → python, pyramid, pyodbc

# N-gram matching
provider = create_autocomplete("ngram", n=3)
provider.add(["python", "java", "javascript"])
hits = provider.search("pyt")
# → python (matched by "pyt" n-gram)

# Fuzzy matching
provider = create_autocomplete("fuzzy", max_distance=2, score_cutoff=60.0)
provider.add(["python", "pyramid", "pyodbc"])
hits = provider.search("pythn")  # typo: missing 'o'
# → python (fuzzy match)
```

### Integration with search middleware

Autocomplete providers can be combined with search middleware for real-time suggestions:

```python
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.wrappers import MiddlewareSearcher

provider = create_autocomplete("inverted")
provider.add(["python", "java", "javascript"])

# In a search handler
def suggest(query: str, limit: int = 5):
    hits = provider.search(query, limit=limit)
    return {"suggestions": [hit.text for hit in hits]}
```


## DOCUMENT: Configuration Engine

# Configuration Engine

Module: `whoosh_modern.config`
Version: 3.1.0

Whoosh-NG includes a **Configuration Engine** (`ConfigEngine`) that loads,
validates, and merges application configuration from YAML or JSON files. It is
built on Pydantic models and supports hierarchical layering so that environment-
specific overrides can cleanly extend base settings.

## Core concepts

### Pydantic models

All configuration is expressed through typed Pydantic models:

- `WhooshNGConfig` — top-level application config
- `FieldConfig` — per-field indexing options
- `SearchConfig` / `FuzzyConfig` / `RankingConfig` / `AIConfig`
- `DataSourceConfigModel` — data source connection and sync settings
- `StorageConfigModel` — storage backend selection

### Loaders

Two loaders are provided:

- `load_yaml(path)` — parse a YAML file into a `dict`
- `load_json(path)` — parse a JSON file into a `dict`
- `load_config(path)` — auto-detect format from extension and return a validated
  `WhooshNGConfig`

### Hierarchical merging

`ConfigEngine.load(path, priority=...)` and `ConfigEngine.merge(overrides, priority=...)`
stack configuration sources with the following precedence (highest wins):

1. `runtime`
2. `instance`
3. `application`
4. `language`

Invalid ``priority`` values raise ``ValueError`` immediately, so misconfigured
layers cannot silently affect the merge order.

Merging is deep: nested dictionaries are merged recursively. Scalar values and
lists are **replaced entirely** by the override values; lists are NOT appended
or combined. For example, a base config with ``{"plugins": ["a", "b"]}``
overridden by ``{"plugins": ["c"]}`` produces ``{"plugins": ["c"]}``, not
``{"plugins": ["a", "b", "c"]}``. If additive list merging is required, handle
it at the application level before calling :meth:`ConfigEngine.merge`.

> [!WARNING]
> **List replacement behavior**: When merging configurations, lists are
> **completely overwritten** by higher-priority layers. They are not appended,
> concatenated, or deduplicated. This is an intentional design choice that
> ensures explicit control over list contents across layers and avoids
> unpredictable merged states. If you need additive behavior (e.g., extending
> a list of plugins or middlewares), perform the merge logic in your application
> code before passing the final dictionary to :meth:`ConfigEngine.merge`.

## Quick start

```python
from whoosh_modern.config import ConfigEngine

engine = ConfigEngine()
engine.load("whoosh-ng.yml", priority="application")
engine.load("whoosh-ng.local.yml", priority="instance")
engine.merge({"search": {"fuzzy": {"distance": 5}}}, priority="runtime")

config = engine.get_config()
print(config.index)
print(config.fields["title"].stemming)
print(config.search.fuzzy.distance)
```

## YAML example

```yaml
# whoosh-ng.yml
index: products
languages:
  default: fr
fields:
  title:
    type: text
    language: fr
    stemming: true
    stored: true
  price:
    type: numeric
    sortable: true
search:
  fuzzy:
    enabled: true
    distance: 2
storage:
  type: file
  path: ./index
```

## JSON example

```json
{
  "index": "products",
  "languages": {"default": "en"},
  "fields": {
    "title": {"type": "text", "language": "en", "stemming": true},
    "price": {"type": "numeric", "sortable": true}
  },
  "search": {"fuzzy": {"enabled": true, "distance": 2}},
  "storage": {"type": "file", "path": "./index"}
}
```

## Complete YAML examples

### Minimal config

```yaml
# whoosh-ng.yml
index: my_index
fields:
  title:
    type: text
    stored: true
storage:
  type: file
  path: ./index
```

### E-commerce catalog with CSV source

```yaml
# whoosh-ng.yml
index: products
fields:
  sku:
    type: text
    stored: true
    unique: true
  name:
    type: text
    language: fr
    stemming: true
    stored: true
  description:
    type: text
    language: fr
    stemming: true
  price:
    type: numeric
    sortable: true
    faceted: true
  category:
    type: text
    faceted: true
  published_at:
    type: datetime
    faceted: true
search:
  fuzzy:
    enabled: true
    distance: 2
data_source:
  type: csv
  path: Datas/products.csv
  delimiter: ","
  encoding: utf-8
  id_field: sku
storage:
  type: file
  path: ./index
```

### Layered configuration (base + instance + runtime)

```yaml
# whoosh-ng.yml  (application layer)
index: app
fields:
  title:
    type: text
    stemming: true
search:
  fuzzy:
    enabled: true
    distance: 2
storage:
  type: file
  path: ./index
```

```yaml
# whoosh-ng.local.yml  (instance layer)
index: app-staging
storage:
  type: file
  path: ./index-staging
```

```python
# runtime override in code
engine = ConfigEngine()
engine.load("whoosh-ng.yml", priority="application")
engine.load("whoosh-ng.local.yml", priority="instance")
engine.merge({"search": {"fuzzy": {"distance": 3}}}, priority="runtime")
app = engine.build()
```

## Complete JSON examples

### Minimal config

```json
{
  "index": "my_index",
  "fields": {
    "title": {"type": "text", "stored": true}
  },
  "storage": {"type": "file", "path": "./index"}
}
```

### Full-stack config with SQL source and hybrid storage

```json
{
  "index": "customers",
  "fields": {
    "customer_id": {"type": "numeric", "stored": true, "sortable": true},
    "first_name": {"type": "text", "language": "en", "stemming": true, "stored": true},
    "last_name": {"type": "text", "language": "en", "stemming": true, "stored": true},
    "city": {"type": "text", "language": "en", "stemming": true, "stored": true},
    "country": {"type": "text", "stored": true},
    "signup_date": {"type": "datetime", "faceted": true}
  },
  "search": {
    "fuzzy": {"enabled": true, "distance": 2},
    "highlight": {"enabled": true, "fragment_size": 200}
  },
  "data_source": {
    "type": "sql",
    "connection_string": "sqlite:///benchmark_data.db",
    "query": "SELECT * FROM customers",
    "id_field": "customer_id"
  },
  "storage": {
    "type": "hybrid",
    "local_path": "./index-cache",
    "remote": {
      "type": "s3",
      "bucket": "my-bucket",
      "prefix": "whoosh-indexes/"
    }
  }
}
```

## Using ConfigEngine.build() for zero-code setup

```python
from whoosh_modern.config import ConfigEngine

engine = ConfigEngine()
engine.load("whoosh-ng.yml")
app = engine.build()
app.build()

# Add documents through the index writer
writer = app.index.writer()
writer.add_document(title="Premier cours de Python", body="...")
writer.add_document(title="Whoosh-NG avancé", body="...")
writer.commit()

# Or use the source directly for streaming/batch indexing
for doc in app._source.iter_documents():
    with app.index.writer() as writer:
        writer.add_document(**doc)

results = app.search("python")
```

## Module reference

| Module | Purpose |
|---|---|
| `whoosh_modern.config.models` | Pydantic models for validation |
| `whoosh_modern.config.loader` | YAML / JSON file loaders |
| `whoosh_modern.config.engine` | `ConfigEngine` with hierarchical merging |

## See Also

- [Storage Providers](storage-providers.md) — Storage backends configurable via `StorageConfigModel`
- [Data Sources](data-sources.md) — `DataSourceConfigModel` and provider configuration


## DOCUMENT: Linguistics

# Synonyms & Linguistics

Module: `whoosh_modern.linguistics.synonyms`, `whoosh_modern.linguistics.stemmers`
Version: 2.0.0

The linguistics module provides a comprehensive synonym expansion engine and language-specific text analyzers. It integrates with the middleware pipeline to expand queries and documents with synonyms at both index time and query time.

## Module Overview

```text
whoosh_modern.linguistics
    ├── synonyms/
    │   ├── provider.py       # SynonymProvider protocol + StaticSynonymProvider
    │   ├── yaml_provider.py  # YAMLSynonymProvider
    │   ├── json_provider.py  # JSONSynonymProvider
    │   ├── store.py          # SQLiteSynonymStore
    │   ├── compiler.py       # SynonymCompiler
    │   ├── manager.py        # SynonymManager
    │   ├── middleware.py     # SynonymExpansionMiddleware
    │   └── languages.py      # LANG_SYNONYMS (FR/EN/DE/ES/IT)
    └── stemmers/
        └── __init__.py       # Language-specific analyzers (FR/EN/DE/ES/IT)
```

## Synonym Providers

### SynonymProvider (Protocol)

The base protocol that all synonym providers implement:

```python
from whoosh_modern.linguistics.synonyms import SynonymProvider

class MyProvider(SynonymProvider):
    def get_synonyms(self, word: str) -> list[str]:
        """Return synonyms for the given word."""
        ...

    def add_synonym(self, word: str, synonyms: list[str]) -> None:
        """Add synonyms for the given word."""
        ...

    def remove_synonym(self, word: str, synonym: str) -> None:
        """Remove a synonym for the given word."""
        ...
```

### StaticSynonymProvider

In-memory provider backed by a dictionary:

```python
from whoosh_modern.linguistics.synonyms import StaticSynonymProvider

provider = StaticSynonymProvider({
    "car": ["automobile", "vehicle", "auto"],
    "house": ["home", "residence"],
})

print(provider.get_synonyms("car"))  # ['automobile', 'vehicle', 'auto']
```

### YAMLSynonymProvider

Loads synonyms from a YAML file:

```yaml
# synonyms.yaml
car:
  - automobile
  - vehicle
  - auto
house:
  - home
  - residence
```

```python
from whoosh_modern.linguistics.synonyms import YAMLSynonymProvider

# Requires: pip install pyyaml
provider = YAMLSynonymProvider("synonyms.yaml")
print(provider.get_synonyms("car"))  # ['automobile', 'vehicle', 'auto']
```

### JSONSynonymProvider

Loads synonyms from a JSON file:

```json
{
    "car": ["automobile", "vehicle", "auto"],
    "house": ["home", "residence"]
}
```

```python
from whoosh_modern.linguistics.synonyms import JSONSynonymProvider

provider = JSONSynonymProvider("synonyms.json")
print(provider.get_synonyms("car"))
```

### SQLiteSynonymStore

Persistent synonym store backed by SQLite:

```python
from whoosh_modern.linguistics.synonyms import SQLiteSynonymStore

store = SQLiteSynonymStore("synonyms.db")

# CRUD operations
store.add_synonym("car", ["automobile", "vehicle"])
print(store.get_synonyms("car"))  # ['automobile', 'vehicle']
store.remove_synonym("car", "automobile")
print(store.get_synonyms("car"))  # ['vehicle']
store.close()
```

### SynonymCompiler

Precompiles raw synonym data into a fast lookup format:

```python
from whoosh_modern.linguistics.synonyms import SynonymCompiler

compiler = SynonymCompiler({"car": ["automobile", "vehicle"]})
compiler.add("house", ["home", "residence"])
compiler.merge({"book": ["publication", "work"]})

compiled = compiler.compile()
print(compiled)
# {'car': ['automobile', 'vehicle'], 'house': ['home', 'residence'], 'book': ['publication', 'work']}
```

## SynonymManager

The `SynonymManager` is the high-level interface for managing synonyms. It wraps a `StaticSynonymProvider` internally and supports import/export:

```python
from whoosh_modern.linguistics.synonyms import SynonymManager

manager = SynonymManager({"car": ["automobile", "vehicle"]})

# CRUD
manager.add_synonyms("house", ["home", "residence"])
print(manager.get_synonyms("house"))  # ['home', 'residence']
manager.remove_synonym("house", "home")

# Import from external sources
manager.import_yaml("synonyms.yaml")   # Requires PyYAML
manager.import_json("synonyms.json")

# Export
manager.export_json("output.json")
```

### Import/Export Workflow

```python
# Import from YAML
manager = SynonymManager()
manager.import_yaml("my_synonyms.yaml")

# Export to JSON (e.g., for migration or backup)
manager.export_json("backup.json")
```

## Prebuilt Language Synonyms

The `LANG_SYNONYMS` dictionary contains starter synonym mappings for five languages:

```python
from whoosh_modern.linguistics.synonyms import LANG_SYNONYMS

# Available languages: fr, en, de, es, it
french_syns = LANG_SYNONYMS["fr"]
print(french_syns["voiture"])  # ['automobile', 'véhicule']

english_syns = LANG_SYNONYMS["en"]
print(english_syns["car"])  # ['automobile', 'vehicle']

# Bootstrap a SynonymManager with a language
manager = SynonymManager(LANG_SYNONYMS["fr"])
```

| Language | Code | Sample Entry                          |
|----------|------|---------------------------------------|
| French   | `fr` | `"voiture": ["automobile", "véhicule"]` |
| English  | `en` | `"car": ["automobile", "vehicle"]`    |
| German   | `de` | `"auto": ["wagen", "fahrzeug"]`       |
| Spanish  | `es` | `"coche": ["automóvil", "vehículo"]`  |
| Italian  | `it` | `"auto": ["automobile", "veicolo"]`   |

> **Note**: These are minimal starter dictionaries for demonstration. Production deployments should load from curated or domain-specific sources.

## SynonymExpansionMiddleware

Integrates synonym expansion into the middleware pipeline. It expands both search queries and indexed document fields:

```python
from whoosh_modern.linguistics.synonyms import (
    SynonymManager,
    SynonymExpansionMiddleware,
)

# Create a manager with your synonyms
manager = SynonymManager({
    "car": ["automobile", "vehicle"],
    "house": ["home", "residence"],
})

# Create the middleware
middleware = SynonymExpansionMiddleware(manager)

# Register with the PluginManager or MiddlewareChain
from whoosh.plugins.manager import PluginManager
PluginManager._default.register_middleware("synonym", middleware)
```

### How It Works

- **`before_search`**: Expands `context.query` by appending synonyms for each token
- **`before_index`**: Expands string values in `context.document` by appending synonyms

```python
# Before: query = "car"
# After:  query = "car automobile vehicle"

# Before: document = {"title": "house for sale"}
# After:  document = {"title": "house for sale home residence"}
```

## Language-Specific Stemming Analyzers

Located in `whoosh_modern.linguistics.stemmers`, these analyzers combine tokenization, stemming, and stop-word removal:

```python
from whoosh_modern.linguistics.stemmers import (
    EnglishAnalyzer,
    FrenchAnalyzer,
    GermanAnalyzer,
    SpanishAnalyzer,
    ItalianAnalyzer,
)

# Each analyzer is an instance of LanguageAnalyzer and is callable: it returns a list of tokens
analyzer = EnglishAnalyzer
tokens = analyzer("The running cats")
# tokens are stemmed: ["run", "cat"] (stop words removed)

# Backward-compatible "class-style" usage also works: calling the analyzer
# with no arguments returns a fresh analyzer instance, so historical code
# written as EnglishAnalyzer()(text) keeps working unchanged.
tokens = EnglishAnalyzer()("The running cats")
```

### Stemmer Backend Selection

Under the hood, the stemmers use `whoosh_modern.analysis.stemmer_providers`:

```python
from whoosh_modern.analysis.stemmer_providers import (
    get_stemmer,
    register_stemmer,
    list_available_backends,
)

# Auto-detect best available stemmer (PyStemmer preferred)
stemmer = get_stemmer("auto", "english")

# Explicit backend
stemmer = get_stemmer("internal", "english")   # Whoosh's built-in stemmer
stemmer = get_stemmer("pystemmer", "english")   # PyStemmer (faster)

# List available backends
print(list_available_backends())
# {'internal': 'available', 'pystemmer': 'available', ...}

# Register a custom stemmer
@register_stemmer("my_stemmer")
class MyStemmer:
    def stem(self, word: str) -> str:
        return word.lower()
```

| Backend       | Requires                          | Speed   |
|---------------|-----------------------------------|---------|
| `auto`        | None (falls back automatically)  | Fastest available |
| `internal`    | None (built-in Porter stemmer)   | Medium  |
| `pystemmer`   | `pip install whoosh-ng[fast-stemming]` | Fast |

## Integration Example: Full Pipeline

```python
from whoosh_modern.linguistics import (
    EnglishAnalyzer,
    LANG_SYNONYMS,
    SynonymExpansionMiddleware,
    SynonymManager,
)
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.wrappers import MiddlewareWriter, MiddlewareSearcher

# 1. Build synonym manager with English synonyms
syn_manager = SynonymManager(LANG_SYNONYMS["en"])
syn_manager.add_synonyms("search", ["query", "find", "lookup"])

# 2. Create synonym expansion middleware
syn_middleware = SynonymExpansionMiddleware(syn_manager)

# 3. Build middleware chain
chain = MiddlewareChain([syn_middleware])

# 4. Wrap writer and searcher
with MiddlewareWriter(ix.writer(), chain) as writer:
    writer.add_document(title="How to search in Whoosh")

with MiddlewareSearcher(ix.searcher(), chain) as searcher:
    # Query "search" is expanded to "search query find lookup"
    results = searcher.search("search")
```

## Language Registry

The `LanguageRegistry` maps language codes to `LanguageProfile` instances, centralizing analyzer, stemmer, synonym provider, and language detector resolution.

```python
from whoosh_modern.linguistics.registry import (
    LanguageRegistry,
    LanguageProfile,
    StemmerRegistry,
    get_default_registry,
)

# Use the pre-populated default registry (FR/EN/DE/ES/IT)
registry = get_default_registry()

# Resolve a language profile
profile = registry.resolve("fr")
print(profile.language)   # "fr"
print(profile.analyzer)   # FrenchAnalyzer instance

# Register a custom language profile
custom = LanguageProfile(
    language="pt",
    analyzer=...,  # your analyzer
    stemmer=...,   # your stemmer
)
registry.register(custom)

# StemmerRegistry adds stemmer-specific helpers
stem_registry = StemmerRegistry(registry._profiles.values())
stemmer = stem_registry.get_stemmer("fr")
```

## Multi-Language Analyzer

`MultiLanguageAnalyzer` applies multiple language analyzers simultaneously for multilingual indexing.

```python
from whoosh_modern.linguistics.analyzers import MultiLanguageAnalyzer

# Default: FR/EN/DE/ES/IT
analyzer = MultiLanguageAnalyzer()

# Custom language set
analyzer = MultiLanguageAnalyzer(languages=["fr", "en"])

tokens = analyzer("hello bonjour")
# Returns combined tokens from all configured analyzers
```

## Language Auto-Detection

`StopwordDetector` and `LangDetectProvider` enable automatic language detection:

```python
from whoosh_modern.linguistics.detection import StopwordDetector

detector = StopwordDetector(supported_languages=["fr", "en", "de"])
lang = detector.detect("Ceci est un texte en français")
print(lang)  # "fr"
```

Use with `SearchApplication` for automatic language resolution:

```python
from whoosh_modern import SearchApplication
from whoosh_modern.linguistics.detection import StopwordDetector

app = SearchApplication(
    source=my_source,
    language_detector=StopwordDetector(),
)

# FieldConfig supports language="auto"
# The detector resolves the language per document
```

## Explain Analyzer

`ExplainAnalyzer` exposes the tokenization/stemming pipeline for Search Studio:

```python
from whoosh_modern.linguistics.explain import ExplainAnalyzer

explainer = ExplainAnalyzer(EnglishAnalyzer)
result = explainer.explain("The running cats")

print(result.text)       # "The running cats"
print(result.tokens)     # ["run", "cat"]
```

## Debugging Analysis with ExplainAnalyzer

`ExplainAnalyzer` wraps any existing analyzer and returns an
`AnalysisExplanation` describing how a text is transformed. It is useful
for debugging complex analyzer chains, especially when mixing multilingual
analyzers, stopword filters, or dictionary stem overrides.

```python
from whoosh_modern.linguistics.explain import ExplainAnalyzer
from whoosh.analysis import StandardAnalyzer

explainer = ExplainAnalyzer(StandardAnalyzer())
explanation = explainer.explain("A quick brown fox jumps over the lazy dog")

print(f"Original text: {explanation.text}")
print(f"Final tokens : {explanation.tokens}")

print("\nStep-by-step explanations:")
for step in explanation.explanations:
    print(
        f"  - {step.step}: '{step.original}' -> '{step.result}'"
    )
```

Example output:

```text
Original text: A quick brown fox jumps over the lazy dog
Final tokens : ['quick', 'brown', 'fox', 'jumps', 'lazy', 'dog']

Step-by-step explanations:
  - tokenize: 'A' -> 'A'
  - lowercase: 'A' -> 'a'
  - stop: 'a' -> ''
  - tokenize: 'quick' -> 'quick'
  - lowercase: 'quick' -> 'quick'
  ...
```

### Interpreting the output

- `explanation.text` — the original input text.
- `explanation.tokens` — the final token list after all analyzer steps.
- `explanation.explanations` — a chronological list of
  `TokenExplanation` objects showing each transformation step.

Use this when:
- an analyzer pipeline behaves differently than expected,
- you need to verify which stopwords or stemming rules are applied,
- you want to compare behavior across languages with `MultiLanguageAnalyzer`.

## Dictionary Stem Override

Override Snowball stemming with business dictionaries:

```python
from whoosh_modern.linguistics.dictionary_stem_override import DictionaryStemOverride

override = DictionaryStemOverride({
    "voiture": "voitur",
    "maison": "maison",
})

print(override.stem("voiture"))  # "voitur"
print(override.stem("maison"))   # "maison"

# Add rules dynamically
override.add_rule("chien", "chien")
```

Use with `SearchApplication`:

```python
from whoosh_modern import SearchApplication

app = SearchApplication(
    source=my_source,
    dictionary_stem_overrides={"voiture": "voitur"},
)
```

## Cached Stemming Analyzer

`CachedStemmingAnalyzer` wraps language analyzers with LRU caching:

```python
from whoosh_modern.analysis.cached_stemming_analyzer import CachedStemmingAnalyzer
from whoosh_modern.linguistics.stemmers import FrenchAnalyzer

cached = CachedStemmingAnalyzer(FrenchAnalyzer, cache_size=50000)
tokens = cached("les maisons")
```

## Stemmer Profiler

Measure stemming impact on vocabulary and performance:

```python
from whoosh_modern.profiling.stemmer_profiler import StemmerProfiler

profiler = StemmerProfiler(stemmer=my_stemmer)
report = profiler.profile(["document 1", "document 2", ...])

print(report.original_tokens)        # Total tokens before stemming
print(report.stemmed_tokens)         # Unique tokens after stemming
print(report.reduction_ratio)        # Vocabulary reduction ratio
print(report.estimated_size_reduction)  # Estimated index size reduction %
print(report.avg_stem_time_ms)       # Average stemming time per token
```

## Analyzer Presets

Preconfigured analyzers for common search scenarios:

```python
from whoosh_modern.analysis.stemmer_presets import AnalyzerPresets

# Autocomplete
autocomplete_analyzer = AnalyzerPresets.autocomplete()

# Partial match
partial_analyzer = AnalyzerPresets.partial_match()

# Ecommerce
ecommerce_analyzer = AnalyzerPresets.ecommerce()

# Blog
blog_analyzer = AnalyzerPresets.blog()

# Multilingual
multilingual_analyzer = AnalyzerPresets.multilingual()

# Get by name
analyzer = AnalyzerPresets.get("autocomplete")
```

## See Also

- [Synonyms](synonyms.md) — Synonym providers, manager, and Wiktionary dictionaries
- [Stemming Guide](stemming-providers.md) — Stemmer providers and language analyzers
- [Middleware Guide](middleware-pipeline.md) — Middleware pipeline integration
- [Provider Integration Guide](provider-integration.md) — Complete pipeline guide for all providers
- [API: Linguistics](../api/modern.md) — Full API reference


## DOCUMENT: Middleware Pipeline

# Middleware & Plugin Pipeline

Module: `whoosh.middleware`, `whoosh.middleware.chain`, `whoosh.middleware.context`, `whoosh_modern.middleware`
Version: 3.0.0

The middleware pipeline allows you to intercept and modify indexing and search operations. It is the primary extension mechanism for cross-cutting concerns like logging, caching, metrics, query rewriting, and security. Middleware can come from both the core `whoosh.middleware` package and from plugins loaded via the `PluginManager`.

## Architecture Overview

```text
Writer/Searcher  ───►  MiddlewareChain
                           ├── Middleware 1 (before hook)
                           ├── Middleware 2 (before hook)
                           ├── ─── core operation ───
                           ├── Middleware 2 (after hook, reverse)
                           └── Middleware 1 (after hook, reverse)
```

- **Before hooks** execute in registration order
- **After hooks** execute in reverse order (like a stack / onion)
- If a hook raises `StopOperation`, the pipeline aborts gracefully
- If `fail_open=False` (default), exceptions propagate immediately

## Core Middleware Classes

### Middleware (Base Class)

Located in `whoosh.middleware.base`. Subclasses implement lifecycle hooks:

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class MyMiddleware(Middleware):
    def startup(self, context: MiddlewareContext) -> None:
        """Called once when middleware is initialized."""
        pass

    def shutdown(self, context: MiddlewareContext) -> None:
        """Called once when middleware is torn down."""
        pass

    def before_index(self, context: MiddlewareContext) -> MiddlewareContext:
        """Called before a document is indexed. Modify context.document."""
        return context

    def after_index(self, context: MiddlewareContext) -> MiddlewareContext:
        """Called after a document is indexed."""
        return context

    def before_delete(self, context: MiddlewareContext) -> MiddlewareContext:
        """Called before a document is deleted."""
        return context

    def after_delete(self, context: MiddlewareContext) -> MiddlewareContext:
        """Called after a document is deleted."""
        return context

    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        """Called before a search query is executed. Modify context.query."""
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        """Called after results are returned. Access context.results."""
        return context

    def on_error(self, context: MiddlewareContext, exc: Exception) -> None:
        """Called when an exception occurs. Re-raise by default."""
        raise exc

    def on_commit(self, context: MiddlewareContext) -> None:
        """Called after a commit operation."""
        pass
```

### MiddlewareContext

Located in `whoosh.middleware.context`. The context object passed to every hook:

```python
class MiddlewareContext:
    def __init__(self, operation: str) -> None:
        self.operation: str           # e.g., "add_document", "search"
        self.index: Any = None        # The Index instance
        self.backend: Any = None       # The storage backend
        self.writer: Any = None        # The IndexWriter (if applicable)
        self.searcher: Any = None      # The Searcher (if applicable)
        self.document: dict[str, Any] | None  # Document being indexed
        self.query: str = ""           # The search query string
        self.collector: Any = None     # The collector (if applicable)
        self.results: Any = None       # Search results
        self.labels: dict[str, Any] = {}    # Arbitrary labels/key-value pairs
        self.metadata: dict[str, Any] = {} # Per-request metadata
```

Use `context.copy()` to create a shallow copy if you need to preserve state.

### MiddlewareChain

Located in `whoosh.middleware.chain`. Orchestrates middleware execution:

```python
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.context import MiddlewareContext

chain = MiddlewareChain([
    MetricsMiddleware(),
    CacheMiddleware(),
])

# Before hooks (in order)
context = MiddlewareContext("search")
context.query = "hello world"
context = chain.run_before("before_search", context)

# ... core search operation ...

# After hooks (in reverse order)
context = chain.run_after("after_search", context)
print(context.results)
```

**Async support**: Use `async_run_before()`, `async_run_after()`, `async_run_on_error()`, and `run_hook()` for async middleware.

### MiddlewareRegistry

Located in `whoosh.middleware.registry`. A class-level registry for named middleware:

```python
from whoosh.middleware.registry import MiddlewareRegistry

MiddlewareRegistry.register("my_mw", MyMiddleware(), owner="my_plugin")
mw = MiddlewareRegistry.get("my_mw")
MiddlewareRegistry.unregister("my_mw")
print(MiddlewareRegistry.list_all())  # ['my_mw', ...]
```

## Middleware Integration

### Wrappers: MiddlewareWriter & MiddlewareSearcher

Located in `whoosh.middleware.wrappers`. These wrap the core writer/searcher to automatically execute middleware hooks:

```python
from whoosh.middleware.wrappers import MiddlewareWriter, MiddlewareSearcher
from whoosh.middleware.chain import MiddlewareChain

chain = MiddlewareChain([MetricsMiddleware(), CacheMiddleware()])

# Wrap a writer
with MiddlewareWriter(ix.writer(), chain) as writer:
    writer.add_document(title="Hello", content="World")

# Wrap a searcher
with MiddlewareSearcher(ix.searcher(), chain) as searcher:
    results = searcher.search(query)
```

### Integration Helpers

Located in `whoosh.middleware.integration`:

```python
from whoosh.middleware.integration import apply_middleware_to_writer, apply_middleware_to_searcher

# Auto-loads middleware from PluginManager if chain is not provided
writer = apply_middleware_to_writer(ix.writer())
searcher = apply_middleware_to_searcher(ix.searcher())
```

## Built-in Middleware

### Core Middleware (`whoosh.middleware.base`)

| Class                  | Hooks              | Description                              |
|------------------------|--------------------|------------------------------------------|
| `CompressionMiddleware` | `before_index`    | Marks documents with `_compressed = True` |
| `EncryptionMiddleware`  | `before_index`    | Marks documents with `_encrypted = True`  |
| `MetricsMiddleware`     | `after_index`, `after_search` | Tracks indexed docs and search count |
| `CacheMiddleware`       | `before_search`, `after_search` | In-memory result caching |

### Observability (`whoosh.middleware.metrics`)

`PrometheusMiddleware` — exports metrics to Prometheus (requires `prometheus-client`):

```python
from whoosh.middleware.metrics import PrometheusMiddleware

# Requires: pip install whoosh-ng[metrics]
prom = PrometheusMiddleware()
# Exports: whoosh_searches_total, whoosh_documents_indexed_total, whoosh_search_duration_seconds
```

### Modern Middleware (`whoosh_modern.middleware`)

#### Resilience Middleware (core subclasses)

`RetryMiddleware`, `LoggingMiddleware`, and `CacheMiddleware` are now **subclasses of the
core `whoosh.middleware.base.Middleware`** (the same base class re-exported as
`whoosh_modern.middleware.Middleware`). They participate in the standard hook pipeline
(`before_index` / `after_index` / `before_search` / `after_search` / `on_error` /
`on_commit`) and additionally keep a `wrap(operation)` helper so arbitrary callables can
still be decorated.

`MiddlewarePipeline` is a thin wrapper around `whoosh.middleware.chain.MiddlewareChain`
that executes a callable through the chain hooks and returns its result. The previous
`whoosh_modern.middleware.pipeline` module has been removed — import these names directly
from `whoosh_modern.middleware`.

| Class                  | Description                              |
|------------------------|------------------------------------------|
| `RetryMiddleware`      | Retries failed operations with exponential backoff |
| `LoggingMiddleware`    | Logs operation execution time and errors |
| `CacheMiddleware`      | Caches operation results (LRU eviction)  |
| `MiddlewarePipeline`   | Chains multiple middlewares via `MiddlewareChain` |

```python
from whoosh_modern.middleware import MiddlewarePipeline, RetryMiddleware, LoggingMiddleware

pipeline = MiddlewarePipeline(
    LoggingMiddleware(),
    RetryMiddleware(attempts=3, backoff="exponential", jitter=True),
)

result = pipeline.execute(lambda: my_index_operation())
```

#### Storage Middleware (`whoosh_modern.middleware.storage`)

| Class                  | Description                              |
|------------------------|------------------------------------------|
| `StorageMiddleware`    | Routes persistence through pluggable storage providers |
| `FileStorageProvider`  | Local filesystem storage                 |
| `SQLiteStorageProvider`| SQLite-backed blob storage               |
| `S3StorageProvider`    | S3 / S3-compatible cloud storage         |

```python
from whoosh_modern.middleware.storage import StorageMiddleware, FileStorageProvider

storage = StorageMiddleware(FileStorageProvider("/data/index"), name="primary")
```

#### Search Middleware (`whoosh_modern.middleware.search`)

| Class                      | Description                              |
|----------------------------|------------------------------------------|
| `QueryRewriteMiddleware`   | Rewrites `context.query` before search   |
| `RankingMiddleware`        | Re-ranks `context.results` after search  |

```python
from whoosh_modern.middleware.search import QueryRewriteMiddleware

def add_synonyms(query: str) -> str:
    # Expand query with synonyms before execution
    return query + " " + get_synonyms(query)

rewriter = QueryRewriteMiddleware(rewriter=add_synonyms)
```

#### Analyzer Middleware (`whoosh_modern.middleware.analyzer`)

| Class                  | Description                              |
|------------------------|------------------------------------------|
| `StemmingMiddleware`   | Applies a stemmer to document fields and query |
| `SynonymMiddleware`    | Expands text with synonyms (placeholder) |

## Creating Custom Middleware

### Hook-Based Middleware

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class RequestLoggingMiddleware(Middleware):
    """Log all search requests with timing."""

    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        import time
        context.metadata["_start_time"] = time.time()
        logger.info(f"[SEARCH] Query: {context.query}")
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        elapsed = time.time() - context.metadata.get("_start_time", time.time())
        result_count = len(context.results) if context.results is not None else 0
        logger.info(f"[RESULTS] Found {result_count} hits in {elapsed:.3f}s")
        return context
```

### Custom Middleware (Hook-Based)

The `Middleware` base class is `whoosh.middleware.base.Middleware`, re-exported from
`whoosh_modern.middleware`. Subclass it and implement the lifecycle hooks:

```python
from whoosh_modern.middleware import Middleware
from whoosh.middleware.context import MiddlewareContext

class RetryMiddleware(Middleware):
    """Retry failed operations with backoff (illustrative)."""

    def __init__(self, attempts: int = 3) -> None:
        self._attempts = attempts

    def on_error(self, context: MiddlewareContext, exc: Exception) -> None:
        # Built-in resilience middlewares already implement this pattern.
        # Custom logic can record failures in context.metadata here.
        context.metadata.setdefault("retry_errors", 0)
        context.metadata["retry_errors"] += 1
        raise exc
```

> Note: the built-in `RetryMiddleware`, `LoggingMiddleware`, and `CacheMiddleware` also
> expose a `wrap(operation)` helper (preserved for backwards compatibility) so they can
> decorate plain callables, but their primary mechanism is the hook pipeline above.

### Middleware with Plugin Integration

Register middleware via a plugin so it's automatically discovered:

```python
from whoosh.plugins.manager import Plugin

class LoggingPlugin(Plugin):
    name = "logging"
    version = "1.0.0"
    middleware = ["whoosh_modern.middleware.LoggingMiddleware"]

    def register(self, manager):
        manager.register_middleware(
            "logging",
            LoggingMiddleware(),
        )
```

## Error Handling

### StopOperation

Abort a pipeline operation gracefully:

```python
from whoosh.middleware.exceptions import StopOperation

class RateLimitMiddleware(Middleware):
    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if not rate_limiter.allow(context):
            raise StopOperation("Rate limit exceeded")
        return context
```

### fail_open Behavior

```python
class ResilientMiddleware(Middleware):
    def on_error(self, context: MiddlewareContext, exc: Exception) -> None:
        try:
            send_to_analytics(context.results)
        except Exception:
            # Log but don't fail the search
            logger.warning("Analytics failed", exc_info=True)
        # Middleware chain continues
```

## Middleware Discovery from Plugins

When `PluginManager.load_plugins()` is called, all plugins that declare a `middleware` list will have those middleware classes imported and instantiated. The `get_middleware_chain()` method builds a `MiddlewareChain` from all registered middleware:

```python
from whoosh.plugins.manager import PluginManager

PluginManager.load_plugins()  # Discovers plugins and their middleware

manager = PluginManager._default
chain = manager.get_middleware_chain()
# chain is a MiddlewareChain ready for use
```

## Best Practices

1. **Statelessness**: Use `context.metadata` for per-request data, not instance attributes
2. **Lightweight hooks**: Keep `before_*` and `after_*` hooks fast; use async for I/O
3. **Order matters**: Place caching before metrics, authentication before routing
4. **Fail fast**: Only use `fail_open=True` for non-critical middleware
5. **Test isolation**: Mock the `MiddlewareContext` to test middleware independently
6. **Clean up**: Implement `shutdown()` for resources like connections and timers

## See Also

- [Plugin System Guide](plugins-advanced.md) — Plugin registration and entry points
- [Provider Integration Guide](provider-integration.md) — Complete pipeline guide for all providers
- [Middleware Examples](../examples/middleware.md) — Practical middleware patterns
- [API: Middleware](../api/middleware.md) — Full API reference
- [API: Middleware Pipeline (modern)](../api/modern.md) — Modern middleware extensions


## DOCUMENT: Middleware

# Middleware

The middleware pipeline allows you to intercept and modify indexing and search operations. It is the primary extension mechanism for cross-cutting concerns like logging, caching, metrics, and security.

## Core Concepts

A middleware is a class that implements hooks into the indexing and search lifecycle:

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class MyMiddleware(Middleware):
    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        # Modify context.query or context.metadata
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        # Access context.results
        return context
```

## Available Hooks

| Hook | When | Common Uses |
|------|------|-------------|
| `startup(context)` | Middleware initialized | Open connections, warm caches |
| `shutdown(context)` | Middleware torn down | Close connections, flush buffers |
| `before_index(context)` | Before document added | Validation, enrichment, compression flags |
| `after_index(context)` | After document added | Metrics, events, cache invalidation |
| `before_delete(context)` | Before document deleted | Audit logging, access control |
| `after_delete(context)` | After document deleted | Metrics, cache invalidation |
| `before_search(context)` | Before query executes | Query rewriting, caching, auth |
| `after_search(context)` | After results returned | Logging, metrics, result modification |
| `on_error(context, exc)` | On exception | Error handling, fallbacks |
| `on_commit(context)` | After commit | Metrics, notifications |

## Built-in Middlewares

### MetricsMiddleware

Tracks basic statistics:

```python
from whoosh.middleware import MetricsMiddleware

metrics = MetricsMiddleware()
# After operations:
stats = metrics.get_metrics()
# Returns: {"documents_indexed": N, "searches_executed": N}
```

### CacheMiddleware

Caches search results in memory:

```python
from whoosh.middleware import CacheMiddleware

cache = CacheMiddleware()

# Check cache
cached = cache.get_cached("user query string")

# Store manually
cache.set_cached("user query string", results)
```

### CompressionMiddleware

Marks documents for compression at the backend level:

```python
from whoosh.middleware import CompressionMiddleware

compression = CompressionMiddleware()
# Sets document["_compressed"] = True
```

### EncryptionMiddleware

Marks documents for encryption at the backend level:

```python
from whoosh.middleware import EncryptionMiddleware

encryption = EncryptionMiddleware()
# Sets document["_encrypted"] = True
```

## MiddlewareChain

Orchestrates middleware execution:

```python
from whoosh.middleware import MiddlewareChain

chain = MiddlewareChain([
    MetricsMiddleware(),
    CacheMiddleware()
])

# Execute before hook
context = MiddlewareContext("search")
context.query = "test"
context = chain.run_before("before_search", context)

# ... core operation ...

# Execute after hook
context = chain.run_after("after_search", context)
```

### Execution Order

- `before_*` hooks run in registration order
- `after_*` hooks run in reverse order
- If a hook raises `StopOperation`, the pipeline aborts
- If `fail_open=False`, exceptions propagate immediately

## Integration

### With Writer

```python
from whoosh.middleware.integration import apply_middleware_to_writer

writer = apply_middleware_to_writer(ix.writer(), chain.middlewares)

with writer:
    writer.add_document(title="Hello", content="World")
```

### With Searcher

```python
from whoosh.middleware.integration import apply_middleware_to_searcher

searcher = apply_middleware_to_searcher(ix.searcher(), chain.middlewares)
results = searcher.search("query")
```

### With PluginManager

```python
from whoosh.plugins.manager import PluginManager

# Plugins can provide middleware
PluginManager.load_plugins()
chain = PluginManager.get_middleware_chain()
```

## Custom Middleware Example

```python
class RequestLoggingMiddleware(Middleware):
    """Log all search requests."""

    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        context.metadata["request_id"] = generate_request_id()
        logger.info(f"Search: {context.query}")
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        logger.info(f"Found {len(context.results)} results")
        return context

class RateLimitMiddleware(Middleware):
    """Abort searches exceeding rate limit."""

    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if not rate_limiter.allow(context):
            raise StopOperation("Rate limit exceeded")
        return context

class QueryEnrichmentMiddleware(Middleware):
    """Add synonyms to the query."""

    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if context.query:
            context.query += " " + get_synonyms(context.query)
        return context
```

## Error Handling

```python
class ResilientMiddleware(Middleware):
    """Continue on non-critical errors."""

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        try:
            send_to_analytics(context.results)
        except Exception:
            # Log but don't fail the search
            logger.warning("Analytics failed", exc_info=True)
        return context
```

## Best Practices

1. **Stateless**: Use `context.metadata` for per-request data
2. **Fail fast**: Only use `fail_open=True` for non-critical middleware
3. **Order matters**: Place caching before metrics, auth before routing
4. **Performance**: Keep hooks lightweight; use async for I/O
5. **Testing**: Mock the context object to test middleware in isolation

## Provider Integration via Middleware

Many Whoosh-NG providers integrate into the indexing and search pipeline through
middleware hooks. This is the standard pattern for cross-cutting concerns that
need to transform documents, queries, or results.

### Provider-to-Middleware Mapping

| Provider | Middleware | Hooks Used | Purpose |
|----------|-----------|------------|---------|
| `StorageProvider` | `StorageMiddleware` | `before_index`, `on_commit` | Tags context with storage backend; writes commit checkpoints |
| `StemmerProvider` | `StemmingMiddleware` | `before_index`, `before_search` | Stems document fields and query text |
| `SynonymProvider` | `SynonymExpansionMiddleware` | `before_index`, `before_search` | Expands documents and queries with synonyms |
| `VectorProvider` | (built into Whoosh core) | N/A (segment format) | Registered in `VectorRegistry`; resolved at search time from segment metadata |
| `AutocompleteProvider` | (standalone or registry) | N/A | Used directly via `.search()` or via `AutocompleteRegistry` |

### How providers flow through the pipeline

```text
                    ┌──────────────────┐
                    │  PluginManager   │
                    │  .load_plugins() │
                    └────────┬─────────┘
                             │
              ┌──────────────┼──────────────┐
              │              │              │
     ┌────────▼──────┐ ┌────▼─────┐ ┌──────▼────────┐
     │ VectorPlugin  │ │AutoPlugin│ │ Other Plugins │
     │               │ │         │ │               │
     │ VectorRegistry│ │AutoReg. │ │ ProviderReg.  │
     │ .register()   │ │.register│ │ .register()   │
     └───────────────┘ └─────────┘ └───────────────┘

                    ┌──────────────────┐
                    │ MiddlewareChain  │
                    │                  │
                    │ ┌──────────────┐ │
                    │ │ before_index │ │
                    │ │  hooks       │ │
                    │ │  (ordered)   │ │
                    │ └──────┬───────┘ │
                    │        │         │
                    │ ┌──────▼───────┐ │
                    │ │   Writer     │ │
                    │ │  .add_doc()  │ │
                    │ └──────┬───────┘ │
                    │        │         │
                    │ ┌──────▼───────┐ │
                    │ │  on_commit   │ │
                    │ │  hooks       │ │
                    │ └──────────────┘ │
                    └──────────────────┘

                    ┌──────────────────┐
                    │  Searcher         │
                    │                  │
                    │ ┌──────────────┐ │
                    │ │before_search │ │
                    │ │  hooks       │ │
                    │ └──────┬───────┘ │
                    │        │         │
                    │ ┌──────▼───────┐ │
                    │ │   Query      │ │
                    │ │  execution   │ │
                    │ └──────┬───────┘ │
                    │        │         │
                    │ ┌──────▼───────┐ │
                    │ │ after_search │ │
                    │ │  hooks       │ │
                    │ └──────────────┘ │
                    └──────────────────┘
```

### Example: Full provider pipeline

```python
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.wrappers import MiddlewareWriter, MiddlewareSearcher
from whoosh_modern.middleware import (
    StorageMiddleware,
    StemmingMiddleware,
    FileStorageProvider,
)
from whoosh_modern.linguistics.synonyms import (
    SynonymExpansionMiddleware,
    SynonymManager,
)
from whoosh_modern.analysis import get_stemmer

# 1. Build providers
storage = FileStorageProvider("/data/index")
stemmer = get_stemmer("auto", "english")
syn_manager = SynonymManager({"car": ["automobile", "vehicle"]})

# 2. Build middleware chain
chain = MiddlewareChain([
    StorageMiddleware(storage, name="primary"),
    StemmingMiddleware(stemmer=stemmer.stem, fields=["title", "content"]),
    SynonymExpansionMiddleware(syn_manager),
])

# 3. Index with middleware
with MiddlewareWriter(ix.writer(), chain) as writer:
    writer.add_document(title="Car for sale", content="House near beach")
    # StorageMiddleware.before_index() tags context
    # StemmingMiddleware stems "Car" → "car"
    # SynonymExpansionMiddleware expands "Car" → "Car automobile vehicle"
    writer.commit()
    # StorageMiddleware.on_commit() writes checkpoint

# 4. Search with middleware
with MiddlewareSearcher(ix.searcher(), chain) as searcher:
    # StemmingMiddleware stems query "cars" → "car"
    # SynonymExpansionMiddleware expands "car" → "car automobile vehicle"
    results = searcher.search("cars")
```

### Key insight: middleware as provider adapter

Middleware acts as the **adapter** between providers and Whoosh's core pipeline:

```text
Provider (domain logic)
    │
    ▼
Middleware (pipeline integration)
    │
    ▼
Whoosh core (generic engine)
```

- **StorageProvider** → **StorageMiddleware** → Whoosh writer/searcher
- **StemmerProvider** → **StemmingMiddleware** → Whoosh document/query
- **SynonymProvider** → **SynonymExpansionMiddleware** → Whoosh text fields

This separation keeps providers simple (single responsibility) while middleware
handles the lifecycle integration (when and how to apply the provider).

## Modern Middleware (Whoosh-NG 2.0)

Whoosh-NG 2.0 adds a modern middleware package (`whoosh_modern.middleware`) with wrap-style resilience middleware (retry, caching, logging) and hook-based middleware for storage, search, and analysis. For full details on the modern middleware architecture, plugin integration, and deployment, see the [Middleware & Plugin Pipeline Guide](middleware-pipeline.md).


## DOCUMENT: Modern Indexing

# Modern Indexing API

Whoosh-NG provides an optimized indexing layer in `whoosh_modern.indexing` for high-throughput document ingestion. These utilities wrap the core Whoosh writer without modifying the library internals.

## BatchIndexWriter

`BatchIndexWriter` wraps a core Whoosh writer with optimizations for batch processing of large datasets.

### Key Optimizations

- Pre-computes schema field names for fast filtering (O(1) per-field validation)
- Skips fields not in the schema (avoiding per-document overhead)
- Uses `multisegment=True` to defer merging during indexing
- Supports configurable batch commits to reduce I/O pressure
- Accepts a callback for post-commit hooks

### Basic Usage

```python
from whoosh_modern.indexing import BatchIndexWriter
from whoosh import index

ix = index.open_dir("indexdir")

writer = BatchIndexWriter(ix, batch_size=5000, commit_every=10)

for batch in source.stream_batches(batch_size=5000):
    writer.add_batch(batch)

writer.close()
```

### Context Manager

```python
with BatchIndexWriter(ix, batch_size=10000) as writer:
    for doc in documents:
        writer.add_document(doc)
```

### Parameters

| Parameter | Default | Description |
|-----------|---------|-------------|
| `batch_size` | 5000 | Number of documents per batch |
| `limitmb` | 512 | Memory limit for the writer (MB) |
| `commit_every` | None | Commit after N batches (None = no auto-commit during indexing) |
| `multisegment` | True | Use multisegment mode to defer merging |
| `callback` | None | Callback invoked after each commit |
| `**writer_kwargs` | None | Additional keyword args passed to `index.writer()` |

### With Commit Profiler

```python
from whoosh_modern.indexing import BatchIndexWriter
from whoosh_modern.profiling import CommitProfilerV2

profiler = CommitProfilerV2()
with BatchIndexWriter(ix, batch_size=5000, commit_every=5, commit_profiler=profiler) as writer:
    for batch in source.stream_batches(batch_size=5000):
        writer.add_batch(batch)

print(profiler.report())
```

---

## AnalyzerCache

`AnalyzerCache` provides an LRU cache for analyzer results, avoiding redundant analysis work on repeated field values.

### Basic Usage

```python
from whoosh_modern.indexing import BatchIndexWriter
from whoosh_modern.profiling import AnalyzerCache

cache = AnalyzerCache(maxsize=50000)
analyzer = StandardAnalyzer()

for doc in docs:
    cache_key = f"title:{doc['title']}"
    tokens = cache.get(cache_key)
    if tokens is None:
        tokens = list(analyzer(doc['title']))
        cache.put(cache_key, tokens)
```

### With get_or_compute

```python
from whoosh_modern.profiling import AnalyzerCache

cache = AnalyzerCache(maxsize=50000)
analyzer = StandardAnalyzer()

for doc in docs:
    tokens = cache.get_or_compute(
        f"title:{doc['title']}",
        lambda: list(analyzer(doc['title']))
    )
```

### Cache Statistics

```python
cache = AnalyzerCache(maxsize=50000)
# ... use cache ...

print(f"Hit rate: {cache.hit_rate:.1%}")
print(f"Size: {cache.size}/{cache.maxsize}")
print(cache.report())
# Analyzer Cache Report
# ==================================================
#   Size: 4823/50000
#   Hits: 12543
#   Misses: 3421
#   Hit rate: 78.6%
```

### Sizing from Profiling Data

```python
from whoosh_modern.profiling import AnalyzerCache, CacheAnalyzer

analyzer = CacheAnalyzer()
analysis = analyzer.analyze(source.iter_documents())

cache = AnalyzerCache.from_profiling(analysis.to_dict())
# Creates an optimally sized cache based on field repetition ratios
```

---

## FieldAnalyzerCache

`FieldAnalyzerCache` wraps an analyzer and caches results per field, automatically generating cache keys from field name and value.

### Basic Usage

```python
from whoosh_modern.profiling import FieldAnalyzerCache

field_cache = FieldAnalyzerCache(
    analyzer=StandardAnalyzer(),
    fields=["Country", "City"],
    cache_size=50000,
)

for doc in docs:
    for field in ["Country", "City"]:
        tokens = field_cache.analyze(field, doc[field])
```

### Invalidating Cache Entries

```python
# Invalidate a specific entry
field_cache.invalidate("Country", "USA")

# Clear entire cache
field_cache.clear()
```

### Cache Statistics

```python
print(f"Hit rate: {field_cache.hit_rate:.1%}")
print(field_cache.report())
# Field Analyzer Cache Report
# ==================================================
#   Fields: ['City', 'Country']
#   Cache size: 4823/50000
#   Hit rate: 96.5%
#   Hits: 12543
#   Misses: 3421
```

---

## Available Data Sources

| Class | Type | Dependencies |
|-------|------|-------------|
| `SQLSource` | SQLite, PostgreSQL, MySQL | `sqlite3` (stdlib) |
| `SQLAlchemySource` | Any SQLAlchemy-supported DB | `sqlalchemy` |
| `RESTSource` | REST APIs | none (stdlib `urllib`) |
| `GraphQLSource` | GraphQL APIs | none (stdlib `urllib`) |
| `FastCSVSource` | CSV files | none |
| `JSONSource` | JSON/JSONL files | none |
| `ParquetSource` | Parquet files | `pyarrow` or `pandas` |
| `PandasSource` | pandas DataFrames | `pandas` |
| `PolarsSource` | Polars DataFrames | `polars` |
| `PeeweeSource` | Peewee ORM | `peewee` |
| `TortoiseSource` | Tortoise ORM | `tortoise-orm` |
| `PydanticSource` | Pydantic models | `pydantic` |


## DOCUMENT: Monitoring

# Monitoring

Whoosh-NG ships with hooks for observability and a Prometheus plugin for production use.

## Built-in Metrics

```python
from whoosh.middleware import MetricsMiddleware, MiddlewareChain
from whoosh.middleware.integration import apply_middleware_to_writer, apply_middleware_to_searcher

chain = MiddlewareChain([MetricsMiddleware()])
writer = apply_middleware_to_writer(ix.writer(), chain.middlewares)
searcher = apply_middleware_to_searcher(ix.searcher(), chain.middlewares)

metrics = chain.get_metrics()
print(metrics)
```

## Prometheus

```bash
pip install whoosh-ng[metrics]
```

| Metric | Type | Description |
|--------|------|-------------|
| `whoosh_documents_indexed_total` | Counter | Total documents indexed |
| `whoosh_searches_executed_total` | Counter | Total searches executed |
| `whoosh_indexing_duration_seconds` | Histogram | Indexing latency |
| `whoosh_search_duration_seconds` | Histogram | Search latency |
| `whoosh_index_size_bytes` | Gauge | Current index size |

## Best practices

1. Add `MetricsMiddleware` early in your base chain.
2. Expose `/metrics` in production.
3. Use `/health` for load balancer health checks.
4. Emit `DocumentIndexed` and `SearchExecuted` events.


## DOCUMENT: Performance

# Performance Benchmarking

Whoosh-NG includes a comprehensive benchmarking toolkit in `whoosh_modern.profiling` for measuring and comparing analyzer performance. This guide explains how to use these tools and documents the optimizations shipped in 2.0.0.

## Quick Start

```python
from whoosh_modern.profiling.benchmarks.regex_tokenizer import run_p5_1
from whoosh_modern.profiling.benchmarks.token_optimization import run_p5_2
from whoosh_modern.profiling.synthetic_datasets import SyntheticDatasetGenerator
from whoosh_modern.profiling.stemmer_benchmark import StemmerBenchmark

# Generate synthetic datasets for consistent benchmarks
gen = SyntheticDatasetGenerator(seed=42)
datasets = gen.generate_all(count=5000)

# Run tokenizer benchmark (P5.1)
run_p5_1(datasets)

# Run token creation benchmark (P5.2)
run_p5_2(token_count=100_000)

# Run stemmer benchmark
bench = StemmerBenchmark()
bench.run(gen.generate_dataset("A", 5000))
print(bench.report())
```

## Benchmarking Tools

### SyntheticDatasetGenerator

Generates deterministic text datasets of varying complexity:

```python
from whoosh_modern.profiling.synthetic_datasets import SyntheticDatasetGenerator

gen = SyntheticDatasetGenerator(seed=42)
datasets = gen.generate_all(count=5000)

# Dataset A: 2 tokens/doc (short)
# Dataset B: 50 tokens/doc (medium)
# Dataset C: 500 tokens/doc (large)
# Dataset D: 1200 tokens/doc (very large)
for name, texts in datasets.items():
    print(f"{name}: {len(texts)} documents")
```

### P5.1: RegexTokenizer Benchmark

Compares different tokenizer implementations:

```python
from whoosh_modern.profiling.benchmarks.regex_tokenizer import run_p5_1

results = run_p5_1(datasets)

# Compare:
# - Current Regex (whoosh default)
# - Compiled Global regex
# - Manual Python tokenizer
# - C extension (re2, if available)
```

### P5.2: Token Optimization Benchmark

Compares Token object implementations:

```python
from whoosh_modern.profiling.benchmarks.token_optimization import run_p5_2

# Compare:
# - Current Token (dict-based)
# - __slots__ optimization
# - namedtuple
# - dataclass(slots=True)
results = run_p5_2(token_count=100_000)
```

### StemmerBenchmark

Compares stemmer backends:

```python
from whoosh_modern.profiling.stemmer_benchmark import StemmerBenchmark

bench = StemmerBenchmark()
bench.run(texts, warmup=True)
print(bench.report())
# Output:
# Stemmer Benchmark
# ==================================================
# Stemmer         Tokens/s        Time (s)    Tokens
# ------------------------------------------------------
# StemFilter      1,004,172       0.1503      150,983
# PyStemmer       2,100,000+      0.0719+     150,983
```

## Performance Optimizations

### 2.0.0 Performance Summary

| Optimization | Component | Measurable Gain |
|---|---|---|
| `__slots__` on Token | `whoosh.analysis.acore` | +35% token creation |
| Global compiled regex | `RegexTokenizer` | +50% regex throughput |
| Compact postings (1-posting) | `W3TermInfo` / `W3PostingsWriter` | +35% commit speed |
| Compact postings (2-8 postings) | `W3TermInfo` / `W3PostingsWriter` | +35% commit speed |
| Field cache in add_postings | `whoosh.codec.base` | -93% write_block calls |
| Varint position encoding | `whoosh.formats` | reduced per-term overhead |
| Stemmer cache tuning | `whoosh.analysis.morph` | 96.5% hit rate, 4.12x on repetitive fields |
| Analyzer cache | `whoosh_modern.profiling.analyzer_cache` | 4.12x on high-repetition fields |
| Batch writer optimization | `whoosh_modern.indexing.batch_writer` | optimized filtered batches |
| Stopword setdefault optimization | `whoosh.formats` | reduced dict overhead |

### Benchmark Results: 20k Documents (`customers_csv`)

```
before:
  commit total      : 18.653s
  analyzing         : 8.641s  (51.5%)
  committing        : 10.012s (27.1%)
  write_postings    : 6.5s
  write_block calls : ~72612

after:
  commit total      : 6.806s   (-63.5%)
  analyzing         : ~2.7s    (-68%)
  committing        : 6.806s   (-32%)
  write_postings    : 6.5s -> reduced allocation
  write_block calls : 7565     (-93%)
  throughput        : 1275 docs/s
```

### Benchmark Results: Stemmer Backends (1.5M tokens)

| Stemmer | Throughput | Relative |
|---|---|---|
| StemFilter (internal) | 1,004,172 tokens/s | 1.0x |
| PyStemmer | ~2,100,000 tokens/s | ~2.1x |

### Benchmark Results: Regex Tokenizer

| Tokenizer | Throughput | Relative |
|---|---|---|
| Current regex | ~1,000,000 tokens/s | 1.0x |
| Compiled global | ~2,300,000 tokens/s | 2.3x |

### Benchmark Results: Token Object

| Implementation | Tokens/sec | Relative |
|---|---|---|
| Current (dict) | 1,000,000 | 1.0x |
| `__slots__` | ~1,350,000 | 1.35x |

## Profiling Tools

### IndexingPipelineProfiler

Profiles the complete indexing pipeline:

```python
from whoosh_modern.profiling.indexing_pipeline_profiler import IndexingPipelineProfiler

profiler = IndexingPipelineProfiler()
for doc in documents:
    profiler.before_tokenize(doc, analyzer)
    analyzer(doc)
    profiler.after_tokenize()

report = profiler.report()
print(report)
```

### CommitProfiler

Profiles commit performance including field writing and posting flushing:

```python
from whoosh_modern.profiling.commit_profiler_v2 import CommitProfiler

profiler = CommitProfiler()
# ... index documents ...
ix.commit()

report = profiler.report()
# Shows: analyze, convert_fields, write_postings, flush, commit breakdown
```

### FieldIndexProfiler

Profiles field conversion costs:

```python
from whoosh_modern.profiling.field_index_profiler import FieldIndexProfiler

profiler = FieldIndexProfiler()
# ... index documents ...
report = profiler.report()
# Identifies expensive field types and conversion bottlenecks
```

### IndexQualityAnalyzer

Analyzes index quality metrics including singleton terms:

```python
from whoosh_modern.profiling.index_quality_analyzer import IndexQualityAnalyzer

analyzer = IndexQualityAnalyzer(index_reader)
report = analyzer.analyze()
print(f"Singleton terms: {report['singleton_terms']}/{report['total_terms']} ({report['singleton_percent']}%)")
```

## Stemmer Provider System

Whoosh-NG provides a pluggable stemmer provider system:

```python
from whoosh_modern.analysis import get_stemmer, StemmingAnalyzer, list_available_backends

# Check available backends
print(list_available_backends())
# {'internal': 'available', 'pystemmer': 'not installed'}

# Use auto-detection (default)
analyzer = StemmingAnalyzer(stemmer="auto")

# Explicit internal stemmer
analyzer = StemmingAnalyzer(stemmer="internal")

# PyStemmer (requires: pip install whoosh-ng[fast-stemming])
analyzer = StemmingAnalyzer(stemmer="pystemmer")

# Custom stemmer provider
from whoosh_modern.analysis.stemmer_providers import StemmerProvider

class MyStemmer(StemmerProvider):
    def __init__(self, language="english"):
        self._lang = language

    def stem(self, word):
        return word.lower()

    @property
    def name(self):
        return "my_stemmer"

    @property
    def language(self):
        return self._lang

analyzer = StemmingAnalyzer(stemmer=MyStemmer())
```

## Performance Recommendations

1. **Use `StemmingAnalyzer`** from `whoosh_modern.analysis` for automatic PyStemmer selection
2. **Enable stemmer cache** for repetitive content (`cachesize=50000` by default)
3. **Minimize TEXT fields** — use KEYWORD or ID for low-cardinality fields
4. **Avoid stored positions/chars** unless highlighting requires them
5. **Use batch indexing** with larger segments for better throughput
6. **Monitor singleton terms** — reduce rare terms via stopword lists

## Running the Full Benchmark Suite

```bash
cd whoosh-ng

# Run all P5 benchmarks
uv run python -m pytest tests/test_regex_tokenizer_unicode.py tests/test_token_slots.py tests/test_stemmer_compatibility.py -v

# Run full test suite
uv run python -m pytest -q
```


## DOCUMENT: Plugins Advanced

# Plugin System

Module: `whoosh.plugins.manager`
Version: 2.0.0

Whoosh-NG's plugin architecture enables external packages to extend the core indexing, search, and analysis pipeline. Plugins are discovered via Python [entry points](https://docs.python.org/3/library/importlib.metadata.html#entry-points) declared in `pyproject.toml` and managed by the `PluginManager`.

## Architecture Overview

```text
PluginManager (singleton)
    ├── load_plugins(group)          # Auto-discover from entry points
    ├── register(plugin)             # Manual registration
    ├── enable(name) / disable(name) # Toggle lifecycle
    ├── get(name) / list_plugins()   # Inspection
    ├── get_middleware_chain()       # Build MiddlewareChain from plugin middleware
    ├── register_datasource()        # Register a datasource provider
    ├── register_vector_provider()   # Register a vector provider
    ├── register_middleware()        # Register a middleware instance
    ├── register_embedding()         # Register an embedding provider
    ├── register_analyzer()          # Register a named analyzer
    └── register_query_rewriter()    # Register a query rewriter
```

## Plugin Base Classes

### Plugin (ABC)

The root plugin class. Subclasses set class-level attributes and implement `register()`.

```python
from whoosh.plugins.manager import Plugin, PluginMetadata

class MyPlugin(Plugin):
    name = "my_plugin"
    version = "1.0.0"

    def register(self, manager: PluginManager) -> None:
        """Called when the plugin is loaded; register providers here."""
        manager.register_middleware("my_module.MyMiddleware", MyMiddleware())

    def register_hooks(self) -> None:
        """Register event hooks (optional)."""
        from whoosh.hooks import hookimpl, register_hook

        @hookimpl
        def on_search(request, response):
            pass
        register_hook("on_search", hookimpl(on_search))
```

### AnalyzerPlugin

For plugins that provide custom tokenizers/analyzers:

```python
from whoosh.plugins.manager import AnalyzerPlugin

class MyAnalyzerPlugin(AnalyzerPlugin):
    name = "my_analyzer"

    def register(self, manager):
        manager.register_analyzer("my_analyzer", MyTokenizer())
```

### QueryRewritePlugin

For plugins that transform queries before execution:

```python
from whoosh.plugins.manager import QueryRewritePlugin

class SynonymRewriterPlugin(QueryRewritePlugin):
    name = "synonym_rewriter"

    def rewrite(self, query, searcher):
        # Return modified query
        return query
```

## PluginMetadata

A dataclass describing plugin metadata:

| Field         | Type              | Description                            |
|---------------|-------------------|----------------------------------------|
| `name`        | `str`             | Unique plugin name                     |
| `version`     | `str`             | SemVer version string                  |
| `depends_on`  | `list[str]`       | Names of required plugins              |
| `priority`    | `int`             | Load ordering priority (higher = later)|
| `middleware`  | `list[str]`       | Dotted paths to middleware classes     |

## Entry Point Groups

The `PluginManager` discovers plugins from these standard entry-point groups:

| Group                | Purpose                              |
|----------------------|--------------------------------------|
| `whoosh.plugins`     | General plugins                      |
| `whoosh.datasources` | Data source providers                |
| `whoosh.vector.providers` | Vector similarity providers     |
| `whoosh.middlewares` | Middleware classes                    |
| `whoosh.embeddings`  | Embedding model providers            |
| `whoosh.language`    | Language-specific analyzers          |
| `whoosh.apps`        | App factories (FastAPI, admin, etc.) |

## Creating and Deploying a Plugin

### Step 1: Define the Plugin Class

```python
# my_plugin/plugin.py
from whoosh.plugins.manager import Plugin
from whoosh.registry import VectorRegistry

class MyVectorPlugin(Plugin):
    name = "my_vector"
    version = "1.0.0"
    depends_on = []
    conflicts_with = []
    priority = 0
    middleware = []

    def register(self, manager):
        """Register a vector provider with the VectorRegistry."""
        provider = MyCustomVectorProvider()
        VectorRegistry.register("my_vector", provider, owner=self.name)

    def register_hooks(self):
        """Register optional hooks (e.g., on_search, on_index)."""
        pass
```

### Step 2: Declare the Entry Point

In your `pyproject.toml`:

```toml
[project]
name = "whoosh-ng-my-vector"
version = "1.0.0"
dependencies = ["whoosh-ng>=2.0"]

[project.entry-points."whoosh_ng.plugins"]
my_vector = "my_plugin.plugin:MyVectorPlugin"
```

### Step 3: Install and Verify

```bash
pip install -e .
```

```python
# Verify the plugin is registered
from whoosh.plugins.manager import PluginManager

PluginManager.load_plugins()  # Auto-discovers all entry points

manager = PluginManager._default
print(manager.list_plugins())
# ['whoosh_autocomplete', 'whoosh_vector', ..., 'my_vector']

# Check the registry
from whoosh.registry import VectorRegistry
print(VectorRegistry.list_keys())
# ['my_vector', 'numpy']
```

## Manual Registration (No Entry Point)

For testing or programmatic use:

```python
from whoosh.plugins.manager import PluginManager

manager = PluginManager()
manager.register(MyVectorPlugin())
manager.enable("my_vector")
```

## Plugin Lifecycle

```
1. Entry point discovered  ───►  2. register() called  ───►  3. register_hooks()
   │                               │                            │
   └── load_plugins(group)          └── register provider/     └── register_hook()
                                      middleware/analyzer
```

### Enabling / Disabling

```python
from whoosh.plugins.manager import PluginManager

manager = PluginManager._default

manager.enable("my_vector")    # Activate a plugin
manager.disable("my_vector")   # Deactivate a plugin
print(manager.list_enabled())  # Only enabled plugins
```

### Version Validation

```python
# Check if a plugin meets a minimum version
ok = manager.validate_version("my_vector", "1.0.0")
print(ok)  # True if plugin version >= 1.0.0
```

### Conflict Detection

```python
# Check if two plugins conflict
if manager.detect_conflicts("plugin_a", "plugin_b"):
    print("These plugins cannot be loaded together")
```

## Plugin Manager API Reference

### `PluginManager.load_plugins(group=None)`

Load all plugins from entry-point groups. If `group` is `None`, loads from all standard groups (`STANDARD_GROUPS`).

### `PluginManager.register(plugin)`

Register a plugin instance. Calls `plugin.register(self)` and `plugin.register_hooks()`. Supports async `register()` via `asyncio`.

### `PluginManager.get_middleware_chain()`

Builds and returns a `MiddlewareChain` from all plugins that declare `middleware` entries. Middleware classes are imported and instantiated by dotted path.

### Registry Registration Methods

| Method                       | Description                          |
|------------------------------|--------------------------------------|
| `register_analyzer(name, analyzer)` | Register a named analyzer   |
| `register_datasource(name, datasource)` | Register a datasource  |
| `register_vector_provider(name, provider)` | Register a vector provider |
| `register_middleware(name, middleware)` | Register a middleware instance |
| `register_embedding(name, embedding)` | Register an embedding provider |
| `register_query_rewriter(plugin)` | Register a query rewriter plugin |

### Lookup Methods

| Method                       | Returns                          |
|------------------------------|----------------------------------|
| `get(name)`                  | `Plugin` instance                |
| `list_plugins()`             | All registered plugin names      |
| `list_enabled()`             | Enabled plugin names             |
| `get_analyzer(name)`         | Analyzer callable                |
| `list_analyzers()`           | Registered analyzer names        |
| `list_datasources()`         | Registered datasource names      |
| `list_vector_providers()`    | Registered vector provider names |
| `list_middlewares()`         | Registered middleware names      |
| `list_embeddings()`          | Registered embedding names       |
| `list_query_rewriters()`     | Registered query rewriter names  |

## Built-in Plugins

| Plugin            | Module                  | Entry-point Group       |
|-------------------|-------------------------|-------------------------|
| `whoosh_autocomplete` | `whoosh_modern.autocomplete.plugin` | `whoosh.plugins` |
| `whoosh_vector`   | `whoosh_modern.vector.plugin`      | `whoosh.plugins` |
| `whoosh_fastapi`  | `whoosh_fastapi`                  | `whoosh.apps` |
| `whoosh_observability` | `whoosh.middleware.metrics`  | `whoosh.middlewares` |
| `whoosh_admin`    | `whoosh_admin`                   | `whoosh.apps` |

## Best Practices

1. **Single responsibility**: One plugin, one feature
2. **Declare dependencies**: Use `depends_on` for required plugins
3. **Semantic versioning**: Increment version for API changes
4. **Graceful degradation**: Check for optional dependencies in `register()`
5. **Avoid side effects in `__init__`**: All setup in `register()`
6. **Clean up**: If applicable, provide teardown logic

## See Also

- [Middleware Guide](middleware-pipeline.md) — Pipeline hooks and custom middleware
- [Provider Integration Guide](provider-integration.md) — Complete pipeline guide for all providers
- [Plugin Development Example](../examples/plugin-dev.md) — Step-by-step plugin tutorial
- [API: Plugins](../api/plugins.md) — Full API reference


## DOCUMENT: Plugins

# Plugins

Whoosh-NG uses a plugin architecture to keep the core lightweight while enabling advanced features. Plugins are loaded via entry points and managed by the `PluginManager`.

## Plugin Architecture

```text
PluginManager
    ├── load_plugins()     # Auto-discover from entry points
    ├── register(name, plugin)  # Manual registration
    ├── enable(name)       # Enable a plugin
    ├── disable(name)      # Disable a plugin
    ├── get(name)          # Retrieve a plugin
    └── list_plugins()     # List all plugins
```

## Built-in Plugins

| Plugin | Package | Description |
|--------|---------|-------------|
| whoosh-ng-vector | `whoosh_modern.vector` | Vector search providers (NumPy, HNSW, Faiss) |
| whoosh-ng-autocomplete | `whoosh_modern.autocomplete` | Edge ngram autocomplete |
| whoosh-ng-fastapi | `whoosh_fastapi` | FastAPI app factory |
| whoosh-ng-observability | `whoosh_observability` | Prometheus metrics |
| whoosh-ng-admin | `whoosh_admin` | Admin UI |

## Creating a Plugin

Every plugin is a subclass of `BasePlugin`:

```python
from whoosh.plugins.base import BasePlugin

class MyPlugin(BasePlugin):
    name = "my_plugin"
    version = "1.0.0"
    dependencies = []

    def setup(self, registry):
        """Called when the plugin is enabled."""
        registry.register("my_provider", MyProvider())

    def teardown(self, registry):
        """Called when the plugin is disabled."""
        registry.unregister("my_provider")

    def middleware(self):
        """Return middleware to inject into the pipeline."""
        return [MyMiddleware()]

    def on_startup(self):
        """Called once at application startup."""
        pass

    def on_shutdown(self):
        """Called once at application shutdown."""
        pass
```

## Plugin Registration

### Via entry_points (pyproject.toml)

```toml
[project.entry-points."whoosh_ng.plugins"]
my_plugin = "my_package.plugin:MyPlugin"
```

### Programmatic

```python
from whoosh.plugins.manager import PluginManager

plugin = MyPlugin()
PluginManager.register("my_plugin", plugin)
PluginManager.enable("my_plugin")
```

## Plugin Lifecycle

```
register() -> setup() -> enable() -> middleware hooks -> teardown() -> disable()
```

## Plugin Dependencies

Plugins can declare dependencies on other plugins:

```python
class VectorPlugin(BasePlugin):
    name = "vector"
    version = "1.0.0"
    dependencies = ["metrics"]  # Requires metrics plugin
```

The `PluginManager` resolves dependency order and detects conflicts.

## Example: Vector Plugin

```python
from whoosh.plugins.base import BasePlugin
from whoosh.vector.base import VectorProvider, VectorField
from whoosh.vector.numpy_provider import NumpyProvider

class VectorPlugin(BasePlugin):
    name = "vector"
    version = "1.0.0"
    dependencies = []

    def setup(self, registry):
        provider = NumpyProvider()
        registry.register("numpy", provider, owner="vector")

    def middleware(self):
        from whoosh.middleware import MetricsMiddleware
        return [MetricsMiddleware()]

    def on_startup(self):
        print("Vector plugin loaded")
```

## Example: FastAPI Plugin

```python
from whoosh.plugins.base import BasePlugin

class FastAPIPlugin(BasePlugin):
    name = "fastapi"
    version = "1.0.0"

    def setup(self, registry):
        self.app = None

    def create_app(self, index, **kwargs):
        from whoosh_fastapi import create_app
        self.app = create_app(index=index, **kwargs)
        return self.app

    def middleware(self):
        from whoosh.middleware import CacheMiddleware
        return [CacheMiddleware()]
```

## Best Practices

1. **Keep plugins small**: One plugin, one responsibility
2. **Declare dependencies**: Help PluginManager resolve load order
3. **Handle conflicts**: Check for existing registrations before adding
4. **Clean up**: Implement `teardown()` to remove registries and middleware
5. **Version your plugin**: Semver for compatibility checking

## Modern Plugin System (Whoosh-NG 2.0)

Whoosh-NG 2.0 introduces an enhanced `PluginManager` with registry support for datasources, vector providers, embeddings, and middleware. For full details on the modern plugin architecture, entry point groups, and deployment, see the [Plugin System Guide](plugins-advanced.md).


## DOCUMENT: Provider Integration

# Provider Integration: Complete Pipeline Guide

Module: `whoosh_modern.storage`, `whoosh_modern.analysis.stemmer_providers`, `whoosh_modern.linguistics.synonyms`, `whoosh_modern.vector`, `whoosh_modern.autocomplete`
Version: 3.0.0

This guide explains how all Whoosh-NG providers integrate into the indexing and
search pipeline. It is the definitive reference for understanding the data flow
from raw documents to search results.

## Overview

Whoosh-NG uses a **provider pattern** to keep the core engine lean while enabling
pluggable behavior for storage, text analysis, vector search, and autocomplete.

```
┌──────────────────────────────────────────────────────────────────────┐
│                        Whoosh-NG Provider Stack                      │
│                                                                      │
│  ┌─────────────┐  ┌──────────────┐  ┌─────────────┐  ┌───────────┐ │
│  │ Storage     │  │ Stemmer      │  │ Synonym     │  │ Vector    │ │
│  │ Providers   │  │ Providers    │  │ Providers   │  │ Providers │ │
│  │             │  │              │  │             │  │           │ │
│  │ FileStorage │  │ Internal     │  │ Static      │  │ Numpy     │ │
│  │ S3Storage   │  │ PyStemmer    │  │ YAML        │  │ HNSW      │ │
│  │ Hybrid      │  │ Identity     │  │ JSON        │  │ Faiss     │ │
│  │ SQLite      │  │ Custom       │  │ SQLite      │  │ Qdrant    │ │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  └─────┬─────┘ │
│         │                │                │                │       │
│         ▼                ▼                ▼                ▼       │
│  ┌─────────────────────────────────────────────────────────────────┐│
│  │              Middleware Pipeline (hooks)                         ││
│  │  ┌─────────────┐  ┌──────────────┐  ┌─────────────────────┐    ││
│  │  │Storage      │  │Stemming      │  │Synonym              │    ││
│  │  │Middleware   │  │Middleware    │  │ExpansionMiddleware  │    ││
│  │  └─────────────┘  └──────────────┘  └─────────────────────┘    ││
│  └─────────────────────────────────────────────────────────────────┘│
│                              │                                       │
│                              ▼                                       │
│  ┌─────────────────────────────────────────────────────────────────┐│
│  │              Whoosh Core Engine                                  ││
│  │  Index │ Writer │ Searcher │ QueryParser │ Segment files         ││
│  └─────────────────────────────────────────────────────────────────┘│
└──────────────────────────────────────────────────────────────────────┘
```

## Complete Indexing Pipeline

### Step-by-step flow

```text
┌─────────────────┐
│   DataSource    │  (SQL, JSON, REST, CSV, DataFrame, etc.)
│   .stream_batches() │
└────────┬────────┘
         │ batches of documents
         ▼
┌─────────────────┐
│  SchemaDiscovery │  Infers Whoosh Schema from data source columns
│  .discover_schema() │
└────────┬────────┘
         │ Schema(TEXT, ID, NUMERIC, VECTOR, ...)
         ▼
┌─────────────────────────────────────────┐
│  Storage Provider Resolution             │
│                                         │
│  storage._root (if any)                 │
│    └──► whoosh.index.create_in(root)    │
│  No root                                 │
│    └──► tempfile.mkdtemp() → create_in() │
└────────┬────────────────────────────────┘
         │ Index instance
         ▼
┌─────────────────────────────────────────┐
│  Writer + MiddlewareChain                │
│                                         │
│  chain.run_before("before_index")        │
│    ├── StorageMiddleware                 │
│    │   └── tags context with provider    │
│    ├── StemmingMiddleware                │
│    │   └── stems document fields         │
│    └── SynonymExpansionMiddleware        │
│        └── expands fields with synonyms  │
│                                         │
│  writer.add_document(**doc)              │
│    └── Whoosh core applies field         │
│        analyzers (TEXT.analyzer)          │
│        and writes to segment             │
│                                         │
│  writer.commit()                         │
│    └── chain.run_after("on_commit")      │
│        └── StorageMiddleware             │
│            └── writes commit checkpoint  │
└────────┬────────────────────────────────┘
         │ Segment files on disk/S3/cache
         ▼
┌─────────────────┐
│  Whoosh Index    │
│  (segments)      │
└─────────────────┘
```

### Concrete example

```python
from whoosh import index, fields
from whoosh_modern import (
    SearchApplication,
    SQLSource,
    HybridStorage,
    S3Storage,
    StemmingAnalyzer,
    get_stemmer,
    SynonymManager,
    SynonymExpansionMiddleware,
    StorageMiddleware,
    StemmingMiddleware,
)
from sqlalchemy import create_engine

# 1. Data source
engine = create_engine("sqlite:///products.db")
source = SQLSource(query="SELECT id, name, description FROM products", connection=engine)

# 2. Storage
remote = S3Storage(bucket="my-index-bucket", prefix="segments")
storage = HybridStorage(local_cache="./cache", remote=remote)

# 3. Schema (auto-discovered from SQL columns)
#    But we customize the analyzer
stemmer = get_stemmer("auto", "english")
schema = fields.Schema(
    name=fields.TEXT(analyzer=StemmingAnalyzer(stemmer=stemmer), stored=True),
    description=fields.TEXT(analyzer=StemmingAnalyzer(stemmer=stemmer)),
    id=fields.ID(stored=True, unique=True),
)

# 4. Create index in storage root
ix = index.create_in(storage._cache_root, schema)

# 5. Build middleware chain
syn_manager = SynonymManager({"laptop": ["notebook", "portable"]})
chain = MiddlewareChain([
    StorageMiddleware(storage, name="products"),
    StemmingMiddleware(stemmer=stemmer.stem),
    SynonymExpansionMiddleware(syn_manager),
])

# 6. Index with middleware
with MiddlewareWriter(ix.writer(), chain) as writer:
    for batch in source.stream_batches():
        for doc in batch:
            writer.add_document(**doc)
    writer.commit()
```

## Complete Search Pipeline

### Step-by-step flow

```text
┌─────────────────┐
│   User Query    │  "running cats"
└────────┬────────┘
         │
         ▼
┌─────────────────────────────────────────┐
│  MiddlewareChain.run_before("search")    │
│                                         │
│  ├── StemmingMiddleware                  │
│  │   └── "running cats" → "run cat"      │
│  ├── SynonymExpansionMiddleware          │
│  │   └── "run cat" → "run cat running feline" │
│  └── QueryRewriteMiddleware              │
│      └── custom rewrites                 │
└────────┬────────────────────────────────┘
         │ Modified query
         ▼
┌─────────────────────────────────────────┐
│  QueryParser.parse(query)                │
│    └── Query object (Term, And, Or...)  │
└────────┬────────────────────────────────┘
         │ Query object
         ▼
┌─────────────────────────────────────────┐
│  Searcher.search(query)                  │
│                                         │
│  ┌───────────────────────────────────┐  │
│  │  Keyword search path              │  │
│  │  └── reads posting lists from     │  │
│  │      segment files                │  │
│  └───────────────────────────────────┘  │
│                                         │
│  ┌───────────────────────────────────┐  │
│  │  Vector search path (if VECTOR)   │  │
│  │  └── VectorRegistry.get(provider) │  │
│  │      └── NumpyProvider.search()   │  │
│  │          └── cosine similarity    │  │
│  └───────────────────────────────────┘  │
│                                         │
│  ┌───────────────────────────────────┐  │
│  │  Autocomplete path                │  │
│  │  └── AutocompleteRegistry.get()   │  │
│  │      └── provider.suggest()       │  │
│  └───────────────────────────────────┘  │
└────────┬────────────────────────────────┘
         │ Raw Results
         ▼
┌─────────────────────────────────────────┐
│  MiddlewareChain.run_after("search")     │
│                                         │
│  └── RankingMiddleware                   │
│      └── re-sorts results                │
└────────┬────────────────────────────────┘
         │ Final Results
         ▼
┌─────────────────┐
│  Hits returned   │
└─────────────────┘
```

### Concrete example

```python
from whoosh.qparser import QueryParser
from whoosh_modern.middleware import (
    StemmingMiddleware,
    RankingMiddleware,
    QueryRewriteMiddleware,
)
from whoosh_modern.analysis import get_stemmer
from whoosh_modern.vector import NumpyProvider
from whoosh_modern.vector.plugin import VectorPlugin
from whoosh.plugins.manager import PluginManager
import numpy as np

# 1. Setup plugins at startup
manager = PluginManager()
VectorPlugin().register(manager)

# 2. Open index
ix = index.open_dir("indexdir")

# 3. Build middleware chain
stemmer = get_stemmer("auto", "english")
chain = MiddlewareChain([
    StemmingMiddleware(stemmer=stemmer.stem),
    QueryRewriteMiddleware(rewriter=lambda q: q + " portable"),  # add synonym
    RankingMiddleware(ranker=lambda r: sorted(r, key=lambda h: h.score, reverse=True)),
])

# 4. Search with middleware
with MiddlewareSearcher(ix.searcher(), chain) as searcher:
    # Query is transformed by middleware before execution
    results = searcher.search("laptop")
    for hit in results:
        print(f"{hit['name']}: {hit.score:.4f}")

    # 5. Vector search (parallel)
    query_vec = np.random.rand(384).tolist()
    vector_results = searcher.vector_search("embedding", query_vec, limit=10)
    for hit in vector_results:
        print(f"doc_id={hit.doc_id}, score={hit.score:.4f}")
```

## Provider Comparison Matrix

| Aspect | Storage | Stemmer | Synonym | Vector | Autocomplete |
|--------|---------|---------|---------|--------|--------------|
| **Integration point** | `StorageMiddleware` + `create_in()` | `StemmingAnalyzer` (field) + `StemmingMiddleware` | `SynonymExpansionMiddleware` | `VectorRegistry` + segment format | `AutocompleteRegistry` + standalone |
| **Registration** | Manual or `__getattr__` | `register_stemmer()` decorator | `SynonymManager` CRUD | `VectorPlugin.register()` | `AutocompletePlugin.register()` |
| **Used at index time** | Yes (commit checkpoints) | Yes (field analyzer + middleware) | Yes (before_index) | Yes (VECTOR field) | No (standalone or post-index) |
| **Used at search time** | Yes (segment reads via filesystem) | Yes (field analyzer + middleware) | Yes (before_search) | Yes (vector_search) | Yes (suggest/search) |
| **Persistence** | Segment files / S3 / SQLite | In-memory (stateless) | In-memory / YAML / JSON / SQLite | Segment files (metadata) | In-memory (phrase list) |
| **Configuration** | Provider class + kwargs | Backend name + language | Mapping dict or file | Provider name + metric | Provider type + params |

## Common Patterns

### Pattern 1: Provider as Field Analyzer

Used by: Stemmer providers, language analyzers

```python
schema = Schema(
    content=TEXT(analyzer=StemmingAnalyzer(stemmer="auto"))
)
```

The provider is wrapped in a Whoosh analyzer and applied automatically.

### Pattern 2: Provider as Middleware

Used by: Storage, Stemmer, Synonyms

```python
chain = MiddlewareChain([
    StorageMiddleware(storage),
    StemmingMiddleware(stemmer=stemmer.stem),
    SynonymExpansionMiddleware(manager),
])
```

The provider is consumed by middleware hooks in the pipeline.

### Pattern 3: Provider as Registry Entry

Used by: Vector, Autocomplete

```python
VectorRegistry.register("numpy", NumpyProvider(), owner="my_app")
provider = VectorRegistry.get("numpy", "my_app")
```

The provider is stored in a global registry and resolved by name at runtime.

### Pattern 4: Provider as Standalone Service

Used by: Autocomplete, Vector (manual mode)

```python
provider = NumpyProvider()
provider.add([(doc_id, vec)])
results = provider.search(query_vec)
```

The provider operates independently of Whoosh's index/searcher.

## Best Practices

1. **Choose the right integration pattern**: Field analyzers for static schemas, middleware for dynamic behavior, registry for pluggable backends.
2. **Avoid double-application**: Don't use both field-level analyzers and middleware for the same transformation (e.g., stemming).
3. **Register providers at startup**: Call `VectorPlugin().register(manager)` and `AutocompletePlugin().register(manager)` before creating indexes.
4. **Use the highest-level API when possible**: `SearchApplication` for end-to-end, `create_autocomplete()` for suggestions, `get_stemmer()` for stemming.
5. **Keep providers stateless**: Providers should not hold index-specific state; use middleware context for per-request data.
6. **Test providers in isolation**: Each provider should be testable without Whoosh core (unit tests for `provider.search()`, `provider.add()`).
7. **Document provider dependencies**: Note optional dependencies (boto3, PyStemmer, PyYAML) in your project's requirements.

## See Also

- [Storage Providers Guide](storage-providers.md) — Storage backend integration
- [Stemming Guide](../core/stemming.md) — Stemmer provider integration
- [Vector Search Guide](vector.md) — Vector provider integration
- [Autocomplete Guide](autocomplete.md) — Autocomplete provider integration
- [Middleware Guide](middleware-pipeline.md) — Pipeline hooks and provider adapters
- [Plugins Guide](plugins-advanced.md) — Plugin registration and entry points
- [API: Modern](../api/modern.md) — Full API reference for all providers


## DOCUMENT: Stemming Providers

# Stemmer Providers

Module: `whoosh_modern.analysis.stemmer_providers`, `whoosh_modern.analysis.stemming_analyzer`, `whoosh_modern.linguistics.stemmers`
Version: 3.0.0

The stemmer provider system gives you flexible control over which stemming backend is used for text analysis. It supports auto-detection, explicit backend selection, and custom stemmer registration—all with a clean plugin-style API.

## Module Overview

```text
whoosh_modern.analysis
    ├── stemmer_providers.py   # StemmerProvider protocol, Internal/PyStemmer backends, register_stemmer, get_stemmer
    └── stemming_analyzer.py   # Enhanced StemmingAnalyzer with plugin support

whoosh_modern.linguistics.stemmers
    └── __init__.py            # Language-specific analyzers (FR/EN/DE/ES/IT)
```

## StemmerProvider Protocol

Located in `whoosh_modern.analysis.stemmer_providers`:

```python
from whoosh_modern.analysis.stemmer_providers import StemmerProvider

class MyStemmer(StemmerProvider):
    def stem(self, word: str) -> str:
        """Stem a single word."""
        ...

    @property
    def name(self) -> str:
        """Return the stemmer name."""
        return "my_stemmer"

    @property
    def language(self) -> str:
        """Return the language code."""
        return "english"
```

## Getting a Stemmer

### Auto-Detection (Recommended)

The `get_stemmer("auto", language)` function automatically selects the best available backend:

```python
from whoosh_modern.analysis.stemmer_providers import get_stemmer

# Auto-detect: prefers PyStemmer if installed, falls back to internal
stemmer = get_stemmer("auto", "english")
print(stemmer.stem("running"))  # "run"
print(stemmer.name)             # "pystemmer" or "internal"
```

**Priority order:**
1. **PyStemmer** (fastest, requires `pip install whoosh-ng[fast-stemming]`)
2. **Internal** stemmer (built-in Porter stemmer, always available)

### Explicit Backend Selection

```python
from whoosh_modern.analysis.stemmer_providers import get_stemmer

# Force internal stemmer
stemmer = get_stemmer("internal", "english")

# Force PyStemmer (requires installation)
stemmer = get_stemmer("pystemmer", "english")
```

### List Available Backends

```python
from whoosh_modern.analysis.stemmer_providers import list_available_backends

backends = list_available_backends()
print(backends)
# {'internal': 'available', 'pystemmer': 'available', 'my_custom': 'registered'}
```

| Backend       | Status String    | Requires                          |
|---------------|------------------|-----------------------------------|
| `internal`    | `"available"`    | None (always bundled)             |
| `pystemmer`   | `"available"` / `"not installed"` | `pip install whoosh-ng[fast-stemming]` |
| Custom        | `"registered"`   | Registered via `@register_stemmer` |

## Built-in Stemmer Providers

### InternalStemmerProvider

Wraps Whoosh's built-in Porter stemmer. Always available (no extra dependencies):

```python
from whoosh_modern.analysis.stemmer_providers import InternalStemmerProvider

stemmer = InternalStemmerProvider("english")
print(stemmer.stem("cats"))    # "cat"
print(stemmer.stem("running")) # "run"
```

### PyStemmerProvider

Wraps the `Stemmer` library for high-performance stemming. Supports all Snowball languages:

```python
from whoosh_modern.analysis.stemmer_providers import PyStemmerProvider

# Requires: pip install whoosh-ng[fast-stemming]
stemmer = PyStemmerProvider("english")
print(stemmer.stem("cats"))    # "cat"
```

**Note**: This provider calls `self._stemmer.stemWord(word)` to stem words. Ensure PyStemmer is installed or auto-detection will fall back to the internal stemmer.

### IdentityStemmerProvider

A no-op stemmer for testing or when stemming is not desired:

```python
from whoosh_modern.analysis.stemmer_providers import IdentityStemmerProvider

stemmer = IdentityStemmerProvider()
print(stemmer.stem("anything"))  # "anything"
```

## Registering a Custom Stemmer

Use the `@register_stemmer` decorator:

```python
from whoosh_modern.analysis.stemmer_providers import register_stemmer

@register_stemmer("simple")
class SimpleStemmer:
    def stem(self, word: str) -> str:
        # Simple suffix stripping
        if word.endswith("s") and len(word) > 3:
            return word[:-1]
        return word

    @property
    def name(self) -> str:
        return "simple"

    @property
    def language(self) -> str:
        return "english"

# Now use it
from whoosh_modern.analysis.stemmer_providers import get_stemmer

stemmer = get_stemmer("simple", "english")
print(stemmer.stem("cats"))  # "cat"
```

## StemmingAnalyzer (Enhanced)

Located in `whoosh_modern.analysis.stemming_analyzer`, this is the main entry point for creating language-aware analyzers:

```python
from whoosh_modern.analysis import StemmingAnalyzer

# Auto-detect best stemmer for English
analyzer = StemmingAnalyzer(stemmer="auto", language="english")

# Explicit internal stemmer
analyzer = StemmingAnalyzer(stemmer="internal", language="english")

# PyStemmer backend (if installed)
analyzer = StemmingAnalyzer(stemmer="pystemmer", language="french")

# Custom stemmer provider
analyzer = StemmingAnalyzer(stemmer=my_stemmer_instance)
```

### StemmingAnalyzer Parameters

| Parameter   | Type                          | Default                  | Description                      |
|-------------|-------------------------------|--------------------------|----------------------------------|
| `expression`| Regex pattern                 | default token pattern    | Tokenization regex              |
| `stoplist`  | Iterable of stop words        | `whoosh.analysis.STOP_WORDS` | Stop words to filter         |
| `minsize`   | `int`                         | `2`                      | Minimum token length            |
| `maxsize`   | `int \| None`                 | `None`                   | Maximum token length            |
| `gaps`      | `bool`                        | `False`                  | Split on expression vs. match  |
| `stemmer`   | `str \| StemmerProvider`      | `"auto"`                 | Stemmer backend                 |
| `language`  | `str`                         | `"english"`              | Language code                   |
| `ignore`    | `set[str] \| None`            | `None`                   | Words to skip                   |
| `cachesize` | `int`                         | `50000`                  | Stem cache size                 |

### Using with Field Types

```python
from whoosh_modern.analysis import StemmingAnalyzer
from whoosh.fields import Schema, TEXT

# English stemmer with stop words
en_analyzer = StemmingAnalyzer("auto", language="english")

# French stemmer
fr_analyzer = StemmingAnalyzer("auto", language="french")

schema = Schema(
    title=TEXT(stored=True),
    content_en=TEXT(analyzer=en_analyzer),
    content_fr=TEXT(analyzer=fr_analyzer),
)
```

## Language-Specific Analyzers

Pre-built analyzers for five languages, available in `whoosh_modern.linguistics.stemmers`:

```python
from whoosh_modern.linguistics.stemmers import (
    EnglishAnalyzer,
    FrenchAnalyzer,
    GermanAnalyzer,
    SpanishAnalyzer,
    ItalianAnalyzer,
)

# Each is callable and returns a list of tokens
en = EnglishAnalyzer()
tokens = en("The quick brown foxes")
# tokens are stemmed: ["quick", "brown", "fox"] (stop words like "the" removed)
```

### Available Language Analyzers

| Class             | Language  | Module                              |
|-------------------|-----------|-------------------------------------|
| `EnglishAnalyzer` | English   | `whoosh_modern.linguistics.stemmers` |
| `FrenchAnalyzer`  | French    | `whoosh_modern.linguistics.stemmers` |
| `GermanAnalyzer`  | German    | `whoosh_modern.linguistics.stemmers` |
| `SpanishAnalyzer` | Spanish   | `whoosh_modern.linguistics.stemmers` |
| `ItalianAnalyzer` | Italian   | `whoosh_modern.linguistics.stemmers` |

Each internally uses `get_stemmer("auto", language)` to select the best available backend and applies language-specific stop words.

## Stemmer Compatibility Validation

Validate that a stemmer provider works correctly with a set of test words:

```python
from whoosh_modern.analysis.stemmer_providers import (
    get_stemmer,
    validate_stemmer_compatibility,
)

stemmer = get_stemmer("auto", "english")
report = validate_stemmer_compatibility(stemmer, ["running", "cats", "jumps", "houses"])

print(report["total_words"])   # 4
print(report["successful"])    # 4 (or fewer if errors)
print(report["failed"])        # 0
print(report["results"])       # [{'word': 'running', 'stemmed': 'run', 'success': True}, ...]
```

### Compatibility Report Structure

| Field          | Type       | Description                          |
|----------------|------------|--------------------------------------|
| `provider`     | `str`      | Stemmer provider name                |
| `language`     | `str`      | Language code                        |
| `total_words`  | `int`      | Total test words                     |
| `successful`   | `int`      | Words stemmed successfully           |
| `failed`       | `int`      | Words that failed                    |
| `results`      | `list[dict]` | Per-word results with `word`, `stemmed`, `success` |

## Integration with StemmingMiddleware

The stemmer providers can be used with the `StemmingMiddleware` from `whoosh_modern.middleware.analyzer`:

```python
from whoosh_modern.analysis.stemmer_providers import get_stemmer
from whoosh_modern.middleware.analyzer import StemmingMiddleware

stemmer = get_stemmer("auto", "english")
middleware = StemmingMiddleware(
    stemmer=stemmer.stem,
    fields=["title", "content"],  # Only stem these fields
    stem_query=True,              # Also stem the search query
)
```

## Migration from Classic Whoosh

### Old API (Whoosh 1.x/2.x)

```python
from whoosh.analysis import StemmingAnalyzer as OldAnalyzer
analyzer = OldAnalyzer("en")  # Hardcoded to "english"
```

### New API (Whoosh-NG 2.0)

```python
from whoosh_modern.analysis import StemmingAnalyzer

# Auto-detect backend (preferred)
analyzer = StemmingAnalyzer("auto", language="en")

# Or use a language-specific analyzer
from whoosh_modern.linguistics.stemmers import EnglishAnalyzer
analyzer = EnglishAnalyzer()
```

> **Note**: The old `StemmingAnalyzer("en")` hardcoded the language to `"english"`. The new `StemmingAnalyzer(stemmer, language)` parameter is explicit and supports all Snowball languages via PyStemmer.

## Installation

```bash
# Without PyStemmer (uses internal stemmer, slower)
pip install whoosh-ng

# With PyStemmer (recommended, faster)
pip install whoosh-ng[fast-stemming]

# Full modern analysis
pip install whoosh-ng[modern]
```

## Stemmer Provider Integration in the Pipeline

The `StemmerProvider` system integrates at **two levels**: field-level analyzers and
pipeline middleware. Understanding both is key to avoiding double-stemming.

### Architecture

```text
┌─────────────────────────────────────────────────────────────────┐
│  StemmingAnalyzer (field-level, in Schema)                      │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ RegexTokenizer() │ StopFilter │ StemmingAnalyzer          │  │
│  │                    (stop words)    │                       │  │
│  │                                   ▼                       │  │
│  │                         stemfn = provider.stem            │  │
│  │                                   │                       │  │
│  │                                   ▼                       │  │
│  │                         Token(stemmed=True)                │  │
│  └───────────────────────────────────────────────────────────┘  │
│                                                                 │
│  Applied by Whoosh core at index time AND query time            │
│  (via QueryParser). Automatic, no middleware needed.             │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│  StemmingMiddleware (pipeline-level)                            │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ before_index(context)                                     │  │
│  │   └── stem all str values in context.document             │  │
│  │                                                             │  │
│  │ before_search(context)                                     │  │
│  │   └── stem context.query if stem_query=True                │  │
│  └───────────────────────────────────────────────────────────┘  │
│                                                                 │
│  Hooked into MiddlewareChain. Manual opt-in.                    │
└─────────────────────────────────────────────────────────────────┘
```

### Level 1: Field-level (automatic)

The `StemmingAnalyzer` wraps Whoosh's built-in `StemmingAnalyzer` and injects
a `StemmerProvider`'s `.stem` method as the `stemfn`. Whoosh core applies it
automatically to the field at both index time and query time.

```python
from whoosh.fields import Schema, TEXT
from whoosh_modern.analysis import StemmingAnalyzer, get_stemmer

# Auto-detect best stemmer (PyStemmer preferred)
stemmer = get_stemmer("auto", "english")

# Create analyzer with the provider's stem function
analyzer = StemmingAnalyzer(stemmer=stemmer)

schema = Schema(
    title=TEXT(stored=True),
    content=TEXT(analyzer=analyzer),
)

# At index time: "running cats" → ["run", "cat"]
# At query time: QueryParser also uses the same analyzer
# so "running cats" matches documents containing "run cat"
```

**Pros**: Automatic, no middleware configuration needed, consistent index/query behavior.

**Cons**: Requires the analyzer to be set on each `TEXT` field. Harder to change at runtime.

### Level 2: Middleware-level (opt-in)

`StemmingMiddleware` applies stemming at the pipeline level, operating on raw
string values in `context.document` and `context.query` before Whoosh's analyzers
see them.

```python
from whoosh_modern.middleware import StemmingMiddleware
from whoosh_modern.analysis import get_stemmer
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.wrappers import MiddlewareWriter

stemmer = get_stemmer("auto", "english")

chain = MiddlewareChain([
    StemmingMiddleware(
        stemmer=stemmer.stem,
        fields=["title", "content"],  # None = all str fields
        stem_query=True,
    ),
])

with MiddlewareWriter(ix.writer(), chain) as writer:
    writer.add_document(title="Running cats", content="Fast dogs")
    # before_index stems: "Running cats" → "run cat"
    writer.commit()
```

**Pros**: Works on any field without modifying the schema. Can be toggled at runtime.

**Cons**: Must be manually wired into the pipeline. Risk of double-stemming if the field also uses `StemmingAnalyzer`.

### Full pipeline example: index + search

```python
from whoosh import index, fields
from whoosh.qparser import QueryParser
from whoosh_modern.analysis import StemmingAnalyzer, get_stemmer
from whoosh_modern.middleware import StemmingMiddleware
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.wrappers import MiddlewareWriter, MiddlewareSearcher

# 1. Schema with field-level analyzer
stemmer = get_stemmer("auto", "english")
schema = fields.Schema(
    title=fields.TEXT(stored=True, analyzer=StemmingAnalyzer(stemmer=stemmer)),
    content=fields.TEXT(analyzer=StemmingAnalyzer(stemmer=stemmer)),
)

ix = index.create_in("indexdir", schema)

# 2. Index with middleware (no double-stemming because
#    we don't use StemmingMiddleware when fields already have StemmingAnalyzer)
with ix.writer() as writer:
    writer.add_document(title="Running cats", content="Fast dogs")
    writer.commit()

# 3. Search: QueryParser applies the same analyzer to the query
with ix.searcher() as searcher:
    qp = QueryParser("content", schema)
    q = qp.parse("running cats")
    results = searcher.search(q)
    # "running" is stemmed to "run" by the analyzer
    # "cats" is stemmed to "cat" by the analyzer
    # Matches document with "run" and "cat"
```

### Avoiding double-stemming

```python
# WRONG: double stemming
schema = Schema(
    content=TEXT(analyzer=StemmingAnalyzer(stemmer="auto")),
)
chain = MiddlewareChain([
    StemmingMiddleware(stemmer=get_stemmer("auto").stem),  # Don't do this!
])
# Result: "running" → "run" (analyzer) → "run" (middleware) — harmless but wasteful

# CORRECT: choose ONE level
# Option A: field-level only (recommended for static schemas)
schema = Schema(content=TEXT(analyzer=StemmingAnalyzer(stemmer="auto")))
# No StemmingMiddleware needed

# Option B: middleware-only (for dynamic fields)
schema = Schema(content=TEXT)  # No analyzer
chain = MiddlewareChain([StemmingMiddleware(stemmer=get_stemmer("auto").stem)])
```

### Custom stemmer provider

```python
from whoosh_modern.analysis import register_stemmer, get_stemmer

@register_stemmer("my_stemmer")
class MyStemmer:
    def stem(self, word: str) -> str:
        return word.lower().rstrip("s")

# Use it like any built-in backend
stemmer = get_stemmer("my_stemmer", "english")
analyzer = StemmingAnalyzer(stemmer=stemmer)
```

## See Also

- [Stemming and Stop Words Guide](../core/stemming.md) — Classic Whoosh stemming guide
- [Synonyms & Linguistics Guide](linguistics.md) — Synonym expansion engine
- [Provider Integration Guide](provider-integration.md) — Complete pipeline guide for all providers
- [API: Modern](../api/modern.md) — Full API reference for analysis extensions


## DOCUMENT: Storage Providers

# Storage Providers

Whoosh-NG provides pluggable storage backends through the
`SyncStorageProvider` / `AsyncStorageProvider` contracts. This allows the
index to be persisted on local disk, SQLite, S3, or a hybrid cache + remote
setup without changing the writer or the index.

## Architecture Overview

### Level 1: SnapshotStorage (Simple)

```
Writer → Local FS → Commit → Upload Segment → S3
Reader → Download Segment → Open locally
```

Very simple to maintain. Use `SnapshotStorage` when you want S3 as a simple
backup/restore target without the complexity of a local cache.

### Level 2: CachedObjectStorage (Recommended for Production)

```
+----------+
|  MinIO   |
+----------+
     ^
     |
 Sync |
     v
+-----------+   Cache Layer   +-----------+
| Searcher  |<--------------->| Writer    |
+-----------+                 +-----------+
        |
        v
 Local SSD
```

- Index lives on SSD
- S3 serves as replication
- Segments are pushed after commit
- Restoration possible at any moment

This is what many modern distributed search systems do.

## Available providers

| Provider | Type | Backend | Use Case |
|----------|------|---------|----------|
| `FileStorage` | sync | local filesystem | Single-node, no cloud |
| `AsyncFileStorage` | async | local filesystem | Single-node async |
| `S3Storage` | sync | S3-compatible | Direct S3 access |
| `SnapshotStorage` | sync | S3-compatible | Simple backup/restore |
| `HybridStorage` | sync | local cache + remote | **Production** (alias: `CachedObjectStorage`) |
| `AsyncHybridStorage` | async | local cache + remote | Production async |
| `CoreStorageAdapter` | sync | core `FileStorage` → modern `SyncStorageProvider` | Bridge legacy core backends into the modern pipeline |

All providers are importable from `whoosh_modern.storage`.

## FileStorage

Local filesystem storage. Keys are relative paths under `root`.

```python
from whoosh_modern.storage import FileStorage

storage = FileStorage("indexdir")
storage.write("segment_1.dat", b"data")
assert storage.read("segment_1.dat") == b"data"
assert storage.exists("segment_1.dat") is True
storage.delete("segment_1.dat")
keys = storage.list_keys()
```

## CoreStorageAdapter

Wraps a core ``whoosh.filedb.filestore.FileStorage`` instance behind the modern
``SyncStorageProvider`` interface. This lets you drop an existing core storage
backend into a modern ``HybridStorage`` or ``StorageMiddleware`` chain without
modifying the core class.

```python
from whoosh.filedb.filestore import FileStorage as CoreFileStorage
from whoosh_modern.storage import CoreStorageAdapter

core = CoreFileStorage("indexdir")
adapter = CoreStorageAdapter(core)

adapter.write("segment_1.dat", b"data")
assert adapter.read("segment_1.dat") == b"data"
assert adapter.exists("segment_1.dat") is True
adapter.delete("segment_1.dat")
```

## AsyncFileStorage

Async variant of `FileStorage`. All operations run on a worker thread via
`asyncio.to_thread` so the event loop is never blocked.

```python
import asyncio
from whoosh_modern.storage import AsyncFileStorage

storage = AsyncFileStorage("indexdir")

async def main() -> None:
    await storage.awrite("segment_1.dat", b"data")
    data = await storage.aread("segment_1.dat")
    await storage.adelete("segment_1.dat")

asyncio.run(main())
```

## S3Storage

S3-compatible blob storage. `boto3` is imported lazily, so it is an optional
dependency. A `client` can be injected for testing.

```python
from whoosh_modern.storage import S3Storage

# Default client (requires boto3 installed and configured)
storage = S3Storage(bucket="my-index-bucket", prefix="segments")

# Or inject a client for testing / custom configuration
storage = S3Storage(
    bucket="my-index-bucket",
    prefix="segments",
    client=my_boto3_client,
)

storage.write("segment_1.dat", b"data")
data = storage.read("segment_1.dat")
keys = storage.list_keys()
```

Install the optional dependency:

```bash
pip install whoosh-ng[s3]
```

## SnapshotStorage

Simple S3 snapshot storage. S3 remains the source of truth; on read, the object is
downloaded from S3 and also persisted under `local_path` so subsequent reads of the same
key can be served from the local scratch copy:

- Write: upload segment directly to S3
- Read: download segment from S3, cache it under `local_path`

Use this when you want S3 as a simple backup/restore target without the
complexity of a local cache.

```python
from whoosh_modern.storage import SnapshotStorage

storage = SnapshotStorage(
    local_path="./index",
    bucket="my-index-bucket",
    prefix="snapshots",
)

storage.write("segment_1.dat", b"data")
data = storage.read("segment_1.dat")
```

## HybridStorage / CachedObjectStorage

`HybridStorage` composes a local cache and a remote backend. The remote is
the source of truth; the local cache is a write-through performance layer.

`CachedObjectStorage` is an alias for `HybridStorage` that better conveys
the intent: a local object cache synchronized with S3.

This is the recommended architecture for production deployments with repeated
read patterns.

```python
from whoosh_modern.storage import HybridStorage, S3Storage

remote = S3Storage(bucket="my-index-bucket", prefix="segments")
storage = HybridStorage(local_cache="./cache", remote=remote)

# Write-through: remote is source of truth, cache is updated on success
storage.write("segment_1.dat", b"data")

# First read: cache miss → fetch from S3, write-through into cache
data = storage.read("segment_1.dat")

# Second read: cache hit → served from local disk, zero network
data = storage.read("segment_1.dat")

# Force refresh from remote
storage.invalidate("segment_1.dat")

# Warm cache proactively
storage.prefetch(["segment_2.dat", "segment_3.dat"])
```

### Read path

1. local cache hit → return immediately
2. cache miss → read from remote, write-through into cache, return

### Write path

- `remote.write(key, data)` (source of truth)
- on success → `local_cache.write(key, data)`
- on failure → raise before polluting cache

### Cache eviction

The local cache is bounded by `max_cache_size_mb` (default 1024 MB). When
the limit is reached, the oldest entries are evicted using an LRU policy.

### `list_keys`

`list_keys()` uses the remote as source of truth because the cache is only
partial. Pass `include_cache=True` to return the union of remote and cache
keys.

## AsyncHybridStorage

Async variant of `HybridStorage`. Remote operations are executed on a worker
thread via `asyncio.to_thread` so the event loop is never blocked.

```python
import asyncio
from whoosh_modern.storage import AsyncHybridStorage, S3Storage

remote = S3Storage(bucket="my-index-bucket", prefix="segments")
storage = AsyncHybridStorage(local_cache="./cache", remote=remote)

async def main() -> None:
    await storage.awrite("segment_1.dat", b"data")
    data = await storage.aread("segment_1.dat")
    await storage.adelete("segment_1.dat")
    keys = await storage.alist_keys()

asyncio.run(main())
```

## Using storage with SearchApplication

```python
from whoosh_modern import SearchApplication, SQLSource
from whoosh_modern.storage import HybridStorage, S3Storage

remote = S3Storage(bucket="my-index-bucket", prefix="segments")
storage = HybridStorage(local_cache="./cache", remote=remote)

app = SearchApplication(
    source=SQLSource(query="SELECT * FROM products", connection=engine),
    storage=storage,
)
app.build()
results = app.index.search("laptop")
```

## Performance Benchmarks

Benchmarks were run against a local MinIO instance using a 28.89 MB Whoosh
index (2 segment files). Results are indicative of relative performance
between strategies on S3-compatible storage.

| Strategy | Backup (MB/s) | Restore (MB/s) | Notes |
|----------|---------------|----------------|-------|
| `1_obj_per_segment` | 39.44 | 139.72 | Best restore throughput; simplest |
| `compressed_zstd` | 31.56 | 133.74 | Lower bandwidth, CPU overhead |
| `hybrid_cache_s3` | 44.97 | 133.61 | Best backup; excellent warm-cache reads |
| `1_obj_per_posting_list` | 0.28 | 4.79 | **Avoid**: millions of small objects kill S3 |

### Recommendations

- **Default**: `S3Storage` with 1 object per segment file. It offers the
  best restore throughput and is the simplest to operate.
- **Production with repeated reads**: `HybridStorage(local_cache, S3Storage)`.
  After the first read, subsequent reads are served from local disk at
  ~133 MB/s.
- **Avoid**: 1 object per posting list. S3 is not optimized for millions of
  tiny objects; latency and cost explode.
- **Compression**: ZSTD reduces transfer size by ~20-30% at the cost of CPU.
  Use it when network bandwidth is the bottleneck, not when CPU is.

### Running the benchmarks

```bash
# Start MinIO
docker run -d --name minio-benchmark -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \
  minio/minio:latest server /data --console-address ":9001"

# Run synthetic benchmark
python benchmark/s3_storage_benchmark.py

# Run real Whoosh index benchmark (requires customers CSV)
python benchmark/s3_storage_benchmark_real.py
```

## How Storage Providers Integrate into the Indexing and Search Pipeline

The storage provider participates in two distinct phases: **index creation** (determining where segments live) and **runtime pipeline integration** (via `StorageMiddleware`).

### Full indexing flow with a storage provider

```text
DataSource.stream_batches()
    │
    ▼
SearchApplication.build()
    │
    ├── source.discover_schema() ──► Whoosh Schema
    │
    ├── storage root resolution
    │       │
    │       ├── FileStorageProvider (exposed as `FileStorage`)
    │       │   └── exposes a public `root` ──► whoosh.index.create_in(root, schema)
    │       │
    │       └── Other providers (S3 / Snapshot / Hybrid, no filesystem root)
    │           └── tempfile.mkdtemp() ──► create_in(tmpdir, schema)
    │
    ├── Writer = index.writer()
    │       │
    │       ├── MiddlewareChain.before_index()
    │       │   └── StorageMiddleware.before_index()
    │       │       ├── context.labels["storage_backend"] = provider.__class__.__name__
    │       │       └── context.metadata["storage_provider"] = self
    │       │
    │       ├── for batch in source.stream_batches():
    │       │       for doc in batch:
    │       │           writer.add_document(**doc)
    │       │
    │       └── writer.commit()
    │           │
    │           └── StorageMiddleware.on_commit()
    │               └── provider.write("commits/{name}/{timestamp}", b"1")
    │
    ▼
Index persisted on disk / S3 / hybrid cache
```

### Full search flow with a storage provider

```text
SearchApplication.search(query)
    │
    ├── index.searcher()
    │       │
    │       └── Whoosh core opens segment files from:
    │           ├── local filesystem (FileStorageProvider root)
    │           ├── SQLite DB (SQLiteStorageProvider)
    │           └── S3 / hybrid cache (S3StorageProvider / HybridStorage)
    │
    ├── QueryParser.parse(query) ──► Query object
    │
    └── searcher.search(query)
        │
        └── Whoosh core reads posting lists from segment files
            └── Returns Results (Hits)
```

### StorageMiddleware hooks in detail

`StorageMiddleware` (`whoosh_modern.middleware.storage`) is the integration point
that routes index persistence through any `SyncStorageProvider` without modifying
the writer.

| Hook | When | What it does |
|------|------|--------------|
| `before_index(context)` | Before each document is added | Tags the context with `storage_backend` label and `storage_provider` metadata |
| `on_commit(context)` | After `writer.commit()` | Writes a commit checkpoint marker (`commits/{name}/{timestamp}`) to the provider |

### Example: StorageMiddleware with a custom chain

```python
from whoosh_modern.middleware import (
    StorageMiddleware,
    FileStorageProvider,
    StemmingMiddleware,
)
from whoosh.middleware.chain import MiddlewareChain
from whoosh_modern.analysis import get_stemmer

# Storage provider
storage = FileStorageProvider("/data/index")

# Create middleware chain
chain = MiddlewareChain([
    StorageMiddleware(storage, name="primary"),
    StemmingMiddleware(stemmer=get_stemmer("auto", "english").stem),
])

# Apply to writer
from whoosh.middleware.wrappers import MiddlewareWriter

with MiddlewareWriter(ix.writer(), chain) as writer:
    writer.add_document(title="Hello", content="World")
    # StorageMiddleware.before_index() tags the context
    # StemmingMiddleware stems the fields
    # writer.commit() triggers StorageMiddleware.on_commit()
    writer.commit()
```

### Key insight: StorageProvider vs StorageMiddleware

| Component | Role |
|-----------|------|
| `SyncStorageProvider` / `AsyncStorageProvider` | **Contract** defining `write()`, `read()`, `delete()`, `exists()`, `list_keys()` |
| `FileStorageProvider`, `S3StorageProvider`, `HybridStorage` | **Implementations** of the contract |
| `StorageMiddleware` | **Integration layer** that calls the provider at specific lifecycle hooks (`before_index`, `on_commit`) |
| `SearchApplication` | **Entry point** that delegates to `SearchView.build()`; when the storage is a `FileStorageProvider` (exposed as `FileStorage`) it uses its public `root` to create the Whoosh index directory, otherwise it falls back to a temporary directory |

The provider itself does **not** intercept Whoosh's internal segment reads. Those reads go through Whoosh's built-in `FileStorage` (`whoosh.filedb.filestore`) which reads from the filesystem path given to `create_in()`. The Whoosh-NG storage provider abstraction is designed for:
- Custom segment routing (S3, SQLite, hybrid cache)
- Commit checkpointing via middleware
- Future: segment-level read/write interception

## See Also

- [Provider Integration Guide](provider-integration.md) — Complete pipeline guide for all providers
- [Middleware Guide](middleware-pipeline.md) — Pipeline hooks and provider adapters


## DOCUMENT: Synonyms

# Synonyms

Module: `whoosh_modern.linguistics.synonyms`
Version: 3.0.0

The synonyms engine provides query-time and index-time synonym expansion through a pluggable provider system. It supports static in-memory mappings, YAML/JSON files, SQLite persistence, and large-scale Wiktionary dictionaries.

## Provider Architecture

All synonym providers implement the `SynonymProvider` protocol:

```python
from whoosh_modern.linguistics.synonyms import SynonymProvider

class MyProvider(SynonymProvider):
    def get_synonyms(self, word: str) -> list[str]: ...
    def add_synonym(self, word: str, synonyms: list[str]) -> None: ...
    def remove_synonym(self, word: str, synonym: str) -> None: ...
```

## Built-in Providers

### StaticSynonymProvider

In-memory provider backed by a dictionary:

```python
from whoosh_modern.linguistics.synonyms import StaticSynonymProvider

provider = StaticSynonymProvider({
    "car": ["automobile", "vehicle"],
    "house": ["home", "residence"],
})
print(provider.get_synonyms("car"))  # ['automobile', 'vehicle']
```

### YAMLSynonymProvider

Loads synonyms from a YAML file:

```yaml
# synonyms.yaml
car:
  - automobile
  - vehicle
house:
  - home
  - residence
```

```python
from whoosh_modern.linguistics.synonyms import YAMLSynonymProvider

provider = YAMLSynonymProvider("synonyms.yaml")
print(provider.get_synonyms("car"))  # ['automobile', 'vehicle']
```

### JSONSynonymProvider

Loads synonyms from a JSON file:

```json
{
    "car": ["automobile", "vehicle"],
    "house": ["home", "residence"]
}
```

```python
from whoosh_modern.linguistics.synonyms import JSONSynonymProvider

provider = JSONSynonymProvider("synonyms.json")
print(provider.get_synonyms("car"))
```

### WiktionarySynonymProvider

Loads synonyms from a kaikki.org JSON Lines dictionary file:

```python
from whoosh_modern.linguistics.synonyms import WiktionarySynonymProvider

provider = WiktionarySynonymProvider(
    "src/whoosh_modern/linguistics/dictionaries/wiktionary/fr.json"
)
print(provider.get_synonyms("voiture"))  # ['automobile', 'véhicule']
```

Each line in the dictionary file is a JSON object:

```json
{"word": "voiture", "s": ["automobile", "véhicule"]}
{"word": "ordinateur", "s": ["pc", "machine"]}
```

The provider filters out:
- Words containing spaces (multi-word expressions)
- Entries with non-standard parts of speech
- Empty or missing synonym lists

### SQLiteSynonymStore

Persistent synonym store backed by SQLite:

```python
from whoosh_modern.linguistics.synonyms import SQLiteSynonymStore

store = SQLiteSynonymStore("synonyms.db")
store.add_synonym("car", ["automobile", "vehicle"])
print(store.get_synonyms("car"))  # ['automobile', 'vehicle']
store.close()
```

## SynonymManager

The `SynonymManager` is the high-level interface for managing synonyms:

```python
from whoosh_modern.linguistics.synonyms import SynonymManager

manager = SynonymManager({"car": ["automobile", "vehicle"]})

# CRUD
manager.add_synonyms("house", ["home", "residence"])
print(manager.get_synonyms("house"))  # ['home', 'residence']
manager.remove_synonym("house", "home")

# Import from external sources
manager.import_yaml("synonyms.yaml")       # Requires PyYAML
manager.import_json("synonyms.json")
manager.import_wiktionary("dictionaries/wiktionary/fr.json")

# Export
manager.export_json("output.json")
```

## Updating Wiktionary Dictionaries

Pre-generated dictionaries live in `src/whoosh_modern/linguistics/dictionaries/wiktionary/`:

```
wiktionary/
├── fr.json
├── en.json
├── de.json
├── es.json
├── it.json
├── manifest.json
└── README.md
```

To regenerate them from the latest kaikki.org dump:

```bash
python scripts/update_wiktionary_dictionaries.py --all
```

Or for a single language:

```bash
python scripts/update_wiktionary_dictionaries.py --lang fr
```

The script downloads `kaikki.org-dictionary-all.jsonl`, extracts synonyms by language, filters by allowed POS tags, and writes compact per-language JSON Lines files.

## SynonymExpansionMiddleware

Integrates synonym expansion into the middleware pipeline:

```python
from whoosh_modern.linguistics.synonyms import (
    SynonymManager,
    SynonymExpansionMiddleware,
)

manager = SynonymManager({
    "car": ["automobile", "vehicle"],
    "house": ["home", "residence"],
})
middleware = SynonymExpansionMiddleware(manager)
```

The middleware expands both search queries and indexed documents:

```python
# Query expansion
ctx = MiddlewareContext(operation="search")
ctx.query = "car"
ctx = middleware.before_search(ctx)
# ctx.query == "car automobile vehicle"

# Document expansion
ctx = MiddlewareContext(operation="index")
ctx.document = {"title": "house for sale"}
ctx = middleware.before_index(ctx)
# ctx.document["title"] == "house for sale home residence"
```

## Prebuilt Language Synonyms

`LANG_SYNONYMS` provides starter dictionaries for five languages:

```python
from whoosh_modern.linguistics.synonyms import LANG_SYNONYMS

french_syns = LANG_SYNONYMS["fr"]
print(french_syns["voiture"])  # ['automobile', 'véhicule']

english_syns = LANG_SYNONYMS["en"]
print(english_syns["car"])  # ['automobile', 'vehicle']
```

| Language | Code | Sample Entry                          |
|----------|------|---------------------------------------|
| French   | `fr` | `"voiture": ["automobile", "véhicule"]` |
| English  | `en` | `"car": ["automobile", "vehicle"]`    |
| German   | `de` | `"auto": ["wagen", "fahrzeug"]`       |
| Spanish  | `es` | `"coche": ["automóvil", "vehículo"]`  |
| Italian  | `it` | `"auto": ["automobile", "veicolo"]`   |

## Integration Example

```python
from whoosh_modern.linguistics import (
    LANG_SYNONYMS,
    SynonymExpansionMiddleware,
    SynonymManager,
)
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.wrappers import MiddlewareWriter, MiddlewareSearcher

# 1. Build synonym manager
syn_manager = SynonymManager(LANG_SYNONYMS["en"])
syn_manager.add_synonyms("search", ["query", "find", "lookup"])

# 2. Create middleware
syn_middleware = SynonymExpansionMiddleware(syn_manager)

# 3. Build middleware chain
chain = MiddlewareChain([syn_middleware])

# 4. Use with writer/searcher
with MiddlewareWriter(ix.writer(), chain) as writer:
    writer.add_document(title="How to search in Whoosh")

with MiddlewareSearcher(ix.searcher(), chain) as searcher:
    results = searcher.search("search")
```

## Wiktionary Indexing Integration

`WiktionaryIndexer` can feed synonyms directly into `SynonymManager` and `SearchApplication`.

### SynonymManager.import_wiktionary_index()

Populate a manager from a built Whoosh index:

```python
from whoosh_modern.linguistics.synonyms import SynonymManager
from whoosh_modern.linguistics.wiktionary_indexer import WiktionaryIndexer

indexer = WiktionaryIndexer("indexdir")
# ... build_index() called earlier ...

manager = SynonymManager()
manager.import_wiktionary_index("indexdir", language="fr")
print(manager.get_synonyms("voiture"))
# ['automobile', 'véhicule']
```

### SearchApplication integration

Pass a `WiktionaryIndexer` to `SearchApplication` to expose a pre-populated `synonym_manager`:

```python
from whoosh_modern import SearchApplication
from whoosh_modern.linguistics.wiktionary_indexer import WiktionaryIndexer

indexer = WiktionaryIndexer("indexdir")
app = SearchApplication(wiktionary_indexer=indexer)

# synonym_manager is lazily populated from the index
manager = app.synonym_manager
```

### SynonymExpansionMiddleware wiring

Combine with the middleware to expand queries at search time:

```python
from whoosh_modern.linguistics.synonyms import SynonymExpansionMiddleware

middleware = SynonymExpansionMiddleware(app.synonym_manager)
```

## See Also

- [Linguistics Overview](linguistics.md) — Stemmers, language analyzers, and full pipeline integration
- [Middleware Pipeline](middleware-pipeline.md) — How middleware chains work
- [Stemming Providers](stemming-providers.md) — Language-specific stemmer backends


## DOCUMENT: Vector

# Vector Search

Module: `whoosh_modern.vector`
Version: 2.1.0

Whoosh-NG supports semantic search through vector embeddings. This guide covers setting up and using vector fields.

## Concept

Vector search lets you find documents based on semantic similarity rather than exact keyword matches. You embed documents and queries into a high-dimensional space, then find nearest neighbors.

```
Query embedding  ----\
                      >--- Cosine Similarity ---> Ranked results
Document embedding ---/
```

## Setup

### Define Schema

```python
from whoosh.fields import Schema, TEXT, VectorField

schema = Schema(
    title=TEXT(stored=True),
    content=TEXT,
    embedding=VectorField(dimensions=384)  # e.g., all-MiniLM-L6-v2
)
```

## Providers

Whoosh-NG includes multiple vector backends:

| Provider | Description | Use Case |
|----------|-------------|----------|
| `NumpyProvider` | Pure NumPy, cosine similarity | Small to medium indexes |
| `HNSWProvider` | Hierarchical navigable small world | Large indexes, fast ANN |
| `FaissProvider` | Facebook AI Similarity Search | Very large indexes |
| `QdrantProvider` | Qdrant vector DB | Distributed |

### NumpyProvider (Default)

```python
from whoosh.vector import NumpyProvider

provider = NumpyProvider()
provider.add_vector(doc_id, embedding)
results = provider.search(query_embedding, limit=10)
```

## Indexing with Vectors

### Generate Embeddings

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

embeddings = model.encode([
    "First document content",
    "Second document content"
])
```

### Write Documents

```python
with ix.writer() as writer:
    writer.add_document(
        title="Doc 1",
        content="Python is great",
        embedding=embeddings[0].tolist()
    )
    writer.commit()
```

## Searching with Vectors

### Hybrid Search (Keyword + Vector)

```python
from whoosh.searching import Searcher
from whoosh.vector import VectorProvider

with ix.searcher() as searcher:
    # Semantic search component
    query_embedding = model.encode(["Python tutorial"])[0]
    vector_results = searcher.vector_search(
        "embedding", query_embedding, limit=20
    )

    # Keyword search component
    keyword_query = QueryParser("content", schema).parse("Python")
    keyword_results = searcher.search(keyword_query, limit=20)

    # Combine (e.g., RRF fusion)
    final_results = fuse_results(vector_results, keyword_results)
```

### Pure Vector Search

```python
with ix.searcher() as searcher:
    query_embedding = model.encode(["search query"])[0]
    results = searcher.vector_search(
        "embedding",
        query_embedding,
        limit=10,
        metric="cosine"  # or "euclidean", "dot"
    )
```

## VectorField Options

```python
embedding_field = VectorField(
    dimensions=384,      # Required: embedding dimension
    metric="cosine",     # Similarity metric: cosine, euclidean, dot
    provider="hnsw"      # Provider name from registry
)
```

## Indexing Stream

```python
from whoosh.vector.indexing import VectorIndexer

indexer = VectorIndexer(ix)
indexer.add_document(
    title="Doc",
    content="Content",
    embedding=embedding.tolist()
)
indexer.commit()
```

## Similarity Metrics

| Metric | Description | Range |
|--------|-------------|-------|
| `cosine` | Cosine similarity | [0, 1] (higher is more similar) |
| `euclidean` | Euclidean distance | [0, inf) (lower is more similar) |
| `dot` | Dot product | [-inf, inf] (higher is more similar) |

## Best Practices

1. **Normalize embeddings**: Use cosine similarity with normalized vectors
2. **Choose provider wisely**: Numpy for &lt;100k vectors, HNSW/Faiss for larger
3. **Hybrid search**: Combine vector and keyword search for best results
4. **Cache embeddings**: Pre-compute and store to avoid recomputation
5. **Batch indexing**: Index vectors in batches for efficiency

## Vector Provider Integration in the Pipeline

The vector search system integrates through Whoosh's plugin registry and segment
format. The provider is stored in the index segment and resolved at search time.

### Architecture

```text
┌─────────────────────────────────────────────────────────────────────┐
│  Registration (startup)                                            │
│                                                                     │
│  VectorPlugin.register(PluginManager)                              │
│    └── VectorRegistry.register("numpy", NumpyProvider(), owner)     │
│                                                                     │
│  The provider is now available for any VECTOR field                │
│  that specifies provider="numpy"                                   │
└─────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────┐
│  Indexation                                                         │
│                                                                     │
│  VECTOR(dimensions=384, provider="numpy")                          │
│       │                                                             │
│       ▼                                                             │
│  PerDocWriter.add_vector_items(fieldname, field, items)            │
│       │                                                             │
│       ▼                                                             │
│  Segment file contains:                                            │
│    - vector bytes (raw)                                            │
│    - provider name ("numpy")                                        │
│    - metric ("cosine")                                              │
│       │                                                             │
│       ▼                                                             │
│  writer.commit() → segments written to disk                        │
└─────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────┐
│  Search                                                          │
│                                                                     │
│  searcher.vector_search("embedding", query_vec, k=10)              │
│       │                                                             │
│       ▼                                                             │
│  Whoosh core reads segment                                         │
│    └── retrieves provider name ("numpy")                            │
│       │                                                             │
│       ▼                                                             │
│  VectorRegistry.get("numpy")                                        │
│       │                                                             │
│       ▼                                                             │
│  NumpyProvider.search(query_vec, k, filter_ids)                    │
│       │                                                             │
│       ▼                                                             │
│  VectorHit[] sorted by cosine similarity                           │
└─────────────────────────────────────────────────────────────────────┘
```

### Full indexing flow

```python
from whoosh import index, fields
from whoosh_modern.vector.plugin import VectorPlugin
from whoosh.plugins.manager import PluginManager
from whoosh_modern.storage import FileStorage
import numpy as np

# 1. Register vector plugin (startup)
manager = PluginManager()
VectorPlugin().register(manager)

# 2. Define schema with VECTOR field
schema = fields.Schema(
    title=fields.TEXT(stored=True),
    embedding=fields.VECTOR(dimensions=384, provider="numpy", stored=True),
)

# 3. Create index (storage determines where segments live)
ix = index.create_in("indexdir", schema)

# 4. Index documents with vectors
np.random.seed(42)
embeddings = {
    "doc1": np.random.rand(384).astype(np.float32).tolist(),
    "doc2": np.random.rand(384).astype(np.float32).tolist(),
}

with ix.writer() as writer:
    for doc_id, vec in embeddings.items():
        writer.add_document(
            title=f"Document {doc_id}",
            embedding=vec,
        )
    writer.commit()
    # Whoosh core serializes vectors to segment files
    # Provider name "numpy" is stored in the segment
```

### Full search flow

```python
from whoosh.qparser import QueryParser
import numpy as np

with ix.searcher() as searcher:
    # 1. Keyword search
    qp = QueryParser("title", schema)
    keyword_results = searcher.search(qp.parse("Document"))

    # 2. Vector search
    query_vec = np.random.rand(384).astype(np.float32).tolist()
    vector_results = searcher.vector_search(
        "embedding",
        query_vec,
        limit=10,
    )

    # 3. Hybrid: combine both
    # Example: Reciprocal Rank Fusion (RRF)
    def rrf(results_list, k=60):
        scores = {}
        for results in results_list:
            for rank, hit in enumerate(results):
                doc_id = hit.doc_id if hasattr(hit, "doc_id") else hit["doc_id"]
                scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
        return sorted(scores.items(), key=lambda x: x[1], reverse=True)

    combined = rrf([keyword_results, vector_results])
```

### Standalone usage (no schema)

```python
from whoosh_modern.vector import NumpyProvider

# Create provider directly
provider = NumpyProvider()

# Add vectors
provider.add([
    ("doc1", [0.1, 0.2, 0.3]),
    ("doc2", [0.4, 0.5, 0.6]),
])

# Search
query_vec = [0.1, 0.2, 0.3]
hits = provider.search(query_vec, k=5)

for hit in hits:
    print(f"doc_id={hit.doc_id}, score={hit.score:.4f}")
```

### Provider resolution chain

When `searcher.vector_search()` is called, Whoosh core:

1. Reads the `VECTOR` field configuration from the schema
2. Opens the segment file containing the vector data
3. Extracts the provider name stored in the segment (e.g., `"numpy"`)
4. Looks up the provider in `VectorRegistry`
5. Calls `provider.search(query_vector, k, filter_ids)`
6. Returns `list[VectorHit]`

If the provider is not registered, the search fails with a registry miss. This
is why `VectorPlugin().register(manager)` (or manual registration) is required
at startup.

## See Also

- [Provider Integration Guide](provider-integration.md) — Complete pipeline guide for all providers
- [Middleware Guide](middleware-pipeline.md) — Pipeline hooks and provider adapters


## DOCUMENT (FR): Analysis

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Analysis API

Classes and functions for turning text into indexable "tokens" (usually words).
Analysis is the first step in the indexing pipeline: an analyzer tokenizes text
and applies zero or more filters to the resulting token stream.

## Overview

Three general categories of objects make up the analysis pipeline:

- **Tokenizers** split text into individual tokens (words, n-grams, identifiers).
  Every tokenizer is callable: `tokenizer(text) -> iterator of Token objects`.
- **Filters** transform one token stream into another. Common operations include
  lowercasing, stop-word removal, stemming, and synonym expansion. Every filter
  is callable: `filter(token_generator) -> token_generator`.
- **Analyzers** compose a tokenizer and zero or more filters into a single unit.
  Every analyzer is callable and can be used directly as a field's `analyzer`
  argument.

Tokenizers and filters are combined using the `|` operator:

```python
my_analyzer = RegexTokenizer() | LowercaseFilter() | StopFilter()
```

The first item must be a tokenizer; subsequent items must be filters.

## Composition

### Composable

```python
class whoosh.analysis.Composable
```

Base class for tokenizers and filters, providing `|` composition.

**Attributes:**
- `is_morph (bool)`: Whether this object performs morphological transformation
  (e.g. stemming). Defaults to `False`.

**Methods:**

#### `__or__(self, other)`

Combines this object with `other` using `CompositeAnalyzer`.

```python
analyzer = RegexTokenizer() | LowercaseFilter() | StopFilter()
```

### CompositeAnalyzer

```python
class whoosh.analysis.CompositeAnalyzer
```

Composed analyzer created by chaining a tokenizer and filters with `|`.

**Example:**
```python
from whoosh.analysis import RegexTokenizer, LowercaseFilter, StopFilter

analyzer = RegexTokenizer() | LowercaseFilter() | StopFilter()
tokens = list(analyzer("Hello world, this is a test"))
```

## Token

```python
class whoosh.analysis.Token
```

Represents a single token (usually a word) extracted from source text.
Tokenizers yield the **same** `Token` object repeatedly (for performance), so
consumers must not hold references between iterations.

**Slots:**

| Attribute | Type | Description |
|-----------|------|-------------|
| `text` | `str` | The text of this token |
| `pos` | `int` | Token position (if `positions=True`) |
| `startchar` | `int` | Start character offset (if `chars=True`) |
| `endchar` | `int` | End character offset (if `chars=True`) |
| `original` | `str` | Original text before filters (if `keeporiginal=True`) |
| `positions` | `bool` | Whether position info was requested |
| `chars` | `bool` | Whether character offsets were requested |
| `stopped` | `bool` | Set by `StopFilter` |
| `boost` | `float` | Token boost factor (default `1.0`) |
| `removestops` | `bool` | Whether stop words should be removed |
| `mode` | `str` | `'index'` or `'query'` |
| `boosts` | `dict` | Per-position boost values (if requested) |
| `tokenize` | `bool` | Whether tokenization should proceed |
| `matched` | `bool` | Used during highlighting |
| `fieldname` | `str` | Field name for this token |

**Methods:**

#### `copy()`

Returns a new `Token` with the same attribute values. Use this if you need to
retain a token between iterations.

```python
def remove_duplicates(stream):
    last = None
    for t in stream:
        if last != t.text:
            yield t
        last = t.text
```

## Utility Functions

### entoken

```python
whoosh.analysis.entoken(
    textstream,
    positions=False,
    chars=False,
    start_pos=0,
    start_char=0,
    **kwargs
) -> Iterator[Token]
```

Converts a sequence of strings into a stream of `Token` objects.

### unstopped

```python
whoosh.analysis.unstopped(tokenstream) -> Iterator[Token]
```

Removes tokens where `token.stopped` is `True`.

## Analyzers

### Analyzer (Base)

```python
class whoosh.analysis.Analyzer
```

Abstract base class for all analyzers. Subclasses implement `__call__`.

### CompositeAnalyzer

Created automatically when you use `|` to compose tokenizers and filters.

### Predefined Analyzers

#### IDAnalyzer

```python
whoosh.analysis.IDAnalyzer(lowercase=False) -> Analyzer
```

Yields the entire input as a single token. Deprecated; use `IDTokenizer` directly.

- `lowercase (bool)`: If True, add a `LowercaseFilter`.

#### KeywordAnalyzer

```python
whoosh.analysis.KeywordAnalyzer(
    lowercase=False,
    commas=False
) -> Analyzer
```

Splits on whitespace or commas. Suitable for field values that are lists of
keywords.

- `lowercase (bool)`: Lowercase each token.
- `commas (bool)`: Split on commas instead of whitespace.

**Example:**
```python
from whoosh.analysis import KeywordAnalyzer

an = KeywordAnalyzer(lowercase=True, commas=True)
list(an("Hello, WORLD, test"))
# => ["hello", "world", "test"]
```

#### RegexAnalyzer

```python
whoosh.analysis.RegexAnalyzer(
    expression=r"\w+(\.?\w+)*",
    gaps=False
) -> Analyzer
```

Deprecated; use `RegexTokenizer` directly.

#### SimpleAnalyzer

```python
whoosh.analysis.SimpleAnalyzer(
    expression=default_pattern,
    gaps=False
) -> Analyzer
```

Composes `RegexTokenizer` with `LowercaseFilter`.

- `expression`: Regex pattern for tokens.
- `gaps`: If True, split on the expression instead of matching it.

**Example:**
```python
an = SimpleAnalyzer()
list(an("Hello there, this is a TEST"))
# => ["hello", "there", "this", "is", "a", "test"]
```

#### StandardAnalyzer

```python
whoosh.analysis.StandardAnalyzer(
    expression=default_pattern,
    stoplist=STOP_WORDS,
    minsize=2,
    maxsize=None,
    gaps=False
) -> Analyzer
```

Composes `RegexTokenizer`, `LowercaseFilter`, and optional `StopFilter`.

- `expression`: Regex pattern for tokens.
- `stoplist`: Words to remove (set to `None` to disable).
- `minsize`: Minimum token length (default `2`).
- `maxsize`: Maximum token length (default `None`, no limit).
- `gaps`: If True, split on the expression instead of matching it.

**Example:**
```python
an = StandardAnalyzer()
list(an("Testing is testing and testing"))
# => ["testing", "testing", "testing"]
```

#### StemmingAnalyzer

```python
whoosh.analysis.StemmingAnalyzer(
    expression=default_pattern,
    stoplist=STOP_WORDS,
    minsize=2,
    maxsize=None,
    gaps=False,
    stemfn=stem,
    ignore=None,
    cachesize=50000
) -> Analyzer
```

Composes `RegexTokenizer`, `LowercaseFilter`, optional `StopFilter`, and
`StemFilter`.

- `expression`: Regex pattern for tokens.
- `stoplist`: Words to remove (set to `None` to disable).
- `minsize`: Minimum token length (default `2`).
- `maxsize`: Maximum token length.
- `gaps`: If True, split on the expression instead of matching it.
- `stemfn`: Stemming function (default: Porter stemmer for English).
- `ignore`: Words to not stem (set).
- `cachesize`: Stem cache size (default `50000`). Use `-1` for unbounded,
  `None` for no cache.

**Example:**
```python
an = StemmingAnalyzer()
list(an("Testing is testing and testing"))
# => ["test", "test", "test"]
```

#### FancyAnalyzer

```python
whoosh.analysis.FancyAnalyzer(
    expression=r"\s+",
    stoplist=STOP_WORDS,
    minsize=2,
    gaps=True,
    splitwords=True,
    splitnums=True,
    mergewords=False,
    mergenums=False
) -> Analyzer
```

Composes `RegexTokenizer`, `IntraWordFilter`, `LowercaseFilter`, and `StopFilter`.
Splits on whitespace and breaks compound words into subwords.

**Example:**
```python
an = FancyAnalyzer()
list(an("Should I call getInt or get_real?"))
# => ["should", "call", "get", "int", "get", "real"]
```

#### LanguageAnalyzer

```python
whoosh.analysis.LanguageAnalyzer(
    lang,
    expression=default_pattern,
    gaps=False,
    cachesize=50000
) -> Analyzer
```

Configures a language-specific analyzer with `LowercaseFilter`, `StopFilter`,
and `StemFilter`.

- `lang`: Language code (e.g., `"en"`, `"es"`, `"fr"`).
- `expression`: Regex pattern for tokens.
- `gaps`: If True, split on the expression instead of matching it.
- `cachesize`: Stem cache size.

Available languages: `ar`, `da`, `nl`, `en`, `fi`, `fr`, `de`, `hu`, `it`,
`no`, `pt`, `ro`, `ru`, `es`, `sv`, `tr`.

See `whoosh.lang` for `has_stemmer()` and `has_stopwords()` helper functions.

## Tokenizers

All tokenizers inherit from `Tokenizer`.

### Tokenizer

```python
class whoosh.analysis.Tokenizer
```

Base class for tokenizers. Each tokenizer is callable and yields `Token`
objects.

### RegexTokenizer

```python
class whoosh.analysis.RegexTokenizer(
    expression=default_pattern,
    gaps=False
)
```

Uses a regular expression to extract tokens from text. Each match of the
expression equals one token; group 0 (the entire match) is used as the text.

- `expression`: Compiled regex or pattern string.
- `gaps`: If True, split on the expression rather than matching it.

**Example:**
```python
from whoosh.analysis import RegexTokenizer

rext = RegexTokenizer()
list(rext("hi there 3.141 big-time under_score"))
# => ["hi", "there", "3.141", "big", "time", "under_score"]
```

### IDTokenizer

```python
class whoosh.analysis.IDTokenizer
```

Yields the entire input string as a single token. Used for indexed but
untokenized fields (e.g., document paths).

### CharsetTokenizer

```python
class whoosh.analysis.CharsetTokenizer(charmap)
```

Tokenizes and translates text according to a character mapping dictionary.
Characters that map to `None` are treated as token break characters.

- `charmap`: Mapping from integer character codes to unicode characters
  (as used by `unicode.translate()`).

### PathTokenizer

```python
class whoosh.analysis.PathTokenizer(expression="[^/]+")
```

Tokenizes path strings into hierarchical prefixes. Given `"/a/b/c"`, yields
`["/a", "/a/b", "/a/b/c"]`.

### NgramTokenizer

```python
class whoosh.analysis.NgramTokenizer(minsize, maxsize=None)
```

Splits input text into N-grams instead of words. Unlike `RegexTokenizer`, this
tokenizer does not use a regex, so grams may include whitespace and punctuation.

- `minsize`: Minimum N-gram size.
- `maxsize`: Maximum N-gram size (defaults to `minsize`).

**Example:**
```python
from whoosh.analysis import NgramTokenizer

ngt = NgramTokenizer(4)
list(ngt("hi there"))
# => ["hi t", "i th", " the", "ther", "here"]
```

### CachedRegexTokenizer

```python
class whoosh.analysis.CachedRegexTokenizer(
    expression=default_pattern,
    gaps=False,
    maxsize=8192
)
```

A `RegexTokenizer` wrapper that caches tokenization results for repeated
strings, trading memory for speed.

- `expression`: Regex pattern.
- `gaps`: If True, split on the expression.
- `maxsize`: Maximum cache size (LRU eviction when exceeded).

### SpaceSeparatedTokenizer

```python
whoosh.analysis.SpaceSeparatedTokenizer() -> RegexTokenizer
```

Returns a `RegexTokenizer` that splits on whitespace.

### CommaSeparatedTokenizer

```python
whoosh.analysis.CommaSeparatedTokenizer() -> CompositeAnalyzer
```

Returns a composed analyzer that splits on commas and strips whitespace.

## Filters

All filters inherit from `Filter`.

### Filter

```python
class whoosh.analysis.Filter
```

Base class for filters. Subclasses implement `__call__(self, tokens)` which
takes a token generator and returns a token generator.

- `is_morph (bool)`: Set to `True` for morphological filters (e.g., stemming).
  This allows the filter to be bypassed during query analysis if desired.

### STOP_WORDS

```python
whoosh.analysis.STOP_WORDS
```

A frozenset of common English stop words: `"a"`, `"an"`, `"and"`, `"the"`, etc.
Used as the default stoplist for `StopFilter` and `StandardAnalyzer`.

### url_pattern

```python
whoosh.analysis.url_pattern
```

A compiled regex useful for URL filtering.

### LowercaseFilter

```python
class whoosh.analysis.LowercaseFilter
```

Lowercases token text using `unicode.lower()`.

**Example:**
```python
rext = RegexTokenizer() | LowercaseFilter()
list(rext("This is a TEST"))
# => ["this", "is", "a", "test"]
```

### StopFilter

```python
class whoosh.analysis.StopFilter(
    stoplist=STOP_WORDS,
    minsize=2,
    maxsize=None,
    renumber=True,
    lang=None
)
```

Marks and optionally removes stop words from the token stream.

- `stoplist`: Set of words to filter out (defaults to `STOP_WORDS`).
- `minsize`: Minimum token length; shorter tokens are removed (default `2`).
- `maxsize`: Maximum token length; longer tokens are removed (default `None`).
- `renumber`: Renumber positions to account for removed tokens (default `True`).
- `lang`: If set, loads stop words for the given language code.

**Example:**
```python
from whoosh.analysis import RegexTokenizer, StopFilter

stopper = RegexTokenizer() | StopFilter()
list(stopper("this is a test"))
# => ["test"]
```

### StripFilter

```python
class whoosh.analysis.StripFilter
```

Calls `unicode.strip()` on each token's text.

### CharsetFilter

```python
class whoosh.analysis.CharsetFilter(charmap)
```

Translates token text using `unicode.translate()` with the given character map.
Useful for case folding and accent folding.

- `charmap`: Dictionary mapping character ordinals to unicode characters.

**Example:**
```python
from whoosh.support.charset import accent_map

rext = RegexTokenizer() | CharsetFilter(accent_map)
list(rext("café"))
# => ["cafe"]
```

### DelimitedAttributeFilter

```python
class whoosh.analysis.DelimitedAttributeFilter(
    delimiter="^",
    attribute="boost",
    default=1.0,
    type=float
)
```

Looks for delimiter characters in token text and extracts data after the
delimiter into a named token attribute.

- `delimiter`: Separator character (default `"^"`).
- `attribute`: Attribute name on the token (default `"boost"`).
- `default`: Default value if no delimiter is found (default `1.0`).
- `type`: Type to cast the extracted value (default `float`).

**Example:**
```python
from whoosh.analysis import RegexTokenizer, DelimitedAttributeFilter

daf = DelimitedAttributeFilter()
an = RegexTokenizer(r"\S+") | daf
for t in an(u"image 3.14^2 render"):
    print(t.text, t.boost)
# image 1.0
# 3.14 2.0
# render 1.0
```

### SubstitutionFilter

```python
class whoosh.analysis.SubstitutionFilter(pattern, replacement)
```

Performs regex substitution on token text using `re.sub()`.

- `pattern`: Pattern string or compiled regex.
- `replacement`: Replacement text.

**Example:**
```python
from whoosh.analysis import RegexTokenizer, SubstitutionFilter

# Remove hyphens
ana = RegexTokenizer(r"\S+") | SubstitutionFilter("-", "")
```

### MultiFilter

```python
class whoosh.analysis.MultiFilter(**kwargs)
```

Selects between two or more sub-filters based on the `mode` attribute of the
token stream. Useful for using different filters during indexing vs. querying.

- Keyword arguments map mode names to filter instances.

**Example:**
```python
from whoosh.analysis import MultiFilter, IntraWordFilter

iwf_index = IntraWordFilter(mergewords=True, mergenums=True)
iwf_query = IntraWordFilter(mergewords=False, mergenums=False)
mf = MultiFilter(index=iwf_index, query=iwf_query)
```

### TeeFilter

```python
class whoosh.analysis.TeeFilter(*filters)
```

Interleaves the results of two or more filter chains. Requires at least two
filters. Note: this filter is slow because it creates token copies.

**Example:**
```python
# Lowercase in one branch, reverse in another
f1 = LowercaseFilter()
f2 = ReverseTextFilter()
ana = RegexTokenizer(r"\S+") | TeeFilter(f1, f2)
```

### ReverseTextFilter

```python
class whoosh.analysis.ReverseTextFilter
```

Reverses the text of each token.

**Example:**
```python
an = RegexTokenizer() | ReverseTextFilter()
list(an("hello there"))
# => ["olleh", "ereht"]
```

### PassFilter

```python
class whoosh.analysis.PassFilter
```

Identity filter; passes tokens through unchanged.

### LoggingFilter

```python
class whoosh.analysis.LoggingFilter(logger=None)
```

Prints debug log entries for every token that passes through.

- `logger`: Logger instance (defaults to `whoosh.analysis` logger).

## Intraword Filters

### IntraWordFilter

```python
class whoosh.analysis.IntraWordFilter(
    delims="-_'\"()!@#$%^&*[]{}<>\\|;:,./?`~+=",
    splitwords=True,
    splitnums=True,
    mergewords=False,
    mergenums=False
)
```

Splits words into subwords and performs optional merging. Based on
WordDelimiterFilter in Solr.

- `delims`: String of delimiter characters.
- `splitwords`: Split at case transitions (e.g., `PowerShot` → `Power`, `Shot`).
- `splitnums`: Split at letter-number transitions (e.g., `SD500` → `SD`, `500`).
- `mergewords`: Merge consecutive alphabetic subwords.
- `mergenums`: Merge consecutive numeric subwords.

### CompoundWordFilter

```python
class whoosh.analysis.CompoundWordFilter(wordset, keep_compound=True)
```

Breaks compound tokens into their constituent parts if they match words in the
given wordset. Useful for agglutinative languages and trademarks.

- `wordset`: A set (or any `__contains__` object) of known words.
- `keep_compound`: If True, keep the original compound token in the stream.

### BiWordFilter

```python
class whoosh.analysis.BiWordFilter(sep="-")
```

Merges adjacent tokens into bigram tokens. Useful for pseudo-phrase searching.

- `sep`: Separator string for bigrams.

### ShingleFilter

```python
class whoosh.analysis.ShingleFilter(size=2, sep="-")
```

Merges N adjacent tokens into multi-word tokens (shingles).

- `size`: Number of tokens to combine.
- `sep`: Separator string.

**Note:** For `size=2`, `BiWordFilter` is faster.

## Morphological Filters

### StemFilter

```python
class whoosh.analysis.StemFilter(
    stemfn=stem,
    lang=None,
    ignore=None,
    cachesize=50000
)
```

Stems tokens using the Porter stemming algorithm (or a language-specific
stemmer if `lang` is specified).

- `stemfn`: Stemming function (default: Porter stemmer).
- `lang`: Language code to override `stemfn` with a Snowball stemmer.
- `ignore`: Set of words to not stem (defaults to stemming all words).
- `cachesize`: Cache size for stemmed words. Use `-1` for unbounded,
  `None` for no cache.

**Example:**
```python
from whoosh.analysis import RegexTokenizer, StemFilter

stemmer = RegexTokenizer() | StemFilter()
list(stemmer("fundamentally willows"))
# => ["fundament", "willow"]
```

### PyStemmerFilter

```python
class whoosh.analysis.PyStemmerFilter(
    lang="english",
    ignore=None,
    cachesize=10000
)
```

Subclass of `StemFilter` that uses the third-party `py-stemmer` library.
Requires the py-stemmer package to be installed.

**Methods:**
- `algorithms()`: Returns available stemming algorithms from py-stemmer.

### DoubleMetaphoneFilter

```python
class whoosh.analysis.DoubleMetaphoneFilter(
    primary_boost=1.0,
    secondary_boost=0.5,
    combine=False
)
```

Encodes tokens using Lawrence Philips's Double Metaphone algorithm. Useful
for phonetic matching of names and places.

- `primary_boost`: Boost factor for the primary code token.
- `secondary_boost`: Boost factor for the secondary code token.
- `combine`: If True, keep the original token alongside the encoded tokens.

## N-gram Filters and Analyzers

### NgramFilter

```python
class whoosh.analysis.NgramFilter(minsize, maxsize=None, at=None)
```

Splits token text into N-grams of varying sizes.

- `minsize`: Minimum N-gram size.
- `maxsize`: Maximum N-gram size (defaults to `minsize`).
- `at`: `'start'` for prefix grams, `'end'` for suffix grams, or `None`
  for all position grams.

### NgramAnalyzer

```python
whoosh.analysis.NgramAnalyzer(minsize, maxsize=None) -> Analyzer
```

Composes `NgramTokenizer` with `LowercaseFilter`.

### NgramWordAnalyzer

```python
whoosh.analysis.NgramWordAnalyzer(
    minsize,
    maxsize=None,
    tokenizer=None,
    at=None
) -> Analyzer
```

Composes `RegexTokenizer`, `LowercaseFilter`, and `NgramFilter`. Use this
when you want sub-word n-grams (without whitespace) rather than raw
character n-grams.


## DOCUMENT (FR): Automata

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Automata API

Module for constructing and manipulating finite state automata (FSAs),
including NFAs, DFAs, finite state transducers (FSTs), Levenshtein
automata, and regular expression automata. Used internally for spelling
correction, fuzzy term queries, and term dictionary operations.

The automata module is a refactored package with submodules. All classes
and functions are importable directly from `whoosh.automata`.

## Module Functions

### `parse_glob`

```python
whoosh.automata.parse_glob(pattern, _glob_multi="*", _glob_single="?", _glob_range1="[", _glob_range2="]") -> NFA
```

Parses a glob-style pattern string and returns an NFA that matches strings
matching the pattern.

**Parameters:**
- `pattern`: Glob pattern string (`*` matches any sequence, `?` matches any
  single character).
- `_glob_multi`, `_glob_single`: Override the wildcard characters.
- `_glob_range1`, `_glob_range2`: Override the range syntax brackets.

### `glob_automaton`

```python
whoosh.automata.glob_automaton(pattern) -> NFA
```

Convenience function that parses a glob pattern and returns an NFA.

## FSA (Finite State Automaton) Classes

### `FSA`

```python
class whoosh.automata.FSA(initial)
```

Base class for finite state automata.

**Constructor:**
- `initial`: The initial state.

**Attributes:**
- `initial`: Initial state.
- `transitions`: Dict mapping source states to dicts mapping labels to
  target states.
- `final_states`: Set of accepting (final) states.

**Methods:**
- `__eq__(other)`: Compares initial state, final states, and transitions.
- `all_states()`: Returns a set of all states reachable in the automaton.
- `all_labels()`: Returns a set of all transition labels.
- `get_labels(src)`: Yields all labels leaving state `src`.
- `generate_all(state=None, sofar="")`: Yields all strings accepted by the
  automaton.
- `move(state, label)`: Returns the state reached by following `label` from
  `state`, or `None`.
- `moves(state, labels)`: Yields `(label, next_state)` pairs.
- `next(state)`: Yields target states reachable from `state` via any label.
- `is_final(state)`: Returns `True` if `state` is a final state.
- `start()`: Returns the initial state.
- `has_path_to(target)`: Returns `True` if there is a path to `target`.

### `Marker`

```python
class whoosh.automata.Marker(name)
```

Marker object used as a special transition label in NFAs (e.g., `ANY`,
`EPSILON`).

### `EPSILON`

```python
whoosh.automata.EPSILON = Marker("EPSILON")
```

Special marker representing an epsilon transition (no input consumed).

### `ANY`

```python
whoosh.automata.ANY = Marker("ANY")
```

Special marker representing a transition that matches any input character.

### `NFA`

```python
class whoosh.automata.NFA(initial)
```

Nondeterministic Finite Automaton. Extends `FSA` with epsilon transitions
and NFA-specific construction methods.

**Methods:**
- `add_transition(src, label, dst)`: Adds a transition from `src` to `dst`
  consuming `label`.
- `add_final_state(state, final=True)`: Marks `state` as a final/accepting
  state.
- `epsilon_closure(state)`: Returns the set of states reachable from `state`
  via epsilon transitions.
- `to_dfa()`: Converts this NFA to an equivalent DFA and returns it.

### `DFA`

```python
class whoosh.automata.DFA(initial)
```

Deterministic Finite Automaton. Extends `FSA` with DFA-specific operations.

**Methods:**
- `next_valid_string(string)`: Finds the lexicographically smallest string
  accepted by the DFA that is greater than or equal to `string`.
- `to_dfa()`: Returns self (already a DFA).

### `renumber_dfa`

```python
whoosh.automata.renumber_dfa(dfa, base=0) -> DFA
```

Renumerates the states of a DFA to integers starting at `base`.

### `u_to_utf8`

```python
whoosh.automata.u_to_utf8(dfa, base=0) -> DFA
```

Converts a Unicode DFA to a UTF-8 DFA.

### `find_all_matches`

```python
whoosh.automata.find_all_matches(dfa, lookup_func, first=unull)
```

Yields all strings accepted by the DFA, using `lookup_func` to determine
which strings exist in the dictionary.

**Parameters:**
- `dfa`: A deterministic finite automaton.
- `lookup_func`: Function called with each candidate string; returns the
  string if found in the dictionary.
- `first`: First string to start matching from (default `chr(0)`).

### `reverse_nfa`

```python
whoosh.automata.reverse_nfa(n) -> NFA
```

Returns the reverse of an NFA (reversed transitions, swapped initial
and final states).

### `product`

```python
whoosh.automata.product(dfa1, op, dfa2) -> DFA
```

Computes the product of two DFAs using a binary operation.

**Parameters:**
- `dfa1`, `dfa2`: Input DFAs.
- `op`: A function `(set1, set2) -> set` computing the output final states
  from the two input final state sets.

### `intersection`

```python
whoosh.automata.intersection(dfa1, dfa2) -> DFA
```

Returns the intersection of two DFAs.

### `union`

```python
whoosh.automata.union(dfa1, dfa2) -> DFA
```

Returns the union of two DFAs.

### `epsilon_nfa`

```python
whoosh.automata.epsilon_nfa() -> NFA
```

Returns an NFA that accepts only the empty string.

### `dot_nfa`

```python
whoosh.automata.dot_nfa() -> NFA
```

Returns an NFA that accepts any single character.

### `basic_nfa`

```python
whoosh.automata.basic_nfa(label) -> NFA
```

Returns an NFA that accepts exactly the string `label`.

### `charset_nfa`

```python
whoosh.automata.charset_nfa(labels) -> NFA
```

Returns an NFA that accepts any single character in `labels`.

### `string_nfa`

```python
whoosh.automata.string_nfa(string) -> NFA
```

Returns an NFA that accepts exactly `string`.

### `choice_nfa`

```python
whoosh.automata.choice_nfa(n1, n2) -> NFA
```

Returns an NFA that accepts strings accepted by either `n1` or `n2`.

### `concat_nfa`

```python
whoosh.automata.concat_nfa(n1, n2) -> NFA
```

Returns an NFA that accepts the concatenation of `n1` and `n2`.

### `star_nfa`

```python
whoosh.automata.star_nfa(n) -> NFA
```

Returns an NFA that accepts zero or more repetitions of `n`.

### `plus_nfa`

```python
whoosh.automata.plus_nfa(n) -> NFA
```

Returns an NFA that accepts one or more repetitions of `n`.

### `optional_nfa`

```python
whoosh.automata.optional_nfa(n) -> NFA
```

Returns an NFA that accepts zero or one occurrence of `n`.

### `strings_dfa`

```python
whoosh.automata.strings_dfa(strings) -> DFA
```

Constructs a minimal DFA that accepts exactly the given strings.

### `add_suffix`

```python
whoosh.automata.add_suffix(dfa, nodes, last, downto, seen)
```

Internal function for adding suffixes to a trie during DFA construction.

## Levenshtein Automata

### `levenshtein_automaton`

```python
whoosh.automata.levenshtein_automaton(term, k, prefix=0) -> NFA
```

Constructs an NFA that matches all strings within edit distance `k` of
`term`. This is the core function for fuzzy term queries and spelling
suggestions.

**Parameters:**
- `term`: The reference string to compute edit distance from.
- `k`: Maximum edit distance (number of insertions, deletions, or
  substitutions).
- `prefix`: If positive, require matched strings to share this length of
  prefix with `term` (speeds up matching significantly).

**Returns:** An NFA that can be converted to a DFA via `.to_dfa()`.

```python
from whoosh.automata import levenshtein_automaton

nfa = levenshtein_automaton("hello", k=1, prefix=0)
dfa = nfa.to_dfa()
```

## RegEx

### `parse`

```python
whoosh.automata.parse(pattern) -> NFA
```

Parses a regular expression pattern string and returns an NFA.

**Parameters:**
- `pattern`: A regex pattern string (Python `re`-style syntax).

### `RegexBuilder`

```python
class whoosh.automata.RegexBuilder(pattern)
```

Helper class for building NFAs from regex patterns.

## FST (Finite State Transducer) Classes

### `Values`

```python
class whoosh.automata.Values
```

Abstract base class for value types stored in FST arcs.

### `IntValues`

```python
class whoosh.automata.IntValues
```

Stores integer values in FST arcs.

### `SequenceValues`

```python
class whoosh.automata.SequenceValues
```

Base class for value types that store sequences of values.

### `BytesValues`

```python
class whoosh.automata.BytesValues
```

Stores byte string values in FST arcs.

### `ArrayValues`

```python
class whoosh.automata.ArrayValues
```

Stores arrays of values in FST arcs.

### `IntListValues`

```python
class whoosh.automata.IntListValues
```

Stores lists of integers in FST arcs.

### `Node`

```python
class whoosh.automata.Node
```

Base class for nodes in an FST.

### `ComboNode`

```python
class whoosh.automata.ComboNode
```

Base class for nodes that combine multiple sub-nodes (intersection, union).

### `UnionNode`

```python
class whoosh.automata.UnionNode
```

A node that represents the union of multiple sub-nodes.

### `IntersectionNode`

```python
class whoosh.automata.IntersectionNode
```

A node that represents the intersection of multiple sub-nodes.

### `BaseCursor`

```python
class whoosh.automata.BaseCursor
```

Base class for cursors that iterate over FST contents.

### `Cursor`

```python
class whoosh.automata.Cursor
```

Concrete cursor for iterating over an FST, supporting `next()`, `find()`,
`text()`, and other navigation methods.

### `UncompiledNode`

```python
class whoosh.automata.UncompiledNode
```

Represents an FST node that has not yet been compiled into a binary
representation. Used during FST construction.

### `Arc`

```python
class whoosh.automata.Arc
```

Represents a single arc in an FST, with a label, target node, and associated
value.

### `GraphWriter`

```python
class whoosh.automata.GraphWriter
```

Writes an FST to a binary file on disk or to an in-memory buffer.

### `BaseGraphReader`

```python
class whoosh.automata.BaseGraphReader
```

Base class for reading FSTs from disk.

### `GraphReader`

```python
class whoosh.automata.GraphReader
```

Concrete reader for FSTs stored on disk. Supports `find()`, `next()`, and
`text()` for navigating the graph.

### `to_labels`

```python
whoosh.automata.to_labels(key)
```

Converts a key (string, int, etc.) into a list of FST arc labels.

### `within`

```python
whoosh.automata.within(graph, text, k=1, prefix=0, address=None)
```

Uses a pre-built FST and a Levenshtein automaton to find all keys in the
graph within edit distance `k` of `text`.

**Parameters:**
- `graph`: A `GraphReader` instance.
- `text`: The search term.
- `k`: Maximum edit distance.
- `prefix`: Required shared prefix length.
- `address`: Optional starting address in the graph.

### `dump_graph`

```python
whoosh.automata.dump_graph(graph, address=None, tab=0, out=None)
```

Debug utility that prints the structure of an FST to stdout or a file.

### `FileVersionError`

```python
class whoosh.automata.FileVersionError
```

Raised when reading an FST file with an incompatible version.

### `InactiveCursor`

```python
class whoosh.automata.InactiveCursor
```

Raised when operating on a cursor that is not at a valid position.


## DOCUMENT (FR): Backends

# API Backends

Architecture de stockage pliable via backends.

## FileBackend (défaut)

```python
class whoosh.backends.file.FileBackend
```

Backend par défaut stockant les segments comme fichiers sur disque.

### Options

| Paramètre | Description |
|-----------|-------------|
| `storage` | Instance de storage (FileStorage par défaut) |
| `limitmb` | Taille maximum des segments (MiB) |

**Exemple:**
```python
from whoosh import index

ix = index.create_in("indexdir", schema)
# Utilise FileBackend implicitement
```

## SQLiteBackend

```python
class whoosh.backends.sqlite.SQLiteBackend
```

Stocke l'index entier dans une base de données SQLite.

### Options

| Paramètre | Description |
|-----------|-------------|
| `storage` | `SQLiteStorage(path)` |
| `writethrough` | Écriture synchrone |

**Exemple:**
```python
from whoosh.backends.sqlite import SQLiteStorage, SQLiteBackend

storage = SQLiteStorage("mon_index.db")
backend = SQLiteBackend(storage=storage)

ix = backend.create_index(schema)
```

## MemoryBackend

```python
class whoosh.backends.memory.MemoryBackend
```

Backend en mémoire (tests uniquement, données perdues au redémarrage).

## Classes Storage

### FileStorage

```python
class whoosh.store.FileStorage
```

Gère les fichiers sur disque.

### SQLiteStorage

```python
class whoosh.store.SQLiteStorage(db_path)
```

Gère le stockage SQLite.

## ProviderRegistry

```python
class whoosh.registry.ProviderRegistry
```

Registre pour les providers de stockage:

```python
from whoosh.registry import ProviderRegistry

ProviderRegistry.register("sqlite", SQLiteBackend(), "mon_app")
```


## DOCUMENT (FR): Classify

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Classify API

Classes and functions for classifying and extracting information from
documents. This module provides query expansion models, similarity
functions (shingling, simhash), and clustering algorithms.

## Expansion Models

### `ExpansionModel`

```python
class whoosh.classify.ExpansionModel(doc_count, field_length)
```

Abstract base class for query expansion models. Subclass to implement custom
expansion scoring.

**Constructor:**
- `doc_count`: Total number of documents in the collection.
- `field_length`: Total length of the field across all documents.

**Computed Attributes:**
- `N`: Document count.
- `collection_total`: Total field length.
- `mean_length`: Average field length (`collection_total / N`).

**Methods:**
- `normalizer(maxweight, top_total)`: Returns a normalization factor.
- `score(weight_in_top, weight_in_collection, top_total)`: Returns the
  expansion score for a term.

### `Bo1Model`

```python
class whoosh.classify.Bo1Model(doc_count, field_length)
```

Bayesian One-Poisson expansion model. One of the standard query expansion
models.

### `Bo2Model`

```python
class whoosh.classify.Bo2Model(doc_count, field_length)
```

Bayesian Two-Poisson expansion model. Another standard query expansion model.

### `KLModel`

```python
class whoosh.classify.KLModel(doc_count, field_length)
```

Kullback-Leibler divergence-based expansion model.

## Expander

### `Expander`

```python
class whoosh.classify.Expander(
    ixreader,
    fieldname,
    model=Bo1Model
)
```

Uses an `ExpansionModel` to expand the set of query terms based on the top N
result documents.

**Constructor:**
- `ixreader`: An `IndexReader` object.
- `fieldname`: The name of the field to expand terms from.
- `model`: An `ExpansionModel` class or instance. Defaults to `Bo1Model`.

**Methods:**

#### `add(vector)`

Adds forward-index information about one of the "top N" documents.

- `vector`: A series of `(text, weight)` tuples, such as is returned by
  `Reader.vector_as("weight", docnum, fieldname)`.

#### `add_document(docnum)`

Adds a document's term vector to the expander. If the field has a term vector,
uses it; otherwise falls back to stored field text.

#### `add_text(string)`

Adds a text string by indexing it with the field's analyzer.

#### `expanded_terms(number, normalize=True)`

Returns the N most important terms in the vectors added so far, ranked by
the expansion model's score.

- `number`: Number of terms to return.
- `normalize`: Whether to normalize weights.
- Returns: List of `(term, weight)` tuples, sorted by weight descending.

```python
from whoosh.classify import Expander, Bo1Model

expander = Expander(ix.reader(), "content")
for docnum in results.ids()[:10]:
    expander.add_document(docnum)

for word, weight in expander.expanded_terms(5):
    print(word, weight)
```

## Similarity Functions

### `shingles`

```python
whoosh.classify.shingles(input, size=2) -> iterable
```

Generates `(shingle, frequency)` pairs from a string by sliding a window of
the given size over the input.

**Parameters:**
- `input`: The input string.
- `size`: The shingle size (default `2`).

```python
from whoosh.classify import shingles

for shingle, freq in shingles("hello world", size=2):
    print(shingle, freq)
```

### `simhash`

```python
whoosh.classify.simhash(features, hashbits=32) -> int
```

Computes a simhash (perceptual hash) from a sequence of weighted features.
Simhashes that are similar produce similar hash values, allowing fast
near-duplicate detection via Hamming distance.

**Parameters:**
- `features`: Iterable of `(feature, weight)` tuples.
- `hashbits`: Number of bits in the hash (default `32`).
- Returns: An integer hash value.

```python
from whoosh.classify import shingles, simhash

h1 = simhash(shingles(text1))
h2 = simhash(shingles(text2))
from whoosh.classify import hamming_distance
dist = hamming_distance(h1, h2)
```

### `hamming_distance`

```python
whoosh.classify.hamming_distance(first_hash, other_hash, hashbits=32) -> int
```

Computes the Hamming distance between two hash values. A small distance
indicates high similarity.

**Parameters:**
- `first_hash`: First hash integer.
- `other_hash`: Second hash integer.
- `hashbits`: Number of bits in the hashes (default `32`).

## Clustering

### `kmeans`

```python
whoosh.classify.kmeans(
    data,
    k,
    t=0.0001,
    distfun=None,
    maxiter=50,
    centers=None
) -> (labels, centroids)
```

One-dimensional K-means clustering. Assigns each data point to the nearest
of `k` centroids and returns cluster labels and final centroids.

**Parameters:**
- `data`: List of data points (numeric values).
- `k`: Number of clusters.
- `t`: Tolerance; stops if centroid changes are below this value.
- `distfun`: Optional distance function (unused if `None`).
- `maxiter`: Maximum iterations (default `50`).
- `centers`: Optional list of initial centroids. If `None`, selects `k`
  random points from `data`.

**Returns:** A tuple `(labels, centroids)` where `labels` is a list of
cluster assignments per data point and `centroids` is the list of final
centroid positions.

### `two_pass_variance`

```python
whoosh.classify.two_pass_variance(data) -> float
```

Computes the sample variance of a data list using the two-pass algorithm
(first pass computes the mean, second pass accumulates squared deviations).

### `weighted_incremental_variance`

```python
whoosh.classify.weighted_incremental_variance(data_weight_pairs) -> float
```

Computes the weighted variance incrementally from a sequence of
`(value, weight)` pairs.

### `swin`

```python
whoosh.classify.swin(data, size) -> list
```

Sliding window clustering that groups data points where the range (max - min)
within a window of `size` is below a threshold. Uses variance for ranking.

**Parameters:**
- `data`: Sorted list of data points.
- `size`: Maximum window range (max - min) for clustering.

**Returns:** A list of `(left, right, count, variance)` tuples representing
clusters, sorted by count descending then by variance ascending.


## DOCUMENT (FR): Codecs

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Codecs API

Classes and interfaces for how Whoosh writes and reads the inverted index,
postings, and per-document values. The codecs module is a refactored package
exposing the same public API as the former monolithic module.

## Module Functions

### `default_codec`

```python
whoosh.codec.default_codec(*args, **kwargs) -> Codec
```

Returns the default codec used by the index. Currently returns a
`W3Codec` instance.

```python
from whoosh.codec import default_codec
codec = default_codec()
```

## Exceptions

### `OutOfOrderError`

```python
whoosh.codec.OutOfOrderError
```

Raised when documents are added to a field out of order. Fields must
receive documents in ascending docnum order.

## Base Classes

### `Codec`

```python
class whoosh.codec.Codec
```

Abstract base class for index codecs. Subclasses implement methods for
writing and reading the index format.

**Class Attributes:**
- `length_stats (bool)`: If `True`, the codec stores per-document field
  length statistics. Default `True`.

**Methods:**

#### `per_document_writer(storage, segment)`

Abstract. Returns a `PerDocumentWriter` for writing per-document values
(columns, term vectors) to the given segment.

#### `field_writer(storage, segment)`

Abstract. Returns a `FieldWriter` for writing postings to the given segment.

#### `postings_writer(dbfile, byteids=False)`

Abstract. Returns a `PostingsWriter` for writing posting lists to `dbfile`.

#### `postings_reader(dbfile, terminfo, format_, term=None, scorer=None)`

Abstract. Returns a `Matcher` for reading postings from `dbfile`.

#### `automata(storage, segment)`

Returns an `Automata` instance for spelling correction using automata-based
edit distance. Default returns a base `Automata()` object.

#### `terms_reader(storage, segment)`

Abstract. Returns a `TermsReader` for reading the term dictionary and
postings of the given segment.

#### `per_document_reader(storage, segment)`

Abstract. Returns a `PerDocumentReader` for reading per-document values
from the given segment.

#### `new_segment(storage, indexname)`

Abstract. Creates and returns a new `Segment` object for the given storage
and index name.

### `WrappingCodec`

```python
class whoosh.codec.WrappingCodec(child)
```

A `Codec` that delegates all operations to a child codec. Useful for
creating codec wrappers that modify or intercept specific operations.

**Constructor:**
- `child`: The underlying `Codec` instance to wrap.

All methods delegate to the child codec:
`per_document_writer()`, `field_writer()`, `postings_writer()`,
`postings_reader()`, `automata()`, `terms_reader()`, `per_document_reader()`,
`new_segment()`.

## Writer Classes

### `PerDocumentWriter`

```python
class whoosh.codec.PerDocumentWriter
```

Abstract base class for writing per-document values (columns, term vectors).

**Methods:**

#### `start_doc(docnum)`

Abstract. Called when starting to write a new document.

#### `add_field(fieldname, fieldobj, value, length)`

Abstract. Adds a field value to the current document.

#### `add_column_value(fieldname, columnobj, value)`

Abstract. Adds a column value. Raises `NotImplementedError` if the codec
doesn't support columns.

#### `add_vector_items(fieldname, fieldobj, items)`

Abstract. Adds term vector items.

#### `add_vector_matcher(fieldname, fieldobj, vmatcher)`

Convenience method that reads items from a `Matcher` and calls
`add_vector_items()`.

#### `finish_doc()`

Called when finishing a document. Default does nothing.

#### `close()`

Called when done writing. Default does nothing.

### `FieldWriter`

```python
class whoosh.codec.FieldWriter
```

Abstract base class for writing postings (inverted index) data.

**Methods:**

#### `add_postings(schema, lengths, items)`

Translates a generator of `(fieldname, btext, docnum, weight, vbytes)`
tuples into calls to `start_field()`, `start_term()`, `add()`,
`finish_term()`, and `finish_field()`.

**Parameters:**
- `schema`: The `Schema` object.
- `lengths`: Optional `FieldLengthTable` for document field lengths.
- `items`: Iterable of posting tuples.

#### `start_field(fieldname, fieldobj)`

Abstract. Called when starting a new field.

#### `start_term(text)`

Abstract. Called when starting a new term within a field.

#### `add(docnum, weight, vbytes, length=None)`

Abstract. Adds a posting to the current term.

#### `add_spell_word(fieldname, text)`

Called to add a word to the spelling index. Default does nothing.

#### `finish_term()`

Abstract. Called when finishing a term.

#### `finish_field()`

Called when finishing a field. Default does nothing.

#### `close()`

Called when done writing. Default does nothing.

### `PostingsWriter`

```python
class whoosh.codec.PostingsWriter
```

Abstract base class for writing posting lists (the inverted index).

**Methods:**

#### `start_postings(format_, terminfo)`

Abstract. Starts writing postings for a new term.

#### `add_posting(id_, weight, vbytes, length=None)`

Abstract. Adds a posting to the current term.

#### `finish_postings(allow_compact=True)`

Called when finished writing postings. Default does nothing.

#### `written()`

Abstract. Returns `True` if this writer has already written to disk.

## Reader Classes

### `FieldCursor`

```python
class whoosh.codec.FieldCursor
```

Abstract base class for iterating over terms in a field.

**Methods:**
- `first()`: Move to the first term.
- `find(string)`: Find a term matching or closest to `string`.
- `next()`: Move to the next term.
- `term()`: Returns the current term's text.

### `EmptyCursor`

```python
class whoosh.codec.EmptyCursor
```

A `FieldCursor` representing an empty field. All methods return `None` or
`False`.

### `TermsReader`

```python
class whoosh.codec.TermsReader
```

Abstract base class for reading the term dictionary and postings of a
segment.

**Methods:**
- `__contains__(term)`: Returns `True` if the term exists.
- `cursor(fieldname, fieldobj)`: Returns a `FieldCursor`.
- `terms()`: Yields `(fieldname, text)` tuples for all terms.
- `terms_from(fieldname, prefix)`: Yields terms from `fieldname` starting
  with `prefix`.
- `items()`: Yields `((fieldname, text), TermInfo)` tuples.
- `items_from(fieldname, prefix)`: Like `items()` but filtered by prefix.
- `term_info(fieldname, text)`: Returns a `TermInfo` for the term.
- `frequency(fieldname, text)`: Returns the total frequency.
- `doc_frequency(fieldname, text)`: Returns the document frequency.
- `matcher(fieldname, text, format_, scorer=None)`: Returns a `Matcher`.
- `indexed_field_names()`: Yields names of indexed fields.
- `close()`: Close the reader.

### `PerDocumentReader`

```python
class whoosh.codec.PerDocumentReader
```

Abstract base class for reading per-document values (columns, term vectors,
stored fields).

**Methods:**
- `close()`: Close the reader.
- `doc_count()`: Returns number of non-deleted documents.
- `doc_count_all()`: Returns total document count (including deleted).
- `has_deletions()`: Returns `True` if any documents are deleted.
- `is_deleted(docnum)`: Returns `True` if docnum is deleted.
- `deleted_docs()`: Yields docnums of deleted documents.
- `all_doc_ids()`: Yields docnums of all non-deleted documents.
- `supports_columns()`: Returns `True` if column storage is supported.
- `has_column(fieldname)`: Returns `True` if field has a column.
- `list_columns()`: Yields names of available columns.
- `column_reader(fieldname, column)`: Returns a column reader.
- `doc_field_length(docnum, fieldname)`: Returns field length for docnum.
- `field_length(fieldname)`: Returns total field length.
- `min_field_length(fieldname)`: Returns minimum field length.
- `max_field_length(fieldname)`: Returns maximum field length.
- `has_vector(docnum, fieldname)`: Returns `True` if docnum has a vector.
- `vector(docnum, fieldname, format_)`: Returns a `Matcher` for the vector.
- `stored_fields(docnum)`: Returns dict of stored field values.
- `all_stored_field()`: Yields stored fields for all documents.

### `MultiPerDocumentReader`

```python
class whoosh.codec.MultiPerDocumentReader(readers, offset=0)
```

Combines multiple `PerDocumentReader` instances into one for multi-segment
indices.

**Constructor:**
- `readers`: List of `PerDocumentReader` instances.
- `offset`: Base document offset (usually `0`).

## Automata

### `Automata`

```python
class whoosh.codec.Automata
```

Provides static methods for automata-based term matching, used by the
spelling corrector.

**Static Methods:**

#### `levenshtein_dfa(uterm, maxdist, prefix=0)`

Returns a deterministic finite automaton (DFA) that matches all edit-distance
variants of `uterm` within `maxdist` edits, optionally requiring a minimum
shared prefix of length `prefix`.

#### `find_matches(dfa, cur)`

Given a DFA and a `FieldCursor`, yields all matching terms.

**Methods:**

#### `terms_within(fieldcur, uterm, maxdist, prefix=0)`

Returns an iterator of matching terms within the given edit distance of
`uterm`.

## Segment

### `Segment`

```python
class whoosh.codec.Segment
```

Represents a segment of the index. Instances are pickled into the TOC file
to describe on-disk files.

**Class Attributes:**
- `COMPOUND_EXT = ".seg"`: Extension for compound segment files.

**Instance Attributes:**
- `indexname`: Base name of the segment.
- `segid`: Random unique ID string.
- `compound (bool)`: Whether this segment uses compound file format.

**Methods:**
- `make_filename(ext)`: Returns `f"{segment_id()}{ext}"`.
- `list_files(storage)`: Lists all files belonging to this segment.
- `create_file(storage, ext, **kwargs)`: Creates a new file for this segment.
- `open_file(storage, ext, **kwargs)`: Opens a file for this segment.
- `create_compound_file(storage)`: Combines all segment files into a
  compound `.seg` file.
- `open_compound_file(storage)`: Opens the compound segment file.
- `doc_count_all()`: Abstract. Returns total document count.
- `doc_count()`: Returns non-deleted document count.
- `set_doc_count(doccount)`: Sets the document count.
- `has_deletions()`: Returns `True` if any documents are deleted.
- `deleted_count()`: Abstract. Returns number of deleted documents.
- `deleted_docs()`: Abstract. Yields docnums of deleted documents.
- `delete_document(docnum, delete=True)`: Abstract. Deletes/undeletes a
  document.
- `is_deleted(docnum)`: Abstract. Returns `True` if docnum is deleted.
- `should_assemble()`: Returns `True` by default. Override to control
  compound file behavior.
- `validate(storage)`: Checks on-disk integrity of this segment.
- `segment_id()`: Returns the unique segment identifier string.
- `is_compound()`: Returns `True` if this segment uses compound file format.

### `WrappingSegment`

```python
class whoosh.codec.WrappingSegment(child)
```

A `Segment` that delegates all operations to a child segment.

**Constructor:**
- `child`: The underlying `Segment` instance to wrap.

## W3 Codec (Default)

The `W3` codec ("Whoosh 3") is the default index format, storing postings in
compressed blocks for efficient reading and skipping.

### `W3Codec`

```python
class whoosh.codec.whoosh3.W3Codec(blocklimit=128, compression=3, inlinelimit=1)
```

The default codec. Uses compressed blocks and term inlining for efficient
storage and fast lookups.

**Constructor:**
- `blocklimit`: Number of postings per block (default `128`).
- `compression`: zlib compression level (default `3`, `0` = no compression).
- `inlinelimit`: Maximum number of postings to inline directly in the term
  info (default `1`).

**File Extensions:**
- `.trm`: Term dictionary
- `.pst`: Postings
- `.vps`: Vector postings
- `.col`: Per-document value columns

### `W3PerDocWriter`

Writer for per-document values using the W3 format. Handles columns,
stored fields, term vectors, and field lengths.

### `W3FieldWriter`

Writer for the inverted term index using the W3 format. Uses a
`OrderedHashWriter` for the term dictionary and posts to a postings file.

### `W3LeafMatcher`

```python
class whoosh.codec.whoosh3.W3LeafMatcher(postfile, startoffset, length, format_, term=None, byteids=None, scorer=None)
```

Reads on-disk postings from the postings file and presents the
`Matcher` interface. Supports block-level skipping and lazy block loading.

**Optimization methods:**
- `block_min_id()`: Returns the first doc ID in the current block.
- `block_max_id()`: Returns the last doc ID in the current block.
- `block_min_length()`: Returns the minimum field length in the current block.
- `block_max_length()`: Returns the maximum field length in the current block.
- `block_max_weight()`: Returns the maximum weight in the current block.
- `skip_to_quality(minquality)`: Skips blocks exceeding a quality threshold.

### `W3TermsReader`

Reader for the term dictionary using the W3 format. Uses an
`OrderedHashReader` for fast lookups.

### `W3TermInfo`

```python
class whoosh.codec.whoosh3.W3TermInfo
```

Stores term statistics and posting location information. Supports inlining
small posting sets directly in the term dictionary for fast lookups.

**Flags:**
- `_FLAG_OFFSET` (0): Postings stored at an offset in the postings file.
- `_FLAG_INLINE_PICKLE` (1): Postings inlined as a pickled tuple.
- `_FLAG_INLINE_COMPACT` (2): Single posting compactly inlined.
- `_FLAG_INLINE_COMPACT_SHORT` (3): Multiple postings compactly inlined.

**Methods:**
- `add_block(block)`: Merges block statistics into this term info.
- `set_extent(offset, length)`: Sets offset and length of postings in file.
- `extent()`: Returns `(offset, length)`.
- `set_inlined(ids, weights, values)`: Sets inlined posting data.
- `set_compact_inline(id_, weight, value)`: Sets single inlined posting.
- `set_compact_short_inline(ids, weights, values)`: Sets multiple compact
  inlined postings.
- `is_inlined()`: Returns `True` if postings are inlined.
- `inlined_postings()`: Returns `(ids, weights, values)` tuples for inlined
  postings.
- `to_bytes()` / `from_bytes()`: Serialize/deserialize.

### `W3Segment`

```python
class whoosh.codec.whoosh3.W3Segment(codec, indexname, doccount=0, segid=None, deleted=None)
```

Segment class for the W3 codec. Stores a reference to the codec, document
count, and deleted document set.

## Plain Text Codec (Debugging)

### `PlainTextCodec`

```python
class whoosh.codec.plaintext.PlainTextCodec
```

A codec that stores the index as human-readable plain text. Intended for
debugging and manual inspection, not for production use.

**Class Attributes:**
- `length_stats = False`

**File extensions:**
- `.dcs`: Document (stored fields, columns, vectors)
- `.trm`: Term dictionary (plain text)

### `PlainPerDocWriter`

Plain text writer for per-document values.

### `PlainPerDocReader`

Plain text reader for per-document values.

### `PlainFieldWriter`

Plain text writer for the inverted index.

### `PlainTermsReader`

Plain text reader for the term dictionary.

### `PlainSegment`

```python
class whoosh.codec.plaintext.PlainSegment(indexname)
```

Segment class for the plain text codec. Does not support compound files
(`should_assume()` returns `False`).

## Memory Codec

### `MemoryCodec`

```python
class whoosh.codec.memory.MemoryCodec
```

An in-memory-only codec for testing. Stores all data in Python objects
rather than on disk.

**Class Attributes:**
- `storage`: A `RamStorage` instance.
- `segment`: A `MemSegment` instance.

**Methods:**
- `writer(schema)`: Returns a `MemWriter`.
- `reader(schema)`: Returns a `SegmentReader`.

### `MemWriter`

```python
class whoosh.codec.memory.MemWriter
```

A `SegmentWriter` subclass that commits immediately without merging.

### `MemPerDocWriter`

In-memory writer for per-document values.

### `MemPerDocReader`

In-memory reader for per-document values.

### `MemFieldWriter`

In-memory writer for the inverted index.

### `MemTermsReader`

In-memory reader for the term dictionary.

### `MemSegment`

```python
class whoosh.codec.memory.MemSegment(codec, indexname)
```

In-memory segment storing all data in Python dictionaries (inverted index,
stored fields, lengths, vectors, term infos). Uses a `Lock` for thread-safe
access.


## DOCUMENT (FR): Collectors

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Collectors API

Classes and functions for gathering search results. Collectors are used
internally by `Searcher.search()` to collect matching documents and build
`Results` objects. The collectors module is a refactored package exposing the
same public API as the former monolithic module.

## Overview

A `Collector` iterates over matching documents in an index, collects
information about them, and produces a `Results` object. The base `Collector`
class defines the interface; specialized subclasses implement different
collection strategies (top-N, unlimited, sorting, filtering, faceting, etc.).

## Core Classes

### `Collector`

```python
class whoosh.collectors.Collector
```

Abstract base class for all collectors. Subclasses must implement `collect()`
and `results()`.

**Methods:**

#### `prepare(top_searcher, q, context)`

Called before a search begins. Sets up `self.top_searcher`, `self.q`,
`self.context`, `self.starttime`, and `self.docset`.

#### `run()`

Iterates over sub-searchers, calling `set_subsearcher()` and
`collect_matches()` for each, then calls `finish()`.

#### `set_subsearcher(subsearcher, offset)`

Called when moving to a new sub-searcher. Sets `self.subsearcher`,
`self.offset`, and `self.matcher`.

#### `collect(sub_docnum)`

Called for every matched document. Must add the document to results and
return a sort key. Subclasses must implement this.

- `sub_docnum`: Segment-relative document number. Add `self.offset` to get
  the top-level document number.

#### `sort_key(sub_docnum)`

Returns a sort key for the current match without the side effect of adding
the document to results. Subclasses must implement this.

#### `collect_matches()`

Calls `matches()` and then `collect()` for each matched document.

#### `matches()`

Yields segment-relative document numbers for matches in the current
sub-searcher.

#### `count()`

Returns the total number of matching documents.

#### `all_ids()`

Returns a sequence of docnums matched in this collector.

#### `computes_count()`

Returns `True` if the collector naturally computes the exact count of
matching documents.

#### `finish()`

Called after the search completes. Sets `self.runtime`.

#### `remove(global_docnum)`

Removes a document from the collector using its global docnum.

#### `results()`

Returns a `Results` object. Subclasses must implement this.

### `ilen`

```python
whoosh.collectors.ilen(iterator) -> int
```

Counts the number of items in an iterator without loading it all into memory.

## Scored Collectors

### `ScoredCollector`

```python
class whoosh.collectors.ScoredCollector(replace=10)
```

Base class for collectors that sort by document score.

**Constructor:**
- `replace`: Number of matches between attempts to replace the matcher with
  a more efficient version.

### `TopCollector`

```python
class whoosh.collectors.TopCollector(
    limit=10,
    usequality=True,
    **kwargs
)
```

A collector that returns only the top N scored results.

**Constructor:**
- `limit`: Maximum number of results to return.
- `usequality`: Whether to use block-quality optimizations for faster
  search. Can be set to `False` for debugging.

**Notes:**
- When `usequality=True`, `computes_count()` returns `False` and
  `all_ids()` requires re-searching.
- Uses a min-heap to efficiently track the top N documents.

### `UnlimitedCollector`

```python
class whoosh.collectors.UnlimitedCollector(reverse=False)
```

A collector that returns **all** scored results. Sorts by score (descending
by default).

**Constructor:**
- `reverse`: If `True`, sort results in ascending order (lowest scores first).

### `UnsortedCollector`

```python
class whoosh.collectors.UnsortedCollector
```

A collector that returns results in document order (no sorting). Used when
the search weighting is `None`.

## Wrapping Collectors

### `WrappingCollector`

```python
class whoosh.collectors.WrappingCollector(child)
```

Base class for collectors that wrap other collectors. Delegates most
operations to the child collector while adding additional behavior.

**Constructor:**
- `child`: The collector to wrap.

**Methods** (all delegated to child):
`top_searcher`, `context`, `prepare`, `set_subsearcher`, `all_ids`,
`count`, `collect_matches`, `sort_key`, `collect`, `remove`, `matches`,
`finish`, `results()`

### `SortingCollector`

```python
class whoosh.collectors.SortingCollector(
    sortedby,
    limit=10,
    reverse=False
)
```

A collector that returns results sorted by a `FacetType` object.

**Constructor:**
- `sortedby`: A `FacetType` or field name to sort by.
- `limit`: Maximum number of results (0 for no limit).
- `reverse`: If `True`, reverse the overall sort order.

### `FilterCollector`

```python
class whoosh.collectors.FilterCollector(
    child,
    allow=None,
    restrict=None
)
```

A collector that allows and/or restricts certain document numbers in
results.

A document is discarded if:
- `allow` is set and the docnum is not in the allowed set, or
- `restrict` is set and the docnum is in the restricted set.

**Constructor:**
- `child`: The collector to wrap.
- `allow`: A query, `Results` object, or set-like of allowed docnums.
  `None` means everything is allowed.
- `restrict`: A query, `Results` object, or set-like of disallowed docnums.
  `None` means nothing is disallowed.

**Attributes:**
- `filtered_count`: Number of documents filtered out.

### `FacetCollector`

```python
class whoosh.collectors.FacetCollector(child, groupedby, maptype=None)
```

A collector that creates groups of documents based on facet objects. Used
when `groupedby` is specified in `Searcher.search()`.

**Constructor:**
- `child`: The collector to wrap.
- `groupedby`: A field name, `FacetType`, dict, or `Facets` object.
- `maptype`: Default `FacetMap` class for facets that don't specify one.

**Attributes:**
- `facetmaps`: Dictionary of facet name to `FacetMap` objects.

### `CollapseCollector`

```python
class whoosh.collectors.CollapseCollector(
    child,
    keyfacet,
    limit=1,
    order=None
)
```

A collector that eliminates all but the top N results sharing the same facet
key. Useful for "dedup" or grouped result views.

**Constructor:**
- `child`: The collector to wrap.
- `keyfacet`: A `FacetType` to collapse on. All but the top N documents
  sharing a key are eliminated.
- `limit`: Maximum documents to keep per key (default `1`).
- `order`: Optional `FacetType` to determine which documents are "top" within
  each group. Defaults to the results order (e.g., highest score).

**Attributes:**
- `collapsed_counts`: Dictionary mapping keys to the number of documents
  eliminated.

### `TimeLimitCollector`

```python
class whoosh.collectors.TimeLimitCollector(
    child,
    timelimit,
    greedy=False,
    use_alarm=True
)
```

A collector that raises a `TimeLimit` exception if the search exceeds a
time limit. Partial results are still available via `results()`.

**Constructor:**
- `child`: The collector to wrap.
- `timelimit`: Maximum search time in seconds.
- `greedy`: If `True`, finish adding the current hit before raising.
- `use_alarm`: If `True` (default), use `signal.SIGALRM` on Unix for
  immediate interruption. On Windows, time is only checked between
  documents.

```python
from whoosh.searching import TimeLimit

uc = collectors.UnlimitedCollector()
tlc = TimeLimitCollector(uc, timelimit=5.8)
try:
    searcher.search_with_collector(myquery, tlc)
except TimeLimit:
    print("Search timed out!")
# Still get partial results:
print(tlc.results())
```

### `TermsCollector`

```python
class whoosh.collectors.TermsCollector(child, settype=set)
```

A collector that records which terms appeared in which matched documents.
Used when `terms=True` in `Searcher.search()`.

**Constructor:**
- `child`: The collector to wrap.
- `settype`: Set type to use for docnum collections (default `set`).

**Attributes:**
- `termdocs`: Dict mapping `(fieldname, text)` tuples to arrays of docnums.
- `docterms`: Dict mapping docnums to lists of `(fieldname, text)` tuples.

## Exceptions

### `TimeLimit`

```python
from whoosh.searching import TimeLimit
```

Raised by `TimeLimitCollector` when the search exceeds the time limit.
Partial results are still available from the collector.


## DOCUMENT (FR): Columns

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Columns API

Classes for storing per-document values (column-oriented storage) used for
fast sorting, faceting, and filtering. Columns are the mechanism by which
Whoosh stores field values alongside the inverted index, in a column-oriented
layout for efficient range access.

The default column type for most fields is `VarBytesColumn`, although numeric
and date fields use `NumericColumn`. Expert users may use other column types
that may be faster or more storage-efficient based on the field contents.

A `Column` object stores configuration information and provides two important
methods: `writer()` to return a `ColumnWriter` and `reader()` to return a
`ColumnReader`.

## Module Functions

### `bytes_column`

```python
whoosh.columns.bytes_column
```

A default `VarBytesColumn` instance used as the column type for string fields.

### `numeric_column`

```python
whoosh.columns.numeric_column
```

A default `NumericColumn` instance used as the column type for numeric fields.

## Base Classes

### `Column`

```python
class whoosh.columns.Column
```

Base class for all column types.

**Class Attributes:**
- `reversible (bool)`: Whether values can be reversed for descending sort.
  Default `False`.

**Methods:**
- `writer(dbfile)`: Returns a `ColumnWriter` for this column type.
- `reader(dbfile, basepos, length, doccount)`: Returns a `ColumnReader` for
  this column type.
- `default_value(reverse=False)`: Returns the default value for documents
  without a column value at index time.
- `stores_lists()`: Returns `True` if the column stores a list of values per
  document instead of a single value.

### `ColumnWriter`

```python
class whoosh.columns.ColumnWriter(dbfile)
```

Base class for writing column values to disk.

**Constructor:**
- `dbfile`: The `StructFile` to write to.

**Methods:**
- `fill(docnum)`: Fills any gap in docnums up to `docnum` with default values.
- `add(docnum, value)`: Adds a value for the given docnum.
- `finish(docnum)`: Called when done writing. Default does nothing.

### `ColumnReader`

```python
class whoosh.columns.ColumnReader(dbfile, basepos, length, doccount)
```

Base class for reading column values from disk.

**Constructor:**
- `dbfile`: The `StructFile` to read from.
- `basepos`: The offset within the file at which the column starts.
- `length`: The length in bytes the column occupies in the file.
- `doccount`: The number of rows (documents) in the column.

**Methods:**
- `__getitem__(docnum)`: Returns the value for the given docnum.
- `sort_key(docnum)`: Returns the value for sorting (defaults to
  `__getitem__`).
- `__iter__()`: Yields values for all documents.
- `load()`: Returns a list of all values.
- `set_reverse()`: Prepares the reader for reverse iteration.

## Concrete Column Types

### `VarBytesColumn`

```python
class whoosh.columns.VarBytesColumn(
    allow_offsets=True,
    write_offsets_cutoff=2**15
)
```

Stores variable-length byte strings. The default value for documents without
a value is `b''` (empty bytes).

**Constructor:**
- `allow_offsets`: Whether to write offsets for faster lookup when there are
  many rows. Default `True`.
- `write_offsets_cutoff`: Write offsets when there are more than this many
  rows (default `2**15`).

### `FixedBytesColumn`

```python
class whoosh.columns.FixedBytesColumn(blocksize, default=emptybytes)
```

Stores fixed-length byte strings, saving space by not storing the length of
each value.

**Constructor:**
- `blocksize`: Fixed size of each value in bytes.
- `default`: Default value for documents without a value.

### `RefBytesColumn`

```python
class whoosh.columns.RefBytesColumn(
    cachesize=1000,
    stable=True,
    default=emptybytes
)
```

Stores references to unique values rather than the values themselves, saving
space when the field has few unique values. Uses a `DocIdSet` to track which
documents contain each value.

**Constructor:**
- `cachesize`: Size of the LRU cache for value lookups (default `1000`).
- `stable`: Whether to use a stable sort of references (default `True`).
- `default`: Default value for missing documents.

### `NumericColumn`

```python
class whoosh.columns.NumericColumn(
    typecode,
    default=None,
    nullable=False
)
```

Stores numbers (int, float, datetime) encoded as binary values. Extends
`FixedBytesColumn`.

**Constructor:**
- `typecode`: A `struct` typecode string (e.g., `"I"` for unsigned int,
  `"q"` for long, `"d"` for float).
- `default`: Default numeric value (None for the type's zero value).
- `nullable`: Whether `None` values are allowed.

### `BitColumn`

```python
class whoosh.columns.BitColumn
```

Stores boolean values as a bitmap. Each value is either `True` (1) or
`False` (0). Uses a `BitSet` internally.

### `CompressedBytesColumn`

```python
class whoosh.columns.CompressedBytesColumn(default=emptybytes)
```

Wraps a `VarBytesColumn` with zlib compression for the value bytes.

### `CompressedBlockColumn`

```python
class whoosh.columns.CompressedBlockColumn
```

Stores values with block-level zlib compression. More efficient for large
columns.

### `StructColumn`

```python
class whoosh.columns.StructColumn(struct, name)
```

Wraps a `FixedBytesColumn` to store structured binary data (e.g., tuples
encoded with `struct`).

**Constructor:**
- `struct`: A `struct.Struct` object defining the format.
- `name`: Field name for error messages.

### `EmptyColumnReader`

```python
class whoosh.columns.EmptyColumnReader(default, doccount)
```

A `ColumnReader` that returns a constant default value for every document.
Used when a field has no column.

### `MultiColumnReader`

```python
class whoosh.columns.MultiColumnReader(readers)
```

Combines multiple `ColumnReader` instances into one for multi-segment indices.

**Constructor:**
- `readers`: List of `ColumnReader` instances (one per segment).

### `TranslatingColumnReader`

```python
class whoosh.columns.TranslatingColumnReader(child, translator)
```

Wraps a `ColumnReader` to apply a translation function to the values.

**Constructor:**
- `child`: The underlying `ColumnReader`.
- `translator`: Function that maps sort keys to human-readable values.

### `WrappedColumn`

```python
class whoosh.columns.WrappedColumn(child)
```

Base class for column wrappers that adapt another column type.

### `WrappedColumnWriter`

```python
class whoosh.columns.WrappedColumnWriter(child)
```

Base class for column writer wrappers.

### `WrappedColumnReader`

```python
class whoosh.columns.WrappedColumnReader(child)
```

Base class for column reader wrappers.

### `ClampedNumericColumn`

```python
class whoosh.columns.ClampedNumericColumn(child, clampfn)
```

Wraps a `NumericColumn` to clamp values to a valid range before sorting.

**Constructor:**
- `child`: The wrapped `NumericColumn`.
- `clampfn`: Function that clamps a value to the valid range.

### `PickleColumn`

```python
class whoosh.columns.PickleColumn(child, ...)
```

Wraps another column to store pickled Python objects.

### `ListColumn`

```python
class whoosh.columns.ListColumn(child)
```

Base class for columns that store multiple values per document.

### `ListColumnReader`

```python
class whoosh.columns.ListColumnReader(child)
```

Reader for list-valued columns.

### `VarBytesListColumn`

```python
class whoosh.columns.VarBytesListColumn
```

A `ListColumn` variant of `VarBytesColumn` that stores lists of byte strings.

### `FixedBytesListColumn`

```python
class whoosh.columns.FixedBytesListColumn(blocksize)
```

A `ListColumn` variant of `FixedBytesColumn` that stores lists of fixed-size
byte strings.


## DOCUMENT (FR): Core

# API Core

Gestion des indexes via les fonctions et classes du module `whoosh.index`.

## Fonctions

### create_in

```python
def create_in(dirname, schema, indexname="MAIN", create=True, **kwargs) -> FileIndex
```

Crée un nouvel index dans le répertoire donné.

**Args:**
- `dirname (str)`: Chemin du répertoire.
- `schema (Schema)`: Objet Schema définissant les champs.
- `indexname (str)`: Nom de l'index.
- `create (bool)`: Si True, crée même si existe (efface l'existant).

**Retourne:**
- `FileIndex`: Objet index.

**Exemple:**
```python
from whoosh.index import create_in
index = create_in("indexdir", schema)
```

### open_dir

```python
def open_dir(dirname, indexname="MAIN", readonly=False, **kwargs) -> FileIndex
```

Ouvre un index existant.

**Exemple:**
```python
index = open_dir("indexdir")
```

### exists_in

```python
def exists_in(dirname, indexname="MAIN", **kwargs) -> bool
```

Vérifie si un index valide existe dans le répertoire.

## Classes

### Index

Classe de base pour les objets index.

#### Méthodes principales

| Méthode | Description |
|---------|-------------|
| `writer(**kwargs)` | Retourne un IndexWriter |
| `searcher(**kwargs)` | Retourne un Searcher |
| `reader()` | Retourne un IndexReader |
| `commit()` | Commit via un writer temporaire |
| `optimize()` | Fusionne tous les segments |
| `add_field()` | Ajoute un champ au schéma |
| `remove_field()` | Supprime un champ du schéma |
| `doc_count()` | Nombre de documents |
| `doc_count_all()` | Nombre total (y compris supprimés) |

## Exceptions

### LockError

Levée quand l'index est verrouillé par un autre writer.

```python
from whoosh.index import LockError

try:
    writer = ix.writer(timeout=5.0)
except LockError:
    print("Index verrouillé, réessayez plus tard")
```

### IndexMissingError

Levée quand l'index n'existe pas.


## DOCUMENT (FR): Events

# API Events

Système d'événements pour un couplage lâche entre les composants.

## EventBus

```python
class whoosh.event_bus.EventBus
```

Registre central des événements et subscribers.

### Méthodes

| Méthode | Description |
|---------|-------------|
| `bus.publish(event)` | Publie un événement |
| `bus.subscribe(fn)` | Abonne un handler |
| `bus.unsubscribe(fn)` | Désabonne un handler |
| `bus.clear()` | Supprime tous les abonnés |

**Exemple:**
```python
from whoosh.event_bus import EventBus

bus = EventBus()

@bus.subscribe
def on_index(event: DocumentIndexed):
    print(f"Indexé: {event.docnum}")

# Publier
bus.publish(DocumentIndexed(docnum=42))
```

## Événements intégrés

### DocumentIndexed

```python
class DocumentIndexed
    docnum: int           # Numéro de document
    schema: Schema        # Schéma utilisé
    timestamp: datetime   # Horodatage
    metadata: dict        # Métadonnées
```

### SearchExecuted

```python
class SearchExecuted
    query: str            # Requête originale
    result_count: int     # Nombre de résultats
    duration_ms: float    # Durée en ms
    user: str | None      # Utilisateur (si auth)
```

### IndexOptimized

```python
class IndexOptimized
    segments_before: int
    segments_after: int
    size_bytes: int
```

## Utilisation avec FastAPI

```python
from fastapi import FastAPI
from whoosh.event_bus import EventBus, DocumentIndexed

app = FastAPI()
bus = EventBus()

@app.on_event("startup")
def startup():
    bus.subscribe(on_index)

@app.post("/documents")
def add_document(doc: dict):
    # Indexation...
    bus.publish(DocumentIndexed(docnum=doc["id"]))
```

## Gestion d'erreurs

```python
@bus.subscribe
def on_error(event: SearchExecuted):
    if event.result_count == 0:
        logger.warning(f"Recherche vide: {event.query}")
```


## DOCUMENT (FR): Fields

# API Champs

Définissez la structure de votre index avec les types de champs.

## Schema

```python
class whoosh.fields.Schema
```

Définit les champs disponibles dans l'index.

### Méthodes

#### `add()`

```python
schema.add(fieldname, fieldtype, glob=False, **kwargs)
```

Ajoute un champ. Si `glob=True`, le nom est traité comme un pattern glob.

#### `remove()`

```python
schema.remove(fieldname, **kwargs)
```

Supprime un champ.

#### `items()`

```python
for name, field in schema.items():
    print(name, field)
```

Retourne les paires (nom, objet champ).

## Types de champs

### TEXT

```python
TEXT(
    stored=False,
    unique=False,
    phrase=True,
    analyzer=None,
    field_boost=1.0
)
```

Texte libre avec tokenisation et recherche de phrase optionnelle.

**Exemple:**
```python
titre = TEXT(stored=True)
corps = TEXT(analyzer=StemmingAnalyzer(), phrase=False)
```

### ID

```python
ID(stored=False, unique=False, field_boost=1.0)
```

Identifiant non tokenisé. Stocke la valeur entière comme terme unique.

**Exemple:**
```python
chemin = ID(stored=True, unique=True)
slug = ID(stored=True)
```

### KEYWORD

```python
KEYWORD(
    stored=False,
    lowercase=False,
    commas=False,
    scorable=False,
    field_boost=1.0
)
```

Mots-clés séparés par espace ou virgule.

**Exemple:**
```python
tags = KEYWORD(lowercase=True, commas=True, stored=True)
```

### STORED

```python
STORED(stored=True)
```

Champ stocké uniquement, non indexé ni searchable.

### NUMERIC

```python
NUMERIC(numtype=int, stored=False, unique=False, field_boost=1.0)
```

Champ numérique (entier ou flottant).

### DATETIME

```python
DATETIME(stored=False, unique=False, field_boost=1.0)
```

Champ date/heure.

### BOOLEAN

```python
BOOLEAN(stored=False, unique=False, field_boost=1.0)
```

Champ booléen. Searchable avec `oui`, `non`, `vrai`, `faux`, `1`, `0`, `t`, `f`.

### VectorField

```python
VectorField(
    dimensions: int,
    metric: str = "cosine",
    provider: str = "numpy",
    stored: bool = False
)
```

Champ pour embeddings vectoriels.

**Exemple:**
```python
embedding = VectorField(dimensions=384, metric="cosine", stored=True)
```

## SchemaBuilder

API fluent pour construire des schémas :

```python
from whoosh.fields import SchemaBuilder, TEXT, ID, NUMERIC

schema = (
    SchemaBuilder()
    .field("titre", TEXT(stored=True))
    .field("chemin", ID(stored=True, unique=True))
    .field("contenu", TEXT)
    .field("note", NUMERIC(float, stored=True))
    .build()
)
```

## Attributs de FieldType

| Attribut | Type | Description |
|----------|------|-------------|
| `format` | `Format` | Définit l'indexation |
| `vector` | `Format` | Format vectoriel optionnel |
| `scorable` | `bool` | Stocke la longueur pour BM25F |
| `stored` | `bool` | Stocke la valeur |
| `unique` | `bool` | Identifie les documents de façon unique |


## DOCUMENT (FR): Filedb Storage

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# File DB / Storage API

Classes for storing and retrieving index data on disk or in memory. The
`Storage` class is the main entry point for persisting an index.

## Storage Classes

### `Storage`

```python
class whoosh.filedb.filestore.Storage(path=None)
```

Abstract base class for storage backends. A `Storage` manages a filesystem-
or memory-based location where index files can be created, read, and
manipulated.

**Constructor:**
- `path`: Optional path string. Subclasses may use this to set the storage
  location.

**Methods:**

#### `create_file(name, **kwargs)`

Creates and returns a file object for writing.

#### `open_file(name, **kwargs)`

Opens and returns a file object for reading.

#### `list()`

Returns a list of all filenames in this storage.

#### `exists(name)`

Returns `True` if a file/named item exists in the storage.

#### `file_exists(name)`

Alias for `exists()`.

#### `file_length(name)`

Returns the length of file `name` in bytes.

#### `rename(src, dst)`

Renames a file from `src` to `dst`.

#### `delete_file(name)`

Deletes file `name` from storage.

#### `destroy()`

Deletes all files and the storage itself.

#### `temp_storage()`

Creates and returns a temporary isolated `Storage` for scratch space.

#### `supports_mmap`

Returns `True` if this storage supports memory-mapped file access.

**Properties:**
- `schema`: The `Schema` for this storage (if it holds an index).
- `lock`: The lock object used for this storage.

### `FileStorage`

```python
class whoosh.filedb.filestore.FileStorage(
    path,
    cachesize_limit=40,
    supports_mmap=None,
    **kwargs
)
```

A `Storage` subclass that uses the operating system's filesystem.

**Constructor:**
- `path`: A `Path` (or string path) to the directory where files are stored.
- `cachesize_limit`: Maximum number of open file handles to cache.
- `supports_mmap`: If `None`, auto-detected; otherwise force enable/disable.

**Methods:** All `Storage` methods plus:
- `create_index(schema, indexname="index", ...)`: Creates and returns a new
  `Index` object.
- `open_index(indexname="index", ...)`: Opens an existing `Index`.
- `lock(name)`: Returns a lock object for the given lock name.

### `RamStorage`

```python
class whoosh.filedb.filestore.RamStorage(cachesize_limit=10)
```

A `Storage` subclass that keeps all files in memory as bytes. Useful for
testing and small indexes.

**Constructor:**
- `cachesize_limit`: Maximum number of files to cache as decoded objects.

**Methods:** All `Storage` methods plus:
- `create_index(schema, ...)`: Creates an in-memory `Index`.
- `save_to_file(filename, ...)`: Saves the entire storage to a file.
- `load_from_file(filename, ...)`: Loads storage contents from a file.

### `OverlayStorage`

```python
class whoosh.filedb.filestore.OverlayStorage(base, overlay)
```

A `Storage` wrapper that presents two storage layers: a base and an overlay.
Files in the overlay take precedence over the base.

**Constructor:**
- `base`: The base `Storage` (e.g., read-only original).
- `overlay`: The overlay `Storage` (e.g., writable copy).

## Storage Exceptions

### `StorageError`

```python
class whoosh.filedb.filestore.StorageError
```

Base exception for storage-related errors.

### `ReadOnlyError`

```python
class whoosh.filedb.filestore.ReadOnlyError(StorageError)
```

Raised when attempting to write to a read-only storage.

## File Tables

### `HashWriter`

```python
class whoosh.filedb.filetables.HashWriter(dbfile, keycoder=None, keydecoder=None, data_encoder=None, data_decoder=None, **kwargs)
```

Writes key-value pairs to a file, with optional indexing by key.

**Constructor:**
- `dbfile`: The `StructFile` to write to.
- `keycoder`: Function to encode keys for storage.
- `keydecoder`: Function to decode keys from storage.
- `data_encoder`: Function to encode values.
- `data_decoder`: Function to decode values.

### `HashReader`

```python
class whoosh.filedb.filetables.HashReader(dbfile, length, keycoder=None, keydecoder=None, data_decoder=None, **kwargs)
```

Reads key-value pairs from a file written by `HashWriter`.

**Constructor:**
- `dbfile`: The `StructFile` to read from.
- `length`: Length of the data section.
- `keycoder`/`keydecoder`/`data_decoder`: Same as `HashWriter`.

**Methods:**
- `__getitem__(key)`: Returns the value for `key`.
- `keys()`: Yields all keys.
- `values()`: Yields all values.
- `items()`: Yields `(key, value)` pairs.
- `keys_from(prefixbytes)`: Yields keys starting at `prefixbytes`.
- `items_from(prefixbytes)`: Yields `(key, value)` pairs starting at prefix.
- `closest_key_pos(key)`: Returns the position of the closest matching key.
- `range_for_key(key)`: Returns `(startpos, endpos)` for a key range.

### `OrderedHashWriter`

```python
class whoosh.filedb.filetables.OrderedHashWriter(HashWriter)
```

A `HashWriter` that maintains keys in sorted order.

### `OrderedHashReader`

```python
class whoosh.filedb.filetables.OrderedHashReader(HashReader)`

A `HashReader` for reading data written by `OrderedHashWriter`. Preserves
key ordering for efficient prefix iteration.

### `FieldedOrderedHashWriter`

```python
class whoosh.filedb.filetables.FieldedOrderedHashWriter(HashWriter)
```

An `OrderedHashWriter` that stores an extra "fieldmap" in the extras dict,
mapping field names to numeric IDs.

### `FieldedOrderedHashReader`

```python
class whoosh.filedb.filetables.FieldedOrderedHashReader(HashReader)
```

Reader for data written by `FieldedOrderedHashWriter`.

## Struct File

### `StructFile`

```python
class whoosh.filedb.structfile.StructFile(name, source, cachesize_limit=40)
```

Wraps a file object and adds methods for reading/writing packed binary
values, arrays, varints, and pickle objects.

**Methods include:**
- `read_int()`, `write_int(n)`: Read/write a 4-byte signed integer.
- `read_long()`, `write_long(n)`: Read/write a 8-byte signed integer.
- `read_uint()`, `write_uint(n)`: Read/write unsigned int.
- `read_ulong()`, `write_ulong(n)`: Read/write unsigned long.
- `read_float()`, `write_float(n)`: Read/write a float.
- `read_ushort()`, `write_ushort(n)`: Read/write unsigned short.
- `read_byte()`, `write_byte(b)`: Read/write a single byte.
- `write_array(arr)`: Write an array of values.
- `get_array(offset, typecode, length)`: Read an array from offset.
- `write_pickle(obj)`: Pickle and write an object.
- `read_pickle()`: Read and unpickle an object.
- `get(offset, length)`: Read `length` bytes from `offset`.
- `get_int()`, `get_uint()`, `get_long()`, `get_float()`, `get_byte()`:
  Read a single value from the given offset.

### `BufferFile`

```python
class whoosh.filedb.structfile.BufferFile
```

A `StructFile` that wraps an in-memory byte buffer.

### `ChecksumFile`

```python
class whoosh.filedb.structfile.ChecksumFile(dbfile)
```

A `StructFile` wrapper that computes a checksum as data is written, for
integrity verification.

## Compound Storage

### `CompoundStorage`

```python
class whoosh.filedb.compound.CompoundStorage(dbfile, use_mmap=True)
```

Treats a single file as a container for multiple sub-files. Used for compound
segment files.

**Methods:**
- `create_file(name)`: Create a sub-file within the compound file.
- `open_file(name)`: Open a sub-file for reading.
- `list()`: List all sub-file names.
- `close()`: Close the compound storage.

### `SubFile`

```python
class whoosh.filedb.compound.SubFile
```

A file-like object representing a sub-file within a `CompoundStorage`.

### `CompoundWriter`

```python
class whoosh.filedb.compound.CompoundWriter(storage)
```

Writes a compound file by assembling multiple files from a storage.

**Methods:**
- `create_file(name)`: Reserve a filename in the compound file.
- `save_as_files(dest_storage, fn_generator)`: Assemble the compound file
  from source files into the destination storage.

## Storage Utility Functions

### `copy_storage`

```python
whoosh.filedb.filestore.copy_storage(sourcestore, deststore)
```

Copies all files from one storage to another.

### `copy_to_ram`

```python
whoosh.filedb.filestore.copy_to_ram(storage)
```

Reads all files from a storage into a `RamStorage` and returns it.


## DOCUMENT (FR): Formats

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Formats API

Classes that control how posting information (frequencies, positions,
character offsets, and weights) is encoded and stored for each field in the
index. The `Format` object is a factory and encoder/decoder for the
value strings stored alongside each posting.

## Module Functions

### `tokens`

```python
whoosh.formats.tokens(value, analyzer, kwargs)
```

Takes a text `value` and an `analyzer`, runs the analyzer on the value, and
returns the resulting token generator (wrapped with `unstopped()` to ignore
`STOP` tokens). Used internally by `Format.word_values()`.

## Format Classes

All format classes accept a `field_boost` parameter (default `1.0`) that
scales the score of all queries matching terms in that field.

### `Format`

```python
class whoosh.formats.Format(field_boost=1.0, **options)
```

Abstract base class for all posting formats. Format objects are
field-level objects: one is created per `Field` and shared across all
postings for that field.

**Attributes:**
- `posting_size (int)`: Fixed byte size of encoded postings, or `None`/`-1`
  if variable-size.
- `textual (bool)`: Whether this format expects string tokens (vs. bytes).
  Default `True`.

**Methods:**

#### `word_values(value, analyzer, **kwargs)`

Abstract. Takes a text value, runs it through the analyzer, and yields
`(tokentext, frequency, weight, valuestring)` tuples.

#### `encode(value)`

Abstract. Encodes raw posting data into the value string bytes.

#### `decode_frequency(valuestring)`

Abstract. Decodes the frequency (term count in document) from the value
string.

#### `decode_weight(valuestring)`

Abstract. Decodes the weight (total boost contribution) from the value string.

#### `combine(valuestrings)`

Abstract. Combines multiple value strings (from overlapping segments) into
a single value string.

#### `supports(name)`

Returns `True` if this format supports interpreting its postings as `name`
(e.g., `"frequency"`, `"positions"`, `"characters"`, `"position_boosts"`,
`"character_boosts"`). Equivalent to `hasattr(self, "decode_" + name)`.

#### `decoder(name)`

Returns the `decode_<name>` method for the given attribute name.

#### `decode_as(astype, valuestring)`

Calls the appropriate `decode_<astype>` method on `valuestring` and returns
the result.

#### `fixed_value_size()`

Returns `self.posting_size` if positive, otherwise `None`.

#### `__eq__(other)`

Returns `True` if `other` is the same class with equal `__dict__`.

### `Existence`

```python
class whoosh.formats.Existence(field_boost=1.0, **options)
```

Indexes only whether a term occurred in a document—not its frequency or
positions. Useful for non-scorable fields like paths.

- `posting_size = 0`
- Supports: `frequency` (always 1), `weight` (always `field_boost`)
- `encode()` returns empty bytes

### `Frequency`

```python
class whoosh.formats.Frequency(field_boost=1.0, boost_as_freq=False, **options)
```

Stores term frequency information (term count per document) for each posting.

- `posting_size = _INT_SIZE` (4 bytes)
- Supports: `frequency`, `weight`
- `encode()` encodes the count as a packed unsigned int
- `boost_as_freq`: If `True`, boosts are interpreted as frequency boosts

```python
from whoosh.formats import Frequency
fmt = Frequency(field_boost=1.0)
```

### `Positions`

```python
class whoosh.formats.Positions(field_boost=1.0, **options)
```

Stores position information (term offsets within the document) in each
posting, enabling phrase queries and "near" queries.

- Supports: `frequency`, `weight`, `positions`, `position_boosts`
- `encode(poslist)` encodes positions using variable-length delta encoding
- Positions are stored as delta-encoded variable-length integers

```python
from whoosh.formats import Positions
fmt = Positions()
```

### `Characters`

```python
class whoosh.formats.Characters(field_boost=1.0, **options)
```

Extends `Positions` to also store character start and end offsets for each
term occurrence, enabling character-precise highlighting.

- Supports: `frequency`, `weight`, `positions`, `position_boosts`,
  `characters`
- `encode()` encodes (position, startchar, endchar) triples with delta
  encoding

### `PositionBoosts`

```python
class whoosh.formats.PositionBoosts(field_boost=1.0, **options)
```

Extends `Positions` to store per-position boost values in addition to
positions.

- Supports: `frequency`, `weight`, `positions`, `position_boosts`
- `encode()` encodes `(position, boost)` pairs

### `CharacterBoosts`

```python
class whoosh.formats.CharacterBoosts(field_boost=1.0, **options)
```

Extends `Characters` to store per-position boost values along with
character offsets.

- Supports: `frequency`, `weight`, `positions`, `position_boosts`,
  `characters`, `character_boosts`
- `encode()` encodes `(position, startchar, endchar, boost)` tuples


## DOCUMENT (FR): Highlight

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Highlight API

Classes and functions for highlighting matches in search result fragments.
The highlight module is a refactored package exposing the same public API as
the former monolithic module.

## Overview

The highlighting system has four components:

- **Fragmenters** split text into fragments.
- **Fragment Scorers** score fragments to determine which to display.
- **Formatters** render fragments as output (HTML, plain text, etc.).
- **Highlighter** ties these together and is used by `Searcher.highlights()`.

## Module-level Functions

### `highlight`

```python
whoosh.highlight.highlight(
    text: str,
    terms: list[str],
    analyzer,
    fragmenter,
    formatter,
    top: int = 3,
    scorer=None,
    minscore: int = 1,
    order=SCORE,
    mode: str = "query"
) -> str
```

Highlights the matched terms in `text` and returns a formatted string.

- `text`: The text to highlight.
- `terms`: A list of matched terms (strings).
- `analyzer`: The analyzer for the field.
- `fragmenter`: A `Fragmenter` instance or class.
- `formatter`: A `Formatter` instance or class.
- `top`: Maximum number of fragments to return.
- `scorer`: Optional fragment scorer (defaults to `BasicFragmentScorer`).
- `minscore`: Minimum score for a fragment to be included.
- `order`: Sort order for fragments (`FIRST`, `SCORE`, `LONGER`, `SHORTER`).
- `mode`: Analysis mode, typically `"query"` or `"index"`.

### `mkfrag`

```python
whoosh.highlight.mkfrag(
    text: str,
    tokens,
    startchar=None,
    endchar=None,
    charsbefore: int = 0,
    charsafter: int = 0
) -> Fragment
```

Returns a `Fragment` object based on `Token` objects in `tokens`.

### `get_text`

```python
whoosh.highlight.get_text(
    original: str,
    token,
    replace: bool
) -> str
```

Returns the text to use for a match when formatting. If `replace` is `False`,
returns the original text between `token.startchar` and `token.endchar`. If
`True`, returns `token.text`.

### `set_matched_filter`

```python
whoosh.highlight.set_matched_filter(
    tokens,
    termset: frozenset
) -> Iterator[Token]
```

Marks tokens as matched if their `text` attribute is in `termset`. Used for
phrase-agnostic highlighting.

### `set_matched_filter_phrases`

```python
whoosh.highlight.set_matched_filter_phrases(
    tokens,
    text: str,
    terms,
    phrases
) -> Iterator[Token]
```

Marks tokens as matched using phrase-aware logic. Highlights only tokens that
are part of matched phrases.

### `top_fragments`

```python
whoosh.highlight.top_fragments(
    fragments,
    count: int,
    scorer,
    order,
    minscore: int = 1
) -> list[Fragment]
```

Returns the best `count` fragments sorted by `order`, filtered by `minscore`.

## Constants

### `DEFAULT_CHARLIMIT`

```python
whoosh.highlight.DEFAULT_CHARLIMIT = 2**15
```

Default character limit for fragments.

### Sort Order Constants

```python
whoosh.highlight.FIRST   # Sort passages from earlier in the document first
whoosh.highlight.SCORE   # Sort higher scored passages first
whoosh.highlight.LONGER  # Sort longer passages first
whoosh.highlight.SHORTER # Sort shorter passages first
```

## Formatters

### `Formatter`

```python
class whoosh.highlight.Formatter
```

Base class for formatters. Subclasses implement `format_token()` to define
how matched tokens are rendered.

**Methods:**

- `format_token(text, token, replace=False)`: Returns formatted text for a
  matched token.
- `format_fragment(fragment, replace=False)`: Returns formatted text for a
  `Fragment`.
- `format(fragments, replace=False)`: Returns formatted text for a list of
  fragments, joined by `between`.

**Attributes:**
- `between`: String inserted between formatted fragments (default `"..."`).

### `NullFormatter`

```python
class whoosh.highlight.NullFormatter(Formatter)
```

A formatter that does not modify the string. Returns fragments unformatted.

### `UppercaseFormatter`

```python
class whoosh.highlight.UppercaseFormatter(between="...")
```

Formats matched terms in uppercase.

### `HtmlFormatter`

```python
class whoosh.highlight.HtmlFormatter(
    tagname="strong",
    between="...",
    classname="match",
    termclass="term",
    maxclasses=5,
    attrquote='"'
)
```

Wraps matched terms in HTML tags with CSS class names. Two classes are
applied to each match: `classname` (same for all matches) and `termclass`
(different for each term, e.g. `term0`, `term1`).

- `tagname`: The HTML tag to wrap matches (default `"strong"`).
- `between`: Text inserted between fragments.
- `classname`: CSS class applied to all matched term tags.
- `termclass`: CSS class prefix for per-term classes.
- `maxclasses`: Maximum number of distinct per-term class numbers.
- `attrquote`: Quote character for attribute values.

**Methods:**
- `clean()`: Clears the internal term-to-classname mapping dictionary.

### `GenshiFormatter`

```python
class whoosh.highlight.GenshiFormatter(qname="strong", between="...")
```

Formats matched terms as Genshi event streams (requires the Genshi library).

## Fragmenters

### `Fragmenter`

```python
class whoosh.highlight.Fragmenter
```

Base class for fragmenters. Subclasses implement `fragment_tokens()` and/or
`fragment_matches()`.

**Methods:**
- `must_retokenize()`: Returns `True` if this fragmenter needs to re-tokenize
  the text (calls `fragment_tokens` with all tokens). Returns `False` if it can
  work from matched token positions alone (calls `fragment_matches`).

### `WholeFragmenter`

```python
class whoosh.highlight.WholeFragmenter(charlimit=DEFAULT_CHARLIMIT)
```

Does not fragment text. Returns the entire text as one fragment. Useful for
highlighting short fields.

```python
results.fragmenter = WholeFragmenter()
```

### `SentenceFragmenter`

```python
class whoosh.highlight.SentenceFragmenter(
    maxchars: int = 200,
    sentencechars=".!?",
    charlimit=DEFAULT_CHARLIMIT
)
```

Breaks text at sentence-ending punctuation (`.`, `!`, `?`).

- `maxchars`: Maximum characters per fragment.
- `sentencechars`: Characters that indicate sentence boundaries.
- `charlimit`: Maximum character position to process.

**Note:** Should be used with an analyzer that does not remove stop words.

### `ContextFragmenter`

```python
class whoosh.highlight.ContextFragmenter(
    maxchars: int = 200,
    surround: int = 20,
    charlimit=DEFAULT_CHARLIMIT
)
```

The default fragmenter. Finds matched terms and includes `surround` characters
of context before and after each match.

- `maxchars`: Maximum characters per fragment.
- `surround`: Number of context characters to include around matches.
- `charlimit`: Maximum character position to process.

### `PinpointFragmenter`

```python
class whoosh.highlight.PinpointFragmenter(
    maxchars: int = 200,
    surround: int = 20,
    autotrim: bool = False,
    charlimit=DEFAULT_CHARLIMIT
)
```

A non-retokenizing fragmenter that builds fragments from character positions of
matched terms. Faster than `ContextFragmenter` because it doesn't need to
re-tokenize text.

- `maxchars`: Maximum characters per fragment.
- `surround`: Number of context characters around matches.
- `autotrim`: If `True`, trims fragments to the nearest spaces.
- `charlimit`: Maximum character position to process.

### `NullFragmeter`

Alias for `WholeFragmenter`.

### `Fragment`

```python
class whoosh.highlight.Fragment(
    text: str,
    matches,
    startchar: int = 0,
    endchar: int = -1
)
```

Represents a fragment (excerpt) from a hit document. Stores the start and end
character offsets and the list of matched term objects.

**Attributes:**
- `text`: The original source text.
- `matches`: List of objects with `startchar` and `endchar` attributes.
- `startchar`: Start index of the fragment.
- `endchar`: End index of the fragment.
- `matched_terms`: Set of text values of matched terms.

**Methods:**
- `overlaps(fragment)`: Returns `True` if this fragment overlaps the given one.
- `overlapped_length(fragment)`: Returns the combined length of overlapping
  fragments.

### `FragmentScorer`

```python
class whoosh.highlight.FragmentScorer
```

Base class for fragment scoring objects. Subclasses implement `__call__()`
to score a `Fragment`.

### `BasicFragmentScorer`

```python
class whoosh.highlight.BasicFragmentScorer
```

Scores fragments by summing the boosts of matched terms, then multiplying by
the number of distinct matched terms (favors diversity).

## Highlighter

### `Highlighter`

```python
class whoosh.highlight.Highlighter(
    fragmenter=None,
    scorer=None,
    formatter=None,
    always_retokenize: bool = False,
    order=SCORE
)
```

Main highlighter object used by `Searcher.highlights()`.

- `fragmenter`: Fragmenter instance (defaults to `ContextFragmenter`).
- `scorer`: Fragment scorer (defaults to `BasicFragmentScorer`).
- `formatter`: Formatter instance (defaults to `HtmlFormatter(tagname="b")`).
- `always_retokenize`: If `True`, always re-tokenize text instead of using
  character offsets from postings.
- `order`: Sort order for fragments.

**Methods:**
- `highlight_hit(hitobj, fieldname, top=3, minscore=1, strict_phrase=False)`:
  Returns the highlighted string for a single hit in a given field.
- `can_load_chars(results, fieldname)`: Returns `True` if the field supports
  "pinpoint" highlighting using stored character offsets.


## DOCUMENT (FR): Idsets

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Idsets API

Specialized set implementations for storing sorted lists of positive
integers (document IDs). These are more memory-efficient than the built-in
`set` for certain use cases, though they are slower for most operations since
they are pure Python.

## Overview

The `DocIdSet` class is the abstract base class. Concrete implementations
include `BitSet`, `OnDiskBitSet`, `SortedIntSet`, `RoaringIdSet`, and
`MultiIdSet`. The `AutoIdSet` function selects the best implementation
based on the contents.

## Module Functions

### `autoset`

```python
whoosh.idsets.autoset
```

A factory that creates an appropriate `DocIdSet` subclass based on the
contents of a given iterable. If all integers in the set are below 10,000,
returns a `BitSet`; otherwise returns a `SortedIntSet`.

## `DocIdSet`

```python
class whoosh.idsets.DocIdSet
```

Abstract base class for set implementations specialized toward storing sorted
lists of positive integers.

**Inheritance:** Inherits from `set`-like interface.

**Methods:**
- `__eq__(other)`: Compares two `DocIdSet` instances by iterating.
- `__len__()`: Returns the number of elements. Override in subclasses.
- `__iter__()`: Yields elements in sorted order. Override in subclasses.
- `__contains__(i)`: Returns `True` if `i` is in the set.
- `__or__(other)`: Returns `self.union(other)`.
- `__and__(other)`: Returns `self.intersection(other)`.
- `__sub__(other)`: Returns `self.difference(other)`.
- `copy()`: Returns a copy of this set.
- `add(n)`: Adds `n` to the set.
- `discard(n)`: Removes `n` from the set (no error if absent).
- `update(other)`: Adds all elements from `other`.
- `intersection_update(other)`: Removes elements not in `other`.
- `difference_update(other)`: Removes all elements in `other`.
- `invert_update(size)`: In-place inversion over the range `[0, size)`.
- `intersection(other)`: Returns a new set with elements in both.
- `union(other)`: Returns a new set with elements from both.
- `difference(other)`: Returns a new set with elements in self but not other.
- `invert(size)`: Returns a new set that is the inversion over `[0, size)`.
- `isdisjoint(other)`: Returns `True` if no elements are shared.
- `before(i)`: Returns the previous integer in the set before `i`, or `None`.
- `after(i)`: Returns the next integer in the set after `i`, or `None`.
- `first()`: Returns the first (lowest) integer.
- `last()`: Returns the last (highest) integer.

## `BaseBitSet`

```python
class whoosh.idsets.BaseBitSet(DocIdSet)
```

Base class for bitmap-backed `DocIdSet` implementations. Uses a bytes-based
bitmap where each bit represents membership of an integer.

**Abstract Methods to Override:**
- `byte_count()`: Returns the number of bytes in the bitmap.
- `_get_byte(i)`: Returns the byte at index `i`.
- `_iter_bytes()`: Yields all bytes in the bitmap.

**Inherited Methods:** All `DocIdSet` methods with efficient bitmap
implementations of `__len__`, `__iter__`, `__contains__`, `first`, and
`last`.

## `OnDiskBitSet`

```python
class whoosh.idsets.OnDiskBitSet(file, doc_count)
```

A `BaseBitSet` that reads the bitmap from a file on disk, using `mmap` for
memory efficiency.

**Constructor:**
- `file`: A file-like object (opened in binary mode) containing the bitmap.
- `doc_count`: Total number of documents (bits) represented.

```python
from whoosh.idsets import OnDiskBitSet

with open("deletions.dat", "rb") as f:
    bs = OnDiskBitSet(f, doc_count=10000)
    if 42 in bs:
        print("Document 42 is deleted")
```

## `BitSet`

```python
class whoosh.idsets.BitSet
```

A `BaseBitSet` that stores the bitmap in memory as a `bytearray`. Fast for
membership tests and set operations on small ranges of integers.

**Constructor:**
- Optional initial iterable of integers.

```python
from whoosh.idsets import BitSet

bs = BitSet([0, 5, 10, 15])
print(5 in bs)  # True
print(bs.first())  # 0
print(len(bs))   # 4
```

**Methods:**
- `from_blob(data)`: Create a `BitSet` from raw bytes.
- `tostring()`: Returns the bitmap as a `bytes` string.
- `set_reverse()`: Prepares the set for reverse iteration.

## `SortedIntSet`

```python
class whoosh.idsets.SortedIntSet
```

A `DocIdSet` that stores integers as a sorted list of Python `int` objects.
More memory-efficient than `BitSet` for sparse sets but slower for membership
tests.

**Constructor:**
- Optional initial iterable of integers.

```python
from whoosh.idsets import SortedIntSet

sis = SortedIntSet([100, 500, 999])
print(500 in sis)  # True
print(sis.after(200))  # 500
```

## `ReverseIdSet`

```python
class whoosh.idsets.ReverseIdSet(child)
```

Wraps another `DocIdSet` to reverse the interpretation of integers. Instead
of representing membership directly, the set represents the *complement* of
the inner set. Useful for representing deleted documents.

**Constructor:**
- `child`: The `DocIdSet` to reverse.

**Example:** If `child` represents documents `{3, 7, 9}`, then
`ReverseIdSet(child)` represents all documents *except* `{3, 7, 9}`.

## `RoaringIdSet`

```python
class whoosh.idsets.RoaringIdSet
```

A `DocIdSet` that partitions integers into 16-bit buckets and uses `BitSet`
within each bucket. More memory-efficient than a single flat `BitSet` for
large, sparse sets of integers.

**Constructor:**
- Optional initial iterable of integers.

**Methods:**
- `from_bytes(data)`: Deserialize from bytes.
- `to_bytes()`: Serialize to bytes.
- `to_bytes_list()`: Returns a list of `(bucket, bytes)` pairs.

## `MultiIdSet`

```python
class whoosh.idsets.MultiIdSet(readers, offsets=None)
```

Combines multiple `DocIdSet` instances into one, handling document ID offsets
automatically. Used for combining deletions across multiple segments.

**Constructor:**
- `readers`: List of `DocIdSet` instances (one per segment).
- `offsets`: Optional list of base docnum offsets for each reader. If
  omitted, offsets are computed automatically.

**Methods:**
- `__contains__(i)`: Checks the appropriate sub-set based on offsets.
- `__iter__()`: Iterates over all integers in all sub-sets.
- `__len__()`: Returns the total count across all sub-sets.


## DOCUMENT (FR): Lang

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Language Support API

Language detection helpers, stemmer selection, stop-word lists, and
language-specific modules (Snowball stemmers, ISRI stemmer, Soundex,
Double Metaphone, etc.).

## Module Overview

The `whoosh.lang` package provides functions for detecting and selecting
language-specific resources (stemmers, stop words) and submodules containing
stemmers for various languages.

## Supported Languages

```python
whoosh.lang.languages = ("ar", "da", "nl", "en", "fi", "fr", "de", "hu",
                         "it", "no", "pt", "ro", "ru", "es", "sv", "tr")
```

Two-letter ISO 639-1 language codes for which stemmers or stop-word lists
are available.

## Language Aliases

```python
whoosh.lang.aliases = { ... }
```

A dictionary mapping alternate language identifiers to their canonical
two-letter codes. Includes ISO 639-3 three-letter codes, English names,
and native-language names.

## Exceptions

### `NoStemmer`

```python
class whoosh.lang.NoStemmer
```

Raised by `stemmer_for_language()` when no stemmer is available for the
given language.

### `NoStopWords`

```python
class whoosh.lang.NoStopWords
```

Raised by `stopwords_for_language()` when no stop-word list is available for
the given language.

## Language Functions

### `two_letter_code`

```python
whoosh.lang.two_letter_code(name) -> str or None
```

Converts a language identifier to its canonical two-letter code. Accepts
two-letter codes, ISO 639-3 codes, English names, and native-language names.

```python
from whoosh.lang import two_letter_code

code = two_letter_code("french")   # 'fr'
code = two_letter_code("deutsch")  # 'de'
code = two_letter_code("español")  # 'es'
```

### `has_stemmer`

```python
whoosh.lang.has_stemmer(lang) -> bool
```

Returns `True` if a stemmer is available for the given language.

### `has_stopwords`

```python
whoosh.lang.has_stopwords(lang) -> bool
```

Returns `True` if a stop-word list is available for the given language.

### `stemmer_for_language`

```python
whoosh.lang.stemmer_for_language(lang) -> callable
```

Returns a stemmer function for the given language. Raises `NoStemmer` if
no stemmer is available.

**Supported languages and stemmers:**
- `"en"` / `"en_porter"`: Original Porter stemmer (`whoosh.lang.porter`)
- `"ar"`: ISRI Arabic stemmer (`whoosh.lang.isri`)
- `"da"`: Danish Snowball stemmer
- `"nl"`: Dutch Snowball stemmer
- `"en"`: English Snowball stemmer
- `"fi"`: Finnish Snowball stemmer
- `"fr"`: French Snowball stemmer
- `"de"`: German Snowball stemmer
- `"hu"`: Hungarian Snowball stemmer
- `"it"`: Italian Snowball stemmer
- `"no"`: Norwegian Snowball stemmer
- `"pt"`: Portuguese Snowball stemmer
- `"ro"`: (no stemmer currently)
- `"ru"`: Russian Snowball stemmer
- `"es"`: Spanish Snowball stemmer
- `"sv"`: Swedish Snowball stemmer
- `"tr"`: (no stemmer currently)

```python
from whoosh.lang import stemmer_for_language

stem = stemmer_for_language("en")
print(stem("running"))  # 'run'
```

### `stopwords_for_language`

```python
whoosh.lang.stopwords_for_language(lang) -> list
```

Returns the stop-word list for the given language. Raises `NoStopWords` if
no stop-word list is available.

```python
from whoosh.lang import stopwords_for_language

stops = stopwords_for_language("en")
```

## Snowball Stemmers

The `whoosh.lang.snowball` subpackage contains stemmers implementing the
Snowball stemming algorithms for various languages.

### Available Stemmers

| Module | Class | Language |
|--------|-------|----------|
| `snowball.english` | `EnglishStemmer` | English |
| `snowball.dutch` | `DutchStemmer` | Dutch |
| `snowball.finnish` | `FinnishStemmer` | Finnish |
| `snowball.french` | `FrenchStemmer` | French |
| `snowball.german` | `GermanStemmer` | German |
| `snowball.hungarian` | `HungarianStemmer` | Hungarian |
| `snowball.italian` | `ItalianStemmer` | Italian |
| `snowball.norwegian` | `NorwegianStemmer` | Norwegian |
| `snowball.portugese` | `PortugueseStemmer` | Portuguese |
| `snowball.russian` | `RussianStemmer` | Russian |
| `snowball.romanian` | `RomanianStemmer` | Romanian |
| `snowball.spanish` | `SpanishStemmer` | Spanish |
| `snowball.swedish` | `SwedishStemmer` | Swedish |
| `snowball.danish` | `DanishStemmer` | Danish |

### Base Classes

```python
class whoosh.lang.snowball.bases._ScandinavianStemmer
class whoosh.lang.snowball.bases._StandardStemmer
```

Internal base classes for Snowball stemmers. User code should use the
language-specific stemmer classes directly.

### `classes`

```python
whoosh.lang.snowball.classes = {"da": DanishStemmer, "nl": DutchStemmer, ...}
```

Dictionary mapping two-letter language codes to Snowball stemmer classes.

## Porter Stemmer

### `whoosh.lang.porter`

The original Porter stemming algorithm, faster but less accurate than
Snowball English stemmer.

#### `stem`

```python
whoosh.lang.porter.stem(w) -> str
```

Stems a single English word using the Porter algorithm.

## ISRI Stemmer

### `whoosh.lang.isri.ISRIStemmer`

```python
class whoosh.lang.isri.ISRIStemmer
```

Arabic stemmer based on the Information Science Research Institute (ISRI)
algorithm. Does not use a root dictionary.

#### `stem`

```python
def ISRIStemmer.stem(word) -> str
```

Stems an Arabic word.

## Double Metaphone

### `whoosh.lang.dmetaphone.double_metaphone`

```python
whoosh.lang.dmetaphone.double_metaphone(text) -> tuple
```

Returns a tuple of `(primary, secondary)` metaphone codes for the given
text, using the Double Metaphone algorithm.

## Soundex

### `whoosh.lang.phonetic`

Soundex implementations for phonetic matching.

#### `soundex_en`

```python
whoosh.lang.phonetic.soundex_en(word) -> str
```

English Soundex encoding.

#### `soundex_esp`

```python
whoosh.lang.phonetic.soundex_esp(word) -> str
```

Spanish Soundex encoding.

#### `soundex_ar`

```python
whoosh.lang.phonetic.soundex_ar(word) -> str
```

Arabic Soundex encoding.

## WordNet Thesaurus

### `whoosh.lang.wordnet.Thesaurus`

```python
class whoosh.lang.wordnet.Thesaurus
```

Provides synonym expansion based on WordNet-style data.

**Methods:**
- `synonyms(word)`: Returns the set of synonyms for `word`.
- `__contains__(word)`: Returns `True` if `word` is in the thesaurus.

### Functions

```python
whoosh.lang.wordnet.parse_file(f) -> dict
whoosh.lang.wordnet.make_index(storage, indexname, word2nums, num2words)
whoosh.lang.wordnet.synonyms(word2nums, num2words, word) -> set
```

## Lovins Stemmer

### `whoosh.lang.lovins`

A suffix-stripping stemmer by Lovins. Functions include:
- `stem(word)`: Main stemming function.
- `remove_ending(word)`: Removes suffixes.
- `fix_ending(word)`: Fixes the word ending after stemming.

## Paice-Husk Stemmer

### `whoosh.lang.paicehusk.PaiceHuskStemmer`

```python
class whoosh.lang.paicehusk.PaiceHuskStemmer(rules)
```

A rule-based stemmer using Paice-Husk rules.

#### `stem`

```python
def PaiceHuskStemmer.stem(word) -> str
```

Stems a word using the Paice-Husk algorithm.

**Usage note:** The module also exposes a pre-configured stemmer:
```python
whoosh.lang.paicehusk.stem = PaiceHuskStemmer(defaultrules).stem
```


## DOCUMENT (FR): Matching

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Matching API

Classes and functions for iterating over and combining result sets during
searching. The matching module is a refactored package exposing the same
public API as the former monolithic module.

## Overview

When you search an index, Whoosh creates `Matcher` objects representing the
postings (document IDs and scores) produced by query objects. Matchers can
be combined (e.g., union, intersection) to build compound queries. The
matching module provides the core `Matcher` class hierarchy, utility
functions, and concrete implementations for various query types.

## Core Matcher Classes

### `Matcher`

```python
class whoosh.matching.Matcher
```

Abstract base class for all matchers. Concrete subclasses implement
`__init__()` and the `_set()` and `_maybe_values()` methods.

**Methods:**

#### `init = property(is_active)`

Property that returns whether the matcher is "active" (at top of segment
postings, not exhausted).

#### `init(view, docnum, score)`

Called when the matcher is initialized.

#### `set(matcher)`

Replaces this matcher with another one.

#### `copy()`

Returns a copy of this matcher.

#### `all_ids()`

Returns a list of docnums matched by this matcher.

#### `matches(matcher)`

Returns `True` if any of the current matches in `self` also match in
`matcher`.

#### `skip_to(docid)`

Advances the matcher to the first match at or after `docid`.

#### `skip_to_intersect(matcher)`

Moves this matcher to the earliest matching docnum that is also matched in
`matcher`.

#### `next()`

Advances the matcher to the next match.

#### `next_in_segment()`

Advances to the next match in the current segment.

#### `next_segment(matcher)`

Advances to the next segment in the context of `matcher`.

#### `is_active(in_segment=False)`

Returns `True` if this matcher has more matches to process.

#### `all_matching_segments()`

Generates `(segment_num, matcher)` pairs for all matching segments.

#### `doc()`

Returns the current document number of this matcher. May advance to next
document if not already on one.

#### `docnum()`

Returns the current docnum (segment-relative) of the matcher.

#### `score()`

Returns the current match's score.

#### `value()`

Returns the current match's value (e.g., the decoded stored value of the
term).

#### `supports()`

Returns `True` if `value()` is supported.

#### `value_matches()`

Returns the value at the current match.

#### `all_values()`

Returns a list of all values in this matcher.

#### `supports_lee()`

Returns `True` if the matcher uses lazy evaluation.

#### `lee`

Returns the current "lazy evaluation extension" value (for term vectors).

#### `spans()`

If the postings include positions, returns a list of `Position` objects for
the current match.

#### `spans()`

Returns the spans (positions) of the match in the current document.

#### `next_type()`

Returns the type of the next match.

#### `copy()`

Returns a shallow copy of this matcher.

### `Child`

```python
class whoosh.matching.Child
```

Mixin class for matchers that wrap other matchers.

### `FilterMixin`

```python
class whoosh.matching.FilterMixin
```

Mixin for matchers used as filters (boolean scoring, no relevance).

### `Custom`

```python
class whoosh.matching.Custom
```

Mixin for matchers that return a custom score from `score()` rather than 1.

### `Constant`

```python
class whoosh.matching.Constant
```

Mixin for matchers whose score is always the same value.

### `Coord`

```python
class whoosh.matching.Coord
```

Mixin for matchers that compute coordination factor (for phrase and other
queries that benefit from it).

## Concrete Matcher Classes

### `ListUnion`

```python
class whoosh.matching.ListUnion(matcher, items, maptype=None)
```

Base class for matchers that combine multiple matchers with a list of keys.

#### `filter`

```python
class whoosh.matching.filter
```

Decorator for creating filter matchers (boolean matchers with no relevance).

### `Union`

```python
class whoosh.matching.Union(matcher, items)
```

Base class for the `OR` operator.

### `Intersection`

```python
class whoosh.matching.Intersection(matcher, items)
```

The `AND` operator. A document matches only if it appears in all the child
matchers.

#### `IntersectionFilter`

```python
class whoosh.matching.IntersectionFilter(matcher, items)
```

A filter (no scoring) version of intersection.

### `And`

```python
class whoosh.matching.And(matcher, items)
```

Alias for `Intersection`.

### `Or`

```python
class whoosh.matching.Or(matcher, items)
```

Alias for `Union`.

### `Not`

```python
class whoosh.matching.Not(matcher, a, b)
```

The `NOT` operator. Matches all documents in `a` that are not in `b`.

### `Require`

```python
class whoosh.matching.Require(matcher, a, b)
```

Matches documents in `a` only if they also appear in `b`, but does not add
`b`'s score.

### `AndNot`

```python
class whoosh.matching.AndNot(matcher, a, b)
```

Matches documents in `a` that are not in `b`.

#### `AndMaybe`

```python
class whoosh.matching.AndMaybe(matcher, a, b)
```

Matches documents in `a`, adding `b`'s score if present.

### `BinaryUnion`

```python
class whoosh.matching.BinaryUnion(items)
```

Efficient intersection of exactly two matchers.

#### `BinaryUnion2`

```python
class whoosh.matching.BinaryUnion2
```

Optimized binary union for two items.

### `TreeMatcher`

```python
class whoosh.matching.TreeMatcher
```

A matcher that wraps a `Tree` object for combining results.

### `NestedParent`

```python
class whoosh.matching.NestedParent(parent, child, bools=False)
```

Matches parent documents that have at least one child document matched by
the child matcher. Used for nested document queries.

### `NestedChildren`

```python
class who which.matching.NestedChildren(parentmatch, child)
```

Matches child documents for a given parent document.

### `LengthMatcher`

```python
class whoosh.matching.LengthMatcher(child, q, polarity=False)
```

Matches documents based on field length (used by `Every` query).

### `Filter`

```python
class whoosh.matching.Filter(matcher)
```

Converts any matcher into a filter (no scoring).

### `AlwaysFilter`

```python
class whoosh.matching.AlwaysFilter
```

A filter that matches all documents.

### `NeverFilter`

```python
class whoosh.matching.NeverFilter
```

A filter that matches no documents.

### `PseudoMatcher`

```python
class whoosh.matching.PseudoMatcher
```

Base class for pseudo-matchers used in span queries.

## Matching Utilities

### `current_spans`

```python
whoosh.matching.current_spans(matcher) -> list
```

Returns a list of `Span` objects for the current match in `matcher`, or an
empty list if the matcher doesn't support positions.

### `disjunction_score`

```python
whoosh.matching.disjunction_score(matcher) -> float
```

Returns the sum of `matcher.score()` and the scores of all child matchers of
type `Union`.

### `intersection_score`

```python
whoosh.matching.intersection_score(matcher) -> float
```

Returns the sum of `matcher.score()` and all child matchers of type
`Intersection`.

### `child_count`

```python
whoosh.matching.child_count(matcher) -> int
```

Returns the number of child matchers in `matcher`.

### `has_quality`

```python
whoosh.matching.has_quality(matcher) -> bool
```

Returns `True` if `matcher` has a `query` attribute (i.e., is a
`QueryMatcher`-derived object, or a combination of such matchers).

### `has_untranslated`

```python
whoosh.matching.has_untranslated(matcher) -> bool
```

Returns `True` if the matcher has an `untranslated` attribute (set by
certain wrapper matchers like `TimeLimited`).

### `wrap`

```python
whoosh.matching.wrap(matcher)
```

Returns `matcher` if it has a `.copy()` method, otherwise wraps it in an
`AutoMatcher`.

### `wrap2`

```python
whoosh.matching.wrap2(a, b, m)
```

Returns either a `BinaryUnion2` or an `AutoMatcher` depending on whether `a`
and `b` are list-compatible.

### `unified`

```python
whoosh.matching.unified(matcher)
```

Returns `matcher` if it has an `untranslated` attribute, otherwise returns
`None`.

### `deletion`

```python
whoosh.matching.deletion(matcher)
```

If `matcher` has a `parent` attribute, returns the parent, otherwise returns
`None`.

### `AutoMatcher`

```python
class whoosh.matching.AutoMatcher(m, **kwargs)
```

A general-purpose matcher that wraps arbitrary objects and adds default
behavior for scoring, docnums, and other features. Created by `wrap()`.

### `MatchingTimeLimit`

```python
class whoosh.matching.MatchingTimeLimit
```

A lightweight exception raised when a query matcher exceeds a time limit.

### `TimeLimited`

```python
class whoosh.matching.TimeLimited(child, maxsteps=100, timeout=None, currenttime=None)
```

Wrapper that wraps a `Matcher` to enforce a time limit. Raises
`MatchingTimeLimit` if the time limit is exceeded.

**Parameters:**
- `child`: The matcher to wrap.
- `maxsteps`: Check time every N documents (default `100`).
- `timeout`: Maximum time in seconds (default `None`, no limit).
- `currenttime`: Optional function to use for getting the current time.

### `TermMatcher`

```python
class whoosh.matching.TermMatcher(postings, text, qname, scorer=None, boost=1.0)
```

Matches documents containing a specific term.

**Constructor:**
- `postings`: A `Postings` object from the index reader.
- `text`: The term text.
- `qname`: The query name for this term.
- `scorer`: Optional `Scorer` object.
- `boost`: Boost factor for this term's score.

### `MultiScorer`

```python
class whoosh.matching.MultiScorer(numgroups, start_i=0)
```

A `Scorer` that combines the scores from multiple scorers into one, weighted
across groups of segments.

### `RangeMatcher`

```python
class whoosh.matching.RangeMatcher(start_matcher, end_matcher, query)
```

Matches documents within a range of term values.

### `RegexMatcher`

```python
class whoosh.matching.RegexMatcher(regex, qname, boost=1.0)
```

Matches documents whose terms match a compiled regex.

### `SpanMatcher`

```python
class whoosh.matching.SpanMatcher(matcher, order=0, end=0)
```

Matches spans (positions) within documents.

### `SpanOverlap`

```python
class whoosh.matching.SpanOverlap(l, r)
```

Matches overlapping spans from two matchers.

### `SpanNear`

```python
class whoosh.matching.SpanNear(l, r, slop=1, ordered=True)
```

Matches spans that are near each other within a document.

### `SpanCondition`

```python
class whoosh.matching.SpanCondition(l, r)
```

Matches a condition on spans.

### `SpanBefore`

```python
class whoosh.matching.SpanBefore(l, r, end=0)
```

Matches spans before a given position.

### `SpanAfter`

```python
class whoosh.matching.SpanAfter(l, r, end=0)
```

Matches spans after a given position.

### `SpanOutside`

```python
class whoosh.matching.SpanOutside(l, r, end=0)
```

Matches spans outside a given range.

### `SpanFirst`

```python
class whoosh.matching.SpanFirst(l, start=0, end=1)
```

Matches spans at the beginning of a document.

### `SpanNot`

```python
class whoosh.matching.SpanNot(l, r)
```

Matches spans in `l` that are not in `r`.

### `SpanOr`

```python
class whoosh.matching.SpanOr(items)
```

Logical OR for span matchers.

### `SpanAnd`

```python
class whoosh.matching.SpanAnd(l, r)
```

Logical AND for span matchers.


## DOCUMENT (FR): Middleware

# API Middleware

Pipeline de middleware pour les opérations d'indexation et de recherche.

## Classes principales

### Middleware

```python
class whoosh.middleware.base.Middleware
```

Classe de base pour tous les middlewares.

#### Méthodes

| Hook | Signature | Appelé quand |
|------|-----------|--------------|
| startup | (context) -> context | Initialisation writer/searcher |
| shutdown | (context) -> context | Nettoyage writer/searcher |
| before_index | (context) -> context | Avant l'ajout d'un document |
| after_index | (context) -> context | Après l'ajout d'un document |
| before_delete | (context) -> context | Avant la suppression |
| after_delete | (context) -> context | Après la suppression |
| before_search | (context) -> context | Avant la recherche |
| after_search | (context) -> context | Après les résultats |
| on_error | (context, exc) -> None | Sur exception |
| on_commit | (context) -> None | Après le commit |

### MiddlewareContext

```python
class whoosh.middleware.context.MiddlewareContext(
    operation: str,
    metadata: dict | None = None
)
```

Attributs:

| Attribut | Type | Description |
|----------|------|-------------|
| `operation` | str | `"index"`, `"search"`, `"delete"`, `"commit"` |
| `query` | Any | Requête (pour `search`) |
| `results` | Any | Résultats de la recherche |
| `document` | dict | Document à indexer (pour `index`) |
| `docnum` | int | Numéro du document (pour `delete`) |
| `metadata` | dict | Données par requête (request_id, trace_id) |

### MiddlewareChain

```python
class whoosh.middleware.base.MiddlewareChain(middlewares)
```

Orchestre l'exécution des middlewares dans l'ordre.

#### Méthodes

| Méthode | Description |
|---------|-------------|
| `chain.add(middleware)` | Ajoute un middleware |
| `chain.run_before(hook, context)` | Exécute les hooks before |
| `chain.run_after(hook, context)` | Exécute les hooks after |
| `chain.run_before_all(hook, context)` | Exécute tous les hooks before |
| `chain.run_after_all(hook, context)` | Exécute tous les hooks after |

### SettingGuard

```python
class whoosh.middleware.base.SettingGuard(
    field: str | None = None,
    default: bool = False
)
```

Vérifie et réinitialise les settings de middleware.

### Skip

```python
class whoosh.middleware.base.Skip(metadata)
```

Exception pour sauter une opération tout en la commitant.

## Intégration

### apply_middleware_to_writer

```python
def apply_middleware_to_writer(
    writer: IndexWriter,
    middlewares: list[Middleware]
) -> IndexWriter
```

Retourne un writer enveloppé par les middlewares.

### apply_middleware_to_searcher

```python
def apply_middleware_to_searcher(
    searcher: Searcher,
    middlewares: list[Middleware]
) -> Searcher
```

Retourne un searcher enveloppé par les middlewares.

## Exceptions

```python
class MiddlewareError(Exception)
class StopOperation(Exception)
class SkipOperation(Skip)
```


## DOCUMENT (FR): Modern

# Fournisseurs de stockage

Whoosh-NG fournit des backends de stockage pluggables via les contrats
`SyncStorageProvider` / `AsyncStorageProvider`. Cela permet de persister
l'index sur disque local, SQLite, S3, ou une configuration hybride cache +
distant sans modifier le writer ni l'index.

## Vue d'ensemble de l'architecture

### Niveau 1 : SnapshotStorage (Simple)

```
Writer → FS local → Commit → Upload Segment → S3
Reader → Download Segment → Open local
```

Très simple à maintenir. Utilisez `SnapshotStorage` quand vous voulez S3
comme cible de sauvegarde/restauration simple sans la complexité d'un cache
local.

### Niveau 2 : CachedObjectStorage (Recommandé pour la production)

```
+----------+
|  MinIO   |
+----------+
     ^
     |
 Sync |
     v
+-----------+   Couche Cache   +-----------+
| Searcher  |<--------------->| Writer    |
+-----------+                 +-----------+
        |
        v
 Local SSD
```

- L'index vit sur SSD
- S3 sert de réplication
- Les segments sont poussés après commit
- Restauration possible à tout moment

C'est exactement ce que font beaucoup de systèmes de recherche distribués
modernes.

## Fournisseurs disponibles

| Fournisseur | Type | Backend | Cas d'usage |
|-------------|------|---------|-------------|
| `FileStorage` | sync | système de fichiers local | Single-node, pas de cloud |
| `AsyncFileStorage` | async | système de fichiers local | Single-node async |
| `S3Storage` | sync | compatible S3 | Accès S3 direct |
| `SnapshotStorage` | sync | compatible S3 | Sauvegarde/restauration simple |
| `HybridStorage` | sync | cache local + distant | **Production** (alias : `CachedObjectStorage`) |
| `AsyncHybridStorage` | async | cache local + distant | Production async |

Tous les fournisseurs sont importables depuis `whoosh_modern.storage`.

## FileStorage

Stockage local sur système de fichiers. Les clés sont des chemins relatifs
sous ``root``.

```python
from whoosh_modern.storage import FileStorage

storage = FileStorage("indexdir")
storage.write("segment_1.dat", b"data")
assert storage.read("segment_1.dat") == b"data"
assert storage.exists("segment_1.dat") is True
storage.delete("segment_1.dat")
keys = storage.list_keys()
```

## AsyncFileStorage

Variante async de ``FileStorage``. Toutes les opérations s'exécutent dans
un thread de travail via ``asyncio.to_thread``.

```python
import asyncio
from whoosh_modern.storage import AsyncFileStorage

storage = AsyncFileStorage("indexdir")

async def main() -> None:
    await storage.awrite("segment_1.dat", b"data")
    data = await storage.aread("segment_1.dat")
    await storage.adelete("segment_1.dat")

asyncio.run(main())
```

## S3Storage

Stockage blobs compatible S3. ``boto3`` est requis uniquement lorsque ce
fournisseur est utilisé ; il est importé paresseusement pour que le reste
de Whoosh-NG n'en dépende pas. Un ``client`` peut être injecté pour les tests.

```python
from whoosh_modern.storage import S3Storage

storage = S3Storage(bucket="mon-bucket", prefix="segments")
storage.write("segment_1.dat", b"data")
data = storage.read("segment_1.dat")
keys = storage.list_keys()
```

Installer la dépendance optionnelle :

```bash
pip install whoosh-ng[s3]
```

## SnapshotStorage

Stockage snapshot S3 simple sans cache local. C'est la stratégie la plus
simple :

- Écriture : upload du segment directement vers S3
- Lecture : download du segment depuis S3 vers un fichier temporaire local

Utilisez ceci quand vous voulez S3 comme cible de sauvegarde/restauration
simple sans la complexité d'un cache local.

```python
from whoosh_modern.storage import SnapshotStorage

storage = SnapshotStorage(
    local_path="./index",
    bucket="mon-bucket",
    prefix="snapshots",
)

storage.write("segment_1.dat", b"data")
data = storage.read("segment_1.dat")
```

## HybridStorage / CachedObjectStorage

`HybridStorage` compose un cache local avec un backend distant. Le distant
est la source de vérité ; le cache local est une couche de performance
write-through.

`CachedObjectStorage` est un alias de `HybridStorage` qui exprime mieux
l'intention : un cache d'objets local synchronisé avec S3.

C'est l'architecture recommandée pour les déploiements production avec des
motifs de lecture répétés.

```python
from whoosh_modern.storage import HybridStorage, S3Storage

distant = S3Storage(bucket="mon-bucket", prefix="segments")
stockage = HybridStorage(local_cache="./cache", remote=distant)

# Write-through : le distant est la source de vérité, le cache est mis à jour
stockage.write("segment_1.dat", b"data")

# Première lecture : miss cache → fetch depuis S3, write-through dans le cache
data = stockage.read("segment_1.dat")

# Deuxième lecture : hit cache → servi depuis le disque local, zéro réseau
data = stockage.read("segment_1.dat")

# Forcer le rafraîchissement depuis le distant
stockage.invalidate("segment_1.dat")

# Pré-chauffer le cache
stockage.prefetch(["segment_2.dat", "segment_3.dat"])
```

### Chemin de lecture

1. hit cache local → retour immédiat
2. miss → lecture depuis le distant, write-through dans le cache, retour

### Chemin d'écriture

- ``distant.write(key, data)`` (source de vérité)
- en cas de succès → ``local_cache.write(key, data)``
- en cas d'échec → lever l'erreur avant de polluer le cache

### Éviction du cache

Le cache local est limité par `max_cache_size_mb` (défaut 1024 Mo). Quand
la limite est atteinte, les entrées les plus anciennes sont évincées selon
une politique LRU.

### `list_keys`

`list_keys()` utilise le distant comme source de vérité car le cache n'est
que partiel. Passez `include_cache=True` pour retourner l'union des clés
distant et cache.

## AsyncHybridStorage

Variante async de ``HybridStorage``. Les opérations distantes s'exécutent
dans un thread de travail via ``asyncio.to_thread`` pour ne jamais bloquer
la boucle d'événements.

```python
import asyncio
from whoosh_modern.storage import AsyncHybridStorage, S3Storage

distant = S3Storage(bucket="mon-bucket", prefix="segments")
stockage = AsyncHybridStorage(local_cache="./cache", remote=distant)

async def main() -> None:
    await stockage.awrite("segment_1.dat", b"data")
    data = await stockage.aread("segment_1.dat")
    await stockage.adelete("segment_1.dat")
    cles = await stockage.alist_keys()

asyncio.run(main())
```

## Utilisation avec SearchApplication

```python
from whoosh_modern import SearchApplication, SQLSource
from whoosh_modern.storage import HybridStorage, S3Storage

distant = S3Storage(bucket="mon-bucket", prefix="segments")
stockage = HybridStorage(local_cache="./cache", remote=distant)

app = SearchApplication(
    source=SQLSource(query="SELECT * FROM produits", connection=engine),
    storage=stockage,
)
app.build()
resultats = app.index.search("laptop")
```

## Benchmarks de performance

Les benchmarks ont été exécutés contre une instance MinIO locale en utilisant
un index Whoosh de 28,89 Mo (2 fichiers de segment). Les résultats indiquent
les performances relatives entre les stratégies sur un stockage compatible S3.

| Stratégie | Sauvegarde (Mo/s) | Restauration (Mo/s) | Notes |
|-----------|-------------------|---------------------|-------|
| `1_obj_per_segment` | 39.44 | 139.72 | Meilleur débit de restauration ; le plus simple |
| `compressed_zstd` | 31.56 | 133.74 | Bande passante réduite, overhead CPU |
| `hybrid_cache_s3` | 44.97 | 133.61 | Meilleure sauvegarde ; lectures cache chaud excellentes |
| `1_obj_per_posting_list` | 0.28 | 4.79 | **À éviter** : millions de petits objets tuent S3 |

### Recommandations

- **Par défaut** : `S3Storage` avec 1 objet par fichier de segment. Il offre
  le meilleur débit de restauration et est le plus simple à exploiter.
- **Production avec lectures répétées** : `HybridStorage(cache_local, S3Storage)`.
  Après le premier accès, les lectures suivantes sont servies depuis le disque
  local à ~133 Mo/s.
- **À éviter** : 1 objet par posting list. S3 n'est pas optimisé pour des
  millions de petits objets ; la latence et le coût explosent.
- **Compression** : ZSTD réduit la taille des transferts de ~20-30% au prix
  d'un overhead CPU. À utiliser quand la bande passante réseau est le
  goulot, pas quand le CPU l'est.

### Exécution des benchmarks

```bash
# Démarrer MinIO
docker run -d --name minio-benchmark -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \
  minio/minio:latest server /data --console-address ":9001"

# Lancer le benchmark synthétique
python benchmark/s3_storage_benchmark.py

# Lancer le benchmark avec un vrai index Whoosh (nécessite le CSV customers)
python benchmark/s3_storage_benchmark_real.py
```


## DOCUMENT (FR): Overview

# Vue d'ensemble API

Cette section fournit une référence complète de l'API publique de Whoosh-NG.

## Modules

| Module | Description |
|--------|-------------|
| `whoosh.index` | Gestion des indexes |
| `whoosh.fields` | Types de champs et schéma |
| `whoosh.writing` | Writers et politiques de fusion |
| `whoosh.searching` | Searcher, Results, collectors |
| `whoosh.query` | Classes de requêtes |
| `whoosh.qparser` | Analyseur de requêtes |
| `whoosh.analysis` | Tokenizers, filtres, analyseurs |
| `whoosh.highlight` | Surbrillance des résultats |
| `whoosh.spelling` | Correction orthographique |
| `whoosh.sorting` | Facettes et tri |
| `whoosh.event_bus` | Système d'événements |
| `whoosh.hooks` | Système de hooks |
| `whoosh.middleware` | Pipeline de middleware |
| `whoosh.plugins` | Système de plugins et registres |
| `whoosh.backends` | Backends de stockage |
| `whoosh.vector` | Providers de recherche vectorielle |
| `whoosh_modern.autocomplete` | Providers d'autocomplétion |
| `whoosh_fastapi` | Intégration FastAPI |

## Référence rapide

### Cycle de vie d'un index

```python
from whoosh.index import create_in, open_dir, exists_in

ix = create_in("indexdir", schema)
ix = open_dir("indexdir")
exists = exists_in("indexdir")
```

### Écriture

```python
with ix.writer() as writer:
    writer.add_document(champ1=val1, champ2=val2)
```

### Lecture

```python
from whoosh.qparser import QueryParser

with ix.searcher() as searcher:
    qp = QueryParser("content", ix.schema)
    q = qp.parse("requête")
    results = searcher.search(q)
```

### Schéma

```python
from whoosh.fields import Schema, TEXT, ID, NUMERIC

schema = Schema(
    titre=TEXT(stored=True),
    chemin=ID(stored=True, unique=True),
    compte=NUMERIC(int, stored=True)
)
```


## DOCUMENT (FR): Plugins

# API Plugins

Système de plugins, registres de providers et discovery.

## PluginManager

```python
class whoosh.plugins.manager.PluginManager
```

Point d'entrée unique pour gérer tous les plugins.

### Méthodes

| Méthode | Description |
|---------|-------------|
| `PluginManager.load_plugins()` | Auto-découvre et charge les entry points |
| `PluginManager.register(name, plugin)` | Enregistre un plugin |
| `PluginManager.enable(name)` | Active un plugin |
| `PluginManager.disable(name)` | Désactive un plugin |
| `PluginManager.get(name)` | Retourne le plugin par nom |
| `PluginManager.list_plugins()` | Liste des plugins enregistrés |

**Exemple:**
```python
from whoosh.plugins.manager import PluginManager

# Programmatique
PluginManager.register("mon_plugin", MonPlugin())
PluginManager.enable("mon_plugin")

# Entry points (dans pyproject.toml)
# [project.entry-points."whoosh_ng.plugins"]
# mon_plugin = "mon_package.plugin:MonPlugin"
PluginManager.load_plugins()
```

## BasePlugin

```python
class whoosh.plugins.base.BasePlugin
```

Classe de base pour tous les plugins.

### Méthodes à implémenter

| Méthode | Appelée quand |
|---------|---------------|
| `setup(registry)` | Plugin activé |
| `teardown(registry)` | Plugin désactivé |
| `on_startup()` | Démarrage application |
| `on_shutdown()` | Arrêt application |

### Attributs

| Attribut | Description |
|----------|-------------|
| `name` | Nom unique |
| `version` | Version du plugin |
| `dependencies` | Liste des noms de plugins requis |

## Registre

```python
class whoosh.registry.Registry
```

Registre global pour les providers de tous types.

### Méthodes

| Méthode | Description |
|---------|-------------|
| `registry.register(name, provider, category)` | Enregistre un provider |
| `registry.get(name, category)` | Récupère un provider |
| `registry.unregister(name, category)` | Supprime un provider |
| `registry.list_providers(category)` | Liste tous les providers d'une catégorie |

**Catégories courantes:**
- `"vector_provider"`
- `"autocomplete_provider"`
- `"storage_provider"`
- `"middleware"`

## VectorRegistry

```python
from whoosh.registry import VectorRegistry

VectorRegistry.register("numpy", NumpyProvider(), "mon_app")
provider = VectorRegistry.get("numpy", "mon_app")
```

## Exception

```python
class PluginNotFoundError(Exception)
```


## DOCUMENT (FR): Query

# API Requêtes

Construisez des requêtes complexes avec l'API de Whoosh-NG.

## Classes principales

### Query

Classe de base pour toutes les requêtes.

```python
class whoosh.query.Query
```

#### Méthodes

| Méthode | Description |
|---------|-------------|
| `q.all(methodname, *args)` | Applique une méthode à tous les termes |
| `q.normalize()` | Normalise la représentation textuelle |
| `q.replace(fieldname, old, new)` | Remplace un terme |
| `q.exclude(term)` | Exclut un terme spécifique |
| `q.fieldname` | Champ principal de la requête |
| `q.children()` | Requêtes enfants (pour opérateurs) |

## Requêtes booléennes

### And

```python
And(require, boost=1.0)
# Tous doivent matcher
```

### Or

```python
Or(require, boost=1.0)
# Un seul doit matcher
```

### Not

```python
Not(require, exclude=None)
```

### Requête Term

```python
Term(fieldname, text, boost=1.0)
```

## Plages

### NumericRange

```python
NumericRange(fieldname, start, end, startexcl=False, endexcl=False)
```

### DateRange

```python
DateRange(fieldname, start, end, startexcl=False, endexcl=False)
```

## Phrase et proximité

### Phrase

```python
Phrase(fieldname, words, slop=1, boost=1.0)
```

### Distance / Near

### Prefix

```python
Prefix(fieldname, text, boost=1.0)
```

## Combinateurs avancés

### Every

```python
Every(fieldname, boost=1.0)
```

### Null

```python
NullQuery
```

## Opérateurs de requête globaux

| Requête | Description |
|---------|-------------|
| `Term` | Égalité exacte (pas d'analyse) |
| `Variations` | Variantes lexicales |
| `FuzzyTerm` | Recherche floue |
| `Wildcard` | Jokers (lent sur grands corpus) |
| `Regex` | Expression régulière |

## Construction manuelle

```python
from whoosh.query import (
    Term, And, Or, Not, Phrase, NumericRange, DateRange
)

q = And([
    Term("status", "published"),
    NumericRange("date", 2020, 2025),
    Or([Term("tags", "python"), Term("tags", "recherche")]),
    Phrase("content", ["tutoriel", "whoosh"])
])
```


## DOCUMENT (FR): Reading

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Reading API

Classes and functions for reading from an index. The reading module is a
refactored package exposing the same public API as the former monolithic
module.

## Overview

The reading module provides classes for accessing documents, terms, and
postings in an index. The main entry points are `IndexReader` objects obtained
from a searcher. These readers allow you to enumerate terms, access stored
fields, iterate postings, and get term frequencies.

## Core Classes

### `IndexReader`

```python
class whoosh.reading.IndexReader
```

Abstract base class for reading index data. Concrete subclasses include
`SegmentReader` and `MultiReader` (which wraps multiple segment readers).

### `MultiReader`

```python
class whoosh.reading.MultiReader(readers, base=None)
```

Combines multiple `IndexReader` instances into one. All docnums are treated
as relative to the combined index.

**Constructor:**
- `readers`: A list of `IndexReader` instances.
- `base`: Optional list of cumulative document count offsets for each reader.

**Methods:**

#### `doc_frequency(fieldname, text)`

Returns the total number of documents that have the given term in the given
field across all sub-readers.

#### `documents()`

Yields dictionaries of stored fields for each document in the index.

#### `stored_fields(docnum)`

Returns a dictionary of stored field values for the given document number
(index-wide docnum).

```python
r = my_index.reader()
print(r.stored_fields(20))
```

#### `all_stored_fields()`

Yields a `(docnum, stored_fields)` tuple for each document in the index.

#### `terms(fieldname)`

Yields `(fieldname, text)` tuples for every term in the given field.

#### `terms_from(segmentreader,fieldnameprefix)`

Low-level method for multi-reader.

#### `has_termvector(docnum, fieldname)`

Returns `True` if the document has a term vector for the given field.

#### `term_vector(docnum, fieldname)`

Returns a `TermVector` for the given document and field.

#### `is_deleted(docnum)`

Returns `True` if the given document (index-wide docnum) is deleted.

#### `all_doc_ids()`

Returns a sorted array of non-deleted document IDs.

#### `min_spam(fieldname)`

Returns the minimum spam value for the given field.

#### `set_spam(fieldname)`

Returns the set spam value for the given field.

#### `has_exact_length(docnum)`

Returns `True` if the exact length is known for `docnum`.

#### `doc_field_length(docnum, fieldname=None, default=1)`

Returns the length of the given field in the given document.

```python
r = my_index.reader()
length = r.doc_field_length(20, "content")
```

#### `max_field_length(fieldname)`

Returns the maximum length of the given field across all documents.

#### `iter_fieldname`

Low-level method for multi-reader.

#### `lexicon(fieldname)`

Returns an array of all unique terms in the given field, sorted.

#### `expanded_lexicon(fieldname)`

Low-level method that yields terms without the overhead of building an array.

#### `term_info(fieldname, text)`

Returns a `TermInfo` object for the given term, or `None` if the term does
not appear in the index.

#### `terminfos(fieldname)`

Yields `(text, TermInfo)` pairs for the given field.

#### `postings(fieldname, text, stype=None)`

Returns a `Matcher` for the postings list of the given term.

```python
r = my_index.reader()
m = r.postings("content", "whoosh")
for docnum, score in m:
    print("doc %d has term" % docnum)
```

#### `_all_postings(fieldname)`

Low-level. Yields `(text, matcher)` pairs for all terms in a field.

#### `_posting_fragments()`

Low-level.

#### `has_vector(docnum, fieldname)`

Returns `True` if the given field has a term vector in the given document.

#### `vectors(docnum)`

Yields `(fieldname, TermVector)` pairs for all term vectors in the document.

#### `all_items(fieldname)`

Yields `(term, weight, docfreq)` tuples for every term in the given field.

#### `frequency(fieldname, text)`

Returns the total frequency of the term across all documents.

#### `idf(term)`

Returns an iterator of `(docnum, idf)` pairs for the given term.

#### `spelling`

Returns a `SpellingAnalyzer` for the given field.

#### `doc_term(slicenum, fieldname, word)`

Returns `(df, weight)` for `word` in `fieldname` in segment `slicenum`.

#### `doc_diff(slicenum, fieldname, text, num)`

Returns `(df, weight)` for `word` in `fieldname` in segment `slicenum`.

### `SegmentReader`

```python
class whoosh.reading.SegmentReader(segment, schema, storage, base=True)
```

Reader for a single segment of the index.

**Constructor:**
- `segment`: The `Segment` object.
- `schema`: The `Schema` object.
- `storage`: The `Storage` instance.
- `base`: Base document number offset (usually `True`, meaning compute it).

### `MultiID3Reader`

```python
class whoosh.reading.MultiID3Reader(readers, base)
```

Combines multiple readers that have ID3 codec.

### `TermInfo`

```python
class whoosh.reading.TermInfo(
    df=0,
    weight=0,
    minlength=0,
    maxlen=0,
    maxnum=0,
    numdocs=0,
    scorable=True
)
```

Information about a term in the index.

**Attributes:**
- `df`: Document frequency (number of documents containing the term).
- `weight`: Total term frequency across all documents.
- `minlength`: Minimum document length where the term appears.
- `maxlength`: Maximum document length where the term appears. This is `0`
  if lengths are not stored.
- `maxnum`: Maximum number of occurrences per document.
- `numdocs`: Number of documents where the term has a non-zero contribution
  to the score.
- `scorable`: Whether this term is scorable.

## Term Vector

### `TermVector`

```python
class whoosh.reading.TermVector(docnum, fieldname, format_, terms, store_term_vector)
```

Represents the term vector for a single document/field pair.

**Methods:**

#### `tokens(text=None)`

Yields `(t, w, v, p)` tuples for terms in this field.

- `t`: The term string.
- `w`: The term weight (frequency in this document).
- `v`: The list of positions where the term occurs. (`None` if positions
  are not stored.)
- `p`: The list of characters where the term occurs. (`None` if character
  vectors are not stored.)

#### `items(text=None)`

Like `tokens()` but includes term strings in the result.

```python
tv = my_index.reader().term_vector(0, "content")
for token, frequency, positions, chars in tv.tokens():
    print(token, frequency, positions)
```

**Parameters:**
- `text`: Optional `Bytes` object. If given, only yield terms starting with
  this text (used for multi-byte tokenization).

## Reader Utilities

### `get_storage`

```python
whoosh.reading.get_storage(searcher) -> Storage
```

Returns the storage object associated with the searcher.

### `get_index_schema`

```python
whoosh.reading.get_index_schema(searcher) -> Schema
```

Returns the schema object associated with the searcher.

### `load_termdocs`

```python
whoosh.reading.load_termdocs(reader, fieldname, text) -> list
```

Returns a list of document numbers that have the given term.

### `read_pattern`

```python
whoosh.reading.read_pattern(reader, fieldname, expression) -> list
```

Returns sorted term list from `reader.lexicon(fieldname)` filtered to those
matching `expression`.

### `read_terminfo`

```python
whoosh.reading.read_terminfo(reader, fieldname, text) -> TermInfo or None
```

Returns a `TermInfo` for the given term, or `None` if not found.


## DOCUMENT (FR): Reference

# Référence API

The Whoosh-NG API reference is générée automatiquement à partir du code source avec
[pydoctor](https://pydoctor.readthedocs.io/), qui analyse les modules Python
et génère une documentation HTML à partir des docstrings.

:::note
If the embedded documentation does not display, the API docs may not have
been generated yet in this deployment. [View on GitHub](https://github.com/dorel14/whoosh-ng/tree/master/website/static/api_docs)
for the full API documentation, or check the
[API modules list](#api-modules) below.
:::

## API Modules

### Core API

| Module | Description |
|--------|-------------|
| `whoosh.index` | High-level index creation, opening, and management |
| `whoosh.fields` | Schema and field type definitions |
| `whoosh.writing` | Writer classes and merge policies |
| `whoosh.searching` | Searcher, Results, and collectors |
| `whoosh.query` | Query classes and parsers |
| `whoosh.qparser` | Query parser implementation |
| `whoosh.analysis` | Tokenizers, filters, and analyzers |
| `whoosh.highlight` | Search result highlighting |
| `whoosh.spelling` | Spelling correction |
| `whoosh.sorting` | Facets and sorting |
| `whoosh.event_bus` | Event system |
| `whoosh.hooks` | Hook system |
| `whoosh.middleware` | Middleware pipeline |
| `whoosh.plugins` | Plugin system and registry |
| `whoosh.backends` | Storage backends |

### Modern API

| Module | Description |
|--------|-------------|
| `whoosh_modern.data_sources` | Data source protocol and implementations |
| `whoosh_modern.views` | SearchView unified interface |
| `whoosh_modern.middleware` | Retry, cache, logging middleware |
| `whoosh_modern.facets` | FacetManager for auto-discovery |
| `whoosh_modern.validation` | 4-level validation framework |
| `whoosh_modern.indexing` | BatchIndexWriter, AnalyzerCache |
| `whoosh_modern.linguistics` | Linguistic engine (stemmers, synonyms) |
| `whoosh_modern.storage` | Storage providers (HybridStorage, etc.) |
| `whoosh_modern.vector` | NumpyProvider for vector similarity |
| `whoosh_modern.autocomplete` | Autocomplete provider plugins |
| `whoosh_fastapi` | FastAPI REST API endpoints |
| `whoosh_admin` | Admin UI dashboard |

:::info
For the full interactive API documentation, run:
```bash
pip install pydoctor
python scripts/generate_api_docs.py
```
Then open `website/static/api_docs/index.html` in your browser.
:::


## DOCUMENT (FR): Searching

# API Recherche

Exécuter des requêtes et récupérer les résultats.

## Searcher

```python
class whoosh.searching.Searcher
```

Interface principale pour lire l'index.

### Méthodes

| Méthode | Description |
|---------|-------------|
| `search(query, limit=10)` | Exécute une requête |
| `search_page(query, pagenum, pagelen=10)` | Récupère une page de résultats |
| `search_with_collector(query, collector)` | Recherche avancée avec collector |
| `find(fieldname, text)` | Recherche dans un champ |
| `documents(**kwargs)` | Documents stockés correspondants |
| `document(**kwargs)` | Un document stocké |
| `lexicon(fieldname)` | Liste des termes d'un champ |
| `all_stored_fields()` | Itère sur tous les champs stockés |

## Results

```python
class whoosh.searching.Results
```

Conteneur de résultats (semblable à une liste).

### Méthodes

| Méthode/attribut | Description |
|------------------|-------------|
| `len(results)` | Nombre total de correspondances |
| `results.scored_length()` | Nombre de résultats scorés |
| `results[0]` | Premier résultat |
| `results[0:10]` | Slice de résultats |
| `results.has_matched_terms()` | Vérifie si les termes matchés sont collectés |
| `results.filtered_count` | Nombre de documents filtrés |
| `results.collapsed_counts` | Comptage par clé d'effondrement |

## Hit

```python
class whoosh.searching.Hit
```

Un document matché.

### Attributs

- `hit["champ"]`: Valeur du champ stocké
- `hit.score`: Score de pertinence
- `hit.docnum`: Numéro interne du document

### Méthodes

| Méthode | Description |
|---------|-------------|
| `hit.highlights("content", top=3)` | Extrait surlignés |
| `hit.matched_terms()` | Termes matchés (si `terms=True`) |

## Highlight

```python
from whoosh.highlight import highlight, Fragment

snippets = hit.highlights("content", top=3)
```

## Collectors

```python
from whoosh.collectors import FacetCollector, TimeLimitCollector
```

## Tri et facettes

```python
from whoosh import sorting

facet = sorting.FieldFacet("categorie")
results = searcher.search(query, sortedby="date")
```


## DOCUMENT (FR): Sorting

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Sorting API

Classes and functions for faceting and sorting search results. The sorting
module is a refactored package exposing the same public API as the former
monolithic module.

## Overview

Sorting and faceting use `FacetType` objects to compute sort keys for documents.
A `FacetType` creates a `Categorizer` that computes a key for each document.
The key is used for sorting and grouping. `FacetMap` objects hold the
results of grouping documents by a facet.

## Facet Types

### `FacetType`

```python
class whoosh.sorting.FacetType
```

Base class for "facets" — aspects that can be sorted and/or faceted.

**Attributes:**
- `maptype`: Default `FacetMap` class to use for this facet.

**Methods:**

#### `categorizer(global_searcher)`

Returns a `Categorizer` corresponding to this facet.

- `global_searcher`: A parent searcher for global document ID references.

#### `map(default=None)`

Returns a `FacetMap` instance for holding facet results.

#### `default_name()`

Returns the default name for this facet (default `"facet"`).

### `Categorizer`

```python
class whoosh.sorting.Categorizer
```

Base class for objects that compute a key value for a document for sorting and
faceting. Created by `FacetType` objects via `categorizer()`.

**Attributes:**
- `allow_overlap (bool)`: If `True`, use `keys_for()` to allow overlapping
  groups. Default `False`.
- `needs_current (bool)`: If `True`, the categorizer needs the matcher to be
  in a valid state when `key_for()` is called. Default `False`.

**Methods:**

#### `set_searcher(segment_searcher, docoffset)`

Called when the collector moves to a new segment. Sets up segment-specific
data.

- `segment_searcher`: The atomic sub-searcher for the current segment.
- `docoffset`: Offset of the segment's docnums relative to the full index.

#### `key_for(matcher, segment_docnum)`

Returns a sort key for the current match.

- `matcher`: A `Matcher` object. If `needs_current` is `False`, do not use
  this object as it may be inconsistent.
- `segment_docnum`: Segment-relative document number.

#### `keys_for(matcher, segment_docnum)`

Yields multiple keys for the current match. Called instead of `key_for()`
when `allow_overlap` is `True`.

#### `key_to_name(key)`

Translates the sort key into a human-readable representation for facet
group names (e.g., converts an integer date sort key to a `datetime`).

### `FieldFacet`

```python
class whoosh.sorting.FieldFacet(
    fieldname,
    reverse=False,
    allow_overlap=False,
    maptype=None
)
```

Sorts/facets by the contents of a field.

**Constructor:**
- `fieldname`: Name of the field to sort/facet on.
- `reverse`: If `True`, reverse the sort order.
- `allow_overlap`: If `True`, allow documents to appear in multiple groups
  when they have multiple terms in the field.
- `maptype`: `FacetMap` class for holding results.

```python
paths = FieldFacet("path", reverse=True)
tags = FieldFacet("tag")
results = searcher.search(myquery, sortedby=paths, groupedby=tags)
```

### `ColumnCategorizer`

Categorizer that reads values from a column for sorting. Used when a field
has a column type.

### `ReversedColumnCategorizer`

Categorizer that reverses column values for fields that are not naturally
reversible.

### `OverlappingCategorizer`

```python
class whoosh.sorting.OverlappingCategorizer
```

Categorizer used when `allow_overlap=True`. A single document can belong to
multiple facet groups.

### `PostingCategorizer`

```python
class whoosh.sorting.PostingCategorizer
```

Categorizer for fields without column values. Builds an array caching the
order of all documents. Used as a fallback; prefer setting
`sortable=True` on fields.

### `QueryFacet`

```python
class whoosh.sorting.QueryFacet(
    querydict: dict,
    other=None,
    allow_overlap=False,
    maptype=None
)
```

Sorts/facets based on the results of a series of queries.

**Constructor:**
- `querydict`: Dictionary mapping keys to `Query` objects.
- `other`: Key to use for documents matching no queries.

### `RangeFacet`

```python
class whoosh.sorting.RangeFacet(
    fieldname,
    start,
    end,
    gap,
    hardend=False,
    maptype=None
)
```

Sorts/facets based on numeric ranges. Ranges are inclusive at the start and
exclusive at the end.

```python
prices = RangeFacet("price", 0, 1000, 100)
results = searcher.search(myquery, groupedby=prices)
```

- `fieldname`: The numeric field to facet on.
- `start`: Start of the entire range.
- `end`: End of the entire range.
- `gap`: Size of each bucket (can be a sequence for progressive gaps).
- `hardend`: If `True`, clamp the last bucket to `end`.

### `DateRangeFacet`

```python
class whoosh.sorting.DateRangeFacet(
    fieldname,
    startdate,
    enddate,
    gap,
    hardend=False,
    maptype=None
)
```

Sorts/facets based on date ranges. Extends `RangeFacet` but uses
`datetime` objects for start/end and `timedelta`/`relativedelta` for gaps.
Generates `DateRange` queries instead of `TermRange` queries.

```python
from datetime import datetime
from whoosh.support.relativedelta import relativedelta

startdate = datetime(1920, 1, 1)
enddate = datetime.now()
gap = relativedelta(years=5)
bdays = DateRangeFacet("birthday", startdate, enddate, gap)
```

### `ScoreFacet`

```python
class whoosh.sorting.ScoreFacet
```

Uses a document's relevance score as a sorting criterion.

```python
tag_score = MultiFacet(["tag", ScoreFacet()])
results = searcher.search(myquery, sortedby=tag_score)
```

### `FunctionFacet`

```python
class whoosh.sorting.FunctionFacet(fn)
```

Lets you pass an arbitrary function that computes the sort key. The function
is called with `(searcher, docid)` where `docid` is an absolute index
document number.

```python
fn = lambda s, docid: s.doc_field_length(docid, "content")
lengths = FunctionFacet(fn)
```

### `TranslateFacet`

```python
class whoosh.sorting.TranslateFacet(fn, *facets)
```

Applies a custom function to the key generated by one or more wrapped facets.
Useful for custom collation, such as Unicode Collation Algorithm (UCA) sorting.

```python
from pyuca import Collator

c = Collator("allkeys.txt")
facet = FieldFacet("name")
facet = TranslateFacet(c.sort_key, facet)
results = searcher.search(myquery, sortedby=facet)
```

**Constructor:**
- `fn`: Function applied to the computed key values.
- `*facets`: One or more `FacetType` objects whose keys are passed to `fn`.

### `StoredFieldFacet`

```python
class whoosh.sorting.StoredFieldFacet(
    fieldname,
    allow_overlap=False,
    split_fn=None,
    maptype=None
)
```

Sorts/groups using the value in an unindexed, stored field (e.g., `STORED`).
Usually slower than using an indexed field.

**Constructor:**
- `fieldname`: Name of the stored field.
- `allow_overlap`: If `True`, when grouping, allow documents to appear in
  multiple groups when they have multiple values (split by `split_fn` or
  `string.split()`).
- `split_fn`: Custom function to split a stored field value into multiple
  facet values (only used when `allow_overlap=True`).

### `MultiFacet`

```python
class whoosh.sorting.MultiFacet(items=None, maptype=None)
```

Sorts/facets by the combination of multiple sub-facets.

```python
facet = MultiFacet([FieldFacet("tag"), FieldFacet("path")])
results = searcher.search(myquery, sortedby=facet)
```

Strings in the items list are treated as field names:

```python
facet = MultiFacet(["tag", "path"])
```

**Methods:**
- `from_sortedby(sortedby)`: Class method that creates a `MultiFacet` from
  a field name, facet, or list thereof.
- `add_field(fieldname, reverse=False)`: Add a `FieldFacet`.
- `add_query(querydict, other=None, allow_overlap=False)`: Add a `QueryFacet`.
- `add_score()`: Add a `ScoreFacet`.
- `add_facet(facet)`: Add an arbitrary `FacetType`.

### `Facets`

```python
class whoosh.sorting.Facets(x=None)
```

Maps facet names to `FacetType` objects for creating multiple independent
groupings of documents.

```python
facets = Facets()
facets.add_field("tag")
facets.add_facet("price", RangeFacet("price", 0, 1000, 100))
results = searcher.search(myquery, groupedby=facets)

tag_groups = results.groups("tag")
price_groups = results.groups("price")
```

**Class Methods:**
- `from_groupedby(groupedby)`: Creates a `Facets` object from a field name,
  `FacetType`, dict, list, or another `Facets` object.

**Methods:**
- `names()`: Returns an iterator of facet names.
- `items()`: Returns a list of `(name, facet)` tuples.
- `add_field(fieldname, **kwargs)`: Adds a `FieldFacet`.
- `add_query(name, querydict, **kwargs)`: Adds a `QueryFacet`.
- `add_facet(name, facet)`: Adds a `FacetType` under the given name.
- `add_facets(facets, replace=True)`: Adds the contents of a `Facets` or
  `dict` to this object.

## Facet Maps

### `FacetMap`

```python
class whoosh.sorting.FacetMap
```

Base class for objects holding the results of grouping search results by a
facet. Use `as_dict()` to access results.

```python
myfacet = FieldFacet("size", maptype=OrderedList)
myfacet = FieldFacet("size", maptype=Count)
```

**Methods:**
- `add(groupname, docid, sortkey)`: Adds a document to the facet results.
- `as_dict()`: Returns a dictionary mapping group names to values.

### `OrderedList`

```python
class whoosh.sorting.OrderedList
```

Stores a list of document numbers for each group, in sorted order.

### `UnorderedList`

```python
class whoosh.sorting.UnorderedList
```

Stores a list of document numbers for each group in arbitrary order. Slightly
faster and more memory-efficient than `OrderedList` when ordering doesn't
matter.

### `Count`

```python
class whoosh.sorting.Count
```

Stores the count of documents in each group.

### `Best`

```python
class whoosh.sorting.Best
```

Stores the "best" (highest sort key) document in each group.

## Sorting Utilities

### `add_sortable`

```python
whoosh.sorting.add_sortable(
    writer,
    fieldname,
    facet,
    column=None
)
```

Adds a per-document value column to an existing field, making it sortable.
Useful for retrofitting fields that were created without `sortable=True`.

**Example:**
```python
from whoosh import index, sorting

ix = index.open_dir("indexdir")
with ix.writer() as w:
    facet = sorting.FieldFacet("price")
    sorting.add_sortable(w, "price", facet)
```

**Parameters:**
- `writer`: An `IndexWriter` object.
- `fieldname`: Name of the field to add sortable values to.
- `facet`: A `FacetType` object to generate per-document values.
- `column`: Optional `ColumnType` to store the values. If omitted, uses the
  field's default column type.


## DOCUMENT (FR): Spelling

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Spelling API

Functions and classes for correcting typos in user queries using edit-distance
(Damerau-Levenshtein) matching against the terms in the index.

## Corrector Objects

### `Corrector`

```python
class whoosh.spelling.Corrector
```

Base class for spelling correction objects. Concrete subclasses implement the
`_suggestions()` method.

**Methods:**

#### `suggest(text, limit=5, maxdist=2, prefix=0)`

Returns a list of suggested corrections for `text`, ranked by edit distance
then by frequency.

- `text`: The text to check. Will **not** be added to suggestions even if it
  appears in the index.
- `limit`: Maximum number of suggestions to return.
- `maxdist`: Maximum edit distance to look at (values > 2 are inefficient).
- `prefix`: Require suggestions to share this length of prefix with `text`.
  Increasing to even `1` dramatically speeds up suggestions.

#### `_suggestions(text, maxdist, prefix)`

Low-level method yielding `(score, suggestion)` tuples. Subclasses must
implement this.

### `ReaderCorrector`

```python
class whoosh.spelling.ReaderCorrector(reader, fieldname, fieldobj)
```

Suggests corrections based on terms in a specific field of an `IndexReader`.

**Ranks suggestions by edit distance, then by highest to lowest frequency.**

**Constructor:**
- `reader`: An `IndexReader` object.
- `fieldname`: The name of the field to get suggestions from.
- `fieldobj`: The `FieldType` for the field.

### `ListCorrector`

```python
class whoosh.spelling.ListCorrector(wordlist)
```

Suggests corrections based on a sorted list of strings.

**Constructor:**
- `wordlist`: A sorted list of words to match against.

### `MultiCorrector`

```python
class whoosh.spelling.MultiCorrector(correctors, op)
```

Merges suggestions from a list of sub-correctors.

**Constructor:**
- `correctors`: List of `Corrector` objects.
- `op`: A function (e.g., `max` or `operator.add`) to combine scores from
  multiple correctors for the same suggestion.

## Query Correction

### `Correction`

```python
class whoosh.spelling.Correction(q, qstring, corr_q, tokens)
```

Represents the corrected version of a user query string.

**Attributes:**
- `query`: The corrected `Query` object.
- `string`: The corrected user query string.
- `original_query`: The original `Query` object.
- `original_string`: The original user query string.
- `tokens`: List of token objects representing corrected words.

**Methods:**

#### `format_string(formatter)`

Highlights corrected words in the original query string using the given
`Formatter`.

```python
from whoosh import highlight

correction = searcher.correct_query(q, qstring)
hf = highlight.HtmlFormatter(classname="change")
html = correction.format_string(hf)
```

- `formatter`: A `Formatter` instance (or class, which will be instantiated).
- Returns: Formatted string, typically with corrections emphasized.

### `QueryCorrector`

```python
class whoosh.spelling.QueryCorrector(fieldname)
```

Base class for objects that correct words in a user query.

**Constructor:**
- `fieldname`: The default field name for corrections.

**Methods:**

#### `correct_query(q, qstring)`

Returns a `Correction` object representing the corrected form of the given
query.

- `q`: The original `Query` tree to be corrected.
- `qstring`: The original user query string (may be `None`).
- Returns: A `Correction` object.

#### `field()`

Returns the field name this corrector operates on.

### `SimpleQueryCorrector`

```python
class whoosh.spelling.SimpleQueryCorrector(
    correctors: dict,
    terms: list,
    aliases=None,
    prefix: int = 0,
    maxdist: int = 2
)
```

A simple query corrector based on a mapping of field names to `Corrector`
objects, and a list of `(fieldname, text)` tuples to correct.

**Constructor:**
- `correctors`: Dictionary mapping field names to `Corrector` objects.
- `terms`: Sequence of `(fieldname, text)` tuples representing terms to be
  corrected.
- `aliases`: Dictionary mapping field names in the query to field names for
  spelling suggestions.
- `prefix`: Suggested replacement words must share this number of initial
  characters. Default `0`.
- `maxdist`: Maximum edit distance for suggestions. Values > 2 may be slow.


## DOCUMENT (FR): Writing

# API Écriture

Écrire, mettre à jour et supprimer des documents via l'interface `IndexWriter`.

## IndexWriter

```python
class whoosh.writing.IndexWriter
```

Classe de base pour l'écriture de documents.

### Contexte manager

```python
with ix.writer() as writer:
    writer.add_document(title="Bonjour", content="Monde")
    # commit() appelé automatiquement
```

### Méthodes

#### `add_document(**fields)`

Ajoute un document.

**Kwargs spéciaux:**
- `_stored_<nom_champ>`: Valeur stockée alternative
- `_<nom_champ>_boost`: Boost spécifique au champ
- `_boost`: Boost global du document

---

#### `update_document(**fields)`

Met à jour/remplace un document. Utilise les champs `unique` pour trouver les documents existants.

---

#### `delete_document(docnum, delete=True)`

Supprime par numéro de document.

---

#### `delete_by_term(fieldname, text) -> int`

Supprime tous les documents avec le terme dans le champ.

**Retourne:**
- `int`: Nombre de documents supprimés.

---

#### `delete_by_query(q, searcher=None) -> int`

Supprime les documents correspondant à la requête.

---

#### `commit(mergetype=None, optimize=False, merge=True)`

Commit les changements sur disque.

**Args:**
- `mergetype`: Fonction de fusion personnalisée
- `optimize`: Fusionner tous les segments en un seul
- `merge`: Si False, ne pas fusionner les segments existants

---

#### `cancel()`

Annule les changements et libère le verrou.

---

#### `add_field(fieldname, fieldtype, **kwargs)`

Ajoute un champ (avant d'ajouter des documents).

---

#### `remove_field(fieldname)`

Supprime un champ du schéma.

---

#### `searcher(**kwargs) -> Searcher`

Retourne un searcher (pour lecture pendant la session d'écriture).

---

#### `reader(**kwargs) -> IndexReader`

Retourne un reader pour l'état actuel.

---

#### `group()`

Context manager pour grouper des documents dans un segment.

## SegmentWriter

Implémentation concrète d'IndexWriter.

## AsyncWriter

Writer threaded qui réessaie automatiquement en cas de contention.

```python
from whoosh.writing import AsyncWriter

writer = AsyncWriter(index, delay=0.25, writerargs={})
```

## BufferedWriter

Buffer les documents en mémoire et commit périodiquement.

```python
from whoosh.writing import BufferedWriter

writer = BufferedWriter(
    index,
    period=60,      # Max secondes entre commits
    limit=100       # Max documents par commit
)
```

## Politiques de fusion

```python
from whoosh.writing import NO_MERGE, MERGE_SMALL, OPTIMIZE, CLEAR

writer.commit(mergetype=NO_MERGE)
writer.commit(mergetype=MERGE_SMALL)
writer.commit(mergetype=OPTIMIZE)
writer.commit(mergetype=CLEAR)
```

## Exceptions

### IndexingError

```python
class whoosh.writing.IndexingError(Exception)
```

Levée quand une opération d'indexation échoue.


## DOCUMENT (FR): Analysis

---
title: "Analyseurs"
sidebar_position: 6
Module: whoosh.analysis
Version: 2.7.4
---

:::info
Suite au renommage de `whoosh-reloaded` en `whoosh-ng`, les nouveaux modules spécifiques à Whoosh-NG se trouvent généralement sous `whoosh_modern`.
Les composants Whoosh core (comme `whoosh.analysis`, `whoosh.index`) restent accessibles directement sous l'espace de noms `whoosh` pour la rétrocompatibilité.
:::

# Analyseurs

## Overview

Un analyseur est une fonction ou une classe callable (une classe avec une méthode `__call__`) qui prend une chaîne unicode et retourne un générateur de tokens. En général, un "token" est un mot, par exemple la chaîne "Mary had a little lamb" peut produire les tokens "Mary", "had", "a", "little", et "lamb". Cependant, les tokens ne sont pas toujours des mots ; cela dépend de l'analyseur utilisé.

## Tokenizers

Un tokenizer est un type d'analyseur qui divise une chaîne de caractères en tokens.

Par exemple, la classe `whoosh.analysis.RegexTokenizer` implémente une expression régulière pour diviser le texte en tokens :

```python
from whoosh.analysis import RegexTokenizer

tokenizer = RegexTokenizer()
for token in tokenizer("Hello there my friend!"):
    print(repr(token.text))
# u'Hello'
# u'there'
# u'my'
# u'friend'
```

## Filtres

Un filtre est un callable qui prend un générateur de Tokens (soit un tokenizer, soit un autre filtre) et retourne à son tour une série de Tokens.

Par exemple, le `whoosh.analysis.LowercaseFilter()` fourni filtre les tokens en convertissant leur texte en minuscules. L'implémentation est très simple :

```python
def LowercaseFilter(tokens):
    """Utilise lower() pour mettre le texte des tokens en minuscules."""
    for t in tokens:
        t.text = t.text.lower()
        yield t
```

Vous pouvez envelopper le filtre autour d'un tokenizer pour le voir en action :

```python
from whoosh.analysis import LowercaseFilter, RegexTokenizer

tokenizer = RegexTokenizer()
for token in LowercaseFilter(tokenizer("These ARE the things I want!")):
    print(repr(token.text))
# u'these'
# u'are'
# u'the'
# u'things'
# u'i'
# u'want'
```

## Analyseurs

Un analyseur est simplement un moyen de combiner un tokenizer et quelques filtres en un seul package.

Vous pouvez implémenter un analyseur comme une classe ou fonction personnalisée, ou composer des tokenizers et filtres en utilisant le caractère `|` :

```python
my_analyzer = RegexTokenizer() | LowercaseFilter() | StopFilter()
```

## Voir aussi

- [API: analysis](../api/analysis) — Référence complète pour les analyseurs, tokenizers et filtres


## DOCUMENT (FR): Backends

# Backends

Whoosh-NG supporte des backends de stockage pluggables via l'architecture Provider. Le backend par défaut stocke les données comme fichiers sur disque, mais vous pouvez utiliser SQLite, PostgreSQL, S3, et plus encore.

## Backends intégrés

| Backend | Description |
|---------|-------------|
| Fichier (défaut) | Stocke l'index comme fichiers sur disque |
| SQLite | Stocke l'index dans une base SQLite |
| Mémoire | Backend en mémoire (tests uniquement) |

## Backend Fichier (défaut)

```python
from whoosh.index import create_in

# Utilise FileBackend par défaut
ix = create_in("indexdir", schema)
```

## Backend SQLite

```python
from whoosh.backends.sqlite import SQLiteBackend
from whoosh.store.sqlite import SQLiteStorage

storage = SQLiteStorage("index.db")
backend = SQLiteBackend(storage=storage)
```

### Avantages

- Index en un seul fichier
- Meilleur pour les charges transactionnelles
- Sauvegardes plus faciles
- Supporte les lectures concurrentes

## Backend Mémoire

```python
from whoosh.backends.memory import MemoryBackend

backend = MemoryBackend()  # Utile pour les tests
```

## Bonnes pratiques

1. **File backend pour production**: Le plus éprouvé
2. **SQLite pour déploiement mono-fichier**: Plus facile à déployer
3. **Mémoire pour les tests**: Rapide, pas de nettoyage nécessaire
4. **Fichiers composés**: Activez pour réduire le nombre de fichiers
5. **Stratégie de sauvegarde**: File = copier le répertoire; SQLite = copier le fichier


## DOCUMENT (FR): Batch

---
title: "Batch Indexing Performance"
sidebar_position: 15
Module: whoosh.index, whoosh.writing
Version: 2.7.4
---
> **Note de traduction** : Cette page n'est pas encore traduite en francais.
> Le contenu anglais est affiche ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Tips for speeding up batch indexing

## Overview

Indexing documents tends to fall into two general patterns: adding documents
one at a time as they are created (as in a web application), and adding a bunch
of documents at once (batch indexing).

The following settings and alternate workflows can make batch indexing faster.

## StemmingAnalyzer cache

The stemming analyzer by default uses a least-recently-used (LRU) cache to
limit the amount of memory it uses, to prevent the cache from growing very
large if the analyzer is reused for a long period of time. However, the LRU
cache can slow down indexing by almost 200% compared to a stemming analyzer
with an "unbounded" cache.

When you're indexing in large batches with a one-shot instance of the analyzer,
consider using an unbounded cache:

> **Note**: For new implementations or complex multilingual scenarios, consider
> using the `CachedStemmingAnalyzer` (from `whoosh_modern.analysis.cached_stemming_analyzer`)
> which offers integrated LRU caching and flexible configuration.

```python
w = myindex.writer()
# Get the analyzer object from a text field
stem_ana = w.schema["content"].analyzer
# Set the cachesize to -1 to indicate unbounded caching
stem_ana.cachesize = -1
# Reset the analyzer to pick up the changed attribute
stem_ana.clear()

# Use the writer to index documents...
```

## The `limitmb` parameter

The `limitmb` parameter to `whoosh.index.Index.writer()` controls the
*maximum* memory (in megabytes) the writer will use for the indexing pool. The
higher the number, the faster indexing will be.

The default value of `128` is actually somewhat low, considering many people
have multiple gigabytes of RAM these days. Setting it higher can speed up
indexing considerably:

```python
from whoosh import index

ix = index.open_dir("indexdir")
writer = ix.writer(limitmb=256)
```

> The actual memory used will be higher than this value because of interpreter
> overhead (up to twice as much!). It is very useful as a tuning parameter, but
> not for trying to exactly control the memory usage of Whoosh.

## The `procs` parameter

The `procs` parameter to `whoosh.index.Index.writer()` controls the number of
processors the writer will use for indexing (via the `multiprocessing` module):

```python
from whoosh import index

ix = index.open_dir("indexdir")
writer = ix.writer(procs=4)
```

When you use multiprocessing, the `limitmb` parameter controls the amount of
memory used by *each process*, so the actual memory used will be
`limitmb * procs`:

```python
# Each process will use a limit of 128, for a total of 512
writer = ix.writer(procs=4, limitmb=128)
```

## The `multisegment` parameter

The `procs` parameter causes the default writer to use multiple processors to
do much of the indexing, but then still uses a single process to merge the pool
of each sub-writer into a single segment.

You can get much better indexing speed by also using the `multisegment=True`
keyword argument, which instead of merging the results of each sub-writer,
simply has them each just write out a new segment:

```python
from whoosh import index

ix = index.open_dir("indexdir")
writer = ix.writer(procs=4, multisegment=True)
```

The drawback is that instead of creating a single new segment, this option
creates a number of new segments **at least** equal to the number of processors
you use. For example, if you use `procs=4`, the writer will create four new
segments.

So, while `multisegment=True` is much faster than a normal writer, you should
only use it for large batch indexing jobs (or perhaps only for indexing from
scratch). It should not be the only method you use for indexing, because
otherwise the number of segments will tend to increase forever!

## See also

- [Indexing](/core/indexing) — Writer options and merge policies
- [API: writing](../api/writing) — `Index.writer()` parameters


## DOCUMENT (FR): Changelog

# Historique des modifications

Notes de version pour Whoosh-NG, generees automatiquement a partir des releases GitHub et des messages de commits.

## v5.3.1 (2026-08-13)
**Tag**: `v5.3.1`

## v5.3.1 (2026-08-13)

_This release is published under the MIT License._

### Bug Fixes

- **linguistics**: Rendre l'import langdetect robuste pour pyright ([`54ab70e`](https://github.com/dorel14/whoosh-ng/commit/54ab70e9247c3ecfdebef592f1107dc794921194))

- **tooling**: Corriger le type:ignore de langdetect pour mypy/pyright ([`e37be1c`](https://github.com/dorel14/whoosh-ng/commit/e37be1ce62ff958011983f8ed3efded56428fda8))

### Documentation

- Update changelog and API docs [skip ci] ([`d884547`](https://github.com/dorel14/whoosh-ng/commit/d884547c17bcaea800e7e2277bcdfc25753ad446))

---

**Detailed Changes**: [v5.3.0...v5.3.1](https://github.com/dorel14/whoosh-ng/compare/v5.3.0...v5.3.1)

### Commits

### Chores

- merge master into dev [skip ci]
- apply pre-commit fixes
- v5.3.1 [skip ci]

### Other

- .
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge pull request #455 from dorel14/dev

### Bug Fixes

- corriger le type:ignore de langdetect pour mypy/pyright
- rendre l'import langdetect robuste pour pyright

### Documentation

- update changelog and API docs [skip ci]


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v5.3.1)

## v5.3.0 (2026-08-13)
**Tag**: `v5.3.0`

## v5.3.0 (2026-08-13)

_This release is published under the MIT License._

### Bug Fixes

- **linguistics**: Corriger les défauts signalés par revue et docs ([`ba05a8d`](https://github.com/dorel14/whoosh-ng/commit/ba05a8d5cbdba8d71f39d8e24920d55f77a8eb32))

### Documentation

- Auto-update llms context files [skip ci] ([`4585b5e`](https://github.com/dorel14/whoosh-ng/commit/4585b5eb665c0329c70da0efe5225b859fc14573))

- Auto-update llms context files [skip ci] ([`c897f1f`](https://github.com/dorel14/whoosh-ng/commit/c897f1f0e6c6bbebee46a7672f5ca6d3d6abebf4))

- Auto-update llms context files [skip ci] ([`a65b926`](https://github.com/dorel14/whoosh-ng/commit/a65b926027472d65244c76f250833af27c4bb8e5))

- Auto-update llms context files [skip ci] ([`4911cf5`](https://github.com/dorel14/whoosh-ng/commit/4911cf57abc555eadfa47cf962943b321fb2457a))

- Auto-update llms context files [skip ci] ([`8605b34`](https://github.com/dorel14/whoosh-ng/commit/8605b344f85f3e40dec93c9f9c7d04e7c10fd9f6))

- Auto-update llms context files [skip ci] ([`8c88da5`](https://github.com/dorel14/whoosh-ng/commit/8c88da5d29097f8a4ee6a9c47ffcc5afdcf2ff55))

- Update changelog and API docs [skip ci] ([`ae57714`](https://github.com/dorel14/whoosh-ng/commit/ae57714558186480f0f15e4aa21e093c61c166bd))

- **config**: Corriger l'exemple ConfigEngine.build() ([#422](https://github.com/dorel14/whoosh-ng/pull/422), [`f09d6da`](https://github.com/dorel14/whoosh-ng/commit/f09d6dac72482868275ad581046909a5b4169885))

- **documentation**: Mettre à jour la documentation pour le renommage en whoosh-ng ([`ec35fbd`](https://github.com/dorel14/whoosh-ng/commit/ec35fbd5182dc0c01e5888d7285f62be8c21a262))

- **linguistics**: Documenter la détection automatique de langue et l'indexation multilingue ([`f86a17b`](https://github.com/dorel14/whoosh-ng/commit/f86a17bd7820a2e5a3f99fc8a8516c4b9b460a18))

- **website**: Restructurer la documentation et déplacer les fonctionnalités classiques vers Core ([`54fb1f6`](https://github.com/dorel14/whoosh-ng/commit/54fb1f61f81b441febda3a0ecfce3fef8edb400f))

### Features

- **analysis**: Add EPIC 5.6 Modern N-Gram Framework Sprint NGRAM-1 ([#440](https://github.com/dorel14/whoosh-ng/pull/440), [`c634718`](https://github.com/dorel14/whoosh-ng/commit/c6347184bb2b6d307da5008c1edf4850eab88d0b))

- **api**: Exposer les nouvelles fonctionnalités pour la version 5.1.0 ([`e306236`](https://github.com/dorel14/whoosh-ng/commit/e306236a71fdcd5f018d9af10cf923398ed852c1))

- **linguistics**: Add language auto-detection for multilingual indexing ([#434](https://github.com/dorel14/whoosh-ng/pull/434), [`a91d6bf`](https://github.com/dorel14/whoosh-ng/commit/a91d6bf59b8faae98fe80aa4d6c5ac4e0813ec0e))

- **linguistics**: Add language auto-detection in SearchApplication ([#435](https://github.com/dorel14/whoosh-ng/pull/435), [`99f4290`](https://github.com/dorel14/whoosh-ng/commit/99f42904e302191de630cfdefe8eec6f5a8b50ca))

- **linguistics**: Add Wiktionary dictionary indexer (#429 #432) ([`726542d`](https://github.com/dorel14/whoosh-ng/commit/726542dd4bf8e79d35dadb36e126f1973207a8b4))

- **linguistics**: Add Wiktionary synonym provider (#425 #426 #427) ([`2961b93`](https://github.com/dorel14/whoosh-ng/commit/2961b93726cf1792faf89b293f2439c5c92e7ee6))

- **linguistics**: Integrate Wiktionary indexer with SearchApplication ([#433](https://github.com/dorel14/whoosh-ng/pull/433), [`f201f5c`](https://github.com/dorel14/whoosh-ng/commit/f201f5c37608df75dc79be5b2526fb71fa45ed36))

- **vector**: Add HnswlibProvider for EPIC 9 Sprint VEC-1 ([#406](https://github.com/dorel14/whoosh-ng/pull/406), [`cd590d4`](https://github.com/dorel14/whoosh-ng/commit/cd590d4601e5c238385be6fe693ee22ebe4a0912))

- **vector,embeddings,config**: Add HnswlibProvider, EmbeddingProvider, ConfigEngine extensions (#406 #407) ([`03a15aa`](https://github.com/dorel14/whoosh-ng/commit/03a15aabcb32822a71635a17b99e9c9545e07962))

---

**Detailed Changes**: [v5.2.0...v5.3.0](https://github.com/dorel14/whoosh-ng/compare/v5.2.0...v5.3.0)

### Commits

### Chores

- merge master into dev [skip ci]
- améliorer la qualité du code et nettoyer la configuration
- v5.3.0 [skip ci]

### Documentation

- update changelog and API docs [skip ci]
- corriger l'exemple ConfigEngine.build() (#422)
- auto-update llms context files [skip ci]
- restructurer la documentation et déplacer les fonctionnalités classiques vers Core
- auto-update llms context files [skip ci]
- documenter la détection automatique de langue et l'indexation multilingue
- auto-update llms context files [skip ci]
- auto-update llms context files [skip ci]
- mettre à jour la documentation pour le renommage en whoosh-ng
- auto-update llms context files [skip ci]
- auto-update llms context files [skip ci]

### Features

- add Wiktionary synonym provider (#425 #426 #427)
- add Wiktionary dictionary indexer (#429 #432)
- integrate Wiktionary indexer with SearchApplication (#433)
- add language auto-detection for multilingual indexing (#434)
- add language auto-detection in SearchApplication (#435)
- add HnswlibProvider for EPIC 9 Sprint VEC-1 (#406)
- add HnswlibProvider, EmbeddingProvider, ConfigEngine extensions (#406 #407)
- add EPIC 5.6 Modern N-Gram Framework Sprint NGRAM-1 (#440)
- exposer les nouvelles fonctionnalités pour la version 5.1.0

### Other

- eat(linguistics): complete EPIC 6.7 Language Auto-Detection & Multilingual Indexing (#430)
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge pull request #454 from dorel14/dev

### Code Refactoring

- améliorer le typage et nettoyer les imports

### Bug Fixes

- corriger les défauts signalés par revue et docs


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v5.3.0)

## v5.2.0 (2026-08-12)
**Tag**: `v5.2.0`

## v5.2.0 (2026-08-12)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **config**: Corriger le passage de booléen à StandardAnalyzer.stoplist ([#422](https://github.com/dorel14/whoosh-ng/pull/422), [`7c3649c`](https://github.com/dorel14/whoosh-ng/commit/7c3649c268d0551793c52f5855b6f1b7cab06a01))

- **config**: Corriger les bugs et améliorer les tests et la documentation ([#422](https://github.com/dorel14/whoosh-ng/pull/422), [`377a531`](https://github.com/dorel14/whoosh-ng/commit/377a531a9e20c7534bb8c54803fd91910b8e2323))

- **config**: Corriger pyright, unused imports et StandardAnalyzer.stoplist ([#422](https://github.com/dorel14/whoosh-ng/pull/422), [`10192bb`](https://github.com/dorel14/whoosh-ng/commit/10192bbd8c7824c3ccd89807c5bc6699c52a29b7))

### Documentation

- Auto-update llms context files [skip ci] ([`d188581`](https://github.com/dorel14/whoosh-ng/commit/d188581f5ca7fd71182ac26f71d06e0d045c2f0f))

- Auto-update llms context files [skip ci] ([`599c979`](https://github.com/dorel14/whoosh-ng/commit/599c979675149203172e73efa587578ffa4dab4f))

- Auto-update llms context files [skip ci] ([`7405bd0`](https://github.com/dorel14/whoosh-ng/commit/7405bd03e8445f547830b461de274d1f73a84cb9))

- Update changelog and API docs [skip ci] ([`9bc3cfd`](https://github.com/dorel14/whoosh-ng/commit/9bc3cfd033c2999aa6dd73ff15bc992c9883dccf))

- Update changelog and API docs [skip ci] ([`2920f65`](https://github.com/dorel14/whoosh-ng/commit/2920f653921efae2075911bd0f9c9a3de43c6808))

- Update changelog and API docs [skip ci] ([`dc32e6f`](https://github.com/dorel14/whoosh-ng/commit/dc32e6fb7b16659c2757ce86ed38c6ee43b8b9df))

### Features

- **config**: Add ConfigEngine sub-engines for Sprint CFG-1 ([#422](https://github.com/dorel14/whoosh-ng/pull/422), [`47c59b4`](https://github.com/dorel14/whoosh-ng/commit/47c59b4d0d66f2105de4de3e0dee5501133ff96e))

- **config**: Add ConfigEngine.build() orchestration for Sprint CFG-1 ([#422](https://github.com/dorel14/whoosh-ng/pull/422), [`1329b42`](https://github.com/dorel14/whoosh-ng/commit/1329b4296a35e2b115580bcad95ac83a394e8911))

- **config**: Ajouter la mise en cache et valider les sources de données ([`add4bb8`](https://github.com/dorel14/whoosh-ng/commit/add4bb8cc4b8d7fcaf4ed06b5c20b93bd3fa312c))

---

**Detailed Changes**: [v5.1.0...v5.2.0](https://github.com/dorel14/whoosh-ng/compare/v5.1.0...v5.2.0)

### Commits

### Chores

- merge master into dev [skip ci]
- synchronize version from pyproject.toml [skip ci]
- merge master into dev [skip ci]
- merge master into dev [skip ci]
- merge master into dev [skip ci]
- synchronize version from pyproject.toml [skip ci]
- apply pre-commit fixes
- v5.2.0 [skip ci]

### Other

- Delete LICENSE_OLD.txt
- Add MIT License to the project
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge pull request #424 from dorel14/dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- style(codebase): uniformiser le formatage du code et améliorer la lisibilité
- .
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- .
- Merge pull request #428 from dorel14/dev

### CI/CD

- restructurer la génération du changelog et simplifier Pages
- documenter le jeton GH_TOKEN et ajouter pydoctor
- documenter le jeton GH_TOKEN et ajouter pydoctor (#422)
- ajouter pydoctor, documenter GH_TOKEN et ignorer les benchmarks JSON (#422)
- factoriser should_release et rendre l'i18n dynamique (#422)

### Documentation

- auto-update llms context files [skip ci]
- update changelog and API docs [skip ci]
- update changelog and API docs [skip ci]
- update changelog and API docs [skip ci]
- auto-update llms context files [skip ci]
- auto-update llms context files [skip ci]

### Code Refactoring

- restructurer la chaîne de déploiement et supprimer les workflows obsolètes
- supprimer le benchmark stemmer français et moderniser les chemins

### Features

- add ConfigEngine sub-engines for Sprint CFG-1 (#422)
- add ConfigEngine.build() orchestration for Sprint CFG-1 (#422)
- ajouter la mise en cache et valider les sources de données

### Bug Fixes

- corriger le passage de booléen à StandardAnalyzer.stoplist (#422)
- corriger les bugs et améliorer les tests et la documentation (#422)
- corriger pyright, unused imports et StandardAnalyzer.stoplist (#422)


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v5.2.0)

## v5.1.0 (2026-08-11)
**Tag**: `v5.1.0`

## v5.1.0 (2026-08-11)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **config,linguistics**: Resolve mypy/pyright issues in config loader and yaml provider ([`3f59f96`](https://github.com/dorel14/whoosh-ng/commit/3f59f9607398547225b090a2eddc0fc7e1f62efd))

- **fastapi**: Correct WebSocket autocomplete test assertion ([`b9b1d57`](https://github.com/dorel14/whoosh-ng/commit/b9b1d57524b6f121bff60833c6b5563264bc4152))

- **storage**: Validate SnapshotStorage key before remote read ([`3ec0b43`](https://github.com/dorel14/whoosh-ng/commit/3ec0b43120e05578a4191d25e0d66b10ab06b914))

- **tests**: Import CoreStorageAdapter in test_storage_providers.py ([`18484b0`](https://github.com/dorel14/whoosh-ng/commit/18484b00c2e3f66e1baab76349d28694a3fde45a))

### Documentation

- Add Configuration Engine docs, update CHANGELOG, FastAPI WebSocket, CoreStorageAdapter, and SnapshotStorage fix ([`5e8ec4a`](https://github.com/dorel14/whoosh-ng/commit/5e8ec4ae49aaa64d66c554751f7cd67d9ca37f31))

- Auto-update llms context files ([`916bb34`](https://github.com/dorel14/whoosh-ng/commit/916bb34e16c4b5adf7cf6cc7a1449864db60873a))

- Auto-update llms context files ([`5b6b9a6`](https://github.com/dorel14/whoosh-ng/commit/5b6b9a63346559febe51c16a23bbe1602188a2c2))

- Auto-update llms context files ([`25f4477`](https://github.com/dorel14/whoosh-ng/commit/25f4477091e5506c58bfe6787e2fa7a6fcc22943))

- Auto-update llms context files [skip ci] ([`127bc4a`](https://github.com/dorel14/whoosh-ng/commit/127bc4aa3277c109f73d932e878be90807c4aafc))

- **config**: Clarify list merge behavior, refactor PyYAML import, document pre-push hook rationale ([`3d122d5`](https://github.com/dorel14/whoosh-ng/commit/3d122d58b85d10b85bf079ad9d3d15acdeab09e2))

- **config**: Clarify list merge rationale and make unsupported format error dynamic ([`25076b6`](https://github.com/dorel14/whoosh-ng/commit/25076b61ab91df44cd81327bf827732325fc1aab))

- **fastapi,config**: Clarify optional FastAPI dependency and warn on list merge behavior ([`e917567`](https://github.com/dorel14/whoosh-ng/commit/e917567509d1b1d84bb27b7ceb75b1b8c15941a6))

### Features

- Add PyYAML extra, make WebSocket limit configurable, validate ConfigEngine priority ([`5fb4a38`](https://github.com/dorel14/whoosh-ng/commit/5fb4a382ccda5e6408ec93ce1e6e2efa8fd583d7))

- **config**: Implement Configuration Engine core with Pydantic models, YAML/JSON loader, and hierarchical merging ([`de7bbca`](https://github.com/dorel14/whoosh-ng/commit/de7bbcadd384391adef090f1944865490b93d8fe))

- **fastapi**: Make WebSocket limit configurable and run autocomplete off the event loop ([`c538ea6`](https://github.com/dorel14/whoosh-ng/commit/c538ea6af2195bd292035c5de3e520577f090bc4))

- **storage**: Add CoreStorageAdapter wrapping core FileStorage for SyncStorageProvider ([`52ab7e6`](https://github.com/dorel14/whoosh-ng/commit/52ab7e614de1f92a7ac8d7afbf95bbc52aaba0e2))

- **website**: Ajouter les fichiers de configuration du site statique ([`103fcff`](https://github.com/dorel14/whoosh-ng/commit/103fcff73199a4e8e126f7f887d0f31bcd003999))

---

**Detailed Changes**: [v5.0.0...v5.1.0](https://github.com/dorel14/whoosh-ng/compare/v5.0.0...v5.1.0)

### Commits

### Code Refactoring

- Sprint D cleanup + P1-1/P3 dedup fixes
- improve WebSocket error handling and document list merge behavior

### Features

- add CoreStorageAdapter wrapping core FileStorage for SyncStorageProvider
- implement Configuration Engine core with Pydantic models, YAML/JSON loader, and hierarchical merging
- ajouter les fichiers de configuration du site statique
- add PyYAML extra, make WebSocket limit configurable, validate ConfigEngine priority
- make WebSocket limit configurable and run autocomplete off the event loop

### Bug Fixes

- correct WebSocket autocomplete test assertion
- validate SnapshotStorage key before remote read
- import CoreStorageAdapter in test_storage_providers.py
- resolve mypy/pyright issues in config loader and yaml provider

### Other

- .
- Rename LICENSE.txt to LICENSE_OLD.txt
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge pull request #418 from dorel14/dev

### Chores

- merge master into dev [skip ci]
- synchronize version from pyproject.toml [skip ci]
- apply pre-commit fixes
- apply pre-commit fixes
- v5.1.0 [skip ci]

### Documentation

- add Configuration Engine docs, update CHANGELOG, FastAPI WebSocket, CoreStorageAdapter, and SnapshotStorage fix
- auto-update llms context files
- auto-update llms context files
- clarify list merge behavior, refactor PyYAML import, document pre-push hook rationale
- auto-update llms context files
- clarify list merge rationale and make unsupported format error dynamic
- clarify optional FastAPI dependency and warn on list merge behavior
- auto-update llms context files [skip ci]

### CI/CD

- regenerate LLM context files only on documentation changes


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v5.1.0)

## v5.0.0 (2026-08-11)
**Tag**: `v5.0.0`

## v5.0.0 (2026-08-11)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **ci**: Ensure git push runs even when pre-commit commit is a no-op ([`b4fa710`](https://github.com/dorel14/whoosh-ng/commit/b4fa710bfb979a764373b1b9dc67e60f0a84ff0e))

- **ci**: Restore || true suppression for git pull --rebase in test workflow ([`6b8ae46`](https://github.com/dorel14/whoosh-ng/commit/6b8ae46f08908aa132b55261643589993de11ba9))

- **indexing**: Avoid NameError when ix.writer() fails in ParallelIndexBuilder ([`b629fed`](https://github.com/dorel14/whoosh-ng/commit/b629feda7398ece400636ef1ab7bf1df742c4255))

- **linguistics**: Restore constructor-style calls for language analyzers ([`8307731`](https://github.com/dorel14/whoosh-ng/commit/8307731b5958654307f6014743fe1e39700ba547))

- **s3**: Corriger l'ordre de validation des chemins dans SnapshotStorage ([`4fdf384`](https://github.com/dorel14/whoosh-ng/commit/4fdf384158249d66937f16976307db01f232ddf0))

- **storage**: Sanitize S3 keys in SnapshotStorage.read to prevent path traversal ([`b57b00d`](https://github.com/dorel14/whoosh-ng/commit/b57b00d84e2adc8d73e83a93bdc9813eeeff69bf))

### Documentation

- Auto-update llms context files ([`6318ddd`](https://github.com/dorel14/whoosh-ng/commit/6318dddda1db67f1d9ffc1492038340eafd2b62e))

- Auto-update llms context files ([`eb618ab`](https://github.com/dorel14/whoosh-ng/commit/eb618ab65aceb212b417a09d5c2ca74e3296bd0d))

### Features

- **workflows**: Amélioration des workflows CI et ajout de la section LLM Context ([`359e19a`](https://github.com/dorel14/whoosh-ng/commit/359e19a79002a1b67ea8faeef868a0cbe877612b))

---

**Detailed Changes**: [v4.3.0...v5.0.0](https://github.com/dorel14/whoosh-ng/compare/v4.3.0...v5.0.0)

### Commits

### Other

- Merge pull request #15 from dorel14/master
- Merge branch 'master' into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'master' into feat/last_updates
- Merge pull request #417 from dorel14/feat/last_updates

### Documentation

- auto-update llms context files
- rename sprint-c/d docs, add provider-integration guide, sync website docs
- auto-update llms context files
- auto-update llms context files
- auto-update llms context files
- auto-update llms context files

### Chores

- synchronize version from pyproject.toml [skip ci]
- synchronize version from pyproject.toml [skip ci]
- apply pre-commit fixes
- synchronize version from pyproject.toml [skip ci]
- v5.0.0 [skip ci]

### Code Refactoring

- modernisation architecturale et unification sur les composants core
- améliorations de la sécurité S3 et du typage statique

### CI/CD

- ajout de la détection des doublons de code

### Bug Fixes

- sanitize S3 keys in SnapshotStorage.read to prevent path traversal
- restore constructor-style calls for language analyzers
- corriger l'ordre de validation des chemins dans SnapshotStorage
- ensure git push runs even when pre-commit commit is a no-op
- restore || true suppression for git pull --rebase in test workflow
- avoid NameError when ix.writer() fails in ParallelIndexBuilder

### Features

- amélioration des workflows CI et ajout de la section LLM Context


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v5.0.0)

## v4.3.0 (2026-08-09)
**Tag**: `v4.3.0`

## v4.3.0 (2026-08-09)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- Add actions: read permission to workflow_run triggers ([`d1cb49c`](https://github.com/dorel14/whoosh-ng/commit/d1cb49c7d5d84b4d790090d8b24250bb0bf5791b))

- Normalize line endings to LF in workflow files ([`d68513b`](https://github.com/dorel14/whoosh-ng/commit/d68513b84b97115f1708412e731c6bfcc7a2b6b1))

- Use repository_dispatch instead of workflow_run for Pages trigger ([`c601ccb`](https://github.com/dorel14/whoosh-ng/commit/c601ccbc6469b4048083ab71c334f937422b2b2f))

- **pages**: Ensure Pages deploys on all master pushes + changelog sidebar fix ([`d5d297f`](https://github.com/dorel14/whoosh-ng/commit/d5d297f17ef941e5575f7f0ac7f5c99d0c188f76))

### Features

- **website**: Ajouter des sidebars dédiées et refondre la page de référence API ([`a69d3c9`](https://github.com/dorel14/whoosh-ng/commit/a69d3c9c9093fa78c246010f79d751ad36c8b9b0))

- **website**: Ajouter des sidebars dédiées par section de documentation ([`ffaf1a9`](https://github.com/dorel14/whoosh-ng/commit/ffaf1a969c53c80fa2d968f5f733b40ae9e2cc00))

---

**Detailed Changes**: [v4.2.3...v4.3.0](https://github.com/dorel14/whoosh-ng/compare/v4.2.3...v4.3.0)

### Commits

### Chores

- synchronize version from pyproject.toml [skip ci]
- synchronize version from pyproject.toml [skip ci]
- apply pre-commit fixes
- synchronize version from pyproject.toml [skip ci]
- v4.3.0 [skip ci]

### CI/CD

- restructurer la chaîne de déploiement CI/CD et optimiser les workflows
- ajouter des garde-fous et corriger le script de synchronisation de version
- améliorer la logique de déclenchement du déploiement GitHub Pages
- ajouter un déclenchement push pour les modifications de documentation

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng

### Features

- ajouter des sidebars dédiées par section de documentation
- ajouter des sidebars dédiées et refondre la page de référence API

### Code Refactoring

- simplifier la configuration des sidebars

### Bug Fixes

- normalize line endings to LF in workflow files
- ensure Pages deploys on all master pushes + changelog sidebar fix
- add actions: read permission to workflow_run triggers
- use repository_dispatch instead of workflow_run for Pages trigger


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v4.3.0)

## v4.2.3 (2026-08-08)
**Tag**: `v4.2.3`

## v4.2.3 (2026-08-08)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **ci**: Add concurrency groups to prevent workflow race conditions ([`76e38f0`](https://github.com/dorel14/whoosh-ng/commit/76e38f03045d19b8d8032f066432af4e9d17d133))

---

**Detailed Changes**: [v4.2.2...v4.2.3](https://github.com/dorel14/whoosh-ng/compare/v4.2.2...v4.2.3)

### Commits

### Chores

- synchronize version from pyproject.toml [skip ci]
- sync version to 4.2.1 [skip ci]
- v4.2.3 [skip ci]

### Other

- Merge: resolve sync-version conflicts (CI and local generated same changes)

### Bug Fixes

- add concurrency groups to prevent workflow race conditions


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v4.2.3)

## v4.2.2 (2026-08-08)
**Tag**: `v4.2.2`

## v4.2.2 (2026-08-08)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **ci**: Fix changelog workflow detached HEAD ([`db4b78e`](https://github.com/dorel14/whoosh-ng/commit/db4b78e6dc51a0e10fde48fd93eb2ce5d1dbecd0))

---

**Detailed Changes**: [v4.2.1...v4.2.2](https://github.com/dorel14/whoosh-ng/compare/v4.2.1...v4.2.2)

### Commits

### Chores

- synchronize version from pyproject.toml [skip ci]
- v4.2.2 [skip ci]

### Bug Fixes

- fix changelog workflow detached HEAD


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v4.2.2)

## v4.2.1 (2026-08-07)
**Tag**: `v4.2.1`

## v4.2.1 (2026-08-07)

_This release is published under the BSD-2-Clause License._

---

**Detailed Changes**: [v4.2.0...v4.2.1](https://github.com/dorel14/whoosh-ng/compare/v4.2.0...v4.2.1)

### Commits

### Bug Fixes

- fix sync-changelog.yml YAML syntax error

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng
- i18n(fr): translate nested.md, storage-providers.md, stemming.md, ngrams.md, glossary.md

### Chores

- synchronize version to 4.2.0 [skip ci]
- synchronize version from pyproject.toml [skip ci]
- v4.2.1 [skip ci]


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v4.2.1)

## v4.2.0 (2026-08-07)
**Tag**: `v4.2.0`

## v4.2.0 (2026-08-07)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **ci**: Fix changelog workflow typo and pages.yml checkout depth ([`d88b82b`](https://github.com/dorel14/whoosh-ng/commit/d88b82b68f7ea2ed28c4530cbda1ecad77308acf))

### Features

- **docs**: Enable dark theme by default and remove edit button ([`2c3d98b`](https://github.com/dorel14/whoosh-ng/commit/2c3d98b27087125073729505cba156d1c2578d66))

---

**Detailed Changes**: [v4.1.0...v4.2.0](https://github.com/dorel14/whoosh-ng/compare/v4.1.0...v4.2.0)

### Commits

### Chores

- synchronize version from pyproject.toml [skip ci]
- v4.2.0 [skip ci]

### Features

- enable dark theme by default and remove edit button

### Bug Fixes

- fix changelog workflow typo and pages.yml checkout depth


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v4.2.0)

## v4.1.0 (2026-08-07)
**Tag**: `v4.1.0`

## v4.1.0 (2026-08-07)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **ci**: Fix broken workflows and Docusaurus build errors after migration ([`0e86e6c`](https://github.com/dorel14/whoosh-ng/commit/0e86e6c51e3d41fec984f0f79a4457e9d5d0b1ef))

- **ci**: Fix sync_version.py NameError and missing imports ([`f809f56`](https://github.com/dorel14/whoosh-ng/commit/f809f563851b102c7220cec0371c560191bdb217))

### Features

- **docs**: Migrate Jekyll/Just the Docs to Docusaurus v3 ([`7e70791`](https://github.com/dorel14/whoosh-ng/commit/7e70791d23fb0f3097e3603ba0ff3fa5c8d822c2))

---

**Detailed Changes**: [v4.0.1...v4.1.0](https://github.com/dorel14/whoosh-ng/compare/v4.0.1...v4.1.0)

### Commits

### Chores

- synchronize version from pyproject.toml [skip ci]
- v4.1.0 [skip ci]

### Features

- migrate Jekyll/Just the Docs to Docusaurus v3

### Bug Fixes

- fix broken workflows and Docusaurus build errors after migration
- fix sync_version.py NameError and missing imports


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v4.1.0)

## v4.0.1 (2026-08-07)
**Tag**: `v4.0.1`

## v4.0.1 (2026-08-07)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **pages**: Pointer Bundler sur docs/Gemfile via BUNDLE_GEMFILE ([`0f3a8f1`](https://github.com/dorel14/whoosh-ng/commit/0f3a8f11087c5df910d1762f31ae1cee4017ed31))

---

**Detailed Changes**: [v4.0.0...v4.0.1](https://github.com/dorel14/whoosh-ng/compare/v4.0.0...v4.0.1)

### Commits

### Chores

- synchronize version from pyproject.toml [skip ci]
- v4.0.1 [skip ci]

### Bug Fixes

- pointer Bundler sur docs/Gemfile via BUNDLE_GEMFILE


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v4.0.1)

## v4.0.0 (2026-08-07)
**Tag**: `v4.0.0`

## v4.0.0 (2026-08-07)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **ci**: Corriger les erreurs de lint, mypy et pyright ([`4283355`](https://github.com/dorel14/whoosh-ng/commit/428335590e641870f6bc399b241b453745e4b9a3))

---

**Detailed Changes**: [v3.0.0...v4.0.0](https://github.com/dorel14/whoosh-ng/compare/v3.0.0...v4.0.0)

### Commits

### Features

- restructurer le système de middleware et étendre PluginManager
- ajouter le module linguistics avec analyseurs multilingues
- add asearch/awriter bridges and AsyncFileStorage
- enhance FastAPI models and Admin Studio modules
- add SearchApplication and FileStorage exports
- add S3Storage, HybridStorage, AsyncHybridStorage
- add SnapshotStorage, CachedObjectStorage alias, and Phase 3 roadmap
- publier la version 3.0.0 et ajouter la documentation LLM

### Documentation

- ajouter les guides Whoosh-NG 2.0 et ajuster la configuration de release
- add Gemfile.lock for reproducible Jekyll build and fix French index permalinks
- add S3 storage benchmarks and documentation
- auto-update llms context files

### Chores

- restructurer les workflows CI/CD et nettoyer le code
- apply pre-commit fixes
- v4.0.0 [skip ci]

### Other

- Merge branch 'master' into dev
- Merge pull request #14 from dorel14/dev

### Bug Fixes

- corriger les erreurs de lint, mypy et pyright


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v4.0.0)

## v3.0.0 (2026-08-06)
**Tag**: `v3.0.0`

## v3.0.0 (2026-08-06)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- Ajouter reportAssignmentType en warning dans pyrightconfig.json ([`3259c76`](https://github.com/dorel14/whoosh-ng/commit/3259c76660a4f99135a8cda1ac26baa3f642a873))

- Corriger docstring PathTokenizer et restaurer Literal pour engine ([`b90222e`](https://github.com/dorel14/whoosh-ng/commit/b90222e5962671dfb20f496d396f546f33c95747))

- Mapper engine pyarrow vers auto pour pandas et exclure tests/ de mypy ([`dadeb91`](https://github.com/dorel14/whoosh-ng/commit/dadeb91cffd9462d0c5c2aff77fd36f4cbac8f4a))

- Ne pas yield de token vide dans RegexTokenizer gaps=True ([`6385a24`](https://github.com/dorel14/whoosh-ng/commit/6385a24cb048777aef9cc5d2bad46f2e5d2e5dfa))

- Remove unreachable dead code after build() return in parallel_builder ([`c2e03c9`](https://github.com/dorel14/whoosh-ng/commit/c2e03c97bda6ac61d78c38c6ac3ad37df4f9345a))

- Rendre test_buffered_threads deterministe en couverture des valeurs ([`77a66f6`](https://github.com/dorel14/whoosh-ng/commit/77a66f6320f9ff7bfe8a1c787c52d1bb0e3cf0d6))

- Resolve mypy errors in CI (parquet, tortoise, hnswlib) ([`514f1f6`](https://github.com/dorel14/whoosh-ng/commit/514f1f62d55edd92633654e0096caf920a9ff3a6))

- Réintégrer assert segment_reader is not None après merge distant ([`c42b065`](https://github.com/dorel14/whoosh-ng/commit/c42b065355bb0004094223748a7c9b6eca810405))

- Résoudre les erreurs pyright dans pre-commit (Token typing, reportAttributeAccessIssue) ([`eff5adf`](https://github.com/dorel14/whoosh-ng/commit/eff5adf55cbbae6169bf2efacc76fddacc1891dd))

- Résoudre les erreurs pyright reportOptionalMemberAccess et reportAssignmentType ([`572529e`](https://github.com/dorel14/whoosh-ng/commit/572529e1fb7f0828f3729ed7920e219a71df139c))

- Supprimer cast redondant et ajouter reportAssignmentType en warning ([`0834923`](https://github.com/dorel14/whoosh-ng/commit/0834923f185a4fb62ded3e5020ea6d36b90bfc42))

- **analysis**: Corriger l'indentation du RegexTokenizer et ajuster les annotations de type ([`69e9ad8`](https://github.com/dorel14/whoosh-ng/commit/69e9ad815192ed9113acea4075e21eaf9e514f8d))

- **deps**: Retirer les modules obsolètes des exclusions mypy ([`f53e157`](https://github.com/dorel14/whoosh-ng/commit/f53e15729647ac7473066f193a33fcbfb2a3dce0))

- **indexing**: Close segment_ix in ParallelIndexBuilder to prevent fd leak ([`bad2928`](https://github.com/dorel14/whoosh-ng/commit/bad2928155d424e19de24e5b00822831695e7449))

- **indexing**: Corriger les fuites de handles et erreurs de nettoyage sous Windows ([`3cb85c7`](https://github.com/dorel14/whoosh-ng/commit/3cb85c7d9f80eccdbb5d2225d270d9f4c15b79bd))

- **indexing**: Merge parallel worker segments into main index ([`6f18248`](https://github.com/dorel14/whoosh-ng/commit/6f182488b18aedaceaf4c886b1b0d87eadc02a99))

- **indexing**: Merge worker segments into main index in ParallelIndexBuilder ([`27b0541`](https://github.com/dorel14/whoosh-ng/commit/27b054140d1a99228a804b75a1d4efd3e4f300ce))

- **mypy**: Restore ignore_missing_imports for pytest, peewee, httpx, re2, psutil ([`699ee4f`](https://github.com/dorel14/whoosh-ng/commit/699ee4f2fb3007012913d5970376f97fc99a0296))

- **profiling**: Add segment_write() and sibling step context managers to CommitProfilerV2 ([`599b650`](https://github.com/dorel14/whoosh-ng/commit/599b6507149332b3e0522b753e547bae718b468e))

- **profiling**: Implement SegmentProfiler to resolve NameError in benchmark.py ([`1b03b3a`](https://github.com/dorel14/whoosh-ng/commit/1b03b3ae7fbc3d03c7223ab4f7759eb500c9c8bf))

### Documentation

- Auto-update llms context files ([`5cca243`](https://github.com/dorel14/whoosh-ng/commit/5cca243cba55d49836e89793af3ac126815b862c))

- Auto-update llms context files ([`55825c1`](https://github.com/dorel14/whoosh-ng/commit/55825c15502f0b15041aec9ac15d7f9c4bc23e6c))

- Auto-update llms context files ([`c0d74a4`](https://github.com/dorel14/whoosh-ng/commit/c0d74a4cda58a44898a0b3f67fdc009fefa565fe))

- Restructurer la documentation et ajouter les pages API et guides ([`4e9b1e2`](https://github.com/dorel14/whoosh-ng/commit/4e9b1e2ce8a494c66f1725cec3b03e5025643e8d))

- **guides**: Ajouter le guide d'indexation moderne et mettre à jour les index de documentation ([`9fbc4ff`](https://github.com/dorel14/whoosh-ng/commit/9fbc4ff41dafc184219f0b4f1da3ee4f3e0ecb71))

### Features

- Ajouter FastCSVSource, indexation par lots, infrastructure de profiling et optimisations du cœur ([`fc63f9c`](https://github.com/dorel14/whoosh-ng/commit/fc63f9c890155fa9ce0b25c03e9db733eb7f0139))

- **analysis**: Ajouter le système de stemmers, FastCSVSource et l'infrastructure de profiling des performances ([`a077319`](https://github.com/dorel14/whoosh-ng/commit/a077319a8fc7435ef283b5c2b5402538c6550819))

- **core**: Ajouter CacheMiddleware et ObservableDataSource ([`508589a`](https://github.com/dorel14/whoosh-ng/commit/508589a8c86c7d1eadeb6ffd044a340fe84363be))

- **data-sources**: Ajouter les sources de données et le pooling de connexions ([`51805d4`](https://github.com/dorel14/whoosh-ng/commit/51805d4c08109b78704f6938deb4c135f57b0674))

- **data-sources**: Améliorer la robustesse et la validation des sources de données ([`35281b5`](https://github.com/dorel14/whoosh-ng/commit/35281b534865111e30d0d1195c492d72b8769e75))

- **profiling**: Ajouter les groupes d'options profiling et fast-stemming, stream_batches et restructurer les chemins d'import des sources de données ([`152e9c4`](https://github.com/dorel14/whoosh-ng/commit/152e9c44b68380a9d238489a29ebfd52e28f97a9))

---

**Detailed Changes**: [v2.0.0...v3.0.0](https://github.com/dorel14/whoosh-ng/compare/v2.0.0...v3.0.0)

### Commits

### Other

- Remove workflows permission from test.yml
- Revise README for version 2.0.0 updates
- Merge pull request #12 from dorel14/master
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Potential fix for pull request finding 'CodeQL / Workflow does not contain permissions'
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- .
- Update src/whoosh_modern/indexing/parallel_builder.py
- codec/base: restore missing out-of-order term check in add_postings
- Update src/whoosh_modern/profiling/segment_profiler.py
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge c42b065355bb0004094223748a7c9b6eca810405 into 9988770df03a2d255a21b74843772776dbbf998a
- Merge pull request #13 from dorel14/dev
- Update commit_parser_options in pyproject.toml
- Add files via upload

### Build System

- automatiser la synchronisation de la version entre les fichiers du projet

### Chores

- apply pre-commit fixes
- simplifier la configuration type-checking et retirer les dépendances inutilisées
- apply pre-commit fixes
- apply pre-commit fixes
- trigger pre-commit workflow
- v3.0.0 [skip ci]

### Features

- ajouter CacheMiddleware et ObservableDataSource
- ajouter les sources de données et le pooling de connexions
- ajouter FastCSVSource, indexation par lots, infrastructure de profiling et optimisations du cœur
- ajouter le système de stemmers, FastCSVSource et l'infrastructure de profiling des performances
- ajouter les groupes d'options profiling et fast-stemming, stream_batches et restructurer les chemins d'import des sources de données
- améliorer la robustesse et la validation des sources de données

### Code Refactoring

- simplifier les expressions multi-lignes et optimiser Token avec __slots__
- nettoyer les annotations de type et supprimer les dépendances inutilisées

### Documentation

- auto-update llms context files
- restructurer la documentation et ajouter les pages API et guides
- auto-update llms context files
- ajouter le guide d'indexation moderne et mettre à jour les index de documentation
- auto-update llms context files

### Bug Fixes

- corriger l'indentation du RegexTokenizer et ajuster les annotations de type
- resolve mypy errors in CI (parquet, tortoise, hnswlib)
- ne pas yield de token vide dans RegexTokenizer gaps=True
- résoudre les erreurs pyright dans pre-commit (Token typing, reportAttributeAccessIssue)
- supprimer cast redondant et ajouter reportAssignmentType en warning
- ajouter reportAssignmentType en warning dans pyrightconfig.json
- corriger docstring PathTokenizer et restaurer Literal pour engine
- mapper engine pyarrow vers auto pour pandas et exclure tests/ de mypy
- remove unreachable dead code after build() return in parallel_builder
- add segment_write() and sibling step context managers to CommitProfilerV2
- implement SegmentProfiler to resolve NameError in benchmark.py
- merge parallel worker segments into main index
- merge worker segments into main index in ParallelIndexBuilder
- rendre test_buffered_threads deterministe en couverture des valeurs
- retirer les modules obsolètes des exclusions mypy
- close segment_ix in ParallelIndexBuilder to prevent fd leak
- résoudre les erreurs pyright reportOptionalMemberAccess et reportAssignmentType
- réintégrer assert segment_reader is not None après merge distant
- restore ignore_missing_imports for pytest, peewee, httpx, re2, psutil
- corriger les fuites de handles et erreurs de nettoyage sous Windows


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v3.0.0)

## v2.0.0 (2026-07-31)
**Tag**: `v2.0.0`

## v2.0.0 (2026-07-31)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- Address review findings in CI/CD workflows and documentation ([`b1e0b89`](https://github.com/dorel14/whoosh-ng/commit/b1e0b89f10ad182ad0120d6d19bcac2f8a04470a))

### Documentation

- Auto-update llms context files ([`10e6c90`](https://github.com/dorel14/whoosh-ng/commit/10e6c90fb6e7ffd9ec9f1b3fa7f4e398a933077b))

- Auto-update llms context files ([`992e4d7`](https://github.com/dorel14/whoosh-ng/commit/992e4d780fd7829bd804d68b215e06f255e9cacf))

- Auto-update llms context files ([`5dfe9dc`](https://github.com/dorel14/whoosh-ng/commit/5dfe9dcfdd725d1ca12f3bba0c09a2f7d163aa88))

- Auto-update llms context files ([`aadce99`](https://github.com/dorel14/whoosh-ng/commit/aadce99460c30cd5950e44c927c07eb6fa5ffde8))

### Features

- **deps**: Add sqlalchemy and sqlmodel to models extra and configure mypy overrides ([`4486b5f`](https://github.com/dorel14/whoosh-ng/commit/4486b5feef2700fd8720a0dec1419985f9479951))

- **models**: Ajouter AutoIndexer et améliorer la génération de schémas ([`919131c`](https://github.com/dorel14/whoosh-ng/commit/919131ce7f475b8584e74d98d9cf41e04891b7c7))

- **models**: ✨ Introduce ModelIndex and SearchField for auto-mapping ([`6c202bc`](https://github.com/dorel14/whoosh-ng/commit/6c202bc6c4e20ad9dba7e4cd81deadfdaf3dcf2a))

- **whoosh_modern**: Add modern API with data sources, schema discovery, facets, validation, middleware, and SearchView ([`36bfbae`](https://github.com/dorel14/whoosh-ng/commit/36bfbaebd6acc18cda06aba83bebade7154c9972))

---

**Detailed Changes**: [v1.3.3...v2.0.0](https://github.com/dorel14/whoosh-ng/compare/v1.3.3...v2.0.0)

### Commits

### Documentation

- merge duplicate FastAPI example into fastapi-search.md
- auto-update llms context files
- auto-update llms context files
- auto-update llms context files
- auto-update llms context files

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng
- ..
- Merge pull request #10 from dorel14/master
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- style(benchmark): ajouter un saut de ligne final manquant dans reuters_modern.py
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Potential fix for pull request finding 'CodeQL / Workflow does not contain permissions'
- Potential fix for pull request finding 'CodeQL / Workflow does not contain permissions'
- Potential fix for pull request finding 'CodeQL / Workflow does not contain permissions'
- .
- .
- .
- Merge pull request #11 from dorel14/dev

### Features

- ✨ Introduce ModelIndex and SearchField for auto-mapping
- ajouter AutoIndexer et améliorer la génération de schémas
- add modern API with data sources, schema discovery, facets, validation, middleware, and SearchView
- add sqlalchemy and sqlmodel to models extra and configure mypy overrides

### Bug Fixes

- address review findings in CI/CD workflows and documentation

### Code Refactoring

- moderniser les annotations de type avec la syntaxe union PEP 604
- moderniser les annotations de type avec Coroutine et ajouter des ignores pyright
- ajouter des annotations de retour aux méthodes replace des matchers
- moderniser la vérification isinstance avec la syntaxe union PEP 604

### CI/CD

- ajouter des extras d'installation et simplifier la couverture

### Chores

- v2.0.0 [skip ci]


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v2.0.0)

## v1.3.3 (2026-07-26)
**Tag**: `v1.3.3`

## v1.3.3 (2026-07-26)

_This release is published under the BSD-2-Clause License._

---

**Detailed Changes**: [v1.3.2...v1.3.3](https://github.com/dorel14/whoosh-ng/compare/v1.3.2...v1.3.3)

### Commits

### Bug Fixes

- reorganize nav_order for coherent navigation (Guides 1-90, API 100-190, Examples 200-270)

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng

### Chores

- v1.3.3 [skip ci]


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.3.3)

## v1.3.2 (2026-07-26)
**Tag**: `v1.3.2`

## v1.3.2 (2026-07-26)

_This release is published under the BSD-2-Clause License._

---

**Detailed Changes**: [v1.3.1...v1.3.2](https://github.com/dorel14/whoosh-ng/compare/v1.3.1...v1.3.2)

### Commits

### Bug Fixes

- remove color_scheme from individual pages, use global config

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng

### Chores

- v1.3.2 [skip ci]


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.3.2)

## v1.3.1 (2026-07-26)
**Tag**: `v1.3.1`

## v1.3.1 (2026-07-26)

_This release is published under the BSD-2-Clause License._

---

**Detailed Changes**: [v1.3.0...v1.3.1](https://github.com/dorel14/whoosh-ng/compare/v1.3.0...v1.3.1)

### Commits

### Bug Fixes

- align _config.yml with taskiq-flow and clean README front matter

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng

### Chores

- v1.3.1 [skip ci]


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.3.1)

## v1.3.0 (2026-07-26)
**Tag**: `v1.3.0`

## v1.3.0 (2026-07-26)

_This release is published under the BSD-2-Clause License._

---

**Detailed Changes**: [v1.2.4...v1.3.0](https://github.com/dorel14/whoosh-ng/compare/v1.2.4...v1.3.0)

### Commits

### Features

- éviter les exécutions inutiles du workflow lors des commits de release

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng

### Chores

- v1.3.0 [skip ci]


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.3.0)

## v1.2.4 (2026-07-26)
**Tag**: `v1.2.4`

## v1.2.4 (2026-07-26)

_This release is published under the BSD-2-Clause License._

---

**Detailed Changes**: [v1.2.3...v1.2.4](https://github.com/dorel14/whoosh-ng/compare/v1.2.3...v1.2.4)

### Commits

### Bug Fixes

- remove invalid parent fields and align config with taskiq-flow

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng

### Chores

- v1.2.4


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.2.4)

## v1.2.3 (2026-07-26)
**Tag**: `v1.2.3`

## v1.2.3 (2026-07-26)

_This release is published under the BSD-2-Clause License._

---

**Detailed Changes**: [v1.2.2...v1.2.3](https://github.com/dorel14/whoosh-ng/compare/v1.2.2...v1.2.3)

### Commits

### Bug Fixes

- remove invalid parent fields from all pages

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng

### Chores

- v1.2.3


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.2.3)

## v1.2.2 (2026-07-26)
**Tag**: `v1.2.2`

## v1.2.2 (2026-07-26)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **docs**: Align _config.yml with taskiq-flow pattern ([`2124be0`](https://github.com/dorel14/whoosh-ng/commit/2124be0cf1fd4669c9ebdaa4ad485eabae22c279))

---

**Detailed Changes**: [v1.2.1...v1.2.2](https://github.com/dorel14/whoosh-ng/compare/v1.2.1...v1.2.2)

### Commits

### Code Refactoring

- supprimer la navigation statique codée en dur

### Other

- Merge branch 'master' of https://github.com/dorel14/whoosh-ng

### Bug Fixes

- align _config.yml with taskiq-flow pattern

### Chores

- v1.2.2


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.2.2)

## v1.2.1 (2026-07-26)
**Tag**: `v1.2.1`

## v1.2.1 (2026-07-26)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **docs**: Restore front matter and add explicit nav config ([`74dc151`](https://github.com/dorel14/whoosh-ng/commit/74dc151104a7a89ffd1459a60e0e28488bf110ff))

### Documentation

- Fix Jekyll links with relative_url and clean deploy workflow ([`2f9afc2`](https://github.com/dorel14/whoosh-ng/commit/2f9afc20f5e41022117a636b0d223f92c660df3d))

---

**Detailed Changes**: [v1.2.0...v1.2.1](https://github.com/dorel14/whoosh-ng/compare/v1.2.0...v1.2.1)

### Commits

### Documentation

- fix Jekyll links with relative_url and clean deploy workflow

### Bug Fixes

- restore front matter and add explicit nav config

### Chores

- v1.2.1


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.2.1)

## v1.2.0 (2026-07-26)
**Tag**: `v1.2.0`

## v1.2.0 (2026-07-26)

_This release is published under the BSD-2-Clause License._

### Documentation

- Ajouter la documentation complète bilingue et le déploiement GitHub Pages ([`a091ced`](https://github.com/dorel14/whoosh-ng/commit/a091cede82bb5d722e5db0ff233c1dcbcecd35ea))

- Auto-update llms context files ([`96e4f2c`](https://github.com/dorel14/whoosh-ng/commit/96e4f2c251ccdde5e9fd7e9cb0ce6b2650ab64d9))

- Auto-update llms context files ([`6abefe4`](https://github.com/dorel14/whoosh-ng/commit/6abefe4b84d99e29c1d6ccb471dd7bf27ea4f31d))

### Features

- **docs**: Ajouter le support multilingue dans la configuration ([`967ed12`](https://github.com/dorel14/whoosh-ng/commit/967ed125e5fcd167e237e1c32b7feeb3ab6f9c4f))

---

**Detailed Changes**: [v1.1.0...v1.2.0](https://github.com/dorel14/whoosh-ng/compare/v1.1.0...v1.2.0)

### Commits

### CI/CD

- restructurer le workflow de release sémantique

### Other

- Merge pull request #7 from dorel14/dev
- revert: supprimer la documentation complète bilingue et restaurer l'état précédent
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge pull request #8 from dorel14/dev
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge pull request #9 from dorel14/dev

### Documentation

- ajouter la documentation complète bilingue et le déploiement GitHub Pages
- auto-update llms context files
- auto-update llms context files

### Features

- ajouter le support multilingue dans la configuration

### Chores

- v1.2.0


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.2.0)

## v1.1.0 (2026-07-26)
**Tag**: `v1.1.0`

## v1.1.0 (2026-07-26)

_This release is published under the BSD-2-Clause License._

### Bug Fixes

- **bench**: 🐛 add type ignores for method overrides in `XappyModule` ([`822e957`](https://github.com/dorel14/whoosh-ng/commit/822e957c986f603e9fd1c2de2799fb60d3c122a9))

- **schema**: 🐛 corriger l'initialisation de l'objet dans `__new__` ([`5a764a7`](https://github.com/dorel14/whoosh-ng/commit/5a764a78751a6a2bb94118340784da074fdcf2c8))

- **stress**: 🐛 ensure correct handling of string encoding in `test_bigtable` ([`822e957`](https://github.com/dorel14/whoosh-ng/commit/822e957c986f603e9fd1c2de2799fb60d3c122a9))

### Build System

- **deps**: Supprimer les dépendances obsolètes de la section models ([`227a59f`](https://github.com/dorel14/whoosh-ng/commit/227a59fc195799aca02289c2cb5f3d60135ea063))

### Features

- **benchmark**: Refonte du système de benchmarks avec nouvelles spécifications ([`417efd1`](https://github.com/dorel14/whoosh-ng/commit/417efd1be034fe8be4377ad2716e7b15028ad929))

- **matching**: ✨ add type hints for `supports_block_quality` methods ([`822e957`](https://github.com/dorel14/whoosh-ng/commit/822e957c986f603e9fd1c2de2799fb60d3c122a9))

- **support**: ✨ add compatibility for Python 3 unicode handling ([`822e957`](https://github.com/dorel14/whoosh-ng/commit/822e957c986f603e9fd1c2de2799fb60d3c122a9))

- **writing**: ✨ Implement segment writing and merging policies ([`ae5fc62`](https://github.com/dorel14/whoosh-ng/commit/ae5fc62d2185ac349b56cbbf006bbb7fd4f92c80))

---

**Detailed Changes**: [v1.0.0...v1.1.0](https://github.com/dorel14/whoosh-ng/compare/v1.0.0...v1.1.0)

### Commits

### Chores

- ✏️ Mise à jour de la version dans le README
- prepare whoosh-ng 1.0.0
- bump version to 1.0.1
- 🔄 Update project dependencies
- apply pre-commit fixes
- apply pre-commit fixes
- v1.1.0

### Other

- Potential fix for code scanning alert no. 1: Workflow does not contain permissions
- Potential fix for code scanning alert no. 1: Workflow does not contain permissions
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge pull request #5 from dorel14/alert-autofix-1
- Merge branch 'master' into dev
- Potential fix for pull request finding 'CodeQL / Workflow does not contain permissions'
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge 3d0187143ec5a2a4e19f62d26a622d2c53efc2c7 into 68e67aaefb0d48f136d53576e00f10d68f77f15a
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge 4473223d510f0cde2d30ca638d284f23dc467b14 into 68e67aaefb0d48f136d53576e00f10d68f77f15a
- Merge branch 'dev' of https://github.com/dorel14/whoosh-ng into dev
- Merge pull request #6 from dorel14/dev

### Features

- ✨ Ajout de la validation de l'intégrité des segments
- ✨ Ajout d'un wrapper Asyncio pour la gestion asynchrone des écritures
- ✨ Isolation des espaces de stockage temporaires pour les écrivains concurrents
- ✨ Ajout de la normalisation des boosts pour les sous-requêtes
- ✨ Ajout d'un module de reporting pour les résultats de benchmark
- ✨ Ajout d'un backend LMDB et d'un support d'autocomplétion
- ✨ Implement segment writing and merging policies
- refonte du système de benchmarks avec nouvelles spécifications
- ✨ add type hints for `supports_block_quality` methods

### Bug Fixes

- 🐛 Ajout d'un type d'ignore pour l'appel de la requête
- 🐛 Amélioration des benchmarks avec un échauffement et ajustement des seuils d'alerte
- 🐛 corriger l'initialisation de l'objet dans `__new__`

### Code Refactoring

- improve _posting_size estimation, fix benchmark CLI, update mypy target to 3.12

### Build System

- supprimer les dépendances obsolètes de la section models


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.1.0)

## v1.0.0 (2026-07-12)
**Tag**: `v1.0.0`

## v1.0.0 (2026-07-12)

_This release is published under the BSD-2-Clause License._

- Initial Release


[View on GitHub](https://github.com/dorel14/whoosh-ng/releases/tag/v1.0.0)


## DOCUMENT (FR): Core Concepts

# Concepts fondamentaux

Whoosh-NG est une bibliothèque de recherche purement Python. Ce guide explique les principaux concepts pour l'utiliser efficacement.

## Architecture

Whoosh-NG suit une architecture en couches :

```text
Application
    ▼
┌─────────────────────────────┐
│       Whoosh-NG Core        │
├─────────────────────────────┤
│ Schema                      │
│ Search Engine               │
│ Plugin Manager              │
│ Registry System             │
│ Middleware Pipeline         │
│ Event Bus                   │
│ Hook System                 │
└─────────────────────────────┘
       ▼
┌─────────────────────────────┐
│           Plugins           │
├─────────────────────────────┤
│ FastAPI                     │
│ Autocomplete                │
│ Vector Search               │
│ PostgreSQL                  │
│ S3                          │
│ Monitoring                  │
│ Admin UI                    │
└─────────────────────────────┘
```

## Composants clés

### Index

Un `Index` est le conteneur de vos documents. Il gère un ou plusieurs segments sur disque.

```python
from whoosh.index import create_in, open_dir

ix = create_in("indexdir", schema)
ix = open_dir("indexdir")
```

### Schema

Le `Schema` définit les champs des documents. Chaque champ a un type qui détermine son indexation et stockage.

```python
from whoosh.fields import Schema, TEXT, ID, NUMERIC

schema = Schema(
    title=TEXT(stored=True),
    path=ID(stored=True, unique=True),
    content=TEXT,
    rating=NUMERIC(float, stored=True)
)
```

### Writer

Un `IndexWriter` permet d'ajouter, modifier et supprimer des documents.

```python
writer = ix.writer()
writer.add_document(title="Bonjour", content="Monde")
writer.commit()
```

### Searcher

Un `Searcher` interroge l'index et retourne des résultats.

```python
with ix.searcher() as s:
    results = s.search("bonjour")
```

### QueryParser

Convertit une chaîne de requête en objet Query.

```python
from whoosh.qparser import QueryParser

qp = QueryParser("content", schema)
query = qp.parse("bonjour monde")
```

## Fonctionnalités modernes

### Système de plugins

Les plugins étendent Whoosh-NG sans modifier le core. Ils peuvent :

- Enregistrer de nouveaux providers vectoriels
- Ajouter des endpoints FastAPI
- Fournir des analyseurs personnalisés
- S'intégrer au pipeline de middleware

```python
from whoosh.plugins.manager import PluginManager

# Auto-découverte depuis les entry points
PluginManager.load_plugins()
```

### Pipeline de middleware

Le middleware intercepte les opérations d'indexation et de recherche :

```python
from whoosh.middleware import Middleware, MiddlewareContext

class LoggingMiddleware(Middleware):
    def before_search(self, context: MiddlewareContext):
        print(f"Recherche: {context.query}")
        return context

    def after_search(self, context: MiddlewareContext):
        print(f"Trouvé: {len(context.results) if context.results else 0} résultats")
        return context
```

### Recherche vectorielle

Permet la recherche sémantique via des embeddings :

```python
from whoosh.fields import Schema, TEXT, VectorField

schema = Schema(
    content=TEXT,
    embedding=VectorField(dimensions=384)
)
```

### Event Bus

Système d'événements pour un couplage lâche :

```python
from whoosh.event_bus import EventBus, DocumentIndexed

bus = EventBus()

@bus.subscribe
def on_document_indexed(event: DocumentIndexed):
    print(f"Document indexé: {event.docnum}")
```

## Principes de conception

1. **Composabilité**: Les composants se combinent via les opérateurs `|` et `+`
2. **Abstractions sans coût**: Pas de middleware = pas de surcoût
3. **Sync-first**: Le core est synchrone; async est optionnel
4. **Isolation des plugins**: Les plugins ne peuvent pas casser le core
5. **Sécurité des types**: Typage complet avec annotations


## DOCUMENT (FR): Dates

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Dates and Numeric Ranges

This guide covers working with `DATETIME` and `NUMERIC` fields, including
range queries, range faceting, and date math.

## DATETIME Fields

`DATETIME` fields store Python `datetime` objects and can be queried with
range queries.

```python
from datetime import datetime
from whoosh import fields, index

schema = fields.Schema(
    title=fields.TEXT(stored=True),
    published_date=fields.DATETIME(stored=True, sortable=True),
)
```

### Indexing Dates

```python
ix = index.create_in("indexdir", schema)
with ix.writer() as w:
    w.add_document(
        title="Article 1",
        published_date=datetime(2024, 6, 15, 14, 30),
    )
```

### Date Range Queries

Use `Range` or `QueryParser` syntax:

```python
from whoosh.qparser import QueryParser
from whoosh.query import Range, Every

# Using QueryParser syntax
qp = QueryParser("published_date", schema=ix.schema)
q = qp.parse("[2024-01-01 TO 2024-12-31]")

# Using Range query directly
from datetime import datetime
q = Range(
    "published_date",
    datetime(2024, 1, 1),
    datetime(2024, 12, 31),
)

with ix.searcher() as searcher:
    results = searcher.search(q)
```

### Sorting by Date

```python
from whoosh.sorting import FieldFacet

# Sort by date, most recent first
results = searcher.search(
    query,
    sortedby=FieldFacet("published_date", reverse=True),
)
```

## NUMERIC Fields

`NUMERIC` fields store integers and floating-point numbers.

```python
schema = fields.Schema(
    title=fields.TEXT(stored=True),
    price=fields.NUMERIC(int, stored=True, sortable=True),
    rating=fields.NUMERIC(float, stored=True),
)
```

### Numeric Range Queries

```python
from whoosh.query import NumericRange

q = NumericRange("price", 100, 500)

# Or with QueryParser
qp = QueryParser("price", schema=ix.schema)
q = qp.parse("[100 TO 500]")
```

### Numeric Faceting

Group results into numeric ranges using `RangeFacet`:

```python
from whoosh.sorting import RangeFacet

price_ranges = RangeFacet("price", 0, 1000, 100)
results = searcher.search(query, groupedby=price_ranges)

for groupname, docnums in results.groups("price").items():
    print(f"Price ${groupname}: {len(docnums)} results")
```

## Date Faceting

Group results by date intervals using `DateRangeFacet`:

```python
from datetime import datetime
from whoosh.sorting import DateRangeFacet

start = datetime(2020, 1, 1)
end = datetime(2026, 1, 1)
date_facet = DateRangeFacet(
    "published_date",
    start,
    end,
    relativedelta(years=1),  # Requires: from dateutil.relativedelta import relativedelta
)
results = searcher.search(query, groupedby=date_facet)

for year_range, docnums in results.groups("published_date").items():
    print(f"Year {year_range}: {len(docnums)} results")
```

## Sorting and Filtering by Numbers

### Sorting

```python
from whoosh.sorting import FieldFacet

# Sort by price ascending
results = searcher.search(query, sortedby=FieldFacet("price"))
```

### Filtering

```python
from whoosh.query import NumericRange

# Only results with price >= 50 and price < 200
filter_q = NumericRange("price", 50, 200)
results = searcher.search(query, filter=filter_q)
```

## Making Date/Numeric Fields Sortable

When defining a schema, set `sortable=True` on `NUMERIC` or `DATETIME` fields
to enable sorting by that field:

```python
schema = fields.Schema(
    title=fields.TEXT(stored=True),
    price=fields.NUMERIC(int, sortable=True),
    date=fields.DATETIME(sortable=True),
)
```

If you forgot to set `sortable=True`, you can add it after indexing:

```python
from whoosh import index, sorting

ix = index.open_dir("indexdir")
with ix.writer() as w:
    sorting.add_sortable(w, "price", sorting.FieldFacet("price"))
```


## DOCUMENT (FR): Fieldcaches

---
title: "Field Caches"
sidebar_position: 16
Module: whoosh.filedb.fieldcache
Version: 2.7.4
---
> **Note de traduction** : Cette page n'est pas encore traduite en francais.
> Le contenu anglais est affiche ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Field caches

The default (`filedb`) backend uses *field caches* in certain circumstances.
The field cache basically pre-computes the order of documents in the index to
speed up sorting and faceting.

Generating field caches can take time the first time you sort/facet on a large
index. The field cache is kept in memory (and by default written to disk when
it is generated) so subsequent sorted/faceted searches should be faster.

The default caching policy never expires field caches, so reused searchers
and/or sorting a lot of different fields could use up quite a bit of memory
with large indexes.

## Customizing cache behaviour

(By default, Whoosh saves field caches to disk. To prevent a reader or
searcher from writing out field caches, do this before you start using it:)

```python
searcher.set_caching_policy(save=False)
```

By default, if caches are written to disk they are saved in the index
directory. To tell a reader or searcher to save cache files to a different
location, create a storage object and pass it to the `storage` keyword
argument:

```python
from whoosh.filedb.filestore import FileStorage

mystorage = FileStorage("path/to/cachedir")
reader.set_caching_policy(storage=mystorage)
```

## Creating a custom caching policy

Expert users who want to implement a custom caching policy (for example, to add
cache expiration) should subclass `whoosh.filedb.fieldcache.FieldCachingPolicy`.
Then you can pass an instance of your policy object to the `set_caching_policy`
method:

```python
searcher.set_caching_policy(MyPolicy())
```

## See also

- [Sorting](/core/sorting) — Facets and sort keys
- [API: sorting](../api/sorting) — Sorting and faceting reference


## DOCUMENT (FR): Glossary

# Glossaire

Un glossaire des termes clés utilisés dans Whoosh.

## Analyse

Le processus de conversion du texte en jetons (unités individuelles comme
les mots ou les termes) pour l'indexation. Implique la tokenisation, la
normalisation (mise en minuscules, racinement) et le filtrage (suppression
des mots vides, etc.).

## Analyseur

Une chaîne d'objets `Tokenizer` et `Filter` qui traite le texte en jetons.
Exemples : `RegexTokenizer`, `NgramTokenizer`, `LowercaseFilter`,
`StopFilter`, `StemmerFilter`.

## Fichier composé

Un format de fichier qui combine plusieurs fichiers de segment d'index
en un seul fichier `.seg`. Cela peut améliorer les performances sur
certains systèmes de fichiers en réduisant l'utilisation des descripteurs
de fichiers. Configuré via le paramètre `should_assemble` du codec.

## Document

Un enregistrement unique dans l'index, similaire à une ligne dans une base
de données. Un document contient des champs (analogues aux colonnes).

## Champ

Un attribut nommé d'un document. Les champs ont un type (défini par
`FieldType`) qui détermine comment la valeur du champ est indexée et
stockée.

## Type de champ

La classe (ex. : `TEXT`, `ID`, `NUMERIC`, `DATETIME`, `BOOLEAN`) qui
définit comment la valeur d'un champ est tokenisée, stockée, indexée, et
rendue triable/facetable.

## Filtre

Un composant d'`Analyzer` qui traite, transforme ou filtre les jetons
après la tokenisation. Exemples : `LowercaseFilter`, `StopFilter`,
`StemmerFilter`.

## Format

Un objet `Format` contrôle comment les informations de posting (fréquence
du terme, positions, décalages de caractères) sont encodées pour chaque
champ dans l'index inversé.
Exemples : `Existence`, `Frequency`, `Positions`, `Characters`.

## Fragmentation

Le processus de sélection des fragments de texte autour des termes
correspondants pour la mise en évidence.

## Mise en évidence (Highlighter)

Le module `whoosh.highlight`, qui fournit des formateurs, fragmenteurs
et évaluateurs pour mettre en évidence les termes de recherche dans les
documents.

## Index

La collection de fichiers de segment qui stockent l'index inversé, les
données de documents et les métadonnées (la table des matières, ou TOC).

## IndexWriter

La classe `IndexWriter` est utilisée pour créer et modifier l'index. Elle
met en mémoire tampon les ajouts et suppressions de documents et les
valide sur le disque.

## Index inversé (Inverted Index)

La structure de données centrale d'un moteur de recherche : pour chaque
terme unique, il stocke une liste de documents (et de positions) où ce
terme apparaît.

## Correspondance (Matcher)

Un objet qui itère sur les documents correspondants dans la liste de
postings pour une requête. Les correspondances peuvent être combinées
(union, intersection, etc.) pour des requêtes composées.

## Posting

Une entrée unique dans l'index inversé : un tuple
(ID de document, fréquence du terme, valeur) pour un terme donné.

## Schéma (Schema)

Définit les champs, leurs types et les options d'indexation. Un schéma
est passé à `Storage.create_index()`.

## Évaluateur (Scorer)

Un objet qui calcule un score de pertinence pour un document donné une
requête et des poids de termes. Les différents modèles de pondération
(BM25, TF-IDF, etc.) utilisent des évaluateurs différents.

## Segment

Une portion autonome de l'index inversé. Un index peut consiste en
plusieurs segments. Les segments sont fusionnés périodiquement (lors de
l'optimisation ou des opérations de fusion) pour améliorer les
performances.

## Clé de tri (Sort Key)

Une valeur calculée par document (via un `FacetType` et son
`Categorizer`) utilisée pour ordonner les résultats lors du tri et de la
facettisation.

## Racinement (Stemming)

Le processus de réduction des mots à leur forme racine (par ex.,
"running" → "run", "cats" → "cat") pour améliorer le rappel en
correspondant les formes inflectées.

## Mots vides (Stop Words)

Des mots à haute fréquence et faible information (ex. : "the", "a",
"and") qui sont généralement filtrés lors de l'indexation.

## Terme

Un couple unique (nom de champ, texte du jeton) dans l'index inversé.

## Vecteur de termes (Term Vector)

Structure de données optionnelle par document stockant les termes (et
optionnellement les positions et décalages de caractères) qui apparaissent
dans le champ d'un document, permettant des fonctionnalités comme la mise
en évidence et les retours de pertinence pseudo.

## Tokeniseur (Tokenizer)

Un composant d'`Analyzer` qui divise le texte d'entrée en jetons.
Exemples : `RegexTokenizer`, `PathTokenizer`, `NgramTokenizer`.

## Requête Whoosh

La syntaxe de requête propre à Whoosh, analysée par `QueryParser`.
Prend en charge la recherche sur champs, les requêtes de phrase, les
jokers, les intervalles et plus encore.


## DOCUMENT (FR): Highlight

---
title: "Highlighting"
sidebar_position: 11
Module: whoosh.highlight
Version: 2.7.4
---
> **Note de traduction** : Cette page n'est pas encore traduite en francais.
> Le contenu anglais est affiche ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Highlighting search result excerpts

## Overview

The highlighting system works as a pipeline, with four component types.

- **Fragmenters** chop up the original text into *fragments*, based on the
  locations of matched terms in the text.
- **Scorers** assign a score to each fragment, allowing the system to rank the
  best fragments by whatever criterion.
- **Order functions** control in what order the top-scoring fragments are
  presented to the user. For example, you can show the fragments in the order
  they appear in the document (`FIRST`) or show higher-scoring fragments first
  (`SCORE`).
- **Formatters** turn the fragment objects into human-readable output, such as
  an HTML string.

## Requirements

Highlighting requires that you have the text of the indexed document available.
You can keep the text in a stored field, or if the original text is available in
a file, database column, etc, just reload it on the fly. Note that you might
need to process the text to remove e.g. HTML tags, wiki markup, etc.

## How to

Get search results and use the `highlights()` method on the
`whoosh.searching.Hit` object to get highlighted snippets:

```python
results = mysearcher.search(myquery)
for hit in results:
    print(hit["title"])
    # Assume "content" field is stored
    print(hit.highlights("content"))
```

If the field is not stored, you need to retrieve the text of the field some
other way, then supply it with the `text` argument:

```python
results = mysearcher.search(myquery)
for hit in results:
    print(hit["title"])
    # Assume the "path" stored field contains a path to the original file
    with open(hit["path"]) as fileobj:
        filecontents = fileobj.read()
    print(hit.highlights("content", text=filecontents))
```

## The character limit

By default, Whoosh only pulls fragments from the first 32K characters of the
text. This prevents very long texts from bogging down the highlighting process
too much. You can change the character limit on the results object:

```python
results = mysearcher.search(myquery)
results.fragmenter.charlimit = 100000
```

To turn off the character limit:

```python
results.fragmenter.charlimit = None
```

If you instantiate a custom fragmenter, you can set the character limit directly:

```python
sf = highlight.SentenceFragmenter(charlimit=100000)
results.fragmenter = sf
```

## Customizing the highlights

### Number of fragments

Use the `top` keyword argument to control the number of fragments returned:

```python
# Show a maximum of 5 fragments from the document
print(hit.highlights("content", top=5))
```

### Fragment size

The default fragmenter has a `maxchars` attribute (default 200) controlling the
maximum length of a fragment, and a `surround` attribute (default 20)
controlling the maximum number of characters of context to add at the beginning
and end of a fragment:

```python
# Allow larger fragments
results.fragmenter.maxchars = 300
# Show more context before and after
results.fragmenter.surround = 50
```

### Fragmenter

A fragmenter controls how to extract excerpts from the original text. The
`highlight` module has the following pre-made fragmenters:

- `whoosh.highlight.ContextFragmenter` (the default) — a "smart" fragmenter
  that finds matched terms and pulls in surround text. Only yields fragments
  that contain matched terms.
- `whoosh.highlight.SentenceFragmenter` — tries to break the text into
  fragments based on sentence punctuation.
- `whoosh.highlight.WholeFragmenter` — returns the entire text as one
  "fragment". Useful for short bits of text.

```python
my_cf = highlight.ContextFragmenter(maxchars=100, surround=30)
results.fragmenter = my_cf
```

### Scorer

A scorer is a callable that takes a `whoosh.highlight.Fragment` object and
returns a sortable value (where higher values represent better fragments). The
default scorer adds up the number of matched terms in the fragment, and adds a
"bonus" for the number of *different* matched terms.

```python
def StandardDeviationScorer(fragment):
    """Gives higher scores to fragments where the matched terms are close together."""
    return 0 - stddev([t.pos for t in fragment.matched])

results.scorer = StandardDeviationScorer
```

### Order

The order is a function that takes a fragment and returns a sortable value used
to sort the highest-scoring fragments before presenting them to the user.

- `FIRST` (the default) — show fragments in document order.
- `SCORE` — show highest scoring fragments first.
- `LONGER` / `SHORTER` — longer/shorter fragments first (less generally useful).

```python
results.order = highlight.SCORE
```

### Formatter

A formatter controls how the highest scoring fragments are turned into a
formatted bit of text. The `highlight` module contains:

- `whoosh.highlight.HtmlFormatter` — outputs HTML with a class attribute around
  matched terms.
- `whoosh.highlight.UppercaseFormatter` — converts matched terms to UPPERCASE.

The easiest way to create a custom formatter is to subclass `highlight.Formatter`
and override `format_token`:

```python
class BracketFormatter(highlight.Formatter):
    """Puts square brackets around the matched terms."""

    def format_token(self, text, token, replace=False):
        tokentext = highlight.get_text(text, token, replace)
        return "[%s]" % tokentext

brf = BracketFormatter()
results.formatter = brf
```

## Highlighter object

Rather than setting attributes on the results object, you can create a reusable
`whoosh.highlight.Highlighter` object:

```python
hi = highlight.Highlighter(fragmenter=my_cf, scorer=sds)
for hit in results:
    print(hit["title"])
    print(hi.highlight_hit(hit))
```

## Speeding up highlighting

Recording which terms matched in which documents during the search may make
highlighting faster:

```python
# Record per-document term matches
results = searcher.search(myquery, terms=True)
```

### PinpointFragmenter

Instead of re-tokenizing the document text, Whoosh can look up the character
positions of the matched terms in the index. To use
`whoosh.highlight.PinpointFragmenter` and avoid re-tokenizing:

1. Index the field with character information (requires re-indexing):

   ```python
   schema = fields.Schema(content=fields.TEXT(stored=True, chars=True))
   ```

2. Record per-document term matches:

   ```python
   results = searcher.search(myquery, terms=True)
   ```

3. Set the `PinpointFragmenter` as the fragmenter:

   ```python
   results.fragmenter = highlight.PinpointFragmenter()
   ```

Use the `autotrim` option to strip whitespace before the first space and after
the last space in the fragments:

```python
results.fragmenter = highlight.PinpointFragmenter(autotrim=True)
```

## Using the low-level API

```python
from whoosh.highlight import highlight

excerpts = highlight(
    text, terms, analyzer, fragmenter, formatter, top=3,
    scorer=BasicFragmentScorer, minscore=1, order=FIRST,
)
```

| Argument | Description |
|----------|-------------|
| `text` | The original text of the document. |
| `terms` | A sequence or set containing the query words to match. |
| `analyzer` | The analyzer to use to break the document text into tokens. |
| `fragmenter` | A `Fragmenter` object. |
| `formatter` | A `Formatter` object. |
| `top` | The number of fragments to include in the output. |
| `scorer` | A `FragmentScorer` object. |
| `minscore` | The minimum score a fragment must have to be included. |
| `order` | An ordering function for the "top" fragments. |

## See also

- [Searching](/core/searching) — The `search()` method and `Hit` objects
- [API: highlight](../api/highlight) — Full `whoosh.highlight` reference


## DOCUMENT (FR): Indexing

:::info
Suite au renommage de `whoosh-reloaded` en `whoosh-ng`, les nouveaux modules spécifiques à Whoosh-NG se trouvent généralement sous `whoosh_modern`.
Les composants Whoosh core (comme `whoosh.analysis`, `whoosh.index`) restent accessibles directement sous l'espace de noms `whoosh` pour la rétrocompatibilité.
:::

# Indexation

Guide pour ajouter, mettre à jour et supprimer des documents.

## Ouvrir un writer

```python
from whoosh import index

ix = index.open_dir("indexdir")

# Writer basique
writer = ix.writer()

# Writer avec options
writer = ix.writer(
    timeout=10.0,
    delay=0.1,
    limitmb=128,
    compound=True
)
```

## Ajouter des documents

```python
with ix.writer() as writer:
    writer.add_document(
        title="Premier document",
        content="Bonjour le monde",
        path="/doc1",
        tags=["python", "recherche"]
    )
    writer.commit()
```

## Mettre à jour

```python
with ix.writer() as writer:
    writer.update_document(
        path="/doc1",
        content="Contenu mis à jour"
    )
```

## Supprimer

```python
# Par numéro de document
writer.delete_document(docnum=42)

# Par terme
writer.delete_by_term("path", "/doc1")

# Par requête
from whoosh.query import Term
q = Term("tags", "deprecated")
writer.delete_by_query(q)

writer.commit()
```

## Bonnes pratiques

- Utilisez `with ix.writer() as writer:` pour le nettoyage automatique
- Commutez par lots pour de meilleures performances
- Utilisez `BufferedWriter` en environnement multi-processus
- Libérez toujours le verrou avec `commit()` ou `cancel()`

## Valeurs stockées vs indexées

Pour les champs qui sont à la fois indexés et stockés, vous pouvez stocker une valeur différente :

```python
writer.add_document(
    title="Title to be indexed",
    _stored_title="Display title to show in results"
)
```

> **Note** : Le préfixe underscore (`_stored_<field>`, `_<field>_boost`) est une
> convention Whoosh pour les overrides par document. Il vous permet de stocker
> une valeur différente pour l'affichage (`_stored_title`) sans modifier ce qui
> est indexé, ou de booster un champ spécifique pour un seul document
> (`_title_boost`) sans affecter le boost au niveau du schéma.

## Boosts de champs

Booster des champs individuels au niveau du document :

```python
writer.add_document(
    title="Important title",
    _title_boost=2.0,   # Double weight for title terms
    content="Body content"
)
```

## Mettre à jour les documents

Utilisez `update_document` pour remplacer les documents correspondant à des champs uniques :

```python
schema = Schema(path=ID(unique=True, stored=True), content=TEXT)
ix = index.create_in("indexdir", schema)

with ix.writer() as writer:
    writer.add_document(path="/doc1", content="Original content")
    writer.commit()

with ix.writer() as writer:
    # Remplace tout document avec path="/doc1"
    writer.update_document(path="/doc1", content="Updated content")
    writer.commit()
```

## Supprimer des documents

```python
# Par numéro de document
writer.delete_document(docnum=42)

# Par terme
writer.delete_by_term("path", "/doc1")

# Par requête
from whoosh.query import Term
q = Term("tags", "deprecated")
writer.delete_by_query(q)

writer.commit()
```


## DOCUMENT (FR): Installation

# Installation

## Prérequis

- Python 3.10+
- Aucune dépendance obligatoire (pur Python)
- Extras optionnels pour les fonctionnalités avancées

## pip install

```bash
pip install whoosh-ng
```

## Extras

| Extra | Description |
|-------|-------------|
| `vector` | Providers de recherche vectorielle (NumPy, HNSW, Faiss) |
| `autocomplete` | Plugin d'autocomplétion |
| `api` | Plugin FastAPI |
| `metrics` | Intégration Prometheus |
| `all` | Installer tout |

```bash
pip install whoosh-ng[all]
```

## Installation pour le développement

```bash
git clone https://github.com/your-org/whoosh-NG.git
cd whoosh-NG
uv sync --extra dev
```

## Vérification

```bash
uv run pytest tests/ -q
uv run ruff check src/ tests/
uv run ruff format --check .
uv run mypy src/whoosh
```

## Prochaines étapes

- [Démarrage rapide](/core/quickstart)
- [Concepts fondamentaux](/core/core-concepts)


## DOCUMENT (FR): Intro

---
title: "Introduction to Whoosh"
sidebar_position: 2
Module: whoosh
Version: 2.7.4
---
> **Note de traduction** : Cette page n'est pas encore traduite en francais.
> Le contenu anglais est affiche ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Introduction to Whoosh

## About Whoosh

Whoosh was created by Matt Chaput. It started as a quick and dirty search
server for the online documentation of the Houdini 3D animation software
package. Side Effects Software generously allowed Matt to open source the code
in case it might be useful to anyone else who needs a very flexible or
pure-Python search engine (or both!).

- Whoosh is fast, but uses only pure Python, so it will run anywhere Python
  runs, without requiring a compiler.
- By default, Whoosh uses the [Okapi BM25F](https://en.wikipedia.org/wiki/Okapi_BM25)
  ranking function, but like most things the ranking function can be easily
  customized.
- Whoosh creates fairly small indexes compared to many other search libraries.
- All indexed text in Whoosh must be **unicode**.
- Whoosh lets you store arbitrary Python objects with indexed documents.

## What is Whoosh?

Whoosh is a fast, pure Python search engine library.

The primary design impetus of Whoosh is that it is pure Python. You should be
able to use Whoosh anywhere you can use Python, no compiler or Java required.

Like one of its ancestors, Lucene, Whoosh is not really a search engine, it's a
programmer library for creating a search engine.

Practically no important behavior of Whoosh is hard-coded. Indexing of text, the
level of information stored for each term in each field, parsing of search
queries, the types of queries allowed, scoring algorithms, etc. are all
customizable, replaceable, and extensible.

## What can Whoosh do for you?

Whoosh lets you index free-form or structured text and then quickly find
matching documents based on simple or complex search criteria.

## Whoosh-NG

Whoosh-NG is the maintained evolution of Whoosh. It preserves the pure-Python
core described above while adding optional, opt-in extensions (vector search,
a plugin system, a middleware pipeline, linguistics, and pluggable storage).
Classic features documented in this section remain backwards-compatible with
Whoosh 1.x/2.x.

## Getting help with Whoosh

You can view outstanding issues on the
[Whoosh-NG GitHub page](https://github.com/dorel14/whoosh-ng) and get help by
opening an issue or discussion there.


## DOCUMENT (FR): Keywords

---
title: "Query Expansion & Keywords"
sidebar_position: 13
Module: whoosh.classify, whoosh.searching
Version: 2.7.4
---
> **Note de traduction** : Cette page n'est pas encore traduite en francais.
> Le contenu anglais est affiche ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Query expansion and keyword extraction

## Overview

Whoosh provides methods for computing the "key terms" of a set of documents.
For these methods, "key terms" basically means terms that are frequent in the
given documents, but relatively infrequent in the indexed collection as a whole.

Because this is a purely statistical operation, not a natural language
processing or AI function, the quality of the results will vary based on the
content, the size of the document collection, and the number of documents for
which you extract keywords.

These methods can be useful for providing the following features to users:

- **Search term expansion.** Extract key terms for the top N results from a
  query and suggest them to the user as additional/alternate query terms.
- **Tag suggestion.** Extracting the key terms for a single document may yield
  useful suggestions for tagging the document.
- **"More like this".** Extract key terms for the top ten or so results from a
  query (and removing the original query terms), and use those key words as the
  basis for another query that may find more documents using terms the user
  didn't think of.

## Usage

### More like this

Get more documents like a certain search hit. *This requires that the field you
want to match on is vectored or stored, or that you have access to the original
text.*

```python
results = mysearcher.search(myquery)
first_hit = results[0]
more_results = first_hit.more_like_this("content")
```

### Key terms from top N results

*This requires that the field is either vectored or stored.*

```python
# Extract five key terms from the "content" field of the top ten documents
keywords = [keyword for keyword, score
            in results.key_terms("content", docs=10, numterms=5)]
```

### Key terms from an arbitrary set of documents

*This requires that the field is either vectored or stored.*

```python
with email_index.searcher() as s:
    docnums = s.document_numbers(emailto="matt@whoosh.ca")
    keywords = [keyword for keyword, score
                in s.key_terms(docnums, "body")]
```

### Key terms from arbitrary text not in the index

```python
with email_index.searcher() as s:
    keywords = [keyword for keyword, score
                in s.key_terms_from_text("body", mytext)]
```

## Expansion models

The `ExpansionModel` subclasses in the `whoosh.classify` module implement
different weighting functions for key words. These models are translated into
Python from original Java implementations in Terrier.

```python
from whoosh.classify import Bo1Model

results = mysearcher.search(myquery)
keywords = results.key_terms("content", docs=10, numterms=5, model=Bo1Model)
```

Available models include `Bo1Model`, `Bo2Model`, and `KLModel`.

## See also

- [Searching](/core/searching) — `Results`, `Hit`, and the `search()` method
- [API: searching](../api/searching) — `key_terms`, `more_like_this` reference


## DOCUMENT (FR): Legacy Cleanup

# Stratégie de nettoyage du code legacy

Ce guide explique comment Whoosh-NG sépare le code moderne typé du code legacy,
et comment le nettoyage progressif est organisé.

## Pourquoi une frontière legacy ?

`whoosh-modern` est la nouvelle surface de Whoosh-NG, entièrement typée.
Le package `whoosh` original fonctionne toujours au runtime, mais il contient
des décennies de motifs de compatibilité Python 2/3, de métaprogrammation
dynamique et d'internes non typés. Forcer des types stricts sur l'ensemble
d'un coup bloquerait le développement.

La stratégie de nettoyage est **incrémentale et opt-in** :

1. `src/whoosh_modern/` est typé et vérifié avec `pyright` et `mypy` en mode strict.
2. `src/whoosh/` est la surface legacy. Elle est divisée en :
   - **modules exclus** (documentés dans `pyrightconfig.json`) — code trop
     dynamique ou vendu pour justifier un passage de types rentable maintenant ;
   - **candidats au nettoyage** — petits fichiers isolés, faciles à annoter et
     à vérifier.
3. Chaque sprint, une vague de candidats est typée, testée, puis sortie de la
   zone de tolérance élevée.

## Seuils actuels (Sprint 2)

| Vérificateur | Portée | Seuil |
|--------------|--------|-------|
| `pyright` | `src/whoosh_modern/` | **0 erreur** (strict) |
| `pyright` | legacy | **≤ 500 erreurs** (tolérant) |
| `mypy` | `src/` | **0 erreur** (via overrides + `ignore_errors`) |

## Justification des exclusions (`pyrightconfig.json`)

La liste `exclude` de `pyrightconfig.json` regroupe les fichiers par thème :

- **Vendu / sans stubs** : `pyparsing.py`, `relativedelta.py`
- **Shims de migration** : `codec/whoosh2.py`, `codec/whoosh3.py`
- **Parsing dynamique** : `qparser/`, `query/`, `analysis/`, `automata/`
- **Stockage fichiers** : `filedb/`, `reading/`, `writing/`
- **Heuristique / data-driven** : `lang/dmetaphone.py`, `lang/lovins.py`,
  `lang/phonetic.py`, `lang/wordnet.py`
- **Objets dynamiques** : `classify.py`, `index.py`, `locking.py`,
  `formats.py`, `middleware/`
- **Bas niveau vendu** : `support/bench.py`, `support/base85.py`,
  `support/bitstream.py`, `support/bitvector.py`, `support/charset.py`,
  `support/levenshtein.py`

## Plan Sprint 2

Pour le Sprint 2, l'accent est mis sur les petits modules utilitaires et de
support, sans dépendances externes ni métaprogrammation lourde.

Vague de candidats :

- `src/whoosh/util/varints.py`
- `src/whoosh/util/text.py`
- `src/whoosh/util/loading.py`
- `src/whoosh/support/bitstream.py`
- `src/whoosh/support/levenshtein.py`

Pour chaque fichier :

1. Supprimer le `# type: ignore` global (si présent).
2. Ajouter des signatures de fonctions précises.
3. Lancer `pyright` et `mypy` pour confirmer **0 nouvelle erreur**.
4. Retirer le fichier des exclusions de `pyrightconfig.json`.
5. Ajouter un test de régression dans `tests/test_legacy_cleanup.py`.

## Objectif long terme

Chaque fichier de `src/whoosh/` doit finir par être vérifiable par `mypy` et
`pyright` sans exclusion globale. D'ici là, la liste d'exclusion est le
registre explicite de la dette, et chaque sprint la réduit.


## DOCUMENT (FR): Migration

# Guide de migration

Ce guide vous aide à migrer depuis Whoosh legacy ou Whoosh-Reloaded 3.x vers Whoosh-NG v3.0.0.
> **Prochaine version** : v4.0.0.dev0 (en développement) ajoute `SchemaBuilder`, la hiérarchie d'exceptions middleware, et plus encore — voir le [CHANGELOG](https://github.com/dorel14/whoosh-ng/blob/master/CHANGELOG.md).

## Depuis Whoosh 1.x/2.x (Legacy)

### Chemins d'import

| Legacy | Whoosh-NG |
|--------|-----------|
| `import whoosh` | `import whoosh` |
| `from whoosh.index import create_in` | `from whoosh.index import create_in` |
| `from whoosh.fields import Schema, TEXT` | `from whoosh.fields import Schema, TEXT` |

L'API core est intentionnellement stable. La plupart du code existant fonctionne sans modification.

## Depuis Whoosh-Reloaded 3.x

Aucun changement cassant. Whoosh-NG est une continuation de Whoosh-Reloaded.

### Migration optionnelle des plugins

```python
# Ancien
from whoosh_modern.vector.numpy_provider import NumpyProvider

# Nouveau (via registre)
from whoosh.vector import NumpyProvider
from whoosh.registry import VectorRegistry

VectorRegistry.register("numpy", NumpyProvider(), "mon_app")
```

### SchemaBuilder (nouveau dans v4.0.0.dev0)

```python
# Ancien
schema = Schema(title=TEXT(stored=True), content=TEXT)

# Nouveau (API fluent)
from whoosh.fields import SchemaBuilder

schema = (
    SchemaBuilder()
    .field("title", TEXT(stored=True))
    .field("content", TEXT)
    .build()
)
```

## Méthode de migration middleware (nouveau dans v4.0.0.dev0)

```python
from whoosh.middleware import Middleware, MiddlewareContext

class LoggingMiddleware(Middleware):
    def before_search(self, context: MiddlewareContext):
        print(f"Query: {context.query}")
        return context

# Envelopper le writer/searcher existant
writer = apply_middleware_to_writer(ix.writer(), [LoggingMiddleware()])
```

## Liste de vérification

1. **Mettre à jour les dépendances**:
   ```bash
   pip install --upgrade whoosh-ng
   ```

2. **Exécuter les tests**:
   ```bash
   uv run pytest tests/ -q
   ```

3. **Mettre à jour les dépendances optionnelles** (si plugins utilisés):
   ```bash
   pip install whoosh-ng[all]
   ```

4. **Revoir le middleware**: Envisagez d'ajouter du middleware pour les préoccupations transverses

## Dépréciations

| Fonctionnalité | Statut | Remplacement |
|----------------|--------|--------------|
| `whoosh_modern.vector` | Déprécié | `whoosh.vector` |
| `whoosh.store` brut | Déprécié | `whoosh.backends` |
| Utilisation directe de `SegmentWriter` | Découragé | Utilisez `IndexWriter` |

## Compatibilité

Whoosh-NG maintient la compatibilité ascendante. Si vous trouvez un changement cassant, signalez-le comme une issue.


## DOCUMENT (FR): Nested

# Documents imbriqués (Nested)

Ce guide couvre l'indexation et la recherche de structures de documents
hiérarchiques imbriqués (par exemple, un document parent contenant plusieurs
documents enfants) en utilisant les fonctionnalités de relation
parent-enfant de Whoosh.

## Définition des documents imbriqués

Vous pouvez indexer des documents parents qui contiennent des documents
enfants en utilisant un champ parent et des champs enfants :

```python
from whoosh import fields, index

schema = fields.Schema(
    type=fields.ID(sortable=True),
    title=fields.TEXT(stored=True),
    content=fields.TEXT,
    section_name=fields.ID,
    section_content=fields.TEXT,
)
```

Le champ `type` distingue les documents parents des documents enfants.

## Indexation des documents imbriqués

Utilisez `IndexWriter.add_all()` avec un générateur qui produit les
documents parents et enfants groupés ensemble :

```python
writer = ix.writer()
writer.add_all([
    parent_doc,
    child_doc_1,
    child_doc_2,
    parent_doc_2,
    child_doc_3,
])
```

Les documents parents ont `type="parent"` et les documents enfants ont
`type="child"`.

## Recherche dans les documents imbriqués

### Recherche-enfants, correspondance-parents

Recherchez dans les documents enfants et faites correspondre leurs
documents parents :

```python
from whoosh.query import Every, Term
from whoosh.sorting import NestedParent

# Correspondre tous les documents parents
parents = NestedParent(Term("type", "parent"))
q = Every("section_content", "hello")
results = searcher.search(q, sortedby=parents)
```

### Recherche-parents, correspondance-enfants

Recherchez les documents parents dont les enfants correspondent :

```python
from whoosh.sorting import NestedChildren

# Correspondre les documents parents qui ont des enfants correspondant à la requête
parent_results = searcher.search(child_query, groupedby=NestedChildren(parent_matcher, child_matcher))
```

## Relations parent-enfant lors de l'indexation

Lors de l'écriture des documents, utilisez le paramètre `parent` pour
lier les enfants aux parents :

```python
writer.add_document(type="parent", title="Chapitre 1", _key="chapter1")
writer.add_document(type="child", section_name="Section 1.1",
                    section_content="...", parent="chapter1")
writer.add_document(type="child", section_name="Section 1.2",
                    section_content="...", parent="chapter1")
```

## Accès aux résultats imbriqués

Pour récupérer les correspondances d'enfants aux côtés des résultats
parents, utilisez la méthode `expand` sur les résultats :

```python
results = searcher.search(parent_query)
expanded = results.expand_child("section")
```

## Facettisation imbriquée

Combinez les relations parent-enfant avec la facettisation en utilisant
`NestedParent` et `NestedChildren` comme facets :

```python
parent_facet = NestedParent(FieldFacet("type"))
results = searcher.search(query, groupedby=parent_facet)
```

## Considérations de performance

- Les jointures parent-enfant sont plus coûteuses que les recherches
  sur des documents plats.
- Utilisez l'option `childperm` du chercheur pour limiter le nombre de
  permutations examinées.
- Considérez si la structure hiérarchique est nécessaire lors de la
  requête, ou si les documents peuvent être aplatis lors de l'indexation.


## DOCUMENT (FR): Ngrams

# N-grammes

Ce guide couvre la tokenisation et l'analyse N-gramme pour la recherche
de sous-chaînes, les requêtes par préfixe et la fonctionnalité
d'autocomplétion.

## Qu'est-ce que les N-grammes ?

Un N-gramme est une séquence continue de N caractères (ou de jetons)
d'une chaîne. Par exemple, les 2-grammes de "hello" sont : "he", "el",
"ll", "lo".

L'analyse N-gramme est utile pour :
- Recherche de sous-chaînes (trouver "ell" dans "hello")
- Autocomplétion / suggestions en temps réel
- Correspondance floue sans calcul de distance d'édition

## NgramTokenizer

Le `NgramTokenizer` divise le texte en N-grammes au niveau des
caractères :

```python
from whoosh.analysis import NgramTokenizer
from whoosh import fields

tokenizer = NgramTokenizer(minsize=2, maxsize=4)

schema = fields.Schema(
    content=fields.TEXT(analyzer=tokenizer),
)
```

### Paramètres de NgramTokenizer

- `minsize` : Longueur minimale des N-grammes (par défaut `2`)
- `maxsize` : Longueur maximale des N-grammes (par défaut `4`)

Avec l'exemple ci-dessus, le texte "hello" produit ces 2-4-grammes :
`he, hel, hell, el, ell, ello, l, ll, llo, l, lo, o`

## NgramFilter

Le `NgramFilter` crée des N-grammes au niveau des mots à partir du
texte tokenisé :

```python
from whoosh.analysis import RegexTokenizer, NgramFilter

analyzer = RegexTokenizer() | NgramFilter(maxsize=2)
```

Cela produit des grammes au niveau des mots : pour "hello world", il
produit ("hello",) et ("hello", "world").

## NgramWordAnalyzer

Un analyseur de commodité qui combine `NgramTokenizer` avec
`LowercaseFilter` :

```python
from whoosh.analysis import NgramWordAnalyzer

analyzer = NgramWordAnalyzer(minsize=2, maxsize=4)

schema = fields.Schema(
    content=fields.TEXT(analyzer=analyzer),
)
```

## Cas d'utilisation

### Recherche de sous-chaînes

Avec l'analyse N-gramme, vous pouvez correspondre des sous-chaînes :

```python
from whoosh.qparser import QueryParser

# Index text with N-grams
# Searching for "ell" matches "hello" because "ell" is a substring
qp = QueryParser("content", schema=ix.schema)
q = qp.parse("ell")
results = searcher.search(q)
```

### Correspondance par préfixe

Définissez `maxsize` à une grande valeur pour créer efficacement des
N-grammes de préfixe :

```python
from whoosh.analysis import NgramWordAnalyzer

# Create N-grams where each word's prefixes become searchable tokens
# e.g., "hello" -> "h", "he", "hel", "hell", "hello"
analyzer = NgramWordAnalyzer(minsize=1, maxsize=10)
```

### Autocomplétion

Les index N-gramme sont couramment utilisés pour l'autocomplétion.
Pour une autocomplétion plus avancée avec des N-grammes de bord
(edge n-grams), envisrez :

```python
from whoosh.analysis import RegexTokenizer, NgramFilter
from whoosh.query import Prefix

# Index with standard tokenization, then use Prefix queries for autocomplete
analyzer = RegexTokenizer()
schema = fields.Schema(
    title=fields.TEXT(stored=True, analyzer=analyzer),
    content=fields.TEXT(analyzer=analyzer),
)

# For autocomplete, query with Prefix
from whoosh.qparser import QueryParser
qp = QueryParser("title", schema=ix.schema)
q = Prefix("title", "hel")  # Find documents where title starts with "hel"
```

## Comparaison avec les N-grammes de bord (Edge N-grams)

Certains moteurs de recherche prennent en charge les "N-grammes de bord"
(uniquement la génération de N-grammes à partir du début des mots).
C'est plus efficace en espace pour l'autocomplétion :

- N-grammes complets : "hello" → "he", "el", "ll", "lo", "hel", "ell", ...
- N-grammes de bord : "hello" → "h", "he", "hel", "hell", "hello"

Le `NgramTokenizer` de Whoosh génère des N-grammes complets (bidirectionnels).
Pour un comportement similaire aux N-grammes de bord, utilisez les
paramètres `minsize` et `maxsize` stratégiquement, ou utilisez des
requêtes `Prefix` sur un champ tokenisé standard.

## Considérations de performance

- Les index N-gramme sont généralement beaucoup plus volumineux que les
  index standard.
- Chaque jeton original produit plusieurs jetons N-gramme, augmentant
  la taille de l'index.
- Choisissez `minsize` et `maxsize` avec soin pour équilibrer la qualité
  de recherche et la taille de l'index.
- Pour l'autocomplétion, envisenez d'utiliser des requêtes `Prefix` sur
  un champ non-N-gramme pour de meilleures performances.


## DOCUMENT (FR): Query

# Langage de requête

Whoosh-NG fournit un langage de requête puissant similaire à Lucene, ainsi qu'une API de requêtes programmatique.

## Syntaxe de requête

### Termes de base

```
bonjour                    # Terme unique
bonjour monde              # Termes multiples (AND par défaut)
bonjour OU monde           # OR explicite
"bonjour monde"            # Phrase
```

### Spécification de champ

```
titre:python               # Recherche dans le champ titre
titre:"Tutoriel Python"    # Phrase dans un champ spécifique
```

### Opérateurs booléens

```
python AND whoosh
python OR whoosh
python AND NOT java
python AND (whoosh OR lucene)
```

### Préfixe et jokers

```
pyth*                     # Requête préfixe
pyth?n                    # Joker caractère unique
```

### Requêtes de plage

```
date:[2020 TO 2025]
prix:[10 TO 50]
rating:[4.0 TO *]         # Plage ouverte
```

### Recherche floue

```
python~2                  # Distance d'édition <= 2
lucene~1                  # Correspondance approximative
```

### Recherche de proximité

```
"bonjour monde"~5         # Dans un rayon de 5 termes
```

### Boost

```
python^2.0 whoosh         # Booster python par 2x
(titre:python)^3 content:python  # Booster les matches dans titre
```

## Classes de requêtes

Construisez des requêtes programmatiquement :

```python
from whoosh.query import *

# Terme simple
q = Term("content", "python")

# AND
q = And([Term("content", "python"), Term("content", "whoosh")])

# OR
q = Or([Term("content", "python"), Term("content", "lucene")])

# Phrase
q = Phrase("content", ["bonjour", "monde"])

# Plage
q = NumericRange("prix", 10, 50)
q = DateRange("date", datetime(2020,1,1), datetime(2025,1,1))

# Préfixe
q = Prefix("content", "pyth")
```

## MultifieldParser

Recherchez plusieurs champs avec des boosts différents :

```python
from whoosh.qparser import MultifieldParser

qp = MultifieldParser(
    ["titre", "content", "tags"],
    schema,
    fieldboosts={"titre": 2.0, "tags": 1.5}
)
q = qp.parse("python recherche")
```

## Échappement des caractères spéciaux

```
titre\:python              # Deux-points littéral
chemin\:\/\/exemple        # Échapper les caractères spéciaux
```


## DOCUMENT (FR): Quickstart

# Démarrage rapide

## Installation

```bash
pip install whoosh-ng
uv pip install whoosh-ng
```

## Exemple basique

```python
from whoosh import index
from whoosh.fields import Schema, TEXT, ID

schema = Schema(id=ID(stored=True), content=TEXT())
ix = index.create_in("indexdir", schema)

with ix.writer() as w:
    w.add_document(id="1", content="hello world")
    w.add_document(id="2", content="goodbye world")

with ix.searcher() as s:
    results = s.search("world")
    for hit in results:
        print(hit["id"], hit.score)
```

## Avec plugins

```bash
pip install whoosh-ng[vector,autocomplete,api]
```

```python
from whoosh.plugins.manager import PluginManager
from whoosh_modern.vector.plugin import VectorPlugin

PluginManager.load_plugins()
```

## Sources de données

```bash
pip install whoosh-ng
```

```python
import sqlite3
from whoosh_modern.data_sources import SQLSource
from whoosh_modern.views import SearchView

# Utiliser les données de benchmark existantes
conn = sqlite3.connect("benchmark/benchmark_data.db")
source = SQLSource(
    connection=conn,
    query="SELECT * FROM reuters_articles",
)

vue = SearchView(name="reuters", source=source)
ix = vue.build("indexdir")
```


## DOCUMENT (FR): Recipes

---
title: "Whoosh Recipes"
sidebar_position: 17
Module: whoosh
Version: 2.7.4
---
> **Note de traduction** : Cette page n'est pas encore traduite en francais.
> Le contenu anglais est affiche ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Whoosh recipes

A collection of small, practical code snippets for common tasks.

## General

### Get the stored fields for a document from the document number

```python
stored_fields = searcher.stored_fields(docnum)
```

## Analysis

### Eliminate words shorter/longer than N

Use a `StopFilter` and the `minsize` and `maxsize` keyword arguments. If you
just want to filter based on size and not common words, set the `stoplist` to
`None`:

```python
sf = analysis.StopFilter(stoplist=None, minsize=2, maxsize=40)
```

### Allow optional case-sensitive searches

Index both the original and lowercased versions of each word. If the user
searches for an all-lowercase word, it acts as a case-insensitive search, but
if they search for a word with any uppercase characters, it acts as a
case-sensitive search:

```python
class CaseSensitivizer(analysis.Filter):
    def __call__(self, tokens):
        for t in tokens:
            yield t
            if t.mode == "index":
                low = t.text.lower()
                if low != t.text:
                    t.text = low
                    yield t

ana = analysis.RegexTokenizer() | CaseSensitivizer()
print([t.text for t in ana("The new SuperTurbo 5000", mode="index")])
# ["The", "the", "new", "SuperTurbo", "superturbo", "5000"]
```

## Searching

### Find every document

```python
myquery = query.Every()
```

### iTunes-style search-as-you-type

Use `whoosh.analysis.NgramWordAnalyzer` as the analyzer for the field you want
to search as the user types. You can save space in the index by turning off
positions in the field using `phrase=False`:

```python
# For example, to search the "title" field as the user types
analyzer = analysis.NgramWordAnalyzer()
title_field = fields.TEXT(analyzer=analyzer, phrase=False)
schema = fields.Schema(title=title_field)
```

See the documentation for the `NgramWordAnalyzer` class for information on the
available options. Also see [N-grams](/core/ngrams).

## Shortcuts

### Look up documents by a field value

```python
# Single document (unique field value)
stored_fields = searcher.document(id="bacon")

# Multiple documents
for stored_fields in searcher.documents(tag="cake"):
    ...
```

## Sorting and scoring

See [Sorting](/core/sorting).

### Score results based on the position of the matched term

The following scoring function uses the position of the first occurrence of a
term in each document to calculate the score, so documents with the given term
earlier in the document will score higher:

```python
from whoosh import scoring

def pos_score_fn(searcher, fieldname, text, matcher):
    poses = matcher.value_as("positions")
    return 1.0 / (poses[0] + 1)

pos_weighting = scoring.FunctionWeighting(pos_score_fn)
with myindex.searcher(weighting=pos_weighting) as s:
    ...
```

## Results

### How many hits were there?

```python
# The number of scored hits
found = results.scored_length()

if results.has_exact_length():
    print("Scored", found, "of exactly", len(results), "documents")
else:
    low = results.estimated_min_length()
    high = results.estimated_length()
    print("Scored", found, "of between", low, "and", high, "documents")
```

### Which terms matched in each hit?

```python
# Use terms=True to record term matches for each hit
results = searcher.search(myquery, terms=True)

for hit in results:
    # Which terms matched in this hit?
    print("Matched:", hit.matched_terms())
    # Which terms from the query didn't match in this hit?
    print("Didn't match:", myquery.all_terms() - hit.matched_terms())
```

## Global information

### How many documents are in the index?

```python
# Including documents that are deleted but not yet optimized away
numdocs = searcher.doc_count_all()
# Not including deleted documents
numdocs = searcher.doc_count()
```

### What fields are in the index?

```python
return myindex.schema.names()
```

### Is term X in the index?

```python
return ("content", "wobble") in searcher
```

### How many times does term X occur in the index?

```python
# Number of times content:wobble appears in all documents
freq = searcher.frequency("content", "wobble")
# Number of documents containing content:wobble
docfreq = searcher.doc_frequency("content", "wobble")
```

### Is term X in document Y?

```python
# Without term vectors
postings = searcher.postings("content", "wobble")
postings.skip_to(500)
return postings.id() == 500

# If field has term vectors
vector = searcher.vector(500, "content")
vector.skip_to("wobble")
return vector.id() == "wobble"
```

## See also

- [Analysis](/core/analysis) — Analyzers, tokenizers, and filters
- [N-grams](/core/ngrams) — Search-as-you-type with N-gram analyzers
- [Searching](/core/searching) — The `search()` method and `Hit` objects


## DOCUMENT (FR): Schema

# Conception de schéma

Le schéma définit la structure des documents dans votre index. Il spécifie les champs existants, leur indexation et leur stockage.

## Types de champs

| Type | Description | Indexé | Stocké |
|------|-------------|--------|--------|
| `TEXT` | Texte libre, tokenisé | Oui | Optionnel |
| `ID` | Identifiant non tokenisé | Oui | Optionnel |
| `KEYWORD` | Mots-clés séparés par espace/virgule | Oui | Optionnel |
| `STORED` | Stocké uniquement, non searchable | Non | Oui |
| `NUMERIC` | Entier ou flottant | Oui | Optionnel |
| `DATETIME` | Dates et heures | Oui | Optionnel |
| `BOOLEAN` | Booléen | Oui | Optionnel |
| `NGRAM` | N-grammes de caractères | Oui | Optionnel |
| `NGRAMWORDS` | N-grammes de mots | Oui | Optionnel |
| `VectorField` | Vecteur d'embedding | Personnalisé | Optionnel |

## Créer un schéma

```python
from whoosh.fields import Schema, TEXT, ID, KEYWORD, STORED, NUMERIC

schema = Schema(
    title=TEXT(stored=True),
    path=ID(stored=True, unique=True),
    content=TEXT,
    tags=KEYWORD(lowercase=True),
    published=NUMERIC(int, stored=True),
    is_published=BOOLEAN,
    icon=STORED
)
```

## Options des champs

### TEXT

```python
content = TEXT(
    stored=False,        # Stocker le texte original ?
    unique=False,        # Utiliser pour remplacer des documents ?
    phrase=True,         # Indexer les positions pour recherche de phrases
    analyzer=None,       # Analyseur personnalisé
    field_boost=1.0      # Boost pour le scoring
)
```

### ID

```python
path = ID(
    stored=True,         # Stocker le chemin
    unique=True          # Utiliser pour remplacement de documents
)
```

### KEYWORD

```python
tags = KEYWORD(
    stored=False,
    lowercase=True,      # Minusculiser automatiquement
    commas=True,         # Séparer par virgules
    scorable=True        # Stocker la longueur pour scoring
)
```

## SchemaBuilder

Whoosh-NG v4.0.0.dev0 (en développement) introduit `SchemaBuilder` pour une API fluide :

```python
from whoosh.fields import SchemaBuilder, TEXT, ID, NUMERIC

schema = (
    SchemaBuilder()
    .field("title", TEXT(stored=True))
    .field("path", ID(stored=True, unique=True))
    .field("content", TEXT)
    .field("rating", NUMERIC(float, stored=True))
    .build()
)
```

## Champs dynamiques

Utilisez des patterns glob pour associer des types :

```python
# Tout champ finissant par "_date" est un DATETIME
schema.add("*_date", DATETIME(stored=True), glob=True)

# Tout champ finissant par "_id" est un ID
schema.add("*_id", ID(stored=True), glob=True)
```

## Modifier le schéma

Ajoutez ou supprimez des champs après création :

```python
writer = ix.writer()

# Ajouter un champ
writer.add_field("description", TEXT(stored=True))

# Supprimer un champ
writer.remove_field("legacy_field")

writer.commit()
```

> Note: Supprimer un champ ne fait que le retirer du schéma. Les données ne sont libérées qu'à l'optimisation.

## Modèles de recherche

Whoosh-NG peut mapper automatiquement des modèles Python (dataclasses, Pydantic, SQLAlchemy, SQLModel, msgspec) vers un `Schema` Whoosh via `ModelIndex`.

### Niveau 1 : Auto-mapping

```python
from dataclasses import dataclass
from whoosh_modern.models import ModelIndex

@dataclass
class Book:
    title: str
    count: int
    tag: str | None = None

idx = ModelIndex(Book)
schema = idx.schema
```

`ModelIndex` inspecte les annotations de type et les mappe vers des champs Whoosh :

| Type Python | Champ Whoosh |
|-------------|--------------|
| `str` | `TEXT` |
| `int` / `float` | `NUMERIC` |
| `bool` | `BOOLEAN` |
| `datetime` / `date` | `DATETIME` |
| `Decimal` | `NUMERIC(int, decimal_places=2)` |
| `Enum` | `KEYWORD` |
| `bytes` | `KEYWORD` (stockage hexadécimal) |
| `list[str]` | `KEYWORD` |
| `Optional[T]` | type mappé ou `STORED` |

Les champs ID sont auto-détectés : `SearchOptions(id=True)` explicite > nom `id`/`ID`/`_id` > premier champ `str`.

### Niveau 2 : Options explicites

Utilisez `SearchField` pour remplacer les valeurs par défaut :

```python
from whoosh_modern.models import SearchField, SearchOptions

class Book:
    title: str = SearchField(fulltext=True, stored=True)
    count: int = SearchField(sortable=True)
    tags: list[str] = SearchField(multi=True)
```

### Niveau 3 : Types annotés

Utilisez `Annotated` pour attacher des métadonnées directement aux annotations :

```python
from typing import Annotated
from whoosh_modern.models import SearchField

class Book:
    title: Annotated[str, SearchField(fulltext=True, stored=True)]
```

### Intégrations

#### Dataclass

```python
from dataclasses import dataclass
from whoosh_modern.models import ModelIndex

@dataclass
class Article:
    title: str
    body: str
    published: datetime.datetime

idx = ModelIndex(Article)
```

#### Pydantic v2

```python
from pydantic import BaseModel
from whoosh_modern.models import register_model

class Article(BaseModel):
    title: str
    body: str
    published: datetime.datetime

    # Métadonnées de recherche par champ via json_schema_extra
    model_config = {"json_schema_extra": {"search": {"fulltext": True}}}

idx = register_model(Article)
```

#### SQLAlchemy

```python
from sqlalchemy import Column, Integer, String, DateTime
from sqlalchemy.orm import DeclarativeBase
from whoosh_modern.models import register_model

class Base(DeclarativeBase):
    pass

class Article(Base):
    __tablename__ = "articles"
    id = Column(Integer, primary_key=True)
    title = Column(String, info={"search": {"fulltext": True, "stored": True}})
    published = Column(DateTime, info={"search": {"sortable": True}})

idx = register_model(Article)
```

#### SQLModel

```python
from sqlmodel import SQLModel, Field
from whoosh_modern.models import register_model

class Article(SQLModel, table=True):
    id: int = Field(primary_key=True)
    title: str = Field(sa_column_kwargs={"info": {"search": {"fulltext": True}}})
    published: datetime.datetime

idx = register_model(Article)
```

#### msgspec

```python
import msgspec
from whoosh_modern.models import register_model

class Article(msgspec.Struct):
    title: str = msgspec.field(metadata={"search": {"fulltext": True}})
    published: datetime.datetime

idx = register_model(Article)
```

### Conversion d'instances

```python
doc = idx.to_whoosh_document(book_instance)
writer.add_document(**doc)
```

`to_whoosh_document` gère :
- dataclass : itération via `dataclasses.fields()`
- Pydantic/SQLModel : itération via `model_fields`
- SQLAlchemy : itération via `__mapper__.columns`
- Valeurs Enum converties en `.value`
- `bytes` convertis en chaîne hexadécimale

## Bonnes pratiques

1. **Minimal** : N'indexez que ce que vous cherchez
2. **STORED avec parcimonie** : Augmente la taille de l'index
3. **Champs uniques** : Utilisez `unique=True` pour les identifiants
4. **Boost de champ** : Boostez les champs importants au niveau schéma
5. **TEXT options** : Désactivez `phrase` si vous n'avez pas besoin de recherche de phrase
6. **Champ ID** : Laissez `ModelIndex` auto-détecter ou marquez explicitement avec `SearchOptions(id=True)`


## DOCUMENT (FR): Searching

# Recherche

Guide pour exécuter des recherches, travailler avec les résultats, le scoring et le tri.

## Recherche basique

```python
from whoosh.qparser import QueryParser

with ix.searcher() as searcher:
    qp = QueryParser("content", ix.schema)
    q = qp.parse("bonjour monde")
    results = searcher.search(q)
    for hit in results:
        print(hit["title"], hit.score)
```

## Le Searcher

Le `Searcher` est l'interface principale pour lire l'index.

```python
# Toujours utiliser le context manager
with ix.searcher() as searcher:
    results = searcher.search(query)

# Ou gestion manuelle
searcher = ix.searcher()
try:
    results = searcher.search(query)
finally:
    searcher.close()
```

## QueryParser

Convertit une chaîne de requête en objet Query :

```python
from whoosh.qparser import QueryParser, OrGroup

# AND par défaut entre termes
qp = QueryParser("content", schema)
q = qp.parse("bonjour monde")  # content:bonjour AND content:monde

# Changer l'opérateur par défaut
qp = QueryParser("content", schema, group=OrGroup)
q = qp.parse("bonjour monde")  # content:bonjour OR content:monde
```

## Méthodes de recherche

### search()

```python
results = searcher.search(
    query,
    limit=10,           # Max résultats (None pour tout)
    sortedby=None,      # Clé(s) de tri
    reverse=False,      # Tri inversé
    terms=False,        # Collecter les termes matchés
    filter=None,        # Autoriser seulement ces docnums
    mask=None,          # Exclure ces docnums
    collapse=None       # Facette d'effondrement
)
```

### search_page()

```python
# Page 1, 10 résultats par page (défaut)
results = searcher.search_page(query, 1)

# Page 3, 20 résultats par page
results = searcher.search_page(query, 3, pagelen=20)
```

## Résultats

`Results` agit comme une liste de documents matchés :

```python
results = searcher.search(query)

# Support de slice
first_five = results[0:5]

# Longueur (peut déclencher un recompte)
total = len(results)

# Longueur scorée (ce qui est réellement retourné)
scored = results.scored_length()
```

### Objet Hit

```python
for hit in results:
    # Champs stockés
    title = hit["title"]
    path = hit["path"]

    # Score
    print(hit.score)

    # Surbrillance
    highlights = hit.highlights("content", top=3)
```

## Scoring

Le modèle de scoring par défaut est BM25F :

```python
from whoosh import scoring

with ix.searcher(weighting=scoring.BM25F()) as s:
    results = s.search(query)
```

### Scoring personnalisé

```python
class MyScorer(scoring.WeightingModel):
    def scorer(self, searcher, fieldname, text, qf=1):
        return MyCustomScorer(searcher, fieldname, text, qf)

with ix.searcher(weighting=MyScorer()) as s:
    results = s.search(query)
```

## Tri

```python
from whoosh import sorting

# Tri par champ unique
results = searcher.search(query, sortedby="date")

# Tri inversé
results = searcher.search(query, sortedby="date", reverse=True)

# Tri multi-champs
results = searcher.search(query, sortedby=[
    sorting.FieldFacet("category"),
    sorting.ScoreFacet()
])
```

## Facettes

```python
from whoosh import sorting

facet = sorting.FieldFacet("category")
with searcher.all_features() as features:
    facets = features.facet(facet)
    for cat, count in facets.most_common():
        print(f"{cat}: {count}")
```

## Filtrage et masquage

```python
from whoosh.query import Term

# Autoriser seulement les documents publiés
filter_q = Term("published", True)
results = searcher.search(query, filter=filter_q)

# Exclure les brouillons
mask_q = Term("draft", True)
results = searcher.search(query, mask=mask_q)
```

## Surbrillance

```python
results = searcher.search(query, terms=True)

for hit in results:
    print(hit.highlights("content", top=2))
```

## Recherches à temps limité

```python
from whoosh.collectors import TimeLimitCollector

with ix.searcher() as s:
    c = s.collector(limit=None)
    tlc = TimeLimitCollector(c, timelimit=5.0)
    try:
        s.search_with_collector(query, tlc)
    except TimeLimit:
        print("Recherche annulée: trop lente")
    results = tlc.results()
```


## DOCUMENT (FR): Sorting

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Sorting

The `whoosh.sorting` module provides facets and sort-key computation for ordering and grouping search results.

## Quick start

```python
from whoosh import sorting

# Sort by a field
results = searcher.search(query, sortedby="date")

# Sort descending
results = searcher.search(query, sortedby=sorting.FieldFacet("price", reverse=True))
```

For the full API reference, see [Sorting API](/api/sorting).


## DOCUMENT (FR): Spelling

---
title: "Did you mean..."
sidebar_position: 12
Module: whoosh.spelling
Version: 2.7.4
---
> **Note de traduction** : Cette page n'est pas encore traduite en francais.
> Le contenu anglais est affiche ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# "Did you mean... ?" Correcting errors in user queries

## Overview

Whoosh can quickly suggest replacements for mis-typed words by returning a list
of words from the index (or a dictionary) that are close to the mis-typed word:

```python
with ix.searcher() as s:
    corrector = s.corrector("text")
    for mistyped_word in mistyped_words:
        print(corrector.suggest(mistyped_word, limit=3))
```

See the `whoosh.spelling.Corrector.suggest()` method documentation for
information on the arguments.

Currently the suggestion engine is more like a "typo corrector" than a real
"spell checker" since it doesn't do the kind of sophisticated phonetic matching
or semantic/contextual analysis a good spell checker might. However, it is
still very useful.

There are two main strategies for correcting words:

- Use the terms from an index field.
- Use words from a word list.

## Pulling suggestions from an indexed field

In Whoosh 2.7 and later, spelling suggestions are available on all fields.
However, if you have an analyzer that modifies the indexed words (such as
stemming), you can add `spelling=True` to a field to have it store separate
unmodified versions of the terms for spelling suggestions:

```python
ana = analysis.StemmingAnalyzer()
schema = fields.Schema(text=TEXT(analyzer=ana, spelling=True))
```

You can then use the `whoosh.searching.Searcher.corrector()` method to get a
corrector for a field:

```python
corrector = searcher.corrector("content")
```

The advantage of using the contents of an index field is that when you are
spell checking queries on that index, the suggestions are tailored to the
contents of the index. The disadvantage is that if the indexed documents
contain spelling errors, then the spelling suggestions will also be erroneous.

## Pulling suggestions from a word list

There are plenty of word lists available on the internet you can use to populate
the spelling dictionary. `word_list` can be a list of unicode strings, or a
file object with one word on each line.

```python
from whoosh.spelling import ListCorrector

# word_list must be a sorted list of unicode strings
corrector = ListCorrector(word_list)
```

## Merging two or more correctors

You can combine suggestions from two sources (for example, the contents of an
index field and a word list) using a `whoosh.spelling.MultiCorrector`:

```python
c1 = searcher.corrector("content")
c2 = spelling.ListCorrector(word_list)
corrector = MultiCorrector([c1, c2])
```

## Correcting user queries

You can spell-check a user query using the
`whoosh.searching.Searcher.correct_query()` method:

```python
from whoosh import qparser

# Parse the user query string
qp = qparser.QueryParser("content", myindex.schema)
q = qp.parse(qstring)

# Try correcting the query
with myindex.searcher() as s:
    corrected = s.correct_query(q, qstring)
    if corrected.query != q:
        print("Did you mean:", corrected.string)
```

The `correct_query` method returns an object with the following attributes:

- `query` — A corrected `whoosh.query.Query` tree. Compare it (`==`) with the
  original parsed query to check if the corrector changed anything.
- `string` — A corrected version of the user's query string.
- `tokens` — A list of corrected token objects representing the corrected terms.

You can use a `whoosh.highlight.Formatter` object to format the corrected query
string, for example the `HtmlFormatter` to format it as HTML:

```python
from whoosh import highlight

hf = highlight.HtmlFormatter()
corrected = s.correct_query(q, qstring, formatter=hf)
```

## See also

- [Highlighting](/core/highlight) — Format corrected query strings with a formatter
- [Query Language](/core/query) — Parsing user queries
- [API: spelling](../api/spelling) — Full `whoosh.spelling` reference


## DOCUMENT (FR): Stemming

# Racinement (Stemming) et mots vides

Ce guide couvre l'utilisation des racines (stemmers), des filtres de
mots vides (stop words) et de l'analyse de texte spécifique à chaque
langue avec Whoosh.

## Racines (Stemmers)

Un racine (stemmer) réduit les mots à leur forme racine (par ex.,
"running" → "run", "cats" → "cat"), afin que les différentes formes
du même mot correspondent lors des recherches.

### Utilisation de StemmerFilter

```python
from whoosh.analysis import RegexTokenizer, StemmerFilter
from whoosh.lang.porter import stem
from whoosh import fields

# English Porter stemmer
stem_analyzer = RegexTokenizer() | StemmerFilter(stemfn=stem)

schema = fields.Schema(
    title=fields.TEXT(stored=True),
    content=fields.TEXT(analyzer=stem_analyzer),
)
```

### Racines Snowball

Whoosh inclut des racines Snowball pour plusieurs langues :

```python
from whoosh.analysis import StemmerFilter
from whoosh.lang.snowball import EnglishStemmer

stem_analyzer = RegexTokenizer() | StemmerFilter(stemfn=EnglishStemmer().stem)
```

### Sélection de racine selon la langue

```python
from whoosh.lang import stemmer_for_language, StemmerFilter
from whoosh.analysis import RegexTokenizer

stem = stemmer_for_language("en")
analyzer = RegexTokenizer() | StemmerFilter(stemfn=stem)

# Ou utilisez l'analyseur StemmingAnalyzer :
from whoosh.analysis import StemmingAnalyzer

analyzer = StemmingAnalyzer("en")
```

### Langues disponibles

```python
from whoosh.lang import languages, has_stemmer, has_stopwords

print(languages)  # ('ar', 'da', 'nl', 'en', 'fi', 'fr', ...)
print(has_stemmer("en"))  # True
print(has_stopwords("en"))  # True
```

## Mots vides (Stop Words)

Les mots vides sont des mots fréquents (comme "the", "a", "and") qui
sont généralement filtrés lors de l'indexation car ils apparaissent
dans trop de documents pour être utiles au classement.

### Utilisation de StopFilter

```python
from whoosh.analysis import RegexTokenizer, StopFilter
from whoosh.lang import stopwords_for_language

# English stop words
stop_words = set(stopwords_for_language("en"))
stop_analyzer = RegexTokenizer() | StopFilter(stoplist=stop_words)

schema = fields.Schema(
    content=fields.TEXT(analyzer=stop_analyzer),
)
```

### Combinaison de racinement et de mots vides

```python
from whoosh.analysis import StemmingAnalyzer

# StemmingAnalyzer charge automatiquement le racine et les mots vides pour la langue
analyzer = StemmingAnalyzer("en")

schema = fields.Schema(
    content=fields.TEXT(analyzer=analyzer),
)
```

### Mots vides personnalisés

```python
from whoosh.analysis import RegexTokenizer, StopFilter

# Liste personnalisée de mots vides
custom_stops = frozenset(["the", "a", "an", "foo", "bar"])
analyzer = RegexTokenizer() | StopFilter(stoplist=custom_stops)
```

## StemmingAnalyzer (Recommandé)

Le `StemmingAnalyzer` combine le tokeniseur, le racinement et le
filtrage des mots vides :

```python
from whoosh.analysis import StemmingAnalyzer

# Utilise automatiquement le bon racine et les mots vides pour la langue
analyzer = StemmingAnalyzer("en")

# Vous pouvez remplacer les valeurs par défaut
analyzer = StemmingAnalyzer("en",
                            use_stopwords=True,
                            use_stems=True)
```

### Options de StemmingAnalyzer

- `lang` : Code de langue (ex. : `"en"`, `"fr"`, `"de"`)
- `use_stopwords` : Charge et applique les mots vides (par défaut `True`)
- `use_stems` : Applique le racinement (par défaut `True`)
- `args` : Arguments passés au tokeniseur
- `kwargs` : Arguments du mot-clé pour le racine ou les mots vides

## Considérations spécifiques selon la langue

### Arabe (ISRI Stemmer)

```python
from whoosh.analysis import StemmerFilter
from whoosh.lang.isri import ISRIStemmer

stem_analyzer = RegexTokenizer() | StemmerFilter(stemfn=ISRIStemmer().stem)
```

### Double Métaphone pour la correspondance phonétique

```python
from whoosh.analysis import RegexTokenizer, DoubleMetaphoneFilter

analyzer = RegexTokenizer() | DoubleMetaphoneFilter()
```

## Racinement côté requête

L'analyseur est appliqué à la fois lors de l'indexation et lors de la
requête (via l'analyseur de requête), donc le racinement est automatiquement
appliqué aux termes de recherche :

```python
from whoosh.qparser import QueryParser

# Si l'index utilise le racinement, les requêtes sont racinées aussi
qp = QueryParser("content", schema=ix.schema)
q = qp.parse("running cats")  # Correspondra à "run", "cat", etc.
```

## Analyse N-gramme

Pour la correspondance de sous-chaînes et les requêtes par préfixe,
utilisez les analyseurs N-gramme :

```python
from whoosh.analysis import NgramWordAnalyzer

analyzer = NgramWordAnalyzer(minsize=2, maxsize=4)
schema = fields.Schema(content=fields.TEXT(analyzer=analyzer))
```

Voir le [Guide N-grammes](ngrams.md) pour plus de détails.

## Fournisseurs de racines modernes (Whoosh-NG 2.0)

Whoosh-NG 2.0 introduit un système de fournisseurs de racines de style
plugin avec détection automatique, support de PyStemmer et d'analyseurs
spécifiques à chaque langue. Pour plus de détails, voir le
[Guide des fournisseurs de racines](stemmers-fournisseurs.md).


## DOCUMENT (FR): Threads

---
title: "Concurrency, Locking & Versioning"
sidebar_position: 14
Module: whoosh.index, whoosh.searching, whoosh.store
Version: 2.7.4
---
> **Note de traduction** : Cette page n'est pas encore traduite en francais.
> Le contenu anglais est affiche ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Concurrency, locking, and versioning

## Concurrency

The `FileIndex` object is "stateless" and should be share-able between threads.

A `Reader` object (which underlies the `Searcher` object) wraps open files and
often individual methods rely on consistent file cursor positions (e.g. they
do two `file.read()`s in a row, so if another thread moves the cursor between
the two read calls Bad Things would happen). You should use one Reader/Searcher
per thread in your code.

Readers/Searchers tend to cache information (such as field caches for
sorting), so if you can share one across multiple search requests, it's a big
performance win.

> Whoosh-NG also provides `AsyncWriter` and `BufferedWriter` in
> `whoosh.writing` (see [Indexing](/core/indexing)) as convenient wrappers for
> multi-process write scenarios.

## Locking

Only one thread/process can write to an index at a time. When you open a
writer, it locks the index. If you try to open a writer on the same index in
another thread/process, it will raise `whoosh.store.LockError`.

In a multi-threaded or multi-process environment your code needs to be aware
that opening a writer may raise this exception if a writer is already open.
Whoosh includes a couple of example implementations
(`whoosh.writing.AsyncWriter` and `whoosh.writing.BufferedWriter`) of ways to
work around the write lock.

While the writer is open and during the commit, **the index is still available
for reading**. Existing readers are unaffected and new readers can open the
current index normally.

### Lock files

Locking the index is accomplished by acquiring an exclusive file lock on the
`<indexname>_WRITELOCK` file in the index directory. The file is not deleted
after the file lock is released, so the fact that the file exists **does not**
mean the index is locked.

## Versioning

When you open a reader/searcher, the reader represents a view of the **current
version** of the index. If someone writes changes to the index, any readers
that are already open **will not** pick up the changes automatically. A reader
always sees the index as it existed when the reader was opened.

If you are re-using a Searcher across multiple search requests, you can check
whether the Searcher is a view of the latest version of the index using
`whoosh.searching.Searcher.up_to_date()`. If the searcher is not up to date,
you can get an up-to-date copy of the searcher using
`whoosh.searching.Searcher.refresh()`:

```python
# If 'searcher' is not up-to-date, replace it
searcher = searcher.refresh()
```

If the searcher has the latest version of the index, `refresh()` simply returns
it. Calling `Searcher.refresh()` is more efficient than closing the searcher
and opening a new one, since it will re-use any underlying readers and caches
that haven't changed.

## See also

- [Indexing](/core/indexing) — Writers, `AsyncWriter`, `BufferedWriter`
- [API: writing](../api/writing) — Writer concurrency helpers


## DOCUMENT (FR): Translation Status

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Translation Completion Tracking

- [x] EN quickstart
- [x] EN guides
- [x] EN API pages
- [x] EN examples
- [x] FR quickstart
- [x] FR guides
- [x] FR API pages
- [x] FR examples


## DOCUMENT (FR): Autocomplete

# Autocomplétion avec Whoosh‑NG

Cet exemple démontre la fonctionnalité **autocomplete/suggestion** avec le plugin `whoosh_modern.autocomplete`.

## 1. Installation

```bash
pip install "whoosh-ng[autocomplete]"
```

## 2. Schéma avec champ Keyword pour les termes

```python
from whoosh import index
from whoosh.fields import Schema, TEXT, KEYWORD
from whoosh_modern.autocomplete.plugin import AutocompletePlugin
from whoosh.plugins.manager import PluginManager

schema = Schema(
    title=TEXT(stored=True),
    tags=KEYWORD(stored=True, commas=True),
)

ix = index.create_in("autocomplete_index", schema)
```

## 3. Enregistrer le plugin

```python
AutocompletePlugin().register(PluginManager())
```

## 4. Indexer les documents

```python
with ix.writer() as w:
    w.add_document(title="Python Programming", tags="python,programming,language")
    w.add_document(title="JavaScript Basics", tags="javascript,programming,web")
    w.add_document(title="Machine Learning", tags="ml,ai,data-science")
    w.commit()
```

## 5. Utiliser l’index inversé pour les suggestions

```python
from whoosh_modern.autocomplete.factory import create_autocomplete
from whoosh.registry import AutocompleteRegistry

provider = AutocompleteRegistry.get("inverted")

with ix.searcher() as s:
    for term in s.lexicon("tags"):
        provider.add_term(term, s.doc_count_all())

suggestions = provider.suggest("py", maxdist=1, limit=5)
print(suggestions)  # ['python', 'programming']
```

## Points clés

- Installez avec `pip install whoosh-ng[autocomplete]`.
- Utilisez des champs `KEYWORD` pour les tags/mots-clés.
- Enregistrez `AutocompletePlugin` pour activer les suggestions.
- Le provider inverted supporte les correspondances floues (`maxdist`).


## DOCUMENT (FR): Basic Indexing

# Indexation de base

Exemples pour indexer des documents dans Whoosh‑NG. Chaque section est un script
autonome **exécutable**.

> **Scénario concret** : Vous construisez un moteur de recherche de blog. Vous avez
> un fichier CSV d'articles (`blog_posts.csv`) avec les colonnes `title`, `url`,
> `tags`, `body` et `published_at`.

## 1. Schéma de production

```python
from whoosh import index
from whoosh.fields import Schema, TEXT, ID, KEYWORD, NUMERIC, DATETIME
from datetime import datetime

schema = Schema(
    doc_id=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    url=ID(stored=True),
    tags=KEYWORD(stored=True, commas=True),
    body=TEXT(stored=True, phrase=True),
    published_at=DATETIME(stored=True, sortable=True),
    word_count=NUMERIC(int, stored=True),
)
```

## 2. Créer l'index

```python
from whoosh import index
from whoosh.fields import Schema, TEXT, ID, KEYWORD, NUMERIC, DATETIME
import shutil

shutil.rmtree("blog_index", ignore_errors=True)
ix = index.create_in("blog_index", schema)
```

## 3. Indexer depuis un CSV

```python
import csv
from datetime import datetime

with open("blog_posts.csv", newline="", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    with ix.writer() as writer:
        for row in reader:
            writer.add_document(
                doc_id=row["doc_id"],
                title=row["title"],
                url=row["url"],
                tags=row["tags"],
                published_at=datetime.fromisoformat(row["published_at"]),
                word_count=int(row["word_count"]),
                body=row["body"],
            )
        writer.commit()
```

## 4. Mise à jour incrémentale

```python
updated_posts = [
    {"doc_id": "1", "title": "Titre mis à jour", "body": "Nouveau contenu..."},
]

with ix.writer() as writer:
    for post in updated_posts:
        writer.update_document(
            doc_id=post["doc_id"],
            title=post["title"],
            url=f"/posts/{post['doc_id']}",
            tags="python,search",
            published_at=datetime(2024, 6, 1),
            word_count=len(post["body"].split()),
            body=post["body"],
        )
    writer.commit()
```

## 5. Suppression

```python
from whoosh.query import Term

with ix.writer() as writer:
    writer.delete_by_term("doc_id", "3")
    writer.commit()
```

## 6. Indexation en bloc (10k+ documents)

```python
from whoosh.writing import BufferedWriter

buffered = BufferedWriter(ix, period=60, limit=500)
try:
    for doc in large_dataset:
        with buffered:
            buffered.add_document(**doc)
finally:
    buffered.close()
```

## 7. Recherche sur les données indexées

```python
from whoosh.qparser import QueryParser

ix = index.open_dir("blog_index")

with ix.searcher() as s:
    qp = QueryParser("body", ix.schema)
    q = qp.parse("moteur de recherche")

    results = s.search(q, limit=10)
    for hit in results:
        print(f"{hit['title']} | {hit['url']} | score={hit.score:.3f}")
```


## DOCUMENT (FR): Data Sources

# Sources de données

Whoosh-NG fournit une couche de sources de données flexible pour l'indexation de documents à partir de bases de données SQL, d'API REST, d'API GraphQL, de fichiers et d'autres fournisseurs.

## Protocole DataSource

Toutes les sources implémentent le protocole `DataSource` :

```python
from whoosh_modern.data_sources import DataSource

class DataSource(Protocol):
    @property
    def name(self) -> str: ...

    def discover_schema(self) -> Schema: ...
    def iter_documents(self) -> Iterator[Document]: ...
    def document_count(self) -> int: ...
    def metadata(self) -> Mapping[str, Any]: ...
```

### Protocoles de capacités

| Protocole | Description |
|----------|-------------|
| `DataSource` | Protocole de base : nom, schéma, itération, métadonnées |
| `IncrementalDataSource` | Supporte `iter_changes(since)` |
| `AsyncDataSource` | Diffusion asynchrone via `aiter_documents()` |
| `RefreshableDataSource` | Support de `refresh()` |
| `CountableDataSource` | `document_count()` |
| `MetadataDataSource` | `metadata()` |
| `ObservableDataSource` | Callbacks d'observation pour les changements de documents |

---

## SQLSource

`SQLSource` se connecte aux bases de données SQL et restitue les documents depuis les résultats de requête, avec pooling de connexions automatique.

### Utilisation de base

```python
from whoosh_modern.data_sources.sql import SQLSource
import sqlite3

conn = sqlite3.connect("mydb.db")
source = SQLSource(
    connection=conn,
    query="SELECT * FROM products",
)

schema = source.discover_schema()
for doc in source.iter_documents():
    print(doc["title"], doc["price"])

count = source.document_count()
```

### Pooling de connexions

```python
from whoosh_modern.data_sources.sql import SQLSource

source = SQLSource(
    connection="sqlite:///mydb.db",
    query="SELECT * FROM products",
    pool_size=10,
    pool_recycle=3600,
)
```

### GROUP BY

```python
source = SQLSource(
    connection=conn,
    query="""
        SELECT category, COUNT(*) as doc_count,
               AVG(price) as avg_price
        FROM products GROUP BY category
    """,
)
```

### JOINs avec alias

```python
source = SQLSource(
    connection=conn,
    query="""
        SELECT p.id AS product_id, p.name AS product_name,
               c.name AS category_name
        FROM products p
        JOIN categories c ON p.category_id = c.id
    """,
)
```

### Synchronisation incrémentale

```python
from datetime import datetime

source = SQLSource(
    connection=conn,
    query="SELECT * FROM articles",
    incremental_field="updated_at",
    id_field="id",
)

for doc in source.iter_changes(since=datetime(2025, 1, 1)):
    print(doc["id"], doc["updated_at"])
```

### SQLAlchemySource

```python
from whoosh_modern.data_sources.sqlalchemy_ds import SQLAlchemySource
from sqlalchemy import create_engine

engine = create_engine("postgresql://user:pass@localhost/mydb")
source = SQLAlchemySource(
    engine=engine,
    query="SELECT * FROM articles",
    incremental_field="updated_at",
    id_field="id",
)
```

### PeeweeSource

```python
from whoosh_modern.data_sources.peewee_ds import PeeweeSource
from peewee import SqliteDatabase

db = SqliteDatabase("mydb.db")
source = PeeweeSource(
    database=db,
    model=MyArticleModel,
    fields=["id", "title", "content"],
)
```

### TortoiseSource (async)

```python
from whoosh_modern.data_sources.tortoise_ds import TortoiseSource

source = TortoiseSource(
    model="myapp.models.Article",
    fields=["id", "title", "content"],
)
```

---

## RESTSource

`RESTSource` récupère les documents depuis des API REST avec pagination et authentification.

### Utilisation de base

```python
from whoosh_modern.data_sources.rest import RESTSource

source = RESTSource(
    url="https://api.example.com/v2/products",
    method="GET",
    headers={"Authorization": "Bearer your_token"},
    pagination="page",
    page_size=50,
)

schema = source.discover_schema()
for doc in source.iter_documents():
    print(doc["name"], doc["price"])
```

### Stratégies de pagination

| Stratégie | Paramètres |
|----------|-----------|
| `page` | `?page=N&size=M` |
| `offset` | `?offset=N&limit=M` |
| `cursor` | `?cursor=XYZ&size=M` |

### Authentification

```python
# Bearer token
source = RESTSource(
    url="https://api.example.com/data",
    headers={"Authorization": "Bearer your_token"},
)

# API key
source = RESTSource(
    url="https://api.example.com/data",
    headers={"X-API-Key": "your_api_key"},
)

# Basic auth
import base64
creds = base64.b64encode(b"user:pass").decode()
source = RESTSource(
    url="https://api.example.com/data",
    headers={"Authorization": f"Basic {creds}"},
)
```

### Document Path

Pour les réponses API imbriquées :

```python
source = RESTSource(
    url="https://api.example.com/api/v2/products",
    document_path="data.results",
    pagination="page",
)
```

---

## GraphQLSource

```python
from whoosh_modern.data_sources.graphql import GraphQLSource

source = GraphQLSource(
    url="https://api.example.com/graphql",
    query="""
        query GetProducts($limit: Int!, $offset: Int!) {
            products(limit: $limit, offset: $offset) {
                id
                name
                price
            }
        }
    """,
    pagination="offset",
    page_size=100,
    headers={"Authorization": "Bearer your_token"},
)
```

---

## Sources de fichiers

### FastCSVSource

```python
from whoosh_modern.data_sources.fast_csv import FastCSVSource

source = FastCSVSource(
    file_path="data/products.csv",
    id_field="id",
    incremental_field="updated_at",
)
```

### JSONSource

```python
from whoosh_modern.data_sources.json import JSONSource

source = JSONSource(file_path="data/products.json")
# ou fichier JSONL
source = JSONSource(file_path="data/logs.jsonl", format="jsonl")
```

### ParquetSource

```python
from whoosh_modern.data_sources.parquet_ds import ParquetSource

source = ParquetSource(
    file_path="data/large_dataset.parquet",
    engine="pyarrow",
    batch_size=1000,
)
```

### PandasSource

```python
from whoosh_modern.data_sources.pandas_ds import PandasSource
import pandas as pd

df = pd.read_csv("data/products.csv")
source = PandasSource(dataframe=df)
```

### PolarsSource

```python
from whoosh_modern.data_sources.polars_ds import PolarsSource
import polars as pl

df = pl.read_csv("data/products.csv")
source = PolarsSource(dataframe=df)
```

---

## DataSourceConfig

Pour une configuration programmatique :

```python
from whoosh_modern.data_sources.config import DataSourceConfig

config = DataSourceConfig(
    source_type="sql",
    connection="sqlite:///mydb.db",
    query="SELECT * FROM products",
    id_field="id",
    incremental_field="updated_at",
    mapping={"db_title": "title"},
    exclude=["description_long"],
)

source = config.create_source()
```

### Sources de données disponibles

| Classe | Type | Dépendances |
|-------|------|-------------|
| `SQLSource` | Bases SQL | `sqlite3` (stdlib) |
| `SQLAlchemySource` | SQLAlchemy | `sqlalchemy` |
| `RESTSource` | API REST | aucune (stdlib `urllib`) |
| `GraphQLSource` | API GraphQL | aucune (stdlib `urllib`) |
| `FastCSVSource` | Fichiers CSV | aucune |
| `JSONSource` | JSON/JSONL | aucune |
| `ParquetSource` | Parquet | `pyarrow` ou `pandas` |
| `PandasSource` | DataFrames pandas | `pandas` |
| `PolarsSource` | DataFrames Polars | `polars` |
| `PeeweeSource` | ORM Peewee | `peewee` |
| `TortoiseSource` | ORM Tortoise | `tortoise-orm` |
| `PydanticSource` | Modèles Pydantic | `pydantic` |


## DOCUMENT (FR): Facets

# Gestionnaire de facettes

`FacetManager` gère la configuration des facettes pour un schéma Whoosh.

## Usage basique

```python
from whoosh.fields import Schema, TEXT, NUMERIC, BOOLEAN
from whoosh_modern.facets import FacetManager, TermsFacet, RangeFacet

schema = Schema(
    title=TEXT(stored=True),
    category=TEXT(sortable=True),
    price=NUMERIC(),
    active=BOOLEAN(),
)

manager = FacetManager(schema)
```

## Auto-découverte

| Type Whoosh | Facette auto-découverte |
|-------------|-------------------------|
| `KEYWORD`, `BOOLEAN`, `ID` | `TermsFacet` |
| `NUMERIC` | `RangeFacet` |
| `DATETIME` | `DateRangeFacet` |

```python
facets = manager.get_facets()
manager.is_facetable("category")  # True
manager.is_facetable("title")     # False
```

## Remplacement manuel

```python
manager.set_manual_override("category", {"type": "terms", "limit": 50})
manager.set_manual_override("price", {"type": "range", "buckets": ["0-10", "10-50"]})
```

## Statistiques

```python
stats = manager.get_facet_stats()
# {"total_fields": 4, "auto_discovered_facets": 2, ...}
```


## DOCUMENT (FR): Fastapi Search

# Intégration FastAPI

Un service FastAPI complet exposant la recherche Whoosh-NG via HTTP.

## 1. Installation

```bash
pip install "whoosh-ng[api]" fastapi uvicorn
```

## 2. Créer l'index

```python
# setup_index.py
import json
from whoosh import index
from whoosh.fields import Schema, TEXT, ID

schema = Schema(
    id=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    content=TEXT,
)

ix = index.create_in("docs_index", schema)

with ix.writer() as w:
    for doc in json.load(open("documents.json")):
        w.add_document(
            id=doc["id"],
            title=doc["title"],
            content=doc["content"],
        )
    w.commit()
```

## 3. Service REST

```python
# main.py
from fastapi import FastAPI, Query
from typing import Optional
from whoosh import index
from whoosh.qparser import QueryParser
from whoosh_fastapi import create_app

ix = index.open_dir("docs_index")

# Option A: Utiliser l'aide
app = create_app(ix, prefix="/api/v1")

# Option B: endpoints manuels
# app = FastAPI(title="Document Search API", version="1.0.0")
#
# @app.get("/api/v1/health")
# async def health():
#     return {"status": "ok"}
#
# @app.post("/api/v1/search")
# async def search(q: str = Query(...), limit: int = 10):
#     with ix.searcher() as s:
#         parser = QueryParser("content", ix.schema)
#         results = s.search(parser.parse(q), limit=limit)
#         return {"hits": [dict(h) for h in results], "total": len(results)}

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)
```

## 4. Démarrer le serveur

```bash
uvicorn main:app --reload --port 8000
```

## 5. Tester l'API

```bash
# Vérification de santé
curl http://localhost:8000/api/v1/health

# Recherche
curl -X POST http://localhost:8000/api/v1/search \
  -H "Content-Type: application/json" \
  -d '{"q": "python recherche"}'

# Get document by ID
curl http://localhost:8000/api/v1/documents/doc1

# Autocomplétion
curl "http://localhost:8000/api/v1/autocomplete?q=py"
```

## 6. Indexation en lot

```python
# Ajouter à main.py pour l'indexation dynamique
from fastapi import FastAPI
from whoosh.writing import BufferedWriter

@app.post("/api/v1/index")
async def index_docs(docs: list[dict]):
    with BufferedWriter(ix, period=30, limit=50) as w:
        for doc in docs:
            w.add_document(**doc)
    return {"indexed": len(docs)}
```

## Points clés

- `create_app()` de `whoosh_fastapi` fournit les endpoints `/health`, `/search` et `/autocomplete`.
- Tous les appels bloquants s'exécutent hors boucle d'événements via `run_sync`.
- Utilisez `BufferedWriter` pour l'indexation en masse.
- `WhooshFastAPI` classe offre une enregistrement par endpoint pour les intégrations personnalisées.


## DOCUMENT (FR): Middleware Pipeline

# Pipeline de middleware

Le pipeline de middleware enveloppe les opérations avec des préoccupations transversales : nouvelle tentative, journalisation, etc.

## Architecture

```python
from whoosh_modern.middleware import Middleware, MiddlewarePipeline, RetryMiddleware, LoggingMiddleware

pipeline = MiddlewarePipeline(
    RetryMiddleware(attempts=3, backoff="exponential"),
    LoggingMiddleware(),
)

result = pipeline.execute(my_operation)
```

## RetryMiddleware

```python
from whoosh_modern.middleware import RetryMiddleware

retry = RetryMiddleware(attempts=3, backoff="exponential")

@retry.wrap
def operation():
    return fetch_data()
```

Stratégies de backoff :
- `"exponential"` : 1s, 2s, 4s, 8s...
- `"linear"` : 1s, 2s, 3s, 4s...

## LoggingMiddleware

```python
from whoosh_modern.middleware import LoggingMiddleware
import logging

logger = logging.getLogger("benchmark")
logging_mw = LoggingMiddleware(logger=logger)

@logging_mw.wrap
def tracked():
    return fetch_data()
```


## DOCUMENT (FR): Middleware

# Exemples de Middleware

Des exemples pratiques pour construire et utiliser les middleware de Whoosh-NG.

## 1. Middleware de Logging

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class LoggingMiddleware(Middleware):
    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        print(f"[RECHERCHE] Requête: {context.query}")
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if context.results is not None:
            print(f"[RÉSULTATS] {len(context.results)} résultats trouvés")
        return context
```

## 2. Middleware de Metrics

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class MetricsMiddleware(Middleware):
    def __init__(self) -> None:
        self._metrics = {}

    def after_index(self, context: MiddlewareContext) -> MiddlewareContext:
        self._metrics["documents_indexés"] = self._metrics.get("documents_indexés", 0) + 1
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        self._metrics["recherches_executées"] = self._metrics.get("recherches_executées", 0) + 1
        return context

    def get_metrics(self) -> dict:
        return dict(self._metrics)
```

## 3. Middleware de Cache

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class SearchCacheMiddleware(Middleware):
    def __init__(self) -> None:
        self._cache = {}

    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if context.query and str(context.query) in self._cache:
            context.metadata["_résultat_cache"] = self._cache[str(context.query)]
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if context.query and context.results is not None:
            self._cache[str(context.query)] = context.results
        return context
```

## 4. Appliquer un Middleware

```python
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.integration import apply_middleware_to_searcher

chain = MiddlewareChain([
    LoggingMiddleware(),
    MetricsMiddleware(),
])

with ix.searcher() as base_searcher:
    searcher = apply_middleware_to_searcher(base_searcher, chain.middlewares)
    results = searcher.search(query)
```

## Points clés

| Hook | Phase | Description |
|------|-------|-------------|
| `startup` | Init | Appelé une fois à l'initialisation |
| `shutdown` | Nettoyage | Appelé à la fermeture |
| `before_index` | Indexation | Avant l'ajout d'un document |
| `after_index` | Indexation | Après l'ajout d'un document |
| `before_delete` | Suppression | Avant la suppression |
| `after_delete` | Suppression | Après la suppression |
| `before_search` | Recherche | Avant l'exécution de la requête |
| `after_search` | Recherche | Après le retour des résultats |
| `on_error` | Erreur | En cas d'exception |
| `on_commit` | Commit | Après writer.commit() |


## DOCUMENT (FR): Movie Search

# Application de Recherche de Films

Exemple complet montrant comment créer une petite **application de recherche de films** avec Whoosh‑NG : conception du schéma, indexation à partir d’un fichier JSON, recherche facettée, mise en évidence et filtrage.

## 1. Schéma

```python
from whoosh.fields import Schema, TEXT, ID, KEYWORD, NUMERIC

schema = Schema(
    id=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    director=TEXT(stored=True),
    genre=KEYWORD(stored=True, commas=True, scorable=True),
    year=NUMERIC(int, stored=True),
    synopsis=TEXT,
)
```

## 2. Indexer les documents

```python
import json
import shutil
from whoosh import index

shutil.rmtree("movies", ignore_errors=True)
ix = index.create_in("movies", schema)

movies = json.load(open("movies.json"))

with ix.writer() as w:
    for m in movies:
        w.add_document(
            id=str(m["id"]),
            title=m["title"],
            director=m["director"],
            genre=",".join(m["genres"]),
            year=m["year"],
            synopsis=m["synopsis"],
        )
    w.commit()
```

Fichier `movies.json` :

```json
[
  {
    "id": 1,
    "title": "Blade Runner",
    "director": "Ridley Scott",
    "genres": ["sci-fi", "thriller"],
    "year": 1982,
    "synopsis": "Un chasseur de replicants questionne l’humanité dans un futur pluvieux."
  }
]
```

## 3. Recherche avec facettes et mise en évidence

```python
from whoosh import index
from whoosh.qparser import MultifieldParser
from whoosh.sorting import FieldFacet

ix = index.open_dir("movies")

qp = MultifieldParser(["title", "synopsis", "director"], ix.schema)

with ix.searcher() as s:
    q = qp.parse("sci-fi")

    results = s.search(
        q,
        sortedby=FieldFacet("year", reverse=True),
        groupedby=FieldFacet("genre", allow_overlap=True),
        limit=20,
    )

    for hit in results:
        print(hit["title"], hit["year"], "|", round(hit.score, 2))
        print("  ", hit.highlights("synopsis"))

    print("\nGenres:", results.groups("genre"))
```

## 4. Filtrage

Filtrer les films de science-fiction après 1990 :

```python
from whoosh import index
from whoosh.qparser import QueryParser
from whoosh.query import Term, And, NumericRange

ix = index.open_dir("movies")
qp = QueryParser("synopsis", ix.schema)

with ix.searcher() as s:
    user_q = qp.parse("future")
    filters = And([
        Term("genre", "sci-fi"),
        NumericRange("year", 1990, None),
    ])
    results = s.search(user_q, filter=filters)
    for hit in results:
        print(hit["title"], hit["year"])
```

## Points clés

- `KEYWORD(commas=True)` stocke des champs multi-valeurs facetables.
- `MultifieldParser` recherche sur plusieurs champs avec des boosts optionnels.
- `FieldFacet` permet les facettes et le tri.
- `hit.highlights()` renvoie des fragments mis en évidence prêts à afficher.


## DOCUMENT (FR): Plugin Dev

# Développement de Plugins

Guide complet pour créer, enregistrer et tester vos propres plugins Whoosh-NG.

## 1. Classe de base des Plugins

Tous les plugins héritent de `whoosh.plugins.base.Plugin` :

```python
from whoosh.plugins.base import Plugin

class MyPlugin(Plugin):
    name = "my_plugin"
    version = "1.0.0"
    depends_on = []
    conflicts_with = []
    priority = 0
    middleware = []

    def register(self, manager):
        """Appelé quand le plugin est chargé."""
        manager.register("my_handler", MyHandler())

    def register_hooks(self):
        """Enregistrer les hooks avec le décorateur hookimpl."""
        from whoosh.hooks import hookimpl, register_hook

        @hookimpl
        def on_search(request, response):
            pass

        register_hook("on_search", hookimpl(on_search))
```

## 2. Enregistrement d’un Plugin

### Enregistrement manuel

```python
from whoosh.plugins.manager import PluginManager

plugin = MyPlugin()
PluginManager.register(plugin)
```

### Auto-découverte via Entry Points

Dans `pyproject.toml` :

```toml
[project]
name = "whoosh-ng-my-plugin"

[project.entry-points."whoosh_ng.plugins"]
my_plugin = "my_package.plugin:MyPlugin"
```

Auto-chargement :

```python
from whoosh.plugins.manager import PluginManager

PluginManager.load_plugins()
```

## 3. Exemple de Plugin Provider

```python
from whoosh.plugins.base import Plugin
from whoosh.registry import VectorRegistry

class MyVectorProvider:
    def search(self, query_vector, k=10):
        return [{"doc_id": "1", "score": 0.95}]

class MyVectorPlugin(Plugin):
    name = "my_vector"
    version = "1.0.0"

    def register(self, manager):
        provider = MyVectorProvider()
        VectorRegistry.register("my_vector", provider, self.name)
```

## 4. Tester son Plugin

```python
import pytest
from whoosh.plugins.manager import PluginManager

class TestMyPlugin:
    def test_register(self):
        plugin = MyPlugin()
        manager = PluginManager()
        plugin.register(manager)
        assert "my_handler" in manager._plugins

    def test_entry_point(self):
        manager = PluginManager()
        manager.register(MyPlugin())
        assert "my_plugin" in manager.list_enabled()
```

## 5. API du PluginManager

```python
from whoosh.plugins.manager import PluginManager

manager = PluginManager()
manager.register(MyPlugin())
manager.enable("my_plugin")
manager.disable("my_plugin")
manager.list_plugins()
manager.list_enabled()
plugin = manager.get("my_plugin")
```

## 6. Plugins Intégrés

- `whoosh_modern.vector` - Recherche vectorielle (NumPy)
- `whoosh_modern.autocomplete` - Autocomplétion par index inversé
- `whoosh_fastapi` - Endpoints REST FastAPI

```python
from whoosh.plugins.manager import PluginManager
from whoosh_modern.vector.plugin import VectorPlugin
from whoosh_modern.autocomplete.plugin import AutocompletePlugin

PluginManager.load_plugins()
```


## DOCUMENT (FR): Schema Discovery

# Découverte de schéma

La découverte de schéma infère un schéma Whoosh à partir des métadonnées de résultats ou d'échantillons de documents.

## Depuis les métadonnées de colonnes

```python
from whoosh_modern.schema_discovery import SchemaDiscovery
import sqlite3

conn = sqlite3.connect("benchmark/benchmark_data.db")
cursor = conn.cursor()
cursor.execute("PRAGMA table_info(reuters_articles)")
columns = [(row[1], row[2]) for row in cursor.fetchall()]

schema = SchemaDiscovery.from_result_set(columns)
```

## Depuis des échantillons de documents

```python
from whoosh_modern.data_sources.sql import SQLSource

source = SQLSource(connection=conn, query="SELECT * FROM reuters_articles LIMIT 10")
docs = list(source.iter_documents())[:10]
schema = SchemaDiscovery.from_sample(docs)
```

## Détection du champ ID

```python
id_field = SchemaDiscovery.detect_id_field(dict(schema))
```

## Détection des doublons

`from_result_set` lève `SchemaDiscoveryError` sur les noms de colonnes dupliqués.


## DOCUMENT (FR): Search Models

# Modèles de recherche

Exemples de mapping automatique de modèles Python vers des schémas Whoosh.

## Dataclass

```python
from dataclasses import dataclass
from whoosh.fields import Schema, TEXT, NUMERIC
from whoosh_modern.models import ModelIndex
import tempfile
import shutil

@dataclass
class Book:
    title: str
    year: int
    tags: list[str] | None = None

idx = ModelIndex(Book)
print(idx.schema)
```

## Pydantic

```python
from pydantic import BaseModel
from whoosh_modern.models import register_model

class BookModel(BaseModel):
    title: str
    year: int
    tags: list[str] | None = None

idx = register_model(BookModel)
schema = idx.schema
```

## SQLAlchemy

```python
from sqlalchemy import Column, Integer, String
from whoosh_modern.models import register_model

class BookSQL:
    __tablename__ = "book"
    title = Column(String, info={"search": {"fulltext": True, "stored": True}})
    year = Column(Integer, info={"search": {"sortable": True}})

idx = register_model(BookSQL)
schema = idx.schema
```

## SQLModel

```python
from sqlmodel import SQLModel, Field
from whoosh_modern.models import register_model

class Book(SQLModel, table=True):
    id: int = Field(primary_key=True)
    title: str = Field(sa_column_kwargs={"info": {"search": {"fulltext": True}}})
    year: int

idx = register_model(Book)
schema = idx.schema
```

## msgspec

```python
import msgspec
from whoosh_modern.models import register_model

class Book(msgspec.Struct):
    title: str = msgspec.field(metadata={"search": {"fulltext": True}})
    year: int

idx = register_model(Book)
schema = idx.schema
```

## Indexation de documents

```python
from whoosh import index

tmp = tempfile.mkdtemp()
ix = index.create_in(tmp, schema)

with ix.writer() as w:
    book = Book(title="Guide Whoosh", year=2024, tags=["python", "recherche"])
    doc = idx.to_whoosh_document(book)
    w.add_document(**doc)
    w.commit()
```

## Auto-indexation avec AutoIndexer

```python
from whoosh_modern.models import AutoIndexer

auto = AutoIndexer(ix, on_error="raise")
auto.register(Book)

# Indexer une instance unique
book = Book(title="Nouveau livre", year=2024, tags=["python"])
auto.index(book)

# Supprimer par ID
auto.remove(book)

# Versions asynchrones
await auto.index_async(book)
await auto.remove_async(book)
```

Pour les modèles SQLAlchemy, `AutoIndexer` se connecte automatiquement aux événements `after_insert`, `after_update` et `after_delete`.

## Nettoyage

```python
shutil.rmtree(tmp)
```


## DOCUMENT (FR): Search View

# SearchView

`SearchView` intègre une source de données avec l'indexation Whoosh.

## Usage basique

```python
from whoosh_modern.views import SearchView
from whoosh_modern.data_sources.sql import SQLSource
import sqlite3

conn = sqlite3.connect("benchmark/benchmark_data.db")
source = SQLSource(
    connection=conn,
    query="SELECT * FROM reuters_articles",
    incremental_field="article_date",
    id_field="id",
)

view = SearchView(
    name="reuters",
    source=source,
)

# Créer l'index
ix = view.build("indexdir")
```

## Rafraîchissement incrémental

```python
# Réindexation complète
count = view.reindex()

# Rafraîchissement incrémental
count = view.refresh()
```

## Validation

```python
results = view.validate()
for result in results:
    print(f"Niveau {result.level}: {'PASS' if result.passed else 'FAIL'}")
```

## Mode strict

```python
view = SearchView(
    name="strict",
    source=source,
    strict=True,  # Lever ValidationError en cas d'échec
)
```


## DOCUMENT (FR): Search

# Recherche

Exemples concrets d'interrogation d'un index Whoosh-NG.

> **Scénario** : Vous avez indexé un catalogue de livres (voir `basic-indexing.md`).
> Les exemples ci-dessous montrent les patterns de recherche produit.

## Prérequis

L'index `book_index/` contient ce schéma :

```python
from whoosh.fields import Schema, TEXT, ID, KEYWORD, NUMERIC

schema = Schema(
    isbn=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    author=TEXT(stored=True),
    content=TEXT,
    genre=KEYWORD(stored=True, commas=True),
    published_year=NUMERIC(int, stored=True, sortable=True),
    rating=NUMERIC(float, stored=True, sortable=True),
)
```

## 1. Recherche basique — « Livres sur Python »

```python
from whoosh import index
from whoosh.qparser import QueryParser

ix = index.open_dir("book_index")

with ix.searcher() as s:
    qp = QueryParser("content", ix.schema)
    q = qp.parse("python")

    results = s.search(q, limit=10)
    for hit in results:
        print(f"{hit['title']} par {hit['author']} — score={hit.score:.2f}")
```

## 2. Recherche multi-champs avec boosts

```python
from whoosh.qparser import MultifieldParser

ix = index.open_dir("book_index")
qp = MultifieldParser(
    ["title", "author", "content"],
    ix.schema,
    fieldboosts={"title": 3.0, "author": 2.0, "content": 1.0},
)

q = qp.parse("clean code")

with ix.searcher() as s:
    results = s.search(q, limit=10)
    for hit in results:
        print(f"{hit['title']} — {hit['author']}")
```

## 3. Pagination — « Page 3 des résultats »

```python
ix = index.open_dir("book_index")
qp = QueryParser("content", ix.schema)
q = qp.parse("machine learning")

with ix.searcher() as s:
    page = s.search_page(q, 3, pagelen=15)

    print(f"Page {page.number} / {page.pagecount} ({page.total} résultats)")
    for hit in page:
        print(f"  {hit['title']}")
```

## 4. Tri et filtres — « Sci-fi noté ≥4 après 2010 »

```python
from whoosh.query import Term, And, NumericRange
from whoosh.sorting import FieldFacet

ix = index.open_dir("book_index")
qp = QueryParser("content", ix.schema)
q = qp.parse("space")

with ix.searcher() as s:
    filters = And([
        Term("genre", "sci-fi"),
        NumericRange("published_year", 2010, None),
    ])

    results = s.search(
        q,
        filter=filters,
        sortedby=FieldFacet("rating", reverse=True),
        limit=20,
    )
    for hit in results:
        print(f"{hit['title']} ({hit['published_year']}) — note: {hit['rating']}")
```

## 5. Mise en évidence — « Où la requête correspond »

```python
ix = index.open_dir("book_index")
qp = QueryParser("content", ix.schema)
q = qp.parse("réseaux de neurones")

with ix.searcher() as s:
    results = s.search(q, limit=5)

    for hit in results:
        snippet = hit.highlights("content", top=2)
        print(f"{hit['title']}:")
        print(f"  {snippet}")
```

## 6. Recherche par plage — « Livres publiés en 2023 »

```python
from whoosh.query import NumericRange

ix = index.open_dir("book_index")

with ix.searcher() as s:
    q = NumericRange("published_year", 2023, 2023)
    results = s.search(q)
    print(f"{results.total} livres publiés en 2023")
```

## 7. Recherche par préfixe — « Titres commençant par "Deep" »

```python
from whoosh.query import Prefix

ix = index.open_dir("book_index")

with ix.searcher() as s:
    q = Prefix("title", "Deep")
    results = s.search(q)
    for hit in results:
        print(hit["title"])
```

## 8. Recherche facetée — « Grouper par genre »

```python
from whoosh.sorting import FieldFacet

ix = index.open_dir("book_index")
qp = QueryParser("content", ix.schema)
q = qp.parse("programming")

with ix.searcher() as s:
    results = s.search(q, groupedby=FieldFacet("genre"))

    for genre, group in results.groups("genre").items():
        print(f"{genre}: {len(group)} résultats")
```

## Points clés

- `QueryParser` analyse une chaîne en objet `Query`.
- `MultifieldParser` recherche plusieurs champs avec des boosts.
- `search_page()` gère la pagination.
- `filter` restreint les résultats sans affecter le score.
- `sortedby` trie par valeur de champ ou par score de pertinence.
- `hit.highlights()` renvoie des extraits mis en surbrillance.


## DOCUMENT (FR): Validation

# Framework de validation

Le framework de validation exécute 4 niveaux de vérifications sur une source de données avant l'indexation.

## Les 4 niveaux

| Niveau | Méthode | Objectif |
|-------|---------|---------|
| **Niveau 1** | `validate_structural(source)` | Disponibilité de la source, détection de schéma |
| **Niveau 2** | `validate_search(schema)` | Champs indexables, compatibilité des analyseurs |
| **Niveau 3** | `validate_performance(schema, source)` | Avertissements de performance (TEXT, etc.) |
| **Niveau 4** | `validate_runtime(source, sample_size)` | Itération d'échantillon, validation de types |

## Usage basique

```python
from whoosh_modern.validation import ValidationFramework, ValidationResult
from whoosh_modern.data_sources.sql import SQLSource
import sqlite3

conn = sqlite3.connect("benchmark/benchmark_data.db")
source = SQLSource(connection=conn, query="SELECT * FROM reuters_articles")

validator = ValidationFramework()
results: list[ValidationResult] = validator.validate(source)

for result in results:
    status = "PASS" if result.passed else "FAIL"
    print(f"Niveau {result.level}: {status}")
```

## Validation individuelle

```python
errors = validator.validate_structural(source)
errors = validator.validate_search(schema)
warnings = validator.validate_performance(schema, source)
errors = validator.validate_runtime(source, sample_size=100)
```


## DOCUMENT (FR): Vector Search

# Recherche Vectorielle avec Whoosh‑NG

Cet exemple montre comment activer la **recherche sémantique/vectorielle** avec l’option `vector` supplémentaire. Nous indexons des embeddings et effectuons une recherche de plus proches voisins (k-NN).

## 1. Installer les dépendances optionnelles

```bash
pip install "whoosh-ng[vector]" numpy
```

## 2. Schéma avec champ Vectoriel

```python
from whoosh.fields import Schema, TEXT, ID, VECTOR

schema = Schema(
    doc_id=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    content=TEXT,
    embedding=VECTOR(stored=True, dim=128),
)
```

## 3. Indexer les vecteurs

```python
import numpy as np
from whoosh import index
import shutil

shutil.rmtree("vector_index", ignore_errors=True)
ix = index.create_in("vector_index", schema)

documents = [
    {"doc_id": "doc1", "title": "Python Basics", "content": "Learn Python programming."},
    {"doc_id": "doc2", "title": "Advanced Python", "content": "Deep dive into decorators."},
    {"doc_id": "doc3", "title": "Data Science", "content": "Pandas and NumPy."},
]

np.random.seed(42)
embeddings = {d["doc_id"]: np.random.rand(128).astype(np.float32) for d in documents}

with ix.writer() as w:
    for doc in documents:
        w.add_document(
            doc_id=doc["doc_id"],
            title=doc["title"],
            content=doc["content"],
            embedding=embeddings[doc["doc_id"]].tobytes(),
        )
    w.commit()
```

## 4. Recherche vectorielle avec NumpyProvider

```python
from whoosh_modern.vector.numpy_provider import NumpyProvider
from whoosh_modern.vector.plugin import VectorPlugin
from whoosh.plugins.manager import PluginManager

VectorPlugin().register(PluginManager())

provider = NumpyProvider()
for doc_id, vec in embeddings.items():
    provider.add([(doc_id, vec.tolist())])

query_vec = embeddings["doc1"]
hits = provider.search(query_vec, k=2)

for hit in hits:
    print(f"doc_id={hit.doc_id}, score={hit.score:.3f}")
```

## 5. Utiliser VectorField pour la sérialisation

```python
from whoosh.vector import VectorField

vf = VectorField(dimension=128, name="embedding")

values = [0.1, 0.2, 0.3, 0.4] + [0.0] * 124
raw = vf.vector_to_bytes(values)
restored = vf.bytes_to_vector(raw)
print(restored == tuple(values))  # True
```

## Points clés

- Installez avec `pip install whoosh-ng[vector]`.
- `VECTOR` stocke les octets bruts ; utilisez `VectorField` pour convertir.
- `NumpyProvider` implémente la similarité cosinus.
- Enregistrez le plugin via `VectorPlugin().register(manager)`.
- Utilisez `filter_ids` dans `provider.search()` pour restreindre les documents.


## DOCUMENT (FR): Autocomplete Fournisseurs

# Fournisseurs d'Autocomplétion

Module: `whoosh_modern.autocomplete`
Version: 2.0.0

Le module d'autocomplétion fournit plusieurs stratégies de fournisseurs pour la suggestion de requêtes et la recherche en tapant. Tous les fournisseurs implémentent une interface commune afin de pouvoir changer de stratégie à l'exécution. Les fournisseurs sont enregistrés via le `AutocompleteRegistry` et chargés via des entry points.

## Vue d'ensemble du module

```text
whoosh_modern.autocomplete
    ├── provider.py   # AutocompleteHit, AutocompleteProvider (Protocole)
    ├── ngram.py      # NGramProvider (basé sur des n-grammes de caractères)
    ├── edge_ngram.py # InvertedIndexAutocomplete (correspondance de préfixe par indice inversé)
    ├── fuzzy.py      # FuzzySuggestProvider (correspondance approximative via rapidfuzz)
    ├── factory.py    # create_autocomplete()
    └── plugin.py     # AutocompletePlugin (plugin via entry point)
```

## AutocompleteProvider (Classe de Base)

Située dans `whoosh_modern.autocomplete.provider` :

```python
from whoosh_modern.autocomplete.provider import AutocompleteProvider, AutocompleteHit

class MyProvider(AutocompleteProvider):
    def add(self, phrases: Iterable[str]) -> None:
        """Ajouter des phrases à l'index du fournisseur."""
        ...

    def search(self, prefix: str, limit: int = 10) -> list[AutocompleteHit]:
        """Retourner les suggestions d'autocomplétion pour le préfixe donné."""
        ...
```

### AutocompleteHit

Un objet de résultat simple retourné par les fournisseurs :

```python
class AutocompleteHit:
    def __init__(self, text: str, score: float) -> None:
        self.text = text    # La phrase correspondante
        self.score = score  # Score de pertinence (plus haut = mieux)
```

## Fournisseurs Intégrés

### InvertedIndexAutocomplete

Situé dans `whoosh_modern.autocomplete.edge_ngram`. Utilise une correspondance simple de préfixe contre une liste en mémoire :

```python
from whoosh_modern.autocomplete.edge_ngram import InvertedIndexAutocomplete

provider = InvertedIndexAutocomplete()
provider.add(["python", "pyramid", "pytorch", "java", "javascript"])

hits = provider.search("py", limit=5)
for hit in hits:
    print(f"{hit.text} (score: {hit.score})")
# Output:
# python (score: 0.45)
# pyramid (score: 0.43)
# pytorch (score: 0.43)
```

**Score** : Les correspondances exactes de préfixe obtiennent un bonus de 1.5x ; le score de base est `1.0 / (len(phrase) + 1)`.

### NGramProvider

Situé dans `whoosh_modern.autocomplete.ngram`. Construit un index de n-grammes de caractères pour une correspondance de sous-chaîne souple :

```python
from whoosh_modern.autocomplete.ngram import NGramProvider

provider = NGramProvider(n=3)
provider.add(["python programming", "java development", "rust language"])

hits = provider.search("pyt", limit=5)
for hit in hits:
    print(f"{hit.text} (score: {hit.score})")
```

**Paramètres :**

| Paramètre | Type | Défaut | Description                          |
|-----------|------|---------|--------------------------------------|
| `n`       | `int` | `3`     | Taille des n-grammes de caractères   |

**Fonctionnement** : Les n-grammes sont extraits de chaque phrase (en minuscules). Lors de la recherche, les n-grammes du préfixe sont comparés à l'index. Les phrases avec plus de n-grammes correspondants obtiennent des scores plus élevés.

### FuzzySuggestProvider

Situé dans `whoosh_modern.autocomplete.fuzzy`. Utilise `rapidfuzz` pour une correspondance approximative (fautes de frappe, correspondances partielles) :

```python
from whoosh_modern.autocomplete.fuzzy import FuzzySuggestProvider

# Nécessite: pip install whoosh-ng[fuzzy]
provider = FuzzySuggestProvider(max_distance=2, score_cutoff=50.0)
provider.add(["python", "pyramid", "pytorch", "java", "javascript"])

hits = provider.search("pythn", limit=5)  # Faute de frappe dans "python"
for hit in hits:
    print(f"{hit.text} (score: {hit.score})")
# Output: python (score: 0.95), ...
```

**Paramètres :**

| Paramètre       | Type  | Défaut  | Description                              |
|-----------------|-------|----------|------------------------------------------|
| `max_distance`  | `int` | `2`      | Distance d'édition maximale (réservé pour une utilisation future) |
| `score_cutoff`  | `float` | `50.0` | Score de similarité minimum (échelle 0-100)   |

**Note** : Nécessite `rapidfuzz` (`pip install whoosh-ng[fuzzy]`). Retourne `ImportError` si non installé.

## Fonction d'Usine

Située dans `whoosh_modern.autocomplete.factory` :

```python
from whoosh_modern.autocomplete import create_autocomplete

# Créer n'importe quel fournisseur par nom
provider = create_autocomplete("inverted")   # InvertedIndexAutocomplete
provider = create_autocomplete("ngram", n=3) # NGramProvider avec n personnalisé
provider = create_autocomplete("fuzzy", max_distance=2, score_cutoff=60.0)
```

**Fournisseurs disponibles :**

| Nom         | Classe                    | Dépendance Optionnelle |
|-------------|---------------------------|------------------------|
| `"inverted"`| `InvertedIndexAutocomplete` | Aucune               |
| `"ngram"`   | `NGramProvider`           | Aucune                |
| `"fuzzy"`   | `FuzzySuggestProvider`    | `rapidfuzz`           |

## Enregistrement dans AutocompleteRegistry

Les fournisseurs sont enregistrés dans `whoosh.registry.AutocompleteRegistry` (une instance de `Registry`) :

```python
from whoosh.registry import AutocompleteRegistry
from whoosh_modern.autocomplete import create_autocomplete

# Enregistrer un fournisseur
provider = create_autocomplete("ngram", n=3)
AutocompleteRegistry.register("ngram-suggester", provider, owner="my_app")

# Le récupérer plus tard
suggester = AutocompleteRegistry.get("ngram-suggester")

# Lister tous les fournisseurs enregistrés
print(AutocompleteRegistry.list_keys())
```

## AutocompletePlugin (Entry Point)

Situé dans `whoosh_modern.autocomplete.plugin`, c'est le plugin intégré enregistré via le groupe d'entry points `whoosh_ng.plugins` :

```python
from whoosh_modern.autocomplete.plugin import AutocompletePlugin

# Automatiquement chargé par PluginManager.load_plugins()
# Enregistre le fournisseur "inverted" dans AutocompleteRegistry
```

### Déclaration d'Entry Point

Dans `pyproject.toml` :

```toml
[project.entry-points."whoosh_ng.plugins"]
whoosh_autocomplete = "whoosh_modern.autocomplete.plugin:AutocompletePlugin"
```

### Détails du Plugin

```python
class AutocompletePlugin(Plugin):
    name = "whoosh_autocomplete"
    version = "3.0.0"

    def register(self, manager):
        # Enregistre InvertedIndexAutocomplete comme "inverted"
        AutocompleteRegistry.register(
            "inverted", create_autocomplete("inverted"), self.name
        )

    def register_hooks(self):
        # Enregistre un hook on_search (actuellement un no-op)
        from whoosh.hooks import hookimpl, register_hook
        register_hook("on_search", hookimpl(on_search))
```

## Exemples d'Utilisation

### Utilisation de Base

```python
from whoosh_modern.autocomplete import create_autocomplete

# Créer et peupler un fournisseur
provider = create_autocomplete("inverted")
provider.add([
    "python programming",
    "python tutorial",
    "java tutorial",
    "javascript framework",
])

# Rechercher des suggestions
hits = provider.search("py", limit=3)
for hit in hits:
    print(f"{hit.text}: {hit.score:.3f}")
```

### Correspondance Floue avec Tolérance aux Fautes

```python
from whoosh_modern.autocomplete import create_autocomplete

provider = create_autocomplete("fuzzy", score_cutoff=70.0)
provider.add(["python", "pytorch", "tensorflow", "keras"])

# Même avec une faute, les suggestions pertinentes sont retournées
hits = provider.search("pyton", limit=5)
for hit in hits:
    print(hit.text, hit.score)
```

### Correspondance par N-grammes pour les Mots Partiels

```python
from whoosh_modern.autocomplete import create_autocomplete

# Utiliser des n-grammes de taille 3 pour une meilleure correspondance de sous-chaînes
provider = create_autocomplete("ngram", n=3)
provider.add(["machine learning", "deep learning", "neural networks"])

# Trouve les phrases contenant les n-grammes de "machin"
hits = provider.search("machin", limit=5)
```

### Intégration avec la Recherche

```python
from whoosh_modern.autocomplete import create_autocomplete

# Construire le fournisseur d'autocomplétion
provider = create_autocomplete("inverted")
provider.add(["python", "java", "javascript", "go", "rust"])

# Utiliser dans un endpoint de recherche
def suggest(prefix: str, limit: int = 5):
    hits = provider.search(prefix, limit=limit)
    return [{"text": h.text, "score": h.score} for h in hits]

# Dans votre endpoint FastAPI/REST :
# GET /api/suggest?q=py&limit=5
# Response: [{"text": "python", "score": 0.45}, ...]
```

## Comparaison des Fournisseurs

| Fournisseur              | Correspondance       | Forces                    | Faiblesses                | Dépendance    |
|--------------------------|----------------------|---------------------------|---------------------------|---------------|
| `inverted`               | Préfixe              | Simple, rapide, pas de deps | Pas de tolérance aux fautes | Aucune          |
| `ngram`                  | Chevauchement n-gramme | Correspondance de sous-chaînes, flexible | Plus lent que préfixe     | Aucune        |
| `fuzzy`                  | Distance d'édition   | Tolérance aux fautes, flexible | Nécessite rapidfuzz    | `rapidfuzz`   |

## Installation

```bash
# Autocomplétion core (inverted + n-gram)
pip install whoosh-ng

# Avec correspondance floue
pip install whoosh-ng[fuzzy]

# Analyse moderne complète
pip install whoosh-ng[modern]
```

## Voir Aussi

- [Guide Système de Plugins](plugins-avances.md) — Enregistrement et découverte de plugins
- [Guide Middleware](middleware-pipeline.md) — Intégration du pipeline de middleware
- [Intégration des Providers](provider-integration.md) — Guide complet du pipeline pour tous les providers
- [API: Moderne](../api/modern.md) — Référence complète de l'API pour les extensions d'autocomplétion


## DOCUMENT (FR): Linguistique

# Synonymes & Linguistique

Module: `whoosh_modern.linguistics.synonyms`, `whoosh_modern.linguistics.stemmers`
Version: 2.0.0

Le module de linguistique fournit un moteur complet d'expansion de synonymes et des analyseurs linguistiques spécifiques à chaque langue. Il s'intègre au pipeline de middleware pour étendre les requêtes et les documents avec des synonymes à la fois à l'indexation et au moment de la recherche.

## Vue d'ensemble du module

```text
whoosh_modern.linguistics
    ├── synonyms/
    │   ├── provider.py       # Protocole SynonymProvider + StaticSynonymProvider
    │   ├── yaml_provider.py  # YAMLSynonymProvider
    │   ├── json_provider.py  # JSONSynonymProvider
    │   ├── store.py          # SQLiteSynonymStore
    │   ├── compiler.py       # SynonymCompiler
    │   ├── manager.py        # SynonymManager
    │   ├── middleware.py     # SynonymExpansionMiddleware
    │   └── languages.py      # LANG_SYNONYMS (FR/EN/DE/ES/IT)
    └── stemmers/
        └── __init__.py       # Analyseurs linguistiques (FR/EN/DE/ES/IT)
```

## Fournisseurs de Synonymes

### SynonymProvider (Protocole)

Le protocole de base que tous les fournisseurs de synonymes implémentent :

```python
from whoosh_modern.linguistics.synonyms import SynonymProvider

class MyProvider(SynonymProvider):
    def get_synonyms(self, word: str) -> list[str]:
        """Retourner les synonymes pour le mot donné."""
        ...

    def add_synonym(self, word: str, synonyms: list[str]) -> None:
        """Ajouter des synonymes pour le mot donné."""
        ...

    def remove_synonym(self, word: str, synonym: str) -> None:
        """Supprimer un synonyme pour le mot donné."""
        ...
```

### StaticSynonymProvider

Fournisseur de synonymes en mémoire basé sur un dictionnaire :

```python
from whoosh_modern.linguistics.synonyms import StaticSynonymProvider

provider = StaticSynonymProvider({
    "car": ["automobile", "vehicle", "auto"],
    "house": ["home", "residence"],
})

print(provider.get_synonyms("car"))  # ['automobile', 'vehicle', 'auto']
```

### YAMLSynonymProvider

Charge les synonymes depuis un fichier YAML :

```yaml
# synonyms.yaml
car:
  - automobile
  - vehicle
  - auto
house:
  - home
  - residence
```

```python
from whoosh_modern.linguistics.synonyms import YAMLSynonymProvider

# Nécessite: pip install pyyaml
provider = YAMLSynonymProvider("synonyms.yaml")
print(provider.get_synonyms("car"))  # ['automobile', 'vehicle', 'auto']
```

### JSONSynonymProvider

Charge les synonymes depuis un fichier JSON :

```json
{
    "car": ["automobile", "vehicle", "auto"],
    "house": ["home", "residence"]
}
```

```python
from whoosh_modern.linguistics.synonyms import JSONSynonymProvider

provider = JSONSynonymProvider("synonyms.json")
print(provider.get_synonyms("car"))
```

### SQLiteSynonymStore

Magasin de synonymes persistant basé sur SQLite :

```python
from whoosh_modern.linguistics.synonyms import SQLiteSynonymStore

store = SQLiteSynonymStore("synonyms.db")

# Opérations CRUD
store.add_synonym("car", ["automobile", "vehicle"])
print(store.get_synonyms("car"))  # ['automobile', 'vehicle']
store.remove_synonym("car", "automobile")
print(store.get_synonyms("car"))  # ['vehicle']
store.close()
```

### SynonymCompiler

Précompile les données de synonymes brutes en un format de recherche rapide :

```python
from whoosh_modern.linguistics.synonyms import SynonymCompiler

compiler = SynonymCompiler({"car": ["automobile", "vehicle"]})
compiler.add("house", ["home", "residence"])
compiler.merge({"book": ["publication", "work"]})

compiled = compiler.compile()
print(compiled)
# {'car': ['automobile', 'vehicle'], 'house': ['home', 'residence'], 'book': ['publication', 'work']}
```

## SynonymManager

Le `SynonymManager` est l'interface de haut niveau pour gérer les synonymes. Il encapsule un `StaticSynonymProvider` en interne et prend en charge l'import/export :

```python
from whoosh_modern.linguistics.synonyms import SynonymManager

manager = SynonymManager({"car": ["automobile", "vehicle"]})

# CRUD
manager.add_synonyms("house", ["home", "residence"])
print(manager.get_synonyms("house"))  # ['home', 'residence']
manager.remove_synonym("house", "home")

# Import depuis des sources externes
manager.import_yaml("synonyms.yaml")   # Nécessite PyYAML
manager.import_json("synonyms.json")

# Export
manager.export_json("output.json")
```

### Flux de Travail d'Import/Export

```python
# Import depuis YAML
manager = SynonymManager()
manager.import_yaml("my_synonyms.yaml")

# Export vers JSON (ex: pour migration ou sauvegarde)
manager.export_json("backup.json")
```

## Synonymes Linguistiques Prédéfinis

Le dictionnaire `LANG_SYNONYMS` contient des mappings de synonymes de démarrage pour cinq langues :

```python
from whoosh_modern.linguistics.synonyms import LANG_SYNONYMS

# Langues disponibles : fr, en, de, es, it
french_syns = LANG_SYNONYMS["fr"]
print(french_syns["voiture"])  # ['automobile', 'véhicule']

english_syns = LANG_SYNONYMS["en"]
print(english_syns["car"])  # ['automobile', 'vehicle']

# Initialiser un SynonymManager avec une langue
manager = SynonymManager(LANG_SYNONYMS["fr"])
```

| Langue   | Code | Exemple                               |
|----------|------|---------------------------------------|
| Français | `fr` | `"voiture": ["automobile", "véhicule"]` |
| Anglais  | `en` | `"car": ["automobile", "vehicle"]`    |
| Allemand | `de` | `"auto": ["wagen", "fahrzeug"]`       |
| Espagnol | `es` | `"coche": ["automóvil", "vehículo"]`  |
| Italien  | `it` | `"auto": ["automobile", "veicolo"]`   |

> **Note** : Ce sont des dictionnaires de démarrage minimaux pour la démonstration et les tests. Les déploiements de production devraient charger depuis des sources élaborées ou spécifiques au domaine.

## SynonymExpansionMiddleware

Intègre l'expansion de synonymes dans le pipeline de middleware. Elle étend à la fois les requêtes de recherche et les champs de documents indexés :

```python
from whoosh_modern.linguistics.synonyms import (
    SynonymManager,
    SynonymExpansionMiddleware,
)

# Créer un gestionnaire avec vos synonymes
manager = SynonymManager({
    "car": ["automobile", "vehicle"],
    "house": ["home", "residence"],
})

# Créer le middleware
middleware = SynonymExpansionMiddleware(manager)

# L'enregistrer auprès du PluginManager ou MiddlewareChain
from whoosh.plugins.manager import PluginManager
PluginManager._default.register_middleware("synonym", middleware)
```

### Fonctionnement

- **`before_search`** : Étend `context.query` en ajoutant les synonymes de chaque token
- **`before_index`** : Étend les valeurs de type chaîne dans `context.document` en ajoutant les synonymes

```python
# Avant : query = "car"
# Après :  query = "car automobile vehicle"

# Avant : document = {"title": "house for sale"}
# Après :  document = {"title": "house for sale home residence"}
```

## Analyseurs Linguistiques Spécifiques

Situés dans `whoosh_modern.linguistics.stemmers`, ces analyseurs combinent tokenisation, stemme et suppression des mots vides :

```python
from whoosh_modern.linguistics.stemmers import (
    EnglishAnalyzer,
    FrenchAnalyzer,
    GermanAnalyzer,
    SpanishAnalyzer,
    ItalianAnalyzer,
)

# Chaque analyseur est appelable et retourne une liste de tokens
analyzer = EnglishAnalyzer()
tokens = analyzer("The running cats")
# tokens sont stemmés: ["run", "cat"] (mots vides supprimés)
```

## Voir Aussi

- [Guide Stemmers](stemmers-fournisseurs.md) — Fournisseurs de stemmers et analyseurs linguistiques
- [Guide Middleware](middleware-pipeline.md) — Intégration du pipeline de middleware
- [Intégration des Providers](provider-integration.md) — Guide complet du pipeline pour tous les providers
- [API: Linguistique](../api/modern.md) — Référence complète de l'API


## DOCUMENT (FR): Plugins Avances

# Système de Plugins

Module: `whoosh.plugins.manager`
Version: 2.0.0

L'architecture à plugins de Whoosh-NG permet aux paquets externes d'étendre le moteur d'indexation, de recherche et d'analyse de texte. Les plugins sont découverts via les [entry points](https://docs.python.org/3/library/importlib.metadata.html#entry-points) Python déclarés dans `pyproject.toml` et gérés par le `PluginManager`.

## Vue d'ensemble de l'architecture

```text
PluginManager (singleton)
    ├── load_plugins(group)          # Auto-découverte depuis entry points
    ├── register(plugin)             # Enregistrement manuel
    ├── enable(name) / disable(name) # Activer/désactiver le cycle de vie
    ├── get(name) / list_plugins()   # Inspection
    ├── get_middleware_chain()       # Construire MiddlewareChain depuis middleware des plugins
    ├── register_datasource()        # Enregistrer un fournisseur de source de données
    ├── register_vector_provider()   # Enregistrer un fournisseur de vecteurs
    ├── register_middleware()        # Enregistrer une instance de middleware
    ├── register_embedding()         # Enregistrer un fournisseur d'embeddings
    ├── register_analyzer()          # Enregistrer un analyseur nommé
    └── register_query_rewriter()    # Enregistrer un réécrivant de requête
```

## Classes de Base des Plugins

### Plugin (ABC)

La classe de base pour tous les plugins. Les sous-classes définissent les attributs de classe et implémentent `register()` :

```python
from whoosh.plugins.manager import Plugin, PluginMetadata

class MyPlugin(Plugin):
    name = "my_plugin"
    version = "1.0.0"

    def register(self, manager: PluginManager) -> None:
        """Appelé quand le plugin est chargé ; enregistrer les fournisseurs ici."""
        manager.register_middleware("my_module.MyMiddleware", MyMiddleware())

    def register_hooks(self) -> None:
        """Enregistrer les hooks d'événements (optionnel)."""
        from whoosh.hooks import hookimpl, register_hook

        @hookimpl
        def on_search(request, response):
            pass
        register_hook("on_search", hookimpl(on_search))
```

### AnalyzerPlugin

Pour les plugins qui fournissent des tokenizers/analyseurs personnalisés :

```python
from whoosh.plugins.manager import AnalyzerPlugin

class MyAnalyzerPlugin(AnalyzerPlugin):
    name = "my_analyzer"

    def register(self, manager):
        manager.register_analyzer("my_analyzer", MyTokenizer())
```

### QueryRewritePlugin

Pour les plugins qui transforment les requêtes avant l'exécution :

```python
from whoosh.plugins.manager import QueryRewritePlugin

class SynonymRewriterPlugin(QueryRewritePlugin):
    name = "synonym_rewriter"

    def rewrite(self, query, searcher):
        # Retourner la requête modifiée
        return query
```

## PluginMetadata

Un dataclass décrivant les métadonnées du plugin :

| Champ         | Type              | Description                            |
|---------------|-------------------|----------------------------------------|
| `name`        | `str`             | Nom unique du plugin                   |
| `version`     | `str`             | Version sémantique                     |
| `depends_on`  | `list[str]`       | Noms des plugins requis                |
| `priority`    | `int`             | Priorité d'ordre de chargement (plus haut = plus tard) |
| `middleware`  | `list[str]`       | Chemins pointés vers les classes de middleware |

## Groupes d'Entry Points

Le `PluginManager` découvre les plugins depuis ces groupes d'entry points standards :

| Groupe                      | Utilisation                          |
|-----------------------------|--------------------------------------|
| `whoosh.plugins`            | Plugins généraux                     |
| `whoosh.datasources`        | Fournisseurs de sources de données   |
| `whoosh.vector.providers`   | Fournisseurs de similarité vectorielle |
| `whoosh.middlewares`        | Classes de middleware                 |
| `whoosh.embeddings`         | Fournisseurs de modèles d'embedding  |
| `whoosh.language`           | Analyseurs linguistiques             |
| `whoosh.apps`               | Usines d'applications (FastAPI, admin, etc.) |

## Créer et Déployer un Plugin

### Étape 1 : Définir la Classe du Plugin

```python
# my_plugin/plugin.py
from whoosh.plugins.manager import Plugin
from whoosh.registry import VectorRegistry

class MyVectorPlugin(Plugin):
    name = "my_vector"
    version = "1.0.0"
    depends_on = []
    conflicts_with = []
    priority = 0
    middleware = []

    def register(self, manager):
        """Enregistrer un fournisseur de vecteurs dans le VectorRegistry."""
        provider = MyCustomVectorProvider()
        VectorRegistry.register("my_vector", provider, owner=self.name)

    def register_hooks(self):
        """Enregistrer les hooks optionnels (ex: on_search, on_index)."""
        pass
```

### Étape 2 : Déclarer l'Entry Point

Dans votre `pyproject.toml` :

```toml
[project]
name = "whoosh-ng-my-vector"
version = "1.0.0"
dependencies = ["whoosh-ng>=2.0"]

[project.entry-points."whoosh_ng.plugins"]
my_vector = "my_plugin.plugin:MyVectorPlugin"
```

### Étape 3 : Installer et Vérifier

```bash
pip install -e .
```

```python
# Vérifier que le plugin est bien enregstré
from whoosh.plugins.manager import PluginManager

PluginManager.load_plugins()  # Découvre tous les entry points

manager = PluginManager._default
print(manager.list_plugins())
# ['whoosh_autocomplete', 'whoosh_vector', ..., 'my_vector']

# Vérifier le registre
from whoosh.registry import VectorRegistry
print(VectorRegistry.list_keys())
# ['my_vector', 'numpy']
```

## Enregistrement Manuel (Sans Entry Point)

Pour les tests ou un usage programmatique :

```python
from whoosh.plugins.manager import PluginManager

manager = PluginManager()
manager.register(MyVectorPlugin())
manager.enable("my_vector")
```

## Cycle de Vie d'un Plugin

```
1. Entry point découvert  ───►  2. register() appelé  ───►  3. register_hooks()
   │                               │                            │
   └── load_plugins(group)          └── register provider/     └── register_hook()
                                      middleware/analyzer
```

### Activation / Désactivation

```python
from whoosh.plugins.manager import PluginManager

manager = PluginManager._default

manager.enable("my_vector")    # Activer un plugin
manager.disable("my_vector")   # Désactiver un plugin
print(manager.list_enabled())  # Seuls les plugins activés
```

### Validation de Version

```python
# Vérifie si un plugin respecte une version minimale
ok = manager.validate_version("my_vector", "1.0.0")
print(ok)  # True si la version du plugin >= 1.0.0
```

### Détection de Conflits

```python
# Vérifie si deux plugins entrent en conflit
if manager.detect_conflicts("plugin_a", "plugin_b"):
    print("Ces plugins ne peuvent pas être chargés ensemble")
```

## API du PluginManager

### `PluginManager.load_plugins(group=None)`

Charge tous les plugins depuis les groupes d'entry points. Si `group` est `None`, charge tous les groupes standards (`STANDARD_GROUPS`).

### `PluginManager.register(plugin)`

Enregistre une instance de plugin. Appelle `plugin.register(self)` et `plugin.register_hooks()`. Supporte les méthodes `register()` asynchrones via `asyncio`.

### `PluginManager.get_middleware_chain()`

Construit et retourne une `MiddlewareChain` à partir de tous les plugins qui déclarent une liste `middleware`. Les classes de middleware sont importées et instanciées par chemin pointé.

### Méthodes d'Enregistrement dans les Registres

| Méthode                       | Description                          |
|-------------------------------|--------------------------------------|
| `register_analyzer(name, analyzer)` | Enregistrer un analyseur nommé   |
| `register_datasource(name, datasource)` | Enregistrer une source de données  |
| `register_vector_provider(name, provider)` | Enregistrer un fournisseur de vecteurs |
| `register_middleware(name, middleware)` | Enregistrer une instance de middleware |
| `register_embedding(name, embedding)` | Enregistrer un fournisseur d'embeddings |
| `register_query_rewriter(plugin)` | Enregistrer un plugin réécrivant des requêtes |

### Méthodes de Recherche

| Méthode                       | Retourne                          |
|-------------------------------|----------------------------------|
| `get(name)`                   | Instance `Plugin`                |
| `list_plugins()`              | Noms de tous les plugins enregistrés |
| `list_enabled()`              | Noms des plugins activés         |
| `get_analyzer(name)`          | Analyseur callable               |
| `list_analyzers()`            | Noms des analyseurs enregistrés  |
| `list_datasources()`          | Noms des sources de données enregistrées |
| `list_vector_providers()`     | Noms des fournisseurs de vecteurs enregistrés |
| `list_middlewares()`          | Noms des middleware enregistrés  |
| `list_embeddings()`           | Noms des fournisseurs d'embeddings enregistrés |
| `list_query_rewriters()`      | Noms des réécrivants enregistrés |

## Plugins Intégrés

| Plugin            | Module                  | Groupe d'Entry Point       |
|-------------------|-------------------------|----------------------------|
| `whoosh_autocomplete` | `whoosh_modern.autocomplete.plugin` | `whoosh.plugins` |
| `whoosh_vector`   | `whoosh_modern.vector.plugin`      | `whoosh.plugins` |
| `whoosh_fastapi`  | `whoosh_fastapi`                  | `whoosh.apps` |
| `whoosh_observability` | `whoosh.middleware.metrics`  | `whoosh.middlewares` |
| `whoosh_admin`    | `whoosh_admin`                   | `whoosh.apps` |

## Bonnes Pratiques

1. **Responsabilité unique** : Un plugin, une fonctionnalité
2. **Déclarez les dépendances** : Utilisez `depends_on` pour les plugins requis
3. **Version sémantique** : Incrémentez la version pour les changements d'API
4. **Degradation gracieuse** : Vérifiez les dépendances optionnelles dans `register()`
5. **Pas d'effets de bord dans `__init__`** : Toute l'initialisation dans `register()`
6. **Nettoyage** : Si applicable, fournissez une logique de teardown

## Voir Aussi

- [Guide Middleware](middleware-pipeline.md) — Pipeline hooks et middleware personnalisé
- [Intégration des Providers](provider-integration.md) — Guide complet du pipeline pour tous les providers
- [Exemple: Développement de Plugin](../examples/plugin-dev.md) — Tutoriel pas à pas
- [API: Plugins](../api/plugins.md) — Référence complète de l'API


## DOCUMENT (FR): Stemmers Fournisseurs

# Fournisseurs de Stemmers

Module: `whoosh_modern.analysis.stemmer_providers`, `whoosh_modern.analysis.stemming_analyzer`, `whoosh_modern.linguistics.stemmers`
Version: 2.0.0

Le système de fournisseurs de stemmers donne un contrôle flexible sur le backend de stemming utilisé pour l'analyse de texte. Il prend en charge la détection automatique, la sélection explicite du backend et l'enregistrement de stemmers personnalisés — le tout avec une API propre de type plugin.

## Vue d'ensemble du module

```text
whoosh_modern.analysis
    ├── stemmer_providers.py   # Protocole StemmerProvider, fournisseurs Internal/PyStemmer, register_stemmer, get_stemmer
    └── stemming_analyzer.py   # StemmingAnalyzer amélioré avec support plugin

whoosh_modern.linguistics.stemmers
    └── __init__.py            # Analyseurs linguistiques (FR/EN/DE/ES/IT)
```

## Protocole StemmerProvider

Situé dans `whoosh_modern.analysis.stemmer_providers` :

```python
from whoosh_modern.analysis.stemmer_providers import StemmerProvider

class MyStemmer(StemmerProvider):
    def stem(self, word: str) -> str:
        """Réduire un mot à sa racine."""
        ...

    @property
    def name(self) -> str:
        """Retourner le nom du stemmer."""
        return "my_stemmer"

    @property
    def language(self) -> str:
        """Retourner le code de langue."""
        return "english"
```

## Obtenir un Stemmer

### Détection Automatique (Recommandé)

La fonction `get_stemmer("auto", language)` sélectionne automatiquement le meilleur backend disponible :

```python
from whoosh_modern.analysis.stemmer_providers import get_stemmer

# Détection automatique : préfère PyStemmer si installé, sinon fallback interne
stemmer = get_stemmer("auto", "english")
print(stemmer.stem("running"))  # "run"
print(stemmer.name)             # "pystemmer" ou "internal"
```

**Ordre de priorité :**
1. **PyStemmer** (le plus rapide, nécessite `pip install whoosh-ng[fast-stemming]`)
2. **Stemmer interne** (Porter, toujours disponible)

### Sélection Explicite du Backend

```python
from whoosh_modern.analysis.stemmer_providers import get_stemmer

# Forcer le stemmer interne
stemmer = get_stemmer("internal", "english")

# Forcer PyStemmer (nécessite l'installation)
stemmer = get_stemmer("pystemmer", "english")
```

### Lister les Backends Disponibles

```python
from whoosh_modern.analysis.stemmer_providers import list_available_backends

backends = list_available_backends()
print(backends)
# {'internal': 'available', 'pystemmer': 'available', 'my_custom': 'registered'}
```

| Backend       | Chaîne de statut    | Nécessite                          |
|---------------|---------------------|-------------------------------------|
| `internal`    | `"available"`       | Aucun (toujours inclus)             |
| `pystemmer`   | `"available"` / `"not installed"` | `pip install whoosh-ng[fast-stemming]` |
| Personnalisé  | `"registered"`      | Enregistré via `@register_stemmer` |

## Fournisseurs de Stemmers Intégrés

### InternalStemmerProvider

Enveloppe le stemmer Porter intégré de Whoosh. Toujours disponible (aucune dépendance externe) :

```python
from whoosh_modern.analysis.stemmer_providers import InternalStemmerProvider

stemmer = InternalStemmerProvider("english")
print(stemmer.stem("cats"))    # "cat"
print(stemmer.stem("running")) # "run"
```

### PyStemmerProvider

Enveloppe la bibliothèque `Stemmer` pour un stemming haute performance. Supporte toutes les langues Snowball :

```python
from whoosh_modern.analysis.stemmer_providers import PyStemmerProvider

# Nécessite: pip install whoosh-ng[fast-stemming]
stemmer = PyStemmerProvider("english")
print(stemmer.stem("cats"))    # "cat"
```

**Note** : Ce fournisseur appelle `self._stemmer.stemWord(word)` pour réduire les mots. Assurez-vous que PyStemmer est installé ou la détection automatique basculera vers le stemmer interne.

### IdentityStemmerProvider

Un stemmer sans opération pour les tests ou lorsque le stemming n'est pas souhaité :

```python
from whoosh_modern.analysis.stemmer_providers import IdentityStemmerProvider

stemmer = IdentityStemmerProvider()
print(stemmer.stem("anything"))  # "anything"
```

## Enregistrer un Stemmer Personnalisé

Utilisez le décorateur `@register_stemmer` :

```python
from whoosh_modern.analysis.stemmer_providers import register_stemmer

@register_stemmer("simple")
class SimpleStemmer:
    def stem(self, word: str) -> str:
        # Suppression simple de suffixe
        if word.endswith("s") and len(word) > 3:
            return word[:-1]
        return word

    @property
    def name(self) -> str:
        return "simple"

    @property
    def language(self) -> str:
        return "english"

# Maintenant l'utiliser
from whoosh_modern.analysis.stemmer_providers import get_stemmer

stemmer = get_stemmer("simple", "english")
print(stemmer.stem("cats"))  # "cat"
```

## StemmingAnalyzer (Amélioré)

Situé dans `whoosh_modern.analysis.stemming_analyzer`, c'est le point d'entrée principal pour créer des analyseurs linguistiques :

```python
from whoosh_modern.analysis import StemmingAnalyzer

# Détection automatique du meilleur stemmer pour l'anglais
analyzer = StemmingAnalyzer(stemmer="auto", language="english")

# Stemmer interne explicite
analyzer = StemmingAnalyzer(stemmer="internal", language="english")

# Backend PyStemmer (si installé)
analyzer = StemmingAnalyzer(stemmer="pystemmer", language="french")

# Instance de fournisseur de stemmer personnalisé
analyzer = StemmingAnalyzer(stemmer=my_stemmer_instance)
```

### Paramètres de StemmingAnalyzer

| Paramètre   | Type                          | Défaut                   | Description                      |
|-------------|-------------------------------|--------------------------|----------------------------------|
| `expression`| Motif regex                   | motif de token par défaut | Tokenisation regex             |
| `stoplist`  | Itérable de mots vides        | `whoosh.analysis.STOP_WORDS` | Mots vides à filtrer         |
| `minsize`   | `int`                         | `2`                      | Longueur minimale du token       |
| `maxsize`   | `int \| None`                 | `None`                   | Longueur maximale du token       |
| `gaps`      | `bool`                        | `False`                  | Diviser sur l'expression vs correspondre |
| `stemmer`   | `str \| StemmerProvider`      | `"auto"`                 | Backend de stemmer               |
| `language`  | `str`                         | `"english"`              | Code de langue                   |
| `ignore`    | `set[str] \| None`            | `None`                   | Mots à ignorer                   |
| `cachesize` | `int`                         | `50000`                  | Taille du cache de stemming      |

### Utilisation avec les Types de Champs

```python
from whoosh_modern.analysis import StemmingAnalyzer
from whoosh.fields import Schema, TEXT

# Stemmer anglais avec mots vides
en_analyzer = StemmingAnalyzer("auto", language="english")

# Stemmer français
fr_analyzer = StemmingAnalyzer("auto", language="french")

schema = Schema(
    title=TEXT(stored=True),
    content_en=TEXT(analyzer=en_analyzer),
    content_fr=TEXT(analyzer=fr_analyzer),
)
```

## Analyseurs Linguistiques Spécifiques

Des analyseurs prêts à l'emploi pour cinq langues, disponibles dans `whoosh_modern.linguistics.stemmers` :

```python
from whoosh_modern.linguistics.stemmers import (
    EnglishAnalyzer,
    FrenchAnalyzer,
    GermanAnalyzer,
    SpanishAnalyzer,
    ItalianAnalyzer,
)

# Chaque analyseur est appelable et retourne une liste de tokens
en = EnglishAnalyzer()
tokens = en("The quick brown foxes")
# les tokens sont stemmés: ["quick", "brown", "fox"] (mots vides comme "the" supprimés)
```

### Analyseurs Linguistiques Disponibles

| Classe             | Langue    | Module                              |
|-------------------|-----------|-------------------------------------|
| `EnglishAnalyzer` | Anglais   | `whoosh_modern.linguistics.stemmers` |
| `FrenchAnalyzer`  | Français  | `whoosh_modern.linguistics.stemmers` |
| `GermanAnalyzer`  | Allemand  | `whoosh_modern.linguistics.stemmers` |
| `SpanishAnalyzer` | Espagnol  | `whoosh_modern.linguistics.stemmers` |
| `ItalianAnalyzer` | Italien   | `whoosh_modern.linguistics.stemmers` |

Chaque analyseur utilise en interne `get_stemmer("auto", language)` pour sélectionner le meilleur backend disponible et applique des mots vides spécifiques à la langue.

## Validation de la Compatibilité des Stemmers

Validez qu'un fournisseur de stemmer fonctionne correctement avec un ensemble de mots de test :

```python
from whoosh_modern.analysis.stemmer_providers import (
    get_stemmer,
    validate_stemmer_compatibility,
)

stemmer = get_stemmer("auto", "english")
report = validate_stemmer_compatibility(stemmer, ["running", "cats", "jumps", "houses"])

print(report["total_words"])   # 4
print(report["successful"])    # 4 (ou moins si erreurs)
print(report["failed"])        # 0
print(report["results"])       # [{'word': 'running', 'stemmed': 'run', 'success': True}, ...]
```

### Structure du Rapport de Compatibilité

| Champ          | Type       | Description                          |
|----------------|------------|--------------------------------------|
| `provider`     | `str`      | Nom du fournisseur de stemmer        |
| `language`     | `str`      | Code de langue                       |
| `total_words`  | `int`      | Nombre total de mots de test         |
| `successful`   | `int`      | Mots stemmés avec succès             |
| `failed`       | `int`      | Mots qui ont échoué                  |
| `results`      | `list[dict]` | Résultats par mot avec `word`, `stemmed`, `success` |

## Intégration avec StemmingMiddleware

Les fournisseurs de stemmers peuvent être utilisés avec le `StemmingMiddleware` de `whoosh_modern.middleware.analyzer` :

```python
from whoosh_modern.analysis.stemmer_providers import get_stemmer
from whoosh_modern.middleware.analyzer import StemmingMiddleware

stemmer = get_stemmer("auto", "english")
middleware = StemmingMiddleware(
    stemmer=stemmer.stem,
    fields=["title", "content"],  # Ne stemmer que ces champs
    stem_query=True,              # Also stemmer la requête de recherche
)
```

## Migration depuis Whoosh Classique

### Ancienne API (Whoosh 1.x/2.x)

```python
from whoosh.analysis import StemmingAnalyzer as OldAnalyzer
analyzer = OldAnalyzer("en")  # Codé en dur sur "english"
```

### Nouvelle API (Whoosh-NG 2.0)

```python
from whoosh_modern.analysis import StemmingAnalyzer

# Détection automatique du backend (recommandé)
analyzer = StemmingAnalyzer("auto", language="en")

# Ou utiliser un analyseur linguistique
from whoosh_modern.linguistics.stemmers import EnglishAnalyzer
analyzer = EnglishAnalyzer()
```

> **Note** : L'ancienne `StemmingAnalyzer("en")` était codée en dur sur la langue `"english"`. La nouvelle `StemmingAnalyzer(stemmer, language)` est explicite et prend en charge toutes les langues Snowball via PyStemmer.

## Installation

```bash
# Sans PyStemmer (utilise le stemmer interne, plus lent)
pip install whoosh-ng

# Avec PyStemmer (recommandé, plus rapide)
pip install whoosh-ng[fast-stemming]

# Analyse moderne complète
pip install whoosh-ng[modern]
```

## Intégration des Fournisseurs de Stemmers dans le Pipeline

Le système `StemmerProvider` s'intègre à **deux niveaux** : les analyseurs de niveau champ et le middleware de pipeline. Comprendre les deux est essentiel pour éviter le double-stemming.

### Architecture

```text
┌─────────────────────────────────────────────────────────────────┐
│  StemmingAnalyzer (niveau champ, dans Schema)                   │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ RegexTokenizer() │ StopFilter │ StemmingAnalyzer          │  │
│  │                    (mots vides)    │                       │  │
│  │                                   ▼                       │  │
│  │                         stemfn = provider.stem            │  │
│  │                                   │                       │  │
│  │                                   ▼                       │  │
│  │                         Token(stemmed=True)                │  │
│  └───────────────────────────────────────────────────────────┘  │
│                                                                 │
│  Appliqué par Whoosh core à l'indexation ET à la recherche     │
│  (via QueryParser). Automatique, aucun middleware nécessaire.   │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│  StemmingMiddleware (niveau pipeline)                           │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ before_index(context)                                     │  │
│  │   └── stemmer toutes les valeurs str dans context.document  │  │
│  │                                                             │  │
│  │ before_search(context)                                     │  │
│  │   └── stemmer context.query si stem_query=True             │  │
│  └───────────────────────────────────────────────────────────┘  │
│                                                                 │
│  Intégré dans MiddlewareChain. Activation manuelle.              │
└─────────────────────────────────────────────────────────────────┘
```

### Niveau 1 : Niveau champ (automatique)

Le `StemmingAnalyzer` encapsule le `StemmingAnalyzer` intégré de Whoosh et injecte
la méthode `.stem` d'un `StemmerProvider` comme `stemfn`. Whoosh core l'applique
automatiquement au champ à la fois à l'indexation et à la recherche.

```python
from whoosh.fields import Schema, TEXT
from whoosh_modern.analysis import StemmingAnalyzer, get_stemmer

# Détection automatique du meilleur stemmer (PyStemmer préféré)
stemmer = get_stemmer("auto", "english")

# Créer un analyseur avec la fonction de stem du provider
analyzer = StemmingAnalyzer(stemmer=stemmer)

schema = Schema(
    title=TEXT(stored=True),
    content=TEXT(analyzer=analyzer),
)

# À l'indexation : "running cats" → ["run", "cat"]
# À la recherche : QueryParser utilise le même analyseur
# donc "running cats" correspond aux documents contenant "run cat"
```

**Avantages** : Automatique, pas de configuration de middleware nécessaire, comportement cohérent index/recherche.

**Inconvénients** : Nécessite que l'analyseur soit défini sur chaque champ TEXT. Plus difficile à changer à l'exécution.

### Niveau 2 : Niveau middleware (activé manuellement)

`StemmingMiddleware` applique le stemming au niveau du pipeline, opérant sur les
valeurs de chaîne brutes dans `context.document` et `context.query` avant que
Whoosh's analyzeurs ne les voient.

```python
from whoosh_modern.middleware import StemmingMiddleware
from whoosh_modern.analysis import get_stemmer
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.wrappers import MiddlewareWriter

stemmer = get_stemmer("auto", "english")

chain = MiddlewareChain([
    StemmingMiddleware(
        stemmer=stemmer.stem,
        fields=["title", "content"],  # None = tous les champs str
        stem_query=True,
    ),
])

with MiddlewareWriter(ix.writer(), chain) as writer:
    writer.add_document(title="Running cats", content="Fast dogs")
    # before_index stemme : "Running cats" → "run cat"
    writer.commit()
```

**Avantages** : Fonctionne avec n'importe quel champ sans modifier le schéma. Peut être activé/désactivé à l'exécution.

**Inconvénients** : Doit être connecté manuellement au pipeline. Risque de double-stemming si le champ utilise aussi `StemmingAnalyzer`.

### Exemple de pipeline complet : indexation + recherche

```python
from whoosh import index, fields
from whoosh.qparser import QueryParser
from whoosh_modern.analysis import StemmingAnalyzer, get_stemmer
from whoosh_modern.middleware import StemmingMiddleware
from whoosh.middleware.chain import MiddlewareChain
from whoosh_modern.analysis import StemmingAnalyzer

# 1. Schéma avec analyseur de niveau champ
stemmer = get_stemmer("auto", "english")
schema = fields.Schema(
    title=fields.TEXT(stored=True, analyzer=StemmingAnalyzer(stemmer=stemmer)),
    content=fields.TEXT(analyzer=StemmingAnalyzer(stemmer=stemmer)),
)

ix = index.create_in("indexdir", schema)

# 2. Indexation (pas de double-stemming car
#    on n'utilise pas StemmingMiddleware quand les champs ont StemmingAnalyzer)
with ix.writer() as writer:
    writer.add_document(title="Running cats", content="Fast dogs")
    writer.commit()

# 3. Recherche : QueryParser applique le même analyseur à la requête
with ix.searcher() as searcher:
    qp = QueryParser("content", schema)
    q = qp.parse("running cats")
    results = searcher.search(q)
    # "running" est stemmé en "run" par l'analyseur
    # "cats" est stemmé en "cat" par l'analyseur
    # Correspond au document avec "run" et "cat"
```

### Éviter le double-stemming

```python
# FAUX : double stemming
schema = Schema(
    content=TEXT(analyzer=StemmingAnalyzer(stemmer="auto")),
)
chain = MiddlewareChain([
    StemmingMiddleware(stemmer=get_stemmer("auto").stem),  # Ne pas faire ça !
])
# Résultat : "running" → "run" (analyseur) → "run" (middleware) — inoffensif mais gaspilleux

# CORRECT : choisir UN niveau
# Option A : niveau champ uniquement (recommandé pour schémas statiques)
schema = Schema(content=TEXT(analyzer=StemmingAnalyzer(stemmer="auto")))
# Aucun StemmingMiddleware nécessaire

# Option B : middleware uniquement (pour champs dynamiques)
schema = Schema(content=TEXT)  # Pas d'analyseur
chain = MiddlewareChain([StemmingMiddleware(stemmer=get_stemmer("auto").stem)])
```

### Provider de stemmer personnalisé

```python
from whoosh_modern.analysis import register_stemmer, get_stemmer

@register_stemmer("my_stemmer")
class MyStemmer:
    def stem(self, word: str) -> str:
        return word.lower().rstrip("s")

# Utilisez-le comme n'importe quel backend intégré
stemmer = get_stemmer("my_stemmer", "english")
analyzer = StemmingAnalyzer(stemmer=stemmer)
```

## Voir Aussi

- [Guide Stemming et Mots Vides](../core/stemming.md) — Guide classique de stemming de Whoosh
- [Guide Synonymes & Linguistique](linguistique.md) — Moteur d'expansion de synonymes
- [Intégration des Providers](provider-integration.md) — Guide complet du pipeline pour tous les providers
- [API: Moderne](../api/modern.md) — Référence complète de l'API pour les extensions d'analyse


## DOCUMENT (FR): Auto Indexing

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Auto-Indexing

Whoosh-NG provides utilities for automatic schema discovery and data-source driven indexing.

## Schema Discovery

The `SchemaDiscovery` utility inspects a data source and auto-generates a Whoosh schema:

```python
from whoosh_modern.discovery import SchemaDiscovery

discovery = SchemaDiscovery(source=data_source)
schema = discovery.discover()
```

See [SearchView](/examples/search-view) and [Data Sources](/examples/data-sources) for usage examples.


## DOCUMENT (FR): Autocomplete Providers

# Providers d'Autocomplétion

Module : `whoosh_modern.autocomplete`
Version : 2.0.0

Le module d'autocomplétion fournit plusieurs stratégies de provider pour les
suggestions de requêtes et la recherche en taper-à-mesure (type-ahead). Tous
les providers implémentent une interface commune afin que vous puissiez
intervertir les stratégies à l'exécution. Les providers sont enregistrés via
l'`AutocompleteRegistry` et chargés par points d'entrée.

## Aperçu du module

```text
whoosh_modern.autocomplete
    ├── provider.py   # AutocompleteHit, AutocompleteProvider (Protocole)
    ├── ngram.py      # NGramProvider (basé sur les n-grammes de caractères)
    ├── edge_ngram.py # InvertedIndexAutocomplete (correspondance de préfixe par index inversé)
    ├── fuzzy.py      # FuzzySuggestProvider (correspondance approximative via rapidfuzz)
    ├── factory.py    # factory create_autocomplete()
    └── plugin.py     # AutocompletePlugin (plugin de point d'entrée)
```

## AutocompleteProvider (classe de base)

Localisé dans `whoosh_modern.autocomplete.provider` :

```python
from whoosh_modern.autocomplete.provider import AutocompleteProvider, AutocompleteHit

class MyProvider(AutocompleteProvider):
    def add(self, phrases: Iterable[str]) -> None:
        """Ajoute des phrases à l'index du provider."""
        ...

    def search(self, prefix: str, limit: int = 10) -> list[AutocompleteHit]:
        """Renvoie les suggestions d'autocomplétion pour le préfixe donné."""
        ...
```

### AutocompleteHit

Un objet résultat simple renvoyé par les providers :

```python
class AutocompleteHit:
    def __init__(self, text: str, score: float) -> None:
        self.text = text    # La phrase correspondante
        self.score = score  # Score de pertinence (plus élevé = meilleur)
```

## Providers intégrés

### InvertedIndexAutocomplete

Localisé dans `whoosh_modern.autocomplete.edge_ngram`. Utilise une
correspondance de préfixe simple contre une liste en mémoire :

```python
from whoosh_modern.autocomplete.edge_ngram import InvertedIndexAutocomplete

provider = InvertedIndexAutocomplete()
provider.add(["python", "pyramid", "pytorch", "java", "javascript"])

hits = provider.search("py", limit=5)
for hit in hits:
    print(f"{hit.text} (score: {hit.score})")
# Sortie :
# python (score: 0.45)
# pyramid (score: 0.43)
# pytorch (score: 0.43)
```

**Scoring** : les correspondances de préfixe exactes reçoivent un bonus 1.5x ;
le score de base est `1.0 / (len(phrase) + 1)`.

### NGramProvider

Localisé dans `whoosh_modern.autocomplete.ngram`. Construit un index de
n-grammes de caractères pour la correspondance approximative de sous-chaînes :

```python
from whoosh_modern.autocomplete.ngram import NGramProvider

provider = NGramProvider(n=3)
provider.add(["python programming", "java development", "rust language"])

hits = provider.search("pyt", limit=5)
for hit in hits:
    print(f"{hit.text} (score: {hit.score})")
```

**Paramètres :**

| Paramètre | Type | Défaut | Description                  |
|-----------|------|--------|------------------------------|
| `n`       | `int` | `3`    | Taille des n-grammes de caractères |

**Fonctionnement** : les n-grammes sont extraits de chaque phrase (en
minuscules). Lors de la recherche, les n-grammes du préfixe sont appariés avec
l'index. Les phrases avec le plus de n-grammes correspondants reçoivent les
scores les plus élevés.

### FuzzySuggestProvider

Localisé dans `whoosh_modern.autocomplete.fuzzy`. Utilise `rapidfuzz` pour la
correspondance approximative de chaînes (fautes de frappe, correspondances
partielles) :

```python
from whoosh_modern.autocomplete.fuzzy import FuzzySuggestProvider

# Nécessite : pip install whoosh-ng[fuzzy]
provider = FuzzySuggestProvider(max_distance=2, score_cutoff=50.0)
provider.add(["python", "pyramid", "pytorch", "java", "javascript"])

hits = provider.search("pythn", limit=5)  # Faute de frappe dans "python"
for hit in hits:
    print(f"{hit.text} (score: {hit.score})")
# Sortie : python (score: 0.95), ...
```

**Paramètres :**

| Paramètre       | Type  | Défaut  | Description                              |
|-----------------|-------|---------|------------------------------------------|
| `max_distance`  | `int` | `2`     | Distance d'édition max (non utilisé directement par rapidfuzz, réservé) |
| `score_cutoff`  | `float` | `50.0` | Score de similarité minimum (échelle 0-100) |

**Note** : nécessite `rapidfuzz` (`pip install whoosh-ng[fuzzy]`). Lève
`ImportError` si non installé.

## Fonction factory

Localisée dans `whoosh_modern.autocomplete.factory` :

```python
from whoosh_modern.autocomplete import create_autocomplete

# Crée n'importe quel provider par nom
provider = create_autocomplete("inverted")   # InvertedIndexAutocomplete
provider = create_autocomplete("ngram", n=3) # NGramProvider avec n personnalisé
provider = create_autocomplete("fuzzy", max_distance=2, score_cutoff=60.0)
```

**Providers disponibles :**

| Nom        | Classe                    | Dépendance optionnelle |
|------------|---------------------------|------------------------|
| `"inverted"`| `InvertedIndexAutocomplete` | Aucune              |
| `"ngram"`   | `NGramProvider`          | Aucune                |
| `"fuzzy"`   | `FuzzySuggestProvider`   | `rapidfuzz`           |

## Enregistrement dans l'AutocompleteRegistry

Les providers sont enregistrés dans `whoosh.registry.AutocompleteRegistry`
(une instance `Registry`) :

```python
from whoosh.registry import AutocompleteRegistry
from whoosh_modern.autocomplete import create_autocomplete

# Enregistre un provider
provider = create_autocomplete("ngram", n=3)
AutocompleteRegistry.register("ngram-suggester", provider, owner="my_app")

# Le récupère plus tard
suggester = AutocompleteRegistry.get("ngram-suggester")

# Liste tous les providers enregistrés
print(AutocompleteRegistry.list_keys())
```

## AutocompletePlugin (point d'entrée)

Localisé dans `whoosh_modern.autocomplete.plugin`, c'est le plugin intégré
enregistré via le groupe de points d'entrée `whoosh_ng.plugins` :

```python
from whoosh_modern.autocomplete.plugin import AutocompletePlugin

# Chargé automatiquement par PluginManager.load_plugins()
# Enregistre le provider "inverted" dans AutocompleteRegistry
```

### Déclaration du point d'entrée

Dans `pyproject.toml` :

```toml
[project.entry-points."whoosh_ng.plugins"]
whoosh_autocomplete = "whoosh_modern.autocomplete.plugin:AutocompletePlugin"
```

### Détails du plugin

```python
class AutocompletePlugin(Plugin):
    name = "whoosh_autocomplete"
    version = "3.0.0"

    def register(self, manager):
        # Enregistre InvertedIndexAutocomplete en tant que "inverted"
        AutocompleteRegistry.register(
            "inverted", create_autocomplete("inverted"), self.name
        )

    def register_hooks(self):
        # Enregistre un hook on_search (actuellement sans effet)
        from whoosh.hooks import hookimpl, register_hook
        register_hook("on_search", hookimpl(on_search))
```

## Exemples d'utilisation

### Utilisation de base

```python
from whoosh_modern.autocomplete import create_autocomplete

# Crée et remplit un provider
provider = create_autocomplete("inverted")
provider.add([
    "python programming",
    "python tutorial",
    "java tutorial",
    "javascript framework",
])

# Recherche des suggestions
hits = provider.search("py", limit=3)
for hit in hits:
    print(f"{hit.text}: {hit.score:.3f}")
```

### Utilisation de la correspondance floue avec tolérance aux fautes

```python
from whoosh_modern.autocomplete import create_autocomplete

provider = create_autocomplete("fuzzy", score_cutoff=70.0)
provider.add(["python", "pytorch", "tensorflow", "keras"])

# Même avec une faute de frappe, des suggestions pertinentes sont renvoyées
hits = provider.search("pyton", limit=5)
for hit in hits:
    print(hit.text, hit.score)
```

### Utilisation des n-grammes pour les mots partiels

```python
from whoosh_modern.autocomplete import create_autocomplete

# Utilise des 3-grammes pour une meilleure correspondance de sous-chaînes
provider = create_autocomplete("ngram", n=3)
provider.add(["machine learning", "deep learning", "neural networks"])

# Trouve les phrases contenant les n-grammes de "machin"
hits = provider.search("machin", limit=5)
```

### Intégration avec la recherche

```python
from whoosh_modern.autocomplete import create_autocomplete

# Construit le provider d'autocomplétion
provider = create_autocomplete("inverted")
provider is None  # (exemple conceptuel)
provider = create_autocomplete("inverted")
provider.add(["python", "java", "javascript", "go", "rust"])

# Utilise dans un endpoint de recherche
def suggest(prefix: str, limit: int = 5):
    hits = provider.search(prefix, limit=limit)
    return [{"text": h.text, "score": h.score} for h in hits]

# Dans votre endpoint FastAPI/REST :
# GET /api/suggest?q=py&limit=5
# Réponse : [{"text": "python", "score": 0.45}, ...]
```

## Comparaison des providers

| Provider              | Correspondance  | Forces                    | Faiblesses               | Dépendance    |
|-----------------------|----------------|---------------------------|--------------------------|---------------|
| `inverted`            | Préfixe         | Simple, rapide, sans deps | Pas de tolérance aux fautes | Aucune     |
| `ngram`               | Recouvrement n-grammes | Correspondance de sous-chaînes, flexible | Plus lent que le préfixe | Aucune |
| `fuzzy`               | Distance d'édition | Tolérance aux fautes, flexible | Nécessite rapidfuzz | `rapidfuzz` |

## Installation

```bash
# Autocomplétion de base (inverted + n-gram)
pip install whoosh-ng

# Avec correspondance floue
pip install whoosh-ng[fuzzy]

# Analyse moderne complète
pip install whoosh-ng[modern]
```

## Voir aussi

- [Guide du Système de Plugins](plugins-advanced.md) — Enregistrement et découverte de plugins
- [Guide du Middleware](middleware-pipeline.md) — Intégration du pipeline de middleware
- [Guide d'Intégration des Providers](provider-integration.md) — Guide complet du pipeline pour tous les providers
- [API : Linguistique](../api/modern.md) — Référence complète de l'API pour les extensions d'autocomplétion


## DOCUMENT (FR): Autocomplete

# Autocomplétion

Couche optionnelle d'autocomplétion par edge-ngram pour Whoosh-NG.

## Installer

```bash
pip install whoosh-ng[autocomplete]
```

## Index minimal

```python
from whoosh.fields import Schema, TEXT, AutocompleteField

schema = Schema(
    titre=TEXT(stored=True),
    query=AutocompleteField()
)

with ix.writer() as writer:
    writer.add_document(titre="Démarrage Python", query="demarrage python")
    writer.commit()
```

## Requête d'autocomplétion

```python
from whoosh_modern.autocomplete import AutocompleteProvider

provider = AutocompleteProvider(ix, "query")
suggestions = provider.suggest("de", limit=5)
print(suggestions)  # ["demarrage python", ...]
```


## DOCUMENT (FR): Configuration Engine

# Moteur de Configuration

Whoosh-NG intègre un **Moteur de Configuration** (`ConfigEngine`) qui charge,
valide et fusionne la configuration applicative depuis des fichiers YAML ou
JSON. Il repose sur des modèles Pydantic et prend en charge une organisation
hiérarchique par couches, permettant à des surcharges spécifiques à
l'environnement d'étendre proprement les réglages de base.

## Concepts de base

### Modèles Pydantic

Toute la configuration est exprimée via des modèles Pydantic typés :

- `WhooshNGConfig` — configuration applicative de plus haut niveau
- `FieldConfig` — options d'indexation par champ
- `SearchConfig` / `FuzzyConfig` / `RankingConfig` / `AIConfig`
- `DataSourceConfigModel` — connexion et synchronisation de la source de données
- `StorageConfigModel` — sélection du backend de stockage

### Chargeurs (loaders)

Deux chargeurs sont fournis :

- `load_yaml(path)` — analyse un fichier YAML en `dict`
- `load_json(path)` — analyse un fichier JSON en `dict`
- `load_config(path)` — détecte automatiquement le format depuis l'extension et
  renvoie un `WhooshNGConfig` validé

### Fusion hiérarchique

`ConfigEngine.load(path, priority=...)` et `ConfigEngine.merge(overrides, priority=...)`
empilent les sources de configuration selon l'ordre de priorité suivant (le plus
haut l'emporte) :

1. `runtime`
2. `instance`
3. `application`
4. `language`

Une valeur de ``priority`` invalide lève immédiatement ``ValueError``, afin
qu'une couche mal configurée ne puisse pas affecter silencieusement l'ordre de
fusion.

La fusion est profonde : les dictionnaires imbriqués sont fusionnés
récursivement. Les valeurs scalaires et les listes sont **remplacées
entièrement** par les valeurs de surcharge ; les listes ne sont PAS ajoutées ni
combinées. Par exemple, une configuration de base ``{"plugins": ["a", "b"]}``
surchargée par ``{"plugins": ["c"]}`` produit ``{"plugins": ["c"]}``, et non
``{"plugins": ["a", "b", "c"]}``. Si une fusion additive de listes est requise,
traitez-la au niveau applicatif avant d'appeler :meth:`ConfigEngine.merge`.

> [!WARNING]
> **Comportement de remplacement des listes** : lors de la fusion de
> configurations, les listes sont **completement écrasées** par les couches de
> priorité supérieure. Elles ne sont ni ajoutées, ni concaténées, ni
> dédupliquées. Il s'agit d'un choix de conception volontaire qui garantit un
> contrôle explicite du contenu des listes entre les couches et évite des états
> fusionnés imprévisibles. Si vous avez besoin d'un comportement additif (par
> exemple étendre une liste de plugins ou de middlewares), effectuez la logique
> de fusion dans votre code applicatif avant de passer le dictionnaire final à
> :meth:`ConfigEngine.merge`.

## Démarrage rapide

```python
from whoosh_modern.config import ConfigEngine

engine = ConfigEngine()
engine.load("whoosh-ng.yml", priority="application")
engine.load("whoosh-ng.local.yml", priority="instance")
engine.merge({"search": {"fuzzy": {"distance": 5}}}, priority="runtime")

config = engine.get_config()
print(config.index)
print(config.fields["title"].stemming)
print(config.search.fuzzy.distance)
```

## Exemple YAML

```yaml
# whoosh-ng.yml
index: products
languages:
  default: fr
fields:
  title:
    type: text
    language: fr
    stemming: true
    stored: true
  price:
    type: numeric
    sortable: true
search:
  fuzzy:
    enabled: true
    distance: 2
storage:
  type: file
  path: ./index
```

## Exemple JSON

```json
{
  "index": "products",
  "languages": {"default": "en"},
  "fields": {
    "title": {"type": "text", "language": "en", "stemming": true},
    "price": {"type": "numeric", "sortable": true}
  },
  "search": {"fuzzy": {"enabled": true, "distance": 2}},
  "storage": {"type": "file", "path": "./index"}
}
```

## Exemples YAML complets

### Configuration minimale

```yaml
# whoosh-ng.yml
index: my_index
fields:
  title:
    type: text
    stored: true
storage:
  type: file
  path: ./index
```

### Catalogue e-commerce avec source CSV

```yaml
# whoosh-ng.yml
index: products
fields:
  sku:
    type: text
    stored: true
    unique: true
  name:
    type: text
    language: fr
    stemming: true
    stored: true
  description:
    type: text
    language: fr
    stemming: true
  price:
    type: numeric
    sortable: true
    faceted: true
  category:
    type: text
    faceted: true
  published_at:
    type: datetime
    faceted: true
search:
  fuzzy:
    enabled: true
    distance: 2
data_source:
  type: csv
  path: Datas/products.csv
  delimiter: ","
  encoding: utf-8
  id_field: sku
storage:
  type: file
  path: ./index
```

### Configuration par couches (base + instance + runtime)

```yaml
# whoosh-ng.yml  (couche application)
index: app
fields:
  title:
    type: text
    stemming: true
search:
  fuzzy:
    enabled: true
    distance: 2
storage:
  type: file
  path: ./index
```

```yaml
# whoosh-ng.local.yml  (couche instance)
index: app-staging
storage:
  type: file
  path: ./index-staging
```

```python
# surcharge runtime dans le code
engine = ConfigEngine()
engine.load("whoosh-ng.yml", priority="application")
engine.load("whoosh-ng.local.yml", priority="instance")
engine.merge({"search": {"fuzzy": {"distance": 3}}}, priority="runtime")
app = engine.build()
```

## Exemples JSON complets

### Configuration minimale

```json
{
  "index": "my_index",
  "fields": {
    "title": {"type": "text", "stored": true}
  },
  "storage": {"type": "file", "path": "./index"}
}
```

### Configuration full-stack avec source SQL et stockage hybride

```json
{
  "index": "customers",
  "fields": {
    "customer_id": {"type": "numeric", "stored": true, "sortable": true},
    "first_name": {"type": "text", "language": "en", "stemming": true, "stored": true},
    "last_name": {"type": "text", "language": "en", "stemming": true, "stored": true},
    "city": {"type": "text", "language": "en", "stemming": true, "stored": true},
    "country": {"type": "text", "stored": true},
    "signup_date": {"type": "datetime", "faceted": true}
  },
  "search": {
    "fuzzy": {"enabled": true, "distance": 2},
    "highlight": {"enabled": true, "fragment_size": 200}
  },
  "data_source": {
    "type": "sql",
    "connection_string": "sqlite:///benchmark_data.db",
    "query": "SELECT * FROM customers",
    "id_field": "customer_id"
  },
  "storage": {
    "type": "hybrid",
    "local_path": "./index-cache",
    "remote": {
      "type": "s3",
      "bucket": "my-bucket",
      "prefix": "whoosh-indexes/"
    }
  }
}
```

## Configuration sans code avec ConfigEngine.build()

```python
from whoosh_modern.config import ConfigEngine

engine = ConfigEngine()
engine.load("whoosh-ng.yml")
app = engine.build()
app.build()

# Ajouter des documents via l'index writer
writer = app.index.writer()
writer.add_document(title="Premier cours de Python", body="...")
writer.add_document(title="Whoosh-NG avancé", body="...")
writer.commit()

# Ou utiliser la source directement pour un indexage en flux/lot
for doc in app._source.iter_documents():
    with app.index.writer() as writer:
        writer.add_document(**doc)

results = app.search("python")
```

## Référence des modules

| Module | Rôle |
|---|---|
| `whoosh_modern.config.models` | Modèles Pydantic de validation |
| `whoosh_modern.config.loader` | Chargeurs de fichiers YAML / JSON |
| `whoosh_modern.config.engine` | `ConfigEngine` avec fusion hiérarchique |

## Voir aussi

- [Providers de Stockage](storage-providers.md) — Backends configurables via `StorageConfigModel`
- [Sources de Données](data-sources.md) — `DataSourceConfigModel` et configuration des providers


## DOCUMENT (FR): Linguistics

# Synonymes & Linguistique

Module : `whoosh_modern.linguistics.synonyms`, `whoosh_modern.linguistics.stemmers`
Version : 2.0.0

Le module de linguistique fournit un moteur complet d'expansion de synonymes
et des analyseurs de texte spécifiques à une langue. Il s'intègre au pipeline de
middleware pour étendre les requêtes et les documents avec des synonymes aussi
bien à l'indexation qu'à la recherche.

## Aperçu du module

```text
whoosh_modern.linguistics
    ├── synonyms/
    │   ├── provider.py       # Protocole SynonymProvider + StaticSynonymProvider
    │   ├── yaml_provider.py  # YAMLSynonymProvider
    │   ├── json_provider.py  # JSONSynonymProvider
    │   ├── store.py          # SQLiteSynonymStore
    │   ├── compiler.py       # SynonymCompiler
    │   ├── manager.py        # SynonymManager
    │   ├── middleware.py      # SynonymExpansionMiddleware
    │   └── languages.py      # LANG_SYNONYMS (FR/EN/DE/ES/IT)
    └── stemmers/
        └── __init__.py       # Analyseurs spécifiques à une langue (FR/EN/DE/ES/IT)
```

## Providers de synonymes

### SynonymProvider (Protocole)

Le protocole de base implémenté par tous les providers de synonymes :

```python
from whoosh_modern.linguistics.synonyms import SynonymProvider

class MyProvider(SynonymProvider):
    def get_synonyms(self, word: str) -> list[str]:
        """Renvoie les synonymes du mot donné."""
        ...

    def add_synonym(self, word: str, synonyms: list[str]) -> None:
        """Ajoute des synonymes au mot donné."""
        ...

    def remove_synonym(self, word: str, synonym: str) -> None:
        """Retire un synonyme du mot donné."""
        ...
```

### StaticSynonymProvider

Provider en mémoire appuyé sur un dictionnaire :

```python
from whoosh_modern.linguistics.synonyms import StaticSynonymProvider

provider = StaticSynonymProvider({
    "car": ["automobile", "vehicle", "auto"],
    "house": ["home", "residence"],
})

print(provider.get_synonyms("car"))  # ['automobile', 'vehicle', 'auto']
```

### YAMLSynonymProvider

Charge les synonymes depuis un fichier YAML :

```yaml
# synonyms.yaml
car:
  - automobile
  - vehicle
  - auto
house:
  - home
  - residence
```

```python
from whoosh_modern.linguistics.synonyms import YAMLSynonymProvider

# Nécessite : pip install pyyaml
provider = YAMLSynonymProvider("synonyms.yaml")
print(provider.get_synonyms("car"))  # ['automobile', 'vehicle', 'auto']
```

### JSONSynonymProvider

Charge les synonymes depuis un fichier JSON :

```json
{
    "car": ["automobile", "vehicle", "auto"],
    "house": ["home", "residence"]
}
```

```python
from whoosh_modern.linguistics.synonyms import JSONSynonymProvider

provider = JSONSynonymProvider("synonyms.json")
print(provider.get_synonyms("car"))
```

### SQLiteSynonymStore

Stockage persistant de synonymes appuyé sur SQLite :

```python
from whoosh_modern.linguistics.synonyms import SQLiteSynonymStore

store = SQLiteSynonymStore("synonyms.db")

# Opérations CRUD
store.add_synonym("car", ["automobile", "vehicle"])
print(store.get_synonyms("car"))  # ['automobile', 'vehicle']
store.remove_synonym("car", "automobile")
print(store.get_synonyms("car"))  # ['vehicle']
store.close()
```

### SynonymCompiler

Précompile les données de synonymes brutes dans un format de recherche rapide :

```python
from whoosh_modern.linguistics.synonyms import SynonymCompiler

compiler = SynonymCompiler({"car": ["automobile", "vehicle"]})
compiler.add("house", ["home", "residence"])
compiler.merge({"book": ["publication", "work"]})

compiled = compiler.compile()
print(compiled)
# {'car': ['automobile', 'vehicle'], 'house': ['home', 'residence'], 'book': ['publication', 'work']}
```

## SynonymManager

Le `SynonymManager` est l'interface de haut niveau pour gérer les synonymes. Il
encapsule en interne un `StaticSynonymProvider` et prend en charge
l'import/export :

```python
from whoosh_modern.linguistics.synonyms import SynonymManager

manager = SynonymManager({"car": ["automobile", "vehicle"]})

# CRUD
manager.add_synonyms("house", ["home", "residence"])
print(manager.get_synonyms("house"))  # ['home', 'residence']
manager.remove_synonym("house", "home")

# Import depuis des sources externes
manager.import_yaml("synonyms.yaml")   # Nécessite PyYAML
manager.import_json("synonyms.json")

# Export
manager.export_json("output.json")
```

### Workflow d'import/export

```python
# Import depuis YAML
manager = SynonymManager()
manager.import_yaml("my_synonyms.yaml")

# Export vers JSON (ex : pour migration ou sauvegarde)
manager.export_json("backup.json")
```

## Dictionnaires de synonymes préconstruits

Le dictionnaire `LANG_SYNONYMS` contient des correspondances de synonymes de
démarrage pour cinq langues :

```python
from whoosh_modern.linguistics.synonyms import LANG_SYNONYMS

# Langues disponibles : fr, en, de, es, it
french_syns = LANG_SYNONYMS["fr"]
print(french_syns["voiture"])  # ['automobile', 'véhicule']

english_syns = LANG_SYNONYMS["en"]
print(english_syns["car"])  # ['automobile', 'vehicle']

# Amorce un SynonymManager avec une langue
manager = SynonymManager(LANG_SYNONYMS["fr"])
```

| Langue  | Code | Entrée d'exemple                          |
|---------|------|-------------------------------------------|
| Français | `fr` | `"voiture": ["automobile", "véhicule"]`    |
| Anglais | `en` | `"car": ["automobile", "vehicle"]`        |
| Allemand | `de` | `"auto": ["wagen", "fahrzeug"]`           |
| Espagnol | `es` | `"coche": ["automóvil", "vehículo"]`      |
| Italien | `it` | `"auto": ["automobile", "veicolo"]`       |

> **Note** : il s'agit de dictionnaires de démarrage minimaux destinés à la
> démonstration. Les déploiements en production devraient charger des sources
> organisées ou spécifiques à un domaine.

## SynonymExpansionMiddleware

Intègre l'expansion de synonymes dans le pipeline de middleware. Il étend à la
fois les requêtes de recherche et les champs de documents indexés :

```python
from whoosh_modern.linguistics.synonyms import (
    SynonymManager,
    SynonymExpansionMiddleware,
)

# Crée un manager avec vos synonymes
manager = SynonymManager({
    "car": ["automobile", "vehicle"],
    "house": ["home", "residence"],
})

# Crée le middleware
middleware = SynonymExpansionMiddleware(manager)

# Enregistre auprès du PluginManager ou de la MiddlewareChain
from whoosh.plugins.manager import PluginManager
PluginManager._default.register_middleware("synonym", middleware)
```

### Fonctionnement

- **`before_search`** : étend `context.query` en ajoutant les synonymes de
  chaque token
- **`before_index`** : étend les valeurs chaîne dans `context.document` en
  ajoutant les synonymes

```python
# Avant : query = "car"
# Après  : query = "car automobile vehicle"

# Avant : document = {"title": "house for sale"}
# Après : document = {"title": "house for sale home residence"}
```

## Analyseurs de stemming spécifiques à une langue

Localisés dans `whoosh_modern.linguistics.stemmers`, ces analyseurs combinent
tokenization, stemming et suppression des mots vides :

```python
from whoosh_modern.linguistics.stemmers import (
    EnglishAnalyzer,
    FrenchAnalyzer,
    GermanAnalyzer,
    SpanishAnalyzer,
    ItalianAnalyzer,
)

# Chaque analyseur est une instance de LanguageAnalyzer et est appelable :
# il renvoie une liste de tokens
analyzer = EnglishAnalyzer
tokens = analyzer("The running cats")
# tokens sont stemmés : ["run", "cat"] (mots vides supprimés)

# L'usage "style classe" rétro-compatible fonctionne aussi : appeler l'analyseur
# sans argument renvoie une nouvelle instance, donc le code historique
# écrit comme EnglishAnalyzer()(text) continue de fonctionner inchangé.
tokens = EnglishAnalyzer()("The running cats")
```

### Sélection du backend de stemming

Sous le capot, les stemmers utilisent `whoosh_modern.analysis.stemmer_providers`
:

```python
from whoosh_modern.analysis.stemmer_providers import (
    get_stemmer,
    register_stemmer,
    list_available_backends,
)

# Auto-détecte le meilleur stemmer disponible (PyStemmer privilégié)
stemmer = get_stemmer("auto", "english")

# Backend explicite
stemmer = get_stemmer("internal", "english")   # Stemmer intégré de Whoosh
stemmer = get_stemmer("pystemmer", "english")   # PyStemmer (plus rapide)

# Liste les backends disponibles
print(list_available_backends())
# {'internal': 'available', 'pystemmer': 'available', ...}

# Enregistre un stemmer personnalisé
@register_stemmer("my_stemmer")
class MyStemmer:
    def stem(self, word: str) -> str:
        return word.lower()
```

| Backend     | Nécessite                       | Vitesse                |
|-------------|---------------------------------|------------------------|
| `auto`      | Aucun (repli automatique)       | Le plus rapide dispo.  |
| `internal`  | Aucun (Porter stemmer intégré)   | Moyenne                |
| `pystemmer` | `pip install whoosh-ng[fast-stemming]` | Rapide         |

## Exemple d'intégration : pipeline complet

```python
from whoosh_modern.linguistics import (
    EnglishAnalyzer,
    LANG_SYNONYMS,
    SynonymExpansionMiddleware,
    SynonymManager,
)
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.wrappers import MiddlewareWriter, MiddlewareSearcher

# 1. Construit le manager de synonymes avec les synonymes anglais
syn_manager = SynonymManager(LANG_SYNONYMS["en"])
syn_manager.add_synonyms("search", ["query", "find", "lookup"])

# 2. Crée le middleware d'expansion de synonymes
syn_middleware = SynonymExpansionMiddleware(syn_manager)

# 3. Construit la chaîne de middleware
chain = MiddlewareChain([syn_middleware])

# 4. Enveloppe le writer et le searcher
with MiddlewareWriter(ix.writer(), chain) as writer:
    writer.add_document(title="How to search in Whoosh")

with MiddlewareSearcher(ix.searcher(), chain) as searcher:
    # La requête "search" est étendue en "search query find lookup"
    results = searcher.search("search")
```

## Registre de langues

`LanguageRegistry` mappe les codes de langue vers des instances `LanguageProfile`, centralisant la résolution d'analyzer, de stemmer, de provider de synonymes et de détecteur de langue.

```python
from whoosh_modern.linguistics.registry import (
    LanguageRegistry,
    LanguageProfile,
    StemmerRegistry,
    get_default_registry,
)

# Utilise le registry pré-peuplé par défaut (FR/EN/DE/ES/IT)
registry = get_default_registry()

# Résout un profil de langue
profile = registry.resolve("fr")
print(profile.language)   # "fr"
print(profile.analyzer)   # instance de FrenchAnalyzer

# Enregistre un profil de langue personnalisé
custom = LanguageProfile(
    language="pt",
    analyzer=...,  # votre analyzer
    stemmer=...,   # votre stemmer
)
registry.register(custom)

# StemmerRegistry ajoute des helpers spécifiques aux stemmers
stem_registry = StemmerRegistry(registry._profiles.values())
stemmer = stem_registry.get_stemmer("fr")
```

## Analyseur multilingue

`MultiLanguageAnalyzer` applique plusieurs analyseurs de langue simultanément pour l'indexation multilingue.

```python
from whoosh_modern.linguistics.analyzers import MultiLanguageAnalyzer

# Par défaut : FR/EN/DE/ES/IT
analyzer = MultiLanguageAnalyzer()

# Ensemble de langues personnalisé
analyzer = MultiLanguageAnalyzer(languages=["fr", "en"])

tokens = analyzer("hello bonjour")
# Retourne les tokens combinés de tous les analyseurs configurés
```

## Auto-détection de langue

`StopwordDetector` et `LangDetectProvider` permettent la détection automatique de langue :

```python
from whoosh_modern.linguistics.detection import StopwordDetector

detector = StopwordDetector(supported_languages=["fr", "en", "de"])
lang = detector.detect("Ceci est un texte en français")
print(lang)  # "fr"
```

Utilisation avec `SearchApplication` pour la résolution automatique de langue :

```python
from whoosh_modern import SearchApplication
from whoosh_modern.linguistics.detection import StopwordDetector

app = SearchApplication(
    source=my_source,
    language_detector=StopwordDetector(),
)

# FieldConfig supporte language="auto"
# Le détecteur résout la langue par document
```

## Analyseur Explain

`ExplainAnalyzer` expose le pipeline de tokenization/stemming pour Search Studio :

```python
from whoosh_modern.linguistics.explain import ExplainAnalyzer

explainer = ExplainAnalyzer(EnglishAnalyzer)
result = explainer.explain("The running cats")

print(result.text)       # "The running cats"
print(result.tokens)     # ["run", "cat"]
```

## Débogage d'analyse avec ExplainAnalyzer

`ExplainAnalyzer` encapsule n'importe quel analyseur existant et retourne une
`AnalysisExplanation` décrivant comment un texte est transformé. C'est un outil
utile pour déboguer des chaînes d'analyseurs complexes, surtout lors de l'utilisation
d'analyseurs multilingues, de filtres stopwords ou de remplacements de stemming
par dictionnaire.

```python
from whoosh_modern.linguistics.explain import ExplainAnalyzer
from whoosh.analysis import StandardAnalyzer

explainer = ExplainAnalyzer(StandardAnalyzer())
explanation = explainer.explain(
    "A quick brown fox jumps over the lazy dog"
)

print(f"Texte original : {explanation.text}")
print(f"Tokens finaux : {explanation.tokens}")

print("\nExplications étape par étape :")
for step in explanation.explanations:
    print(
        f"  - {step.step} : '{step.original}' -> '{step.result}'"
    )
```

Exemple de sortie :

```text
Texte original : A quick brown fox jumps over the lazy dog
Tokens finaux : ['quick', 'brown', 'fox', 'jumps', 'lazy', 'dog']

Explications étape par étape :
  - tokenize : 'A' -> 'A'
  - lowercase : 'A' -> 'a'
  - stop : 'a' -> ''
  - tokenize : 'quick' -> 'quick'
  - lowercase : 'quick' -> 'quick'
  ...
```

### Interprétation de la sortie

- `explanation.text` — le texte d'entrée original.
- `explanation.tokens` — la liste finale de tokens après toutes les étapes de l'analyseur.
- `explanation.explanations` — une liste chronologique d'objets
  `TokenExplanation` montrant chaque transformation.

Utilisez ceci lorsque :
- une chaîne d'analyseurs se comporte différemment de ce qui est attendu,
- vous avez besoin de vérifier quels stopwords ou règles de stemming sont appliqués,
- vous voulez comparer le comportement entre langues avec `MultiLanguageAnalyzer`.

## Override de stem par dictionnaire

Remplace le stemming Snowball par des dictionnaires métier :

```python
from whoosh_modern.linguistics.dictionary_stem_override import DictionaryStemOverride

override = DictionaryStemOverride({
    "voiture": "voitur",
    "maison": "maison",
})

print(override.stem("voiture"))  # "voitur"
print(override.stem("maison"))   # "maison"

# Ajoute des règles dynamiquement
override.add_rule("chien", "chien")
```

Utilisation avec `SearchApplication` :

```python
from whoosh_modern import SearchApplication

app = SearchApplication(
    source=my_source,
    dictionary_stem_overrides={"voiture": "voitur"},
)
```

## Analyseur de stemming avec cache

`CachedStemmingAnalyzer` encapsule les analyseurs de langue avec un cache LRU :

```python
from whoosh_modern.analysis.cached_stemming_analyzer import CachedStemmingAnalyzer
from whoosh_modern.linguistics.stemmers import FrenchAnalyzer

cached = CachedStemmingAnalyzer(FrenchAnalyzer, cache_size=50000)
tokens = cached("les maisons")
```

## Profileur de stemmer

Mesure l'impact du stemming sur le vocabulaire et les performances :

```python
from whoosh_modern.profiling.stemmer_profiler import StemmerProfiler

profiler = StemmerProfiler(stemmer=my_stemmer)
report = profiler.profile(["document 1", "document 2", ...])

print(report.original_tokens)        # Tokens totaux avant stemming
print(report.stemmed_tokens)         # Tokens uniques après stemming
print(report.reduction_ratio)        # Ratio de réduction du vocabulaire
print(report.estimated_size_reduction)  # Réduction estimée de la taille d'index %
print(report.avg_stem_time_ms)       # Temps moyen de stemming par token
```

## Préréglages d'analyseurs

Analyseurs préconfigurés pour des scénarios de recherche courants :

```python
from whoosh_modern.analysis.stemmer_presets import AnalyzerPresets

# Autocomplétion
autocomplete_analyzer = AnalyzerPresets.autocomplete()

# Correspondance partielle
partial_analyzer = AnalyzerPresets.partial_match()

# E-commerce
ecommerce_analyzer = AnalyzerPresets.ecommerce()

# Blog
blog_analyzer = AnalyzerPresets.blog()

# Multilingue
multilingual_analyzer = AnalyzerPresets.multilingual()

# Accès par nom
analyzer = AnalyzerPresets.get("autocomplete")
```

## Voir aussi

- [Synonymes](synonyms.md) — Providers de synonymes, manager et dictionnaires Wiktionary
- [Guide des Stemmers](stemming-providers.md) — Providers de stemming et analyseurs de langue
- [Guide du Middleware](middleware-pipeline.md) — Intégration du pipeline de middleware
- [Guide d'Intégration des Providers](provider-integration.md) — Guide complet du pipeline pour tous les providers
- [API : Linguistique](../api/modern.md) — Référence complète de l'API


## DOCUMENT (FR): Middleware Pipeline

# Middleware & Pipeline de Plugins

Module: `whoosh.middleware`, `whoosh.middleware.chain`, `whoosh.middleware.context`, `whoosh_modern.middleware`
Version: 2.0.0

Le pipeline de middleware permet d'intercepter et de modifier les opérations d'indexation et de recherche. C'est le mécanisme d'extension principal pour les préoccupations transverses comme la journalisation, la mise en cache, les métriques, la réécriture de requêtes et la sécurité. Le middleware peut provenir à la fois du package core `whoosh.middleware` et des plugins chargés via le `PluginManager`.

## Vue d'ensemble de l'architecture

```text
Writer/Searcher  ───►  MiddlewareChain
                           ├── Middleware 1 (hook before)
                           ├── Middleware 2 (hook before)
                           ├── ─── opération core ───
                           ├── Middleware 2 (hook after, inverse)
                           └── Middleware 1 (hook after, inverse)
```

- Les **hooks `before_*`** s'exécutent dans l'ordre d'enregistrement
- Les **hooks `after_*`** s'exécutent dans l'ordre inverse (comme une pile / oignon)
- Si un hook lève `StopOperation`, le pipeline s'arrête gracieusement
- Si `fail_open=False` (défaut), les exceptions se propagent immédiatement

## Classes de Base du Middleware

### Middleware (Classe de Base)

Localisée dans `whoosh.middleware.base`. Les sous-classes implémentent les hooks du cycle de vie :

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class MyMiddleware(Middleware):
    def startup(self, context: MiddlewareContext) -> None:
        """Appelé une fois quand le middleware est initialisé."""
        pass

    def shutdown(self, context: MiddlewareContext) -> None:
        """Appelé une fois quand le middleware est détruit."""
        pass

    def before_index(self, context: MiddlewareContext) -> MiddlewareContext:
        """Appelé avant qu'un document soit indexé. Modifier context.document."""
        return context

    def after_index(self, context: MiddlewareContext) -> MiddlewareContext:
        """Appelé après qu'un document a été indexé."""
        return context

    def before_delete(self, context: MiddlewareContext) -> MiddlewareContext:
        """Appelé avant la suppression d'un document."""
        return context

    def after_delete(self, context: MiddlewareContext) -> MiddlewareContext:
        """Appelé après la suppression d'un document."""
        return context

    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        """Appelé avant l'exécution d'une requête. Modifier context.query."""
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        """Appelé après le retour des résultats. Accéder à context.results."""
        return context

    def on_error(self, context: MiddlewareContext, exc: Exception) -> None:
        """Appelé quand une exception survient. Re-raise par défaut."""
        raise exc

    def on_commit(self, context: MiddlewareContext) -> None:
        """Appelé après une opération de commit."""
        pass
```

### MiddlewareContext

Localisé dans `whoosh.middleware.context`. L'objet contexte passé à chaque hook :

```python
class MiddlewareContext:
    def __init__(self, operation: str) -> None:
        self.operation: str           # ex: "add_document", "search"
        self.index: Any = None        # L'instance Index
        self.backend: Any = None       # Le backend de stockage
        self.writer: Any = None        # L'IndexWriter (si applicable)
        self.searcher: Any = None      # Le Searcher (si applicable)
        self.document: dict[str, Any] | None  # Document à indexer
        self.query: str = ""           # La chaîne de requête de recherche
        self.collector: Any = None     # Le collecteur (si applicable)
        self.results: Any = None       # Résultats de recherche
        self.labels: dict[str, Any] = {}    # Paires clé-valeur arbitraires
        self.metadata: dict[str, Any] = {} # Métadonnées par requête
```

Utilisez `context.copy()` pour créer une copie superficielle si vous devez préserver l'état.

### MiddlewareChain

Localisé dans `whoosh.middleware.chain`. Ordonnance l'exécution des middleware :

```python
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.context import MiddlewareContext

chain = MiddlewareChain([
    MetricsMiddleware(),
    CacheMiddleware(),
])

# Hooks before (dans l'ordre)
context = MiddlewareContext("search")
context.query = "hello world"
context = chain.run_before("before_search", context)

# ... opération de recherche core ...

# Hooks after (dans l'ordre inverse)
context = chain.run_after("after_search", context)
print(context.results)
```

**Support asynchrone** : Utilisez `async_run_before()`, `async_run_after()`, `async_run_on_error()` et `run_hook()` pour un middleware asynchrone.

### MiddlewareRegistry

Localisé dans `whoosh.middleware.registry`. Un registre au niveau de la classe pour les middleware nommés :

```python
from whoosh.middleware.registry import MiddlewareRegistry

MiddlewareRegistry.register("my_mw", MyMiddleware(), owner="my_plugin")
mw = MiddlewareRegistry.get("my_mw")
MiddlewareRegistry.unregister("my_mw")
print(MiddlewareRegistry.list_all())  # ['my_mw', ...]
```

## Intégration du Middleware

### Wrappers: MiddlewareWriter & MiddlewareSearcher

Localisés dans `whoosh.middleware.wrappers`. Ces wrappers enveloppent le writer/searcher core pour exécuter automatiquement les hooks de middleware :

```python
from whoosh.middleware.wrappers import MiddlewareWriter, MiddlewareSearcher
from whoosh.middleware.chain import MiddlewareChain

chain = MiddlewareChain([MetricsMiddleware(), CacheMiddleware()])

# Envelopper un writer
with MiddlewareWriter(ix.writer(), chain) as writer:
    writer.add_document(title="Hello", content="World")

# Envelopper un searcher
with MiddlewareSearcher(ix.searcher(), chain) as searcher:
    results = searcher.search(query)
```

### Assistants d'Intégration

Localisés dans `whoosh.middleware.integration` :

```python
from whoosh.middleware.integration import apply_middleware_to_writer, apply_middleware_to_searcher

# Charge automatiquement le middleware depuis PluginManager si chain non fournie
writer = apply_middleware_to_writer(ix.writer())
searcher = apply_middleware_to_searcher(ix.searcher())
```

## Middleware Intégrés

### Middleware Core (`whoosh.middleware.base`)

| Classe                  | Hooks              | Description                              |
|------------------------|--------------------|------------------------------------------|
| `CompressionMiddleware` | `before_index`    | Marque les documents avec `_compressed = True` |
| `EncryptionMiddleware`  | `before_index`    | Marque les documents avec `_encrypted = True`  |
| `MetricsMiddleware`     | `after_index`, `after_search` | Compte les documents indexés et les recherches |
| `CacheMiddleware`       | `before_search`, `after_search` | Mise en cache en mémoire des résultats |

### Observabilité (`whoosh.middleware.metrics`)

`PrometheusMiddleware` — exporte des métriques vers Prometheus (nécessite `prometheus-client`) :

```python
from whoosh.middleware.metrics import PrometheusMiddleware

# Nécessite: pip install whoosh-ng[metrics]
prom = PrometheusMiddleware()
# Exporte: whoosh_searches_total, whoosh_documents_indexed_total, whoosh_search_duration_seconds
```

### Middleware Moderne (`whoosh_modern.middleware`)

#### Middleware de Résilience (sous-classes du core)

`RetryMiddleware`, `LoggingMiddleware` et `CacheMiddleware` sont désormais des **sous-classes
du core `whoosh.middleware.base.Middleware`** (la même classe de base ré-exportée sous
`whoosh_modern.middleware.Middleware`). Ils participent au pipeline de hooks standard
(`before_index` / `after_index` / `before_search` / `after_search` / `on_error` /
`on_commit`) et conservent en plus un helper `wrap(operation)` permettant de décorer de
simples callables.

`MiddlewarePipeline` est un fin wrapper autour de `whoosh.middleware.chain.MiddlewareChain`
qui exécute un callable à travers les hooks de la chaîne et renvoie son résultat. L'ancien
module `whoosh_modern.middleware.pipeline` a été supprimé — importez ces noms directement
depuis `whoosh_modern.middleware`.

| Classe                  | Description                              |
|------------------------|------------------------------------------|
| `RetryMiddleware`      | Réessaie les opérations échouées avec backoff exponentiel |
| `LoggingMiddleware`    | Journalise le temps d'exécution et les erreurs |
| `CacheMiddleware`      | Met en cache les résultats d'opérations (éviction LRU) |
| `MiddlewarePipeline`   | Enchaîne plusieurs middleware via `MiddlewareChain` |

```python
from whoosh_modern.middleware import MiddlewarePipeline, RetryMiddleware, LoggingMiddleware

pipeline = MiddlewarePipeline(
    LoggingMiddleware(),
    RetryMiddleware(attempts=3, backoff="exponential", jitter=True),
)

result = pipeline.execute(lambda: my_index_operation())
```

#### Middleware de Stockage (`whoosh_modern.middleware.storage`)

| Classe                  | Description                              |
|------------------------|------------------------------------------|
| `StorageMiddleware`    | Redirige la persistance vers des fournisseurs de stockage pluginables |
| `FileStorageProvider`  | Stockage sur système de fichiers local   |
| `SQLiteStorageProvider`| Stockage blob SQLite                    |
| `S3StorageProvider`    | Stockage cloud S3 / S3-compatible       |

```python
from whoosh_modern.middleware.storage import StorageMiddleware, FileStorageProvider

storage = StorageMiddleware(FileStorageProvider("/data/index"), name="primary")
```

#### Middleware de Recherche (`whoosh_modern.middleware.search`)

| Classe                      | Description                              |
|----------------------------|------------------------------------------|
| `QueryRewriteMiddleware`   | Réécrit `context.query` avant la recherche |
| `RankingMiddleware`        | Re-classe `context.results` après la recherche |

```python
from whoosh_modern.middleware.search import QueryRewriteMiddleware

def add_synonyms(query: str) -> str:
    # Étendre la requête avec des synonymes avant l'exécution
    return query + " " + get_synonyms(query)

rewriter = QueryRewriteMiddleware(rewriter=add_synonyms)
```

#### Middleware d'Analyse (`whoosh_modern.middleware.analyzer`)

| Classe                  | Description                              |
|------------------------|------------------------------------------|
| `StemmingMiddleware`   | Applique un stemmer aux champs de document et à la requête |
| `SynonymMiddleware`    | Étend le texte avec des synonymes (placeholder) |

## Créer un Middleware Personnalisé

### Middleware Basé sur des Hooks

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class RequestLoggingMiddleware(Middleware):
    """Journaliser toutes les recherches avec le timing."""

    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        import time
        context.metadata["_start_time"] = time.time()
        logger.info(f"[RECHERCHE] Requête: {context.query}")
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        elapsed = time.time() - context.metadata.get("_start_time", time.time())
        result_count = len(context.results) if context.results is not None else 0
        logger.info(f"[RÉSULTATS] Trouvé {result_count} résultats en {elapsed:.3f}s")
        return context
```

### Middleware Personnalisé (Basé sur des Hooks)

La classe de base `Middleware` est `whoosh.middleware.base.Middleware`, ré-exportée depuis
`whoosh_modern.middleware`. Sous-classez-la et implémentez les hooks du cycle de vie :

```python
from whoosh_modern.middleware import Middleware
from whoosh.middleware.context import MiddlewareContext

class RetryMiddleware(Middleware):
    """Réessaie les opérations échouées avec backoff (illustratif)."""

    def __init__(self, attempts: int = 3) -> None:
        self._attempts = attempts

    def on_error(self, context: MiddlewareContext, exc: Exception) -> None:
        # Les middleware de résilience intégrés implémentent déjà ce pattern.
        context.metadata.setdefault("retry_errors", 0)
        context.metadata["retry_errors"] += 1
        raise exc
```

> Note : les `RetryMiddleware`, `LoggingMiddleware` et `CacheMiddleware` intégrés exposent
> aussi un helper `wrap(operation)` (préservé pour compatibilité ascendante) leur permettant
> de décorer de simples callables, mais leur mécanisme principal reste le pipeline de hooks
> ci-dessus.

### Middleware avec Intégration de Plugin

Enregistrer du middleware via un plugin pour qu'il soit automatiquement découvert :

```python
from whoosh.plugins.manager import Plugin

class LoggingPlugin(Plugin):
    name = "logging"
    version = "1.0.0"
    middleware = ["whoosh_modern.middleware.LoggingMiddleware"]

    def register(self, manager):
        manager.register_middleware(
            "logging",
            LoggingMiddleware(),
        )
```

## Gestion des Erreurs

### StopOperation

Abandonner une opération de pipeline gracieusement :

```python
from whoosh.middleware.exceptions import StopOperation

class RateLimitMiddleware(Middleware):
    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if not rate_limiter.allow(context):
            raise StopOperation("Limite de taux dépassée")
        return context
```

### Comportement fail_open

```python
class ResilientMiddleware(Middleware):
    def on_error(self, context: MiddlewareContext, exc: Exception) -> None:
        try:
            send_to_analytics(context.results)
        except Exception:
            # Journaliser mais ne pas échouer la recherche
            logger.warning("Analytics failed", exc_info=True)
        # La chaîne de middleware continue
```

## Découverte de Middleware depuis les Plugins

Quand `PluginManager.load_plugins()` est appelé, tous les plugins qui déclarent une liste `middleware` auront leurs classes de middleware importées et instanciées. La méthode `get_middleware_chain()` construit une `MiddlewareChain` à partir de tous les middleware enregistrés :

```python
from whoosh.plugins.manager import PluginManager

PluginManager.load_plugins()  # Découvre les plugins et leurs middleware

manager = PluginManager._default
chain = manager.get_middleware_chain()
# chain est une MiddlewareChain prête à l'emploi
```

## Bonnes Pratiques

1. **Sans état** : Utilisez `context.metadata` pour les données par requête, pas les attributs d'instance
2. **Hooks légers** : Gardez les hooks `before_*` et `after_*` rapides ; utilisez async pour les E/S
3. **L'ordre compte** : Placez le cache avant les métriques, l'authentification avant le routage
4. **Fail fast** : N'utilisez `fail_open=True` que pour les middleware non critiques
5. **Testabilité** : Mockez le `MiddlewareContext` pour tester le middleware indépendamment
6. **Nettoyage** : Implémentez `shutdown()` pour les ressources comme les connexions et les minuteurs

## Voir Aussi

- [Guide Système de Plugins](plugins-avances.md) — Enregistrement et entry points des plugins
- [Intégration des Providers](provider-integration.md) — Guide complet du pipeline pour tous les providers
- [Exemples: Middleware](../examples/middleware.md) — Patterns de middleware pratiques
- [API: Middleware](../api/middleware.md) — Référence complète de l'API
- [API: Middleware Pipeline (moderne)](../api/modern.md) — Extensions middleware modernes


## DOCUMENT (FR): Middleware

# Middleware

Le pipeline de middleware permet d'intercepter et modifier les opérations d'indexation et de recherche. C'est le mécanisme d'extension principal pour les préoccupations transverses comme le logging, le cache, les métriques et la sécurité.

## Concepts de base

Un middleware est une classe qui implémente des hooks dans le cycle de vie :

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class MonMiddleware(Middleware):
    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        # Modifier context.query ou context.metadata
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        # Accéder à context.results
        return context
```

## Hooks disponibles

| Hook | Quand | Utilisations courantes |
|------|------|------------------------|
| `startup(context)` | Initialisation | Ouvrir connexions, remplir caches |
| `shutdown(context)` | Nettoyage | Fermer connexions, flush buffers |
| `before_index(context)` | Avant indexation | Validation, enrichissement, flags compression |
| `after_index(context)` | Après indexation | Métriques, événements, invalidation cache |
| `before_delete(context)` | Avant suppression | Journalisation audit, contrôle d'accès |
| `after_delete(context)` | Après suppression | Métriques, invalidation cache |
| `before_search(context)` | Avant recherche | Réécriture de requête, cache, auth |
| `after_search(context)` | Après résultats | Logging, métriques, modification résultats |
| `on_error(context, exc)` | Sur exception | Gestion d'erreur, fallbacks |
| `on_commit(context)` | Après commit | Métriques, notifications |

## Classes intégrées

### MetricsMiddleware

```python
from whoosh.middleware import MetricsMiddleware

metrics = MetricsMiddleware()
# Après opérations:
stats = metrics.get_metrics()
# Retourne: {"documents_indexed": N, "searches_executed": N}
```

### CacheMiddleware

```python
from whoosh.middleware import CacheMiddleware

cache = CacheMiddleware()
cached = cache.get_cached("requête utilisateur")
cache.set_cached("requête utilisateur", results)
```

## MiddlewareChain

```python
from whoosh.middleware import MiddlewareChain

chain = MiddlewareChain([
    MetricsMiddleware(),
    CacheMiddleware()
])

# Exécuter un hook before
context = MiddlewareContext("search")
context.query = "test"
context = chain.run_before("before_search", context)

# ... opération core ...

# Exécuter un hook after
context = chain.run_after("after_search", context)
```

## Intégration

### Avec Writer

```python
from whoosh.middleware.integration import apply_middleware_to_writer

writer = apply_middleware_to_writer(ix.writer(), chain.middlewares)

with writer:
    writer.add_document(title="Bonjour", content="Monde")
```

### Avec Searcher

```python
from whoosh.middleware.integration import apply_middleware_to_searcher

searcher = apply_middleware_to_searcher(ix.searcher(), chain.middlewares)
results = searcher.search("query")
```

## Exemple: middleware personnalisé

```python
class RequestLoggingMiddleware(Middleware):
    """Journaliser toutes les recherches."""

    def before_search(self, context: MiddlewareContext):
        context.metadata["request_id"] = generate_request_id()
        logger.info(f"Recherche: {context.query}")
        return context

    def after_search(self, context: MiddlewareContext):
        logger.info(f"Trouvé: {len(context.results)} résultats")
        return context

class RateLimitMiddleware(Middleware):
    """Abandonner les recherches dépassant la limite."""

    def before_search(self, context: MiddlewareContext):
        if not rate_limiter.allow(context):
            raise StopOperation("Limite de taux dépassée")
        return context
```

## Gestion des erreurs

```python
class ResilientMiddleware(Middleware):
    """Continuer malgré les erreurs non critiques."""

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        try:
            send_to_analytics(context.results)
        except Exception:
            logger.warning("Analytics failed", exc_info=True)
        return context
```

## Bonnes pratiques

1. **Sans état**: Utilisez `context.metadata` pour les données par requête
2. **Fail fast**: Utilisez `fail_open=True` uniquement pour middleware non critique
3. **L'ordre compte**: Placez le cache avant les métriques, l'auth avant le routage
4. **Performance**: Gardez les hooks légers; utilisez async pour les I/O
5. **Testabilité**: Mockez le contexte pour tester le middleware isolément

## Middleware Moderne (Whoosh-NG 2.0)

Whoosh-NG 2.0 ajoute un package middleware moderne (`whoosh_modern.middleware`) avec un middleware de résilience de type wrapper (réessaissance, cache, journalisation) et un middleware basé sur des hooks pour le stockage, la recherche et l'analyse. Pour plus de détails sur l'architecture moderne du middleware, l'intégration de plugins et le déploiement, consultez le [Guide Middleware & Pipeline de Plugins](middleware-pipeline.md).


## DOCUMENT (FR): Modern Indexing

# Indexation Moderne

Whoosh-NG fournit une couche d'indexation optimisée dans `whoosh_modern.indexing` pour l'absorption de grands volumes de documents. Ces utilitaires encapsulent l'écrivain Whoosh de base sans modifier les internals de la bibliothèque.

## BatchIndexWriter

`BatchIndexWriter` encapsule un écrivain Whoosh avec les optimisations suivantes pour le traitement par lots de grands ensembles de données.

### Optimisations clés

- Pré-calcule les noms de champs du schéma pour un filtrage rapide (O(1) par champ)
- Ignore les champs non présents dans le schéma (évitant le surcoût par document)
- Utilise `multisegment=True` pour reporter les fusions pendant l'indexation
- Supporte des commits par lots configurables pour réduire la pression I/O
- Accepte un callback pour les hooks post-commit

### Utilisation de base

```python
from whoosh_modern.indexing import BatchIndexWriter
from whoosh import index

ix = index.open_dir("indexdir")

writer = BatchIndexWriter(ix, batch_size=5000, commit_every=10)

for batch in source.stream_batches(batch_size=5000):
    writer.add_batch(batch)

writer.close()
```

### Context Manager

```python
with BatchIndexWriter(ix, batch_size=10000) as writer:
    for doc in documents:
        writer.add_document(doc)
```

### Paramètres

| Paramètre | Par défaut | Description |
|-----------|---------|-------------|
| `batch_size` | 5000 | Nombre de documents par lot |
| `limitmb` | 512 | Limite mémoire pour l'écrivain (Mo) |
| `commit_every` | None | Commit après N lots (None = pas de commit auto) |
| `multisegment` | True | Utiliser le mode multisegment |
| `callback` | None | Callback invoqué après chaque commit |
| `**writer_kwargs` | None | Arguments supplémentaires pour `index.writer()` |

### Avec Commit Profiler

```python
from whoosh_modern.indexing import BatchIndexWriter
from whoosh_modern.profiling import CommitProfilerV2

profiler = CommitProfilerV2()
with BatchIndexWriter(ix, batch_size=5000, commit_every=5, commit_profiler=profiler) as writer:
    for batch in source.stream_batches(batch_size=5000):
        writer.add_batch(batch)

print(profiler.report())
```

---

## AnalyzerCache

`AnalyzerCache` fournit un cache LRU pour les résultats d'analyse, évitant les travaux d'analyse redondants sur des valeurs de champ répétées.

### Utilisation de base

```python
from whoosh_modern.indexing import BatchIndexWriter
from whoosh_modern.profiling import AnalyzerCache

cache = AnalyzerCache(maxsize=50000)
analyzer = StandardAnalyzer()

for doc in docs:
    cache_key = f"title:{doc['title']}"
    tokens = cache.get(cache_key)
    if tokens is None:
        tokens = list(analyzer(doc['title']))
        cache.put(cache_key, tokens)
```

### Avec get_or_compute

```python
from whoosh_modern.profiling import AnalyzerCache

cache = AnalyzerCache(maxsize=50000)
analyzer = StandardAnalyzer()

for doc in docs:
    tokens = cache.get_or_compute(
        f"title:{doc['title']}",
        lambda: list(analyzer(doc['title']))
    )
```

### Statistiques du cache

```python
print(f"Hit rate: {cache.hit_rate:.1%}")
print(f"Size: {cache.size}/{cache.maxsize}")
print(cache.report())
```

### Dimensionnement à partir de données de profiling

```python
from whoosh_modern.profiling import AnalyzerCache, CacheAnalyzer

analyzer = CacheAnalyzer()
analysis = analyzer.analyze(source.iter_documents())

cache = AnalyzerCache.from_profiling(analysis.to_dict())
```

---

## FieldAnalyzerCache

`FieldAnalyzerCache` encapsule un analyseur et met en cache les résultats par champ.

### Utilisation de base

```python
from whoosh_modern.profiling import FieldAnalyzerCache

field_cache = FieldAnalyzerCache(
    analyzer=StandardAnalyzer(),
    fields=["Country", "City"],
    cache_size=50000,
)

for doc in docs:
    for field in ["Country", "City"]:
        tokens = field_cache.analyze(field, doc[field])
```

### Invalidation du cache

```python
# Invalider une entrée spécifique
field_cache.invalidate("Country", "USA")

# Vider le cache entier
field_cache.clear()
```

### Statistiques du cache

```python
print(f"Hit rate: {field_cache.hit_rate:.1%}")
print(field_cache.report())
```

---

## Sources de données disponibles

| Classe | Type | Dépendances |
|-------|------|-------------|
| `SQLSource` | Bases SQL | `sqlite3` (stdlib) |
| `SQLAlchemySource` | SQLAlchemy | `sqlalchemy` |
| `RESTSource` | API REST | aucune (stdlib `urllib`) |
| `GraphQLSource` | API GraphQL | aucune (stdlib `urllib`) |
| `FastCSVSource` | Fichiers CSV | aucune |
| `JSONSource` | JSON/JSONL | aucune |
| `ParquetSource` | Parquet | `pyarrow` ou `pandas` |
| `PandasSource` | DataFrames pandas | `pandas` |
| `PolarsSource` | DataFrames Polars | `polars` |
| `PeeweeSource` | ORM Peewee | `peewee` |
| `TortoiseSource` | ORM Tortoise | `tortoise-orm` |
| `PydanticSource` | Modèles Pydantic | `pydantic` |


## DOCUMENT (FR): Monitoring

# Monitoring

Whoosh-NG inclut des hooks d'observabilité intégrés et un plugin Prometheus pour le monitoring en production.

## Métriques intégrées

### MetricsMiddleware

```python
from whoosh.middleware import MetricsMiddleware, MiddlewareChain
from whoosh.middleware.integration import apply_middleware_to_writer, apply_middleware_to_searcher

chain = MiddlewareChain([MetricsMiddleware()])

writer = apply_middleware_to_writer(ix.writer(), chain.middlewares)
searcher = apply_middleware_to_searcher(ix.searcher(), chain.middlewares)

# Obtenir les métriques
metrics = chain.get_metrics()
```

## Plugin Prometheus

### Installation

```bash
pip install whoosh-ng[metrics]
```

### Métriques exposées

| Métrique | Type | Description |
|----------|------|-------------|
| `whoosh_documents_indexed_total` | Counter | Total documents indexés |
| `whoosh_searches_executed_total` | Counter | Total recherches exécutées |
| `whoosh_indexing_duration_seconds` | Histogram | Temps d'indexation |
| `whoosh_search_duration_seconds` | Histogram | Temps de recherche |
| `whoosh_index_size_bytes` | Gauge | Taille actuelle de l'index |
| `whoosh_cache_hits_total` | Counter | Cache hits |
| `whoosh_cache_misses_total` | Counter | Cache misses |

## Event Bus pour monitoring

```python
from whoosh.event_bus import EventBus, DocumentIndexed, SearchExecuted

bus = EventBus()

@bus.subscribe
def on_indexed(event: DocumentIndexed):
    stats.increment("documents.indexed")

@bus.subscribe
def on_searched(event: SearchExecuted):
    stats.timing("search.duration", event.duration)
```

## Bonnes pratiques

1. **Ajoutez MetricsMiddleware tôt**: Incluez-le dans votre chaîne de base
2. **Exportez via Prometheus**: En production, exposez l'endpoint `/metrics`
3. **Endpoint health**: Utilisez `/health` pour les health checks load balancer
4. **Logging structuré**: Corrélez les événements search/index avec des request IDs
5. **Alerting**: Définissez des alertes sur les taux d'erreur et les latences


## DOCUMENT (FR): Performance

# Performance et Benchmarking

Whoosh-NG inclut un ensemble complet d'outils de benchmarking dans `whoosh_modern.profiling` pour mesurer et comparer les performances des analyseurs. Ce guide explique comment utiliser ces outils et documente les optimisations livrées dans la version 2.0.0.

## Démarrage rapide

```python
from whoosh_modern.profiling.benchmarks.regex_tokenizer import run_p5_1
from whoosh_modern.profiling.benchmarks.token_optimization import run_p5_2
from whoosh_modern.profiling.synthetic_datasets import SyntheticDatasetGenerator
from whoosh_modern.profiling.stemmer_benchmark import StemmerBenchmark

# Générer des datasets synthétiques pour des benchmarks cohérents
gen = SyntheticDatasetGenerator(seed=42)
datasets = gen.generate_all(count=5000)

# Exécuter le benchmark du tokeniseur (P5.1)
run_p5_1(datasets)

# Exécuter le benchmark de création de tokens (P5.2)
run_p5_2(token_count=100_000)

# Exécuter le benchmark du stemmer
bench = StemmerBenchmark()
bench.run(gen.generate_dataset("A", 5000))
print(bench.report())
```

## Outils de benchmarking

### SyntheticDatasetGenerator

Génère des datasets de texte déterministes de complexité variable :

```python
from whoosh_modern.profiling.synthetic_datasets import SyntheticDatasetGenerator

gen = SyntheticDatasetGenerator(seed=42)
datasets = gen.generate_all(count=5000)

# Dataset A: 2 tokens/doc (court)
# Dataset B: 50 tokens/doc (moyen)
# Dataset C: 500 tokens/doc (grand)
# Dataset D: 1200 tokens/doc (très grand)
for name, texts in datasets.items():
    print(f"{name}: {len(texts)} documents")
```

### P5.1 : Benchmark du RegexTokenizer

Compare différentes implémentations de tokeniseur :

```python
from whoosh_modern.profiling.benchmarks.regex_tokenizer import run_p5_1

results = run_p5_1(datasets)

# Compare :
# - Current Regex (whoosh par défaut)
# - Compiled Global regex
# - Manual Python tokenizer
# - C extension (re2, si disponible)
```

### P5.2 : Benchmark d'optimisation des tokens

Compare différentes implémentations d'objets Token :

```python
from whoosh_modern.profiling.benchmarks.token_optimization import run_p5_2

# Compare :
# - Current Token (dict-based)
# - __slots__ optimization
# - namedtuple
# - dataclass(slots=True)
results = run_p5_2(token_count=100_000)
```

### StemmerBenchmark

Compare les backends de stemming :

```python
from whoosh_modern.profiling.stemmer_benchmark import StemmerBenchmark

bench = StemmerBenchmark()
bench.run(texts, warmup=True)
print(bench.report())
# Résultats :
# Stemmer         Tokens/s        Time (s)    Tokens
# ------------------------------------------------------
# StemFilter      1,004,172       0.1503      150,983
# PyStemmer       ~2,100,000+     0.0719+     150,983
```

## Optimisations de performance

### Résumé des gains 2.0.0

| Optimisation | Composant | Gain mesurable |
|---|---|---|
| `__slots__` sur Token | `whoosh.analysis.acore` | +35% création de tokens |
| Regex globale compilée | `RegexTokenizer` | +50% débit regex |
| Postings compactés (1 posting) | `W3TermInfo` / `W3PostingsWriter` | +35% vitesse de commit |
| Postings compactés (2-8 postings) | `W3TermInfo` / `W3PostingsWriter` | +35% vitesse de commit |
| Cache de champ dans add_postings | `whoosh.codec.base` | -93% appels write_block |
| Encodage varint des positions | `whoosh.formats` | réduction de l'overhead par terme |
| Cache de stemmer | `whoosh.analysis.morph` | taux de hit 96,5 %, 4,12x sur champs répétitifs |
| Cache d'analyseur | `whoosh_modern.profiling.analyzer_cache` | 4,12x sur champs hautement répétitifs |
| Écrivain par lots | `whoosh_modern.indexing.batch_writer` | lots filtrés optimisés |
| Optimisation setdefault stopwords | `whoosh.formats` | réduction de l'overhead dict |

### Résultats de benchmark : 20 000 documents (`customers_csv`)

```
Avant :
  commit total      : 18,653 s
  analyzing         : 8,641 s  (51,5 %)
  committing        : 10,012 s  (27,1 %)
  write_postings    : 6,5 s
  write_block calls : ~72 612

Après :
  commit total      : 6,806 s  (-63,5 %)
  analyzing         : ~2,7 s   (-68 %)
  committing        : 6,806 s  (-32 %)
  write_postings    : 6,5 s -> allouations réduites
  write_block calls : 7 565   (-93 %)
  débit             : 1 275 docs/s
```

### Résultats de benchmark : Backends de stemming (1,5M de tokens)

| Stemmer | Débit | Relatif |
|---|---|---|
| StemFilter (interne) | 1 004 172 tokens/s | 1,0x |
| PyStemmer | ~2 100 000 tokens/s | ~2,1x |

### Résultats de benchmark : Tokeniseur regex

| Tokeniseur | Débit | Relatif |
|---|---|---|
| Current regex | ~1 000 000 tokens/s | 1,0x |
| Compiled global | ~2 300 000 tokens/s | 2,3x |

### Résultats de benchmark : Objet Token

| Implémentation | Tokens/s | Relatif |
|---|---|---|
| Current (dict) | 1 000 000 | 1,0x |
| `__slots__` | ~1 350 000 | 1,35x |

## Outils de profiling

### IndexingPipelineProfiler

Profile le pipeline d'indexation complet :

```python
from whoosh_modern.profiling.indexing_pipeline_profiler import IndexingPipelineProfiler

profiler = IndexingPipelineProfiler()
for doc in documents:
    profiler.before_tokenize(doc, analyzer)
    analyzer(doc)
    profiler.after_tokenize()

report = profiler.report()
print(report)
```

### CommitProfiler

Profile les performances de commit :

```python
from whoosh_modern.profiling.commit_profiler_v2 import CommitProfiler

profiler = CommitProfiler()
# ... indexer des documents ...
ix.commit()

report = profiler.report()
# Affiche : analyze, convert_fields, write_postings, flush, commit
```

### FieldIndexProfiler

Profile les coûts de conversion des champs :

```python
from whoosh_modern.profiling.field_index_profiler import FieldIndexProfiler

profiler = FieldIndexProfiler()
# ... indexer des documents ...
report = profiler.report()
```

### IndexQualityAnalyzer

Analyse les métriques de qualité de l'index :

```python
from whoosh_modern.profiling.index_quality_analyzer import IndexQualityAnalyzer

analyzer = IndexQualityAnalyzer(index_reader)
report = analyzer.analyze()
print(f"Termes singletons: {report['singleton_terms']}/{report['total_terms']} ({report['singleton_percent']}%)")
```

## Système de fournisseurs de stemmer

Whoosh-NG propose un système de fournisseurs de stemmer :

```python
from whoosh_modern.analysis import get_stemmer, StemmingAnalyzer, list_available_backends

# Vérifier les backends disponibles
print(list_available_backends())
# {'internal': 'available', 'pystemmer': 'not installed'}

# Utiliser la détection automatique (par défaut)
analyzer = StemmingAnalyzer(stemmer="auto")

# Stemmer interne explicite
analyzer = StemmingAnalyzer(stemmer="internal")

# PyStemmer (nécessite: pip install whoosh-ng[fast-stemming])
analyzer = StemmingAnalyzer(stemmer="pystemmer")
```

## Recommandations de performance

1. **Utilisez** `StemmingAnalyzer` de `whoosh_modern.analysis` pour une sélection automatique de PyStemmer
2. **Activez le cache de stemmer** pour le contenu répétitif (`cachesize=50000` par défaut)
3. **Minimisez les champs TEXT** — utilisez KEYWORD ou ID pour les champs à faible cardinalité
4. **Évitez les positions/chars stockés** sauf si la mise en évidence l'exige
5. **Utilisez l'indexation par lots** avec des segments plus grands pour de meilleurs débits
6. **Surveillez les termes singletons** — réduisez les termes rares via des listes de stopwords

## Exécution de la suite complète de benchmarks

```bash
cd whoosh-ng

# Exécuter tous les tests P5
uv run python -m pytest tests/test_regex_tokenizer_unicode.py tests/test_token_slots.py tests/test_stemmer_compatibility.py -v

# Suite de tests complète
uv run python -m pytest -q
```


## DOCUMENT (FR): Plugins Advanced

# Système de Plugins

Module : `whoosh.plugins.manager`
Version : 2.0.0

L'architecture à plugins de Whoosh-NG permet à des paquets externes d'étendre
le pipeline central d'indexation, de recherche et d'analyse. Les plugins sont
découverts via des [points d'entrée](https://docs.python.org/3/library/importlib.metadata.html#entry-points)
Python déclarés dans `pyproject.toml` et gérés par le `PluginManager`.

## Aperçu de l'architecture

```text
PluginManager (singleton)
    ├── load_plugins(group)          # Auto-découverte via les points d'entrée
    ├── register(plugin)             # Enregistrement manuel
    ├── enable(name) / disable(name) # Bascule du cycle de vie
    ├── get(name) / list_plugins()   # Inspection
    ├── get_middleware_chain()       # Construit la MiddlewareChain depuis les middlewares des plugins
    ├── register_datasource()        # Enregistre un provider de source de données
    ├── register_vector_provider()   # Enregistre un provider vectoriel
    ├── register_middleware()        # Enregistre une instance de middleware
    ├── register_embedding()         # Enregistre un provider d'embeddings
    ├── register_analyzer()          # Enregistre un analyseur nommé
    └── register_analyzer()          # Enregistre un rewriteur de requêtes
```

## Classes de base des plugins

### Plugin (ABC)

La classe de plugin racine. Les sous-classes définissent des attributs au niveau
classe et implémentent `register()`.

```python
from whoosh.plugins.manager import Plugin, PluginMetadata

class MyPlugin(Plugin):
    name = "my_plugin"
    version = "1.0.0"

    def register(self, manager: PluginManager) -> None:
        """Appelé au chargement du plugin ; enregistrez les providers ici."""
        manager.register_middleware("my_module.MyMiddleware", MyMiddleware())

    def register_hooks(self) -> None:
        """Enregistre les hooks d'événements (optionnel)."""
        from whoosh.hooks import hookimpl, register_hook

        @hookimpl
        def on_search(request, response):
            pass
        register_hook("on_search", hookimpl(on_search))
```

### AnalyzerPlugin

Pour les plugins fournissant des tokenizers/analyseurs personnalisés :

```python
from whoosh.plugins.manager import AnalyzerPlugin

class MyAnalyzerPlugin(AnalyzerPlugin):
    name = "my_analyzer"

    def register(self, manager):
        manager.register_analyzer("my_analyzer", MyTokenizer())
```

### QueryRewritePlugin

Pour les plugins transformant les requêtes avant exécution :

```python
from whoosh.plugins.manager import QueryRewritePlugin

class SynonymRewriterPlugin(QueryRewritePlugin):
    name = "synonym_rewriter"

    def rewrite(self, query, searcher):
        # Renvoie la requête modifiée
        return query
```

## PluginMetadata

Une dataclass décrivant les métadonnées du plugin :

| Champ         | Type              | Description                            |
|---------------|-------------------|----------------------------------------|
| `name`        | `str`             | Nom de plugin unique                   |
| `version`     | `str`             | Chaîne de version SemVer               |
| `depends_on`  | `list[str]`       | Noms des plugins requis                |
| `priority`    | `int`             | Priorité d'ordre de chargement (plus élevé = plus tard) |
| `middleware`  | `list[str]`       | Chemins pointés vers les classes de middleware |

## Groupes de points d'entrée

Le `PluginManager` découvre les plugins depuis ces groupes de points d'entrée
standard :

| Groupe                | Rôle                                 |
|----------------------|--------------------------------------|
| `whoosh.plugins`     | Plugins généraux                     |
| `whoosh.datasources` | Providers de source de données       |
| `whoosh.vector.providers` | Providers de similarité vectorielle |
| `whoosh.middlewares` | Classes de middleware                 |
| `whoosh.embeddings`  | Providers de modèles d'embeddings     |
| `whoosh.language`    | Analyseurs spécifiques à une langue  |
| `whoosh.apps`        | Usines d'applications (FastAPI, admin, etc.) |

## Créer et déployer un plugin

### Étape 1 : Définir la classe du plugin

```python
# my_plugin/plugin.py
from whoosh.plugins.manager import Plugin
from whoosh.registry import VectorRegistry

class MyVectorPlugin(Plugin):
    name = "my_vector"
    version = "1.0.0"
    depends_on = []
    conflicts_with = []
    priority = 0
    middleware = []

    def register(self, manager):
        """Enregistre un provider vectoriel dans le VectorRegistry."""
        provider = MyCustomVectorProvider()
        VectorRegistry.register("my_vector", provider, owner=self.name)

    def register_hooks(self):
        """Enregistre des hooks optionnels (ex : on_search, on_index)."""
        pass
```

### Étape 2 : Déclarer le point d'entrée

Dans votre `pyproject.toml` :

```toml
[project]
name = "whoosh-ng-my-vector"
version = "1.0.0"
dependencies = ["whoosh-ng>=2.0"]

[project.entry-points."whoosh_ng.plugins"]
my_vector = "my_plugin.plugin:MyVectorPlugin"
```

### Étape 3 : Installer et vérifier

```bash
pip install -e .
```

```python
# Vérifie que le plugin est enregistré
from whoosh.plugins.manager import PluginManager

PluginManager.load_plugins()  # Auto-découvre tous les points d'entrée

manager = PluginManager._default
print(manager.list_plugins())
# ['whoosh_autocomplete', 'whoosh_vector', ..., 'my_vector']

# Vérifie le registre
from whoosh.registry import VectorRegistry
print(VectorRegistry.list_keys())
# ['my_vector', 'numpy']
```

## Enregistrement manuel (sans point d'entrée)

Pour les tests ou un usage programmatique :

```python
from whoosh.plugins.manager import PluginManager

manager = PluginManager()
manager.register(MyVectorPlugin())
manager.enable("my_vector")
```

## Cycle de vie des plugins

```
1. Point d'entrée découvert  ───►  2. register() appelé  ───►  3. register_hooks()
   │                               │                            │
   └── load_plugins(group)          └── enregistre provider/     └── register_hook()
                                       middleware/analyzer
```

### Activation / Désactivation

```python
from whoosh.plugins.manager import PluginManager

manager = PluginManager._default

manager.enable("my_vector")    # Active un plugin
manager.disable("my_vector")   # Désactive un plugin
print(manager.list_enabled())  # Uniquement les plugins activés
```

### Validation de version

```python
# Vérifie si un plugin respecte une version minimale
ok = manager.validate_version("my_vector", "1.0.0")
print(ok)  # True si la version du plugin >= 1.0.0
```

### Détection de conflits

```python
# Vérifie si deux plugins sont en conflit
if manager.detect_conflicts("plugin_a", "plugin_b"):
    print("Ces plugins ne peuvent pas être chargés ensemble")
```

## Référence de l'API du Plugin Manager

### `PluginManager.load_plugins(group=None)`

Charge tous les plugins depuis les groupes de points d'entrée. Si `group` est
`None`, charge depuis tous les groupes standards (`STANDARD_GROUPS`).

### `PluginManager.register(plugin)`

Enregistre une instance de plugin. Appelle `plugin.register(self)` et
`plugin.register_hooks()`. Supporte un `register()` asynchrone via `asyncio`.

### `PluginManager.get_middleware_chain()`

Construit et renvoie une `MiddlewareChain` à partir de tous les plugins
déclarant des entrées `middleware`. Les classes de middleware sont importées et
instanciées par chemin pointé.

### Méthodes d'enregistrement dans les registres

| Méthode                       | Description                          |
|------------------------------|--------------------------------------|
| `register_analyzer(name, analyzer)` | Enregistre un analyseur nommé   |
| `register_datasource(name, datasource)` | Enregistre une source de données |
| `register_vector_provider(name, provider)` | Enregistre un provider vectoriel |
| `register_middleware(name, middleware)` | Enregistre une instance de middleware |
| `register_embedding(name, embedding)` | Enregistre un provider d'embeddings |
| `register_query_rewriter(plugin)` | Enregistre un plugin de rewriteur de requêtes |

### Méthodes de recherche

| Méthode                       | Renvoie                          |
|------------------------------|----------------------------------|
| `get(name)`                  | Instance `Plugin`                |
| `list_plugins()`             | Tous les noms de plugins enregistrés |
| `list_enabled()`             | Noms des plugins activés         |
| `get_analyzer(name)`         | Analyseur appelable              |
| `list_analyzers()`           | Noms des analyseurs enregistrés  |
| `list_datasources()`         | Noms des sources de données enregistrées |
| `list_vector_providers()`    | Noms des providers vectoriels enregistrés |
| `list_middlewares()`         | Noms des middlewares enregistrés |
| `list_embeddings()`          | Noms des embeddings enregistrés  |
| `list_query_rewriters()`     | Noms des rewriteurs de requêtes enregistrés |

## Plugins intégrés

| Plugin            | Module                  | Groupe de point d'entrée |
|-------------------|-------------------------|--------------------------|
| `whoosh_autocomplete` | `whoosh_modern.autocomplete.plugin` | `whoosh.plugins` |
| `whoosh_vector`   | `whoosh_modern.vector.plugin`      | `whoosh.plugins` |
| `whoosh_fastapi`  | `whoosh_fastapi`                  | `whoosh.apps` |
| `whoosh_observability` | `whoosh.middleware.metrics`  | `whoosh.middlewares` |
| `whoosh_admin`    | `whoosh_admin`                   | `whoosh.apps` |

## Bonnes pratiques

1. **Responsabilité unique** : un plugin, une fonctionnalité
2. **Déclarer les dépendances** : utilisez `depends_on` pour les plugins requis
3. **Versionnement sémantique** : incrémentez la version pour les changements d'API
4. **Dégradation gracieuse** : vérifiez les dépendances optionnelles dans `register()`
5. **Éviter les effets de bord dans `__init__`** : toute l'initialisation dans `register()`
6. **Nettoyage** : si applicable, fournissez une logique de teardown

## Voir aussi

- [Guide du Middleware](middleware-pipeline.md) — Hooks du pipeline et middleware personnalisés
- [Guide d'Intégration des Providers](provider-integration.md) — Guide complet du pipeline pour tous les providers
- [Exemple de Développement de Plugin](../examples/plugin-dev.md) — Tutoriel pas à pas d'un plugin
- [API : Plugins](../api/plugins.md) — Référence complète de l'API


## DOCUMENT (FR): Plugins

# Plugins

Whoosh-NG utilise une architecture à plugins pour garder le core léger tout en permettant des fonctionnalités avancées. Les plugins sont chargés via des entry points et gérés par le `PluginManager`.

## Architecture des plugins

```text
PluginManager
    ├── load_plugins()           # Auto-découvrir depuis entry points
    ├── register(name, plugin)   # Enregistrement manuel
    ├── enable(name)            # Activer un plugin
    ├── disable(name)           # Désactiver un plugin
    ├── get(name)               # Récupérer un plugin
    └── list_plugins()          # Lister tous les plugins
```

## Plugins intégrés

| Plugin | Description |
|--------|-------------|
| whoosh-ng-vector | Recherche vectorielle (NumPy, HNSW, Faiss) |
| whoosh-ng-autocomplete | Autocomplétion par edge n-gram |
| whoosh-ng-fastapi | Factory d'app FastAPI |
| whoosh-ng-observability | Métriques Prometheus |
| whoosh-ng-admin | Interface d'administration |

## Créer un plugin

Tout plugin hérite de `BasePlugin` :

```python
from whoosh.plugins.base import BasePlugin

class MonPlugin(BasePlugin):
    name = "mon_plugin"
    version = "1.0.0"
    dependencies = []

    def setup(self, registry):
        """Appelé quand le plugin est activé."""
        registry.register("mon_provider", MonProvider())

    def teardown(self, registry):
        """Appelé quand le plugin est désactivé."""
        registry.unregister("mon_provider")

    def middleware(self):
        """Middleware à injecter dans le pipeline."""
        return [MonMiddleware()]

    def on_startup(self):
        """Appelé une fois au démarrage."""
        pass

    def on_shutdown(self):
        """Appelé une fois à l'arrêt."""
        pass
```

## Enregistrement de plugin

### Via entry_points (pyproject.toml)

```toml
[project.entry-points."whoosh_ng.plugins"]
mon_plugin = "mon_package.plugin:MonPlugin"
```

### Programmatique

```python
from whoosh.plugins.manager import PluginManager

plugin = MonPlugin()
PluginManager.register("mon_plugin", plugin)
PluginManager.enable("mon_plugin")
```

## Cycle de vie d'un plugin

```
register() -> setup() -> enable() -> hooks middleware -> teardown() -> disable()
```

## Dépendances entre plugins

```python
class VectorPlugin(BasePlugin):
    name = "vector"
    dependencies = ["metrics"]  # Requiert le plugin metrics
```

Le `PluginManager` résout l'ordre de chargement et détecte les conflits.

## Bonnes pratiques

1. **Un plugin, une responsabilité**: Gardez les plugins petits et focalisés
2. **Déclarez les dépendances**: Aidez PluginManager à résoudre l'ordre
3. **Nettoyez bien**: Implémentez `teardown()` pour supprimer les registres

## Système de Plugins Moderne (Whoosh-NG 2.0)

Whoosh-NG 2.0 introduit un `PluginManager` amélioré avec support de registres pour les sources de données, les fournisseurs de vecteurs, les embeddings et le middleware. Pour plus de détails sur l'architecture moderne des plugins, les groupes d'entry points et le déploiement, consultez le [Guide Système de Plugins](plugins-avances.md).


## DOCUMENT (FR): Provider Integration

# Intégration des Providers : Guide Complet du Pipeline

Module: `whoosh_modern.storage`, `whoosh_modern.analysis.stemmer_providers`, `whoosh_modern.linguistics.synonyms`, `whoosh_modern.vector`, `whoosh_modern.autocomplete`
Version: 2.0.0

Ce guide explique comment tous les providers de Whoosh-NG s'intègrent dans le
pipeline d'indexation et de recherche. Il est la référence définitive pour
comprendre le flux de données des documents bruts aux résultats de recherche.

## Vue d'ensemble

Whoosh-NG utilise un **pattern de provider** pour garder le moteur de recherche
léger tout en activant un comportement pluggable pour le stockage, l'analyse de
texte, la recherche vectorielle et l'autocomplétion.

```
┌──────────────────────────────────────────────────────────────────────┐
│                        Pile de Providers Whoosh-NG                   │
│                                                                      │
│  ┌─────────────┐  ┌──────────────┐  ┌─────────────┐  ┌───────────┐ │
│  │ Stockage    │  │ Stemmer      │  │ Synonyme    │  │ Vecteur   │ │
│  │ Providers   │  │ Providers    │  │ Providers   │  │ Providers │ │
│  │             │  │              │  │             │  │           │ │
│  │ FileStorage │  │ Internal     │  │ Static      │  │ Numpy     │ │
│  │ S3Storage   │  │ PyStemmer    │  │ YAML        │  │ HNSW      │ │
│  │ Hybride     │  │ Identity     │  │ JSON        │  │ Faiss     │ │
│  │ SQLite      │  │ Custom       │  │ SQLite      │  │ Qdrant    │ │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  └─────┬─────┘ │
│         │                │                │                │       │
│         ▼                ▼                ▼                ▼       │
│  ┌─────────────────────────────────────────────────────────────────┐│
│  │              Pipeline de Middleware (hooks)                     ││
│  │  ┌─────────────┐  ┌──────────────┐  ┌─────────────────────┐    ││
│  │  │Stockage     │  │Stemming      │  │Synonyme             │    ││
│  │  │Middleware   │  │Middleware    │  │ExpansionMiddleware  │    ││
│  │  └─────────────┘  └──────────────┘  └─────────────────────┘    ││
│  └─────────────────────────────────────────────────────────────────┘│
│                              │                                       │
│                              ▼                                       │
│  ┌─────────────────────────────────────────────────────────────────┐│
│  │              Moteur Core Whoosh                                 ││
│  │  Index │ Writer │ Searcher │ QueryParser │ Fichiers segment     ││
│  └─────────────────────────────────────────────────────────────────┘│
└──────────────────────────────────────────────────────────────────────┘
```

## Pipeline d'Indexation Complet

### Flux étape par étape

```text
┌─────────────────┐
│   DataSource    │  (SQL, JSON, REST, CSV, DataFrame, etc.)
│   .stream_batches() │
└────────┬────────┘
         │ lots de documents
         ▼
┌─────────────────┐
│  SchemaDiscovery │  Infère le schéma Whoosh depuis les colonnes de la source
│  .discover_schema() │
└────────┬────────┘
         │ Schema(TEXT, ID, NUMERIC, VECTOR, ...)
         ▼
┌─────────────────────────────────────────┐
│  Résolution du Provider de Stockage      │
│                                         │
│  storage._root (si présent)             │
│    └──► whoosh.index.create_in(root)    │
│  Pas de root                             │
│    └──► tempfile.mkdtemp() → create_in() │
└────────┬────────────────────────────────┘
         │ Instance Index
         ▼
┌─────────────────────────────────────────┐
│  Writer + MiddlewareChain                │
│                                         │
│  chain.run_before("before_index")        │
│    ├── StorageMiddleware                 │
│    │   └── marque le contexte            │
│    ├── StemmingMiddleware                │
│    │   └── stemme les champs             │
│    └── SynonymExpansionMiddleware        │
│        └── expande les champs            │
│                                         │
│  writer.add_document(**doc)              │
│    └── Whoosh core applique les analyzeurs│
│        (TEXT.analyzer)                    │
│        et écrit le segment               │
│                                         │
│  writer.commit()                         │
│    └── chain.run_after("on_commit")      │
│        └── StorageMiddleware             │
│            └── écrit un point de commit  │
└────────┬────────────────────────────────┘
         │ Fichiers segment sur le disque/S3/cache
         ▼
┌─────────────────┐
│  Index Whoosh    │
│  (segments)      │
└─────────────────┘
```

### Exemple concret

```python
from whoosh import index, fields
from whoosh_modern import (
    SearchApplication,
    SQLSource,
    HybridStorage,
    S3Storage,
    StemmingAnalyzer,
    get_stemmer,
    SynonymManager,
    SynonymExpansionMiddleware,
    StorageMiddleware,
    StemmingMiddleware,
)
from sqlalchemy import create_engine

# 1. Source de données
engine = create_engine("sqlite:///products.db")
source = SQLSource(query="SELECT id, name, description FROM products", connection=engine)

# 2. Stockage
remote = S3Storage(bucket="my-index-bucket", prefix="segments")
storage = HybridStorage(local_cache="./cache", remote=remote)

# 3. Schéma (auto-découvert des colonnes SQL)
#    Mais on personnalise l'analyseur
stemmer = get_stemmer("auto", "english")
schema = fields.Schema(
    name=fields.TEXT(analyzer=StemmingAnalyzer(stemmer=stemmer), stored=True),
    description=fields.TEXT(analyzer=StemmingAnalyzer(stemmer=stemmer)),
    id=fields.ID(stored=True, unique=True),
)

# 4. Créer l'index dans la racine du stockage
ix = index.create_in(storage._cache_root, schema)

# 5. Construire la chaîne de middleware
syn_manager = SynonymManager({"laptop": ["notebook", "portable"]})
chain = MiddlewareChain([
    StorageMiddleware(storage, name="products"),
    StemmingMiddleware(stemmer=stemmer.stem),
    SynonymExpansionMiddleware(syn_manager),
])

# 6. Indexer avec le middleware
with MiddlewareWriter(ix.writer(), chain) as writer:
    for batch in source.stream_batches():
        for doc in batch:
            writer.add_document(**doc)
    writer.commit()
```

## Pipeline de Recherche Complet

### Flux étape par étape

```text
┌─────────────────┐
│   Requête Utilisateur │  "running cats"
└────────┬────────┘
         │
         ▼
┌─────────────────────────────────────────┐
│  MiddlewareChain.run_before("search")    │
│                                         │
│  ├── StemmingMiddleware                  │
│  │   └── "running cats" → "run cat"      │
│  ├── SynonymExpansionMiddleware          │
│  │   └── "run cat" → "run cat running feline" │
│  └── QueryRewriteMiddleware              │
│      └── réécritures personnalisées      │
└────────┬────────────────────────────────┘
         │ Requête modifiée
         ▼
┌─────────────────────────────────────────┐
│  QueryParser.parse(query)                │
│    └── Objet Query (Term, And, Or...)    │
└────────┬────────────────────────────────┘
         │ Objet Query
         ▼
┌─────────────────────────────────────────┐
│  Searcher.search(query)                  │
│                                         │
│  ┌───────────────────────────────────┐  │
│  │  Chemin de recherche par mot-clé  │  │
│  │  └── lit les listes de postings │  │
│  │      depuis les fichiers segment   │  │
│  └───────────────────────────────────┘  │
│                                         │
│  ┌───────────────────────────────────┐  │
│  │  Chemin de recherche vectorielle  │  │
│  │  └── VectorRegistry.get(provider) │  │
│  │      └── NumpyProvider.search()   │  │
│  │          └── similarité cosinus   │  │
│  └───────────────────────────────────┘  │
│                                         │
│  ┌───────────────────────────────────┐  │
│  │  Chemin d'autocomplétion            │  │
│  │  └── AutocompleteRegistry.get()   │  │
│  │      └── provider.suggest()       │  │
│  └───────────────────────────────────┘  │
└────────┬────────────────────────────────┘
         │ Résultats bruts
         ▼
┌─────────────────────────────────────────┐
│  MiddlewareChain.run_after("search")     │
│                                         │
│  └── RankingMiddleware                   │
│      └── réclasse les résultats          │
└────────┬────────────────────────────────┘
         │ Résultats finaux
         ▼
┌─────────────────┐
│  Hits retournés   │
└─────────────────┘
```

### Exemple concret

```python
from whoosh.qparser import QueryParser
from whoosh_modern.middleware import (
    StemmingMiddleware,
    RankingMiddleware,
    QueryRewriteMiddleware,
)
from whoosh_modern.analysis import get_stemmer
from whoosh_modern.vector import NumpyProvider
from whoosh_modern.vector.plugin import VectorPlugin
from whoosh.plugins.manager import PluginManager
import numpy as np

# 1. Configurer les plugins au démarrage
manager = PluginManager()
VectorPlugin().register(manager)

# 2. Ouvrir l'index
ix = index.open_dir("indexdir")

# 3. Construire la chaîne de middleware
stemmer = get_stemmer("auto", "english")
chain = MiddlewareChain([
    StemmingMiddleware(stemmer=stemmer.stem),
    QueryRewriteMiddleware(rewriter=lambda q: q + " portable"),  # ajouter synonyme
    RankingMiddleware(ranker=lambda r: sorted(r, key=lambda h: h.score, reverse=True)),
])

# 4. Rechercher avec le middleware
with MiddlewareSearcher(ix.searcher(), chain) as searcher:
    # La requête est transformée par le middleware avant l'exécution
    results = searcher.search("laptop")
    for hit in results:
        print(f"{hit['name']}: {hit.score:.4f}")

    # 5. Recherche vectorielle (parallèle)
    query_vec = np.random.rand(384).tolist()
    vector_results = searcher.vector_search("embedding", query_vec, limit=10)
    for hit in vector_results:
        print(f"doc_id={hit.doc_id}, score={hit.score:.4f}")
```

## Matrice de Comparaison des Providers

| Aspect | Stockage | Stemmer | Synonyme | Vecteur | Autocomplétion |
|--------|----------|---------|----------|---------|----------------|
| **Point d'intégration** | `StorageMiddleware` + `create_in()` | `StemmingAnalyzer` (champ) + `StemmingMiddleware` | `SynonymExpansionMiddleware` | `VectorRegistry` + format de segment | `AutocompleteRegistry` + autonome |
| **Enregistrement** | Manuel ou `__getattr__` | Décorateur `register_stemmer()` | CRUD `SynonymManager` | `VectorPlugin.register()` | `AutocompletePlugin.register()` |
| **Utilisé à l'indexation** | Oui (points de commit) | Oui (analyseur de champ + middleware) | Oui (before_index) | Oui (champ VECTOR) | Non (autonome ou post-indexation) |
| **Utilisé à la recherche** | Oui (lectures de segments via le système de fichiers) | Oui (analyseur de champ + middleware) | Oui (before_search) | Oui (vector_search) | Oui (suggest/search) |
| **Persistance** | Fichiers segment / S3 / SQLite | En mémoire (sans état) | En mémoire / YAML / JSON / SQLite | Fichiers segment (métadonnées) | En mémoire (liste de phrases) |
| **Configuration** | Classe provider + kwargs | Nom du backend + langue | Dict de mapping ou fichier | Nom du provider + métrique | Type de provider + paramètres |

## Patterns Communs

### Pattern 1 : Provider comme Analyseur de Champ

Utilisé par: Stemmer providers, analyzeurs linguistiques

```python
schema = Schema(
    content=TEXT(analyzer=StemmingAnalyzer(stemmer="auto"))
)
```

Le provider est encapsulé dans un analyzeur Whoosh et appliqué automatiquement.

### Pattern 2 : Provider comme Middleware

Utilisé par: Stockage, Stemmer, Synonymes

```python
chain = MiddlewareChain([
    StorageMiddleware(storage),
    StemmingMiddleware(stemmer=stemmer.stem),
    SynonymExpansionMiddleware(manager),
])
```

Le provider est consommé par des hooks de middleware dans le pipeline.

### Pattern 3 : Provider comme Entrée du Registre

Utilisé par: Vecteur, Autocomplétion

```python
VectorRegistry.register("numpy", NumpyProvider(), owner="my_app")
provider = VectorRegistry.get("numpy", "my_app")
```

Le provider est stocké dans un registre global et résolu par nom à l'exécution.

### Pattern 4 : Provider comme Service Autonome

Utilisé par: Autocomplétion, Vecteur (mode manuel)

```python
provider = NumpyProvider()
provider.add([(doc_id, vec)])
results = provider.search(query_vec)
```

Le provider fonctionne indépendamment de l'index/searcher Whoosh.

## Bonnes Pratiques

1. **Choisir le bon pattern d'intégration** : Les analyseurs de champs pour les schémas statiques, le middleware pour le comportement dynamique, le registre pour les backends interchangeables.
2. **Éviter les double-applications** : Ne pas utiliser à la fois les analyseurs de niveau champ et le middleware pour la même transformation (ex: stemming).
3. **Enregistrer les providers au démarrage** : Appeler `VectorPlugin().register(manager)` et `AutocompletePlugin().register(manager)` avant de créer les index.
4. **Utiliser l'API de plus haut niveau quand c'est possible** : `SearchApplication` pour le bout en bout, `create_autocomplete()` pour les suggestions, `get_stemmer()` pour le stemming.
5. **Garder les providers sans état** : Les providers ne devraient pas contenir d'état spécifique à l'index; utilisez le contexte de middleware pour les données par requête.
6. **Tester les providers en isolation** : Chaque provider devrait être testable sans Whoosh core (tests unitaires pour `provider.search()`, `provider.add()`).
7. **Documenter les dépendances des providers** : Noter les dépendances optionnelles (boto3, PyStemmer, PyYAML) dans les exigences de votre projet.

## Voir Aussi

- [Guide des Fournisseurs de Stockage](storage-providers.md) — Intégration des backends de stockage
- [Guide des Stemmers](stemmers-fournisseurs.md) — Intégration des fournisseurs de stemmers
- [Guide de Recherche Vectorielle](vector.md) — Intégration des fournisseurs de vecteurs
- [Guide d'Autocomplétion](autocomplete-providers.md) — Intégration des fournisseurs d'autocomplétion
- [Guide Middleware](middleware-pipeline.md) — Pipeline hooks et adaptateurs de providers
- [Guide Plugins](plugins-advanced.md) — Enregistrement et entry points des plugins
- [API: Moderne](../api/modern.md) — Référence complète de l'API pour tous les providers


## DOCUMENT (FR): Stemming Providers

# Providers de Stemmers

Module : `whoosh_modern.analysis.stemmer_providers`, `whoosh_modern.analysis.stemming_analyzer`, `whoosh_modern.linguistics.stemmers`
Version : 2.0.0

Le système de providers de stemmers vous donne un contrôle flexible sur le
backend de stemming utilisé pour l'analyse de texte. Il prend en charge
l'auto-détection, la sélection explicite de backend et l'enregistrement de
stemmers personnalisés — le tout avec une API propre de style plugin.

## Aperçu du module

```text
whoosh_modern.analysis
    ├── stemmer_providers.py   # Protocole StemmerProvider, backends Internal/PyStemmer, register_stemmer, get_stemmer
    └── stemming_analyzer.py   # StemmingAnalyzer enrichi avec support de plugins

whoosh_modern.linguistics.stemmers
    └── __init__.py            # Analyseurs spécifiques à une langue (FR/EN/DE/ES/IT)
```

## Protocole StemmerProvider

Localisé dans `whoosh_modern.analysis.stemmer_providers` :

```python
from whoosh_modern.analysis.stemmer_providers import StemmerProvider

class MyStemmer(StemmerProvider):
    def stem(self, word: str) -> str:
        """Stem un seul mot."""
        ...

    @property
    def name(self) -> str:
        """Renvoie le nom du stemmer."""
        return "my_stemmer"

    @property
    def language(self) -> str:
        """Renvoie le code de langue."""
        return "english"
```

## Obtenir un stemmer

### Auto-détection (recommandée)

La fonction `get_stemmer("auto", language)` sélectionne automatiquement le
meilleur backend disponible :

```python
from whoosh_modern.analysis.stemmer_providers import get_stemmer

# Auto-détecte : privilégie PyStemmer si installé, repli sur le stemmer interne
stemmer = get_stemmer("auto", "english")
print(stemmer.stem("running"))  # "run"
print(stemmer.name)             # "pystemmer" ou "internal"
```

**Ordre de priorité :**
1. **PyStemmer** (le plus rapide, nécessite `pip install whoosh-ng[fast-stemming]`)
2. **Stemmer interne** (Porter stemmer intégré, toujours disponible)

### Sélection explicite du backend

```python
from whoosh_modern.analysis.stemmer_providers import get_stemmer

# Force le stemmer interne
stemmer = get_stemmer("internal", "english")

# Force PyStemmer (nécessite l'installation)
stemmer = get_stemmer("pystemmer", "english")
```

### Lister les backends disponibles

```python
from whoosh_modern.analysis.stemmer_providers import list_available_backends

backends = list_available_backends()
print(backends)
# {'internal': 'available', 'pystemmer': 'available', 'my_custom': 'registered'}
```

| Backend     | Chaîne de statut    | Nécessite                          |
|-------------|---------------------|------------------------------------|
| `internal`  | `"available"`       | Aucun (toujours fourni)            |
| `pystemmer` | `"available"` / `"not installed"` | `pip install whoosh-ng[fast-stemming]` |
| Custom      | `"registered"`      | Enregistré via `@register_stemmer` |

## Providers de stemmers intégrés

### InternalStemmerProvider

Encapsule le Porter stemmer intégré de Whoosh. Toujours disponible (aucune
dépendance supplémentaire) :

```python
from whoosh_modern.analysis.stemmer_providers import InternalStemmerProvider

stemmer = InternalStemmerProvider("english")
print(stemmer.stem("cats"))    # "cat"
print(stemmer.stem("running")) # "run"
```

### PyStemmerProvider

Encapsule la bibliothèque `Stemmer` pour un stemming haute performance. Prend en
charge toutes les langues Snowball :

```python
from whoosh_modern.analysis.stemmer_providers import PyStemmerProvider

# Nécessite : pip install whoosh-ng[fast-stemming]
stemmer = PyStemmerProvider("english")
print(stemmer.stem("cats"))    # "cat"
```

**Note** : ce provider appelle `self._stemmer.stemWord(word)` pour stemmer les
mots. Assurez-vous que PyStemmer est installé, sinon l'auto-détection repliera
sur le stemmer interne.

### IdentityStemmerProvider

Un stemmer sans effet pour les tests ou quand le stemming n'est pas souhaité :

```python
from whoosh_modern.analysis.stemmer_providers import IdentityStemmerProvider

stemmer = IdentityStemmerProvider()
print(stemmer.stem("anything"))  # "anything"
```

## Enregistrer un stemmer personnalisé

Utilisez le décorateur `@register_stemmer` :

```python
from whoosh_modern.analysis.stemmer_providers import register_stemmer

@register_stemmer("simple")
class SimpleStemmer:
    def stem(self, word: str) -> str:
        # Suppression simple de suffixe
        if word.endswith("s") and len(word) > 3:
            return word[:-1]
        return word

    @property
    def name(self) -> str:
        return "simple"

    @property
    def language(self) -> str:
        return "english"

# Utilisez-le maintenant
from whoosh_modern.analysis.stemmer_providers import get_stemmer

stemmer = get_stemmer("simple", "english")
print(stemmer.stem("cats"))  # "cat"
```

## StemmingAnalyzer (enrichi)

Localisé dans `whoosh_modern.analysis.stemming_analyzer`, c'est le point d'entrée
principal pour créer des analyseurs conscient de la langue :

```python
from whoosh_modern.analysis import StemmingAnalyzer

# Auto-détecte le meilleur stemmer pour l'anglais
analyzer = StemmingAnalyzer(stemmer="auto", language="english")

# Stemmer interne explicite
analyzer = StemmingAnalyzer(stemmer="internal", language="english")

# Backend PyStemmer (si installé)
analyzer = StemmingAnalyzer(stemmer="pystemmer", language="french")

# Provider de stemmer personnalisé
analyzer = StemmingAnalyzer(stemmer=my_stemmer_instance)
```

### Paramètres de StemmingAnalyzer

| Paramètre   | Type                          | Défaut                  | Description                      |
|-------------|-------------------------------|-------------------------|----------------------------------|
| `expression`| Motif regex                  | motif de token par défaut | Regex de tokenization          |
| `stoplist`  | Itérable de mots vides       | `whoosh.analysis.STOP_WORDS` | Mots vides à filtrer       |
| `minsize`   | `int`                         | `2`                     | Longueur minimale des tokens    |
| `maxsize`   | `int \| None`                 | `None`                  | Longueur maximale des tokens    |
| `gaps`      | `bool`                        | `False`                 | Découpe sur l'expression vs. le match |
| `stemmer`   | `str \| StemmerProvider`      | `"auto"`                | Backend de stemming             |
| `language`  | `str`                         | `"english"`             | Code de langue                  |
| `ignore`    | `set[str] \| None`            | `None`                  | Mots à ignorer                  |
| `cachesize` | `int`                         | `50000`                 | Taille du cache de stemming     |

### Utilisation avec les types de champs

```python
from whoosh_modern.analysis import StemmingAnalyzer
from whoosh.fields import Schema, TEXT

# Stemmer anglais avec mots vides
en_analyzer = StemmingAnalyzer("auto", language="english")

# Stemmer français
fr_analyzer = StemmingAnalyzer("auto", language="french")

schema = Schema(
    title=TEXT(stored=True),
    content_en=TEXT(analyzer=en_analyzer),
    content_fr=TEXT(analyzer=fr_analyzer),
)
```

## Analyseurs spécifiques à une langue

Analyseurs préconstruits pour cinq langues, disponibles dans
`whoosh_modern.linguistics.stemmers` :

```python
from whoosh_modern.linguistics.stemmers import (
    EnglishAnalyzer,
    FrenchAnalyzer,
    GermanAnalyzer,
    SpanishAnalyzer,
    ItalianAnalyzer,
)

# Chacun est appelable et renvoie une liste de tokens
en = EnglishAnalyzer()
tokens = en("The quick brown foxes")
# tokens sont stemmés : ["quick", "brown", "fox"] (mots vides comme "the" supprimés)
```

### Analyseurs de langue disponibles

| Classe             | Langue   | Module                              |
|--------------------|----------|-------------------------------------|
| `EnglishAnalyzer`  | Anglais  | `whoosh_modern.linguistics.stemmers` |
| `FrenchAnalyzer`   | Français | `whoosh_modern.linguistics.stemmers` |
| `GermanAnalyzer`   | Allemand | `whoosh_modern.linguistics.stemmers` |
| `SpanishAnalyzer`  | Espagnol | `whoosh_modern.linguistics.stemmers` |
| `ItalianAnalyzer`  | Italien  | `whoosh_modern.linguistics.stemmers` |

Chacun utilise en interne `get_stemmer("auto", language)` pour sélectionner le
meilleur backend disponible et applique les mots vides spécifiques à la langue.

## Validation de compatibilité des stemmers

Validez qu'un provider de stemmer fonctionne correctement avec un ensemble de
mots de test :

```python
from whoosh_modern.analysis.stemmer_providers import (
    get_stemmer,
    validate_stemmer_compatibility,
)

stemmer = get_stemmer("auto", "english")
report = validate_stemmer_compatibility(stemmer, ["running", "cats", "jumps", "houses"])

print(report["total_words"])   # 4
print(report["successful"])    # 4 (ou moins en cas d'erreur)
print(report["failed"])        # 0
print(report["results"])       # [{'word': 'running', 'stemmed': 'run', 'success': True}, ...]
```

### Structure du rapport de compatibilité

| Champ         | Type       | Description                          |
|---------------|------------|--------------------------------------|
| `provider`    | `str`      | Nom du provider de stemming          |
| `language`    | `str`      | Code de langue                       |
| `total_words` | `int`      | Nombre total de mots de test         |
| `successful`  | `int`      | Mots stemmés avec succès             |
| `failed`      | `int`      | Mots en échec                        |
| `results`     | `list[dict]` | Résultats par mot avec `word`, `stemmed`, `success` |

## Intégration avec StemmingMiddleware

Les providers de stemmers peuvent être utilisés avec le `StemmingMiddleware`
depuis `whoosh_modern.middleware.analyzer` :

```python
from whoosh_modern.analysis.stemmer_providers import get_stemmer
from whoosh_modern.middleware.analyzer import StemmingMiddleware

stemmer = get_stemmer("auto", "english")
middleware = StemmingMiddleware(
    stemmer=stemmer.stem,
    fields=["title", "content"],  # Stemme uniquement ces champs
    stem_query=True,              # Stemme aussi la requête de recherche
)
```

## Migration depuis le Whoosh classique

### Ancienne API (Whoosh 1.x/2.x)

```python
from whoosh.analysis import StemmingAnalyzer as OldAnalyzer
analyzer = OldAnalyzer("en")  # Code en dur sur "english"
```

### Nouvelle API (Whoosh-NG 2.0)

```python
from whoosh_modern.analysis import StemmingAnalyzer

# Auto-détecte le backend (recommandé)
analyzer = StemmingAnalyzer("auto", language="en")

# Ou utilisez un analyseur spécifique à une langue
from whoosh_modern.linguistics.stemmers import EnglishAnalyzer
analyzer = EnglishAnalyzer()
```

> **Note** : l'ancien `StemmingAnalyzer("en")` codait en dur la langue sur
> `"english"`. Le nouveau paramètre `StemmingAnalyzer(stemmer, language)` est
> explicite et prend en charge toutes les langues Snowball via PyStemmer.

## Installation

```bash
# Sans PyStemmer (utilise le stemmer interne, plus lent)
pip install whoosh-ng

# Avec PyStemmer (recommandé, plus rapide)
pip install whoosh-ng[fast-stemming]

# Analyse moderne complète
pip install whoosh-ng[modern]
```

## Intégration des providers de stemming dans le pipeline

Le système `StemmerProvider` s'intègre à **deux niveaux** : les analyseurs de
niveau champ et le middleware de pipeline. Comprendre les deux est essentiel
pour éviter le double stemming.

### Architecture

```text
┌─────────────────────────────────────────────────────────────────┐
│  StemmingAnalyzer (niveau champ, dans le Schema)                │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ RegexTokenizer() │ StopFilter │ StemmingAnalyzer          │  │
│  │                    (mots vides)   │                       │  │
│  │                                   ▼                       │  │
│  │                         stemfn = provider.stem            │  │
│  │                                   │                       │  │
│  │                                   ▼                       │  │
│  │                         Token(stemmed=True)                │  │
│  └───────────────────────────────────────────────────────────┘  │
│                                                                 │
│  Appliqué par le cœur de Whoosh à l'indexation ET à la recherche  │
│  (via QueryParser). Automatique, aucun middleware nécessaire.     │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│  StemmingMiddleware (niveau pipeline)                            │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ before_index(context)                                     │  │
│  │   └── stemme toutes les valeurs str dans context.document │  │
│  │                                                             │  │
│  │ before_search(context)                                     │  │
│  │   └── stemme context.query si stem_query=True             │  │
│  └───────────────────────────────────────────────────────────┘  │
│                                                                 │
│  Accroché à la MiddlewareChain. Opt-in manuel.                  │
└─────────────────────────────────────────────────────────────────┘
```

### Niveau 1 : niveau champ (automatique)

Le `StemmingAnalyzer` encapsule le `StemmingAnalyzer` intégré de Whoosh et
injecte la méthode `.stem` d'un `StemmerProvider` comme `stemfn`. Le cœur de
Whoosh l'applique automatiquement au champ à l'indexation et à la recherche.

```python
from whoosh.fields import Schema, TEXT
from whoosh_modern.analysis import StemmingAnalyzer, get_stemmer

# Auto-détecte le meilleur stemmer (PyStemmer privilégié)
stemmer = get_stemmer("auto", "english")

# Crée l'analyseur avec la fonction de stemming du provider
analyzer = StemmingAnalyzer(stemmer=stemmer)

schema = Schema(
    title=TEXT(stored=True),
    content=TEXT(analyzer=analyzer),
)

# À l'indexation : "running cats" → ["run", "cat"]
# À la recherche : QueryParser utilise aussi le même analyseur
# donc "running cats" correspond aux documents contenant "run cat"
```

**Avantages** : automatique, aucune configuration de middleware requise,
comportement index/requête cohérent.

**Inconvénients** : nécessite de définir l'analyseur sur chaque champ `TEXT`.
Plus difficile à modifier à l'exécution.

### Niveau 2 : niveau middleware (opt-in)

`StemmingMiddleware` applique le stemming au niveau du pipeline, opérant sur
les valeurs chaîne brutes dans `context.document` et `context.query` avant que
les analyseurs de Whoosh ne les voient.

```python
from whoosh_modern.middleware import StemmingMiddleware
from whoosh_modern.analysis import get_stemmer
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.wrappers import MiddlewareWriter

stemmer = get_stemmer("auto", "english")

chain = MiddlewareChain([
    StemmingMiddleware(
        stemmer=stemmer.stem,
        fields=["title", "content"],  # None = tous les champs str
        stem_query=True,
    ),
])

with MiddlewareWriter(ix.writer(), chain) as writer:
    writer.add_document(title="Running cats", content="Fast dogs")
    # before_index stemme : "Running cats" → "run cat"
    writer.commit()
```

**Avantages** : fonctionne sur n'importe quel champ sans modifier le schéma.
Peut être activé/désactivé à l'exécution.

**Inconvénients** : doit être câblé manuellement dans le pipeline. Risque de
double stemming si le champ utilise aussi `StemmingAnalyzer`.

### Exemple de pipeline complet : index + recherche

```python
from whoosh import index, fields
from whoosh.qparser import QueryParser
from whoosh_modern.analysis import StemmingAnalyzer, get_stemmer
from whoosh_modern.middleware import StemmingMiddleware
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.wrappers import MiddlewareWriter, MiddlewareSearcher

# 1. Schéma avec analyseur de niveau champ
stemmer = get_stemmer("auto", "english")
schema = fields.Schema(
    title=fields.TEXT(stored=True, analyzer=StemmingAnalyzer(stemmer=stemmer)),
    content=fields.TEXT(analyzer=StemmingAnalyzer(stemmer=stemmer)),
)

ix = index.create_in("indexdir", schema)

# 2. Index avec middleware (pas de double stemming car
#    on n'utilise pas StemmingMiddleware quand les champs ont déjà StemmingAnalyzer)
with ix.writer() as writer:
    writer.add_document(title="Running cats", content="Fast dogs")
    writer.commit()

# 3. Recherche : QueryParser applique le même analyseur à la requête
with ix.searcher() as searcher:
    qp = QueryParser("content", schema)
    q = qp.parse("running cats")
    results = searcher.search(q)
    # "running" est stemmé en "run" par l'analyseur
    # "cats" est stemmé en "cat" par l'analyseur
    # Correspond au document contenant "run" et "cat"
```

### Éviter le double stemming

```python
# FAUX : double stemming
schema = Schema(
    content=TEXT(analyzer=StemmingAnalyzer(stemmer="auto")),
)
chain = MiddlewareChain([
    StemmingMiddleware(stemmer=get_stemmer("auto").stem),  # À ne pas faire !
])
# Résultat : "running" → "run" (analyseur) → "run" (middleware) — inoffensif mais gaspilleur

# CORRECT : choisissez UN seul niveau
# Option A : niveau champ uniquement (recommandé pour les schémas statiques)
schema = Schema(content=TEXT(analyzer=StemmingAnalyzer(stemmer="auto")))
# Aucun StemmingMiddleware nécessaire

# Option B : middleware uniquement (pour les champs dynamiques)
schema = Schema(content=TEXT)  # Pas d'analyseur
chain = MiddlewareChain([StemmingMiddleware(stemmer=get_stemmer("auto").stem)])
```

### Provider de stemmer personnalisé

```python
from whoosh_modern.analysis import register_stemmer, get_stemmer

@register_stemmer("my_stemmer")
class MyStemmer:
    def stem(self, word: str) -> str:
        return word.lower().rstrip("s")

# Utilisez-le comme n'importe quel backend intégré
stemmer = get_stemmer("my_stemmer", "english")
analyzer = StemmingAnalyzer(stemmer=stemmer)
```

## Voir aussi

- [Guide Stemming et Mots Vides](../core/stemming.md) — Guide classique de stemming de Whoosh
- [Guide Synonymes & Linguistique](linguistics.md) — Moteur d'expansion de synonymes
- [Guide d'Intégration des Providers](provider-integration.md) — Guide complet du pipeline pour tous les providers
- [API : Linguistique](../api/modern.md) — Référence complète de l'API pour les extensions d'analyse


## DOCUMENT (FR): Storage Providers

# Fournisseurs de stockage

Whoosh-NG fournit des backends de stockage modulaires via les contrats
`SyncStorageProvider` / `AsyncStorageProvider`. Cela permet de persister
l'index sur le disque local, SQLite, S3, ou une configuration hybride
(cache local + distant) sans modifier l'écrivain ou l'index.

## Aperçu de l'architecture

### Niveau 1 : SnapshotStorage (Simple)

```
Writer → Local FS → Commit → Upload Segment → S3
Reader → Download Segment → Open locally
```

Très simple à maintenir. Utilisez `SnapshotStorage` quand vous voulez
utiliser S3 comme cible de sauvegarde/restauration simple sans la
complexité d'un cache local.

### Niveau 2 : CachedObjectStorage (Recommandé pour la production)

```
+----------+
|  MinIO   |
+----------+
    ^
    |
Sync |
    v
+-----------+   Cache Layer   +-----------+
| Searcher  |<--------------->| Writer    |
+-----------+                 +-----------+
         |
         v
  Local SSD
```

- L'index réside sur un SSD
- S3 sert de réplication
- Les segments sont poussés après le commit
- La restauration est possible à tout moment

C'est ce que font de nombreux systèmes de recherche distribués modernes.

## Fournisseurs disponibles

| Fournisseur | Type | Backend | Cas d'utilisation |
|-------------|------|---------|--------------------|
| `FileStorage` | sync | système de fichiers local | Nœud unique, pas de cloud |
| `AsyncFileStorage` | async | système de fichiers local | Nœud unique async |
| `S3Storage` | sync | compatible S3 | Accès direct S3 |
| `SnapshotStorage` | sync | compatible S3 | Sauvegarde/restauration simple |
| `HybridStorage` | sync | cache local + distant | **Production** (alias : `CachedObjectStorage`) |
| `AsyncHybridStorage` | async | cache local + distant | Production async |

Tous les fournisseurs sont importables depuis `whoosh_modern.storage`.

## FileStorage

Stockage sur le système de fichiers local. Les clés sont des chemins
relatifs sous `root`.

```python
from whoosh_modern.storage import FileStorage

storage = FileStorage("indexdir")
storage.write("segment_1.dat", b"data")
assert storage.read("segment_1.dat") == b"data"
assert storage.exists("segment_1.dat") is True
storage.delete("segment_1.dat")
keys = storage.list_keys()
```

## AsyncFileStorage

 Variante asynce de `FileStorage`. Toutes les opérations s'exécutent sur
un thread de travail via `asyncio.to_thread` afin de ne jamais bloquer
la boucle d'événements.

```python
import asyncio
from whoosh_modern.storage import AsyncFileStorage

storage = AsyncFileStorage("indexdir")

async def main() -> None:
    await storage.awrite("segment_1.dat", b"data")
    data = await storage.aread("segment_1.dat")
    await storage.adelete("segment_1.dat")

asyncio.run(main())
```

## S3Storage

Stockage d'objets compatible S3. `boto3` est importé de manière paresseuse,
il s'agit donc d'une dépendance optionnelle. Un `client` peut être injecté
pour les tests.

```python
from whoosh_modern.storage import S3Storage

# Client par défaut (nécessite boto3 installé et configuré)
storage = S3Storage(bucket="my-index-bucket", prefix="segments")

# Ou injecter un client pour tests / configuration personnalisée
storage = S3Storage(
    bucket="my-index-bucket",
    prefix="segments",
    client=my_boto3_client,
)

storage.write("segment_1.dat", b"data")
data = storage.read("segment_1.dat")
keys = storage.list_keys()
```

Installez la dépendance optionnelle :

```bash
pip install whoosh-ng[s3]
```

## SnapshotStorage

Stockage d'instantané S3 simple sans cache local. C'est la stratégie de
stockage S3 la plus simple :

- Écriture : téléverse le segment directement vers S3
- Lecture : télécharge le segment depuis S3 vers un fichier temporaire local

Utilisez cela quand vous voulez utiliser S3 comme cible de
sauvegarde/restauration simple sans la complexité d'un cache local.

```python
from whoosh_modern.storage import SnapshotStorage

storage = SnapshotStorage(
    local_path="./index",
    bucket="my-index-bucket",
    prefix="snapshots",
)

storage.write("segment_1.dat", b"data")
data = storage.read("segment_1.dat")
```

## HybridStorage / CachedObjectStorage

`HybridStorage` compose un cache local et un backend distant. Le backend
distant est la source de vérité ; le cache local est un anneau de
performance en écriture-transparente.

`CachedObjectStorage` est un alias pour `HybridStorage` qui exprime
mieux l'intention : un cache d'objets local synchronisé avec S3.

C'est l'architecture recommandée pour les déploiements de production
avec des modèles de lecture répétés.

```python
from whoosh_modern.storage import HybridStorage, S3Storage

remote = S3Storage(bucket="my-index-bucket", prefix="segments")
storage = HybridStorage(local_cache="./cache", remote=remote)

# Write-through : le distant est la source de vérité, le cache est mis à jour en cas de succès
storage.write("segment_1.dat", b"data")

# Première lecture : cache manquant → récupère depuis S3, écrit dans le cache
data = storage.read("segment_1.dat")

# Deuxième lecture : cache hit → servi depuis le disque local, zéro réseau
data = storage.read("segment_1.dat")

# Forcer le rafraîchissement depuis le distant
storage.invalidate("segment_1.dat")

# Préchauffer le cache de manière proactive
storage.prefetch(["segment_2.dat", "segment_3.dat"])
```

### Chemin de lecture

1. cache local hit → retourne immédiatement
2. cache manquant → lit depuis le distant, écrit dans le cache, retourne

### Chemin d'écriture

- `remote.write(key, data)` (source de vérité)
- en cas de succès → `local_cache.write(key, data)`
- en cas d'échec → lève l'erreur avant de polluer le cache

### Éviction du cache

Le cache local est borné par `max_cache_size_mb` (par défaut 1024 Mo).
Lorsque la limite est atteinte, les entrées les plus anciennes sont
évictées selon une politique LRU.

### `list_keys`

`list_keys()` utilise le backend distant comme source de vérité car le
cache n'est qu'partiel. Passez `include_cache=True` pour retourner
l'union des clés distantes et du cache.

## AsyncHybridStorage

 Variante asynce de `HybridStorage`. Les opérations distantes sont
exécutées sur un thread de travail via `asyncio.to_thread` afin de ne
jamais bloquer la boucle d'événements.

```python
import asyncio
from whoosh_modern.storage import AsyncHybridStorage, S3Storage

remote = S3Storage(bucket="my-index-bucket", prefix="segments")
storage = AsyncHybridStorage(local_cache="./cache", remote=remote)

async def main() -> None:
    await storage.awrite("segment_1.dat", b"data")
    data = await storage.aread("segment_1.dat")
    await storage.adelete("segment_1.dat")
    keys = await storage.alist_keys()

asyncio.run(main())
```

## Utilisation du stockage avec SearchApplication

```python
from whoosh_modern import SearchApplication, SQLSource
from whoosh_modern.storage import HybridStorage, S3Storage

remote = S3Storage(bucket="my-index-bucket", prefix="segments")
storage = HybridStorage(local_cache="./cache", remote=remote)

app = SearchApplication(
    source=SQLSource(query="SELECT * FROM products", connection=engine),
    storage=storage,
)
app.build()
results = app.index.search("laptop")
```

## Benchmarks de performance

Les benchmarks ont été exécutés contre une instance MinIO locale en
utilisant un index Whoosh de 28,89 Mo (2 fichiers de segment). Les
résultats sont indicatives de la performance relative entre les
stratégies sur du stockage compatible S3.

| Stratégie | Sauvegarde (Mo/s) | Restauration (Mo/s) | Notes |
|-----------|-------------------|---------------------|-------|
| `1_obj_per_segment` | 39.44 | 139.72 | Meilleur débit de restauration ; le plus simple |
| `compressed_zstd` | 31.56 | 133.74 | Moins de bande passante, surcharge CPU |
| `hybrid_cache_s3` | 44.97 | 133.61 | Meilleur sauvegarde ; lectures excellentes avec cache chaud |
| `1_obj_per_posting_list` | 0.28 | 4.79 | **À éviter** : des millions de petits objets tuent S3 |

### Recommandations

- **Par défaut** : `S3Storage` avec 1 objet par fichier de segment. Offre
  le meilleur débit de restauration et est le plus simple à exploiter.
- **Production avec lectures répétées** : `HybridStorage(local_cache, S3Storage)`.
  Après la première lecture, les lectures suivantes sont servies depuis le
  disque local à ~133 Mo/s.
- **À éviter** : 1 objet par liste de postings. S3 n'est pas optimisé
  pour des millions de petits objets ; la latence et les coûts explosent.
- **Compression** : ZSTD réduit la taille de transfert d'environ 20-30%
  au coût du CPU. Utilisez-le quand la bande passante réseau est le
  goulot d'étranglement, pas quand c'est le CPU.

### Exécution des benchmarks

```bash
# Start MinIO
docker run -d --name minio-benchmark -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \
  minio/minio:latest server /data --console-address ":9001"

# Run synthetic benchmark
python benchmark/s3_storage_benchmark.py

# Run real Whoosh index benchmark (requires customers CSV)
python benchmark/s3_storage_benchmark_real.py
```

## Comment les Fournisseurs de Stockage s'Intègre dans le Pipeline d'Indexation et de Recherche

Le fournisseur de stockage participe à deux phases distinctes : **la création de l'index** (déterminer où les segments vivent) et **l'intégration en temps réel** (via `StorageMiddleware`).

### Flux d'indexation complet avec un fournisseur de stockage

```text
DataSource.stream_batches()
    │
    ▼
SearchApplication.build()
    │
    ├── source.discover_schema() ──► Whoosh Schema
    │
    ├── résolution de storage._root
    │       │
    │       ├── HybridStorage/S3Storage/FileStorage
    │       │   └── a _root ? ──► whoosh.index.create_in(root, schema)
    │       │
    │       └── Pas de _root (S3 pur/SnapshotStorage)
    │           └── tempfile.mkdtemp() ──► create_in(tmpdir, schema)
    │
    ├── Writer = index.writer()
    │       │
    │       ├── MiddlewareChain.before_index()
    │       │   └── StorageMiddleware.before_index()
    │       │       ├── context.labels["storage_backend"] = provider.__class__.__name__
    │       │       └── context.metadata["storage_provider"] = self
    │       │
    │       ├── for batch in source.stream_batches():
    │       │       for doc in batch:
    │       │           writer.add_document(**doc)
    │       │
    │       └── writer.commit()
    │           │
    │           └── StorageMiddleware.on_commit()
    │               └── provider.write("commits/{name}/{timestamp}", b"1")
    │
    ▼
Index persistsé sur le disque / S3 / cache hybride
```

### Flux de recherche complet avec un fournisseur de stockage

```text
SearchApplication.search(query)
    │
    ├── index.searcher()
    │       │
    │       └── Whoosh core ouvre les fichiers segment depuis :
    │           ├── système de fichiers local (racine FileStorageProvider)
    │           ├── base de données SQLite (SQLiteStorageProvider)
    │           └── S3 / cache hybride (S3StorageProvider / HybridStorage)
    │
    ├── QueryParser.parse(query) ──► Objet Query
    │
    └── searcher.search(query)
        │
        └── Whoosh core lit les listes de postings depuis les fichiers segment
            └── Retourne Results (Hits)
```

### Hooks détaillés de StorageMiddleware

`StorageMiddleware` (`whoosh_modern.middleware.storage`) est le point d'intégration
qui redirige la persistance de l'index via n'importe quel `SyncStorageProvider` sans
modifier le writer.

| Hook | Quand | Ce qu'il fait |
|------|------|--------------|
| `before_index(context)` | Avant qu'un document soit ajouté | Marque le contexte avec l'étiquette `storage_backend` et les métadonnées `storage_provider` |
| `on_commit(context)` | Après `writer.commit()` | Écrit un point de commit (`commits/{name}/{timestamp}`) dans le provider |

### Insight clé : StorageProvider vs StorageMiddleware

| Composant | Rôle |
|-----------|------|
| `SyncStorageProvider` / `AsyncStorageProvider` | **Contrat** définissant `write()`, `read()`, `delete()`, `exists()`, `list_keys()` |
| `FileStorageProvider`, `S3StorageProvider`, `HybridStorage` | **Implémentations** du contrat |
| `StorageMiddleware` | **Couche d'intégration** qui appelle le provider aux hooks de cycle de vie (`before_index`, `on_commit`) |
| `SearchApplication` | **Point d'entrée** qui extrait `_root` du provider pour créer le répertoire d'index Whoosh |

Le provider ne **n'intercepte pas** les lectures internes de segments de Whoosh. Ces lectures passent par le `FileStorage` intégré de Whoosh (`whoosh.filedb.filestore`) qui lit depuis le chemin du système de fichiers donné à `create_in()`. L'abstraction provider de Whoosh-NG est conçue pour :
- Le routage personnalisé de segments (S3, SQLite, cache hybride)
- Le pointage de commit via le middleware
- Futur : l'interception de lectures/écritures au niveau des segments

## Voir Aussi

- [Intégration des Providers](provider-integration.md) — Guide complet du pipeline pour tous les providers
- [Guide Middleware](middleware-pipeline.md) — Pipeline hooks et adaptateurs de providers


## DOCUMENT (FR): Synonyms

# Synonymes

Module: `whoosh_modern.linguistics.synonyms`
Version: 3.0.0

Le moteur de synonymes fournit une expansion de synonymes au moment de la requête et de l'indexation via un système de providers pluggables. Il supporte les mappings statiques en mémoire, les fichiers YAML/JSON, la persistance SQLite, et les dictionnaires Wiktionary à grande échelle.

## Architecture des providers

Tous les providers de synonymes implémentent le protocole `SynonymProvider` :

```python
from whoosh_modern.linguistics.synonyms import SynonymProvider

class MyProvider(SynonymProvider):
    def get_synonyms(self, word: str) -> list[str]: ...
    def add_synonym(self, word: str, synonyms: list[str]) -> None: ...
    def remove_synonym(self, word: str, synonym: str) -> None: ...
```

## Providers intégrés

### StaticSynonymProvider

Provider en mémoire sauvegardé par un dictionnaire :

```python
from whoosh_modern.linguistics.synonyms import StaticSynonymProvider

provider = StaticSynonymProvider({
    "voiture": ["automobile", "véhicule"],
    "maison": ["domicile", "résidence"],
})
print(provider.get_synonyms("voiture"))  # ['automobile', 'véhicule']
```

### YAMLSynonymProvider

Charge les synonymes depuis un fichier YAML :

```yaml
# synonyms.yaml
voiture:
  - automobile
  - véhicule
maison:
  - domicile
  - résidence
```

```python
from whoosh_modern.linguistics.synonyms import YAMLSynonymProvider

provider = YAMLSynonymProvider("synonyms.yaml")
print(provider.get_synonyms("voiture"))  # ['automobile', 'véhicule']
```

### JSONSynonymProvider

Charge les synonymes depuis un fichier JSON :

```json
{
    "voiture": ["automobile", "véhicule"],
    "maison": ["domicile", "résidence"]
}
```

```python
from whoosh_modern.linguistics.synonyms import JSONSynonymProvider

provider = JSONSynonymProvider("synonyms.json")
print(provider.get_synonyms("voiture"))
```

### WiktionarySynonymProvider

Charge les synonymes depuis un fichier JSON Lines kaikki.org :

```python
from whoosh_modern.linguistics.synonyms import WiktionarySynonymProvider

provider = WiktionarySynonymProvider(
    "src/whoosh_modern/linguistics/dictionaries/wiktionary/fr.json"
)
print(provider.get_synonyms("voiture"))  # ['automobile', 'véhicule']
```

Chaque ligne du fichier dictionnaire est un objet JSON :

```json
{"word": "voiture", "s": ["automobile", "véhicule"]}
{"word": "ordinateur", "s": ["pc", "machine"]}
```

Le provider filtre :
- Les mots contenant des espaces (expressions multi-mots)
- Les entrées avec parties du discours non standard
- Les listes de synonymes vides ou manquantes

### SQLiteSynonymStore

Store de synonymes persistant sauvegardé par SQLite :

```python
from whoosh_modern.linguistics.synonyms import SQLiteSynonymStore

store = SQLiteSynonymStore("synonyms.db")
store.add_synonym("voiture", ["automobile", "véhicule"])
print(store.get_synonyms("voiture"))  # ['automobile', 'véhicule']
store.close()
```

## SynonymManager

`SynonymManager` est l'interface de haut niveau pour gérer les synonymes :

```python
from whoosh_modern.linguistics.synonyms import SynonymManager

manager = SynonymManager({"voiture": ["automobile", "véhicule"]})

# CRUD
manager.add_synonyms("maison", ["domicile", "résidence"])
print(manager.get_synonyms("maison"))  # ['domicile', 'résidence']
manager.remove_synonym("maison", "domicile")

# Import depuis des sources externes
manager.import_yaml("synonyms.yaml")       # Requiert PyYAML
manager.import_json("synonyms.json")
manager.import_wiktionary("dictionaries/wiktionary/fr.json")

# Export
manager.export_json("output.json")
```

## Mise à jour des dictionnaires Wiktionary

Les dictionnaires pré-générés se trouvent dans `src/whoosh_modern/linguistics/dictionaries/wiktionary/` :

```
wiktionary/
├── fr.json
├── en.json
├── de.json
├── es.json
├── it.json
├── manifest.json
└── README.md
```

Pour les régénérer depuis le dernier dump kaikki.org :

```bash
python scripts/update_wiktionary_dictionaries.py --all
```

Ou pour une seule langue :

```bash
python scripts/update_wiktionary_dictionaries.py --lang fr
```

Le script télécharge `kaikki.org-dictionary-all.jsonl`, extrait les synonymes par langue, filtre par tags POS autorisés, et écrit des fichiers JSON Lines compacts par langue.

## SynonymExpansionMiddleware

Intègre l'expansion de synonymes dans le pipeline de middleware :

```python
from whoosh_modern.linguistics.synonyms import (
    SynonymManager,
    SynonymExpansionMiddleware,
)

manager = SynonymManager({
    "voiture": ["automobile", "véhicule"],
    "maison": ["domicile", "résidence"],
})
middleware = SynonymExpansionMiddleware(manager)
```

Le middleware étend à la fois les requêtes de recherche et les documents indexés :

```python
# Expansion de requête
ctx = MiddlewareContext(operation="search")
ctx.query = "voiture"
ctx = middleware.before_search(ctx)
# ctx.query == "voiture automobile véhicule"

# Expansion de document
ctx = MiddlewareContext(operation="index")
ctx.document = {"title": "maison à vendre"}
ctx = middleware.before_index(ctx)
# ctx.document["title"] == "maison à vendre domicile résidence"
```

## Synonymes préconstruits par langue

`LANG_SYNONYMS` fournit des dictionnaires de démarrage pour cinq langues :

```python
from whoosh_modern.linguistics.synonyms import LANG_SYNONYMS

french_syns = LANG_SYNONYMS["fr"]
print(french_syns["voiture"])  # ['automobile', 'véhicule']

english_syns = LANG_SYNONYMS["en"]
print(english_syns["car"])  # ['automobile', 'vehicle']
```

| Langue   | Code | Entrée exemple                          |
|----------|------|----------------------------------------|
| Français | `fr` | `"voiture": ["automobile", "véhicule"]` |
| Anglais  | `en` | `"car": ["automobile", "vehicle"]`      |
| Allemand | `de` | `"auto": ["wagen", "fahrzeug"]`         |
| Espagnol | `es` | `"coche": ["automóvil", "vehículo"]`    |
| Italien  | `it` | `"auto": ["automobile", "veicolo"]`     |

## Exemple d'intégration

```python
from whoosh_modern.linguistics import (
    LANG_SYNONYMS,
    SynonymExpansionMiddleware,
    SynonymManager,
)
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.wrappers import MiddlewareWriter, MiddlewareSearcher

# 1. Construit le manager de synonymes
syn_manager = SynonymManager(LANG_SYNONYMS["fr"])
syn_manager.add_synonyms("recherche", ["query", "cherche"])

# 2. Crée le middleware
syn_middleware = SynonymExpansionMiddleware(syn_manager)

# 3. Construit la chaîne de middleware
chain = MiddlewareChain([syn_middleware])

# 4. Utilise avec writer/searcher
with MiddlewareWriter(ix.writer(), chain) as writer:
    writer.add_document(title="Comment faire une recherche dans Whoosh")

with MiddlewareSearcher(ix.searcher(), chain) as searcher:
    results = searcher.search("recherche")
```

## Intégration de l'indexation Wiktionary

`WiktionaryIndexer` peut alimenter les synonymes directement dans `SynonymManager` et `SearchApplication`.

### SynonymManager.import_wiktionary_index()

Peuple un manager depuis un index Whoosh construit :

```python
from whoosh_modern.linguistics.synonyms import SynonymManager
from whoosh_modern.linguistics.wiktionary_indexer import WiktionaryIndexer

indexer = WiktionaryIndexer("indexdir")
# ... build_index() appelé précédemment ...

manager = SynonymManager()
manager.import_wiktionary_index("indexdir", language="fr")
print(manager.get_synonyms("voiture"))
# ['automobile', 'véhicule']
```

### Intégration SearchApplication

Passe un `WiktionaryIndexer` à `SearchApplication` pour exposer un `synonym_manager` pré-peuplé :

```python
from whoosh_modern import SearchApplication
from whoosh_modern.linguistics.wiktionary_indexer import WiktionaryIndexer

indexer = WiktionaryIndexer("indexdir")
app = SearchApplication(wiktionary_indexer=indexer)

# synonym_manager est peuplé paresseusement depuis l'index
manager = app.synonym_manager
```

### Câblage de SynonymExpansionMiddleware

Combine avec le middleware pour étendre les requêtes au moment de la recherche :

```python
from whoosh_modern.linguistics.synonyms import SynonymExpansionMiddleware

middleware = SynonymExpansionMiddleware(app.synonym_manager)
```

## Voir aussi

- [Vue d'ensemble linguistique](linguistics.md) — Stemmers, analyseurs de langue, et intégration complète du pipeline
- [Pipeline de middleware](middleware-pipeline.md) — Fonctionnement des chaînes de middleware
- [Providers de stemming](stemming-providers.md) — Backends de stemming spécifiques aux langues


## DOCUMENT (FR): Vector

# Recherche vectorielle

Whoosh-NG supporte la recherche sémantique via des embeddings vectoriels. Ce guide couvre la configuration et l'utilisation des champs vectoriels.

## Concept

La recherche vectorielle permet de trouver des documents par similarité sémantique plutôt que par correspondance exacte de mots-clés.

```
Embedding requête  ----\
                       >--- Similarité cosinus ---> Résultats classés
Embedding document ---/
```

## Configuration

```python
from whoosh.fields import Schema, TEXT, VectorField

schema = Schema(
    title=TEXT(stored=True),
    content=TEXT,
    embedding=VectorField(dimensions=384)  # ex: all-MiniLM-L6-v2
)
```

## Providers

| Provider | Description | Cas d'usage |
|----------|-------------|-------------|
| `NumpyProvider` | NumPy pur, similarité cosinus | Petits/moyens indexes |
| `HNSWProvider` | Hierarchical Navigable Small World | Gros indexes, ANN rapide |
| `FaissProvider` | Facebook AI Similarity Search | Très gros indexes |
| `QdrantProvider` | Qdrant vector DB | Distribué |

## Indexation avec vecteurs

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode([
    "Premier document",
    "Deuxième document"
])

with ix.writer() as writer:
    writer.add_document(
        title="Doc 1",
        content="Python est génial",
        embedding=embeddings[0].tolist()
    )
    writer.commit()
```

## Recherche hybride (mots-clés + vecteur)

```python
with ix.searcher() as searcher:
    # Composante sémantique
    query_embedding = model.encode(["Tutoriel Python"])[0]
    vector_results = searcher.vector_search(
        "embedding", query_embedding, limit=20
    )

    # Composante mots-clés
    keyword_query = QueryParser("content", schema).parse("Python")
    keyword_results = searcher.search(keyword_query, limit=20)

    # Combiner (ex: fusion RRF)
    final_results = fuse_results(vector_results, keyword_results)
```

## Bonnes pratiques

1. **Normalisez les embeddings**: Utilisez la similarité cosinus avec des vecteurs normalisés
2. **Choisissez le provider wisely**: Numpy pour &lt;100k vecteurs, HNSW/Faiss pour plus
3. **Recherche hybride**: Combinez vecteur et mots-clés pour de meilleurs résultats
4. **Cachez les embeddings**: Pré-calculez et stockez pour éviter de recalculer
5. **Indexation par lots**: Indexez les vecteurs en lots pour l'efficacité

## Intégration des Fournisseurs de Vecteurs dans le Pipeline

Le système de recherche vectorielle s'intègre via le registre de plugins de Whoosh
et le format de segment. Le provider est stocké dans le segment d'index et résolu
au moment de la recherche.

### Architecture

```text
┌─────────────────────────────────────────────────────────────────────┐
│  Enregistrement (démarrage)                                         │
│                                                                     │
│  VectorPlugin.register(PluginManager)                              │
│    └── VectorRegistry.register("numpy", NumpyProvider(), owner)     │
│                                                                     │
│  Le provider est maintenant disponible pour tout champ VECTOR      │
│  qui spécifie provider="numpy"                                     │
└─────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────┐
│  Indexation                                                          │
│                                                                     │
│  VECTOR(dimensions=384, provider="numpy")                          │
│       │                                                             │
│       ▼                                                             │
│  PerDocWriter.add_vector_items(fieldname, field, items)            │
│       │                                                             │
│       ▼                                                             │
│  Le fichier segment contient :                                     │
│    - octets vectoriels (bruts)                                     │
│    - nom du provider ("numpy")                                     │
│    - métrique ("cosine")                                           │
│       │                                                             │
│       ▼                                                             │
│  writer.commit() → segments écrits sur le disque                  │
└─────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────┐
│  Recherche                                                         │
│                                                                     │
│  searcher.vector_search("embedding", query_vec, k=10)              │
│       │                                                             │
│       ▼                                                             │
│  Whoosh core lit le segment                                        │
│    └── récupère le nom du provider ("numpy")                        │
│       │                                                             │
│       ▼                                                             │
│  VectorRegistry.get("numpy")                                        │
│       │                                                             │
│       ▼                                                             │
│  NumpyProvider.search(query_vec, k, filter_ids)                    │
│       │                                                             │
│       ▼                                                             │
│  VectorHit[] trié par similarité cosinus                           │
└─────────────────────────────────────────────────────────────────────┘
```

### Chaîne de résolution du provider

Quand `searcher.vector_search()` est appelé, Whoosh core :

1. Lit la configuration du champ `VECTOR` depuis le schéma
2. Ouvre le fichier segment contenant les données vectorielles
3. Extrait le nom du provider stocké dans le segment (ex: `"numpy"`)
4. Recherche le provider dans `VectorRegistry`
5. Appelle `provider.search(query_vector, k, filter_ids)`
6. Retourne `list[VectorHit]`

Si le provider n'est pas enregistré, la recherche échoue avec un manquant de registre.
C'est pourquoi `VectorPlugin().register(manager)` (ou l'enregistrement manuel) est
requise au démarrage.

## Voir Aussi

- [Intégration des Providers](provider-integration.md) — Guide complet du pipeline pour tous les providers
- [Guide Middleware](middleware-pipeline.md) — Pipeline hooks et adaptateurs de providers


# Code Examples