Skip to content

Completion FSTs to be loaded off-heap at all times #14364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Mar 19, 2025
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -25,17 +25,9 @@
* @lucene.experimental
*/
public class Completion101PostingsFormat extends CompletionPostingsFormat {
/** Creates a {@link Completion101PostingsFormat} that will load the completion FST on-heap. */
/** Creates a {@link Completion101PostingsFormat} that will load the completion FST off-heap. */
public Completion101PostingsFormat() {
this(FSTLoadMode.ON_HEAP);
}

/**
* Creates a {@link Completion101PostingsFormat} that will use the provided <code>fstLoadMode
* </code> to determine if the completion FST should be loaded on or off heap.
*/
public Completion101PostingsFormat(FSTLoadMode fstLoadMode) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find that these constructors taking the fst load mode are trappy, in that they make readers think that calling them allows to override the load mode. In practice, the load mode is only ever used at read time, and that codepath goes through SPI loading, so via the default constructor. I think removing these is the best path forward to avoid confusion.

super("Completion101", fstLoadMode);
super("Completion101", FSTLoadMode.OFF_HEAP);
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,17 +27,9 @@
* @lucene.experimental
*/
public class Completion50PostingsFormat extends CompletionPostingsFormat {
/** Creates a {@link Completion50PostingsFormat} that will load the completion FST on-heap. */
/** Creates a {@link Completion50PostingsFormat} that will load the completion FST off-heap. */
public Completion50PostingsFormat() {
this(FSTLoadMode.ON_HEAP);
}

/**
* Creates a {@link Completion50PostingsFormat} that will use the provided <code>fstLoadMode
* </code> to determine if the completion FST should be loaded on or off heap.
*/
public Completion50PostingsFormat(FSTLoadMode fstLoadMode) {
super("completion", fstLoadMode);
super("completion", FSTLoadMode.OFF_HEAP);
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,17 +27,9 @@
* @lucene.experimental
*/
public class Completion84PostingsFormat extends CompletionPostingsFormat {
/** Creates a {@link Completion84PostingsFormat} that will load the completion FST on-heap. */
/** Creates a {@link Completion84PostingsFormat} that will load the completion FST off-heap. */
public Completion84PostingsFormat() {
this(FSTLoadMode.ON_HEAP);
}

/**
* Creates a {@link Completion84PostingsFormat} that will use the provided <code>fstLoadMode
* </code> to determine if the completion FST should be loaded on or off heap.
*/
public Completion84PostingsFormat(FSTLoadMode fstLoadMode) {
super("Completion84", fstLoadMode);
super("Completion84", FSTLoadMode.OFF_HEAP);
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,17 +27,9 @@
* @lucene.experimental
*/
public class Completion90PostingsFormat extends CompletionPostingsFormat {
/** Creates a {@link Completion90PostingsFormat} that will load the completion FST on-heap. */
/** Creates a {@link Completion90PostingsFormat} that will load the completion FST off-heap. */
public Completion90PostingsFormat() {
this(FSTLoadMode.ON_HEAP);
}

/**
* Creates a {@link Completion90PostingsFormat} that will use the provided <code>fstLoadMode
* </code> to determine if the completion FST should be loaded on or off heap.
*/
public Completion90PostingsFormat(FSTLoadMode fstLoadMode) {
super("Completion90", fstLoadMode);
super("Completion90", FSTLoadMode.OFF_HEAP);
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,17 +27,9 @@
* @lucene.experimental
*/
public class Completion912PostingsFormat extends CompletionPostingsFormat {
/** Creates a {@link Completion912PostingsFormat} that will load the completion FST on-heap. */
/** Creates a {@link Completion912PostingsFormat} that will load the completion FST off-heap. */
public Completion912PostingsFormat() {
this(FSTLoadMode.ON_HEAP);
}

/**
* Creates a {@link Completion912PostingsFormat} that will use the provided <code>fstLoadMode
* </code> to determine if the completion FST should be loaded on or off heap.
*/
public Completion912PostingsFormat(FSTLoadMode fstLoadMode) {
super("Completion912", fstLoadMode);
super("Completion912", FSTLoadMode.OFF_HEAP);
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,17 +27,9 @@
* @lucene.experimental
*/
public class Completion99PostingsFormat extends CompletionPostingsFormat {
/** Creates a {@link Completion99PostingsFormat} that will load the completion FST on-heap. */
/** Creates a {@link Completion99PostingsFormat} that will load the completion FST off-heap. */
public Completion99PostingsFormat() {
this(FSTLoadMode.ON_HEAP);
}

/**
* Creates a {@link Completion99PostingsFormat} that will use the provided <code>fstLoadMode
* </code> to determine if the completion FST should be loaded on or off heap.
*/
public Completion99PostingsFormat(FSTLoadMode fstLoadMode) {
super("Completion99", fstLoadMode);
super("Completion99", FSTLoadMode.OFF_HEAP);
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@
final class CompletionFieldsProducer extends FieldsProducer implements Accountable {

private FieldsProducer delegateFieldsProducer;
private Map<String, CompletionsTermsReader> readers;
private final Map<String, CompletionsTermsReader> readers;
private IndexInput dictIn;

// copy ctr for merge instance
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -122,11 +122,6 @@ public enum FSTLoadMode {

private final FSTLoadMode fstLoadMode;

/** Used only by core Lucene at read-time via Service Provider instantiation */
public CompletionPostingsFormat(String name) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was unused and redundant.

this(name, FSTLoadMode.ON_HEAP);
}

/**
* Creates a {@link CompletionPostingsFormat} that will use the provided <code>fstLoadMode</code>
* to determine if the completion FST should be loaded on or off heap.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -951,7 +951,15 @@ static IndexWriterConfig iwcWithSuggestField(Analyzer analyzer, final Set<String
new FilterCodec(TestUtil.getDefaultCodec().getName(), TestUtil.getDefaultCodec()) {
final CompletionPostingsFormat.FSTLoadMode fstLoadMode =
RandomPicks.randomFrom(random(), CompletionPostingsFormat.FSTLoadMode.values());
final PostingsFormat postingsFormat = new Completion101PostingsFormat(fstLoadMode);
//FST load mode can only be overridden via a custom completion postings format
final PostingsFormat postingsFormat = new CompletionPostingsFormat("Completion101", fstLoadMode) {
final CompletionPostingsFormat delegate = new Completion101PostingsFormat();

@Override
protected PostingsFormat delegatePostingsFormat() {
return delegate.delegatePostingsFormat();
}
};

@Override
public PostingsFormat postingsFormat() {
Expand Down
Loading