Store arrays offsets for numeric fields natively with synthetic source #124594

jordan-powers · 2025-03-11T20:22:57Z

This patch builds on the work in #122999 and #113757 to natively store array offsets for numeric fields instead of falling back to ignored source when source_keep_mode: arrays.

Nested contexts don't work right with the current offset context logic. For now, we can disable native synthetic source support and fall back to the ignored source mechanism. We can revisit to support native synthetic source within nested contexts at a later date.

This patch removes the context.isImmediateParentAnArray() check when deciding whether to store an array offset. This is necessary in case we are parsing fields within an object array. F.e. for the document `{"path":[{"int_value":10},{"int_value":20}]}` with `synthetic_source_keep: arrays` on the `int_value` field, we'd want to store the offsets for `int_value` even though the immediate parent is an object and not an array.

Single-element arrays can be unwraped into arrayless field values, so we need to handle that case.

…ng-numbers

elasticsearchmachine · 2025-03-11T20:27:12Z

Hi @jordan-powers, I've created a changelog YAML for you.

elasticsearchmachine · 2025-03-11T20:27:12Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

jordan-powers · 2025-03-11T20:49:00Z

server/src/main/java/org/elasticsearch/index/mapper/NumberFieldMapper.java

+        } else {
+            value = fieldType().nullValue;
+        }
+        if (offsetsFieldName != null && context.canAddIgnoredField()) {


I don't check context.isImmediateParentAnArray() because we also need to record the offsets for object arrays.

There was a failing test IgnoredSourceFieldMapperTests#testIndexStoredArraySourceSingleLeafElementInObjectArray

It had the following:
mapping: {"path": {"type": "object", "properties": { "int_value": { "type": "long", "synthetic_source_keep": "arrays" }}}}
document: {"path":[{"int_value":10},{"int_value":20}]}

The values for int_value were not immediate children of arrays, and so the DocumentParser would fall back to the ignored source mechanism. Then, when the synthetic _source was generated, the resultant document would look like this:
{"path":{"int_value":[10, 20]}}

Then, when this document was round-tripped and indexed in a new synthetic source index, the values would be immediate children of arrays, so the offsets would be recorded.

This caused a mismatch in the index writers (since one had the offsets field and the other did not), and so the test would fail.

My solution is to always just always record the offsets, even when the immediate parent is not an array.

This has a couple of drawbacks:

This does create an inefficiency where we're potentially storing the offsets even for single-value arrays. We could probably fix that by adding a check to FieldArrayContext#addToLuceneDocument to skip recording the offsets if there's only one non-null value.

In the source loader, we can't rely anymore on the presence of the offset field to differentiate between a single value (f.e. field: 5) and a single-value array (f.e. field: [5]). As such, my current implementation of the loader always unwraps single-value arrays into a single value. I think this is fine because it matches the ignored source implementation.

I think we need to check for context.isImmediateParentAnArray() here. The current offset encoding doesn't work well if there the immediate parent is not an array. If there is an object array higher up in the tree, then the array is pushed down to the leaf when synthesizing. This is incorrect, as this is not how the document was provided during indexing. This is why we fall back to ignored source in this case.

I suspect that the reason the test fails (with context.isImmediateParentAnArray() check) is that initially the document gets index using an array, but SortedNumericWithOffsetsDocValuesSyntheticFieldLoader normalizes that array to a value, then the round trip indexing doesn't have an array and therefor there is no offset field. (because context.isImmediateParentAnArray() returns false)

The problem is actually the other way around.

Here's the error:
java.lang.AssertionError: round trip {"path":{"int_value":[10,20]}} expected:<[_seq_no, _version, _primary_term, path.int_value]> but was:<[_seq_no, path.int_value.offsets, _version, _primary_term, path.int_value]>

The first time the document is indexed, the document has the structure {"path":[{"int_value":10},{"int_value":20}]}. Since 10 and 20 are not immediate children of arrays, instead the fallback ignored source mechanism kicks in (via the check in DocumentParser#parseObjectOrField()).

However, one of the modifications made by synthetic source is that arrays are moved to leaf fields. So when the synthetic source is returned, instead the document has the structure {"path":{"int_value":[10, 20]}}.
When this modified document is indexed, the values 10 and 20 are now immediate children of an array, and so the offset encoding happens.

This means that the offsets field does not exist the first time the document is indexed, but it exists the second time after the roundtrip.

jordan-powers · 2025-03-11T20:52:25Z

server/src/test/java/org/elasticsearch/index/mapper/NativeArrayIntegrationTestCase.java

+            } else {
+                assertThat(actualArray, Matchers.contains(expected));
+            }
+        } else {


Had to update this check since single-value arrays are unwrapped to single-values

So single slot arrays should be retained and now be normalized to a single value. This works for ip and keyword field as well. So this should work for number field types too?

This was also part of the change removing the check for context.isImmediateParentAnArray(), since the change caused single-slot arrays to be returned as single values.

Since we're rethinking that solution, I'll probably end up reverting this too.

martijnvg

Thanks @jordan-powers, I did my first review round. This looks into the right direction.

martijnvg · 2025-03-12T08:21:37Z

server/src/test/java/org/elasticsearch/index/mapper/NativeArrayIntegrationTestCase.java

@@ -49,7 +48,7 @@ public void testSynthesizeEmptyArray() throws Exception {
    public void testSynthesizeArrayRandom() throws Exception {
        var arrayValues = new Object[randomInt(64)];
        for (int j = 0; j < arrayValues.length; j++) {
-            arrayValues[j] = NetworkAddress.format(randomIp(true));
+            arrayValues[j] = getRandomValue();


martijnvg · 2025-03-12T08:39:06Z

server/src/main/java/org/elasticsearch/index/mapper/NumberFieldMapper.java

+        if (offsetsFieldName != null && context.canAddIgnoredField()) {
+            if (value != null) {
+                final long sortableLongValue = type.sortableLongValue(value);
+                context.getOffSetContext().recordOffset(offsetsFieldName, sortableLongValue);


I think that casting the value variable to Comparable works too?

Suggested change

context.getOffSetContext().recordOffset(offsetsFieldName, sortableLongValue);

context.getOffSetContext().recordOffset(offsetsFieldName, (Comparable<?>) value);

This way we don't need the type.sortableLongValue(...) method?

I'll give it a shot

martijnvg · 2025-03-12T08:45:06Z

server/src/main/java/org/elasticsearch/index/mapper/DocumentParser.java

-                        || (sourceKeepMode == Mapper.SourceKeepMode.ARRAYS && context.inArrayScope())
+                        || (sourceKeepMode == Mapper.SourceKeepMode.ARRAYS
+                            && context.inArrayScope()
+                            && fieldMapper.supportStoringArrayOffsets() == false)


Can you explain why this change is needed?

This was part of the change removing the check for context.isImmediateParentAnArray(). Adding this to the check disabled the fallback source for that object array so that we could use the offset encoding.

Since we're rethinking that solution, I'll probably end up reverting this.

martijnvg · 2025-03-12T08:46:40Z

server/src/main/java/org/elasticsearch/index/mapper/FieldArrayContext.java

@@ -97,6 +97,7 @@ static String getOffsetsFieldName(
            && sourceKeepMode == Mapper.SourceKeepMode.ARRAYS
            && hasDocValues
            && isStored == false
+            && context.isInNestedContext() == false


This makes sense. Can you add a test to NativeArrayIntegrationTestCase, which checks that if leaf array field has nested parent field, then we always fall back to ignored source?

martijnvg · 2025-03-12T09:52:56Z

server/src/test/java/org/elasticsearch/index/mapper/NativeArrayIntegrationTestCase.java

+            } else {
+                assertThat(actualArray, Matchers.contains(expected));
+            }
+        } else {


So single slot arrays should be retained and now be normalized to a single value. This works for ip and keyword field as well. So this should work for number field types too?

martijnvg · 2025-03-12T09:55:08Z

server/src/test/java/org/elasticsearch/index/mapper/OffsetDocValuesLoaderTestCase.java

@@ -159,25 +159,27 @@ public void testOffsetArrayWithNulls() throws Exception {
    }

    public void testOffsetArrayRandom() throws Exception {
-        StringBuilder values = new StringBuilder();


Is this change required to make tests pass?

The original implementation converted everything to quoted strings (f.e. ["4", "-1", "20"]). However, elasticsearch would convert them to numbers when returning the synthetic source (f.e. [4, -1, 20]), causing tests to fail. I needed some way to convert the random values into quoted or unquoted strings depending on the type. I figured that since XContentBuilder already has that logic, I'd just pass it down.

Alternatively I could change the return type of getRandomValue() to Object, then use the array(String name, Object... values) method on XContentBuilder.

martijnvg · 2025-03-12T11:36:58Z

...va/org/elasticsearch/index/mapper/SortedNumericWithOffsetsDocValuesSyntheticFieldLoader.java

+
+import java.io.IOException;
+
+class SortedNumericWithOffsetsDocValuesSyntheticFieldLoader extends SourceLoader.DocValuesBasedSyntheticFieldLoader {


I think we should instead implement this is a CompositeSyntheticFieldLoader.DocValuesLayer. (like SortedSetWithOffsetsDocValuesSyntheticFieldLoaderLayer)

martijnvg · 2025-03-12T11:40:00Z

...va/org/elasticsearch/index/mapper/SortedNumericWithOffsetsDocValuesSyntheticFieldLoader.java

+                return;
+            }
+
+            int count = count();


If this class implements CompositeSyntheticFieldLoader.DocValuesLayer then we don't need the logic here that determine whether something needs to be serialized as array. And we can serialize regular number values as regular number values (instead of wrapping them in an array)

martijnvg · 2025-03-12T11:42:28Z

server/src/main/java/org/elasticsearch/index/mapper/NumberFieldMapper.java

+        } else {
+            value = fieldType().nullValue;
+        }
+        if (offsetsFieldName != null && context.canAddIgnoredField()) {


I think we need to check for context.isImmediateParentAnArray() here. The current offset encoding doesn't work well if there the immediate parent is not an array. If there is an object array higher up in the tree, then the array is pushed down to the leaf when synthesizing. This is incorrect, as this is not how the document was provided during indexing. This is why we fall back to ignored source in this case.

martijnvg · 2025-03-12T11:54:09Z

server/src/main/java/org/elasticsearch/index/mapper/NumberFieldMapper.java

+        } else {
+            value = fieldType().nullValue;
+        }
+        if (offsetsFieldName != null && context.canAddIgnoredField()) {


I suspect that the reason the test fails (with context.isImmediateParentAnArray() check) is that initially the document gets index using an array, but SortedNumericWithOffsetsDocValuesSyntheticFieldLoader normalizes that array to a value, then the round trip indexing doesn't have an array and therefor there is no offset field. (because context.isImmediateParentAnArray() returns false)

jordan-powers added 13 commits March 7, 2025 15:20

WIP numeric fields native synthetic array support

6213d87

Fix hasValue logic for case of empty array

3d79319

Fix missing field name for all-null arrays

e3961ec

Add LongOffsetDocValuesLoaderTests.java

484dabb

Use abstract getRandomValue for testSynthesizeArrayRandom

188a478

Use correct fieldType in verifySyntheticArrayInObject

1ea276b

Avoid string-encoding expected values

0656ba6

Don't assume response will be array in NativeArrayIntegrationTestCase

b790084

Single-element arrays can be unwraped into arrayless field values, so we need to handle that case.

Add LongSyntheticSourceNativeArrayIntegrationTests

2533b0b

Fix x-pack build issues

578fc42

Merge remote-tracking branch 'upstream/main' into array-offset-encodi…

7bfd82b

…ng-numbers

jordan-powers requested a review from martijnvg March 11, 2025 20:22

elasticsearchmachine added needs:triage Requires assignment of a team area label v9.1.0 labels Mar 11, 2025

jordan-powers added >enhancement auto-backport Automatically create backport pull requests when merged test-full-bwc Trigger full BWC version matrix tests :StorageEngine/Mapping The storage related side of mappings v8.19.0 labels Mar 11, 2025

elasticsearchmachine added Team:StorageEngine and removed needs:triage Requires assignment of a team area label labels Mar 11, 2025

Update docs/changelog/124594.yaml

3d7347c

github-actions bot deployed to docs-preview March 11, 2025 20:27 View deployment

jordan-powers commented Mar 11, 2025

View reviewed changes

martijnvg reviewed Mar 12, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store arrays offsets for numeric fields natively with synthetic source #124594

Store arrays offsets for numeric fields natively with synthetic source #124594

jordan-powers commented Mar 11, 2025

elasticsearchmachine commented Mar 11, 2025

elasticsearchmachine commented Mar 11, 2025

jordan-powers Mar 11, 2025

martijnvg Mar 12, 2025

martijnvg Mar 12, 2025

jordan-powers Mar 12, 2025

jordan-powers Mar 11, 2025

martijnvg Mar 12, 2025

jordan-powers Mar 12, 2025

martijnvg left a comment

martijnvg Mar 12, 2025

martijnvg Mar 12, 2025

jordan-powers Mar 12, 2025

martijnvg Mar 12, 2025

jordan-powers Mar 12, 2025

martijnvg Mar 12, 2025

martijnvg Mar 12, 2025

martijnvg Mar 12, 2025

jordan-powers Mar 12, 2025

martijnvg Mar 12, 2025

martijnvg Mar 12, 2025

martijnvg Mar 12, 2025

martijnvg Mar 12, 2025

	context.getOffSetContext().recordOffset(offsetsFieldName, sortableLongValue);
	context.getOffSetContext().recordOffset(offsetsFieldName, (Comparable<?>) value);


		import java.io.IOException;

		class SortedNumericWithOffsetsDocValuesSyntheticFieldLoader extends SourceLoader.DocValuesBasedSyntheticFieldLoader {

Store arrays offsets for numeric fields natively with synthetic source #124594

Are you sure you want to change the base?

Store arrays offsets for numeric fields natively with synthetic source #124594

Conversation

jordan-powers commented Mar 11, 2025

elasticsearchmachine commented Mar 11, 2025

elasticsearchmachine commented Mar 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martijnvg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment