Customize the search rebuild process
Describes how to add a custom field to a search product document in search rebuild V2.
The search rebuild version 2 process improves speed and performance of index builds. Optimizely encourages customers to upgrade to version 2. However, some customizations may require additional effort to ensure proper functionality. Customers should verify upgradability with their partner. Customers will need to opt in to V2, although the new default for resets to default search will be V2.
Users with an ISC_Implementer role can opt in to V2 in the Admin Console.
-
Go to Administration > System > Settings.
-
Click the Search tab.
-
Select Version 2 on the Build Version dropdown.
-
Click Save.
Warning
V1 search no longer supported. For details on customizing V1 see Customize search rebuild V1.
Modify the search build
The general strategy for modifying the Search Build process follows:
-
Use a
PrepareToRetrieveIndexableProducts
extension to gather data needed for modification of the base results. This data should be attached to theRetrieveIndexableProductsPreparation
property of the result. This value is carried into the next extension points.
For best performance, this typically means using a custom entity framework database query executed with the general form of.ToDictionary(record => record.ProductId)
, storing this dictionary inRetrieveIndexableProductsPreparation
, and in later extensions, using.TryGetValue
to look up the product's associated extension data. For best performance and reliability, it's recommended to select only the needed columns from the database as the amount of data could be large if there are many products and this will be retained in system memory for the duration of the search build process. -
GetIndexableProducts
– Base code cannot be overridden for Search Build Version 2 and doing this is not recommended in Version 1 due to performance and compatibility risks. Instead, create a new pipeline after the base code in the pipeline sequence that modifies theIndexableProducts
property of the result. The data provided inPrepareToRetrieveIndexableProducts
is available to you in theRetrieveIndexableProductsPreparation
property of the parameter type. C#'s "iterator methods" feature is the most convenient way to add/remove/modify the content ofIndexableProducts
by enumerating it in your extension and yielding the modified results. You should not do any database queries or other network activity inside your iteration as this will cripple performance--instead, gather the data you need in aPrepareToRetrieveIndexableProducts
and use it here. If you're not adding new records, you may be able to achieve your goals using an extension ofCreateElasticsearchProductResult
alone. -
CreateElasticsearchProductResult
– Similar toGetIndexableProducts
, theRetrieveIndexableProductsPreparation
object is available to you. Within this method, you can modify a record destined for Elasticsearch or suppress it by returning null. You can extend the record by inheritingElasticsearchProduct
and adding your fields. You should not do any database queries or other network activity inside this method as this will cripple performance; instead, gather the data you need in aPrepareToRetrieveIndexableProducts
extension and use it here.
Extension example code
-
Add a class extending from
ElasticsearchProduct
with the custom property:namespace Extensions.Search.Elasticsearch.DocumentTypes.Product { using Insite.Search.Elasticsearch.DocumentTypes.Product; using Nest; [ElasticsearchType(Name = "product")] public class ElasticsearchProductCustom : ElasticsearchProduct { public ElasticsearchProductCustom(ElasticsearchProduct source) : base(source) // This constructor copies all base code properties. { } [Keyword(Name = "myCustomField ", Index = true)] public int MyCustomField { get; set; } } }
-
Gather data needed for the customization in an extension of
PrepareToRetrieveIndexableProducts
:namespace Extensions.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Pipes.PrepareToRetrieveIndexableProducts { using Insite.Core.Interfaces.Data; using Insite.Core.Plugins.Pipelines; using Insite.Data.Entities; using Insite.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Parameters; using Insite.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Results; using System.Linq; public sealed class PrepareToRetrieveIndexableProducts : IPipe<PrepareToRetrieveIndexableProductsParameter, PrepareToRetrieveIndexableProductsResult> { public int Order => 0; // This pipeline has no base code so Order can be anything. public PrepareToRetrieveIndexableProductsResult Execute(IUnitOfWork unitOfWork, PrepareToRetrieveIndexableProductsParameter parameter, PrepareToRetrieveIndexableProductsResult result) { result.RetrieveIndexableProductsPreparation = unitOfWork .GetRepository<CustomerOrder>() .GetTableAsNoTracking() .Where(order => order.Status == "Submitted") .SelectMany(order => order.OrderLines) .GroupBy(orderLine => orderLine.ProductId) .ToDictionary(group => group.Key, group => group.Count()); return result; } } }
-
This scenario doesn’t require extension of
GetIndexableProducts
, so the final step is to extract the data from our preparation result inCreateElasticsearchProductResult
and apply it to our extended record:namespace Extensions.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Pipes.CreateElasticsearchProduct { using System; using System.Collections.Generic; using Insite.Core.Interfaces.Data; using Insite.Core.Plugins.Pipelines; using Insite.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Parameters; using Insite.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Results; public sealed class ExtendElasticsearchProduct : IPipe<CreateElasticsearchProductParameter, CreateElasticsearchProductResult> { public int Order => 150; public CreateElasticsearchProductResult Execute(IUnitOfWork unitOfWork, CreateElasticsearchProductParameter parameter, CreateElasticsearchProductResult result) { var elasticsearchProductCustom = new ElasticsearchProductCustom(result.ElasticsearchProduct); if (((Dictionary<Guid, int>)parameter.RetrieveIndexableProductsPreparation).TryGetValue(elasticsearchProductCustom.ProductId, out var count)) { elasticsearchProductCustom.MyCustomField = count; } result.ElasticsearchProduct = elasticsearchProductCustom; return result; } } }
Use the new field
With this code in place, the new field will be added to your elasticsearch index product documents. You can then use this data in the query pipelines however you see fit.
This is an example of using the new field to influence the sort order of results adding a pipe in the RunProductSearch pipeline. This pipe runs after the FormSortOrder pipe (Order 300) and it modifies the SortOrderFields field on RunProductSearchResult, when the user is looking at a category page in the default Relevance sort order.
namespace Extensions.Plugins.Search.Elasticsearch.DocumentTypes.Product.Query.Pipelines.Pipes.RunProductSearch
{
using System;
using Insite.Core.Interfaces.Data;
using Insite.Core.Plugins.Pipelines;
using Insite.Core.Plugins.Search;
using Insite.Search.Elasticsearch.DocumentTypes.Product;
using Insite.Search.Elasticsearch.DocumentTypes.Product.Query.Pipelines.Parameters;
using Insite.Search.Elasticsearch.DocumentTypes.Product.Query.Pipelines.Results;
public class FormCustomSortOrder : IPipe<RunProductSearchParameter, RunProductSearchResult>
{
public int Order => 310;
public RunProductSearchResult Execute(
IUnitOfWork unitOfWork,
RunProductSearchParameter parameter,
RunProductSearchResult result)
{
if (result.SortOrderFields == null)
{
return result;
}
if (parameter.ProductSearchParameter.SortBy != SortOrderType.Relevance ||
parameter.ProductSearchParameter.SearchCriteria.IsNotBlank() ||
parameter.ProductSearchParameter.SearchCriteria.IsNotBlank())
{
return result;
}
result.SortOrderFields = new[]
{
new SortOrderField(nameof(ElasticsearchProduct.SortOrder).ToCamelCase(), true, true),
new SortOrderField(nameof(ElasticsearchProductCustom.MyCustomField).ToCamelCase(), true, true),
new SortOrderField(nameof(ElasticsearchProduct.ShortDescriptionSort).ToCamelCase())
};
return result;
}
}
}
All of the available pipelines on the product query side are:
- RunProductSearch – The main product query builder.FormProductFilterBuilds up filters for product search over category, language, attribute values, brands, price range, product line, restriction group and website. The generated filter is used by the other search pipelines.
- RunProductFacetSearch – Performs faceted searches over brand, product line, and category.
- RunBrandSearch – Searches for brands within the product index.FormRestrictionGroupFilter Builds the restriction group filter for FormProductFilter pipeline.
Note
If you have access to the Optimizely Dogfood sample repository in GitHub, it includes an example of adding a multivalue field to the index and using it to filter the products based on which warehouses have stock. It demonstrates how to add a new filter pipe to the FormProductFilter pipeline and how to pass a custom parameter from the external API into the search provider.
Updated 3 days ago