Customize the search rebuild process

The search rebuild version 2 process improves speed and performance of index builds. Optimizely recommends upgrading to version 2. However, some customizations may require additional effort to ensure proper functionality. You or your partner should verify upgradability before opting in to version 2.

📘
Note
When using Elasticsearch version 7.10, search rebuild version 2 is the default configuration and cannot be changed. You should upgrade any custom search extensions to search rebuild version 2 and Elasticsearch version 7.10.

Opt in to version 2

Users with an ISC_Implementer role can opt in to version 2 in the Admin Console.

Go to Administration > System > Settings.
Click the Search tab.
Select Version 2 on the Build Version dropdown.
Click Save.

🚧
Important
V1 search no longer supported. For details on customizing version 1, see Customize search rebuild V1.

Modify the search build

Use a PrepareToRetrieveIndexableProducts extension to gather the data needed for modification of the base results. This data should be attached to the RetrieveIndexableProductsPreparation property of the result. This value is carried into the next extension points.
For best performance, you should use a custom entity framework database query executed with the general form of .ToDictionary(record => record.ProductId), storing this dictionary in RetrieveIndexableProductsPreparation, and, in later extensions, using .TryGetValue to look up the product's associated extension data. You should select only the needed columns from the database to prevent large amounts of data slowing down performance during the search rebuild process.
Create a new pipeline after the base code in the pipeline sequence that modifies the IndexableProducts property of the result for GetIndexablePRoducts. You cannot override base code in version 2, and doing this in version 1 can vause performance and compatibility issues. The data provided in PrepareToRetrieveIndexableProducts is available in the RetrieveIndexableProductsPreparation property of the parameter type. C#'s "iterator methods" feature is the most convenient way to add/remove/modify the content of IndexableProducts by enumerating it in your extension and yielding the modified results. You should not do any database queries or other network activity inside your iteration as this will cripple performance--instead, gather the data you need in a PrepareToRetrieveIndexableProducts and use it here. If you are not adding new records, you may be able to use an extension of CreateElasticsearchProductResult alone.
Use a CreateElasticsearchProductResult extensions to modify a record destined for Elasticsearch or suppress it by returning null. You can extend the record by inheriting ElasticsearchProduct and adding your fields. You should not do any database queries or other network activity inside this method as this will cripple performance. Instead, gather the data you need in a PrepareToRetrieveIndexableProducts extension and use it here.

Extension example code

Add a class extending from ElasticsearchProduct with the custom property:

namespace Extensions.Search.Elasticsearch.DocumentTypes.Product
{
 using Insite.Search.Elasticsearch.DocumentTypes.Product;
 using Nest;

 [ElasticsearchType(Name = "product")]
 public class ElasticsearchProductCustom : ElasticsearchProduct
 {
     public ElasticsearchProductCustom(ElasticsearchProduct source)
         : base(source) // This constructor copies all base code properties.
     {
     }

     [Keyword(Name = "myCustomField ", Index = true)]
     public int MyCustomField { get; set; }
 }
}

namespace Extensions.Search.ElasticsearchV7.DocumentTypes.Product
{
 using Insite.Search.ElasticsearchV7.DocumentTypes.Product;
 using Nest7;

 [ElasticsearchType(RelationName = "product")]
 public class ElasticsearchProductCustom : ElasticsearchProduct
 {
     public ElasticsearchProductCustom(ElasticsearchProduct source)
         : base(source) // This constructor copies all base code properties.
     {
     }

     [Keyword(Name = "myCustomField ", Index = true)]
     public int MyCustomField { get; set; }
 }
}

Gather data needed for the customization in an extension of PrepareToRetrieveIndexableProducts:

namespace Extensions.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Pipes.PrepareToRetrieveIndexableProducts
{
 using Insite.Core.Interfaces.Data;
 using Insite.Core.Plugins.Pipelines;
 using Insite.Data.Entities;
 using Insite.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Parameters;
 using Insite.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Results;
 using System.Linq;

 public sealed class PrepareToRetrieveIndexableProducts : IPipe<PrepareToRetrieveIndexableProductsParameter, PrepareToRetrieveIndexableProductsResult>
 {
     public int Order => 0; // This pipeline has no base code so Order can be anything.

     public PrepareToRetrieveIndexableProductsResult Execute(IUnitOfWork unitOfWork, PrepareToRetrieveIndexableProductsParameter parameter, PrepareToRetrieveIndexableProductsResult result)
     {
         result.RetrieveIndexableProductsPreparation = unitOfWork
             .GetRepository<CustomerOrder>()
             .GetTableAsNoTracking()
             .Where(order => order.Status == "Submitted")
             .SelectMany(order => order.OrderLines)
             .GroupBy(orderLine => orderLine.ProductId)
             .ToDictionary(group => group.Key, group => group.Count());

         return result;
     }
 }
}

namespace Extensions.Search.ElasticsearchV7.DocumentTypes.Product.Index.Pipelines.Pipes.PrepareToRetrieveIndexableProducts
{
 using Insite.Core.Interfaces.Data;
 using Insite.Core.Plugins.Pipelines;
 using Insite.Data.Entities;
 using Insite.Search.ElasticsearchV7.DocumentTypes.Product.Index.Pipelines.Parameters;
 using Insite.Search.ElasticsearchV7.DocumentTypes.Product.Index.Pipelines.Results;
 using System.Linq;

 public sealed class PrepareToRetrieveIndexableProducts : IPipe<PrepareToRetrieveIndexableProductsParameter, PrepareToRetrieveIndexableProductsResult>
 {
     public int Order => 0; // This pipeline has no base code so Order can be anything.

     public PrepareToRetrieveIndexableProductsResult Execute(IUnitOfWork unitOfWork, PrepareToRetrieveIndexableProductsParameter parameter, PrepareToRetrieveIndexableProductsResult result)
     {
         result.RetrieveIndexableProductsPreparation = unitOfWork
             .GetRepository<CustomerOrder>()
             .GetTableAsNoTracking()
             .Where(order => order.Status == "Submitted")
             .SelectMany(order => order.OrderLines)
             .GroupBy(orderLine => orderLine.ProductId)
             .ToDictionary(group => group.Key, group => group.Count());

         return result;
     }
 }
}

Extract the data from the preparation result in CreateElasticsearchProductResult and apply it to the extended record (this example does not require GetIndexableProducts):

namespace Extensions.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Pipes.CreateElasticsearchProduct
{
 using System;
 using System.Collections.Generic;
 using Insite.Core.Interfaces.Data;
 using Insite.Core.Plugins.Pipelines;
 using Insite.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Parameters;
 using Insite.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Results;

 public sealed class ExtendElasticsearchProduct : IPipe<CreateElasticsearchProductParameter, CreateElasticsearchProductResult>
 {
     public int Order => 150;

     public CreateElasticsearchProductResult Execute(IUnitOfWork unitOfWork, CreateElasticsearchProductParameter parameter, CreateElasticsearchProductResult result)
     {
         var elasticsearchProductCustom = new ElasticsearchProductCustom(result.ElasticsearchProduct);

         if (((Dictionary<Guid, int>)parameter.RetrieveIndexableProductsPreparation).TryGetValue(elasticsearchProductCustom.ProductId, out var count))
         {
             elasticsearchProductCustom.MyCustomField = count;
         }

         result.ElasticsearchProduct = elasticsearchProductCustom;

         return result;
     }
 }
}

namespace Extensions.Search.ElasticsearchV7.DocumentTypes.Product.Index.Pipelines.Pipes.CreateElasticsearchProduct
{
 using System;
 using System.Collections.Generic;
 using Insite.Core.Interfaces.Data;
 using Insite.Core.Plugins.Pipelines;
 using Insite.Search.ElasticsearchV7.DocumentTypes.Product.Index.Pipelines.Parameters;
 using Insite.Search.ElasticsearchV7.DocumentTypes.Product.Index.Pipelines.Results;

 public sealed class ExtendElasticsearchProduct : IPipe<CreateElasticsearchProductParameter, CreateElasticsearchProductResult>
 {
     public int Order => 150;

     public CreateElasticsearchProductResult Execute(IUnitOfWork unitOfWork, CreateElasticsearchProductParameter parameter, CreateElasticsearchProductResult result)
     {
         var elasticsearchProductCustom = new ElasticsearchProductCustom(result.ElasticsearchProduct);

         if (((Dictionary<Guid, int>)parameter.RetrieveIndexableProductsPreparation).TryGetValue(elasticsearchProductCustom.ProductId, out var count))
         {
             elasticsearchProductCustom.MyCustomField = count;
         }

         result.ElasticsearchProduct = elasticsearchProductCustom;

         return result;
     }
 }
}

Use the new field

This code adds a new field to your Elasticsearch index product documents. You can then use this data in the query pipelines however you see fit.

This is an example of using the new field to influence the sort order of results adding a pipe in the RunProductSearch pipeline. This pipe runs after the FormSortOrder pipe (Order 300) and modifies the SortOrderFields field on RunProductSearchResult, when the user is looking at a category page in the default Relevance sort order.

namespace Extensions.Plugins.Search.Elasticsearch.DocumentTypes.Product.Query.Pipelines.Pipes.RunProductSearch
{
 using System;
 using Insite.Core.Interfaces.Data;
 using Insite.Core.Plugins.Pipelines;
 using Insite.Core.Plugins.Search;
 using Insite.Search.Elasticsearch.DocumentTypes.Product;
 using Insite.Search.Elasticsearch.DocumentTypes.Product.Query.Pipelines.Parameters;
 using Insite.Search.Elasticsearch.DocumentTypes.Product.Query.Pipelines.Results;

 public class FormCustomSortOrder : IPipe<RunProductSearchParameter, RunProductSearchResult>
 {
     public int Order => 310;

     public RunProductSearchResult Execute(
         IUnitOfWork unitOfWork,
         RunProductSearchParameter parameter,
         RunProductSearchResult result)
     {

         if (result.SortOrderFields == null)
         {
             return result;
         }

         if (parameter.ProductSearchParameter.SortBy != SortOrderType.Relevance ||
             parameter.ProductSearchParameter.SearchCriteria.IsNotBlank() || 
             parameter.ProductSearchParameter.SearchCriteria.IsNotBlank())
         {
             return result;
         }

         result.SortOrderFields = new[]
         {
             new SortOrderField(nameof(ElasticsearchProduct.SortOrder).ToCamelCase(), true, true),
             new SortOrderField(nameof(ElasticsearchProductCustom.MyCustomField).ToCamelCase(), true, true),
             new SortOrderField(nameof(ElasticsearchProduct.ShortDescriptionSort).ToCamelCase())
         };

         return result;
     }
 }
}

namespace Extensions.Plugins.Search.ElasticsearchV7.DocumentTypes.Product.Query.Pipelines.Pipes.RunProductSearch
{
 using System;
 using Insite.Core.Interfaces.Data;
 using Insite.Core.Plugins.Pipelines;
 using Insite.Core.Plugins.Search;
 using Insite.Search.ElasticsearchV7.DocumentTypes.Product;
 using Insite.Search.Shared.DocumentTypes.Product.Query.Pipelines.Parameters;
 using Insite.Search.Shared.DocumentTypes.Product.Query.Pipelines.Results;

 public class FormCustomSortOrder : IPipe<RunProductSearchParameter, RunProductSearchResult>
 {
     public int Order => 310;

     public RunProductSearchResult Execute(
         IUnitOfWork unitOfWork,
         RunProductSearchParameter parameter,
         RunProductSearchResult result)
     {

         if (result.SortOrderFields == null)
         {
             return result;
         }

         if (parameter.ProductSearchParameter.SortBy != SortOrderType.Relevance ||
             parameter.ProductSearchParameter.SearchCriteria.IsNotBlank() || 
             parameter.ProductSearchParameter.SearchCriteria.IsNotBlank())
         {
             return result;
         }

         result.SortOrderFields = new[]
         {
             new SortOrderField(nameof(ElasticsearchProduct.SortOrder).ToCamelCase(), true, true),
             new SortOrderField(nameof(ElasticsearchProductCustom.MyCustomField).ToCamelCase(), true, true),
             new SortOrderField(nameof(ElasticsearchProduct.ShortDescriptionSort).ToCamelCase())
         };

         return result;
     }
 }
}

The available pipelines on the product query side are:

RunProductSearch – The main product query builder.FormProductFilter builds up filters for product search over category, language, attribute values, brands, price range, product line, restriction groups, and website. The generated filter is used by the other search pipelines.
RunProductFacetSearch – This pipeline performs faceted searches over brand, product line, and category.
RunBrandSearch – This pipeline searches for brands within the product. index.FormRestrictionGroupFilter builds the restriction group filter for the FormProductFilter pipeline.

📘
Note
The Optimizely Dogfood sample repository in GitHub includes an example of adding a multi-value field to the index and using it to filter the products based on which warehouses have stock. It demonstrates how to add a new filter pipe to the FormProductFilter pipeline and how to pass a custom parameter from the external API into the search provider. The example is available for both Elasticsearch version 5 (Nest5, Search Rebuild Version 1) and Elasticsearch version 7 (Nest7, Search Rebuild Version 2).