Customize the search rebuild process

The search rebuild version 2 process improves speed and performance of index builds. Optimizely encourages customers to upgrade to version 2. However, some customizations may require additional effort to ensure proper functionality. Customers should verify upgradability with their partner. Customers will need to opt in to V2, although the new default for resets to default search will be V2.

Users with an ISC_Implementer role can opt in to V2 in the Admin Console.

Go to Administration > System > Settings.
Click the Search tab.
Select Version 2 on the Build Version dropdown.
Click Save.

🚧
Warning
V1 search no longer supported. For details on customizing V1 see Customize search rebuild V1.

Modify the search build

The general strategy for modifying the Search Build process follows:

Use a PrepareToRetrieveIndexableProducts extension to gather data needed for modification of the base results. This data should be attached to the RetrieveIndexableProductsPreparation property of the result. This value is carried into the next extension points.
For best performance, this typically means using a custom entity framework database query executed with the general form of .ToDictionary(record => record.ProductId), storing this dictionary in RetrieveIndexableProductsPreparation, and in later extensions, using .TryGetValue to look up the product's associated extension data. For best performance and reliability, it's recommended to select only the needed columns from the database as the amount of data could be large if there are many products and this will be retained in system memory for the duration of the search build process.
GetIndexableProducts – Base code cannot be overridden for Search Build Version 2 and doing this is not recommended in Version 1 due to performance and compatibility risks. Instead, create a new pipeline after the base code in the pipeline sequence that modifies the IndexableProducts property of the result. The data provided in PrepareToRetrieveIndexableProducts is available to you in the RetrieveIndexableProductsPreparation property of the parameter type. C#'s "iterator methods" feature is the most convenient way to add/remove/modify the content of IndexableProducts by enumerating it in your extension and yielding the modified results. You should not do any database queries or other network activity inside your iteration as this will cripple performance--instead, gather the data you need in a PrepareToRetrieveIndexableProducts and use it here. If you're not adding new records, you may be able to achieve your goals using an extension of CreateElasticsearchProductResult alone.
CreateElasticsearchProductResult – Similar to GetIndexableProducts, the RetrieveIndexableProductsPreparation object is available to you. Within this method, you can modify a record destined for Elasticsearch or suppress it by returning null. You can extend the record by inheriting ElasticsearchProduct and adding your fields. You should not do any database queries or other network activity inside this method as this will cripple performance; instead, gather the data you need in a PrepareToRetrieveIndexableProducts extension and use it here.

Extension example code

Add a class extending from ElasticsearchProduct with the custom property:

namespace Extensions.Search.Elasticsearch.DocumentTypes.Product
{
 using Insite.Search.Elasticsearch.DocumentTypes.Product;
 using Nest;

 [ElasticsearchType(Name = "product")]
 public class ElasticsearchProductCustom : ElasticsearchProduct
 {
     public ElasticsearchProductCustom(ElasticsearchProduct source)
         : base(source) // This constructor copies all base code properties.
     {
     }

     [Keyword(Name = "myCustomField ", Index = true)]
     public int MyCustomField { get; set; }
 }
}

Gather data needed for the customization in an extension of PrepareToRetrieveIndexableProducts:

namespace Extensions.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Pipes.PrepareToRetrieveIndexableProducts
{
 using Insite.Core.Interfaces.Data;
 using Insite.Core.Plugins.Pipelines;
 using Insite.Data.Entities;
 using Insite.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Parameters;
 using Insite.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Results;
 using System.Linq;

 public sealed class PrepareToRetrieveIndexableProducts : IPipe<PrepareToRetrieveIndexableProductsParameter, PrepareToRetrieveIndexableProductsResult>
 {
     public int Order => 0; // This pipeline has no base code so Order can be anything.

     public PrepareToRetrieveIndexableProductsResult Execute(IUnitOfWork unitOfWork, PrepareToRetrieveIndexableProductsParameter parameter, PrepareToRetrieveIndexableProductsResult result)
     {
         result.RetrieveIndexableProductsPreparation = unitOfWork
             .GetRepository<CustomerOrder>()
             .GetTableAsNoTracking()
             .Where(order => order.Status == "Submitted")
             .SelectMany(order => order.OrderLines)
             .GroupBy(orderLine => orderLine.ProductId)
             .ToDictionary(group => group.Key, group => group.Count());

         return result;
     }
 }
}

This scenario doesn’t require extension of GetIndexableProducts, so the final step is to extract the data from our preparation result in CreateElasticsearchProductResult and apply it to our extended record:

namespace Extensions.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Pipes.CreateElasticsearchProduct
{
 using System;
 using System.Collections.Generic;
 using Insite.Core.Interfaces.Data;
 using Insite.Core.Plugins.Pipelines;
 using Insite.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Parameters;
 using Insite.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Results;

 public sealed class ExtendElasticsearchProduct : IPipe<CreateElasticsearchProductParameter, CreateElasticsearchProductResult>
 {
     public int Order => 150;

     public CreateElasticsearchProductResult Execute(IUnitOfWork unitOfWork, CreateElasticsearchProductParameter parameter, CreateElasticsearchProductResult result)
     {
         var elasticsearchProductCustom = new ElasticsearchProductCustom(result.ElasticsearchProduct);

         if (((Dictionary<Guid, int>)parameter.RetrieveIndexableProductsPreparation).TryGetValue(elasticsearchProductCustom.ProductId, out var count))
         {
             elasticsearchProductCustom.MyCustomField = count;
         }

         result.ElasticsearchProduct = elasticsearchProductCustom;

         return result;
     }
 }
}

Use the new field

With this code in place, the new field will be added to your elasticsearch index product documents. You can then use this data in the query pipelines however you see fit.

This is an example of using the new field to influence the sort order of results adding a pipe in the RunProductSearch pipeline. This pipe runs after the FormSortOrder pipe (Order 300) and it modifies the SortOrderFields field on RunProductSearchResult, when the user is looking at a category page in the default Relevance sort order.

namespace Extensions.Plugins.Search.Elasticsearch.DocumentTypes.Product.Query.Pipelines.Pipes.RunProductSearch
{
 using System;
 using Insite.Core.Interfaces.Data;
 using Insite.Core.Plugins.Pipelines;
 using Insite.Core.Plugins.Search;
 using Insite.Search.Elasticsearch.DocumentTypes.Product;
 using Insite.Search.Elasticsearch.DocumentTypes.Product.Query.Pipelines.Parameters;
 using Insite.Search.Elasticsearch.DocumentTypes.Product.Query.Pipelines.Results;

 public class FormCustomSortOrder : IPipe<RunProductSearchParameter, RunProductSearchResult>
 {
     public int Order => 310;

     public RunProductSearchResult Execute(
         IUnitOfWork unitOfWork,
         RunProductSearchParameter parameter,
         RunProductSearchResult result)
     {

         if (result.SortOrderFields == null)
         {
             return result;
         }

         if (parameter.ProductSearchParameter.SortBy != SortOrderType.Relevance ||
             parameter.ProductSearchParameter.SearchCriteria.IsNotBlank() || 
             parameter.ProductSearchParameter.SearchCriteria.IsNotBlank())
         {
             return result;
         }

         result.SortOrderFields = new[]
         {
             new SortOrderField(nameof(ElasticsearchProduct.SortOrder).ToCamelCase(), true, true),
             new SortOrderField(nameof(ElasticsearchProductCustom.MyCustomField).ToCamelCase(), true, true),
             new SortOrderField(nameof(ElasticsearchProduct.ShortDescriptionSort).ToCamelCase())
         };

         return result;
     }
 }
}

All of the available pipelines on the product query side are:

RunProductSearch – The main product query builder.FormProductFilterBuilds up filters for product search over category, language, attribute values, brands, price range, product line, restriction group and website. The generated filter is used by the other search pipelines.
RunProductFacetSearch – Performs faceted searches over brand, product line, and category.
RunBrandSearch – Searches for brands within the product index.FormRestrictionGroupFilter Builds the restriction group filter for FormProductFilter pipeline.

📘
Note
If you have access to the Optimizely Dogfood sample repository in GitHub, it includes an example of adding a multivalue field to the index and using it to filter the products based on which warehouses have stock. It demonstrates how to add a new filter pipe to the FormProductFilter pipeline and how to pass a custom parameter from the external API into the search provider.

🚧Warning

Modify the search build

Extension example code

Use the new field

📘Note

🚧
Warning

📘
Note