Customize search rebuild V1

🚧
Important
Optimizely Configured Commerce no longer supports search rebuild version 1. See Customize the search rebuild process to upgrade to version 2.

For this article, assume you want to add a new field to the product document called MyCustomField set to the number of order lines included each product. This field could then be used in product searches to modify the sort order.

📘
Note
Your extension project must reference NEST – a strongly typed interface to Elasticsearch.

The product indexing process has 3 main pipelines, used by the ElasticsearchProductRepository to transfer data from database into the Elasticsearch index:

GetIndexableProducts – Makes optimized SQL queries to fetch all of the products and associated data that is needed for searching and returns them as a list of IndexableProduct objects.
CreateElasticsearchProduct – Converts an IndexableProduct into an ElasticsearchProduct. ElasticsearchProduct is the form of the product on Elasticsearch and includes Nest attributes which define the field names and indexing parameters. After all data is converted into a list of ElasticsearchProduct objects, the data is posted to the Elasticsearch server.
MapElasticSearchProductProperty – Handles the auto indexing of product custom properties and is not used in this example.

Add classes to the extension project

In your extension project, add two new classes:

1. A class inherited from IndexableProduct – This allows you to add new fields but keep the base fields.

namespace Extensions.Plugins.Search.Elasticsearch.DocumentTypes.Product
{
using Insite.Search.Elasticsearch.DocumentTypes.Product.Index;

public class IndexableProductCustom : IndexableProduct
{
  public int MyCustomField { get; set; }
}
}

A class inherited from ElasticsearchProduct – The constructor should copy the data in all fields in the base class.

namespace Extensions.Plugins.Search.Elasticsearch.DocumentTypes.Product
{
 using Nest;

 using Insite.Search.Elasticsearch.DocumentTypes.Product;

 [ElasticsearchType(Name = "product")]
 public class ElasticsearchProductCustom : ElasticsearchProduct
 {
     public ElasticsearchProductCustom(ElasticsearchProduct elasticsearchProduct)
     {
         this.BasicListPrice = elasticsearchProduct.BasicListPrice;
         this.BasicSaleEndDate = elasticsearchProduct.BasicSaleEndDate;
         this.BasicSalePrice = elasticsearchProduct.BasicSalePrice;
         this.BasicSaleStartDate = elasticsearchProduct.BasicSaleStartDate;
         this.BrandFacet = elasticsearchProduct.BrandFacet;
         this.BrandId = elasticsearchProduct.BrandId;
         this.BrandIsSponsored = elasticsearchProduct.BrandIsSponsored;
         this.BrandManufacturer = elasticsearchProduct.BrandManufacturer;
         this.BrandName = elasticsearchProduct.BrandName;
         this.BrandNameFirstCharacter = elasticsearchProduct.BrandNameFirstCharacter;
         this.BrandNameSort = elasticsearchProduct.BrandNameSort;
         this.BrandProductLineFacet = elasticsearchProduct.BrandProductLineFacet;
         this.BrandSearchBoost = elasticsearchProduct.BrandSearchBoost;
         this.BrandUrlSegment = elasticsearchProduct.BrandUrlSegment;
         this.Boost = elasticsearchProduct.Boost;
         this.Categories = elasticsearchProduct.Categories;
         this.CategoryNames = elasticsearchProduct.CategoryNames;
         this.CategoryTree = elasticsearchProduct.CategoryTree;
         this.Content = elasticsearchProduct.Content;
         this.CustomProperties = elasticsearchProduct.CustomProperties;
         this.CustomerNames = elasticsearchProduct.CustomerNames;
         this.Customers = elasticsearchProduct.Customers;
         this.DefaultVisibility = elasticsearchProduct.DefaultVisibility;
         this.DocumentNames = elasticsearchProduct.DocumentNames;
         this.ErpDescription = elasticsearchProduct.ErpDescription;
         this.ErpNumber = elasticsearchProduct.ErpNumber;
         this.ErpNumberNgram = elasticsearchProduct.ErpNumberNgram;
         this.ErpNumberNgramWithoutSpecialCharacters = elasticsearchProduct.ErpNumberNgramWithoutSpecialCharacters;
         this.ErpNumberWithoutSpecialCharacters = elasticsearchProduct.ErpNumberWithoutSpecialCharacters;
         this.FilterNames = elasticsearchProduct.FilterNames;
         this.Filters = elasticsearchProduct.Filters;
         this.Id = elasticsearchProduct.Id;
         this.ImageAltText = elasticsearchProduct.ImageAltText;
         this.IsSponsored = elasticsearchProduct.IsSponsored;
         this.LanguageCode = elasticsearchProduct.LanguageCode;
         this.ManufacturerItem = elasticsearchProduct.ManufacturerItem;
         this.ManufacturerItemNgram = elasticsearchProduct.ManufacturerItemNgram;
         this.ManufacturerItemNgramWithoutSpecialCharacters = elasticsearchProduct.ManufacturerItemNgramWithoutSpecialCharacters;
         this.ManufacturerItemWithoutSpecialCharacters = elasticsearchProduct.ManufacturerItemWithoutSpecialCharacters;
         this.MediumImagePath = elasticsearchProduct.MediumImagePath;
         this.MetaDescription = elasticsearchProduct.MetaDescription;
         this.MetaKeywords = elasticsearchProduct.MetaKeywords;
         this.ModelNumber = elasticsearchProduct.ModelNumber;
         this.ModifiedOn = elasticsearchProduct.ModifiedOn;
         this.Name = elasticsearchProduct.Name;
         this.PackDescription = elasticsearchProduct.PackDescription;
         this.PageTitle = elasticsearchProduct.PageTitle;
         this.Price = elasticsearchProduct.Price;
         this.PriceFacet = elasticsearchProduct.PriceFacet;
         this.ProductCode = elasticsearchProduct.ProductCode;
         this.ProductId = elasticsearchProduct.ProductId;
         this.ProductUrlSegment = elasticsearchProduct.ProductUrlSegment;
         this.RestrictionGroups = elasticsearchProduct.RestrictionGroups;
         this.SearchLookup = elasticsearchProduct.SearchLookup;
         this.ShippingWeight = elasticsearchProduct.ShippingWeight;
         this.ShortDescription = elasticsearchProduct.ShortDescription;
         this.ShortDescriptionSort = elasticsearchProduct.ShortDescriptionSort;
         this.Sku = elasticsearchProduct.Sku;
         this.SmallImagePath = elasticsearchProduct.SmallImagePath;
         this.SortOrder = elasticsearchProduct.SortOrder;
         this.Specifications = elasticsearchProduct.Specifications;
         this.SpellingCorrection = elasticsearchProduct.SpellingCorrection;
         this.StyledChildren = elasticsearchProduct.StyledChildren;
         this.UnitOfMeasure = elasticsearchProduct.UnitOfMeasure;
         this.UnitOfMeasureDescription = elasticsearchProduct.UnitOfMeasureDescription;
         this.Unspsc = elasticsearchProduct.Unspsc;
         this.UpcCode = elasticsearchProduct.UpcCode;
         this.Vendor = elasticsearchProduct.Vendor;
         this.Version = elasticsearchProduct.Version;
         this.Websites = elasticsearchProduct.Websites;
     }

     [Keyword(Name = "myCustomField ", Index = true)]
     public int MyCustomField  { get; set; }
 }
}

Define the pipes

Next you must define three pipes to modify the indexing process:

Make a GetIndexableProducts pipe which sets the CustomFields property on the result object. This property should be set to a SQL fragment which is inserted into the main indexing SELECT query and fetches the additional needed data. Alias the data with the name of the custom field (in this case, MyCustomField).

This pipe should run between Order 100 and 200 so that it is after the GetIndexableProductsSqlStatementParts pipe (which sets the SqlStatement field to the large SQL block) and before the pipe CombineIndexableProductsSqlStatementParts (which builds the final query.)
If you need to make more radical changes to the main indexing SQL query, you can modify the result.SqlStatement field directly (or replace GetIndexableProductsSqlStatementParts entirely). However, you should avoid doing this to prevent upgradability issues.
Doing large subqueries can adversely affect indexing performance. This is example is for instructional purposes.

namespace Extensions.Plugins.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Pipes.GetIndexableProducts
{
 using Insite.Core.Interfaces.Data;
 using Insite.Core.Plugins.Pipelines;
 using Insite.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Parameters;
 using Insite.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Results;

 public class AddCustomFieldsToSqlStatement : IPipe<GetIndexableProductsParameter, GetIndexableProductsResult>
 {
     public int Order => 150;

     public GetIndexableProductsResult Execute(
         IUnitOfWork unitOfWork,
         GetIndexableProductsParameter parameter,
         GetIndexableProductsResult result)
     {
         result.CustomFields = @"
             (select count(*) from OrderLine ol with (nolock)
             inner join CustomerOrder o on o.id = ol.CustomerOrderId 
             where p.Id = ol.ProductId and o.Status = 'Submitted') 
             as MyCustomField";

         return result;
     }
 }
}

Replace the pipe PerformIndexableProductsSqlQuery at Order 300 to deserialize the results of the SQL query into the IndexableProductCustom type.

namespace Extensions.Plugins.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Pipes.GetIndexableProducts
{
 using Insite.Core.Interfaces.Data;
 using Insite.Core.Plugins.Pipelines;
 using Insite.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Parameters;
 using Insite.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Results;

 public class PerformIndexableProductsSqlQuery : IPipe<GetIndexableProductsParameter, GetIndexableProductsResult>
 {
     public int Order => 300;

     public GetIndexableProductsResult Execute(IUnitOfWork unitOfWork,
                                               GetIndexableProductsParameter parameter,
                                               GetIndexableProductsResult result)
     {
         result.IndexableProducts = unitOfWork.DataProvider.SqlQuery(
             result.FormattedSqlStatement, null, false, parameter.QueryTimeout);

         return result;
     }
 }
}

Create a new pipe in the CreateElasticsearchProduct pipeline to copy the custom data field from the IndexableProduct to the ElasticsearchProduct.

namespace Extensions.Plugins.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Pipes.CreateElasticsearchProduct
{
 using Insite.Core.Interfaces.Data;
 using Insite.Core.Plugins.Pipelines;
 using Insite.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Parameters;
 using Insite.Search.Elasticsearch.DocumentTypes.Product.Index.Pipelines.Results;

 public class SetCustomFields : IPipe<CreateElasticsearchProductParameter, CreateElasticsearchProductResult>
 {
     public int Order => 150;

     public CreateElasticsearchProductResult Execute(
         IUnitOfWork unitOfWork,
         CreateElasticsearchProductParameter parameter,
         CreateElasticsearchProductResult result)
     {
         var elasticsearchProductCustom = new ElasticsearchProductCustom(result.ElasticsearchProduct);
         var indexableProductCustom = parameter.IndexableProduct as IndexableProductCustom;
         elasticsearchProductCustom.MyCustomField = indexableProductCustom?.MyCustomField ?? 0;

         result.ElasticsearchProduct = elasticsearchProductCustom;

         return result;
     }
 }
}

Use the new field

Now that you have added the new field, you can use this data in the query pipelines however you see fit. This example uses the new field to influence the sort order of results by adding a pipe in the RunProductSearch pipeline. This pipe runs after the FormSortOrder pipe (Order 300) and modifies the SortOrderFields field on RunProductSearchResult when the user is looking at a category page in the default Relevance sort order.

namespace Extensions.Plugins.Search.Elasticsearch.DocumentTypes.Product.Query.Pipelines.Pipes.RunProductSearch
{
    using System;
    using Insite.Core.Interfaces.Data;
    using Insite.Core.Plugins.Pipelines;
    using Insite.Core.Plugins.Search;
    using Insite.Search.Elasticsearch.DocumentTypes.Product;
    using Insite.Search.Elasticsearch.DocumentTypes.Product.Query.Pipelines.Parameters;
    using Insite.Search.Elasticsearch.DocumentTypes.Product.Query.Pipelines.Results;

    public class FormCustomSortOrder : IPipe<RunProductSearchParameter, RunProductSearchResult>
    {
        public int Order => 310;

        public RunProductSearchResult Execute(
            IUnitOfWork unitOfWork,
            RunProductSearchParameter parameter,
            RunProductSearchResult result)
        {

            if (result.SortOrderFields == null)
            {
                return result;
            }

            if (parameter.ProductSearchParameter.SortBy != SortOrderType.Relevance ||
                parameter.ProductSearchParameter.SearchCriteria.IsNotBlank() || 
                parameter.ProductSearchParameter.SearchCriteria.IsNotBlank())
            {
                return result;
            }

            result.SortOrderFields = new[]
            {
                new SortOrderField(nameof(ElasticsearchProduct.SortOrder).ToCamelCase(), true, true),
                new SortOrderField(nameof(ElasticsearchProductCustom.MyCustomField).ToCamelCase(), true, true),
                new SortOrderField(nameof(ElasticsearchProduct.ShortDescriptionSort).ToCamelCase())
            };

            return result;
        }
    }
}

All available pipelines on the product query side are:

RunProductSearch – The main product query builder.FormProductFilterBuilds up filters for product search over category, language, attribute values, brands, price range, product line, restriction groups, and website. The generated filter is used by the other search pipelines.
RunProductFacetSearch – This pipeline performs faceted searches over brand, product line, and category.
RunBrandSearch – This pipeline searches for brands within the product index.FormRestrictionGroupFilter Builds the restriction group filter for FormProductFilter pipeline.

📘
Note
The Optimizely Dogfood sample repository in GitHub includes an example of adding a multi-value field to the index and using it to filter the products based on which warehouses have stock. It demonstrates how to add a new filter pipe to the FormProductFilter pipeline and how to pass a custom parameter from the external API into the search provider.

Customize search rebuild V1

🚧
Important

📘
Note

Add classes to the extension project

Define the pipes

Use the new field

📘
Note

🚧Important

📘Note

Add classes to the extension project

Define the pipes

Use the new field

📘Note

🚧
Important

📘
Note

📘
Note