Elasticsearch

🚧

IMPORTANT

Optimizely Configured CommerceTM 4.4 upgraded to Elasticsearch 5.5 and Nest version 5. Any code from previous versions will require updates. See upgrading to 4.4.

Given the complexity around B2B digital commerce it is common to extend the document types within Configured Commerce and Elasticsearch. The product types are the only supported extension types.

Elasticsearch extensions

ElasticsearchProduct

This is the base class located in the Insite.Search.Elasticsearch.DocumentTypes.Product namespace. This class represents the document type within the Elasticsearch index. Represented as JSON objects returned from the REST API as a collection, the ElasticsearchProduct is the class that the NEST API maps the result.

To extend the product document type, or schema, derive a new class using the ElasticsearchProduct as the base class. Decorate the class with the ElasticType attribute located in the Nest.ElastictypeAttribute namespace. Add new properties to the derived class to expand the document type for products that are indexed.

For example, the following extends the definition of a product document to include a closeout property:

[ElasticType(Name = "product")] 
public class ElasticsearchProduct_Custom : ElasticsearchProduct 
{      /// Gets or sets the Closeout product value.      public int Closeout { get; set; } 
}

ElasticsearchProductFactory

The ElasticsearchProductFactory maps the IndexableProduct to ElasticsearchProduct. This factory is responsible for mapping the result of the SQL query and maps it into the format that is loaded into Elasticsearch. For example, the following maps the additional properties to the derived Elasticsearch product document type:

[DependencyOrder(1)] 
public class ElasticsearchProductFactory_Custom : 
ElasticsearchProductFactoryBase<IndexableProduct_Custom, ElasticsearchProduct_Custom>
{      /// Initializes a new instance of the <see cref="ElasticsearchProductFactory_Custom" /> class.
  ///The boost helper.      ///The unit of work factory.      public ElasticsearchProductFactory_Custom(IBoostHelper boostHelper, IUnitOfWorkFactory unitOfWorkFactory)            : base(boostHelper, unitOfWorkFactory)
  {
  } 
  /// The map additional properties.      ///The indexable product.      ///The elasticsearch product.      /// The <see cref="ElasticsearchProduct_Custom" />.      protected override ElasticsearchProduct_Custom 
  MapAdditionalProperties(IndexableProduct_Custom indexableProduct, 
  ElasticsearchProduct_Custom elasticsearchProduct)      {
    elasticsearchProduct.Closeout = indexableProduct.Closeout;
    return elasticsearchProduct;
  }
} 

ElasticsearchProductMapping

The ElasticsearchProductMapping class is responsible for extending the product document type in Elasticsearch. As described earlier, Elasticsearch mapping is comparable to extending the data schema for the product types. The Get Mapping method can be used to retrieve the default schema, or mapping, for the product document type in Elasticsearch. The RootObjectMapping from the base class can be used to extend the product document type by adding a new mapping to the properties collection.

/// <summary>  
/// This class overrides the default ElasticsearchProductMapping. With this you can  
/// change Elasticsearch mappings for the standard ElasticsearchProduct /// properties, or new custom properties.  
/// </summary>  
[DependencyOrder(1)]  
[DependencyName("ElasticsearchProductMapping")]  
public class ElasticsearchProductMapping_Custom : ElasticsearchProductMapping  
  {      /// <summary>The type of the class you are creating a mapping for.</summary>  
  public override Type MappingType { get; } = typeof(ElasticsearchProduct_Custom);  
  /// <summary>The get mapping.</summary>      /// <returns>The <see cref="RootObjectMapping"/>.</returns>      public override RootObjectMapping GetMapping()      {            var mapping = base.GetMapping();     mapping.Properties.Add(nameof(ElasticsearchProduct_Custom.Closeout).ToCamelCase(), new  
    StringMapping { Index = FieldIndexOption.Analyzed });     return mapping;      }  
}

IndexableProduct The IndexableProduct class is the object that the SQL query is bound to. This class can also be perceived as a DTO (data transfer object).

For example, the following class describes the objects that are returned from the SQL query, including the Closeout custom property.

/// <summary> 
/// This is the object that the SQL query gets bound to. If you add custom fields to the 
/// SQL query (See IndexableProductRepository_Custom.cs) you will also 
/// need to add the corresponding properties to this class. 
/// </summary>
public class IndexableProduct_Custom : IndexableProduct 
{      /// <summary>Gets or sets the Closeout product value.</summary>      public int Closeout { get; set; } 
} 

IndexableProductRepository

The IndexableProductRepository is primarily responsible for populating the IndexableProduct with values from the database. When extending the ElasticsearchProduct and IndexableProduct classes, it is necessary to also derive a new class from IndexableProductRepository and override the AllProductQueryCustomFields property with the SQL query. This query is then appended to the full SQL query used to populate the IndexableProduct object.

For example, the following query uses custom properties that have been extended in the Application Dictionary of the Admin Console. The closeout property value of each product is retrieved.

[DependencyOrder(1)] 
public class IndexableProductRepository_Custom : IndexableProductRepository 
{      public IndexableProductRepository_Custom(IUnitOfWorkFactory unitOfWorkFactory)            : base(unitOfWorkFactory)    {
  }
         /// <summary>The all product query custom fields.</summary>    protected override string AllProductQueryCustomFields => @"
  CONVERT(INT, ISNULL((
    SELECT pp.Value
    FROM CustomProperty pp WITH (NOLOCK)
    WHERE pp.ParentId = p.Id AND pp.Name = 'Closeout'
  ), '0')) 
  AS Closeout";
} 

ProductSearchIndexerElasticsearch

The ProductSearchIndexerElasticsearch is responsible for indexing the indexable products into the Elasticsearch index. This will map the DTO objects of IndexableProduct to ElasticsearchProduct which resolves to the document type stored in the Elasticsearch index.

When extending the mapping of a product in Elasticsearch it is also required to derive a new ProductSearchIndexerElasticsearch class which overrides the BuildProductSearchIndex method. Within this method it should map the derived IndexableProduct class to the derived ElasticsearchProduct class using the IndexProducts method located in the base class.

This class also needs to be decorated with the same DependencyApplicationSetting as the ProductSearchProviderElasticsearch.

For example, the following will map the appropriate derived classes to leverage the Closeout custom property.

[DependencyApplicationSetting("ProductSearchProvider", "ElasticsearchCustom")]  
public class ProductSearchIndexerElasticsearch_Custom : ProductSearchIndexerElasticsearch  
{  
  /// <summary>Initializes a new instance of the <see cref="ProductSearchIndexerElasticsearch_Custom"/> class.</summary>      /// <param name="index">The index.</param>     /// <param name="unitOfWorkFactory">The unit of work factory.</param>     /// <param name="indexableProductRepository">The indexable product repository.</param>      public ProductSearchIndexerElasticsearch_Custom(IElasticsearchIndex index,  
    IUnitOfWorkFactory unitOfWorkFactory, IIndexableProductRepository  
    indexableProductRepository)            : base(index, unitOfWorkFactory, indexableProductRepository)      {  
  }  
   /// <summary>The build product search index.</summary>      /// <param name="indexName">The index name.</param>      /// <param name="incremental">The incremental.</param>      /// <param name="logMessage">The log message.</param>      public override void BuildProductSearchIndex(string indexName = null, bool  
  incremental = false, Action<string> logMessage = null)      {            indexName = indexName ?? this.Index.Alias;     try            {                  this.IndexProducts\<IndexableProduct_Custom,  
      ElasticsearchProduct_Custom>(indexName, logMessage, incremental);           }  
    catch (Exception ex)  
    {                  logMessage?.Invoke($"Product search indexer exception: {ex.Message}  
      {ex.StackTrace}");                  throw;            }      }  
}

📘

Note

The code snippet above relies on the Application Setting called ProductSearchProvider to be set with the value of ElasticsearchCustom in order for the code to run.

ProductSearchProviderElasticsearch ProductSearchProviderElasticsearch is the core class used as the query mechanism for search. The DependencyApplicationSetting class attribute value is used to configure the search provider by name. The name of the default search provider is called Elasticsearch. The application setting used for the search provider is called ProductSearchProvider.

This class is responsible for sorting and filtering the results of the product set returned by a query. This includes conditional queries that include customer product restrictions, products by website, language, categories, and more. Ultimately the result is to limit which products are searched.

Methods for Elasticsearch

The following are overridable methods for the search provider.

SortOptions

The SortOptions property is used to filter the query in a particular order. This includes by relevance, alphabetically, or by price. Override this property to add custom sort by values in the product list drop down list.

GetSponsoredResults

Gets the sponsored product results. This returns an ISearchResponse of the generic type of ElasticsearchProduct.

SetSuggestions

Transfers suggestions in search response to service result.

ConvertAggregationToPriceRangeFacets

Converts aggregation to price range facets.

ConvertAggregationToCategoryFacets

Convert Elasticsearch aggregations to category facet collection.

ConvertAggregationToAttributeTypeFacets

Convert Elasticsearch aggregations to AttributeType facet collection.

GetAggregationHits

Extracts a list of facets from an Elasticsearch aggregation.

AddSortOrder

Adds the sort order to a nest SearchDescriptor. This method uses the SortOrder array and for each sort order sorts in ascending based on the "_score" unless the Reverse property of the sort order is set to true. Otherwise the search descriptor is sorted using the search field.

AddAggregations

Adds aggregation requests to a search request.

MakeCustomFilter

Used to filter the results with a custom query which must match.

GetExactMatchFields

Used to modify the document fields which should be searched by MakeMultiMatchQuery.

GetPrefixMatchFields

Used to modify the document fields which should be searched by MakeMultiMatchPrefixQuery

GetPhraseMatchFields

Used to modify the document fields which should be searched by MakeMultiMatchPhraseQuery.

GetFuzzyMatchFields

Used to modify the document fields which should be searched by MakeMultiMatchFuzzyQuery.

MakeFilterQuery

Creates a query on the filtered, AND attributes and OR attribute values within each section.

MakeCustomerProductClause

Used to filter products, and hide, that are restricted by the Customer Product Include clause.

MakeWebsiteClause

Used filter products in the search query by website.

MakeLanguageClause

Used to filter products in the search query by language.

MakeRestrictionGroupClause

Used to filter products in the search query by restriction group.

MakeRestrictionGroupFilters

Tests for ProductGroup/ProductSet constraints for the current customer and shipto

MakeCategoryClause

Used to filter products in the search query by category.

MakePriceRangeFilter

Make a query over a range of prices that works on fields of type NumericField

MakeCustomSortOrder

Use to provide custom sorting rules.

GetSortOrder

Returns a search order priority based on an input sort order.

For example, the following derived version of the ProductSearchProviderElasticsearch class is used to create a custom sort order on the product list page. This sorts by closeout products.

[DependencyApplicationSetting("ProductSearchProvider", "ElasticsearchCustom")] 
public class ProductSearchProviderElasticSearch_Custom : 
  Insite.Search.Elasticsearch.DocumentTypes.Product.Query.ProductSearchProviderElasticsearch
{      public ProductSearchProviderElasticSearch_Custom(            IElasticsearchIndex index,                 ICacheManager cacheManager,                ICatalogCacheKeyProvider catalogCacheKeyProvider,                  IPerRequestCacheManager perRequestCacheManager,            IUnitOfWorkFactory unitOfWorkFactory,            IProductSearchFacetProcessor facetProcessor,            IElasticsearchQueryBuilder queryBuilder,            IPhraseSuggestConfiguration phraseSuggestConfiguration,            IBoostHelper boostHelper,            IApplicationSettingProvider applicationSettingProvider) : base(index,              cacheManager,              catalogCacheKeyProvider,              perRequestCacheManager,              unitOfWorkFactory,              facetProcessor,              queryBuilder,              phraseSuggestConfiguration,              boostHelper,              applicationSettingProvider)      {            base.SortOptions.Add(new SortOrderDto() { DisplayName = "Closeout", SearchOnly = 
    false, SortType = "Closeout" });      }      protected override SortOrderField[] MakeCustomSortOrder(string sortBy, 
    IProductSearchParameter parameter, out bool manualSort)      {            manualSort = false; // for price sorting
    // sortBy string is from the sort options SortType property
    if (sortBy == "Closeout")            {                 // secondary sort                  // determine the order of items added to searchLookup cannot be sorted on. Only indexed.                  // name of the sort order field is the custom property called Closeout. Name of whatever we added to the index as. 
      // if numeric field then it is the type of field.                  // if numeric it will default to 0,1 and then when reverse is set it is saying that Closeout is at the top because 1 is true.                  // thus rendering 1,0
      return new SortOrderField[]                              {
        new SortOrderField("Closeout",false, true),                                    new SortOrderField(SortOrderField.Score, true, true)                              };
    }     return null;
  }
} 

📘

Note

The code snippet above relies on the Application Setting called ProductSearchProvider to be set with the value of ElasticsearchCustom in order for the code to run.

The Elasticsearch Index objects provide access to the search index using the NEST IElasticClient interface.