Starting with FEP-7, WCS provided REST Services to interact with Solr Server directly without going through BOD / Component Services layer. This post explores the new search framework in detail and touches upon various customization points provided by the framework to implement customer requirements.
What happens if we want to return products within certain price range, belonging to certain category and want the result set to be sorted by price or relevance ? The query starts getting more and more complicated and difficult to manage.
Let us see how the Solr query looks in this case.
And then tell Solr to search in name and short description fields. Also products where searchTerm appears in name are more relevant than products where search term appears in shortDescription field.
As you can see Solr provides you a simple yet powerful query syntax to query the index and organize the results the way you want. You can set the fields to query, set the fields to retrieve, boost the results based on field names, add sorting, spell check, highlighting features to result set, get the stats about the result set, control the facets, filter query based on various conditions etc., quite easily.
Once the final Solr Query is built, it will be executed by SolrRESTSearchExpressionProcessor class. But who is responsible for building this Solr Query ? The entire query is NOT built by a single java class. Instead the query is split into separate logical parts like q, fq, fl, qf etc., and each part of the query is built by a separate java class known as Providers and PreProcessors. The parts of query are then assembled by SolrRESTSearchExpressionProcessor and executed against Solr Server. We will look at the roles of Providers and PreProcessors in detail in coming section.
A sample REST Search request from JSP
As part of the framework, new JSTL Tag wcf:rest
was introduced so that store front can easily invoke these REST
Services. (The tag also takes care of local binding / remote binding of
the services)
Below code snippet shows an example of how to use this tag to invoke productview/bySearchTerm GET REST service:
<wcf:rest var="catalogNavigationView" url="${searchHostNamePath}${searchContextPath}/store/${WCParam.storeId}/productview/bySearchTerm/*">
<wcf:param name="pageSize" value="${pageSize}" />
<wcf:param name="pageNumber" value="${pageNumber + 1}" />
<wcf:param name="profileName" value="IBM_findProductsBySearchTerm_Summary" /> <wcf:param name="searchType" value="${searchType}" />
<wcf:param name="searchTerm" value="${searchTerm}" />
<wcf:param name="langId" value="${langId}"/>
<wcf:param name="responseFormat" value="json"/>
<wcf:param name="catalogId" value="${WCParam.catalogId}"/>
<wcf:param name="categoryId" value="${currentCategoryId}" />
<wcf:param name="manufacturer" value="${newManufacturer}" />
<wcf:param name="minPrice" value="${WCParam.minPrice}" />
<wcf:param name="maxPrice" value="${WCParam.maxPrice}" />
</wcf:rest>
Note:
<wcbase:usebean> - To invoke data beans
<wcf:getData> - To invoke GET data component services
<wcf:rest> - To invoke REST services.
Required parameters to build the final Sol Query like searchTerm, pageNumber, pageSize, categoryId, minPrice, maxPrice etc., are passed to the REST Service using <wcf:param> tag.
These external query parameter names are converted into internal names based on the mapping defined in:
workspace\Search\xml\config\com.ibm.commerce.catalog\wc-component.xml
Sample Mapping:
<_config:valuemapping externalName="SearchControlParameterMapping" internalName="SearchControlParameterMapping">
<_config:valuemap externalValue="searchType" internalValue="_wcf.search.type" />
<_config:valuemap externalValue="categoryId" internalValue="_wcf.search.category" />
<_config:valuemap externalValue="catalogId" internalValue="_wcf.search.catalog" />
<_config:valuemap externalValue="storeId" internalValue="_wcf.search.store.online" />
<_config:valuemap externalValue="facet" internalValue="_wcf.search.facet" />
<_config:valuemap externalValue="facetLimit" internalValue="_wcf.search.facet.field.limit" />
<_config:valuemap externalValue="minPrice" internalValue="_wcf.search.price.minimum" />
<_config:valuemap externalValue="maxPrice" internalValue="_wcf.search.price.maximum" />
<_config:valuemap externalValue="searchTerm" internalValue="_wcf.search.term" />
<_config:valuemap externalValue="profileName" internalValue="_wcf.search.profile" /></_config:valuemapping>
A Selection Criteria object will be built using these internal names which will be accessible to all the downstream processors like provider, pre-processor and post-processor
Along with the request, we also pass a special parameter by name "profileName". wc-search.xml (inside workspace\Search\xml\config\com.ibm.commerce.catalog) defines various search profiles which controls how Solr queries are built, the meta-data for the query, the return fields etc.,
A sample profile will look like:
<_config:profile name="IBM_findProductsBySearchTerm_Summary" indexName="CatalogEntry">
<_config:query>
<_config:param name="maxRows" value="500" />
<_config:provider classname="com.ibm.solr.SolrRESTSearchBasedMerchandisingExpressionProvider"/>
<_config:provider classname="com.ibm.xxx.solr.SolrRESTSearchTermAssociationExpressionProvider"/>
<_config:preprocessor classname="com.ibm.xxx.solr.SolrSearchDebugQueryPreprocessor"/>
<_config:preprocessor classname="com.ibm.xxx.solr.SolrSearchResultFieldQueryPreprocessor"/>
<_config:field name="name"/>
<_config:field name="defaultSearch"/>
<_config:field name="categoryname"/>
<_config:field name="shortDescription"/>
<_config:postprocessor classname="com.ibm.xxx.solr.SolrRESTSearchPreviewQueryPostprocessor" />
<_config:postprocessor classname="com.ibm.solr.SolrRESTSearchExperimentQueryPostprocessor" />
</_config:query>
<_config:result>
<_config:field name="catentry_id"/>
<_config:field name="storeent_id"/>
<_config:field name="buyable"/>
<_config:field name="mfName_ntk"/>
<_config:field name="catenttype_id_ntk_cs"/>
<_config:field name="price_*"/>
<_config:field name="listprice_*"/>
<_config:field name="parentCatgroup_id_facet"/>
<_config:field name="childCatentry_id"/>
</_config:result>
</_config:profile>
- <_config:query> defines a set of providers and pre-processor. Each of these providers and pre-processor will contribute a part of Solr query. SolrRESTSearchExpressionProcessor will build the final query aggregating all the part queries built by these providers and pre-processor and executes it against the Solr index.
- <_config:param> is used to add name/value parameters to final Solr Query directly and it can be used to control how query is executed or fine tune the query further.
- <confg:provider> and <config:preprocessor> are used to define the providers and pre-processor used while building the query.
- <config:field> defines the index field names against which search should be performed. (qf fields)
- <config:result> defines the fields to be returned in the result set (fl fields)
- You can also use <_config:sort>, <_config:highlight>, <_config:spellCheck> and other configurations to fine tune the Solr Query.
Search framework will retrieve the profileName from query parameters and identifies the suitable profile defined in wc-search.xml. It then invokes all the providers and pre-processors + default providers and pre-processors defined for the profile, to build the Solr Query which will then be executed against the Solr Index. Once Solr returns back the result set, the search framework invokes the post-processors defined against the profile in a sequential order. Each of these post-processors will modify the result set returned by the Solr. ( Add / modify / remove data from the result set ).
Finally the result set will be converted into required responseFormat ( JSON / XML ) and returned back to JSP.
Customizing Solr Services
WCS
- Solr framework provides various touch points to customize the
existing services or create new services. Some of the customization
touch points which we will explore in detail are:
- JSP
- wc-search.xml
- Provider
- Pre-Processor
- Post-Processor
Customization using JSP
The first step of customization is JSP, where the actual Search REST request is made. You can pass various parameters to control how the search is performed. You can control the amount of data returned by Solr Server by using appropriate profileName. (Use xxx_Summary profiles when you want to retrieve lesser data).
Let us look at how some advanced customization scenarios can be achieved.
Scenario 1: By default MinimumMatch property is set to "1" in wc-search.xml. Which means, when a shopper searches for "red apple" all products which either contains a word "red" or "apple" will be returned. So shopper will see - apples, red dress, red potatoes, red handbags, red ginsing etc., in the result set.
Let us assume that you want to control this parameter value from JSP while doing the search and set it to different values based on some logic.
SolrSearchEDismaxQueryPreProcessor is the PreProcessor class which looks for
"_wcf.search.edismax.mm" parameter in the selection criteria object and uses the value to set "mm" parameter in the final Solr Query executed against Search Server. So you can pass this parameter from JSP using <wcf:param> tag.
<wcf:rest var="catalogNavigationView" url="${searchHostNamePath}${searchContextPath}/store/${storeId}/productview/bySearchTerm">
<wcf:param name="pageSize" value="${pageSize}" />
<wcf:param name="searchTerm" value="${term}" />
xxx
xxx
<wcf:param name="_wcf.search.edismax.mm" value="2"/>
</wcf:rest>
With this param set in request, when you search for "red apple", you will see that results contains only products which matches both the words - red and apple.
You can always update wc-component.xml to map this internal name "_wcf.search.edismax.mm" to a user friendly external name like "mm" and start using "mm" in the JSP file.
Scenario 2: Let us assume that you have defined an attribute "Clearance" and made it searchable and facetable using CMC tooling. Now when someone searches for a product, you would like to take advantage of this attribute
and return only those products which has this attribute.
To achieve this first identify the SearchFieldName ( index field name) for the "Clearance" attribute using below query:
select SRCHFIELDNAME from ATTRDICTSRCHCONF where attr_id in (select attr_id from attr where identifier = 'Clearance' and storeent_id = <storeId>);
Lets assume the above query returned SearchFieldName as: ads_f10101.
Search framework provides SolrRESTSearchByCustomExpressionProvider which allows you to add optional query ( OR constraint ), mandatory query ( AND constraint ) and Filter query ( fq ) part to the final Solr query. This custom provider looks for query parameter with name "_wcf.search.expr", "_wcf.search.mandatory.expr" and "_wcf.search.filter.expr". You can directly use these param names in JSP or create a mapping in wc-component.xml as below:
<_config:valuemap externalValue="minMatch" internalValue="_wcf.search.edismax.mm" />
<_config:valuemap externalValue="optionalQuery" internalValue="_wcf.search.expr" />
<_config:valuemap externalValue="mandatoryQuery" internalValue="_wcf.search.mandatory.expr" />
<_config:valuemap externalValue="filterQuery" internalValue="_wcf.search.filter.expr" />
With this mapping in place you can pass these additional parameters to <wcf:rest> tag in JSP.
// Return products which matches SearchTerm
AND which has this attribute set to Clearance.
<wcf:param name="mandatoryQuery" value="ads_f10101_ntk_cs:(\"Clerance\")"/>
OR
// Return products which matches search term OR the products which has this attribute set to Cleranace
<wcf:param name="optionalQuery" value="ads_f10101_ntk_cs:(\"Clerance\")"/>
OR
// Return products which matches search term, filter out the results to return products with attribute Clerarance.
<wcf:param name="filterQuery" value="ads_f10101_ntk_cs:(\"Clerance\")"/>
FilterQuery is similar to ManatoryQuery, but filter Query performs the filter on the returned result set and hence maintains the relevance score better.
So by making use of these additional parameters and appropriate search profiles you can control the modify the solr query to suit your needs.
Customization using wc-search.xml
If needed you can define your own custom profiles and use it in JSP pages while making REST Service calls.
Create custom profiles in wc-search.xml ( in workspace\Search\xml\config\com.ibm.commerce.catalog\ext folder ).
You can use existing providers, pre-processors, post-processors or create your own custom providers / pre-processors / post-processors and use it in the search profile. You can make use of various other aspects of wc-search.xml like <_config:field>, <_config:param>, <_config:sort> to customize the solr query.
Customization using custom Providers
Sometimes custom scenarios may be complex and you may not be able achieve required features just by using the existing profiles or providers. Passing in mandatoryQuery, optionalQuery or filterQuery may not be sufficient or relevant for some customization scenarios.
If the customization calls for modifying the Solr Query to add additional constraints or boost the result set o modify the query based on certain conditions, then you can always create a custom provider.
Let us look at the following REST service which retrieves product data byId.
http://localhost/search/resources/store/10001/productview/byIds?id=10034&id=10014&id=10906&profileName=IBM_findProductByIds_Summary
When you inspect the result set from this service, you will notice that the products are not in the same order in which the product ids were passed (10034, 10014, 10906). The order is completely random (probably based on the order in which the data was indexed).
Now if you would like to maintain the order of products, then one option is to use the boost query to boost the products as below, so that the sequence order is maintained and id1 is returned first and id3 is returned at the end.
bq=catentry_id:"id1"^10.0 catentry_id:"id2"^9.0 catentry_id:"id3"^8.0
Lets create a provider which will add this query part to the Selection Criteria object, so that it can be consumed by the final ExpressionProcessor while building the Solr Query.
 |
| Sample Provider Class |
Above listing shows a sample Provider Class which extends from AbstractSolrSearchExpressionProvider and implements SearchExpressionProvider interface.
It overrides invoke method and execute some custom logic to build the boost query string. getControlParameterValue gives you access to the input query parameters. Notice that we are using SearchServiceConstants.CTRL_PARAM_SEARCH_TERM as the key name while retrieving the product ids.
Once the boost search query is built, it is added back to the Selection Criteria object using addControlParameterValue method. The key used in this case is SearchServiceConstants.CTRL_PARAM_SEARCH_INTERNAL_BOOST_QUERY. This is the predefined name which is used by the Expression Processors to build the "bq" Solr Query.
Once the provider is built.. TODO
Customization using custom Pre-processor - TODO
Customization using custom Post-processor - TODO
