Search · Solr · Technical

Solr caching settings


Caching is always a burning and desirable topic in every application. Searching in Millions of records cost some time and performance though we can use caching to save this time again for same query. As a good product, Solr OOB provides caching options. Below are the details of different caching options and other caching configuration.

Version: 3.1.0
Solr Cache Class: solr.LRUCache

1. filterCache – The filter cache stores the results of any filter queries (“fq” parameters) that Solr is explicitly asked to execute. (Each filter is executed and cached separately. When it’s time to use them to limit the number of results returned by a query, this is done using set intersections.)

<filterCache
        size="512"
        initialSize="512"
        autowarmCount="128"/>

2. fieldValueCache – it supports multiple values per document (either multivalued fields or fields with multiple terms due to tokenization). If the fieldValueCache is not declared in solrconfig.xml then it is generated for you automatically with an initial size 10, a max size of 10000, and no autowarming.

<fieldValueCache
        size="10000"
        initialSize="10"
        autowarmCount="0"/>

3. queryResultCache – This cache stores ordered sets of document IDs — the top N results of a query ordered by some criteria. The memory usage for the queryResultCache is significantly less than that of the filterCache because it only stores document IDs that were returned to the user by the query.

<queryResultCache
        size="16384"
        initialSize="4096"
        autowarmCount="1024"/>

4. documentCache – The documentCache stores Lucene Document objects that have been fetched from disk.

<documentCache
        size="16384"
        initialSize="16384"/>

4.1 enableLazyFieldLoading=true – Document objects fetched from the IndexReader will only contain the Fields specified in the fl. All other Fields will be marked as “LOAD_LAZY”. Use this setting if schema attributes are many.

5. newSearcher and firstSearcher Event Listeners – A firstSearcher event is fired whenever a new searcher is being prepared but there is no current registered searcher to handle requests or to gain autowarming data from (ie: on Solr startup). A newSearcher event is fired whenever a new searcher is being prepared and there is a current searcher handling requests (aka registered).
NOTE: We need to identify common query request we want to autowarm when server starts or new searcher is created

<listener event="newSearcher">
    <arr name="queries">
        <!-- seed common sort fields -->
        <lst>
            <str name="q">anything</str>
            <str name="sort">name desc price desc populartiy desc</str>
        </lst>
    </arr>
</listener>
<listener event="firstSearcher">
    <arr name="queries">
        <!-- seed common sort fields -->
        <lst>
            <str name="q">anything</str>
            <str name="sort">name desc, price desc, populartiy desc</str>
        </lst>
    </arr>
</listener>

6. queryResultWindowSize – An optimization for use with the queryResultCache. When a search is requested, a superset of the requested number of document ids are collected. For example, of a search for a particular query requests matching documents 10 through 19, and queryWindowSize is 50, then documents 0 through 50 will be collected and cached. Any further requests in that range can be satisfied via the cache.

<queryResultWindowSize>50</queryResultWindowSize>

7. HashDocSet – This entry enables an int hash representation for filters (DocSets) when the number of items in the set is less than maxSize. For smaller sets, this representation is more memory efficient, more efficient to iterate over, and faster to take intersections.

<HashDocSet maxSize="3000" loadFactor="0.75"/>
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s