Looks like interest to Sitecore implementation of Lucene index has raised since Dream Core event and developers have run into an issue with old data being kept in the index repository. In this article I want to show you how to go around this issue.
First of all let’s see why it’s happening. I ran into this issue when I started playing with new implementation of Lucene index in Sitecore 6. When I created an output of the results I saw duplicates of my data in there. I stated debugging my code and found that Lucene somehow recognizes raw GUID’s which breaks search criteria that Sitecore uses to find items during update/delete procedure.
To solve this issue I had to create additional field for Lucene index (_shorttemplateid) and store there short GUID for an item (item.ID.ToShortID()). Then override AddMatchCriteria method and dependent properties to use short template GUID for matching criteria. Below is the code example.

This should solve this issue as well as add Lucene recognizable format for Sitecore date and datetime field types. Also it will allow to build Combined and Boolean search queries.

Update. Code for the Constants class:

   1: namespace LuceneExamples

   2: {

   3:    public class Constants

   4:    {

   5:       // special field for template id in ShortID format

   6:       public const string ShortTemplate = "_shorttemplateid";

   7:  

   8:       // searchable date-time format. All datetime field

   9:       public const string DateTimeFormat = "yyyyMMddHHmmss";

  10:  

  11:       // Path to lucene setting items: /sitecore/system/Settings/Lucene

  12:       public const string LuceneSettingsPath = "{89783047-026C-45B5-AB5B-338E4A22446C}";

  13:    }

  14: }

Hope it saves someone a minute or two.

Sitecore Gadgets

© admin for Sitecore Fetch Squad, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Feed enhanced by Better Feed from Ozh

Previously I published an article explaining how to fix a glitch in Sitecore index that keeps old data after updating an item. In this article I updated the code since some issues were fixed in Sitecore 6.2 (Update-4) and 6.3.

Starting from 6.2 (Update-4) and 6.3 the _template field in Lucene index stores an item ID in ShortID format. Now it’s even easier to customize the DatabaseCrawler as the following methods were made virtual: AddItem, IndexVersion, DeleteItem and DeleteVersion. Also fields _hasIncludes, _hasExludes and _templateFilder were made protected which helps to shrink the fix even more.

Here is the code that fixes the outlined problem for these releases:

Basically you need to make sure that the result of GetItemID method is in lower case as well as a term for _path field, in AddMatchCriteria method, is constructed with lower case query.

Sitecore Gadgets

© admin for Sitecore Fetch Squad, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Feed enhanced by Better Feed from Ozh

In this post I want to show how to address a missing feature that was a part of “old” lucene index implementation. This article will provide an example how one can customize Lucene search configuration so that it’s possible to add custom fields to the index.

First off, let’s create a configuration that would allow us to add additional fields to the indexed data.

<index id="News" type="Sitecore.Search.Index, Sitecore.Kernel">  <param desc="name">$(id)</param>  <param desc="folder">_news</param>  <Analyzer ref="search/analyzer" />  <locations hint="list:AddCrawler">    <examples-news type="LuceneExamples.DatabaseCrawler,LuceneExamples">      <Database>web</Database>      <Root>/sitecore/content</Root>      <IndexAllFields>true</IndexAllFields>      <include hint="list:IncludeTemplate">        <news>{788EF1BE-B71E-4D59-9276-50519BD4F641}</news>        <tag>{4DD970FB-2695-4E50-96F3-A766F7D6CAF1}</tag>      </include>      <fields hint="raw:AddCustomField">        <field luceneName="author" storageType="no" indexType="tokenized">__updated by</field>        <field luceneName="changed" storageType="yes" indexType="untokenized">__updated</field>      </fields>    </examples-news>  </locations></index>

There is a new configuration section in this example. It’s <fields> section that introduces two fields “author” and “changed”. These fields will be added to a fields collection of each indexed item. Basically, there is AddCustomField method that gets called for every <field> configuration entry to identify a custom field that is going to be added to the fields collection.

Description of configuration attributes:

• luceneName  is a field name that appears in lucene index.
• storageType  is a storage type for lucene field. It can have the following values:
  • no
  • yes
  • compress
• indexType  is an index type for lucene field. It can have the following values:
  • no
  • tokenized
  • untokenized
  • nonorms

Refere to Lucene documentation to find out what each of these options mean: store and index.

Now all you need to do is to loop through the collection of custom fields in the overridden AddAllFields method and add them to the indexed data.

I created a custom class called CustomField that helps to manage custom field entries. Below is the example of this class as well as additional methods for extended DatabaseCrawler. Since code for the DatabaseCrawler was already published in this blog post, I’m not going to duplicate it here.

Here is a code for CustomField class.

using System.Xml;using Sitecore.Data;using Sitecore.Data.Items;using Sitecore.Xml;using Lucene.Net.Documents;

namespace LuceneExamples{   public class CustomField   {      public CustomField()      {         FieldID = ID.Null;         FieldName = "";         LuceneFieldName = "";      }

      public ID FieldID      {         get;         private set;      }

      public string FieldName { get; private set; }

      public Field.Store StorageType { get; set; }

      public Field.Index IndexType { get; set; }

      public string LuceneFieldName { get; private set; }

      public static CustomField ParseConfigNode(XmlNode node)      {         CustomField field = new CustomField();         string fieldName = XmlUtil.GetValue(node);         if (ID.IsID(fieldName))         {            field.FieldID = ID.Parse(fieldName);         }         else         {            field.FieldName = fieldName;         }         field.LuceneFieldName = XmlUtil.GetAttribute("luceneName", node);         field.StorageType = GetStorageType(node);         field.IndexType = GetIndexType(node);

         if (!IsValidField(field))         {            return null;         }

         return field;      }

      public string GetFieldValue(Item item)      {         if (!ID.IsNullOrEmpty(FieldID))         {            return item[ID.Parse(FieldID)];         }         if(!string.IsNullOrEmpty(FieldName))         {            return item[FieldName];         }         return string.Empty;      }

      private static bool IsValidField(CustomField field)      {         if ((!string.IsNullOrEmpty(field.FieldName) || !ID.IsNullOrEmpty(field.FieldID)) && !string.IsNullOrEmpty(field.LuceneFieldName))         {            return true;         }         return false;      }

      private static Field.Index GetIndexType(XmlNode node)      {         string indexType = XmlUtil.GetAttribute("indexType", node);         if (!string.IsNullOrEmpty(indexType))         {            switch (indexType.ToLowerInvariant())            {               case "no":                  return Field.Index.NO;               case "tokenized":                  return Field.Index.TOKENIZED;               case "untokenized":                  return Field.Index.UN_TOKENIZED;               case "nonorms":                  return Field.Index.NO_NORMS;            }         }         return Field.Index.TOKENIZED;      }

      private static Field.Store GetStorageType(XmlNode node)      {         string storage = XmlUtil.GetAttribute("storageType", node);         if (!string.IsNullOrEmpty(storage))         {            switch (storage.ToLowerInvariant())            {               case "no":                  return Field.Store.NO;               case "yes":                  return Field.Store.YES;               case "compress":                  return Field.Store.COMPRESS;            }         }         return Field.Store.NO;      }   }}

And the code for additional methods for DatabaseCrawler.

/// <summary>/// Loops through the collection of custom fields and adds them to fields collection of each indexed item./// </summary>/// <param name="document">Lucene document</param>/// <param name="item">Sitecore data item</param>private void AddCustomFields(Document document, Item item){   foreach(CustomField field in _customFields)   {      document.Add(CreateField(field.LuceneFieldName, field.GetFieldValue(item), field.StorageType, field.IndexType, Boost));   }}

/// <summary>/// Creates a Lucene field./// </summary>/// <param name="fieldKey">Field name</param>/// <param name="fieldValue">Field value</param>/// <param name="storeType">Storage option</param>/// <param name="indexType">Index type</param>/// <param name="boost">Boosting parameter</param>/// <returns></returns>private Fieldable CreateField(string fieldKey, string fieldValue, Field.Store storeType, Field.Index indexType, float boost){   Field field = new Field(fieldKey, fieldValue, storeType, indexType);   field.SetBoost(boost);   return field;}

/// <summary>/// Parses a configuration entry for a custom field and adds it to a collection of custom fields./// </summary>/// <param name="node">Configuration entry</param>public void AddCustomField(XmlNode node){   CustomField field = CustomField.ParseConfigNode(node);   if (field == null)   {      throw new InvalidOperationException("Could not parse custom field entry: " + node.OuterXml);   }   _customFields.Add(field);}

Last thing that is left to do is to call AddCustomFields method from AddAllFields one.

protected override void AddAllFields (Documentdocument, Itemitem, bool versionSpecific)
{
    ………………………………………
    AddCustomFields(document, item);
}

You can take it even further and add support for some field interpreter for each field configuration entry.

Hope you’ll find it useful.

Sitecore Gadgets

© admin for Sitecore Fetch Squad, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Feed enhanced by Better Feed from Ozh

Recently I happened to help one of our clients to create a custom Multilist field that gives you selecting options only if item has  at least one translated version in a current content language. From content author’s point of view it makes a lot of sense. Why to give them an option that is not useful?

It’s being a while since I created a custom field that has to take into account content language selection. The main challenge for me was to retrieve that content language. I did remember that there is a property that the Content Editor (CE) sets at run-time for every field but could not recall its name. So after several minutes of browsing my code storage of all samples for all Sitecore versions, I finally found it. So, here are the properties that you need to define in your custom field if you are planning to use them afterwards:

- ItemLanauage – represents a content language selected in the CE.

- ItemID – contains the item ID the field belongs to.

- ItemVersion – contains selected item version.

- ReadOnly – indicates if field is a readonly. If it is, then it will be grayed out and not editable.

- Source – represents the source value from field definition on a data template.

- FieldID – contains the field ID.

In my example I needed only ItemLanguage property. When I put things together the code looked like this:

using Sitecore.Shell.Applications.ContentEditor;using Sitecore.Data.Items;

namespace Sitecore.Shell.Applications.ContentEditor.CustomExtensions{    public class LanguageFilteredMultilist : Sitecore.Shell.Applications.ContentEditor.MultilistEx    {        #region Overrides

        protected override Item[] GetItems(Item current)        {                          Item[] items = base.GetItems(current);

            var filteredItems = items.Where(item =>                     {                        string lang = ItemLanguage;                        if (!string.IsNullOrEmpty(lang))                        {                            var versions = Sitecore.Data.Managers.ItemManager.GetVersions(item, Sitecore.Data.Managers.LanguageManager.GetLanguage(lang, item.Database));                            return versions.Count > 0;                        }                        return false;                    }                );   

            return filteredItems.ToArray<Item>();        }

        #endregion Overrides

        // Content Editor sets this property. It has a content language from the Content Editor.        public string ItemLanguage        {            get;            set;        }    }}

Simple enough isn’t it. Don’t forget to add your custom field to /App_Config/FieldTypes.config file if it’s supposed to contain references to other items. In my case it should be added since it’s a multilist. If you forget to do it, the LinkDatabase won’t update references for your field.

That’s all for now.

Sitecore Gadgets

© admin for Sitecore Fetch Squad, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Feed enhanced by Better Feed from Ozh

For those who consider this so easy so that it’s now worth to create a VS project I’ve built a ready-to-go Sitecore package with a dll and include file inside. Just install it and give it a try .

Sitecore Gadgets

© admin for Sitecore Fetch Squad, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Feed enhanced by Better Feed from Ozh

Sitecore Gadgets

© admin for Sitecore Fetch Squad, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Feed enhanced by Better Feed from Ozh

Sitecore Gadgets

© admin for Sitecore Fetch Squad, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Feed enhanced by Better Feed from Ozh

Here are possible ways to do it:

• Sitecore API
• Sitecore Query
• SQL custom stored procedures
• Lucene index

Let’s take a look at each option briefly.

Sitecore API – the most popular way to get an item. All you need is an item ID and simple call of GetItem() method of database instance will get you the latest item version in current language. If you want an item in specific language, not a problem: GetItem(, ). Want specify version, not a problem either: GetItem(, , ). There are many ways to get an item through Sitecore API. My point is that it’s the most easiest way to do it.

Sitecore Query – easy way to get a bunch of items filtered by some criteria. Build a string query, which kinda look like XPath query, and run it on a database instance. Read more about Sitecore Query.

SQL custom stored procedures – we all know how to create a SQL stored procedure. Connections strings already exist in Sitecore solution. All you need is to use some SQL management studio to create a stored procedure for the database. The question is why would you do it if API already exists? It makes sense only if you run complex query with lots of filtering conditions so that SQL will return you only items that you’re searching for.

Lucene index – why would I use separate data storage/data index to get an item from Sitecore database? It does not make sense. Agree, for an item it does not but for a collection of items it has a perfect sense to do it. Why? Maybe because performance is much better when you are working with big amount of data. Moreover that’s a search index. Which means that data are perfectly organized to be searched.

Let’s talk about performance characteristics for each of these options and compare them.

Sitecore API – uses Sitecore data provider to pull out information from the database. If item that we’re looking for is cached the call to SQL database is avoided and item gets retrieved from one of Sitecore caches. Using Sitecore caches gives you huge performance benefit. That’s why we always worried about caches configuration in Sitecore solution. What if you need to get a number of items by some criteria, let’s say template ID and belonging to some category (from here on I’m gonna use this condition for all item retrieval options). You run foreach or for or any other logic that goes through the content tree and checks for a TemplateID property and category field of every single item. If item resides in cache, it will be quick enough but still the code will have to request every single item from the content tree.

Sitecore Query – the picture with Sitecore Query is very similar to Sitecore API but it get’s more slower when number of items is growing. Kim has a good article that explains Sitecore Query performance.

NOTE: I’m not talking about fast Sitecore Query introduced in Sitecore 6.

SQL stored procedures – it seems to be a good approach to go with if you’re searching for items through the content tree. The thing is there will be only one stored procedure that executes query by specified criteria. Also you will have to consider caching option for retrieved items if you have very deep content tree and items of some specific time are not gathered under one branch. My point is that it can cost you more then the benefits that you will get. I would go with this approach only if I cannot do it with Lucene resources.

Lucene index – again it’s search index. It perfectly fits searching options. When you search for items with specified criteria it neither requests an item instance nor runs query over database tables. It goes through the fields that you have in your index and compares their values to your search conditions. The only thing if search query uses custom fields to filter data by, the fields must be added to your index. Otherwise you will get zero results. It’s very easy to do though. I will describe it in the next part of the article.

Conclusion: Let’s answer those questions in the topic of this part.

Why to use lucene index?

The approach is one of the most (maybe event the best) efficient ones. Search is quick enough that in most cases you don’t even need to implement custom caching for the retrieved data.

When to use lucene index?

When you need to run search queries with specific criteria on huge number of data.

Don’t forget the rule – the more complex search query gets the slower it works .

In next part we will look into Lucene search index introduced in Sitecore 6.

Information about “old” Lucene index (it was introduced in Sitecore 5) you can find here.

Sitecore Gadgets

© admin for Sitecore Fetch Squad, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Feed enhanced by Better Feed from Ozh

Download the package.

Sitecore Gadgets

© admin for Sitecore Fetch Squad, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Feed enhanced by Better Feed from Ozh

• Check our cache configuration reference
• The following shared source modules should be useful in this exercise:
  http://trac.sitecore.net/CachingManager
  http://trac.sitecore.net/CacheTuner

2. Ensure client caching is properly configured (MediaResponse.XXX settings).
3. Media Library structure – make sure the tree is balanced.
4. Change MediaLinkPrefix from ~/media to something else, for example “-/media”:

- set the value of the “Media.MediaLinkPrefix” to alternative prefix:

<setting name="Media.MediaLinkPrefix" value="-/media" /> 

-add the following line to the customHandlers section:

<customHandlers>
   <handler trigger="-/media/" handler="sitecore_media.ashx" />
   <handler trigger="~/media/" handler="sitecore_api.ashx" />
   …
</customHandlers>

This really helps for high traffic websites. Apparently tilde in the path forces different way of resolving the system permissions on a resource on Windows Server.

6. Consider adding other edge caching options in front of the web servers.
7. Leverage CDN like AKAMAI.

Some links and food for thought:

• http://sdn.sitecore.net/Scrapbook/Changing%20media%20URLs%20in%20Sitecore%206.aspx
• http://www.sitecoredevelopment.com/en/MarkGraber/Using-Akamai-Net-Storage-for-Sitecore-Media-Items.aspx
• http://sdn.sitecore.net/Forum/ShowPost.aspx?postid=18768
• http://sdn.sitecore.net/SDN5/Forum/ShowPost.aspx?postid=27646
• http://sdn.sitecore.net/SDN5/Forum/ShowPost.aspx?postid=15132
• http://sdn.sitecore.net/SDN5/Forum/ShowPost.aspx?postid=26647

8. Use dedicated “Sitecore media server” (e.g. replace hostname in media links to point to another Sitecore instance; would require some customization).

Some great stuff here:

• http://mcore.wordpress.com/2009/11/02/dedicated-image-server-and-sitecore/
• http://mcore.wordpress.com/2011/01/02/dedicated-image-server-in-sitecore-part-2/

Thanks to the guys from our brilliant customer service for helping me to come up with a more complete list.

Consulting and Supporting Sitecore Developer Community

© admin for Sitecore Fetch Squad, 2011. | Permalink | No comment | Add to del.icio.us
Post tags:

Feed enhanced by Better Feed from Ozh