« Amazon Web Services | Main | Movable Type Resource Sites »

How to Make a Subject Index for Your Movable Type Blog

With MT3.3's new tagging feature, you no longer need all the plugins and method described here to make a subject index. If you are running MT3.3, see How to Make a Subject Index Using Tags. ~Elise, August 21, 2006

This tutorial is written by LMT contributor Mike Everett-Lane of Ishbadiddle.

A Subject Index can give context to your posts and makes it easier for your readers to browse what you've written on specific topics. Unlike Categories, which are limited, top-down, and hierarchical, Subject tags are open-ended and limitless. While your blog's Category system is like the Table of Contents of a book, a Subject Index is like the book's Index, one that is constantly updated.

First, you might want to read my blog post on how the use of Subject Indexes can improve the organization of your blog. You can see it in action there as well as here on LMT. For instance, here's the index of all my subjects, and the index of all my posts on the subject of "Movable Type".

Basically, what this method does is link your blog posts together by keyword. (In this setup, "keyword," "subject" and "tag" all mean pretty much the same thing.) Since this kind of index isn't native to MT, to create it we'll be hacking the MT-Search function and creating a special search template.

Here's how to do it:

1. Code your posts.

Use the "Keywords" field to enter your tags. (If you don't see it, you'll have to select "Customize display of this page" on your New Entry page.) This system assumes that your keywords are formatted thusly:

Penguins, George W. Bush, New York City

Each keyword is separated by a comma and a space. Capitalization matters (I generally use Sentence Case in all my keywords). Anything can go into a keyword, although ampersands make it go kind of wonky. You can have as many keywords as you want for each post.

Here are some tips for successful "tagging":

A. Be specific.

Ambiguous keywords only muddy the waters. Try to be as specific as possible. If you've tagged one post with "The Beatles," avoid tagging others with "Beatles". Use "George W. Bush" to differentiate him from his father. Check your complete subject index to see if a keyword's already being used.

B. Be redundant.

Not every Matrix-related post will also have Science Fiction as a keyword. But if at least one does, then the two subjects are related. (Look over on the sidebar of one of my subject indexes and you'll see the related subjects.) With an open-ended scheme, we will end up with subjects that are synonyms, or subjects (Animals) that contain others (Cats.) Redundant keywording ensures that readers are only two jumps away from related posts on any given topic.

C. Go back in time

The more of your archives you've got tagged, the more useful your subject index will be. I went back and created keywords for all the posts in my archives. (Yes, it took a while, but it's worth it!)

OK, so you've started adding your keywords. On to the code!

2. Get the Plugins

You'll need the following plugins: IfEmpty, MTLoop, Compare, and MTCollate. Follow the instructions to install the plugins.

3. Make a new keyword-only alternative search template

Next, you'll need to modify your search function so that you can limit the search to the keyword field. Follow the instructions for the hack on Stepan Riha's site to change the lib/MT/App/Search.pm file, which I've reproduced here:

A. Open the file Search.pm in the lib/MT/App directory. B. Find the section that says:

sub _search_hit {
    my($app, $entry) = @_;
    my @text_elements;
    if ($app->{searchparam}{SearchElement} ne 'comments') {
        @text_elements = ($entry->title, $entry->text, $entry->text_more,
                          $entry->keywords);
    }
    if ($app->{searchparam}{SearchElement} ne 'entries') {
        my $comments = $entry->comments;
        for my $comment (@$comments) {
            push @text_elements, $comment->text, $comment->author,
                                 $comment->url;
        }
    }
    return 1 if $app->is_a_match(join("\n", map $_ || '', @text_elements));
}

C. Replace it with the following code:

sub _search_hit {
    my($app, $entry) = @_;
    my @text_elements;
    if ($app->{searchparam}{SearchElement} ne 'comments') {
        if(my @fields = $app->{query}->param('SearchField')) {
            foreach my $field (@fields) {
                push @text_elements, $entry->$field;
            }
        }
        @text_elements = ($entry->title, $entry->text, $entry->text_more,
                          $entry->keywords) unless @text_elements;
    }
    if ($app->{searchparam}{SearchElement} ne 'entries') {
        my $comments = $entry->comments;
        for my $comment (@$comments) {
            push @text_elements, $comment->text, $comment->author,
                                 $comment->url;
        }
    }
    return 1 if $app->is_a_match(join("\n", map $_ || '', @text_elements));
}

You'll then create a new template, just to use for keyword extraction. Create a copy of your regular search template (default.tmpl) and name it subject.tmpl. (See LMT's MT Keyword Search tutorial for more information on creating an alternative search template.) My basic code for the template is:

<MTSearchResults>

<MTBlogResultHeader>
Subject Index for <$MTSearchString$>
</MTBlogResultHeader>

<a href="<$MTEntryPermalink$>"><$MTEntryTitle remove_html="1"$></a>
<$MTEntryExcerpt$><br />
<$MTEntryAuthor$> wrote this on <$MTEntryDate format="&#37;B &#37;e, &#37;Y"$><br /><br />

</MTSearchResults>

Of course you can change the styling for your own blog, and list the contents however you choose (full posts? excerpts? chronological? reverse chronological? It's all up to you.)

The other addition to your keyword search template is the list of related keywords. I put this in my sidebar.

<b>Other Subjects Related to "<$MTSearchString$>"</b><br/>
<MTCollateCollect>
<MTSearchResults>
<MTLoop values="[MTEntryKeywords]" delimiter=", ">
<MTCollateRecord>
<MTCollateSetField name="subject_keyword">[MTLoopValue]</MTCollateSetField>
</MTCollateRecord>
</MTLoop>
</MTSearchResults>
</MTCollateCollect>
<MTCollateList sort="subject_keyword&#58;+&#58;i&#58;d">
<MTIfNotEqual a="[MTSearchString]" b="[MTCollateField name=&#39;subject_keyword&#39;]">
<a href="http&#58;//WWW.YOURURL.NET/mt/mt-search.cgi?IncludeBlogs=1&amp;SearchField=keywords&amp;Template=subject&amp;CaseSearch=1&amp;ResultDisplay=Ascending&amp;search=<MTCollateField name="subject_keyword">" title="Index of posts relating to <MTCollateField name="subject_keyword">"><MTCollateField name="subject_keyword"></a><br />
</MTIfNotEqual>
</MTCollateList>

This creates a list of keywords that is related to the one on the page they're looking at. It's generated by looking at all the keywords that occur in posts along with the current keyword. Fun to browse! (Note: if nothing appears, try removing the "MTIfNotEquals" lines.)

Upload subject.tmpl into the proper folder (see the MT manual for instructions.)

4. Create an index of all of your subjects

Make a new template, one that will auto-update. The following code will make a list of all the subject keywords used on your blog:

<MTCollateCollect>
<MTEntries lastn="9999">
<MTLoop values="[MTEntryKeywords]" delimiter=", ">
<MTCollateRecord>
<MTCollateSetField name="subject_keyword">[MTLoopValue]</MTCollateSetField>
</MTCollateRecord>
</MTLoop>
</MTEntries>
</MTCollateCollect>
<MTCollateList sort="subject_keyword&#58;+&#58;i&#58;d">
<a href="http&#58;//WWW.YOURURL.NET/mt/mt-search.cgi?IncludeBlogs=1&amp;SearchField=keywords&amp;Template=subject&amp;CaseSearch=1&amp;ResultDisplay=Ascending&amp;search=<MTCollateField name="subject_keyword">" title="Index of posts relating to <MTCollateField name="subject_keyword">"><MTCollateField name="subject_keyword"></a><br />
</MTCollateList>

I use similar code to create category-specific sets of keywords, which are much more browsable than the full list. This I put in a new category archive.

<$MTArchiveCategory$> Subject Index
<i>This is the list of subjects covered in <$MTArchiveCategory$> posts. You can also read <a href="<$MTBlogURL$>archives/<$MTArchiveCategory dirify="1"$>">all the posts in the <$MTArchiveCategory$> category here.</a></i>
<br/><br/>
<MTCollateCollect>
<MTEntries>
<MTLoop values="[MTEntryKeywords]" delimiter=", ">
<MTCollateRecord>
<MTCollateSetField name="subject_keyword">[MTLoopValue]</MTCollateSetField>
</MTCollateRecord>
</MTLoop>
</MTEntries>
</MTCollateCollect>
<MTCollateList sort="subject_keyword&#58;+&#58;i&#58;d">
<a href="http&#58;//WWW.YOURURL.NET/mt/mt-search.cgi?IncludeBlogs=1&amp;SearchField=keywords&amp;Template=subject&amp;CaseSearch=1&amp;ResultDisplay=Ascending&amp;search=<MTCollateField name="subject_keyword">" title="Index of posts relating to <MTCollateField name="subject_keyword">"><MTCollateField name="subject_keyword"></a><br />
</MTCollateList>

5. Coding your blog

Finally, you'll want to put this code into your main blog index, as part of your code for each post within MTEntries. The code listed shows "IncludeBlogs=1". If you have more than one blog on your MT installation, use the blog number for the blog you are working on (you can find it easily by looking at the URL in the browser window when you are in the edit screen for that blog.)

<MTIfNotEmpty var="EntryKeywords">
<br /><br />See also&#58; <MTLoop values="[MTEntryKeywords]" delimiter=", "> 
<a href="http&#58;//WWW.YOURURL.NET/mt/mt-search.cgi?IncludeBlogs=1&amp;SearchField=keywords&amp;Template=subject&amp;CaseSearch=1&amp;ResultDisplay=Ascending&amp;search=[MTLoopValue]" title="Index of posts relating to [MTLoopValue]">[MTLoopValue]</a>
 |</MTLoop></MTIfNotEmpty>

That will create a list of keywords, separated by "|"s, each of which will link to a list of posts on that subject.

Pretty neat, huh?

You'll want to include that code on your Individual Archive pages as well.

6. Options

A. If you want to get fancy, you can create a list of recent topics for your sidebar:

<!--Recent Topics-->
<a name="recenttopics"></a>
<b>What We&#39;re Talking About Lately</b><br>
<MTCollateCollect>
<MTEntries lastn="20">
<MTLoop values="[MTEntryKeywords]" delimiter=", "><MTCollateRecord>
<MTCollateSetField name="subject_keyword">[MTLoopValue]</MTCollateSetField>
</MTCollateRecord>
</MTLoop>
</MTEntries>
</MTCollateCollect>
<MTCollateList sort="subject_keyword&#58;+&#58;i&#58;d">
<a href="http&#58;//WWW.YOURURL.NET/mt/mt-search.cgi?IncludeBlogs=1&amp;SearchField=keywords&amp;Template=subject&amp;CaseSearch=1&amp;ResultDisplay=Ascending&amp;search=<MTCollateField name="subject_keyword">" title="Index of posts relating to <MTCollateField name="subject_keyword">"><MTCollateField name="subject_keyword"></a><br />
</MTCollateList>

B. Meta-tagging.

You'll need Brad Choate's Regex plugin for this. Put the following in the header information for your Individual Entry Archive:

<MTRegexDefine name="delimiter">s|, |" /><meta name="DC.subject" content="|g</MTRegexDefine>
<MTIfNotEmpty var="EntryKeywords"><meta name="DC.subject" content="<$MTEntryKeywords regex="delimiter"$>" /></MTIfNotEmpty>

That will put the metadata onto each entry page, using the Dublin Core standard.

C. Technorati links

Technorati tracks tags from blogs, Flickr, del.icio.us, and Furl, all in one place. (See this LMT post for details.) You can create an automatic link to the Technorati tag on your main blog page:

<MTIfNotEmpty var="EntryKeywords"><br /><br />
See also&#58; <MTLoop values="[MTEntryKeywords]" delimiter=", "> <a href="http&#58;//www.YOURSITE/mt/mt-search.cgi?IncludeBlogs=1&amp;SearchField=keywords&amp;Template=subject&amp;CaseSearch=1&amp;ResultDisplay=Ascending&amp;search=[MTLoopValue]" title="Index of posts relating to [MTLoopValue]">[MTLoopValue]</a><a href="http&#58;//technorati.com/tag/[MTLoopValue]" rel="tag" title="Technorati tag page for [MTLoopValue]"><img src="http&#58;//www.YOURSITE/images/technotag.gif" border=0 alt="Technorati icon"></a> |
</MTLoop>
</MTIfNotEmpty>

I use this image for the Technorati links: (Please upload this to your own server.)

Throw the same code into your atom.xml template and it should ping Technorati. I also put the following code in that template, which will properly meta-tag each item in your feed:

<MTRegexDefine name="subj_delimiter">s|, |</dc.subject><dc.subject>|g</MTRegexDefine>
<MTIfNotEmpty var="EntryKeywords"><dc.subject><$MTEntryKeywords regex="subj_delimiter"$></dc.subject></MTIfNotEmpty>

Again, you'll need Regex to make that work.

D. Auto-suggest tags

I haven't tried this myself (yet), but Dan & Sherree have a hack so that your current tags will pop-up as suggestions.

E. "Tag clouds"

Also in the "I haven't tried yet" column, Al-Muhajabah has the code to create a "tag cloud" that will show all your tags, with more frequently used ones appearing larger.

F. External Resources

Another idea I borrowed stole from Al-Muhajabah is a list of external resources relating to a subject page, so your reader can jump to a Google search, Wikipedia entry, del.icio.us tag, or Technorati tag on that subject. You'll need the MTCgi plugin for this one. I put it in my sidebar just below the list of related subjects:

More Resources On "<$MTSearchString$>"

<MTCgiParams include="search">
<a href="http://technorati.com/tag/<$MTSearchString$>" title="Blog posts, photos, and links from around the web on <$MTSearchString$>" class="menutext">Technorati's Tags on "<$MTSearchString$>"</a><br>
<a href="http://del.icio.us/tag/<MTCgiParamValue dirify="1">" title="links tagged with <$MTSearchString$>" class="menutext">del.icio.us's Tags on "<$MTSearchString$>"</a><br>
<a href="http://www.google.com/search?q=<$MTSearchString$>" title="search Google for <$MTSearchString$>" class="menutext">Google search on "<$MTSearchString$>"</a><br>
<a href="http://en.wikipedia.org/wiki/<$MTSearchString$>" title="read Wikipedia's encylcopedic article on <$MTSearchString$>" class="menutext">Wikipedia article on "<$MTSearchString$>"</a><br>
</MTCgiParams>

I hope that Subject Indexes are useful for you! Let me know if you've got questions about it.

Comments (16)

Great info here - thank you so much. I've almost got this done on my blog over at www.withashout.net.....

Although I've hit a wall at #3, where you create the subject.tmpl template. When I click on a subject on one of my entries, to take me to the search results, I get a mess of code, and I'm not quite sure what's wrong. here's a sample link of what I'm seeing: http://www.withashout.net/mt/mt-search.cgi?IncludeBlogs=1&SearchField=keywords&Template=subject&CaseSearch=1&ResultDisplay=Ascending&search=Meta

I really appreciate the help. This is going to be a lot of fun when it's all said and done! :^)

Aaron -- it looks like you didn't change the search template. Try using the code in part 3 under MTSearchResults, instead of the default search results code, and then re-upload the template.

bert:

Hi Elise and Mike,
I want to use the keyword tag in the meta keywords field but I can't really. I have a script to make my page titles shorter and it requires to put the wanted name of the page in the keywords field between [ ].

Is there any way to not include the words between the [] in the meta keywords ?

Thank you.

That's a toughie, bert. I think you'd need to use this method to compare the first letter of each keyword with "[" and ignore all keywords that start with "[". I haven't tested this out at all, but I think it would look something like this:

<MTIfNotEmpty var="EntryKeywords">

See also: <MTLoop values="[MTEntryKeywords]" delimiter=", ">

<MTTagInvoke tag_name="MTSetVar">

<MTTagAttribute name="name">first_key_letter</MTTagAttribute>

<MTTagAttribute name="value"><$MTLoopValue trim_to="1"$></MTTagAttribute>
</MTTagInvoke>

<MTIfNotEqual a="[MTGetVar name='first_key_letter']" b="/[">

<a href="http://WWW.YOURURL.NET/mt/mt-search.cgi?IncludeBlogs=1&SearchField=keywords&Template=subject&CaseSearch=1&ResultDisplay=Ascending&search=[MTLoopValue]" title="Index of posts relating to [MTLoopValue]">[MTLoopValue]</a>
</MTIfNotEqual>
|</MTLoop>
</MTIfNotEmpty>

(breaks added for clarity)

You'd need to do the same with the output for the indexes. Let me know how it goes!

bert:

An error occurred:

Can't call method "id" on an undefined value at lib/MT/App/Search.pm line 174.

I have no idea what I'm doing. Sorry !

jane:

this is such a great tutorial! thanks!
my only problem (I think) as a total newbie programmer is that when i click a keyword it brings up the error "No alternate template is specified for the Template 'subject'". I cannot for hte life of me figure out why this is... Hope someone out there might have an idea...
Thanks.

bert: did you install the TagInvoke plugin?

jane: you need to create a template called "subject.tmpl" and upload it to the right directory. See Step 3 above. Sorry if that's confusing; there are a lot of steps involved to make this work!

Jane:

Thanks for the help! Did that (made and uploaded subject.tmpl) and now I get this error when I try to do a search:

Can't call method "id" on an undefined value at lib/MT/App/Search.pm line 173.

Sorry to be a trouble and thanks again for all the help!

Jane, you need to keep the old "defaul.tmpl". Your search function is trying to call the "subject.tmpl" instead of the "default."

jane:

thanks again! i am sorry to ask so many questions. But it still says same thing and i made sure that the index template says:
so where do i indicate that it should search the default.tmpl. I've been around this so many times, I just can't find it...
Apologies for total newbie questions. I will be so thrilled when this works. The keyword thing works great, tho I need to figure out how to connect to my style sheet and make it look good. But once I do, it will be super!
Thanks

Jane:

replace this in your main template, where the search is:

input type="hidden" name="IncludeBlogs" value="Deep North"

with this:

input type="hidden" name="IncludeBlogs" value="1"

And it should work. Also, fyi, you have an extra "/MTEntryIfExtended" in your template.

Thanks for the tutorial and plugin ideas. Great for a couple of blogs I am revamping and getting into top gear. I think this type of navigation around a constantly changing site like a blog that also has the potential to cover hundreds of relevant topics for a reader is just what I need. It took a while to get my head round a couple of the ideas but it now works perfectly. I will be setting it up on another blog today and also trying out the technorati tags thing. Thanks

elise [TypeKey Profile Page]:

Hi Mike,
The Subject Index has been acting funky since the MT3.2 upgrade. Basically results are being returned that don't have the keyword. For example, if you click on 404 error in the Subject Index, the first result returned is "What is Trackback". Yet this tutorial does not have 404 error in its keyword list. Any idea what might be causing this behavior? I'm guessing that it's one of the plugins that isn't functioning quite right in MT3.2.

elise [TypeKey Profile Page]:

Hi Mike,
I figured it out. I forgot that when I upgraded to MT3.2, I needed to also redo the search.pm hack. Works perfectly now.

That's good -- because I hadn't even started to think about what might be going on!

Coolness. I've been explicitly using keywords over the last several months. Now, I'd like to go back and add keywords old posts. To make this easier, does anyone have a "power edit" for keywords? I quickly looked at the code that creates the entries list, and decided I don't have the skillz required to change the power-edit mode to include other fields. It would be even cooler if this listing also had a link to the entry, so I could view it while adding tags.

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)