Yahoo Articles

I implemented full text search with the Yahoo! Search BOSS API on top of CakePHP in one night, ending at 5AM - lol.  It has auto-suggest for spelling errors, and I added paging as well. The BOSS API impressed me, it was very clean and I was able to implement seamless full text search into my website.

The full source is at the bottom of the article, but let me walk you through the basic parts.

Start by gathering the results. I'm sure there is a better way to do this in CakePHP, but for now I'm using straight PHP5.  I got most of this off Webmaster-Source, but I added a few things including: setting the site search context site:domain.com as well as getting the total results found totalhits.  You can see all the xml data fields that you get back on the search response fields page.

Here is the code for gathering the search results, make sure you get $search_term from the params, then switch out the BOSS API key your_API_key and domain.com with yours. You get the API key by registering on the Yahoo! BOSS website.

PHP:

//Gather data and prepare query
$thequery = urlencode($search_term);
$yhost = 'http://boss.yahooapis.com';
$apikey = 'your_API_key';
$site = 'site:yourdomain.com';
$url = "$yhost/ysearch/web/v1/$thequery+$site?appid=$apikey&format=xml";
// if (!empty($start)) $url .= "&start=$start"; Add this to provide paging
 
//Get the search results
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, 0);
$data = curl_exec($ch);
curl_close($ch);
$results = new SimpleXmlElement($data, LIBXML_NOCDATA);
$total = $results->resultset_web->attributes();
$total = $total['totalhits'];
$results = $results->resultset_web->result;
 
$yhost = 'http://boss.yahooapis.com';
$url = "$yhost/ysearch/spelling/v1/$thequery?appid=$apikey&format=xml";

We can loop through our results in our view. $results will contain title, dispurl and abstract. I use the shortcut, <? which is shorthand for <?php echo.

PHP:

<? $i=1; foreach($results as $result): if(!empty($result->title)): ?>
    <li>
        <span class="count">#<?= $i++; ?>:</span>
        <? $title = str_replace('"', '"', $result->title); ?>
        <a class="search_result" title="<?= $title; ?>" href="<?= $result->clickurl; ?>">
            <?= $title; ?>
        </a><br />
        <span class="abstract">
            <?= str_replace(
                array('<b>yourdomain.com</b>', '<b>YourDomain.com</b>', '<b>...</b>'),
                array('yourdomain.com', 'YourDomain.com', '...'), 
                $result->abstract
            ); ?>
        </span><br />
        <div class="url"><?= strip_tags($result->dispurl); ?></div>
    </li>
<? endif; endforeach; ?>

Next, we want to get any suggestions in case they spelled something wrong.  This is obviously optional, but I think it is a key to search. People are going to misspell things (including me). Using the spelling API, look up the suggested search term.

PHP:

//Get the suggested term
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, 0);
$data = curl_exec($ch);
curl_close($ch);
$suggest = new SimpleXmlElement($data, LIBXML_NOCDATA);
$suggest_term = $suggest->resultset_spell->result->suggestion;

Then it is easy to display the suggestion in our view:

PHP:

<? if(!empty($suggestTerm)): ?>
    <strong>Did you mean: <?= $html->link($suggestTerm, '/searches?q=' . $suggestTerm); ?></strong>
<? endif; ?>

I also implemented paging in the view, this is probably pretty crappy code... written very close to 5AM so you may want to do this your own way. It basically uses the $start and $total variables in the URL to calculate the pages, and what page you are on.

PHP:

<?php
    if ($start > 0):
 
        if ($start == 10)
            $startParam = '';
        else
            $startParam = "&start=" . ($start - 10);
       
        echo $html->link('Prev', $self . "?q=$searchTerm" . $startParam) . "&nbsp;&nbsp; ";
    endif;
 
    $page = 0;
    for ($i=0; $i<$total; $i=$i+10) {
        $page++;
   
        if ($i > ($start-50) && $i < ($start+50)) {
            if ($i == $start) {
                echo " $page ";
            } else if ($i === 0) {
                echo $html->link($page, $self . "?q=$searchTerm");
            } else {
                echo $html->link($page, $self . "?q=$searchTerm&start=$i");
            }
        } else if ($i == ($start-60)) {
            echo '... ';
        } else if ($i == ($start+60)) {
            echo '... '; break;
        }
    }
    echo " &nbsp;&nbsp;";
 
    if (count( $results ) === 10):
        echo $html->link('Next', $self . "?q=$searchTerm&start=$next");
    endif;
 
    $countPages = ceil($total / 10);
    echo "&nbsp;&nbsp; ($total results on $countPages pages)";
endif;
?>

Ok, putting it all together here is the full source.

Full source of controllers/searches_controller.php

PHP:

<?php
class SearchesController extends AppController {
    var $name = 'Searches';
    var $uses = array();
 
        public function index() {
        
        params['url']['q']) ? $this->params['url']['q'] : null;
        $start = isset($this->params['url']['start']) ? $this->params['url']['start'] : 0;
        
        if (!empty($search_term)) {
        
            //Gather data and prepare query
            $thequery = urlencode($search_term);
            $yhost = 'http://boss.yahooapis.com';
            $apikey = 'your_API_key';
            $site = 'site:yourdomain.com';
            $url = "$yhost/ysearch/web/v1/$thequery+$site?appid=$apikey&format=xml";
            if (!empty($start)) $url .= "&start=$start";
 
            //Get the search results
            $ch = curl_init($url);
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
            curl_setopt($ch, CURLOPT_HEADER, 0);
            $data = curl_exec($ch);
            curl_close($ch);
            $results = new SimpleXmlElement($data, LIBXML_NOCDATA);
            $total = $results->resultset_web->attributes();
            $total = $total['totalhits'];
            $results = $results->resultset_web->result;
            
            $yhost = 'http://boss.yahooapis.com';
            $url = "$yhost/ysearch/spelling/v1/$thequery?appid=$apikey&format=xml";
            
            //Get the suggested term
            $ch = curl_init($url);
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
            curl_setopt($ch, CURLOPT_HEADER, 0);
            $data = curl_exec($ch);
            curl_close($ch);
            $suggest = new SimpleXmlElement($data, LIBXML_NOCDATA);
            $suggest_term = $suggest->resultset_spell->result->suggestion;
            
            $next = $start + 10;
            $this->set(compact('search_term', 'results', 'suggest_term', 'start', 'next', 'total'));
            
        }
    }
    
}
 
?>

Full source of views/searches/index.ctp

PHP:

<?php if (!empty($searchTerm)): ?>
    
<?php
    $this->pageTitle = "Search results for, \"$searchTerm\"";
?>
 
    <h2>Search results for, "<?= $searchTerm; ?>"</h2>
    <? if(!empty($suggestTerm)): ?>
        <strong>Did you mean: <?= $html->link($suggestTerm, '/searches?q=' . $suggestTerm); ?></strong>
    <? endif; ?>
    <div class="container search_results">
        <? if(empty($results)): ?>
            No results found for "<?= $searchTerm; ?>". 
        <? else: ?>
            <h4>Results <?= $start+1 . '-' . (count($results)+$start) . ' of ' . $total; ?></h4>
            <ul>
                <? $i=$start+1; foreach($results as $result): if(!empty($result->title)): ?>
                    <li>
                        <span class="count">#<?= $i++; ?>:</span>
                        <? $title = str_replace('"', '&quot;', $result->title); ?>
                        <a class="search_result" title="<?= $title; ?>" href="<?= $result->clickurl; ?>">
                            <?= $title; ?>
                        </a><br />
                        <span class="abstract">
                            <?= str_replace(
                                array('<b>yourdomain.com</b>', '<b>YourDomain.com</b>', '<b>...</b>'),
                                array('yourdomain.com', 'YourDomain.com', '...'), 
                                $result->abstract
                            ); ?>
                        </span><br />
                        <div class="url"><?= strip_tags($result->dispurl); ?></div>
                    </li>
                <? endif; endforeach; ?>
            </ul>
            
            <? 
            
            if ($total > 10):
                if ($start > 0):
                
                    if ($start == 10)
                        $startParam = '';
                    else
                        $startParam = "&start=" . ($start - 10);
                        
                    echo $html->link('Prev', $self . "?q=$searchTerm" . $startParam) . "&nbsp;&nbsp; "; 
                endif;
                
                $page = 0;
                for ($i=0; $i<$total; $i=$i+10) {
                    $page++;
                    
                    if ($i > ($start-50) && $i < ($start+50)) {
                        if ($i == $start) {
                            echo " $page ";
                        } else if ($i === 0) {
                            echo $html->link($page, $self . "?q=$searchTerm");
                        } else {
                            echo $html->link($page, $self . "?q=$searchTerm&start=$i");
                        }
                    } else if ($i == ($start-60)) {
                        echo '... ';
                    } else if ($i == ($start+60)) {
                        echo '... '; break;
                    }
                }
                echo " &nbsp;&nbsp;";
                
                if (count( $results ) === 10):
                    echo $html->link('Next', $self . "?q=$searchTerm&start=$next");
                endif;
                
                $countPages = ceil($total / 10);
                echo "&nbsp;&nbsp; ($total results on $countPages pages)";
            endif;
            ?>
        <? endif; ?>
    </div>
    
    
 
<?php endif; ?>

Yahoo just released version 4 of Yahoo Widgets. View the overview of enhancements below. If you want more detail, check out the Yahoo blog.

Since I am working on a widget as we speak, I dug through the Reference Manual and found these changes.

Basic CSS Support

Yahoo Widgets now supports basic cascading style sheet attributes.

Canvas

You can draw lines, rectangles, circles, etc. This makes my job much easier on the widget that I'm currently developing!

SQLite

This is cool, you can now communicate with a SQL database. Nice! Now I could make a widget to manage my blog or maybe I will convert Ajax Messages into a widget... hmm... interesting doctor jones.

DOM API

You can now use appendChild() and removeChild() to manipulate the DOM.

My Notes

The new widget docker is nice to keep everything organized. The canvas makes drawing much simpler (graphs, etc). They sped up the engine (Up to 40% CPU Use Reduction), speed has been an issue since the Konfabulator days. The addition of SQLite allows developers to make more diverse and dynamic widgets. I noticed when you download a widget from the widget gallery it instantly open, simple but cool!

Yahoo remains to be the only cross-platform widget engine. If you have some time check out the new Yahoo! Widgets.

In 2005, Yahoo purchased "Konfabulator" and renamed it Yahoo! Widgets. Yahoo's widgets run on your desktop and can do everything from forecast the weather to validate your javascript. I tried it a few years back and it slow, but things are different now. The code-base has improved and the widgets don't slow down your computer as much. Companies have been rumored to "steal" the idea of desktop widgets from Konfabulator. Apple was the first rumored to copy the widget idea, but now Microsoft and Google have their own "gadgets" as well.

Users can create their own widgets and share them with everyone in the widget gallery. Here is a list of some of my favorites.

• Paper - Even more simple than Notepad.
• Screen Shooter - Easily take screenshots of a portion of your desktop and save it to an image.
• iTunes Bar - Controls iTunes from a slick little bar.
• My Volume - Controls system volume and iTunes volume.
• Quick Dock - Quick launch replacement that looks like the Mac OSX dock.
• PowerXP - Easy access to lock, shutdown, restart, or bring your pc to standby mode.
• Widget Maker - Aids in building your own widget.
• JSLint - Validates your javascript code.
• AWStats Bar - Displays unique and total visitors from an AWStats installation.
• Colors - HTML color picker.
• IconXtractor - Converts between images and icon files.

Now that I have seen and used some great widgets, it is time to create a widget. I am tasked at work with creating one of these widgets to monitor website traffic. Wish me luck!