ON THE INTERNET
Estimating relevancy
By Robert Berkman The Information Advisor
In the information world, units of related information are called "records." A record can be an article, an abstract of an article, a bibliographic citation, a Web page, and so on. A "field" is a specified portion of a record. For example, for Web pages, fields may include the Web page's title; the date that Web page was last modified; the Web page's URL; or the Web page's meta tags. Meta tags are the textual description — though invisible to the Web page, written by its creator to inform search engines what the page is about. Some search engines allow you to instruct the engine to look for your keywords only in a specified field. When would you want to do this kind of limited field search? If you did a keyword for, say, "Vietnam," you'd be swamped with sites. By restricting your search to the title field, you'd help insure that the keyword is the primary focus of the Web pages returned.
If you want to search for recent events, some search engines, such as Alta Vista, let you search for a date field. You can specify that you only want recently updated Web pages.
Sometimes you may only want to see Web sites produced by a specific organization or type of institution. In these cases, you can search the URL field.
Say, for instance, that you're doing a story on Sony, and you want to locate Web pages produced by the company. Zero in on Sony's own sites by searching for "Sony" only in the URL, which normally contains the name of the institution that created the page.
Another example is when you want to see sites only from a particular type of institution. If you were looking for statistics on aircraft deaths but only wanted data from official governmental sites, you could instruct the search engine to return only sites that end in ".gov".
Know when to modify
As any librarian knows, getting good results from an online search often means tinkering with the initial search statement. If an initial search turns up no Web pages, too many, or irrelevant ones, it's a good idea to examine possibly alter that keyword search statement. Try the following:
Check the spelling. If you are searching unfamiliar territory, make sure the words you've used are standard terminology. For example, if you were searching for Web pages on "static electricity," it would be important to know that the technical term for that phenomenon is "electrostatics."
Double-check your syntax if you used Boolean operators. If you didn't get enough hits, take out any AND operators you entered. If you got too many hits, either add an AND operator to restrict your search, eliminate one or more keywords, or limit your keywords to a title field.
One-way bridges
A caution: even if you've done a perfectly good search, you still can't assume the Web pages that the search engines returned are really the most relevant. Why? Some site creators are manipulating their pages and the keywords on their sites with the specific goal of getting high rankings by the search engines.
According to Danny Sullivan, who publishes an e-mail newsletter called Search Engine Watch (http://www.searchenginewatch.com), some sites do this by developing what are called "bridge" pages. A bridge page is created specifically to receive a high ranking from a search engine. When you click on the decoy-like page, you are routed to the site's actual home page.
In his newsletter, Sullivan found, for example, that State Farm Insurance submitted several pages to a variety of search engines. Each page was different and specifically written to rank well in search engines' ranking algorithms. Several pages were submitted to each search engine and written to "capture" a particular insurance-related keyword like auto insurance or boat insurance.
Cyber consultants
Web site owners also are contracting the services of consulting firms that claim they can help the site get higher rankings.
These companies take their clients' Web pages, run them against the major search engines, analyze placement of the page, and then provide specific strategies on how to improve rankings. One of these firms, called Webposition (http://www.webposition.com), also provides free advice on its site on how to improve rankings on a search engine.
Web illusions
Things are not, then, always what they appear to be in the brave new world of the Web. On a traditional online service, like Dialog, you could assume that if you conducted a good search, you'd pretty much get what you expect. But on the Web, it appears the data creator's intentions are another force making an impact on what you'll retrieve.
Although popular search engines like to claim that their robots scour the entire Web, the fact of the matter is no search engine can actually index all the pages out there. The April 2 issue of Science magazine reported on an extensive study by Drs. Steve Lawrence and C. Lee Giles of the NEC Research Institute. The researchers posed over 550 scientific search questions to the five largest search engines and discovered that there are about 320 million Web pages — but that search engines index only about 40% of them.
Information overload
If you feel overwhelmed by the enormous number of hits you get back from Alta Vista or one of the other well-known search engines, try a newer search engine called Northern Light (http://www.nlsearch.com). Northern Light does an admirable job in making a dent in information overload by grouping the sites it retrieves into logical folders, which contain related information.
Robert Berkman can be reached at (413) 549-9924 or by e-mail at rberkman@javanet.com.