OPIM101 - Case 1 - Part1 - Balanced Scorecard


Note to graders: The information provided below is separated into the six question groups.


For Part 1 of the first case for OPIM101, I was asked to use five search engines to determine which provided the most relevant results for the subject of the "Balanced Scorecard". Using Altavista, Google, Hotbot, Lycos and Excite, I explored the various sites until I discovered enough information to answer the following questions:
  1. What is Activity Based Costing (Balanced Scorecard)?
    2. bullet"The Balanced Scorecard is an approach to performance measurement that combines traditional financial measures with non-financial measures to provide managers with richer and more relevant information about activities they are managing.”
  2. What are the advantages of this approach over traditional methodologies?
    3. bullet“Financial measures alone do not provide enough information to properly manage complex organizations operating in a complex environment. A balanced view of organizational performance must include measures that indicate performance in at least four areas: Financial, Customer, Internal Business Processes, and Learning and Growth. A Balanced Scorecard overcomes key barriers to strategy implementation by: (a) making the strategy specific and actionable, (b) engaging everyone in the organization to set stretch targets and resource allocation plans that specifically relate to the strategy, and (c) providing strategic feedback and learning.”
  3. Which companies have implemented Activity Based Costing (Balanced Scorecard)? (Limit to 3 names)
    4. bullet Boeing
    bullet Microsoft
    bullet NASA
  4. Which companies offer consulting services in implementing this methodology? (Limit to 3 names)
    5. bullet 2GC Active Management
    bullet Foresight International
    bullet PdK Consulting

 My main sources of information to answer the above questions were:
bullethttp://www.bscnews.com/balanced_scorecard/balancescorecard.htm
bullethttp://www.ceoreview.com/books/scorecard.htm
bullethttp://www.2gc.co.uk/Pub-FAQ.asp


To enrich my knowledge and understanding of search engines, I was asked to use the search string "Balanced Scorecard" with and without quotations in order to determine the recall and precision of the 5 search engines. You can see in the tables below the number of hits returned by each search engine on two different occasions:

Results as of January 19th, 2002

Website Search Results: # of Hits Returned
Search Engine Balanced Scorecard "Balanced Scorecard"
Altavista
833,246
30,102
Google
96,700
88,400
Hotbot
34,600
29,200
Lycos
45,792
42,660
Excite
240
240


Results as of January 30th, 2002

Website Search Results: # of Hits Returned
Search Engine Balanced Scorecard "Balanced Scorecard"
Altavista
40,125
33,167
Google
103,000
93,300
Hotbot
34,700
31,700
Lycos
45,708
42,576
Excite
240
240


[Note: There appears to have been an overall INCREASE in the number of sites that mention "Balanced Scorecard" in the past week and a half for most of the sites. Also, the increases were greater for the search w/ quotes than without.]


Obviously, browsing through all tens and hundreds of thousands of websites returned would be impossible and unfeasible. However, the results returned by the different search engines reveals how "good" each one is in comparison to the others. One would hope that amidst the thousands of pages returned, the first 20 would be completely relevant. Unfortunately, in the top 20 results from all five search engines were foreign sites that I couldn't possibly comprehend. Hotbot and Lycos each returned 1, Google and Excite returned 4, and Altavista returned an astonishing 8 foreign sites! One of the sites I encountered in the top 20 from Hotbot was a site called "Social Rejects." An obvious anomaly! Here is a list of a few of the common sites returned by at least two of the search engines:
bullethttp://www.bscol.com
bullethttp://www.bscnews.com - provided links that led me to other informative sites.
bullethttp://www.balancedscorecard.org
bullethttp://www.balanced-scorecard.nl - This site is one from the Netherlands that doesn't help at all... primarily since I can't read the language!


The five search engines that we were directed to use produced varying degrees of recall and precision. Based on the numbers of documents returned, it would appear that Google has the best precision while Altavista has the greatest recall (while it is possible that Google is best in both!). Of the five search engines, I would first throw out Excite as the worst of the group. First, it caps the results at 240 making it impossible to find all possible documents if only 240 can be returned. Since the formulas for calculating recall and precision are:
bulletRecall = (Number of Relevant and Retrieved) / (Total Number Relevant)
bulletPrecision = (Number of Relevant and Retrieved) / (Total Number Retrieved)
Therefore, determining the Recall value of the sites is impossible since we do not know the Total Number of Relevant Documents.

Note: My Precision ratings based on English websites. If I were multilingual, the Precision ratings for the top 20 sites returned would be higher because the foreign language sites appear to be relevant (just not to me).

    The rankings for precision are as follows:
  1. Google - Having a huge database, it is able to retrieve documents with high precision. Google's PageRank technology is explained below.
    My Precision rating: 0.75
  2. Altavista - Producing the highest number of hits(833,246), it may have the highest recall in returning ALL relevant documents. The fact that it returned 8 foreign language sites further demonstrates it's high recall in returning foreign sites that other search engines may easily miss.(Altavista also has an option to return only English sites).
    [edited (1/30/02): The number of total hits returned by Altavista dramatically dropped between the time period from 1/19/02 and 1/30/02. Perhaps Altavista editted their search software. Anyway, the reduced number of hits seems to indicate that extraneous sites were removed and that Altavista now has even higher recall and precision.]
    My Precision rating: 0.6
  3. Lycos - Although producing less total documents than Google, this engine also was able to locate foreign language sites(It also has an option to choose just English sites, but under Advanced Options).
    My Precision rating: 0.6
  4. Hotbot - Apparently, a site owned by the Terra Lycos Network, this engine is less effective in recall and precision than it's parent site Lycos. It does not appear to have language options.
    My Precision rating: 0.55
  5. Excite - Claims to have Precision Search, but seemed to be the weakest search engine as explained above.
    My Precision rating: 0.5
The University of California at Berkeley has a internet search guide that spells out why Google is the best search engine.
The site is located at: UCBerkeley rates Google
The primary reason is Google's ranking method of: PageRank(tm)
For searches requiring the ability to trace the occurence of words within pages with other words, the UCB site recommends Altavista.
Altavista's Search Technology explained here: Altavista


In the hypothetical situation in which the precision (based on the first 20 items returned) is very high among all search engines for the given search string, one can deduce that the search string was very specific in its subject matter. This type of search string is what is desired when searching for data because a high precision will result in less digging through useless documents that do not provide useful information.


If we used the search string 'balance score cards' (without the quotes) in place of 'Balanced Scorecard,' we drastically reduce the precision of almost all five search engines that we used. Most of the hits returned on Hotbot, Excite and Lycos deal with credit cards. Altavista and Google also return one or two credit card webpages and Altavista even returns a gymnastics website based on the word "balance"! Overall, the precision of all the search engines is drastically reduced by changing "balanced" to "balance" and separating "scorecard" into "score cards" because the additional words "score" and "cards" produce many more hits.

Back to Dan's Homepage

Back to OPIM101 - Case 1 - Part 2

Contact: Daniel Park (parkdani@alumni.upenn.edu)