CAVAN MCCARTHY

INTERNET AS A SOURCE FOR COMPETITIVE INTELLIGENCE

A new field of activity for Special Libraries and Information Centers


A paper presented to the
Sixth Annual Conference of the
Special Libraries Association
Arabian Gulf Chapter
Bahrain, November 1998



ABSTRACT

1. INTRODUCTION

2. DEFINITIONS

3. THE VALUE OF COMPETITIVE INTELLIGENCE

4. COMPETITIVE INTELLIGENCE IN THE SPECIAL LIBRARY

5. COMPETITIVE INTELLIGENCE AND COMPANY WEBSITES

6. OTHER INTERNET SERVICES

7. THE ETHICS OF COMPETITIVE INTELLIGENCE

8. ORGANIZING COMPETITIVE INTELLIGENCE INFORMATION

9. PROFESSIONAL ORGANIZATIONS AND PUBLICATIONS

10. CONCLUSIONS

BIBLIOGRAPHY
 
 
 

ABSTRACT

Special and above all corporate libraries have always reported the activities of companies which compete with their host companies. These activities are referred to as the monitoring of competitive intelligence, but until recently special libraries could only apply them in a limited manner, although such information is now critical due to globalization and rapid change. This problem has now been solved, because most companies now have Internet sites, which reveal a significant amount of information about their activities. Any corporate librarian can enter sites of competing companies and download information on product launches, company policy, prices, organizational structure, consumer relations etc. Further information can be found in online news services and company oriented information services. Information retrieval via search engine is simplified because search terms are normally company or product names. Competitive intelligence can be undertaken by special libraries located in any part of the world, and be aimed at libraries globally. Although special libraries have up to now concentrated their activities on more formal sources of information, they need to be aware of the potential of Internet for monitoring competitive intelligence.
 
 

1. INTRODUCTION

The principle objective of special libraries is to support the aims of the institutions to which they are subordinate. When they are part of industrial, commercial, financial and other profit-oriented companies, they naturally act in such a way as to help maximize the profits of their parent companies. For this reason they have always tried to remain aware of the activities of other companies in the same area, especially when they compete directly with their parent companies. It was often possible to find useful information in the periodicals scanned or the databases regularly searched by the library. These activities are generally referred to as the monitoring of competitive intelligence.
 
 

2. DEFINITIONS

Gulliford (1998), in one of the few papers written so far on this subject from a business school perspective, begins by defining competitors as other organizations offering the same or similar products or services now, or that could offer such products or services in the future, or that could remove the need for a product or service. He cites two possible definitions for competitive intelligence: "Information and knowledge to be obtained and used to formulate strategies for effectively dealing with competitors, customers and regulators" or "any information obtained from sources external to the firm that can help improve the firm's performance". Kahaner (1996) was careful to distinguish between information and intelligence. Information consisted of factual items; only when collections of information pieces had been filtered, distilled, and analyzed, only when they had been turned into something a manager could act upon, could they be considered intelligence (Kahaner, 1996). In these terms only the first of Gulliford's definitions would be appropriate for competitive intelligence.

Other definitions of competitive intelligence are careful to point out the public nature of the information being sought: "the use of public sources to develop information about the competition, competitors and market environment" (McGonagle, 1993, p. 3); "… gathering of information, the majority of which is readily available" (Attaway, 1998).

Related terms include business monitoring, business intelligence, or competitor intelligence, but the term competitive intelligence clearly predominates. These terms often seem to be used with very similar meanings; Kelly (1987, p. 4) stated clearly that competitive and competitor intelligence could be used interchangeably. Miller (1996) attempted to differentiate between three concepts. Business intelligence monitored a wide range of activities across the external environment of an organization. Competitive intelligence focused on the strengths, weaknesses, and activities of the organizations with similar products or services within a single industry, while competitor intelligence emphasized the acquisition of detailed and current information regarding a specific organization (Miller, 1996). This level of differentiation seems excessive; it is difficult to imagine that busy professionals, who frequently refer to their activities by the acronym CI, would have time to apply such categories in practice. Perhaps the best definition comes from Gordon's pioneering book on the subject, published in 1989. He firmly rejected the term competitor intelligence in favor of competitive intelligence, which he defined as: "The process of obtaining and analyzing publicly available data to develop the information necessary to serve as input to competitive strategy development" (Gordon, 1989, p. 9). This seems to unite all significant elements and can be accepted as a standard definition.
 
 

3. THE VALUE OF COMPETITIVE INTELLIGENCE

Kahaner (1996) offered several motives for undertaking competitive intelligence: anticipate changes in the marketplace, including anticipating actions of competitors; discovering new or potential customers; learning from the successes and failures of others; learning about new technologies, products, and processes or learning about political, legislative, or regulatory changes (Kahaner, 1996).

He also suggested motives why companies needed competitive intelligence more than ever: the quickening pace of business; information overload; increased global competition; more aggressive competition and rapid technological and political change. (Kahaner, 1996).

Despite such pressing motives, less than seven percent of large American companies had a full-scale competitive intelligence division; eighty percent of these were established in the last five years (Kahaner, 1996). Activities of this nature were most firmly established in Japan, where they had been routine since the end of the Second World War, or in France, where the government regularly collected information on foreign companies.

Kahaner also attempted to explain why some companies do not use competitive intelligence. Amongst motives cited were: many managers thought they knew everything about their industries, or believed that nothing worth watching could happen outside America; some considered competitive intelligence spying; the subject was not taught in business schools in the US; its benefits could be easily quantified; American managers were financially, not technologically, oriented (Kahaner, 1996).
 
 

4. COMPETITIVE INTELLIGENCE IN THE SPECIAL LIBRARY

Competitive intelligence has always been considered a valid activity for a special librarian, even though it was until recently considered a relatively minor element amongst professional duties. The basic textbook, by Ellis Mount, only devotes a couple of brief paragraphs to the subject (Mount, 1995, pp. 114-5). The activity was given professional respectability for special librarians by a paper on "Competitive Intelligence and the Information Center" by Greene, published in "Special Libraries" as long ago as 1988. As might be expected from a paper of that era, databases were considered central to competitive intelligence gathering and organization. Other sources suggested by Greene were subscriptions to local newspapers from cities where competitors had facilities, personal contacts, advertisements and even the yellow pages.

Until recently competitive intelligence remained a relatively peripheral activity in special libraries because its techniques could only be applied in a relatively limited manner, due to the lack of in-depth indexing of periodicals and the limited coverage of databases. This was despite a rapidly increasing demand for up-to-date information on the activities of competitors.
 
 

5. COMPETITIVE INTELLIGENCE AND COMPANY WEBSITES

The recent exponential growth of the Internet has totally changed the global informational environment. Greene in 1988 compared competitive intelligence work to a mosaic; the World Wide Web has literally created a giant, computerized, informational mosaic which vastly increases the potential for competitive monitoring by special libraries and information centers. Most large and many small companies now have extensive Internet sites, which publicly reveal a significant amount of information about their activities. Although the Internet is popularly considered to be largely a source of entertainment and even titillation, in fact by far the fastest growth is in the area of business sites.

The potential of Internet for this field has now been widely recognized. A newsletter in the marketing field was highly excited by the potential of Internet: "a few clicks of the mouse can provide an avalanche of free - but priceless - information about competitors. Ironically, much of the most useful information is provided by competitors" (Yovovich, 1997). This was a journalistic, rather than a formal source, but it included an accurate summary of nine categories of information available from corporate sites:

1 Customer and client lists;

2 Detailed product and pricing information, as well as reams of product specifications and technical data;

3 Specifics about business goals and the strategies companies plan to use to achieve them;

4 New product plans and R&D efforts;

5 Extensive company job postings that shine a spotlight on new business emphases;

6 Details about manufacturing processes and quality-control efforts;

7 Company organizational structure and biographical information about top management;

8 Comprehensive information about business locations, offices, distributors, and service companies;

9 Information about partnerships, joint ventures and strategic alliances. (Yovovich, 1997)

The author concluded that "The Web presents firms with a dilemma. To leverage its power to communicate with prospective customers, employees and investors, companies feel they must post plenty of information on their sites. One the other hand, information so posted is available to anyone who seeks it out" (Yovovich, 1997).

Two recent papers help librarians and information make best use of Internet resources for competitive intelligence. Kassler (1997) was an important and comprehensive source, taking the researcher into a company web site, and onwards to other business information sources, based on the knowledge of a person with a deep experience of the practical side of seeking competitive intelligence. Volpo and Piggott (1996) attempted an equally wide coverage, but with less depth; their paper can still be useful as an introduction to the field.

Internet is an ephemeral communication channel, and sites require constant updating. The advantage for competitive intelligence activities is that the updating process itself can offer important information to the alert observer. New staff appointments, the creation of new divisions, price alterations, or the opening of plants or facilities in specific locations can give important insights into the directions which competing companies are taking. Software such as Nearsite (http://www.nearsite.com) will aid in automatically retrieving and filing relevant sites.
 
 

6. OTHER INTERNET SERVICES

Further information can be found in online news services and company oriented information services. There is now a considerable number of these, such as the Infoseek News Center (http:// www. infoseek. com). Companies Online (http:// www. companiesonline. com), KnowX (http:// www. knowx. com) available either free or against a modest subscription. Most of these are indexed, either by major search engines or by specialized systems. Information retrieval is simplified because the search term is normally the name of a company or its products.

Special librarians do not always consider discussion groups and Usenet newsgroups as serious information sources, because open forums of this nature tend to attract disorganized and poorly considered messages. But precisely because contributors feel that they can speak openly in these groups, they sometimes offer critical information about weaknesses of competitors' products and of their customer service departments. Information of this type must, of course, be used with care, but it is now common to monitor discussion lists and newsgroups. In fact most companies today keep a close watch on discussion lists which relate to their own products, correct any misinformation and privately contact dissatisfied customers. Experienced users frequent discussion lists and they often contribute detailed descriptions of how to use products. These can reveal critical weaknesses of competing products or suggest new lines of development. There is little difficulty in locating discussion lists, which are described in several sources, notably "Liszt" (http://www.liszt.com). Newsgroups are thoroughly indexed by publicly available search engines; Dejanews (http://www.dejanews.com) is said to be the most thorough indexer of newsgroups; again information retrieval in this area is simplified because company or product names are normally used as search keys.

Often Internet will open up fascinating new vistas for researchers. One practitioner found Internet very useful for what she called primary resources: "well-informed people with knowledge that hasn't yet reached published form. These can include researchers in the field, vendors, association members, former employees, and reporters" (Kassler, 1997). This is a category of information which an office-bound librarian could only obtain via Internet.
 
 

7. THE ETHICS OF COMPETITIVE INTELLIGENCE

In every discussion of competitive intelligence the problem of professional ethics soon arises. Even the two paragraph coverage of competitive intelligence in Ellis Mount's special library textbook was careful to point out that the "service is perfectly legal, not at all unethical" (Mount, 1995, p. 114). Many sources include significant discussion of the subject; for instance McGonagle's book (1993) dedicates its third chapter to "Ethical and legal issues". The second of the Frequently Asked Questions on the Society of Competitive Intelligence Professionals' web site is headed "Is CI espionage?" (SCIP. Frequently…, 1998). Good definitions of competitive intelligence point out that the field deals with publicly available information. One major review went so far as to define competitive intelligence as the "ethical and legal gathering of information, the majority of which is readily available" (Attaway, 1998). Illegal or unethical activities were therefore automatically labeled as espionage. The two areas are so close that Attaway considered it necessary to list activities which he clearly considered as espionage, such as stealing laptops from competitors, intercepting their communications or stealing trash from outside their homes. Note that there are fine distinctions here: stealing laptops is a crime everywhere; stealing trash is not a crime in most communities, but would be unethical if undertaken for intelligence gathering purposes. McGonagle defines industrial espionage as "the collection of information by illegal means, such as breaking and entering, "hacking" or collecting information that it is illegal to have, such as other companies' 'confidential' papers" (McGonagle, 1993, p. 3). The latter category would presumably cover documents put into trash. The Society of Competitive Intelligence Professionals' Code of Ethics requires members "to avoid all unethical practices", but does not define such practices in detail; members must however continually strive to increase respect and recognition for the profession (SCIP Code…, 1998). Specific requirements are that members must comply with all applicable laws; accurately disclose all relevant information, including identity and organization, prior to all interviews, and fully respect all requests for confidentiality (SCIP Code…, 1998).

In many borderline cases the intention is more important than the act. Interviewing former employees of competitors for jobs, and asking them what projects they worked on, is ethical if there is a real intention of offering employment, unethical if the objective is simply information gathering. Contacting a competitor, discussing a possible joint venture, and obtaining information from them is ethical, as long as there is a real intention of entering into a joint venture. It would however be unethical if done just to obtain information, trade secrets, designs etc. McGonagle suggested a "golden rule" to distinguish the ethical from the unethical: executives put themselves in the place of the recipient of the information gathering activity that is being considered and should ask themselves whether they would feel wronged or done an injustice (McGonagle, 1993, p. 40). There is clearly some room for subjectivity here, but this is perhaps the closest that one can come to an answer to this dilemma.

There is considerable concern about such problems, but the majority of executives, when surveyed, state that they would be unwilling to engage in unethical intelligence gathering practices (Gordon, 1989, p. 19). It is also unnecessary, especially in the context of Internet, which is above all a publicly available channel. McGonagle emphasized that "A key maxim of CI is that 90% of all information that you and your business need to make key decisions and to understand your market and your competitors is already public or can be collected from public sources" (McGonagle, 1993, p. 3). Professionals working in special libraries and information centers will be so busy exploring the riches of the Internet that they will not have time for or interest in unethical activities.
 
 

8. ORGANIZING COMPETITIVE INTELLIGENCE INFORMATION

One of the most obvious features of Internet, especially when used with major search engines, is its ability to reveal large quantities of non-standardized information. To avoid information overload it is necessary to focus on specific sites, companies, products etc. This is not difficult in CI, where much information gathering is done by company or product name. Evaluating data is a further problem. Many of the clues which librarians have traditionally used to identify quality information, such as affiliation of the author, reputation of the publisher or journal, are absent from Internet. Each specific item must be evaluated from the point of view of the context where it was found, and also in relation to other information on the same subject, either from Internet or from a database, periodical etc. Information from a source generally considered of good quality, which coincides with what is learnt from other sources, can be considered accurate. Final verification depends on somebody going out into the field and confirming the information via observation or personal contact. As McCune (1997) warned: "Do not believe everything that you read on the Internet; use the Internet as a launching pad, not the end-all for your intelligence gathering".

Specialized Internet software, such as push or channeling software, also supports competitive intelligence monitoring activities. Push technology is the Internet equivalent of SDI; information from specific channels can be "pushed" to a user's microcomputer. This was widely discussed recently, but has yet to achieve wide acceptance (Arnold, 1997; Cahlin, 1998). The basic problem with push technology is that early systems, such are PointCast, gained a bad reputation for clogging already slow Internet connections with even more data, in order to overload users with excessive quantities of realtime information. Much work is being done on Push, notably in the sense of filtering information before it reaches the end user, but it seems to be a technology with great potential whose time has not quite arrived. Farcast (http:// www.farcast.com) is now often cited as a useful source of current information.

Information downloaded must also be properly filed. Clearly a flexible electronic database system would be called for here; Lotus Notes has been cited in the professional literature (Sawka, 1996). AskSam, a free-form textual data base, would also be a valid choice; web pages and traditional materials can be archived together in its database. Filing a mass of varied information so that it can be retrieved easily is clearly a major problem in this field. Books on competitive intelligence offer checklists or worksheets to help organize data; Kelly (1987) devoted a significant part of his book to this, offering no less than 31 worksheets, of up to five pages in length. Such worksheets could be used as a starting point for the design of databases to file competitive intelligence information.

Competitive intelligence can be undertaken by special libraries located in any part of the world, and be aimed at libraries globally. North American or European libraries have few specific advantages in this area, apart from quicker Internet connections. Some Internet sources now charge small amounts for specialized documents, detailed company descriptions etc. It is common to make payments of this mature by credit card. This could presumably be arranged for corporate libraries, but might well be impossible in government institutions. Competitive intelligence activities could easily be outsourced, especially in countries where communication charges are significantly lower at night. The activities would be entirely appropriate for home workers who need flexible schedules.
 
 

9. PROFESSIONAL ORGANIZATIONS AND PUBLICATIONS

Although a new field, competitive intelligence already has a firmly established professional organization, the Society of Competitive Intelligence Professionals, created in 1986. Their desire to be considered as making a serious contribution to society is clearly demonstrated by the inclusion of the word "Professionals" in the name of their society. They also have a well-organized web site (http://www.scip.org). At the time this paper was being prepared their site stated that the society had 6,000 members, a number which, SCIP claimed, had been increasing at a rate of 40% each year (Frequently asked questions; 1998). About a quarter of the members were from 44 countries outside the US. Job titles for professionals in these areas include competitive intelligence monitor or analyst; Piggott (1996) noted the emergence of the more impressively titled "Business Intelligence Strategist". The principal periodical in the field is the quarterly "Competitive Intelligence Review", first published in 1989. A more journalistic approach is offered by "Information Strategy: Europe's magazine for business advantage", from the Economist Group (http://www.info-strategy.com).
 
 

10. CONCLUSIONS

A constant process of absorbing, or even pioneering, new technology, has dominated the history of special libraries over the last 150 years. Catalog cards, subject headings, specialized classifications and abstracting were in use in special libraries over a century ago. Mechanized indexing systems such as optical coincidence and Uniterm were invented in special libraries and laid the foundations for post-coordinate indexing. Databases and SDI owe much to the special library environment, notably to the Lockheed library that gave birth to Dialog. The first CD-Roms widely used were Medline discs in medical libraries. Now special libraries are at the forefront of using Intranets to disseminate information within their organization. Note that all this technology is never used for its own sake, but above all to improve the quality of information services and enable effective access to a wider range of resources. Internet searching for competitive intelligence purposes is just another example of how the special library and information center can exploit new technology to give the organization it serves an information edge, permitting it to survive and prosper in difficult, rapidly changing global conditions.
 
 

BIBLIOGRAPHY

Arnold, Stephen E. and Erik S. Arnold. "Push Technology." Database 20, no. 4 (1997): 36-46.

Attaway Sr, Morris C. "A Review of Issues Related to Gathering and Assessing Competitive Intelligence." American Business Review 16, no. 1 (1998): 25-35.

Cahlin, Michael. "Pushing Information." PC World 16, no. 3 (1998): 166-71.

Gordon, Ian. Beat the Competition: How to Use Competitive Intelligence. Oxford: Basil Blackwell, 1989.

Greene, Frances H. "Competitive Intelligence and the Information Center." Special Libraries 79, no. 4 (1988): 285-95.

Gulliford, James. "The Challenge of Competitor Intelligence." Management Services 42, no. 1 (1998): 20-22.

Kahaner, Larry. Competitive Intelligence. New York: Simon Schuster, 1996.

Kassler, Helene S. "Mining the Internet for Competitive Intelligence." Online 21, no. 5 (1997): 34-45.

Kelly, John M. How to Check Out Your Competition: a Complete Plan for Investigating Your Market. New York: Wiley, 1987.

McCune, Jenny C. "Snooping on the Internet." Management Review 86, no. 7 (1997): 58-59.

McGonagle Jr., John J., and Carolyn M. Vella. Outsmarting the Competition: Practical Approaches to Finding and Outsmarting the Competition. New ed. London: McGraw-Hill, 1993.

Miller, Jerry. "Information Science and Competitive Intelligence: Possible Collaborators?" Bulletin of the American Society for Information Science 23, no. 3 (1996): 11-13.

Mount, Ellis. Special libraries and information centers: an introductory text. 3rd. ed. Washington, Special Libraries Association, 1995.

Piggott, Sylvia E. A. "New Roles for the Information Professional." Specialist 19, no. 9 (1996): 8.

Sawka, Kenneth A. "Demystifying Business Intelligence." Management Review 85, no. 10 (1996): 47-51.

Society of Professional Intelligence Professionals. "Frequently asked questions." Web page, 1998 [accessed 6 October 1998]. Available at http//www.scip.org/faq.htm.

Society of Professional Intelligence Professionals. "SCIP code of ethics for CI professionals." Web page, 1998 [accessed 6 October 1998]. Available at http//www.scip.org/ethics.htm.

Volpo, Dan, and Sylvia Piggott. "Surfing for Corporate Intelligence." Business Information Review 13, no. 1 (1996): 39-47.

Yovovich, B. G. "Browsers Get a Look at Rivals' Secrets." Marketing News 31, no. 23 (1997): 1, 6.
 

Note: this paper was prepared while I was teaching in the Library and Information Science Program, College of Graduate Studies, Kuwait University.
 
 
 

Updated:  1999 Nov. 29          Conditions of use
Return to opening page of Prof. McCarthy's site
Click here to send e-mail to Prof. McCarthy