Fast Text Analysis Using Symbol Enumeration and Hashing Methodology

Abstract

This paper is focusing on reducing the time for text processing operations by taking the advantage of enumerating each string using the multi hashing methodology. Text analysis is an important subject for any system that deals with strings (sequences of characters from an alphabet) and text processing (e.g., word-processor, text editor and other text manipulation systems). Many problems have been arisen when dealing with string operations which consist of an unfixed number of characters (e.g., the execution time); this due to the overhead embedded-operations (like, symbols matching and conversion operations). The execution time largely depends on the string characteristics; especially its length (i.e., the number of characters consisting the strings plus the number of words in the sentence). In other words, the variable length of strings is an obstacle to achieve processing uniformity when manipulating strings. Many of string matching algorithms were introduced in the literature to deal with fixed length of characters of each string. In this paper, some test results are provided for a number of string operations (such as, simple string matching, hashing indexing systems, stop-words collection and text extractions). To understand the advantage of the proposed method, these operations were applied on different sizes of text files. A comparison is made with the results of using traditional methods that deal with strings only. The overall results demonstrate the positive effectiveness of the proposed approach.