SearchPDF for Millions of Documents

<body topmargin=0 leftmargin=0 marginheight=0 marginwidth=0> <table cellpadding=0 cellspacing=0 border=0><tr> <td valign="top" rowspan="2" width=10% BGCOLOR="#FFFBF0" TEXT="#080000" background="354767e3.jpg"> <div> <A HREF="index.htm" TARGET="_top" TITLE="SearchPDF Home Page"><U>Home</U></A></div> <div> <A HREF="products.htm" TARGET="_top" TITLE="SearchPDF Products"><U>Products</U></A></div> <div> <A HREF="store.htm" TARGET="_top" TITLE="SearchPDF Product Store"><U>Store</U></A></div> <div> <A HREF="studies.htm" TARGET="_top" TITLE="Case Studies"><U>Studies</U></A></div> <div> <A HREF="papers.htm" TARGET="_top" TITLE="Papers"><U>Papers</U></A></div> <div> <A HREF="partners.htm" TARGET="_top" TITLE="Partners"><U>Partners</U></A></div> <div> <A HREF="about.htm" TARGET="_top" TITLE="About SearchPDF"><U>About</U></A></div> <bR> <div> <B>PRODUCTS</B></div> <div> <A HREF="search.htm" TARGET="_top" TITLE="SearchPDF"><U>Search</U></A></div> <div> <A HREF="metamaker.htm" TARGET="_top" TITLE="PDFMetamaker"><U>Metamaker</U></A></div> <div> <A HREF="publish.htm" TARGET="_top" TITLE="PDFPublish"><U>Publish</U></A></div> <div> <A HREF="firstpage.htm" TARGET="_top" TITLE="JRA FirstPage"><U>FirstPage</U></A></div> <div> <A HREF="compress.htm" TARGET="_top" TITLE="Compression"><U>Compress</U></A></div> <div> <A HREF="support.htm" TARGET="_top" TITLE="Support"><U>Support</U></A></div> <div> <A HREF="services.htm" TARGET="_top" TITLE="Services"><U>Services</U></A></div> <bR> </td> <td valign="top" height=130 BGCOLOR="#FFFFFF" TEXT="#080000"> <div> <IMG SRC="96be83c0.jpg" border=0 width="488" height="60" ALT="SearchPDF.com" ALIGN="BOTTOM" HSPACE="0" VSPACE="0"></div> <div> <A HREF="index.htm" TARGET="_top" TITLE="SearchPDF Home Page"><U>Home</U></A><FONT SIZE="2" COLOR="#111111" FACE="Symbol" >   ·   </FONT><A HREF="products.htm" TARGET="_top" TITLE="SearchPDF Products"><U>Products</U></A><FONT SIZE="2" COLOR="#111111" FACE="Symbol" >   ·   </FONT><A HREF="store.htm" TARGET="_top" TITLE="SearchPDF Product Store"><U>Store</U></A><FONT SIZE="2" COLOR="#111111" FACE="Symbol" >   ·   </FONT><A HREF="studies.htm" TARGET="_top" TITLE="Case Studies"><U>Studies</U></A><FONT SIZE="2" COLOR="#111111" FACE="Symbol" >   ·   </FONT><A HREF="papers.htm" TARGET="_top" TITLE="Papers"><U>Papers</U></A><FONT SIZE="2" COLOR="#111111" FACE="Symbol" >   ·   </FONT><A HREF="partners.htm" TARGET="_top" TITLE="Partners"><U>Partners</U></A><FONT SIZE="2" COLOR="#111111" FACE="Symbol" >   ·   </FONT><A HREF="about.htm" TARGET="_top" TITLE="About SearchPDF"><U>About</U></A></div> <div> <A HREF="papers.htm" TARGET="_top" TITLE="Papers"><U>Back to Papers</U></A></div> </td> <td valign="top" rowspan="2" width=10% BGCOLOR="#FFFBF0" TEXT="#080000" bgproperties="fixed" background="354767e3.jpg"> <bR> <div> Questions?</div> <div> <A HREF="mailto:sales@searchpdf.com" TARGET="_top" TITLE="mailto:sales@searchpdf.com"><U><IMG SRC="24a3c150.png" border=0 width="60" height="21" ALT="E-Mail SearchPDF" ALIGN="BOTTOM" HSPACE="0" VSPACE="0"></U></A></div> <div> (610) 983-3682</div> </td> </tr><tr> <td valign="top" height=350 BGCOLOR="#FFFFFF" TEXT="#080000" bgproperties="fixed" background="16880803.png"> <div> <TABLE BORDER="1" CELLPADDING="12" CELLSPACING="1" WIDTH="588" BORDERCOLORLIGHT="#C0C0C0" BORDERCOLORDARK="#808080" FRAME="BOX" RULES="ALL" HSPACE="0" VSPACE="0" BGCOLOR=#FFFBF0 > <tR> <tD VALIGN=TOP HEIGHT= 2025 WIDTH="558"><div> <B>SearchPDF for Millions of Documents</B></div> <bR> <div> <B>Overview</B></div> <div> SearchPDF can be used to create a searchable index of millions of documents when the documents are correctly organized for optimal performance. This paper discusses the organization of documents and search-indexes when the document collection is comprised of millions of pages.</div> <bR> <div> SearchPDF uses dtSearch Indexes to perform a search. dtSearch Indexes are generated in a batch process before making the collection available for searching.</div> <bR> <div> <B>dtSearch Index Size Limits</B></div> <div> There are some practical limits to the size of a dtSearch Index. A single dtSearch index can hold up to 4-8 Gigabytes of text. Technically, the limit is that none of the individual *.ix files in a dtSearch index can exceed 2 Gigabytes in size. When more than 4-8 Gigabytes are to be indexed, we recommend that multiple indexes be built. One common approach is to have one or more "archive" indices, organized by date (e.g., one for 1998 documents, one for 1999 documents, etc.) and a "Current" index with the documents that are still being added.</div> <bR> <div> <B>Maximum number of files/Folder structure</B></div> <div> File access under Windows 9X (using the Fat32 file system) can become quite slow when folders contain more than a few thousand documents. Therefore, for document collections containing hundreds of thousands of documents, we recommend that the documents be organized into sub-folders, preferably with less than 1,000 documents in each folder (this number will vary depending on your Windows environment). The maximum number of files and subfolders under a single folder using Fat32 is 64,000.</div> <div> Fortunately, today you will most likely be using WindowsXP or Windows2000 with the NTFS file system, and NTFS does not have the limits on number of files in a folder that FAT32 has. The maximum number of files and subfolders in a single folder under NTFS is about 6 billion. Still, using NTFS, we recommend that you limit the number of document files in one folder to about 200,000 for best performance.</div> <bR> <div> <B>Divide and Conquer</B></div> <div> One user has reported building a single dtSearch Index for more than 2 million documents. However, this dtSearch Index took a long time to generate (and we assume that the maximum size limit on a dtSearch Index was not reached, otherwise it would not have worked).  For better performance, and to avoid reaching dtSearch Index size limits, it is better to create muliple, smaller dtSearch indexes, and then search these multiple dtSearch indexes concurrently. There is no limit on the number of dtSearch indexes that can be searched concurrently in SearchPDF.</div> <bR> <div> <B>An Example Strategy for a 5 Million Document Collection</B></div> <div> OK. You have 5 million documents.  This is not a problem if you organize the documents and search-indexes correctly, as discussed below.</div> <bR> <div> <B>Logical Division of Documents and Indexes</B></div> <div> The first thing to do is to determine if these documents can be logically divided into collections.  For example, if the collection is shipping receipts and invoices, you really have two collections or sub-collections: receipts and invoices. Now store each sub-collection under a separate folder tree.</div> <bR> <div> Next, see if each sub-collection can be further logically divided. In our example, we might divide invoices by year.  In this case, we can now, under the Invoices folder, have further "year" subfolders, and the invoice documents can now be sorted into the year subfolders.</div> <bR> <div> Once you have complete the logical division of the document collection, it is time to check the number of files in each document subfolder.  It the total documents in a subfolder exceeds 200,000, then it will be good to further divide these documents purely because of volume into further subfolders, perhaps named "vol1", "vol2", etc.</div> <bR> <div> Now that we have our 5 millions documents organized in a folder tree, we can define the multiple search-indexes that will be generated for the collection. Define the dtSearch Indexes so that they correspond to the logical subdivision of documents. In our example, we will then have a "Invoices-2001-Part A" index, a "Invoices - 2001 - Part B" index, etc.</div> <bR> <div> <B>Division of Documents and Indexes based only on Volume</B></div> <div> If your documents cannot be logically divided into groups, it is fine to divide the documents based only on volume as a criteria.</div> <bR> <div> In another example, let's say that our 5 million documents are all "discovery documents". There is no logical subdivision of these documents. </div> <bR> <div> Here is what we can do.  We will create 25 subfolders named "001", "002", etc. We will place 200,000 documents into each of these subfolders. Now, we will generate a dtSearch Index for each of these subfolders, named "001", "002", etc.</div> <bR> <div> Now on our SearchPDF search page, we will be searching 25 indexes concurrently, the indexes named "001" through "025".</div> <bR> <div> <B>Summary</B></div> <div> SearchPDF can be used to perform search and retrieval operations on millions of documents, as long as the folder structure for the storage of these documents is organized correctly, and as long as multiple search-indexes are used, so as not to exceed the size-limits for a single dtSearch Index.</div> <bR> <div> <B>Public Example</B></div> <div> There is one website that is public and accessible, and that searches millions of documents using dtSearch Indexes.</div> <bR> <div> <A HREF="http://tobaccodocuments.org" TARGET="_top" TITLE="http://tobaccodocuments.org"><U>http://tobaccodocuments.org</U></A></div> <bR> <div> It's a collection of over 4 million internal tobacco industry documents, with citations and OCR. It's a fairly custom configuration (the searches run in parallel across several machines, the data is all in xml, etc.), but it does show the power of dtSearch with large collections.</div> <bR> <div> <B>Multiple Server Configuration for Performance</B></div> <bR> <div> Search query processsing can be configured for multiple servers in SearchPDF, and this approach is recommended for document collections that exceed 5 million documents. In this design, the search query that is specified by the user is submitted to multiple servers, each of which supports search-indexes for a subset of the complete document collection. The search results are then aggregated on the primary server and returned to the user in aggregated form.</div> <bR> <div> The purpose of using multiple servers is for performance reasons. It is a "divide and conquer" strategy for rapid, parallel processing of search queries.</div> <bR> <bR> <bR> <bR> <bR> <bR> <bR> </tD> </tR> </TABLE></div> <bR> <bR> <bR> </td> </tr></table></body>