org.carrot2.source

Class MultipageSearchEngine

java.lang.Object

org.carrot2.core.ProcessingComponentBase

org.carrot2.source.SearchEngineBase

org.carrot2.source.MultipageSearchEngine

All Implemented Interfaces:

IDocumentSource, IProcessingComponent

Direct Known Subclasses:

Bing3DocumentSource, GoogleDocumentSource, IdolDocumentSource, OpenSearchDocumentSource

@Bindable
public abstract class MultipageSearchEngine
extends SearchEngineBase

A base class facilitating implementation of IDocumentSources wrapping external search engines with remote/ network-based interfaces. This class implements helper methods for concurrent querying of search services that limit the number of search results returned in one request.

See Also:

SimpleSearchEngine

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	MultipageSearchEngine.SearchEngineResponseCallable An implementation of Callable that increments page request count statistics before the actual search is made.
static class	MultipageSearchEngine.SearchMode Search mode for data source components that implement parallel request to some search service.
protected static class	MultipageSearchEngine.SearchRange A single result window to fetch.

Field Summary

Fields

Modifier and Type	Field and Description
MultipageSearchEngine.SearchMode	searchMode Search mode defines how fetchers returned from createFetcher(org.carrot2.source.MultipageSearchEngine.SearchRange) are called.

Fields inherited from class org.carrot2.source.SearchEngineBase

compressed, documents, POSTPROCESSING, query, results, resultsTotal, SERVICE, start, statistics

Constructor Summary

Constructors

Constructor and Description
MultipageSearchEngine()

Method Summary

Methods

Modifier and Type	Method and Description
protected void	collectDocuments(Collection<Document> collector, SearchEngineResponse[] responses) Collects documents from an array of search engine's responses.
protected abstract Callable<SearchEngineResponse>	createFetcher(MultipageSearchEngine.SearchRange bucket) Subclasses should override this method and return a Callable instance that fetches search results in the given range.
protected void	process(MultipageSearchEngineMetadata metadata, ExecutorService executor) Run a request the search engine's API, setting documents to the set of returned documents.
protected SearchEngineResponse[]	runQuery(String query, int start, int results, MultipageSearchEngineMetadata metadata, ExecutorService executor) This method implements the logic of querying a typical search engine.

Methods inherited from class org.carrot2.source.SearchEngineBase

afterFetch, clean, urlEncode

Methods inherited from class org.carrot2.core.ProcessingComponentBase

afterProcessing, beforeProcessing, dispose, getContext, getSharedExecutor, init, process

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.carrot2.core.IProcessingComponent

afterProcessing, beforeProcessing, dispose, init, process

Field Detail

searchMode

@Processing
@Input
@Attribute(key="search-mode")
@Level(value=ADVANCED)
@Label(value="Search Mode")
@Group(value="Data source paging")
public MultipageSearchEngine.SearchMode searchMode

Search mode defines how fetchers returned from createFetcher(org.carrot2.source.MultipageSearchEngine.SearchRange) are called.

See Also:

MultipageSearchEngine.SearchMode

Constructor Detail

MultipageSearchEngine

public MultipageSearchEngine()

Method Detail

process

protected void process(MultipageSearchEngineMetadata metadata,
           ExecutorService executor)
                throws ProcessingException

Run a request the search engine's API, setting documents to the set of returned documents.

Throws:

ProcessingException

createFetcher

protected abstract Callable<SearchEngineResponse> createFetcher(MultipageSearchEngine.SearchRange bucket)

Subclasses should override this method and return a Callable instance that fetches search results in the given range.

Note the query (if any is required) should be passed at the concrete class level. We are not concerned with it here.

Parameters:

bucket - The search range to fetch.

collectDocuments

protected final void collectDocuments(Collection<Document> collector,
                    SearchEngineResponse[] responses)

Collects documents from an array of search engine's responses.

runQuery

protected final SearchEngineResponse[] runQuery(String query,
                              int start,
                              int results,
                              MultipageSearchEngineMetadata metadata,
                              ExecutorService executor)
                                         throws ProcessingException

This method implements the logic of querying a typical search engine. If the number of requested results is higher than the number of results on one response page, then multiple (possibly concurrent) requests are issued via the provided ExecutorService.

Throws:

ProcessingException