The exception occurs when Full Text indexing is disabled on the database and the number of items in the database is huge.
When we try to execute a FTSearch method on such a database a temporary Full Text index is created on the database. The number of items which can be indexed by this temporary index is set to some default value and if the number of items in the database exceeds this default value then we get this exception. We can override this setting in server's notes.ini using the following parameter
TEMP_INDEX_MAX_DOC=number
Once we set this number to a value greater then number of items in the database the API should start working.
More details can be found here.
~Abhishek