Take too much time with large amount of dataset(over 5,000,000)

[minku1024]

Query : select Meta().id, * from TestDB

DB : TestDB -> has 5,120,000 documents.

Take too much time(almost 5 hrs) to get result.

I think it's problem of fetchsize(Resultset take all dataset at once).

Is there something solution to avoid it?

code spend almost all time in CharArrayBuffer.append and CharArrayBuffer.expand.

    at org.apache.http.util.CharArrayBuffer.expand(CharArrayBuffer.java:62)
    at org.apache.http.util.CharArrayBuffer.append(CharArrayBuffer.java:92)
    at org.apache.http.util.EntityUtils.toString(EntityUtils.java:248)
    at org.apache.http.util.EntityUtils.toString(EntityUtils.java:291)
    at com.couchbase.jdbc.core.ProtocolImpl.handleResponse(ProtocolImpl.java:412)
    at com.couchbase.jdbc.core.ProtocolImpl.query(ProtocolImpl.java:366)
    at com.couchbase.jdbc.CBStatement.executeQuery(CBStatement.java:75)

[j-larson]

There is no realistic way to prevent this. If you need to see results faster, you could make multiple queries using LIMIT and OFFSET clauses to get the results piece by piece. In that case it would make sense to add an ORDER BY clause. You might also consider whether you need the whole objects you are fetching, or just a few fields. A more specific SELECT clause should make things faster.

Going beyond that, there is the Couchbase Java SDK, which may be what you are looking for. But fetching and parsing and building objects for 5 million objects is always going to be slow.

If you are planning to put the results in a file somewhere, you might try the Couchbase command-line tools.