Influx db java client batch does not write to DB #378

IdanFridman · 2017-10-24T11:24:11Z

I am trying to write points to influxDB using their Java client.
Batch is important to me.

If I use the influxDB.enableBatch with influxDB.write(Point) no data is inserted.
If I use the BatchPoints and influxDB.write(batchPoints) - data is inserted successfully.

Both code samples are taken from: https://github.com/influxdata/influxdb-java/tree/influxdb-java-2.7

    InfluxDB influxDB = InfluxDBFactory.connect(influxUrl, influxUser, influxPassword);
    influxDB.setDatabase(dbName);
    influxDB.setRetentionPolicy("autogen");

    // Flush every 2000 Points, at least every 100ms
    influxDB.enableBatch(2000, 100, TimeUnit.MILLISECONDS);
    influxDB.write(Point.measurement("cpu")
            .time(System.currentTimeMillis(), TimeUnit.MILLISECONDS)
            .addField("idle", 90L)
            .addField("user", 9L)
            .addField("system", 1L)
            .build());

    Query query = new Query("SELECT idle FROM cpu", dbName);
    QueryResult result = influxDB.query(query);

Returns nothing.

    BatchPoints batchPoints = BatchPoints.database(dbName).tag("async", "true").build();
    Point point1 = Point
            .measurement("cpu")
            .tag("atag", "test")
            .addField("idle", 90L)
            .addField("usertime", 9L)
            .addField("system", 1L)
            .build();
    batchPoints.point(point1);
    influxDB.write(batchPoints);
    Query query = new Query("SELECT * FROM cpu ", dbName);
    QueryResult result = influxDB.query(query);

This returns data successfully.

As mentioned, I need the first way to function.
How can I achieve that?

versions:
influxdb-1.3.6
influxdb-java:2.7

Thanks

The text was updated successfully, but these errors were encountered:

majst01 · 2017-10-24T13:35:27Z

of course, because:

    // Flush every 2000 Points, at least every 100ms
    influxDB.enableBatch(2000, 100, TimeUnit.MILLISECONDS);

Will tell influxdb-java to wait 100ms or 2000Points, whichever occurs first before sending the points to influxdb.
To get your points immediatly after writing them to a batching enabled influxdb-java instance, you can call influxDB.flush(), which writes the points to influxdb, before executing the query.

IdanFridman · 2017-10-24T14:14:08Z

@majst01 Sorry but I dont understand:

Will tell influxdb-java to wait 100ms or 2000Points

After another 100ms it should have flushed the points already. we were waiting couple of seconds after the .write() method and even invoked write() couple of times but still the database remains empty.

majst01 · 2017-10-24T16:52:54Z

ok, add more logging, set:

influxDB.setLogLevel(Level.Debug)

and post the output of you run.

idobarash · 2017-10-25T06:55:52Z

Hello, I am working with @IdanFridman.

These are the logs. I also printed the result after querying:

Oct 25, 2017 9:48:48 AM okhttp3.internal.platform.Platform log
INFO: --> GET https://<url>:<port>/query?u=<username>&p=<password>&db=<db>&q=SELECT+*+FROM+cpu http/1.1
Oct 25, 2017 9:48:48 AM okhttp3.internal.platform.Platform log
INFO: --> END GET
Oct 25, 2017 9:48:48 AM okhttp3.internal.platform.Platform log
INFO: --> POST https://<url>:<port>/write?u=<username>&p=<password>&db=<db>&rp=autogen&precision=n&consistency=one http/1.1
Oct 25, 2017 9:48:48 AM okhttp3.internal.platform.Platform log
INFO: Content-Type: text/plain; charset=utf-8
Oct 25, 2017 9:48:48 AM okhttp3.internal.platform.Platform log
INFO: Content-Length: 51
Oct 25, 2017 9:48:48 AM okhttp3.internal.platform.Platform log
INFO: 
Oct 25, 2017 9:48:48 AM okhttp3.internal.platform.Platform log
INFO: cpu idle=90i,system=1i,user=9i 1508914128778000000

Oct 25, 2017 9:48:48 AM okhttp3.internal.platform.Platform log
INFO: --> END POST (51-byte body)
Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: <-- 200 OK https://<url>:<port>/query?u=<username>&p=<password>&db=<db>&q=SELECT+*+FROM+cpu (316ms)
Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: Content-Type: application/json
Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: Request-Id: 8e1cfb48-b950-11e7-929f-000000000000
Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: X-Influxdb-Version: 1.3.5-6-g6f43c63
Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: Date: Wed, 25 Oct 2017 06:48:49 GMT
Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: Transfer-Encoding: chunked
Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: Server: Aiven InfluxDB
Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: Strict-Transport-Security: max-age=15768000
Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: 
Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: {"results":[{"statement_id":0}]}

Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: <-- END HTTP (33-byte body)
{"results":[{}]}
09:48:49.099 INFO  i.v.e.w.h.i.LoggerHandlerImpl:153 - POST /api/v1/test 200 4 - 321 ms
Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: <-- 204 No Content https://<url>:<port>/write?u=<username>&p=<password>&db=<db>&rp=autogen&precision=n&consistency=one (316ms)
Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: Content-Type: application/json
Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: Request-Id: 8e256541-b950-11e7-92a0-000000000000
Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: X-Influxdb-Version: 1.3.5-6-g6f43c63
Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: Date: Wed, 25 Oct 2017 06:48:49 GMT
Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: Server: Aiven InfluxDB
Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: Strict-Transport-Security: max-age=15768000
Oct 25, 2017 9:48:49 AM okhttp3.internal.platform.Platform log
INFO: <-- END HTTP

Second point is the same:


Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: --> GET https://<url>:<port>/query?u=<username>&p=<password>&db=<db>&q=SELECT+*+FROM+cpu http/1.1
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: --> END GET
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: --> POST https://<url>:<port>/write?u=<username>&p=<password>&db=<db>&rp=autogen&precision=n&consistency=one http/1.1
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: Content-Type: text/plain; charset=utf-8
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: Content-Length: 51
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: 
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: cpu idle=90i,system=1i,user=9i 1508914326257000000

Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: --> END POST (51-byte body)
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: <-- 200 OK https://<url>:<port>/query?u=<username>&p=<password>&db=<db>&q=SELECT+*+FROM+cpu (673ms)
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: Content-Type: application/json
{"results":[{}]}
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: Request-Id: 04083bf0-b951-11e7-92a1-000000000000
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: X-Influxdb-Version: 1.3.5-6-g6f43c63
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: Date: Wed, 25 Oct 2017 06:52:06 GMT
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: Transfer-Encoding: chunked
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: Server: Aiven InfluxDB
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: Strict-Transport-Security: max-age=15768000
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: 
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: {"results":[{"statement_id":0}]}

Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: <-- END HTTP (33-byte body)
09:52:06.934 INFO  i.v.e.w.h.i.LoggerHandlerImpl:153 - POST /api/v1/test 200 4 - 677 ms
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: <-- 204 No Content https://<url>:<port>/write?u=<username>&p=<password>&db=<db>&rp=autogen&precision=n&consistency=one (607ms)
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: Content-Type: application/json
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: Request-Id: 0408f126-b951-11e7-92a2-000000000000
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: X-Influxdb-Version: 1.3.5-6-g6f43c63
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: Date: Wed, 25 Oct 2017 06:52:06 GMT
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: Server: Aiven InfluxDB
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: Strict-Transport-Security: max-age=15768000
Oct 25, 2017 9:52:06 AM okhttp3.internal.platform.Platform log
INFO: <-- END HTTP

majst01 · 2017-10-25T06:57:44Z

I can only see selects in the logs, no writes ! So no surprise that not result returns.

idobarash · 2017-10-25T06:58:42Z

what about?
INFO: <-- 204 No Content https://<url>:<port>/write?u=<username>&p=<password>&db=<db>&rp=autogen&precision=n&consistency=one (607ms)

idobarash · 2017-10-25T07:00:50Z

And if there are no writes, what did we do wrong in the code?

majst01 · 2017-10-25T07:02:47Z

influxdb.flush() was not fired because either 100ms, nor 2000 points have been passed.

idobarash · 2017-10-25T07:07:19Z

How can it be... I waited some times before sending events - 100MS must have passed. I retried now.

It creates the MEASUREMENT CPU but it inserts nothing.

show MEASUREMENTS:
name: measurements
name
----
cpu

SELECT * FROM cpu

bkdonline · 2017-10-25T07:47:46Z

Can you add

Thread.sleep(200); after your write, and before you select?

InfluxDB influxDB = InfluxDBFactory.connect(influxUrl, influxUser, influxPassword);
influxDB.setDatabase(dbName);
influxDB.setRetentionPolicy("autogen");

// Flush every 2000 Points, at least every 100ms
influxDB.enableBatch(2000, 100, TimeUnit.MILLISECONDS);
influxDB.write(Point.measurement("cpu")
        .time(System.currentTimeMillis(), TimeUnit.MILLISECONDS)
        .addField("idle", 90L)
        .addField("user", 9L)
        .addField("system", 1L)
        .build());

Thread.sleep(200); //add this
Query query = new Query("SELECT idle FROM cpu", dbName);
QueryResult result = influxDB.query(query);

idobarash · 2017-10-25T08:01:53Z

Added the Thread.sleep(200).
Unfortunantly it is the same:

`Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: --> POST https://:/write?u=&p=&db=&rp=autogen&precision=n&consistency=one http/1.1
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: Content-Type: text/plain; charset=utf-8
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: Content-Length: 51
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO:
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: cpu idle=90i,system=1i,user=9i 1508918378183000000

Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: --> END POST (51-byte body)
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: --> GET https://:/query?u=&p=&db=&q=SELECT++FROM+cpu http/1.1
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: --> END GET
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: <-- 200 OK https://:/query?u=&p=&db=&q=SELECT++FROM+cpu (332ms)
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: Content-Type: application/json
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: Request-Id: 739b3060-b95a-11e7-92c6-000000000000
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: X-Influxdb-Version: 1.3.5-6-g6f43c63
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: Date: Wed, 25 Oct 2017 07:59:39 GMT
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: Transfer-Encoding: chunked
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: Server: Aiven InfluxDB
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: Strict-Transport-Security: max-age=15768000
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO:
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: {"results":[{"statement_id":0}]}

Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: <-- END HTTP (33-byte body)
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: <-- 204 No Content https://:/write?u=&p=&db=&rp=autogen&precision=n&consistency=one (411ms)
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: Content-Type: application/json
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: Request-Id: 739cc68e-b95a-11e7-92c7-000000000000
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: X-Influxdb-Version: 1.3.5-6-g6f43c63
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: Date: Wed, 25 Oct 2017 07:59:39 GMT
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: Server: Aiven InfluxDB
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: Strict-Transport-Security: max-age=15768000
Oct 25, 2017 10:59:38 AM okhttp3.internal.platform.Platform log
INFO: <-- END HTTP
{"results":[{}]}
10:59:38.747 INFO i.`

IdanFridman · 2017-11-05T13:39:42Z

I found out that the batch-API doesnt fully work. you can use batch insert but its configuration isnt really working.
for example if you set this:
influxDB.enableBatch(4, 5000, TimeUnit.MILLISECONDS);

it wont affect anything.

the only way the batch is working would be when you execute influxDB.write(batchPoints);
on that point the data will be written to the database(with all the batchPoints at once). the configuration above doesnt affect anyhow on the actual batching execution against the database

bkdonline · 2017-11-05T16:13:27Z

The batch API works as expected for me. Just reporting.

IdanFridman · 2017-11-05T16:27:03Z

Guys, I found the issue. and I couldnt find it documented anywhere..
In order to really enjoy the batch-api we must use the right write () method signature

you have 2 methods that will actually use the batching API(if enable)

1.public void write(final String database, final String retentionPolicy, final Point point);
2. public void write(final int udpPort, final Point point);

only those has inside (InfluxDBImpl) this line:

if (this.batchEnabled.get())

in your document you specify only
public void write(final BatchPoints batchPoints);

that write() method doesnt really queue anything in it and therefor no usage of:
influxDB.enableBatch(4, 5000, TimeUnit.MILLISECONDS);

at all!

fmachado · 2017-11-07T09:47:16Z

I agree with you the current documentation is not clear enough about what will happen when you enableBatch, write a few points and immediately query the database without (1) waiting enough time or (2) writing enough points to trigger the flush.

You are also correct when you say that not all methods will use the "batching mechanism": there is no mention to it on javadocs and you have to inspect the source-code (as you did) to understand how to properly use the built-in batching mechanism.

Do you think you could make a pull-request with some documentation improvements?

About the log output that was posted here:

I could not reproduce any incorrect behavior: when I used enableBatch and waited enough time to trigger the flush, my points were properly written. There is also a test case for these scenarios: org.influxdb.impl.BatchProcessorTest
Every successful write to InfluxDB will result with HTTP code 204. Check https://docs.influxdata.com/influxdb/v1.3/guides/writing_data/#http-response-summary for more information.

IdanFridman · 2017-11-09T08:17:39Z

@fmachado I dont think it's just document. I would change the interface. (e.g writeBatch or something similar). people who using that code should understand according the methods the expected behavior

dubsky · 2017-11-29T08:41:25Z

I think that batching should be a configuration option only - it would be nice if the user's code would work independently on the fact batching was enabled or not.

majst01 · 2017-11-29T08:47:10Z

You have the choice, if do not call influxdb.enableBatch() your writes will be synchronous.

dubsky · 2017-11-29T09:25:39Z

That is true. On the other hand, you may call influxdb.enableBatch() but some write calls would be always executed synchronously. The mentioned write(batchPoints) and the writes that consume directly line protocol lines are the case. I am not saying it is big deal but you can see that some people might get confused.

fmachado · 2017-11-30T08:31:22Z

That's why I've already mentioned before about simplifying our API, including deprecation of write(...) methods for protocol lines (I really doubt there is anyone using it) and improving the docs. We can talk more about this in our call tomorrow.

fmachado added the kind/bug label Nov 7, 2017

fmachado added the help wanted label Nov 7, 2017

fmachado mentioned this issue Nov 9, 2017

Why we must collect points to use the batch api? #381

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Influx db java client batch does not write to DB #378

Influx db java client batch does not write to DB #378

IdanFridman commented Oct 24, 2017

majst01 commented Oct 24, 2017

IdanFridman commented Oct 24, 2017

majst01 commented Oct 24, 2017

idobarash commented Oct 25, 2017 •

edited

Loading

majst01 commented Oct 25, 2017

idobarash commented Oct 25, 2017 •

edited

Loading

idobarash commented Oct 25, 2017

majst01 commented Oct 25, 2017

idobarash commented Oct 25, 2017 •

edited

Loading

bkdonline commented Oct 25, 2017

idobarash commented Oct 25, 2017

IdanFridman commented Nov 5, 2017 •

edited

Loading

bkdonline commented Nov 5, 2017

IdanFridman commented Nov 5, 2017 •

edited

Loading

fmachado commented Nov 7, 2017

IdanFridman commented Nov 9, 2017

dubsky commented Nov 29, 2017

majst01 commented Nov 29, 2017

dubsky commented Nov 29, 2017

fmachado commented Nov 30, 2017

Influx db java client batch does not write to DB #378

Influx db java client batch does not write to DB #378

Comments

IdanFridman commented Oct 24, 2017

majst01 commented Oct 24, 2017

IdanFridman commented Oct 24, 2017

majst01 commented Oct 24, 2017

idobarash commented Oct 25, 2017 • edited Loading

majst01 commented Oct 25, 2017

idobarash commented Oct 25, 2017 • edited Loading

idobarash commented Oct 25, 2017

majst01 commented Oct 25, 2017

idobarash commented Oct 25, 2017 • edited Loading

bkdonline commented Oct 25, 2017

idobarash commented Oct 25, 2017

IdanFridman commented Nov 5, 2017 • edited Loading

bkdonline commented Nov 5, 2017

IdanFridman commented Nov 5, 2017 • edited Loading

fmachado commented Nov 7, 2017

IdanFridman commented Nov 9, 2017

dubsky commented Nov 29, 2017

majst01 commented Nov 29, 2017

dubsky commented Nov 29, 2017

fmachado commented Nov 30, 2017

idobarash commented Oct 25, 2017 •

edited

Loading

idobarash commented Oct 25, 2017 •

edited

Loading

idobarash commented Oct 25, 2017 •

edited

Loading

IdanFridman commented Nov 5, 2017 •

edited

Loading

IdanFridman commented Nov 5, 2017 •

edited

Loading