TPCC load from "remote" node causes errors

[tylarb]

When I run a TPCC load from a node that is undersized and in a different zone as the cluster, I get an error like the following:

java.sql.BatchUpdateException: Batch entry 0 INSERT INTO ORDER_LINE VALUES (100, 5, 2397, 2, 23927, NULL, 6837.2998046875, 100, 5.0, 'xtidaggxzqrblbkwcmvgnsm'),(100, 5, 2397, 3, 45671, NULL, 3648.889892578125, 100, 5.0, 'sklxmxsteqlvdzfccapyopi'),(100, 5, 2397, 4, 66621, NULL, 582.9000244140625, 100, 5.0, 'nnknnvlwjytcvflupvgocqu'),(100, 5, 2397, 5, 68014, NULL, 5221.0498046875, 100, 5.0, 'qfzmrxbqxxkrydypflcjzgt'),(100, 5, 2397, 6, 19077, NULL, 6319.66015625, 100, 5.0, 'jehcyolsplncttdfiofqqmx'),(100, 5, 2397, 7, 37807, NULL, 3811.47998046875, 100, 5.0, 'lzkbqvpgesfsbinrqtkxrkz'),(100, 5, 2397, 8, 37202, NULL, 9973.919921875, 100, 5.0, 'ofnwjtyllvoigkywehkamuk'),(100, 5, 2397, 9, 9547, NULL, 5458.16015625, 100, 5.0, 'ociihrrrguwynnlibyqpymt'),(100, 5, 2397, 10, 66289, NULL, 8420.669921875, 100, 5.0, 'rrotdnfoihcwdkslgbqsuph'),(100, 5, 2397, 11, 16512, NULL, 4311.43994140625, 100, 5.0, 'unlnfmxvrjfqwwgqdykxaee'),(100, 5, 2397, 12, 52677, NULL, 9055.2900390625, 100, 5.0, 'gglxnvehtyhawifexnvfylq'),(100, 5, 2397, 13, 26786, NULL, 5675.43017578125, 100, 5.0, 'queozuwhdedivxsbcttvtro'),(100, 5, 2397, 14, 66286, NULL, 7765.259765625, 100, 5.0, 'jdqzwxshfonoorfiqcmcxzi'),(100, 5, 2397, 15, 58563, NULL, 6690.2998046875, 100, 5.0, 'heuuaaqmxxgbnnpmkyrnoco'),(100, 5, 2398, 1, 97237, NULL, 9282.240234375, 100, 5.0, 'bxyudrflxqoxcyyperpqlwg'),(100, 5, 2398, 2, 53710, NULL, 7994.89990234375, 100, 5.0, 'lalydrefatpabmwawafmmym'),(100, 5, 2398, 3, 94667, NULL, 1452.300048828125, 100, 5.0, 'hqdjscoeseddsyqzjftdzpl'),(100, 5, 2398, 4, 9154, NULL, 2855.889892578125, 100, 5.0, 'yhrnhszzihajvkcotedoiiz'),(100, 5, 2398, 5, 15724, NULL, 5966.52001953125, 100, 5.0, 'pucoasjugpsbvaxubretztb'),(100, 5, 2398, 6, 82044, NULL, 8115.56005859375, 100, 5.0, 'wndldacfveccjqhrzeioyxt'),(100, 5, 2398, 7, 9215, NULL, 4488.97998046875, 100, 5.0, 'tuqztdizmhnogxwunppizds'),(100, 5, 2398, 8, 72394, NULL, 9693.7099609375, 100, 5.0, 'xivtzoqlkmikxjourtwktlo'),(100, 5, 2398, 9, 20024, NULL, 871.1799926757812, 100, 5.0, 'vyykzakxyuqtwwtuwzhbvik'),(100, 5, 2398, 10, 38960, NULL, 7032.0400390625, 100, 5.0, 'xzqfaxpmwglttcxpwokjksw'),(100, 5, 2399, 1, 99063, NULL, 1821.1099853515625, 100, 5.0, 'ellkskwpxrmgfurpowdlbge'),(100, 5, 2399, 2, 68952, NULL, 2603.0, 100, 5.0, 'wypkuqhafupsrsfumsgmmmd'),(100, 5, 2399, 3, 501, NULL, 6561.85986328125, 100, 5.0, 'jjxujrenrdmesmvddevjkvd'),(100, 5, 2399, 4, 11525, NULL, 63.45000076293945, 100, 5.0, 'ueoejkxvpcbgibaktneqjrt'),(100, 5, 2399, 5, 6281, NULL, 2081.97998046875, 100, 5.0, 'kmzdjfpcdvlnxpujucsrxoh'),(100, 5, 2400, 1, 34045, NULL, 537.5900268554688, 100, 5.0, 'gavhhxtxrojjpnxcnrfdvuc'),(100, 5, 2400, 2, 67127, NULL, 8112.39990234375, 100, 5.0, 'chzkzjssbtacpslzcwawlqy'),(100, 5, 2400, 3, 44207, NULL, 7267.580078125, 100, 5.0, 'xnxxltqunnkdoteinwhiiwa'),(100, 5, 2400, 4, 2772, NULL, 5974.81005859375, 100, 5.0, 'czhudmrisvmgtbrxlnnhuhh'),(100, 5, 2400, 5, 91037, NULL, 2170.389892578125, 100, 5.0, 'yxmhosxysbupmuvgmzgzfnm'),(100, 5, 2400, 6, 66402, NULL, 4073.409912109375, 100, 5.0, 'wfsdyoryebjyxbrejwffnjc'),(100, 5, 2400, 7, 59733, NULL, 357.05999755859375, 100, 5.0, 'ngcvtlsxaknbngpmreiwces'),(100, 5, 2401, 1, 78337, NULL, 3864.949951171875, 100, 5.0, 'wxnfspimsghahaenvwogqip'),(100, 5, 2401, 2, 13291, NULL, 1420.9000244140625, 100, 5.0, 'cgtistplsyfkvxshvxtfkqt'),(100, 5, 2401, 3, 72470, NULL, 9503.76953125, 100, 5.0, 'onthvhadihatbbesqdtyjah'),(100, 5, 2401, 4, 95167, NULL, 2889.1298828125, 100, 5.0, 'uqynhumjnwjvwreubldjonq'),(100, 5, 2401, 5, 11385, NULL, 3797.1201171875, 100, 5.0, 'zhhvzhbbrpevqdiuejnrshm'),(100, 5, 2401, 6, 22631, NULL, 261.6600036621094, 100, 5.0, 'xrynlbnxeujbnewfocrwtpy'),(100, 5, 2401, 7, 39342, NULL, 7196.89990234375, 100, 5.0, 'eybmyimyvezchuquligzcgt'),(100, 5, 2401, 8, 64767, NULL, 140.00999450683594, 100, 5.0, 'bkgboafvgbwjkknmxigzfdg'),(100, 5, 2401, 9, 6054, NULL, 8708.3896484375, 100, 5.0, 'viikbglerjanrdwfqkcwioi'),(100, 5, 2401, 10, 41637, NULL, 5414.0498046875, 100, 5.0, 'plfvhokawyyqyoxksprwiyu'),(100, 5, 2401, 11, 58700, NULL, 164.94000244140625, 100, 5.0, 'euokdnsqvpurcwphayehrxz'),(100, 5, 2402, 1, 62540, NULL, 4443.990234375, 100, 5.0, 'soywsnbwnowwazmqltmuwqu'),(100, 5, 2402, 2, 72410, NULL, 4648.169921875, 100, 5.0, 'ygpybmcypdehrhexrwtlmtr'),(100, 5, 2402, 3, 83847, NULL, 159.74000549316406, 100, 5.0, 'vvbijmuubfddbjkpgudykrg'),(100, 5, 2402, 4, 13416, NULL, 4187.919921875, 100, 5.0, 'kyjxxwqabvuasbizpnehnri'),(100, 5, 2402, 5, 70453, NULL, 3106.409912109375, 100, 5.0, 'atahohmlpyujncoldindsmn'),(100, 5, 2402, 6, 19865, NULL, 2820.489990234375, 100, 5.0, 'ogaruxqsnrbfpaasbmajfgi'),(100, 5, 2403, 1, 56000, NULL, 7594.85986328125, 100, 5.0, 'dqzngqudfvrlyzpsmushkhp'),(100, 5, 2403, 2, 58758, NULL, 6756.89990234375, 100, 5.0, 'glxzalsghpynqfyeislobki'),(100, 5, 2403, 3, 97499, NULL, 3684.489990234375, 100, 5.0, 'lcnydgfvqlvlipqefxoiowl'),(100, 5, 2403, 4, 75713, NULL, 8723.490234375, 100, 5.0, 'xrqhjjmrwckxrgrauiwartp'),(100, 5, 2403, 5, 16686, NULL, 1411.050048828125, 100, 5.0, 'kxhfzcyytbsxmpgpmahzhgy'),(100, 5, 2403, 6, 58785, NULL, 6468.97998046875, 100, 5.0, 'kvsibsfyqnmlcfxetobxsud'),(100, 5, 2403, 7, 99537, NULL, 6104.75, 100, 5.0, 'fczizdhzpafccoeprzujqsx'),(100, 5, 2403, 8, 92818, NULL, 5439.240234375, 100, 5.0, 'dwdwvxwryhbbdstzxudemuu'),(100, 5, 2403, 9, 89776, NULL, 5000.97998046875, 100, 5.0, 'gsymfzczrsuxnlwfrlatodt'),(100, 5, 2403, 10, 56544, NULL, 8533.919921875, 100, 5.0, 'kjfiqpgecunwdjfvgcrmmew'),(100, 5, 2403, 11, 7283, NULL, 5715.419921875, 100, 5.0, 'asxgzmcvprzifugojtbodpi') was aborted: ERROR: Already present: [Already present (yb/consensus/retryable_requests.cc:241): Duplicate request]  Call getNextException to see other errors in the batch.
        at org.postgresql.jdbc.BatchResultHandler.handleError(BatchResultHandler.java:148)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2212)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:509)
        at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:853)
        at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1546)
        at com.oltpbenchmark.benchmarks.tpcc.TPCCLoader.loadOrders(TPCCLoader.java:701)
        at com.oltpbenchmark.benchmarks.tpcc.TPCCLoader$2.load(TPCCLoader.java:112)
        at com.oltpbenchmark.api.Loader$LoaderThread.run(Loader.java:64)
        at com.oltpbenchmark.util.ThreadUtil$LatchRunnable.run(ThreadUtil.java:343)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.postgresql.util.PSQLException: ERROR: Already present: [Already present (yb/consensus/retryable_requests.cc:241): Duplicate request]
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2468)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2211)
        ... 10 more

Here is the cluster setup and tpcc load module:
cluster:

US-West
3 node
i3.8xlarge, 32core, ~240Gb ram

Node which tpcc is running from:

US-east
c5.xlarge 4vcpus, 8GB RAM

Issue happens with small numbers of threads.
Tested and failing with 4, 8, 48 threads

100 warehouses

Command for example:

time ./tpccbenchmark --create=true --load=true --nodes=[node1, node2, node3]  --warehouses=100   --loaderthreads 8

The tpcc benchmark success but with the above errors.

[tylarb]

@psudheer21

[tylarb]

I've had this issue again, but this time with a node in the same zone.

cluster:

us-west-2 / us-west-2c
3 node
3.8xlarge, 32core, ~240Gb ram

Node which tpcc is running from:

us-west-2 / us-west-2cc5.xlarge
48CPU, ~96GB ram

[tylarb]

We tried with a load from same setup as just above, but this time loading to single node, and we get hundreds of these errors:

        at com.oltpbenchmark.benchmarks.tpcc.TPCCLoader$2.load(TPCCLoader.java:112)
        at com.oltpbenchmark.api.Loader$LoaderThread.run(Loader.java:64)
        at com.oltpbenchmark.util.ThreadUtil$LatchRunnable.run(ThreadUtil.java:343)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.postgresql.util.PSQLException: ERROR: Catalog Version Mismatch: A DDL occurred while processing this query. Try Again.

[tylarb]

Error already present persists on data load to single node as well.