Fio reports higher runtime than the configured value

[raj-prince]

Please acknowledge the following before creating a ticket

• I have read the GitHub issues section of REPORTING-BUGS.

Description of the bug:
Fio reports higher runtime than the configured one

Environment: Ubuntu 20

fio version: fio-3.16

Reproduction steps
Fio job specs - https://github.com/GoogleCloudPlatform/gcsfuse/blob/ce0a93d1ddfe33ae1bfeabff98eba7befab03a9f/perfmetrics/scripts/job_files/seq_rand_read_write.fio

Ouput json:

{
      "jobname": "2_thread",
      "groupid": 1,
      "error": 0,
      "eta": 0,
      "elapsed": 3043,
      "job options": {
        "startdelay": "310",
        "bs": "16k",
        "directory": "gcs/256kb",
        "filesize": "256k",
        "rw": "write",
        "numjobs": "40"
      },
      "read": {
        "io_bytes": 0,
        "io_kbytes": 0,
        "bw_bytes": 0,
        "bw": 0,
        "iops": 0,
        "runtime": 0,
        "total_ios": 0,
        "short_ios": 0,
        "drop_ios": 0,
        "slat_ns": {
          "min": 0,
          "max": 0,
          "mean": 0,
          "stddev": 0
        },
        "clat_ns": {
          "min": 0,
          "max": 0,
          "mean": 0,
          "stddev": 0
        },
        "lat_ns": {
          "min": 0,
          "max": 0,
          "mean": 0,
          "stddev": 0,
          "percentile": {
            "1.000000": 0,
            "5.000000": 0,
            "10.000000": 0,
            "20.000000": 0,
            "30.000000": 0,
            "40.000000": 0,
            "50.000000": 0,
            "60.000000": 0,
            "70.000000": 0,
            "80.000000": 0,
            "90.000000": 0,
            "95.000000": 0,
            "99.000000": 0,
            "99.500000": 0,
            "99.900000": 0,
            "99.950000": 0,
            "99.990000": 0
          }
        },
        "bw_min": 0,
        "bw_max": 0,
        "bw_agg": 0,
        "bw_mean": 0,
        "bw_dev": 0,
        "bw_samples": 0,
        "iops_min": 0,
        "iops_max": 0,
        "iops_mean": 0,
        "iops_stddev": 0,
        "iops_samples": 0
      },
      "write": {
        "io_bytes": 141639680,
        "io_kbytes": 138320,
        "bw_bytes": 1651023,
        "bw": 1612,
        "iops": 71.36113,
        "runtime": 85789,
        "total_ios": 6122,
        "short_ios": 0,
        "drop_ios": 0,
        "slat_ns": {
          "min": 359311,
          "max": 839801153,
          "mean": 32299912.179265,
          "stddev": 14276734.695224
        },
        "clat_ns": {
          "min": 4766,
          "max": 51078766478,
          "mean": 21817528028.5705,
          "stddev": 8055171405.52465
        },
        "lat_ns": {
          "min": 35401665,
          "max": 51107629055,
          "mean": 21844414912.6291,
          "stddev": 8058098755.87145,
          "percentile": {
            "1.000000": 2332033024,
            "5.000000": 7616856064,
            "10.000000": 11072962560,
            "20.000000": 14294188032,
            "30.000000": 17112760320,
            "40.000000": 17112760320,
            "50.000000": 17112760320,
            "60.000000": 17112760320,
            "70.000000": 17112760320,
            "80.000000": 17112760320,
            "90.000000": 17112760320,
            "95.000000": 17112760320,
            "99.000000": 17112760320,
            "99.500000": 17112760320,
            "99.900000": 17112760320,
            "99.950000": 17112760320,
            "99.990000": 17112760320
          }
        },

Here, we can see the runtime for job_name: 2_thread is 85s, but we have configured 60s runtime for every job.

Expectation:
We want to calculate the start and end_time of every configured job. Currently we are calculating the start and end-time using the runtime value reported by fio, which is not correct. Could you please help me in calculating the correct start and end-time of the fio job.

[vincentkfu]

Please try a current version of fio. The latest is 3.35.

[sethiay]

@vincentkfu As far as I understand, the latest version available for Ubuntu 20.04 is 3.16 (https://packages.ubuntu.com/search?keywords=fio).

[sethiay]

@vincentkfu is there any way we can update to fio 3.35 on Ubuntu 20.04 ?

[gargnitingoogle]

I installed fio 3.35 by building it from source code in this repo with tag fio-3.35 as @vincentkfu suggested in #1640 (comment) . And then I ran the jobs that @raj-prince shared in #1640 (comment) , but the indiviaul fio job runtimes still overshoot the runtime set in the job file (60 seconds).

cat fio-output.json | grep '"runtime"' | grep -v ': 0'
    "runtime" : "60s",
        "runtime" : 60074,
        "runtime" : 64764,
        "runtime" : 60173,
        "runtime" : 61647,
        "runtime" : 60200,
        "runtime" : 61768,
        "runtime" : 60259,
        "runtime" : 61922,
        "runtime" : 60139,
        "runtime" : 70952,
        "runtime" : 60429,
        "runtime" : 62345,
        "runtime" : 60270,
        "runtime" : 62062,
        "runtime" : 60373,
        "runtime" : 62306,

@vincentkfu is this a known issue in fio ?
I'm going by the spec at https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-runtime.

[vincentkfu]

You have a heavily loaded system and it takes fio some time to record the timestamp when a job completes.

Within each group fio reports the highest runtime as the group runtime. Set group_reporting=0 and you will see the actual runtimes for each job. Some will exceed 60s.

[gargnitingoogle]

You have a heavily loaded system and it takes fio some time to record the timestamp when a job completes.

Within each group fio reports the highest runtime as the group runtime. Set group_reporting=0 and you will see the actual runtimes for each job. Some will exceed 60s.

Thanks @vincentkfu . I'll try out group_reporting=0 . I was also wondering if the increased time has something to do with the ramp_time=10 also in our job.

Some will exceed 60s.

Could you please expand on this ? Will some runtimes exceed 60s even after setting group_reporting to 0 ?

[gargnitingoogle]

You have a heavily loaded system and it takes fio some time to record the timestamp when a job completes.

Within each group fio reports the highest runtime as the group runtime. Set group_reporting=0 and you will see the actual runtimes for each job. Some will exceed 60s.

On running group_reporting=0, I got most of runtimes at around 60 seconds and some higher, as @vincentkfu noted.

$ cat fio-output.json | grep '"runtime"' | grep -v ': 0'
    "runtime" : "60s",
        "runtime" : 60018,
        "runtime" : 60015,
        "runtime" : 60002,
        "runtime" : 60001,
        "runtime" : 60044,
        "runtime" : 60027,
        "runtime" : 60031,
        "runtime" : 60025,
        "runtime" : 60019,
        "runtime" : 60023,
        "runtime" : 60023,
        "runtime" : 60036,
        "runtime" : 60019,
        "runtime" : 60016,
        "runtime" : 60001,
        "runtime" : 60005,
        "runtime" : 60023,
        "runtime" : 60001,
        "runtime" : 60001,
        "runtime" : 60057,
        "runtime" : 60027,
        "runtime" : 60008,
        "runtime" : 60008,
        "runtime" : 60027,
        "runtime" : 60015,
        "runtime" : 60056,
        "runtime" : 60024,
        "runtime" : 60045,
        "runtime" : 60033,
        "runtime" : 60024,
        "runtime" : 60038,
        "runtime" : 60011,
        "runtime" : 60069,
        "runtime" : 60018,
        "runtime" : 60020,
        "runtime" : 60002,
        "runtime" : 60035,
        "runtime" : 60019,
        "runtime" : 60001,
        "runtime" : 60072,
        "runtime" : 60063,
        "runtime" : 60101,
        "runtime" : 62174,
        "runtime" : 60187,
        "runtime" : 60270,
        "runtime" : 60214,
        "runtime" : 60100,
        "runtime" : 65621,
        "runtime" : 61947,
        "runtime" : 60584,
        "runtime" : 60054,
        "runtime" : 60164,
        "runtime" : 64516,
        "runtime" : 60119,
        "runtime" : 60168,
        "runtime" : 60503,
        "runtime" : 65962,
        "runtime" : 60086,
        "runtime" : 60032,
        "runtime" : 60051,
        "runtime" : 62464,
        "runtime" : 60309,
        "runtime" : 60917,
        "runtime" : 60045,
        "runtime" : 60034,
        "runtime" : 60021,
        "runtime" : 60009,
        "runtime" : 60177,
        "runtime" : 60346,
        "runtime" : 60206,
        "runtime" : 60308,
        "runtime" : 60031,
        "runtime" : 60053,
        "runtime" : 61741,
        "runtime" : 60149,
        "runtime" : 60081,
        "runtime" : 60097,
        "runtime" : 60376,
        "runtime" : 60098,
        "runtime" : 60104,
        "runtime" : 60090,
        "runtime" : 60008,
        "runtime" : 60052,
        "runtime" : 60049,
        "runtime" : 60002,
        "runtime" : 60004,
        "runtime" : 60013,
        "runtime" : 60108,
        "runtime" : 60153,
        "runtime" : 60119,
        "runtime" : 60120,
        "runtime" : 60114,
        "runtime" : 60133,
        "runtime" : 60049,
        "runtime" : 60064,
        "runtime" : 60026,
        "runtime" : 60019,
        "runtime" : 60064,
        "runtime" : 60173,
        "runtime" : 60006,
        "runtime" : 60142,
        "runtime" : 60053,
        "runtime" : 60141,
        "runtime" : 60086,
        "runtime" : 60004,
        "runtime" : 60008,
        "runtime" : 60007,
        "runtime" : 60001,
        "runtime" : 60111,
        "runtime" : 60006,
        "runtime" : 60026,
        "runtime" : 60020,
        "runtime" : 60109,
        "runtime" : 60120,
        "runtime" : 60103,
        "runtime" : 60011,
        "runtime" : 60068,
        "runtime" : 60010,
        "runtime" : 60011,
        "runtime" : 60111,
        "runtime" : 60793,
        "runtime" : 60354,
        "runtime" : 60043,
        "runtime" : 60127,
        "runtime" : 60092,
        "runtime" : 60758,
        "runtime" : 60542,
        "runtime" : 60695,
        "runtime" : 60512,
        "runtime" : 60974,
        "runtime" : 60160,
        "runtime" : 60023,
        "runtime" : 60303,
        "runtime" : 61024,
        "runtime" : 62135,
        "runtime" : 60247,
        "runtime" : 60361,
        "runtime" : 60378,
        "runtime" : 60533,
        "runtime" : 60334,
        "runtime" : 60740,
        "runtime" : 60539,
        "runtime" : 60188,
        "runtime" : 60379,
        "runtime" : 61192,
        "runtime" : 60533,
        "runtime" : 60343,
        "runtime" : 60225,
        "runtime" : 60035,
        "runtime" : 60079,
        "runtime" : 60891,
        "runtime" : 60631,
        "runtime" : 60379,
        "runtime" : 60152,
        "runtime" : 60393,
        "runtime" : 62009,
        "runtime" : 60313,
        "runtime" : 60565,
        "runtime" : 60409,
        "runtime" : 60458,
        "runtime" : 60141,
        "runtime" : 60037,
        "runtime" : 60088,
        "runtime" : 60057,
        "runtime" : 60002,
        "runtime" : 60017,
        "runtime" : 60038,
        "runtime" : 60091,
        "runtime" : 60073,
        "runtime" : 60020,
        "runtime" : 60121,
        "runtime" : 60005,
        "runtime" : 60038,
        "runtime" : 60027,
        "runtime" : 60076,
        "runtime" : 60130,
        "runtime" : 60018,
        "runtime" : 60024,
        "runtime" : 60108,
        "runtime" : 60086,
        "runtime" : 60104,
        "runtime" : 60076,
        "runtime" : 60041,
        "runtime" : 60040,
        "runtime" : 60004,
        "runtime" : 60036,
        "runtime" : 60032,
        "runtime" : 60018,
        "runtime" : 60014,
        "runtime" : 60118,
        "runtime" : 60091,
        "runtime" : 60035,
        "runtime" : 60007,
        "runtime" : 60028,
        "runtime" : 60011,
        "runtime" : 60111,
        "runtime" : 60139,
        "runtime" : 60027,
        "runtime" : 60090,
        "runtime" : 60004,
        "runtime" : 60586,
        "runtime" : 61452,
        "runtime" : 60393,
        "runtime" : 60253,
        "runtime" : 60520,
        "runtime" : 60292,
        "runtime" : 61269,
        "runtime" : 60752,
        "runtime" : 60457,
        "runtime" : 60434,
        "runtime" : 60016,
        "runtime" : 60405,
        "runtime" : 60450,
        "runtime" : 60849,
        "runtime" : 60464,
        "runtime" : 60856,
        "runtime" : 60021,
        "runtime" : 60429,
        "runtime" : 60667,
        "runtime" : 60867,
        "runtime" : 60837,
        "runtime" : 60567,
        "runtime" : 60133,
        "runtime" : 61840,
        "runtime" : 60524,
        "runtime" : 60372,
        "runtime" : 60328,
        "runtime" : 60637,
        "runtime" : 60112,
        "runtime" : 61537,
        "runtime" : 60333,
        "runtime" : 60008,
        "runtime" : 60211,
        "runtime" : 60650,
        "runtime" : 61722,
        "runtime" : 60293,
        "runtime" : 61663,
        "runtime" : 60778,
        "runtime" : 60945,
        "runtime" : 60702,
        "runtime" : 60156,
        "runtime" : 60043,
        "runtime" : 60020,
        "runtime" : 60033,
        "runtime" : 60002,
        "runtime" : 60002,
        "runtime" : 60002,
        "runtime" : 60002,
        "runtime" : 60034,
        "runtime" : 60023,
        "runtime" : 60006,
        "runtime" : 60005,
        "runtime" : 60288,
        "runtime" : 60003,
        "runtime" : 60002,
        "runtime" : 60002,
        "runtime" : 60003,
        "runtime" : 60011,
        "runtime" : 60014,
        "runtime" : 60002,
        "runtime" : 60005,
        "runtime" : 60011,
        "runtime" : 60014,
        "runtime" : 60007,
        "runtime" : 60005,
        "runtime" : 60007,
        "runtime" : 60006,
        "runtime" : 60009,
        "runtime" : 60010,
        "runtime" : 60013,
        "runtime" : 60004,
        "runtime" : 60156,
        "runtime" : 60001,
        "runtime" : 60016,
        "runtime" : 60022,
        "runtime" : 60041,
        "runtime" : 60019,
        "runtime" : 60344,
        "runtime" : 60005,
        "runtime" : 60005,
        "runtime" : 60849,
        "runtime" : 61137,
        "runtime" : 60365,
        "runtime" : 61462,
        "runtime" : 62386,
        "runtime" : 60460,
        "runtime" : 61894,
        "runtime" : 60245,
        "runtime" : 61091,
        "runtime" : 60803,
        "runtime" : 60119,
        "runtime" : 61140,
        "runtime" : 61508,
        "runtime" : 60426,
        "runtime" : 61087,
        "runtime" : 60066,
        "runtime" : 62101,
        "runtime" : 60501,
        "runtime" : 60488,
        "runtime" : 62057,
        "runtime" : 61132,
        "runtime" : 60839,
        "runtime" : 61389,
        "runtime" : 60333,
        "runtime" : 60571,
        "runtime" : 61164,
        "runtime" : 60785,
        "runtime" : 61764,
        "runtime" : 61221,
        "runtime" : 62133,
        "runtime" : 60550,
        "runtime" : 60395,
        "runtime" : 60475,
        "runtime" : 61046,
        "runtime" : 60241,
        "runtime" : 60727,
        "runtime" : 61184,
        "runtime" : 61167,
        "runtime" : 61392,
        "runtime" : 61045,
        "runtime" : 60011,
        "runtime" : 60057,
        "runtime" : 60002,
        "runtime" : 60220,
        "runtime" : 60047,
        "runtime" : 60068,
        "runtime" : 60047,
        "runtime" : 60018,
        "runtime" : 60049,
        "runtime" : 60030,
        "runtime" : 60105,
        "runtime" : 60148,
        "runtime" : 60032,
        "runtime" : 60007,
        "runtime" : 60037,
        "runtime" : 60050,
        "runtime" : 60031,
        "runtime" : 60064,
        "runtime" : 60085,
        "runtime" : 60045,
        "runtime" : 60012,
        "runtime" : 60009,
        "runtime" : 60114,
        "runtime" : 60014,
        "runtime" : 60014,
        "runtime" : 60004,
        "runtime" : 60017,
        "runtime" : 60058,
        "runtime" : 60014,
        "runtime" : 60047,
        "runtime" : 60007,
        "runtime" : 60045,
        "runtime" : 60076,
        "runtime" : 60043,
        "runtime" : 60011,
        "runtime" : 60005,
        "runtime" : 60060,
        "runtime" : 60128,
        "runtime" : 60056,
        "runtime" : 60038,
        "runtime" : 60283,
        "runtime" : 60465,
        "runtime" : 60050,
        "runtime" : 60150,
        "runtime" : 60018,
        "runtime" : 60049,
        "runtime" : 60212,
        "runtime" : 60301,
        "runtime" : 64969,
        "runtime" : 60289,
        "runtime" : 62119,
        "runtime" : 60202,
        "runtime" : 60098,
        "runtime" : 60088,
        "runtime" : 60350,
        "runtime" : 60049,
        "runtime" : 60075,
        "runtime" : 60437,
        "runtime" : 65994,
        "runtime" : 63444,
        "runtime" : 62165,
        "runtime" : 62410,
        "runtime" : 60345,
        "runtime" : 60042,
        "runtime" : 60204,
        "runtime" : 60227,
        "runtime" : 61692,
        "runtime" : 60284,
        "runtime" : 60046,
        "runtime" : 60286,
        "runtime" : 60135,
        "runtime" : 60135,
        "runtime" : 67337,
        "runtime" : 62702,
        "runtime" : 61239,
        "runtime" : 60150,
        "runtime" : 62971,
        "runtime" : 60015,
        "runtime" : 60055,
        "runtime" : 60628,
        "runtime" : 60221,
        "runtime" : 60056,
        "runtime" : 60047,
        "runtime" : 60092,
        "runtime" : 60240,
        "runtime" : 60137,
        "runtime" : 60016,
        "runtime" : 60191,
        "runtime" : 60196,
        "runtime" : 60106,
        "runtime" : 60062,
        "runtime" : 60096,
        "runtime" : 60022,
        "runtime" : 60197,
        "runtime" : 60047,
        "runtime" : 60098,
        "runtime" : 60101,
        "runtime" : 60066,
        "runtime" : 60195,
        "runtime" : 60081,
        "runtime" : 60001,
        "runtime" : 60015,
        "runtime" : 60116,
        "runtime" : 60041,
        "runtime" : 60070,
        "runtime" : 60163,
        "runtime" : 60145,
        "runtime" : 60138,
        "runtime" : 60237,
        "runtime" : 60116,
        "runtime" : 60130,
        "runtime" : 60150,
        "runtime" : 60047,
        "runtime" : 60132,
        "runtime" : 60063,
        "runtime" : 60091,
        "runtime" : 60259,
        "runtime" : 60002,
        "runtime" : 60123,
        "runtime" : 60174,
        "runtime" : 60416,
        "runtime" : 60080,
        "runtime" : 60022,
        "runtime" : 60128,
        "runtime" : 60163,
        "runtime" : 60252,
        "runtime" : 61601,
        "runtime" : 60049,
        "runtime" : 61260,
        "runtime" : 60144,
        "runtime" : 60627,
        "runtime" : 61857,
        "runtime" : 60211,
        "runtime" : 60087,
        "runtime" : 60283,
        "runtime" : 60403,
        "runtime" : 60737,
        "runtime" : 62576,
        "runtime" : 60400,
        "runtime" : 60100,
        "runtime" : 60215,
        "runtime" : 60628,
        "runtime" : 60102,
        "runtime" : 60897,
        "runtime" : 60818,
        "runtime" : 60354,
        "runtime" : 60124,
        "runtime" : 61218,
        "runtime" : 60814,
        "runtime" : 60034,
        "runtime" : 60062,
        "runtime" : 60814,
        "runtime" : 60077,
        "runtime" : 62581,
        "runtime" : 60175,
        "runtime" : 60401,
        "runtime" : 60281,
        "runtime" : 60137,
        "runtime" : 60165,
        "runtime" : 60474,
        "runtime" : 60104,
        "runtime" : 60134,
        "runtime" : 60057,
        "runtime" : 60109,
        "runtime" : 60215,
        "runtime" : 60267,
        "runtime" : 60267,
        "runtime" : 60035,
        "runtime" : 60019,
        "runtime" : 60020,
        "runtime" : 60147,
        "runtime" : 60274,
        "runtime" : 60136,
        "runtime" : 60165,
        "runtime" : 60041,
        "runtime" : 60044,
        "runtime" : 60124,
        "runtime" : 60177,
        "runtime" : 60085,
        "runtime" : 60134,
        "runtime" : 60148,
        "runtime" : 60055,
        "runtime" : 60109,
        "runtime" : 60090,
        "runtime" : 60110,
        "runtime" : 60064,
        "runtime" : 60097,
        "runtime" : 60033,
        "runtime" : 60022,
        "runtime" : 60237,
        "runtime" : 60114,
        "runtime" : 60224,
        "runtime" : 60216,
        "runtime" : 60205,
        "runtime" : 60052,
        "runtime" : 60070,
        "runtime" : 60350,
        "runtime" : 60059,
        "runtime" : 60004,
        "runtime" : 60210,
        "runtime" : 60359,
        "runtime" : 60908,
        "runtime" : 60274,
        "runtime" : 60033,
        "runtime" : 60372,
        "runtime" : 60046,
        "runtime" : 60216,
        "runtime" : 60186,
        "runtime" : 60998,
        "runtime" : 60406,
        "runtime" : 61867,
        "runtime" : 60176,
        "runtime" : 60473,
        "runtime" : 60151,
        "runtime" : 60224,
        "runtime" : 60313,
        "runtime" : 61443,
        "runtime" : 60376,
        "runtime" : 60156,
        "runtime" : 60665,
        "runtime" : 60287,
        "runtime" : 60205,
        "runtime" : 60332,
        "runtime" : 60337,
        "runtime" : 61513,
        "runtime" : 60208,
        "runtime" : 60315,
        "runtime" : 60138,
        "runtime" : 60517,
        "runtime" : 60467,
        "runtime" : 60080,
        "runtime" : 60401,
        "runtime" : 60351,
        "runtime" : 60654,
        "runtime" : 60054,
        "runtime" : 60102,
        "runtime" : 60908,
        "runtime" : 61258,
        "runtime" : 60096,
        "runtime" : 60072,
        "runtime" : 60018,
        "runtime" : 60031,
        "runtime" : 60176,
        "runtime" : 60099,
        "runtime" : 60125,
        "runtime" : 60003,
        "runtime" : 60072,
        "runtime" : 60330,
        "runtime" : 60005,
        "runtime" : 60192,
        "runtime" : 60129,
        "runtime" : 60300,
        "runtime" : 60185,
        "runtime" : 60216,
        "runtime" : 60018,
        "runtime" : 60200,
        "runtime" : 60030,
        "runtime" : 60045,
        "runtime" : 60028,
        "runtime" : 60094,
        "runtime" : 60223,
        "runtime" : 60143,
        "runtime" : 60175,
        "runtime" : 60232,
        "runtime" : 60200,
        "runtime" : 60307,
        "runtime" : 60296,
        "runtime" : 60020,
        "runtime" : 60266,
        "runtime" : 60130,
        "runtime" : 60242,
        "runtime" : 60241,
        "runtime" : 60180,
        "runtime" : 60043,
        "runtime" : 60053,
        "runtime" : 60204,
        "runtime" : 60293,
        "runtime" : 60051,
        "runtime" : 60121,
        "runtime" : 60102,
        "runtime" : 61498,
        "runtime" : 60005,
        "runtime" : 60443,
        "runtime" : 61413,
        "runtime" : 60717,
        "runtime" : 60388,
        "runtime" : 60181,
        "runtime" : 62094,
        "runtime" : 60758,
        "runtime" : 60451,
        "runtime" : 60723,
        "runtime" : 60479,
        "runtime" : 60846,
        "runtime" : 60510,
        "runtime" : 60583,
        "runtime" : 61340,
        "runtime" : 61792,
        "runtime" : 60707,
        "runtime" : 61286,
        "runtime" : 60768,
        "runtime" : 60191,
        "runtime" : 61032,
        "runtime" : 61940,
        "runtime" : 60226,
        "runtime" : 60691,
        "runtime" : 60842,
        "runtime" : 60132,
        "runtime" : 61311,
        "runtime" : 60042,
        "runtime" : 60866,
        "runtime" : 60759,
        "runtime" : 60332,
        "runtime" : 60906,
        "runtime" : 61352,
        "runtime" : 61356,
        "runtime" : 60768,
        "runtime" : 60204,
        "runtime" : 60805,
        "runtime" : 61806,
        "runtime" : 60427,

[gargnitingoogle]

@vincentkfu is it possible to get the start and end times of individual job-groups (shown in below sample fio job file) if they are are all stonewalled e.g. below?

[global]
...

[job-group-1]
stonewall
...

[job-group-2]
stonewall
...

Currently we are calculating them (start, end times of job-groups) in reverse order by progressively subtracting the the inserted delays i.e. the ramptimes, runtimes, and startDelays from the final timestamp in the fio output. However the total runtime of fio (calculated by $date after fio command - $date before fio command) comes out to be a lot higher than the sum-total of all the ramptimes, runtimes, and startDelays, which tells us that we're doing something wrong.

[vincentkfu]

In the JSON output is a new job start time key that was added via a recent pull request: #1621

Might this help? You will need to update to the tip of the repository to see this feature.