rudder : has error on upload/proof submission: %Mint.TransportError{reason: :timeout}

[ruangnode]

rudder | [info] 17710035 has error on upload/proof submission: %Mint.TransportError{reason: :timeout}
rudder | [warning] key 1_17710035_7bf280ee14755a012fd9c891d5f1450579792c9785445c9d671b39817b3274b3_2a93084faad06cddd77fb25606878ea543df76230de81bdefcd05e95817e02f0 written to backlog with ipfs://bafybeih477vv3xhjah64ekm77xol47q4pbuicmp2pntvhsgl3ldc7imsvuipfs://bafybeih477vv3xhjah64ekm77xol47q4pbuicmp2pntvhsgl3ldc7imsvu; error: %Mint.TransportError{reason: :timeout}
rudder | [info] 17712765 has error on upload/proof submission: %Mint.TransportError{reason: :timeout}
rudder | [warning] key 1_17712765_f1bbe816cfa804109cd0ba59f9df473fdea092dd009608910df912a1458dd5a0_c6c94fbd9006ddabb1ce69d5d9af65998d267d3eb79a973827d8e48f4b53b202 written to backlog with ipfs://bafybeihxkvjnsqhsbrnjctiehxj33iz4ne4x44hvicslssmqdhg42z4tviipfs://bafybeihxkvjnsqhsbrnjctiehxj33iz4ne4x44hvicslssmqdhg42z4tvi; error: %Mint.TransportError{reason: :timeout}
rudder | [info] 17716020 has error on upload/proof submission: %Mint.TransportError{reason: :timeout}
rudder | [warning] key 1_17716020_f6351276bfb94ea3ffee5b9e45785cb0e7df121f9a59ff952383e66b2ebc2ce1_106ae000a1cbb2a6a3d2b377d709b73b757ddee520e6de8663a16a906604c981 written to backlog with ipfs://bafybeibhol5fynfuwdm62m7lxuuiajuj5lvxm3vbdg7uoysvetzqvkyoxiipfs://bafybeibhol5fynfuwdm62m7lxuuiajuj5lvxm3vbdg7uoysvetzqvkyoxi; error: %Mint.TransportError{reason: :timeout}
rudder | [info] 17722960 has error on upload/proof submission: %Mint.TransportError{reason: :timeout}
rudder | [warning] key 1_17722960_a176a93f93f81196e40a58a9c598fe19be1d4c107b8371aa171bda1b900e0623_ee6e2bed6ffb8e48db8563250df3a650fdfbb8df54ea090797bd1e2bda0809b4 written to backlog with ipfs://bafybeidm4kupyievvu3ocr7sxqdw54yocifdqpw2ijoeahsdho452albrmipfs://bafybeidm4kupyievvu3ocr7sxqdw54yocifdqpw2ijoeahsdho452albrm; error: %Mint.TransportError{reason: :timeout}
rudder | [info] 17724760 has error on upload/proof submission: %Mint.TransportError{reason: :timeout}
rudder | [warning] key 1_17724760_1b7ee711fdd7628e101cc369c054068f44a4391b9aec2b4ba1ba724eaed005fb_ad21544f4b652ca33d1295efa998159881d86b4b12ff3c2d20106b8ba7dbd1b9 written to backlog with ipfs://bafybeibrpokdcp5h6ghyqn6gqtnkssf3h77ubdzdgly6lftzqjw4t4up4eipfs://bafybeibrpokdcp5h6ghyqn6gqtnkssf3h77ubdzdgly6lftzqjw4t4up4e; error: %Mint.TransportError{reason: :timeout}
rudder | [info] 17724940 has error on upload/proof submission: %Mint.TransportError{reason: :timeout}
rudder | [warning] key 1_17724940_0cb3e5fc4055174832e686fe3e8851fda132b5a086794bb14032a37d30bc7746_c021a8c6ee7ea67b1266c6860b45416c8eb5ccb98ef407785715e7dceb5c48c8 written to backlog with ipfs://bafybeic3u5hzm7cpk2geotwiguvjg6oypk6wvo7hjsgl5b2wzf4bpr4miuipfs://bafybeic3u5hzm7cpk2geotwiguvjg6oypk6wvo7hjsgl5b2wzf4bpr4miu; error: %Mint.TransportError{reason: :timeout}
rudder | [info] 17725240 has error on upload/proof submission: %Mint.TransportError{reason: :timeout}
rudder | [warning] key 1_17725240_6bf5474f28768196975b02aff4e1f2d3c7c1b8e86e03bb3d5dc7078156af0e2a_4cecf5d4d8088ea55835ef21a5b75cfce28f646e26c0566c209d2db847a421d5 written to backlog with ipfs://bafybeigabh57hfkevjxanhass4z3d3s2bgpwrw5u524a64ller56rhbjtyipfs://bafybeigabh57hfkevjxanhass4z3d3s2bgpwrw5u524a64ller56rhbjty; error: %Mint.TransportError{reason: :timeout}
rudder | [info] 17724090 has error on upload/proof submission: %Mint.TransportError{reason: :timeout}
rudder | [warning] key 1_17724090_aef73bc27bf53794c6a17f10842f3f2841d68c3915d3a334968a4ea5adfaeeea_317b3a24a56d54eff2e352ba8de1df166300e376eb6de5fd3985347b730c22b8 written to backlog with ipfs://bafybeigqayeggn2rh76fzhrki34jgkxbkjh2bdpzq5hczpy2n46unudh24; error: %Mint.TransportError{reason: :timeout}
rudder | [info] 17730490 has error on upload/proof submission: %Mint.TransportError{reason: :timeout}
rudder | [warning] key 1_17730490_0493f0a5eaafa82b526c5776950fc271e61e48d55dfa24c3b994dd1d3d22b181_c765b2304fee2f2808590a1b0b76c88d594866c5ce8ccd435b4036cf8265d4a5 written to backlog with ipfs://bafybeifft3cvghcfccfhrosxyzyoelmagmyc2cjwuczvxqbve3dllslcee; error: %Mint.TransportError{reason: :timeout}
rudder | [info] 17733070 has error on upload/proof submission: %Mint.TransportError{reason: :timeout}
rudder | [warning] key 1_17733070_f5e71c2ebdeaeed8a92883ecc7206fee67d2ebf35d66364dd10507850b8d40a3_7e75e30c1013d1237acc4939a805ae4b8370cde00f75858247d1d0eee5df9adb written to backlog with ipfs://bafybeie7b47nwrl4z6walffxyfakmuopfc7b5hmzhmrulupnxhbxtxzgkuipfs://bafybeie7b47nwrl4z6walffxyfakmuopfc7b5hmzhmrulupnxhbxtxzgku; error: %Mint.TransportError{reason: :timeout}

[ruangnode]

How to solve it ? Because I have a gap between upload_success: 1910 & pipeline_success: 1920. What caused this issue?

[ruangnode]

I got error again rudder | [info] 17743030 has error on upload/proof submission: %Mint.TransportError{reason: :timeout} rudder | [warning] key 1_17743030_45bfc6940c877a51589c89ee7833ec1dbdfde9e1f57706585c117b4f65700e41_45fe839b1b2534b44f7f96597595765d9ebb313e4d85e1be02eefc13bbdb353b written to backlog with ipfs://bafybeif5qklfj2nqy7j7fbduwiuxmdxlhei26wd4ll4jmvc6k5s4de34m4ipfs://bafybeif5qklfj2nqy7j7fbduwiuxmdxlhei26wd4ll4jmvc6k5s4de34m4; error: %Mint.TransportError{reason: :timeout}

I always encounter such errors every day at almost the same time, which is from 00:01 AM to 01:01 AM UTC.

[sudeepdino008]

right now, not much can be done about it. The errors happen when IPFS gateways go down. We've logic to retry such failed keys on rudder restart, but the retries don't happen when the rudder is running, and the ipfs service is back up.
This is something that will be fixed in #147

[ruangnode]

This morning, I encountered numerous failures that caused a drastic decrease in proof blocks.

rudder | [info] 17766159 has error on upload/proof submission: %Mint.TransportError{reason: :timeout} rudder | [warning] key 1_17766159_38567cb8d3d7595ba819a8eaaec972861d9ff38bacc057e650f00daaf6d823c9_a33f6a7a82a13037a8378830d8953b2284d10c50ed17d6f7820e30466af0069b written to backlog with ipfs://bafybeihwvhlgb4vvluot3e3kgltslcqkpomqizmoedk6vz6g5jiipqwevy; error: %Mint.TransportError{reason: :timeout} rudder | [info] 17761299 has error on upload/proof submission: %Mint.TransportError{reason: :timeout} rudder | [warning] key 1_17761299_831d56cbfc32c51dd29d992d3d6d47e6f0b7b66187dc075f39c075a965e6c7f0_6d802e0462e32fabb99857c89dc1e6f06f01e077dbaf1a2d10d3cd239b114868 written to backlog with ipfs://bafybeianpl2afxzser7y674w3ey2bbves66bvevfkogfyaum2hgo5r3yiu; error: %Mint.TransportError{reason: :timeout} rudder | [info] 17766162 has error on upload/proof submission: %Mint.TransportError{reason: :timeout} rudder | [warning] key 1_17766162_5ed1cce862295ad9cc93279c1d65a4b24e80b6fc60e3b0336f0c796318358da6_b9d020be7170e0e1d14f3a37b37a2324383c93492363b46eee6307221bd5909c written to backlog with ipfs://bafybeif2aqitbdcywltclaldst4clzmnxcgj3kcd64ex3v4vkx3kq72j4e; error: %Mint.TransportError{reason: :timeout} rudder | [info] 17761302 has error on upload/proof submission: %Mint.TransportError{reason: :timeout} rudder | [warning] key 1_17761302_8d475016b341417a745c4a406e0a169219987f3f48090db9a33fb65f36d94402_d6a5b1507d9e24dbb4897180e149bb795ec630144285f4baecb0926f2ed6e348 written to backlog with ipfs://bafybeih35mnnetpdybposo6gjtdgt3w52yewphr7easyl3sru7cx6ve3uq; error: %Mint.TransportError{reason: :timeout} rudder | [info] 17766165 has error on upload/proof submission: %Mint.TransportError{reason: :timeout} rudder | [warning] key 1_17766165_f0f90985a697c7dc85e3a4bd61ace623fdaf646d3277787c6d400d7f8bd5b0a0_e4d0fff7942ca13bda01e42ad6b4d33d80ddff7063f0cc472cb49cbace8d698b written to backlog with ipfs://bafybeigwwqkdlbprwwif32utlyebecvfghepckg2izr6mleb5crizz65di; error: %Mint.TransportError{reason: :timeout} rudder | [info] 17761311 has error on upload/proof submission: %Mint.TransportError{reason: :timeout} rudder | [warning] key 1_17761311_5a4ef08217fad41a528e966d2aac0f221fb53abc8c4486e129ea61cc9f307e77_229972a3b2659daab75bf44365e5fdc857b617ade4ffac1cabed03bd18a4c918 written to backlog with ipfs://bafybeicmijtqwssw2jgmauskqrqsuxtumed6l54dhq6huze4neggizmer4; error: %Mint.TransportError{reason: :timeout} rudder | [info] 17761314 has error on upload/proof submission: %Mint.TransportError{reason: :timeout} rudder | [warning] key 1_17761314_205322a0a392afc7590bd0e3f5d5b1af925b84e4d7428ad3e83dfd7ebbce3067_0b1e8c1aa6a7fa9384f1a963ec9fdf5899d53c20f930971b4ff8d363b1d530c5 written to backlog with ipfs://bafybeieqcwbqolafze45z4vkqmdo5rsah65ojmpjgoandlb3jumoctn2o4; error: %Mint.TransportError{reason: :timeout} rudder | [info] 17766168 has error on upload/proof submission: %Mint.TransportError{reason: :timeout} rudder | [warning] key 1_17766168_f25c0d21b1a9b662d987e12e29139d6a7352250a4c0612e692e00f8047e06b9c_b0fd8aa58fb4916e6a38e7104e6831a3c784969ac58a0f52a33d95da82f83e44 written to backlog with ipfs://bafybeigr36gm7bbgxndkdoxygpmtt3km4jjyq7yq4nux7dyeavcr4fbvga; error: %Mint.TransportError{reason: :timeout} rudder | [info] 17761317 has error on upload/proof submission: %Mint.TransportError{reason: :timeout} rudder | [warning] key 1_17761317_814fe7314004eced308c467dccc1c6c8ae6dee952db310ea43e0899de511df49_e0a0b1d1848ce4ee31b777525d7f6dad287ebfea4947d152236c966cdd166758 written to backlog with ipfs://bafybeifeklb7wavdn4impmomiiexu26qebbxlapl5cqjldcqnklih3yxui; error: %Mint.TransportError{reason: :timeout}