-
Notifications
You must be signed in to change notification settings - Fork 70
-
Hi all, I'm working with a team with an unusual and large log file. Most of the log file is junk, meaningless text that no one appears to care for but there are certain blobs of text within that log that are meaningful and someone might need to look at. I've tried extracting those blobs of text using the parse_regex function and have had success filtering out all of the junk but having issues emitting each pattern match as it's own individual log event. I've messed around on vrl playground and keep getting to the same spot and unsure the best approach to proceed. Can you anyone help? |
Beta Was this translation helpful? Give feedback.
All reactions
Replies: 1 comment · 4 replies
-
Hi @jrosado48 ! You might find https://vector.dev/highlights/2021-07-16-remap-multiple/ useful. Basically, if you set
|
Beta Was this translation helpful? Give feedback.
All reactions
-
Hi @jszwedko ! Thank you for your response, I tried something similar to this and got the same result but only seeing the first regex capture group come into DD. Apologies for the formatting, having trouble sharing my VRL playground link VRL
|
Beta Was this translation helpful? Give feedback.
All reactions
-
Example log
|
Beta Was this translation helpful? Give feedback.
All reactions
-
Thanks for sharing the additional examples! When I use your Configuration: sources:
source0:
type: stdin
decoding:
codec: json
transforms:
transform0:
type: remap
inputs:
- source0
source: |
str = string(.message) ?? ""
.value = parse_regex_all(str, r'(?P<value>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{6}.*\|\|\s*)|(?P<value2>Number_PRSDUP:\s*\d+\s+Avg_QTime:\s*\d+\.\d{3}\s+Max_Parallel:\s*\d+\s+Parallel_Idle_time:\s*\d+\.\d{3}\s+Delete_Alert:\s*\d+(?:\s+Delete_Alert_failures:\s*\d+)?)') ?? {}
newval = []
for_each(.value) -> |_, item| {
if item.value != null {
newval = push(newval, item.value)
} else if item.value2 != null {
newval = push(newval, item.value2)
}
}
. = newval
sinks:
sink0:
type: console
inputs:
- transform0
encoding:
codec: json Input file ( { "message": "2024-11-13 14:51:41.742191 PRSDUP|GDSD1|CCNMETA|CHNHY|PIDMIARM|RSCN|CURUSD|BDCHR|RMLD|SDT28JAN25|EDT28JAN25|RPCRACK|RTYDDBL|FPLOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO|LOS1|RMR699.00|XA1699.00|XA2724.00|XA3749.00|XC1699.00|LOS2|RMR524.50|XA1524.50|XA2549.50|XA3574.50|XC1524.50|LOS3|RMR459.34|XA1459.34|XA2484.34|XA3509.34|XC1459.34|LOS4|RMR492.00|XA1492.00|XA2517.00|XA3542.00|XC1492.00|LOS5|RMR516.40|XA1516.40|XA2541.40|XA3566.40|XC1516.40|LOS6|RMR492.50|XA1492.50|XA2517.50|XA3542.50|XC1492.50|LOS7|RMR479.29|XA1479.29|XA2504.29|XA3529.29|XC1479.29|LOS8|RMR468.00|XA1468.00|XA2493.00|XA3518.00|XC1468.00|LOS9|RMR493.67|XA1493.67|XA2518.67|XA3543.67|XC1493.67|LOS10|RMR493.50|XA1493.50|XA2518.50|XA3543.50|XC1493.50|LOS11|RMR507.00|XA1507.00|XA2532.00|XA3557.00|XC1507.00|LOS12|RMR515.25|XA1515.25|XA2540.25|XA3565.25|XC1515.25|LOS13|RMR506.39|XA1506.39|XA2531.39|XA3556.39|XC1506.39|LOS14|RMR508.08|XA1508.08|XA2533.08|XA3558.08|XC1508.08|GMT2024-11-13T14:50:20Z|PGMIDeas Overbook|USRIDOBINTF|IDX1639813742|TYP01|MSN20241113145128787475||\nqueued in D14 status=0\nGdsAriEventState state change event=4 from state 3 to 2 pollTmo=0 outstandingRequests=7 2024-11-13 14:51:41.746504\nGdsAriEventState state change event=3 from state 2 to 3 pollTmo=15 outstandingRequests=7 2024-11-13 14:51:41.746562\nRecieived PRSDUP uuid=f417b488-a200-11ef-95b8-fb48a6a6bb6a seqNo=116 isLast=0 host=pdbcrsa22 server_index=103 2024-11-13 14:51:41.747326\nNew alert records - 2024-11-13 14:57:26.784895: (fetchId=1731531446784895)\nGDS HOTEL typ mkt_key rcat_key rp_key dbg dates idx rtyp_key dest act amt day_cd dow los ins_tm program usr_nam\n--- ----- ---- ------- -------- -------- -------- --------------------- ---------- -------- ---- --- -------- ------ --- ---- ------------------- -------------------- --------\nDENDV 20 2 110 0 7d5 2025-02-12/2025-02-13 1639914884 1975 7 127 0 2024-11-13 14:57:25 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-13/2025-02-14 1639914886 1975 7 127 0 2024-11-13 14:57:25 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-14/2025-02-15 1639914887 1975 7 127 0 2024-11-13 14:57:25 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-02/2025-02-03 1639914890 1730 7 127 0 2024-11-13 14:57:25 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-03/2025-02-04 1639914891 1730 7 127 0 2024-11-13 14:57:25 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-04/2025-02-05 1639914893 1730 7 127 0 2024-11-13 14:57:25 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-05/2025-02-06 1639914894 1730 7 127 0 2024-11-13 14:57:25 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-06/2025-02-07 1639914895 1730 7 127 0 2024-11-13 14:57:25 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-07/2025-02-08 1639914896 1730 7 127 0 2024-11-13 14:57:25 m16_rms_rec EGEGHMLA\nEnd new alert records\nFinal List:\nTotal Number of Requests: 0\nidx 1639914884 could not be processed because AlertCore has no partners\nidx 1639914886 could not be processed because Request dates 2025-02-13/2025-02-14 after last inventory day 1970-01-01\nidx 1639914887 could not be processed because Request dates 2025-02-14/2025-02-15 after last inventory day 1970-01-01\nidx 1639914890 could not be processed because Request dates 2025-02-02/2025-02-03 after last inventory day 1970-01-01\nidx 1639914891 could not be processed because Request dates 2025-02-03/2025-02-04 after last inventory day 1970-01-01\nidx 1639914893 could not be processed because Request dates 2025-02-04/2025-02-05 after last inventory day 1970-01-01\nidx 1639914894 could not be processed because Request dates 2025-02-05/2025-02-06 after last inventory day 1970-01-01\nidx 1639914895 could not be processed because Request dates 2025-02-06/2025-02-07 after last inventory day 1970-01-01\nidx 1639914896 could not be processed because Request dates 2025-02-07/2025-02-08 after last inventory day 1970-01-01\ndeleting instance indices: 1639914884 1639914886-1639914887 1639914890-1639914891 1639914893-1639914896\nGdsAriEventState state change event=2 from state 5 to 7 pollTmo=0 outstandingRequests=0 2024-11-13 14:57:26.805729\nARI CONTR: <READS: 12> <WRITES: 4> <OPENS: 4> <FETCHES: 44> <INIT: 0.000> <SPIRIT: 0.321> <HTL_LOCK: 0.000> <ARI_SORT: 0.000> <ARI_CONSOLIDATE: 0.000> <ARI_WAIT: 0.000> <DEBUG: 0.000> <OTHER: 0.007> <TOTAL: 0.329>\nNumber_PRSDUP: 0\nAvg_QTime: 0.000\nMax_Parallel: 0\nParallel_Idle_time: 0.000\nDelete_Alert: 9\nAri processing complete - 2024-11-13 14:57:26.806123\nGdsAriEventState state change event=6 from state 7 to 5 pollTmo=0 outstandingRequests=0 2024-11-13 14:57:26.812513\nNew alert records - 2024-11-13 14:57:27.139014: (fetchId=1731531447139014)\nGDS HOTEL typ mkt_key rcat_key rp_key dbg dates idx rtyp_key dest act amt day_cd dow los ins_tm program usr_nam\n--- ----- ---- ------- -------- -------- -------- --------------------- ---------- -------- ---- --- -------- ------ --- ---- ------------------- -------------------- --------\nDENDV 20 2 110 0 7d5 2025-02-07/2025-02-08 1639914918 4084 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-08/2025-02-09 1639914920 4084 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-09/2025-02-10 1639914921 4084 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-10/2025-02-11 1639914922 4084 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-11/2025-02-12 1639914924 4084 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-12/2025-02-13 1639914925 4084 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-13/2025-02-14 1639914926 4084 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-14/2025-02-15 1639915271 4084 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nEnd new alert records\nFinal List:\nTotal Number of Requests: 0\nidx 1639914918 could not be processed because AlertCore has no partners\nidx 1639914920 could not be processed because Request dates 2025-02-08/2025-02-09 after last inventory day 1970-01-01\nidx 1639914921 could not be processed because Request dates 2025-02-09/2025-02-10 after last inventory day 1970-01-01\nidx 1639914922 could not be processed because Request dates 2025-02-10/2025-02-11 after last inventory day 1970-01-01\nidx 1639914924 could not be processed because Request dates 2025-02-11/2025-02-12 after last inventory day 1970-01-01\nidx 1639914925 could not be processed because Request dates 2025-02-12/2025-02-13 after last inventory day 1970-01-01\nidx 1639914926 could not be processed because Request dates 2025-02-13/2025-02-14 after last inventory day 1970-01-01\nidx 1639915271 could not be processed because Request dates 2025-02-14/2025-02-15 after last inventory day 1970-01-01\ndeleting instance indices: 1639914918 1639914920-1639914922 1639914924-1639914926 1639915271\nGdsAriEventState state change event=2 from state 5 to 7 pollTmo=0 outstandingRequests=0 2024-11-13 14:57:27.213859\nARI CONTR: <READS: 11> <WRITES: 4> <OPENS: 4> <FETCHES: 43> <INIT: 0.000> <SPIRIT: 0.394> <HTL_LOCK: 0.000> <ARI_SORT: 0.000> <ARI_CONSOLIDATE: 0.000> <ARI_WAIT: 0.000> <DEBUG: 0.000> <OTHER: 0.005> <TOTAL: 0.401>\nNumber_PRSDUP: 0\nAvg_QTime: 0.000\nMax_Parallel: 0\nParallel_Idle_time: 0.000\nDelete_Alert: 8\nAri processing complete - 2024-11-13 14:57:27.213980\nGdsAriEventState state change event=6 from state 7 to 5 pollTmo=0 outstandingRequests=0 2024-11-13 14:57:27.235508\nNew alert records - 2024-11-13 14:57:27.510011: (fetchId=1731531447510011)\nGDS HOTEL typ mkt_key rcat_key rp_key dbg dates idx rtyp_key dest act amt day_cd dow los ins_tm program usr_nam\n--- ----- ---- ------- -------- -------- -------- --------------------- ---------- -------- ---- --- -------- ------ --- ---- ------------------- -------------------- --------\nDENDV 20 2 110 0 7d5 2025-02-09/2025-02-10 1639916802 4343 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-10/2025-02-11 1639916803 4343 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-11/2025-02-12 1639916805 4343 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-12/2025-02-13 1639916806 4343 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-13/2025-02-14 1639916808 4343 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-14/2025-02-15 1639916809 4343 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nEnd new alert records\nFinal List:\nTotal Number of Requests: 0\nidx 1639916802 could not be processed because AlertCore has no partners\nidx 1639916803 could not be processed because Request dates 2025-02-10/2025-02-11 after last inventory day 1970-01-01\nidx 1639916805 could not be processed because Request dates 2025-02-11/2025-02-12 after last inventory day 1970-01-01\nidx 1639916806 could not be processed because Request dates 2025-02-12/2025-02-13 after last inventory day 1970-01-01\nidx 1639916808 could not be processed because Request dates 2025-02-13/2025-02-14 after last inventory day 1970-01-01\nidx 1639916809 could not be processed because Request dates 2025-02-14/2025-02-15 after last inventory day 1970-01-01\ndeleting instance indices: 1639916802-1639916803 1639916805-1639916806 1639916808-1639916809\nGdsAriEventState state change event=2 from state 5 to 7 pollTmo=0 outstandingRequests=0 2024-11-13 14:57:27.559391\nARI CONTR: <READS: 9> <WRITES: 3> <OPENS: 4> <FETCHES: 41> <INIT: 0.000> <SPIRIT: 0.319> <HTL_LOCK: 0.000> <ARI_SORT: 0.000> <ARI_CONSOLIDATE: 0.000> <ARI_WAIT: 0.000> <DEBUG: 0.000> <OTHER: 0.003> <TOTAL: 0.324>\nNumber_PRSDUP: 0\nAvg_QTime: 0.000\nMax_Parallel: 0\nParallel_Idle_time: 0.000\nDelete_Alert: 6\nAri processing complete - 2024-11-13 14:57:27.559497\nGdsAriEventState state change event=6 from state 7 to 5 pollTmo=0 outstandingRequests=0 2024-11-13 14:57:27.574164\nNew alert records - 2024-11-13 14:57:27.848626: (fetchId=1731531447848626)\nGDS HOTEL typ mkt_key rcat_key rp_key dbg dates idx rtyp_key dest act amt day_cd dow los ins_tm program usr_nam\n--- ----- ---- ------- -------- -------- -------- --------------------- ---------- -------- ---- --- -------- ------ --- ---- ------------------- -------------------- --------\nDENDV 20 2 110 0 7d5 2025-02-02/2025-02-03 1639916810 4128 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-03/2025-02-04 1639916811 4128 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-04/2025-02-05 1639916815 4128 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-05/2025-02-06 1639916816 4128 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-06/2025-02-07 1639916818 4128 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-07/2025-02-08 1639916819 4128 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-08/2025-02-09 1639916820 4128 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-09/2025-02-10 1639916823 4128 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-10/2025-02-11 1639916824 4128 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-11/2025-02-12 1639916825 4128 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-12/2025-02-13 1639916826 4128 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nEnd new alert records\nFinal List:\nTotal Number of Requests: 0\nidx 1639916810 could not be processed because AlertCore has no partners\nidx 1639916811 could not be processed because Request dates 2025-02-03/2025-02-04 after last inventory day 1970-01-01\nidx 1639916815 could not be processed because Request dates 2025-02-04/2025-02-05 after last inventory day 1970-01-01\nidx 1639916816 could not be processed because Request dates 2025-02-05/2025-02-06 after last inventory day 1970-01-01\nidx 1639916818 could not be processed because Request dates 2025-02-06/2025-02-07 after last inventory day 1970-01-01\nidx 1639916819 could not be processed because Request dates 2025-02-07/2025-02-08 after last inventory day 1970-01-01\nidx 1639916820 could not be processed because Request dates 2025-02-08/2025-02-09 after last inventory day 1970-01-01\nidx 1639916823 could not be processed because Request dates 2025-02-09/2025-02-10 after last inventory day 1970-01-01\nidx 1639916824 could not be processed because Request dates 2025-02-10/2025-02-11 after last inventory day 1970-01-01\nidx 1639916825 could not be processed because Request dates 2025-02-11/2025-02-12 after last inventory day 1970-01-01\nidx 1639916826 could not be processed because Request dates 2025-02-12/2025-02-13 after last inventory day 1970-01-01\ndeleting instance indices: 1639916810-1639916811 1639916815-1639916816 1639916818-1639916820 1639916823-1639916826\nGdsAriEventState state change event=2 from state 5 to 7 pollTmo=0 outstandingRequests=0 2024-11-13 14:57:27.883925\nARI CONTR: <READS: 14> <WRITES: 4> <OPENS: 4> <FETCHES: 46> <INIT: 0.000> <SPIRIT: 0.303> <HTL_LOCK: 0.000> <ARI_SORT: 0.000> <ARI_CONSOLIDATE: 0.000> <ARI_WAIT: 0.000> <DEBUG: 0.000> <OTHER: 0.005> <TOTAL: 0.310>\nNumber_PRSDUP: 0\nAvg_QTime: 0.000\nMax_Parallel: 0\nParallel_Idle_time: 0.000\nDelete_Alert: 9\nDelete_Alert_failures: 5\nAri processing complete - 2024-11-13 14:57:27.884021\nGdsAriEventState state change event=6 from state 7 to 5 pollTmo=0 outstandingRequests=0 2024-11-13 14:57:27.892542\nNew alert records - 2024-11-13 14:57:28.165282: (fetchId=1731531448165282)\nGDS HOTEL typ mkt_key rcat_key rp_key dbg dates idx rtyp_key dest act amt day_cd dow los ins_tm program usr_nam\n--- ----- ---- ------- -------- -------- -------- --------------------- ---------- -------- ---- --- -------- ------ --- ---- ------------------- -------------------- --------\nDENDV 20 2 110 0 7d5 2025-02-04/2025-02-05 1639916835 4081 7 127 0 2024-11-13 14:57:26 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-05/2025-02-06 1639916837 4081 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-06/2025-02-07 1639916838 4081 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-07/2025-02-08 1639916839 4081 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-09/2025-02-10 1639916840 4081 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 202å5-02-10/2025-02-11 1639916842 4081 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-11/2025-02-12 1639916844 4081 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-12/2025-02-13 1639916846 4081 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-13/2025-02-14 1639916848 4081 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nEnd new alert records\nFinal List:\nTotal Number of Requests: 0\nidx 1639916835 could not be processed because AlertCore has no partners\nidx 1639916837 could not be processed because Request dates 2025-02-05/2025-02-06 after last inventory day 1970-01-01\nidx 1639916838 could not be processed because Request dates 2025-02-06/2025-02-07 after last inventory day 1970-01-01\nidx 1639916839 could not be processed because Request dates 2025-02-07/2025-02-08 after last inventory day 1970-01-01\nidx 1639916840 could not be processed because Request dates 2025-02-09/2025-02-10 after last inventory day 1970-01-01\nidx 1639916842 could not be processed because Request dates 2025-02-10/2025-02-11 after last inventory day 1970-01-01\nidx 1639916844 could not be processed because Request dates 2025-02-11/2025-02-12 after last inventory day 1970-01-01\nidx 1639916846 could not be processed because Request dates 2025-02-12/2025-02-13 after last inventory day 1970-01-01\nidx 1639916848 could not be processed because Request dates 2025-02-13/2025-02-14 after last inventory day 1970-01-01\ndeleting instance indices: 1639916835 1639916837-1639916840 1639916842 1639916844 1639916846 1639916848\nGdsAriEventState state change event=2 from state 5 to 7 pollTmo=0 outstandingRequests=0 2024-11-13 14:57:28.195935\nARI CONTR: <READS: 12> <WRITES: 6> <OPENS: 4> <FETCHES: 44> <INIT: 0.000> <SPIRIT: 0.291> <HTL_LOCK: 0.000> <ARI_SORT: 0.000> <ARI_CONSOLIDATE: 0.000> <ARI_WAIT: 0.000> <DEBUG: 0.000> <OTHER: 0.009> <TOTAL: 0.302>\nNumber_PRSDUP: 0\nAvg_QTime: 0.000\nMax_Parallel: 0\nParallel_Idle_time: 0.000\nDelete_Alert: 9\nAri processing complete - 2024-11-13 14:57:28.196010\nGdsAriEventState state change event=6 from state 7 to 5 pollTmo=0 outstandingRequests=0 2024-11-13 14:57:28.200841\nNew alert records - 2024-11-13 14:57:28.518388: (fetchId=1731531448518388)\nGDS HOTEL typ mkt_key rcat_key rp_key dbg dates idx rtyp_key dest act amt day_cd dow los ins_tm program usr_nam\n--- ----- ---- ------- -------- -------- -------- --------------------- ---------- -------- ---- --- -------- ------ --- ---- ------------------- -------------------- --------\nDENDV 20 2 110 0 7d5 2025-02-02/2025-02-03 1639916852 4124 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-03/2025-02-04 1639916855 4124 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-04/2025-02-05 1639916856 4124 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-05/2025-02-06 1639916858 4124 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-06/2025-02-07 1639916859 4124 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-07/2025-02-08 1639916860 4124 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-08/2025-02-09 1639916862 4124 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-09/2025-02-10 1639916863 4124 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-10/2025-02-11 1639916865 4124 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-11/2025-02-12 1639916866 4124 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-12/2025-02-13 1639916867 4124 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-13/2025-02-14 1639916869 4124 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-14/2025-02-15 1639916871 4124 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-02/2025-02-03 1639916874 4129 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-03/2025-02-04 1639916875 4129 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-04/2025-02-05 1639916877 4129 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-05/2025-02-06 1639916878 4129 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-06/2025-02-07 1639916879 4129 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-07/2025-02-08 1639916881 4129 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-08/2025-02-09 1639916882 4129 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-09/2025-02-10 1639916883 4129 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-10/2025-02-11 1639916885 4129 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-11/2025-02-12 1639916886 4129 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nDENDV 20 2 110 0 7d5 2025-02-12/2025-02-13 1639916887 4129 7 127 0 2024-11-13 14:57:27 m16_rms_rec EGEGHMLA\nEnd new alert records\nFinal List:\nTotal Number of Requests: 0\nidx 1639916852 could not be processed because AlertCore has no partners\nidx 1639916855 could not be processed because Request dates 2025-02-03/2025-02-04 after last inventory day 1970-01-01\nidx 1639916856 could not be processed because Request dates 2025-02-04/2025-02-05 after last inventory day 1970-01-01\nidx 1639916858 could not be processed because Request dates 2025-02-05/2025-02-06 after last inventory day 1970-01-01\nidx 1639916859 could not be processed because Request dates 2025-02-06/2025-02-07 after last inventory day 1970-01-01\nidx 1639916860 could not be processed because Request dates 2025-02-07/2025-02-08 after last inventory day 1970-01-01\nidx 1639916862 could not be processed because Request dates 2025-02-08/2025-02-09 after last inventory day 1970-01-01\nidx 1639916863 could not be processed because Request dates 2025-02-09/2025-02-10 after last inventory day 1970-01-01\nidx 1639916865 could not be processed because Request dates 2025-02-10/2025-02-11 after last inventory day 1970-01-01\nidx 1639916866 could not be processed because Request dates 2025-02-11/2025-02-12 after last inventory day 1970-01-01\nidx 1639916867 could not be processed because Request dates 2025-02-12/2025-02-13 after last inventory day 1970-01-01\nidx 1639916869 could not be processed because Request dates 2025-02-13/2025-02-14 after last inventory day 1970-01-01\nidx 1639916871 could not be processed because Request dates 2025-02-14/2025-02-15 after last inventory day 1970-01-01" } Output:
Here we can see multiple messages. You mentioned you were sending the logs to Datadog. Is it possible some processing pipeline on that side is dropping or not indexing some of the logs? |
Beta Was this translation helpful? Give feedback.
All reactions
-
Ahh thank you for this, it seems I've been chasing my own tail. There's other transforms before and after the transform I'm currently working that are having unwanted affects and thus dropping the below messages. Will need to look into that next but for the sake of this discussion the above works. Thank you very much for your help @jszwedko ! |
Beta Was this translation helpful? Give feedback.
All reactions
-
❤️ 1
Thanks for sharing the additional examples! When I use your
remap
I do see multiple events being emitted.Configuration: