Lua script for split big PCAP file in few little PCAP's by tcp stream id with one tshark run. It's much faster than:
pcap="very-big-file.pcap"
mkdir -p "$pcap.parts/"
for tcp_stream in $(tshark -n -r "$pcap" -T fields -e tcp.stream | sort -un | tail -1); do
tshark -Y "tcp.stream eq ${tcp_stream}" -r "$pcap" -w "$pcap.parts/$tcp_stream.pcap"
donebecause you don't need to reread entire PCAP for each tcp stream.
tshark -X lua_script:tcp-stream-splitter.lua -X lua_script1:very-big-file.pcap -n -r very-big-file.pcapOutput files will be stored by pattern $PWD/very-big-file.pcap.parts/$CLIENT_IP-$CLIENT_PORT_$SERVER_IP-$SERVER_PORT_$TCP_STREAM_ID.pcap.
If there's a lot concurrent tcp streams in one big PCAP you may avoid fail with to many opened file descriptor by set ulimit to maximal available value:
MacOS:
ulimit -n 2048
Some linux may allow bigger value:
ulimit -n 4096
If there's a really lot of streams probably nothing will help you. You can use shell-script above (and add some "parallelism) with python/coproc) and have nice cup of coffee. If you can suggest an better solution of this problem feel free to open an issue or send pull request.
- tcp stream count doesn't include corrupted (not full) tcp streams.
| Size of PCAP, Mbytes | tcp packet count | tcp stream count | time | hardware | ulimit -n | remarks |
|---|---|---|---|---|---|---|
| 0.004 | 23 | 1 | 00:00:00.342 | Macbook Pro 2015 | 256 | -- |
| 0.224 | 1000 | 18 | 00:00:00.332 | Macbook Pro 2015 | 256 | -- |
| 21 | 96682 | 13832 | 00:00:09.000 | Macbook Pro 2015 | 2048 | -- |
| 41 | 302868 | 14465 | 00:00:19.000 | Macbook Pro 2015 | 2048 | -- |
| 283 | 967707 | 49239 | 00:01:10.303 | Macbook Pro 2015 | 8192 | failed after 967707 packet. exit by ^C |