Digger is a web application that offers a graphical user interface to explore Suricata Extensible Event Format (EVE) outputs. Its primary focus is to help Capture-the-Flag players analyse network flows during stressful and time-limited attack-defense games, such as FAUSTCTF, ENOWARS, CyberChallengeIT or ECSC. Digger is a fork of Shovel made by Pwnzer0tt1.
You might also want to have a look at these other awesome traffic analyser tools:
- https://github.com/secgroup/flower (first commit in 2018)
- https://github.com/eciavatta/caronte (first commit in 2020)
- https://github.com/OpenAttackDefenseTools/tulip (fork from flower in May 2022)
- https://github.com/FCSC-FR/shovel (repository on which Digger is based)
Compared to these traffic analyser tools, excluding Shovel, Digger only relies on Suricata while making opinionated choices for the frontend. This has a few nice implications:
- dissection of all application protocols supported by Suricata (HTTP2, modbus, SMB, DNS, etc),
- flows payloads and dissections are stored inside a PostgreSQL database for fast queries,
- ingest can be a folder of pcaps for non-root CTF, or a live capture (less delay),
- tags are defined using Suricata rules (regex, libmagic match, HTTP header, etc),
- no heavy build tools needed, Digger is easy to tweak.
- a Grafana dashboard for real-time statistics
Moreover, Digger is batteries-included with some Suricata alert rules.
+--->stats.log
|
| +---->suricata.log ctf_config.json
| | +---------------
| | +---->pcaps/* |
| | | |
+--+-----+-----+----------+ +------v-------+
device | | | |
or pcaps | | | |
-----------> Suricata with plugins | | Webapp |
| | | |
| | | |
+------^--------+---------+ +------+-------+
| | |
| | +------------------+ |
suricata.rules | | | | |
---------------+ | | | |
+--------> PostgreSQL <------+
| |
| |
+--------^---------+
|
| +--------------+
| | |
+------| Grafana |
| |
+--------------+
- The webapp is made with Svelte and Actix Web
- The UI has a fresh style with some new features: hexdump viewer, code highlighting, stats page
- CTF related settings (e.g. tick length, datetime start) are stored using a JSON file that can be edited from the webapp.
- A
run.pythat automatically starts Digger - A single PostgreSQL container is used in place of two separated SQLite files
- Digger is distributed with a GPL-3.0 license
- TCP raw data is handled by a Rust plugins instead of Lua (work in progress for UDP data)
Most of the original code of Shovel has been completly changed.
The webapp shares some similiarities but none of the original files remained, everything is new.
The part related to Suricata is what remain in common with Shovel, except for the use of PostgreSQL as database and Diesel ORM and other changes like the Rust plugins.
An important part that remained the same is the template used for registering the plugins. Thanks to FCSC-FR, in particular erdnaxe, we figured how to use Suricata 8.0.0 in Digger without breaking stuff.
The files that are currently in common with Shovel are all found inside suricata, if a file was modified by us you will find a comment that specify that.
CTF related infos are stored in a JSON file and can be edited directly from the webapp:
- tick length
- available services
- start datetime
- end datetime
Add the flag format in suricata/rules/suricata.rules if needed.
If you modify this file after starting Suricata, you may reload rules using
pkill -USR2 suricata.
Digger currently implements 3 capture modes:
- Mode A: pcap replay (slower, for archives replay or rootless CTF).
- Mode B: capture interface (fast, requires root on vulnbox and in Docker).
- Mode C: PCAP-over-IP (fast, requires root on vulnbox).
Please prefer mode B or C to get the best latency between the game network and Suricata. Use mode A only if you are not root on the vulnbox and have access to pcap files indirectly.
Place pcap files in a folder such as input_pcaps/.
If you are continuously adding new pcap, add --pcap-file-continuous to
Suricata command line.
A sample configuration can be found in docker-compose-a.yml.
Warning
Please note that restarting Suricata will cause all network capture files to be loaded again. It might add some delay before observing new flows.
This mode requires to have direct access to the game network interface.
This can be achieved by mirroring vulnbox traffic through a tunnel,
see FAQ for more details.
Here this device is named tun5.
A sample configuration can be found in docker-compose-b.yml.
Warning
Please note that stopping Suricata will stop network capture.
You may also run sudo tcpdump -n -i tun5 -G 30 -w trace-%Y-%m-%d_%H-%M-%S.pcap
for archiving purposes.
This mode requires to have access to a TCP listener exposing PCAP-over-IP. By the default pcap-broker will connect with SSH to the vulnbox and run the command:
tcpdump -U --immediate-mode -ni game -s 65535 -w - not tcp port 22If you need to route PCAP-over-IP to multiple clients, you should consider using
pcap-broker.
A sample configuration is given in docker-compose-c.yml.
Warning
Please note that stopping Suricata will stop network capture.
Digger provides a convenient ./run.py script to manage containers and configuration. The script supports multiple commands with a user-friendly interface.
If you run ./run.py without arguments, it will enter interactive mode where you can select actions and configure parameters through guided prompts.
Here you can find a list of available commands to use with ./run.py:
start- Build and start containersstop- Stop running containersclear- Clear data (Suricata output, config files, PCAP files) and stop containersstatus- Show container statuslogs- Follow container logshelp- Show help information
To start Digger, use the start command with one of the following capture modes:
--mode-a- Mode A, for pcap replay mode--mode-b- Mode B, for live capture interface mode--mode-c- Mode C, for live capture using PCAP-over-IP (default if not specified)
When using Mode C, you can specify additional parameters:
--target-ip TARGET_IP(or-ip): IP address of the vulnbox (MANDATORY for Mode C)--key ALGORITHM(or-k): SSH private key path used for connection
Here you can find an example with the full configuration:
./run.py start --mode-c \
--target-ip 10.60.0.1 \
--key ~/.ssh/id_ed25519--no-build: Skip building Docker images (use existing images)--no-clean: Skip cleaning environment (keep existing data)
To stop running containers:
./run.py stopThe clear command provides granular control over data cleanup. You can use the interactive clear by running:
./run.py clearAlternatively, you can specify what to clean:
./run.py clear --suricata # (or -s): Clear Suricata output and stop containersTo rapidly delete everything listed above you can use the flag --all (or -A).
To avoid the cleanup, you can add the flag --no-clean to the startup command.
Check container status:
./run.py statusFollow container logs:
# Follow all container logs
./run.py logs
# Follow logs with tail limit
./run.py logs --tail 100
# Follow specific service logs
./run.py logs webapp --tail 50For a complete list of commands and options, run:
./run.py helpCommon command examples:
# Quick start Mode C with target IP
./run.py start --mode-c --target-ip 10.60.2.1
# Start without rebuilding images
./run.py start --mode-c --no-build
# Start preserving existing data
./run.py start --mode-c --no-clean
# Stop everything
./run.py stop
# Clean everything and stop
./run.py clear --all
# Monitor containers
./run.py status
./run.py logsYou can customize the configuration directly from the web interface, all the changes are saved in file called ctf_config.json.
Click on the settings icon in the top left corner to open the Service Manager modal. Here you can add:
- service name,
- ip:port used (we support services with multiple IPs and ports),
- colour (helps to identify the service in the flow list).
In order to work properly and display correct informations Digger requires specific configuration about the CTF you are playing:
- tick length (default to 120 seconds)
- start datetime (default to present datetime)
- end datetime (default to present datetime + 8 hours)
Digger includes an auto-refresh feature that automatically refreshes the flow list with an interval specified in the configuration. After updating, the interface will scroll to the previously selected flow (if any), by default is every 60 seconds. The auto-refresh is client specific, that means every user can decide the refresh interval.
The auto-refresh feature can be toggled by clicking the Auto-Update button in the top left corner of the web interface.
You can update this values at runtime from Settings in the web interface.
As of Suricata 8.0.3, flow_id is calculated using the hash of the flow and the timestamp in seconds and micro-seconds. See the source code in flow.h: https://doxygen.openinfosecfoundation.org/flow_8h_source.html#l00627
Most CTF uses OpenVPN or Wireguard for the "game" network interface on the vulnbox,
which means you can duplicate the traffic to an OpenSSH tun tunnel.
Using this method, Digger can run on another machine in live capture mode.
Warning
If you need to clone a physical Ethernet interface such as eth0,
you will need to use -o Tunnel=ethernet -w 5:5 in the SSH command line to create a tap.
To achieve traffic mirroring, you may use these steps as reference:
-
Enable SSH tunneling in vulnbox OpenSSH server:
echo -e 'PermitTunnel yes' | sudo tee -a /etc/ssh/sshd_config systemctl restart ssh
-
Create
tun5tunnel from the local machine to the vulnbox and uptun5on vulnbox:sudo ip tuntap add tun5 mode tun user $USER ssh -w 5:5 root@10.20.9.6 ip link set tun5 up
-
Up
tun5on the local machine and starttcpdumpto create pcap files:sudo ip link set tun5 up sudo tcpdump -n -i tun5 -G 30 -Z root -w trace-%Y-%m-%d_%H-%M-%S.pcap -
Mirror
gametraffic totun5on the vulnbox. This can be done using Nftables netdevdupoption oningressandegress.
You can edit suricata rules in suricata/rules/suricata.rules, then reload the rules
using:
pkill -USR2 suricataThis project is a fork of Shovel, which included code under the following licenses:
In accordance with the "or later" clause of the GPL-2.0-or-later license, and the GPL compatibility of CC0 and MIT, this fork is distributed under the terms of the GNU General Public License, version 3.0 (GPL-3.0).
All future contributions to this repository will be licensed under GPL-3.0.
The logo is the icon of the D-765 Mining Drill from Industrial Craft 2, that is a mod of Minecraft. Check the wiki: https://wiki.industrial-craft.net/index.php/Mining_Drill