dir_exporter

dir_exporter is a small Prometheus exporter that recursively scans a directory tree and exposes per-directory file counts and byte totals as time-series metrics.

It is designed for:

• Tracking how datasets grow over time
• Detecting stalled or runaway ingestion pipelines
• Storage growth and capacity planning
• Snapshot-style filesystem telemetry (not event-based)

The exporter performs periodic scans, caches the results, and serves them efficiently to Prometheus.

---

Features

• Recursive directory scanning from a single root
• Per-directory metrics (file count + total bytes)
• Bounded parallelism using goroutines
• Configurable scan interval (decoupled from scrape interval)
• Optional maximum directory depth
• Prometheus-native metrics
• Single static Go binary

---

Metrics Exposed

Directory metrics

disk_directory_file_count{path="/data/qlir_data/binance/klines"} 123456
disk_directory_bytes_total{path="/data/qlir_data/binance/klines"} 987654321

Both metrics are recursive — values include all files under the directory.

Exporter telemetry

disk_directory_scan_duration_seconds 1.42
disk_directory_last_scan_timestamp_seconds 1739000000

These allow you to monitor exporter health and scan performance.

---

Usage

Build

go mod init dir_exporter
go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promhttp
go mod tidy

go build -o dir_exporter

This produces a single static binary.

---

Run

./dir_exporter \
  --root /data/qlir_data \
  --scan-interval 1m \
  --scan-workers 8 \
  --max-depth 6 \
  --listen :9105

---

Command-line Flags

Flag	Description
--root	Root directory to scan recursively (required)
--scan-interval	How often to rescan the filesystem
--scan-workers	Number of parallel file stat workers
--max-depth	Maximum directory depth (0 = unlimited)
--listen	HTTP listen address

Or With the Makefile

make tidy
make build
make linux
make clean

---

Repo Structure

dir_exporter/
├── README.md
├── LICENSE
├── Makefile
├── go.mod
├── go.sum
├── main.go
│
├── ops/
│   ├── systemd/
│   │   └── dir_exporter.service
│   │
│   ├── prometheus/
│   │   └── dir_exporter.scrape.yaml
│   │
│   ├── grafana/
│   │   ├── dashboards/
│   │   │   └── dir_exporter_overview.json
│   │   └── README.md
│   │
│   └── alerts/
│       └── dir_exporter.rules.yaml
│
└── docs/
    └── architecture.md

Operational Toolkit

This repository includes optional operational tooling under ops/ for running dir_exporter in production environments:

• systemd unit files
• Prometheus scrape configuration
• Grafana dashboards
• Alerting rules

These are provided as reference implementations and are not required to use the exporter.

---

Prometheus Configuration

- job_name: dir_exporter
  static_configs:
    - targets: ['localhost:9105']

Scraping is cheap — filesystem IO only occurs during scheduled scans, not per scrape.

---

Grafana Examples

Directory growth over time

disk_directory_file_count

Growth rate (files per second)

rate(disk_directory_file_count[5m])

Storage growth

rate(disk_directory_bytes_total[5m])

Exporter health

disk_directory_scan_duration_seconds
time() - disk_directory_last_scan_timestamp_seconds

---

Design Notes

• Scans are snapshot-based, not event-driven
• Parallelism is bounded to avoid disk thrashing
• Aggregation is deterministic and scrape-safe
• Directory paths are treated as stable, finite labels
• Intended for structured data trees, not arbitrary user home directories

---

When This Is a Good Fit

• Data lakes / research datasets
• Time-partitioned ingestion pipelines
• Append-only storage layouts
• Monitoring completeness and growth trends

---

When This Is NOT a Good Fit

• Per-file monitoring
• Extremely high-churn ephemeral directories
• Environments with unbounded or user-generated paths

---

Future Extensions

Planned or easy additions:

• Exclude path globs
• Incremental or cached directory scans