Columnar Packet Traces for Scalable Encrypted-Internet Measurement

Abstract

Internet traffic volumes continue to grow while transport and application encryption increasingly limit payload visibility. However, modern transport protocols such as QUIC complicate traditional packet-trace analytics built around pcap (e.g., tshark-to-CSV), and row-oriented extraction increases the cost of repeated analysis. This paper makes repeated offline mea- surement practical by converting packet traces into a column- oriented representation that separates one-time parsing from downstream analytics. Concretely, we introduce an advanced benchmarked software implementation that converts pcap files to the column-oriented Apache parquet format. To the best of our knowledge, this is among the first end-to-end pipelines that (i) converts pcap/pcapng traces into parquet with a schema aimed at repeated offline measurement and (ii) is engineered around a convert-once, query-many workflow with measured end-to-end time-to-insight benefits. The pipeline leverages par- allel parsing to produce a columnar dataset that can be queried efficiently by standard analytics engines. We benchmark end- to-end time-to-insight and storage costs across representative workloads and diverse datasets. Our results show that after a one-time conversion, subsequent analyses can exploit column pruning and vectorized scans to reduce query time compared with re-parsing pcap or re-extracting row-oriented text formats, while maintaining storage costs competitive with compressed pcap and CSV baselines.