Columnar Packet Traces for Scalable Encrypted-Internet Measurement

Abstract

Internet traffic volumes continue to grow while transport and application encryption increasingly limit payload visibility. However, modern transport protocols such as QUIC complicate traditional packet-trace analytics built around \PCAP (e.g., tshark-to-CSV), and row-oriented extraction increases the cost of repeated analysis. This paper makes repeated offline measurement practical by converting packet traces into a column-oriented representation that separates one-time parsing from downstream analytics. Concretely, we introduce the architecture of a software pipeline that converts \PCAP files to the column-oriented Apache \Parquet format. To the best of our knowledge, this is the first end-to-end pipeline that (i) converts \texttt{pcap}/\texttt{pcapng} traces into \texttt{parquet} with a schema aimed at repeated offline measurement and (ii) is engineered around a convert-once, query-many workflow with measured end-to-end time-to-insight benefits. The pipeline leverages parallel parsing to produce a columnar dataset that can be queried efficiently by standard analytics engines. We benchmark end-to-end time-to-insight and storage costs across representative workloads and diverse datasets. Our results show that after a one-time conversion, subsequent analyses can exploit column pruning and vectorized scans to reduce query time compared with re-parsing \PCAP or re-extracting row-oriented text formats, while maintaining storage costs competitive with compressed \PCAP and CSV baselines.