גיל אינציגר

Due to the high data volume and large number of distinct elements, memory capacity is often the primary bottleneck in stream processing systems. Many such systems rely on counter-based data structures, making counter representation a key target for memory optimization. However, selecting counter sizes presents a fundamental trade-off: small counters risk overflow, while large counters waste space and limit scalability. In this work, we propose an efficient encoding scheme that adapts each counter's size to its actual requirements. Our approach partitions memory into fixed-size pools (e.g., a single machine word), with each pool dynamically managing multiple counters. The design prioritizes both space efficiency and runtime performance. We demonstrate substantial memory savings and across a range of streaming algorithms and workloads.