https://nyadgar.com/tags/bigdata/ Recent content in Bigdata on nyadgar.com Hugo en-us Wed, 13 Nov 2024 15:50:12 +0200 https://nyadgar.com/posts/the-beautiful-math-of-bloom-filters/ Wed, 13 Nov 2024 15:50:12 +0200 https://nyadgar.com/posts/the-beautiful-math-of-bloom-filters/ <p>Probabilistic functions can model many algorithms and procedures. They can help us optimize procedures to produce the best results. Experienced software engineers know that at some point, almost any software will reach some level of non-determinism, where a solution is not absolute but approaches the best results if using the optimal configuration. Mathematically, such a solution usually boils down to finding <em>minima</em>, <em>maxima</em> or <em>limits</em> of some probabilistic functions.</p> <p>In this article, we will explore the beautiful math behind Bloom filters. We will explore the accuracy and trade-offs of Bloom filters and see why a Bloom filter can be an excellent choice for some cases, especially in <em>Big Data</em>, <a href="https://en.wikipedia.org/wiki/Online_analytical_processing">OLAP</a> systems, dealing with huge and fairly static datasets.</p>