support for Cuckoo filter commands

.github/workflows/test.yml

@@ -43,7 +43,7 @@ jobs:
     name: >
       py:${{ matrix.python-version }},${{ matrix.redis-image }},
       redis-py:${{ matrix.redis-py }},cov:${{ matrix.coverage }},
-      lupa:${{ matrix.lupa }}, json:${{matrix.extra}}
+      extra:${{matrix.extra}}
     needs:
       - "lint"
     runs-on: ubuntu-latest
@@ -58,14 +58,12 @@ jobs:
           - python-version: "3.11"
             redis-image: "redis/redis-stack:6.2.6-v10"
             redis-py: "5.0.1"
-            lupa: true
-            extra: true
+            extra: true  # json, bf, lupa, cf
             hypothesis: true
           - python-version: "3.11"
             redis-image: "redis/redis-stack-server:7.2.0-v0"
             redis-py: "5.0.1"
-            lupa: true
-            extra: true  # json, bf
+            extra: true  # json, bf, lupa, cf
             coverage: true
             hypothesis: true
 
@@ -97,14 +95,10 @@ jobs:
           echo "$HOME/.poetry/bin" >> $GITHUB_PATH
           poetry install
           poetry run pip install redis==${{ matrix.redis-py }}
-      - name: Install lupa
-        if: ${{ matrix.lupa }}
-        run: |
-          poetry run pip install "fakeredis[lua]"
       - name: Install json
         if: ${{ matrix.extra }}
         run: |
-          poetry run pip install "fakeredis[json,bf]"
+          poetry run pip install "fakeredis[json,bf,cf,lua]"
       - name: Get version
         id: getVersion
         shell: bash

docs/about/changelog.md

@@ -5,16 +5,22 @@ description: Change log of all fakeredis releases
 
 ## Next release
 
-## v2.20.2
+## v2.21.0
+
+### 🚀 Features
+
+- Implement all cuckoo filter commands #276
 
 ### 🐛 Bug Fixes
 
 - Fix XREAD blocking bug #274 #275
 
 ### 🧰 Maintenance
 
-- Support for redis-py 5.1.0b1
+- Support for redis-py 5.1.0b3
 - Improve `@testtools.run_test_if_redispy_ver`
+- Refactor bloom filter commands implementation to use [pyprobables](https://github.com/barrust/pyprobables) instead of
+  pybloom_live
 
 ## v2.20.1
 

docs/redis-commands/Redis.md

@@ -619,7 +619,7 @@ Closes the connection.
 Resets the connection.
 
 
-## `bitmap` commands (6/7 implemented)
+## `bitmap` commands (6/6 implemented)
 
 ### [BITCOUNT](https://redis.io/commands/bitcount/)
 
@@ -646,13 +646,6 @@ Returns a bit value by offset.
 Sets or clears the bit at offset of the string value. Creates the key if it doesn't exist.
 
 
-### Unsupported bitmap commands 
-> To implement support for a command, see [here](../../guides/implement-command/) 
-
-#### [BITFIELD_RO](https://redis.io/commands/bitfield_ro/) <small>(not implemented)</small>
-
-Performs arbitrary read-only bitfield integer operations on strings.
-
 
 ## `list` commands (22/22 implemented)
 

docs/redis-commands/RedisBloom.md

@@ -44,60 +44,59 @@ Returns the cardinality of a Bloom filter
 
 
 
+## `cf` commands (12/12 implemented)
 
-### Unsupported cf commands 
-> To implement support for a command, see [here](../../guides/implement-command/) 
-
-#### [CF.RESERVE](https://redis.io/commands/cf.reserve/) <small>(not implemented)</small>
+### [CF.RESERVE](https://redis.io/commands/cf.reserve/)
 
 Creates a new Cuckoo Filter
 
-#### [CF.ADD](https://redis.io/commands/cf.add/) <small>(not implemented)</small>
+### [CF.ADD](https://redis.io/commands/cf.add/)
 
 Adds an item to a Cuckoo Filter
 
-#### [CF.ADDNX](https://redis.io/commands/cf.addnx/) <small>(not implemented)</small>
+### [CF.ADDNX](https://redis.io/commands/cf.addnx/)
 
 Adds an item to a Cuckoo Filter if the item did not exist previously.
 
-#### [CF.INSERT](https://redis.io/commands/cf.insert/) <small>(not implemented)</small>
+### [CF.INSERT](https://redis.io/commands/cf.insert/)
 
 Adds one or more items to a Cuckoo Filter. A filter will be created if it does not exist
 
-#### [CF.INSERTNX](https://redis.io/commands/cf.insertnx/) <small>(not implemented)</small>
+### [CF.INSERTNX](https://redis.io/commands/cf.insertnx/)
 
 Adds one or more items to a Cuckoo Filter if the items did not exist previously. A filter will be created if it does not exist
 
-#### [CF.EXISTS](https://redis.io/commands/cf.exists/) <small>(not implemented)</small>
+### [CF.EXISTS](https://redis.io/commands/cf.exists/)
 
 Checks whether one or more items exist in a Cuckoo Filter
 
-#### [CF.MEXISTS](https://redis.io/commands/cf.mexists/) <small>(not implemented)</small>
+### [CF.MEXISTS](https://redis.io/commands/cf.mexists/)
 
 Checks whether one or more items exist in a Cuckoo Filter
 
-#### [CF.DEL](https://redis.io/commands/cf.del/) <small>(not implemented)</small>
+### [CF.DEL](https://redis.io/commands/cf.del/)
 
 Deletes an item from a Cuckoo Filter
 
-#### [CF.COUNT](https://redis.io/commands/cf.count/) <small>(not implemented)</small>
+### [CF.COUNT](https://redis.io/commands/cf.count/)
 
 Return the number of times an item might be in a Cuckoo Filter
 
-#### [CF.SCANDUMP](https://redis.io/commands/cf.scandump/) <small>(not implemented)</small>
+### [CF.SCANDUMP](https://redis.io/commands/cf.scandump/)
 
 Begins an incremental save of the bloom filter
 
-#### [CF.LOADCHUNK](https://redis.io/commands/cf.loadchunk/) <small>(not implemented)</small>
+### [CF.LOADCHUNK](https://redis.io/commands/cf.loadchunk/)
 
 Restores a filter previously saved using SCANDUMP
 
-#### [CF.INFO](https://redis.io/commands/cf.info/) <small>(not implemented)</small>
+### [CF.INFO](https://redis.io/commands/cf.info/)
 
 Returns information about a Cuckoo Filter
 
 
 
+
 ### Unsupported cms commands 
 > To implement support for a command, see [here](../../guides/implement-command/) 
 

fakeredis/_fakesocket.py

@@ -1,4 +1,4 @@
-from fakeredis.stack import JSONCommandsMixin, BFCommandsMixin
+from fakeredis.stack import JSONCommandsMixin, BFCommandsMixin, CFCommandsMixin
 from ._basefakesocket import BaseFakeSocket
 from .commands_mixins.bitmap_mixin import BitmapCommandsMixin
 from .commands_mixins.connection_mixin import ConnectionCommandsMixin
@@ -41,6 +41,7 @@ class FakeSocket(
     JSONCommandsMixin,
     GeoCommandsMixin,
     BFCommandsMixin,
+    CFCommandsMixin,
 ):
     def __init__(self, server, db):
         super(FakeSocket, self).__init__(server, db)

fakeredis/stack/__init__.py

@@ -11,13 +11,18 @@ class JSONCommandsMixin:  # type: ignore # noqa: E303
         pass
 
 try:
-    import pybloom_live  # noqa: F401
+    import probables  # noqa: F401
 
     from ._bf_mixin import BFCommandsMixin  # noqa: F401
+    from ._cf_mixin import CFCommandsMixin  # noqa: F401
 except ImportError as e:
-    if e.name == "fakeredis.stack._bf_mixin":
+    if e.name == "fakeredis.stack._bf_mixin" or e.name == "fakeredis.stack._cf_mixin":
         raise e
 
 
     class BFCommandsMixin:  # type: ignore # noqa: E303
         pass
+
+
+    class CFCommandsMixin:  # noqa: E303
+        pass

fakeredis/stack/_bf_mixin.py

@@ -1,30 +1,36 @@
 """Command mixin for emulating `redis-py`'s BF functionality."""
 import io
 
-import pybloom_live
+from probables import ExpandingBloomFilter
 
 from fakeredis import _msgs as msgs
 from fakeredis._command_args_parsing import extract_args
 from fakeredis._commands import command, Key, CommandItem, Float, Int
 from fakeredis._helpers import SimpleError, OK, casematch
 
 
-class ScalableBloomFilter(pybloom_live.ScalableBloomFilter):
+class ScalableBloomFilter(ExpandingBloomFilter):
     NO_GROWTH = 0
 
-    def __init__(self, *args, **kwargs):
-        super().__init__(*args, **kwargs)
-        self.filters.append(
-            pybloom_live.BloomFilter(
-                capacity=self.initial_capacity,
-                error_rate=self.error_rate * self.ratio))
+    def __init__(self, capacity: int = 100, error_rate: float = 0.001, scale: int = 2):
+        super().__init__(capacity, error_rate)
+        self.scale: int = scale
 
-    def add(self, key):
+    def add(self, key: bytes, force: bool = False) -> bool:
         if key in self:
             return True
-        if self.scale == self.NO_GROWTH and self.filters and self.filters[-1].count >= self.filters[-1].capacity:
+        if self.scale == self.NO_GROWTH and self.elements_added >= self.estimated_elements:
             raise SimpleError(msgs.FILTER_FULL_MSG)
-        return super(ScalableBloomFilter, self).add(key)
+        super(ScalableBloomFilter, self).add(key)
+        return False
+
+    @classmethod
+    def frombytes(cls, b: bytes, **kwargs) -> "ScalableBloomFilter":
+        size, est_els, added_els, fpr = cls._parse_footer(b)
+        blm = ScalableBloomFilter(capacity=est_els, error_rate=fpr)
+        blm._parse_blooms(b, size)
+        blm._added_elements = added_els
+        return blm
 
 
 class BFCommandsMixin:
@@ -64,7 +70,7 @@ def bf_madd(self, key, *values):
         repeat=(),
     )
     def bf_card(self, key):
-        return len(key.value)
+        return key.value.elements_added
 
     @command(
         name="BF.EXISTS",
@@ -147,10 +153,10 @@ def bf_info(self, key: CommandItem, *args: bytes):
             raise SimpleError(msgs.SYNTAX_ERROR_MSG)
         if len(args) == 0:
             return [
-                b'Capacity', key.value.capacity,
-                b'Size', key.value.capacity,
-                b'Number of filters', len(key.value.filters),
-                b'Number of items inserted', key.value.count,
+                b'Capacity', key.value.estimated_elements,
+                b'Size', key.value.elements_added,
+                b'Number of filters', key.value.expansions + 1,
+                b'Number of items inserted', key.value.elements_added,
                 b'Expansion rate', key.value.scale if key.value.scale > 0 else None,
             ]
         if casematch(args[0], b'CAPACITY'):
@@ -178,10 +184,7 @@ def bf_scandump(self, key: CommandItem, iterator: int):
         f = io.BytesIO()
 
         if iterator == 0:
-            key.value.tofile(f)
-            f.seek(0)
-            s = f.read()
-            f.close()
+            s = bytes(key.value)
             return [1, s]
         else:
             return [0, None]
@@ -195,8 +198,5 @@ def bf_scandump(self, key: CommandItem, iterator: int):
     def bf_loadchunk(self, key: CommandItem, iterator: int, data: bytes):
         if key.value is not None and type(key.value) is not ScalableBloomFilter:
             raise SimpleError(msgs.NOT_FOUND_MSG)
-        f = io.BytesIO(data)
-        key.value = ScalableBloomFilter.fromfile(f)
-        f.close()
-        key.updated()
+        key.update(ScalableBloomFilter.frombytes(data))
         return OK