Jack's blog

Debugging flakey tests with pytest-repeat

pytest-repeat is something I reach for whenever I (or someone on my team) encounter a flakey CI test.

The Problem

Just this morning I ran into a flakey test which looked something like:

def request_callback(request):
    payload = json.loads(request.body)
    num = random.randint(1, 100)
    payload["id"] = num
    payload["href"] = f"https://api.placeholder.com/posts/{num}"
    return (200, {}, json.dumps(payload))


def test_flakey(...):
    mocked_responses.add_callback(
        responses.POST,
        "https://api.placeholder.com/posts",
        callback=request_callback,
    )
    create_post("a") # Makes POST request to /posts, saves to DB
    create_post("b")

Can you spot where/why this might fail?

$ pytest test_flakey.py --count=100 -x --pdb -q
...................................F

In my case, I was using the randomly generated ID random.randint(1, 100) as the primary key for objects in DB, which was causing a UniqueViolation

(psycopg.errors.UniqueViolation) duplicate key value violates unique constraint "post_data_pkey"

Since the test failures were so intermittent pytest-repeat was super helpful in tracking down & reproducing this specific bug.

The Fix

The fix here was to instead use something deterministic to generate IDs returned from the mocked API.

+import itertools
+
+id_counter = itertools.count()
+
+
 def request_callback(request):
-    num = random.randint(1, 100)
+    num = next(id_counter)
     payload["id"] = num
     payload["href"] = f"https://api.placeholder.com/{num}"