How long are immutable objects stored in Python?

As far as I understand Python‘s memory management, when initialising e.g. some float

k = 1.0

Python creates an immutable object. Even when redefining the name „k“

k = 1.1

the memory space is not released, but rather a second one allocated for the „new k“.

I was wondering, how long this immutable object is saved. Since there exists no name that references to it, it could as be well deleted immediately. But I have read that this is not the case. Am I right in assuming that with this behaviour, some „simple“ routine like

from datetime import datetime

while True:
    k = datetime.now()

would eventually crash as all memory has been littered with unnamed immutable objects. What is the conceptual advantage of such memory management; meaning why not release the memory immediately?