It's a bug in iwd and/or the underlying broadcom driver.
I understand you have that wifi issue, but you should think a bit out of the box. It's a very large number of the users with RPi, LE and wifi connection. So, in the case of the clear iwd/broadcom bug, this forum would be filled with bug reports and probably solved long time ago. I don't say all it's perfect with iwd, just this probably isn't a simple software "bug" case.
It doesn't have to do with signal strength or WPA mode as others suggested.
I had this wifi "thing" or something very the same, but at least I tried few things. I moved to external antenna (far better singal), changed the router (anyway was one old stuff), and I don't have any wifi issue, even after "dirty" restart. With external antenna, I only disconnected that tiny and weak on-board antenna and connected one with "normal size", the broadcom wifi and the iwd remains the same (nightly build LE, so the iwd it's more or less up to date). I don't know if in my case what triggered that issue, was the signal strenght or something in the router... but now it's working well. And I don't think to be alone who had, but no more have it that issue.