To my knowledge, you're the first RPi4 1GB user here, so we only have experimental recommendations.
Going back to an older version will probably not solve the problem (you've already reduced the cache size without luck).
You said it works fine without WiFi, and obviously you don't want to create a swap file. Then try this:
Switch WiFi off, and buy a WiFi-to-Ethernet adapter: https://www.amazon.com/dp/B07Z1SGWL5