You should continue not impairing your skins ability to function by avoiding cleansers and topicals that alter PH and remove lipids. The dead skin thing is irrelevant.

I've read around and it seems like most people just exfoliate and wash their faces once a month to get rid of the skin, and afterwards continue with the caveman regimen.

For me the dead skin layer is visible only when I sweat heavily for more than 30 minutes, which happens during an prolonged aerobic exercise. The dripping wet dead skin turns into a white paste making me look extremely pale.