55L here and weigh 85 kg
I downsized from an 85L , and my conclusion is that the trade off in low end for maneuverability is too great to justify going below ~70-80L at my weight. However, my 55L is pretty heavy and almost weighs the same as my 85L, which is light for it's size. My conclusion is that weight makes the biggest difference for maneuverability, wave riding, and pumping.
If I'd do it all over again I'd say make sure you prioritize weight and don't bother pushing volume as low as possible. My opinion is you don't get much in return for loss in low end and safety (floatation in case the wind dies).