Avoid holding HRTF accumulation samples per-source