VERTEX_BUFFER_SIZE

The VERTEX_BUFFER_SIZE is the limit on the size of a single vertex buffer. Multiple buffers are rendered per scene, so this isn't a limit that you should ever encounter. If you are hitting it, you've most likely got a bug in your mod's code or content.

Resolution to VERTEX_BUFFER_SIZE

This is an example of a brush with too many vertices. Note that the brush in this image is also non-convex. It has incorrect geometry!

This happens when your map is clogged up with too many brushes/brushes with many vertices. Try making small brushes that are close to each other a func_detail. Do this to as much of the map as possible to stop this occuring.