Concave
A concave brush is any brush that is not convex. If a brush is convex, then any imaginary line you could draw from one vertex to another will never be outside the brush. Therefore, if such a line can be drawn, the brush is concave. To put it another way, a brush is concave if any of its vertices sticks inward instead of outward.
Concave brushes are not allowed in the BSP format. Hammer and the compile tools will report concave brushes so you can repair them (by reshaping solids) if your map is still open. However, Hammer will not be able to load concave brushes from a saved map. If Hammer cannot fix them when the map is opened, they will be ignored.
Note how the left and middle brushes are convex (legal) while the right brush is concave (illegal).