RGB stands for red, green, blue. In a TGA or VTF, each pixel of the image is defined by three bytes that define how much red, green, and blue light are added together to produce the color in that pixel. The three separate values for each color are called channels. Shaders in Source can manipulate each channel individually to produce color effects.
Image files can also have a fourth channel, called alpha, which can define other information for each pixel, usually opacity.
Within materials, an RGB colour is a material vector.