A fillet is a small piece of wood that acts as a decorative edge. It does not have a rabbet edge like picture frames do.
This is the profile of a common fillet.
In this diagram you can see how the fillet works with the mat.
In the above picture you can see the fillet is attached to the top mat and then there is a matboard underneath it.
You can also attach fillets to the lip of a frame as you can see in the picture below.
There are a lot of different things you can do with a fillet in your framing design. The most important thing to remember is to make the fillet work with the design and for you to like what you see.