Short version: You should.
Long version: It might have been stuck to your drain plug. Subaru uses a copper crush gasket that squeezes flat at the proper torque and makes a nice seal (saving your threads from stripping but letting you know when it's tight). If you buy a filter at Subaru, the gasket may come with it (which has confused many people who thought it went on the filter). Otherwise, they're a buch or two (American).