I own the E-P1, E-PL1s (Japan exclusive obtained via eBay), and E-P3. I used to have the E-PL1 but recently traded it. Honestly, unless you spend all day pixel peeping there is virtually zero difference between the way the current Olympus Pens render images. The E-P1 and E-P2 are "a little" softer since they have stronger AA filters on the sensors, but I really can't see much difference in image quality in practical use.
The E-P3 (and E-PL3 and E-PM1) have faster AF and faster art filter processing than the older Pens, but I rarely use the in-camera art filters and the slowest focusing Pen (the original E-P1) is just as fast as the E-P2, E-PL1, E-PL1s and E-PL2 after you update the E-P1 with the latest firmware. Lenses can make a big difference, though (the older 14-42mm kit lens is slower than the newest 14-42mm IIR kit lens).
Honestly, I tend to use the E-P3 and E-P1 more than my E-PL1s just because they handle in much the same way so I don't have to give much thought to using the cameras. If I was regularly switching between the E-PL1 and E-PM1 I might find myself stopping to remember how I change settings on one camera versus the other.
Getting back to the image quality, here are images taken with the E-P1, E-PL1, E-PL1s, and E-P3. Other than using different lenses, I can't really tell which camera is "better" in terms of how the sensor renders the image.
Join to see EXIF info for this image (if available)
Join to see EXIF info for this image (if available)
Join to see EXIF info for this image (if available)
Join to see EXIF info for this image (if available)