I suspect for the type of edge you are looking for you may need to use a vector program, or try to make your sharp edged areas using the shapes function of painter.
Painter being a raster prorgam may be a bit more difficult. Try using some of the pens and the liquid ink brushes for the drawing and the softer brushes of the tinting.
What you see as pixelation is normal for many of the painterly brushes.
We have but one life, live it well!!
AMD FX(tm) 8 core processor, 16GB RAM, Radeon HD 7900 series video card 1 Gig RAM, Dual Monitors