A little bit of plaster dust shouldn't cause too much trouble, but any kind of oil or similar will make a weaker bond.  Also, if the blocks aren't fully dry before you glue, the glue can soften a thin layer of plaster at the glue interface, which flakes off easily.

As for a level surface, build your floors face-down, so the top surface will be flat.  The bottom can be cleaned up a bit later, if needed, with a big flat file, rather than sandpaper.  I find that a medium-to-coarse file is fairly easy to clean out with a bristle brush, and it doesn't need to be incredibly sharp to smooth plaster (which it won't be after a few uses).