What type of plaster are you using? That also might explain why they aren't drying very well. My first build was made from plaster of paris, and the blocks were still very wet and crumbly, even though they were cast the day before.

I use Merlin's Magic and Elmer's wood glue. I can usually glue pieces together right after I cast them and I don't have a lot of problems, as long as I'm careful.

When I am building, I tend to add an entire layer of bricks, and then let is sit for an hour or so before I add the next layer.