I use Elmers Carpenders glue or Iliens Tacky glue. Either seems to dry quickly (two hours or so) even with undried bricks.

I would guess its your glue... You may want to give some simple PVA (Elmers or equivalent) a try.

Phaze

We don't stop playing because we grow old; we grow old because we stop playing. Author unknown