If the plaster is not fully dry when you glue them, you get a thin layer of soft plaster at the interface. This makes for a weak joint, and may be what you are experiencing.

Here in damp and dreary Pennsylvania, it usually takes 2 full days for the blocks to dry completely, and it might take up to a week of these misty, off-and-on rainy days like we're getting right now.

Don't put your blocks in a dryer or low-temperature oven right away, give them at least a couple of hours for the water and plaster to fully react. That way, the chemical reaction has time to complete itself before you start removing the water. If there's any water still present by that point, it's a safe bet that the gypsum has absorbed all the water that it's going to.