2009 update: Safer's De-Moss, specifically made for removing moss, is still available but considered a controlled herbicide in many jurisdictions. If you cannot find it in stores, check GOOGLE.
You can't mechanically remove this moss without bothering all the gravel, perhaps even the roof. You need to chemically break it down so you can rinse it away. Some companies do make specific moss cleaner products but when you can't find them use deck restorer chemicals, not deck cleaners. Deck cleaners are basically just soap and bleach. Deck restorers will dissolve the dry cellulose on the deck boards, and will break up live moss and allow it to flow away. Then scatter zinc pellets on the flat roof to prevent new moss from growing.