A prefabricated chimney always has insulation in the chimney itself. Yet, when it is on the outside of the house, and when there is no fire burning, it gets as cold as outdoors. That?s when all that cold air wants to drop into the basement, giving you that downdraft every time you try to start a fire. Simply boxing in and insulating the chimney more won't make a very big difference, unless the chimney gets some heat from the house somehow. So if you were building a new house, and could leave out the insulation in the house wall between the house and the chimney -- leaving a blanket of insulation along the wall, around the chimney and then along the wall on the other side -- this would warm up the chimney and stop the down drafting. Essentially you would now have the chimney inside the house, or at least inside the thermal envelope.
In an existing house, the best way to prevent down drafting is simply to install an insulated metal chimney inside the house. That way only the portion of the chimney that is above the ceiling gets cold -- usually less than half the total chimney height. This will generally create an updraft force, which makes getting the fire started very easy. Of course it is more difficult to cut straight up through the ceilings and the roof, but the chimney always works much better.
Exhaust fans in a house can be causing downdrafts as well. In fact fans can affect any "Spillage Susceptible"