Boil half of the potatoes in salted water until you can run a fork through them (about 20 minutes).
In the meantime, peel and grate the other half of the potatoes and set aside.
In a separate bowl, whisk together the flours, baking powder, salt, and rosemary.
When the potatoes are cool enough to handle, slip the skins off.
Place the boiled potatoes in a large bowl and mash.
Add the grated potatoes and the dry ingredients. Mix to combine.
Add the milk and mix everything together.
Preheat the oven to 350 degrees F. Line a baking sheet with parchment paper.
Place the dough on the baking sheet and pat into a round or oval shape.
Use a sharp knife to score some slashes across the top of the dough.
Place in the oven and bake for 1 hour. Check how moist the bread is inside. If it still seems sticky, turn the oven off and leave the bread in until the oven is cool.