Prepare the vanilla bean by slicing it lengthwise and scraping the inside of the pod with the back of a knife, careful to remove only the soft seeds and not the fibrous hull. Peel and slice two of the peaches, removing the pit. Puree these peaches with the honey, vanilla bean contents, brandy, lemon juice, and vanilla in a food processor until smooth.
In a saucepan over med heat, cook 2 tablespoons of butter until light golden brown. Add peach puree to the butter and simmer until slightly thickened, about 5 minutes. Remove from heat and set aside.
For the topping, combine the flour, oats, sugar, cinnamon, pecans, and sea salt. Mix-in the softened butter by working the butter into the mixture with your fingertips until incorporated. Set aside.
Halve the remaining peaches, remove pit, leaving the skin intact. On the cut side, drizzle olive oil over peaches and sprinkle with a little cinnamon and nutmeg.
Grill peaches over med-high heat on a preheated grill, starting cut side down. Grill on each side about 3-5 minutes or until peaches are softened.
Transfer peaches to a cutting board. Once cool enough to handle, remove the skin and slice, arranging slices in an 8x8 buttered casserole dish.
Pour peach sauce over sliced peaches and top with crumb mixture. Bake at 350° for 20 minutes to brown topping and meld flavors. Or if using a metal baking pan, place back on the grill over indirect heat until browned and bubbly.
Serve warm with whipped cream or vanilla ice cream.