Put the bacon in a large skillet or on a griddle and cook, flipping it once, over medium heat until golden brown and crisp, about 10 minutes. Remove to a paper-towel-lined-plate. Break each piece in half and set aside.
Heat a cast-iron griddle or cast-iron pan over medium heat.
Spread some butter on one side of each slice of bread. Flip over 4 of the bread slices and layer each with 3 to 4 slices Brie, 2 slices green tomato, 2 slices goat cheese and 4 slices bacon. Sprinkle with salt and pepper. Place the remaining bread slices on top, butter-side up, and cook on the griddle until the bottom is golden brown, 3 to 4 minutes. Flip the sandwich over, cover with a heatproof bowl (to melt the cheese), and continue cooking until the bottom is golden brown and the Brie has melted, 3 to 4 minutes.