Lightly oil broiler pan. In small bowl, combine lemon juice, oil, fennel seed, salt and pepper; mix well. Place snapper on oiled pan. Brush with lemon juice mixture.
Broil 4 inches from heat for 5 minutes. Top each fillet with lemon slice; broil an additional 3 to 5 minutes or until fish is no longer translucent in center and flakes easily with fork. Sprinkle with olives.