Hey there, Dave.
I've not read through all of the responses so apologies if I'm only reiterating what's already been said.
1) FOR THAT UGLY GRAVEL PATCH YOU DO (OR DON'T) SEE ...
The age old science of "a body in motion tends to stay in motion" has its equivalent truths in motorcycling. Wether you've got your eyes on the prize, meaning looking ahead and understand "yep, that's gravel" or are rounding a corner and the shift of geometry catches you off guard, the key ... the BIG key ... is to remain calm and:
LET THE BIKE CORRECT ITSELF.
One of my own early-riding-career-gurus used to say "chicken wings, chicken wings". Keep those arms LOOSE. Keep your attention AHEAD. Keep the throttle in PACE ... meaning don't chop it (wring it out) and don't "spazz" and roll off out of paranoia.
NOTHING about this is comfortable but it happens. It straight up "heart attack" happens ... and it's up to YOU to maintain a separation of mind/body so that your physics remain intact enough to remedy themselves.
I'm with you in that I hate "that feeling" of "Welp, it's up to the bike now" but so long as you're looking ahead and processing the horizon, 9 times out of 10 the geometry will regain it's center of balance.
2) FOR THAT GRAVEL ROAD THAT'S DEEP PEET (MARBLE SIZED TACK AND SEAL) ...
All of the above but with a bit more confidence. You know the bike is going to "dance". It's your job to pick a path of travel that allows it's sudden shifts to buffer themselves ... AGAIN ... eyeing the road ahead and ONLY the road ahead. KEEP YOUR GRIP LOOSE AND YOUR ELBOWS RELAXED. Eyes on the prize ... chicken wings ... yada yada.
3) FOR THAT HARD PACKED DIRT ...
Treat it with respect and you can almost behave as if it's pavement. The difference? That "pavement" has a "traffic police officer" that, rather than writing you a speeding ticket, will hurl your bike onto the ground. Point: don't experiment and don't over do it. Keep things five miles below the speed limit or anticipated hazards pace.
I hope this helps. Not intending to be blow hard. I just want your bike (and you) to remain intact.
PS ...
LEARN TO RIDE DIRT AND BECOME A BETTER STREET RIDER AS A SIDE ORDER