I'm slowly moving away from Gmail and Google Docs/Drive, but in the meantime entering this into uBlock Origin seems to help:
mail.google.com##div[jscontroller][jsaction]:has(> span > button[aria-label="Try Gemini"])
drive.google.com##div[jscontroller][jsaction]:has(> span > button[aria-label="Try Gemini"])
docs.google.com##div[jscontroller][jsaction]:has(> span > button[aria-label="Try Gemini"])