Settle the argument
Can you pull the flag of a player without the ball?
No. Only the runner’s flags.
A defender can only pull the flags of the runner. Pulling a flag from anyone else is a foul.
There are two exceptions: a player who pretends to be the runner (on a fake handoff, for example), and a player who has touched a pass in flight.
The rule
IFAF Flag Football Rules 2023, R 9-2-2-d
The call
S52 Flag guarding, illegal flag pullRaise an open hand, sweep it down to the hip.
Illegal flag pull: 5 yards from the basic spot (signal S52).

Other rulebooks
NFL FLAG and local leagues can differ. Check the rulebook your league plays.
Watch this play in the free flag football rules quiz.
Send this to whoever you’re arguing with
Think you know the rules?
Watch animated plays and make the call in the free rules quiz. Refs: print every penalty on a pocket card.