Of course. It could work totally different from bash.

To be fair I saw some cool fish stuff and thought about using it but when maintaining a few thousand servers with bash its not viable to load an alternative shell to all of them.

In that case, I think you want

rm -rf "'folder'\$'\\003'"

Note the double escape before 003, which will render to the character literal \

Alternatively, start typing

rm -rf \'fo

And then hit tab until fish autocompletes the directory you want to kill, and run it.

Side note: i would absolutely not tolerate directories named like that lol

Is this some school task…? Why is there a folder with such a name 😅