In home assistant every entity has “assist” config, you can remove any entity from assist which in turn will prevent it from being included in “all” assist/voice target.

So you can have a plug for server without worry but need to remove it from assist.