Unloading a loaded plugin is not possible. I don’t know of any workarounds that would make unloading possible with the plugin package. There are a couple of alternatives to the plugin package (I did a writeup some time ago here), but all these approaches only provide out-of-process connections between the plugin and its client.
Sun looked at this back in the LD_PRELOAD days, and found that unlinking something merely caused the dynamic linker to try to re-resolve the links. There was one use-case where that made sense, but in general it was pretty insane to undo something and then have it re-done the next time the application called the entry-point.
An application would need to manage all its plug-ins explicitly to avoid unlinking just being a no-op, but if it did, it could refrain from ever calling the plug-in again. A “mere matter of programming” for your application (;-))