I want specific control over the users who can use the add-in

You can do so in the registry.
By default the add-in will be loaded for all users. However, this behavior can be changed easily by removing the add-in from the HKLM and adding it to the HKEY_CURRENT_USER.

https://www.dragdrop.com/download/DragDrop_UserInstall_x86.reg (use with 32bit Outlook)

https://www.dragdrop.com/download/DragDrop_UserInstall_x64.reg (use with 64bit Outlook)

The x86 and x64 are linked to your OFFICE version (NOT to Windows!)

Please note that the removal of the keys from the HKEY_LOCAL_MACHINE might require admin/installation permissions.

Windows Registry Editor Version 5.00

[-HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office\Outlook\Addins\DragDrop]
[-HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Office\Outlook\Addins\DragDrop]

[HKEY_CURRENT_USER\Software\Microsoft\Office\Outlook\Addins\DragDrop]
"RequireShutdownNotification"=dword:00000001
"LoadBehavior"=dword:00000003
"FriendlyName"="DragDrop for Outlook"
"Manifest"="C:\\Program Files (x86)\\DragDrop\\DragDrop for Outlook - Enterprise\\x64\\DragDrop.vsto|vstolocal"
"Description"="Enables you to drag emails or attachments to your browser (Current User mode)"