This may or may not be plausible, but here goes nothing:
Hardware plugins are in the form of native DLLs. The DLL should specify an initialization routine, shutdown routine, and an ioctl routine.
68k side interface:
A trap task (TT) for connecting, accepting a pointer to the name of the DLL, returning the Device ID.
Another one for disconnecting, accepting the Device ID.
And finally, one for controlling the device, accepting a Device ID,an IOCTL code, and an argument, returning the result of the IOCTL function.
Native interface (for the actual plugin writers):
OnConnect: Function to run during connect TT. Accepts char*, returns int.
OnDisconnect: Function to run during disconnect TT. Accepts int, returns void.
OnControl: Function to run when control TT is invoked. Accepts three ints, returns int.
Sim_Memory(): returns a pointer to the 68K's main memory. Index this pointer to read and write the sim's memory.
Interrupt(int): Trigger a 68K interrupt in user code. The argument is the index into the 68K's interrupt table (I.E 0=Reset, 1=Bus Error, 2=Address Error, etc.)
Sim_D(): Returns a pointer to the Dn registers. Index this to get the specific register.
Sim_A(): Returns a pointer to the An registers. Index this to get the specific register.
Sim_SR(): Returns a pointer to the System Register.
Sim_PC(): Returns a pointer to the Program Counter.
Something that I also thought of: Using the high byte of a 68k pointer as a Device ID to do memory-mapped IO. If the device isn't found, just raise a Bus Error.