Let me preface by saying I'm very much a newbie. So much so that I have not written M68k code yet (I have only read it). I have also only thought in integers (so division is a new game for me).
An iterative strategy may be necessary if you don't have access to higher math functions: https://en.wikipedia.org/wiki/Nth_root_algorithm
With this algorithm, you could perform a square root by combining multiplication, division and addition. Somewhere there is learning material about the packed decimal storage strategies which allow for non-integer numbers too.
;put the value we are interested in finding the square root of in D0
;use 1 as the iteration difference tolerance -> when the difference between loop results is smaller than this, we're done
MOVE.L #1,D1 ;we need a first guess for the square root value (this is x_k)
MOVE.L D0,D3 ;store a temporary version of the number which we are finding the square root of
DIVU.L D1,D3 ;D3 contains -> refer to wiki article: A/x (the remainder is discarded)
ADD.L D1,D3 ;D3 now contains -> refer to wiki article: (x+A/x)
DIVU.L #2,D3 ;D3 now contains -> refer to wiki article: 1/2*(x+A/x) -> this is our next guess for the square root, x_(k+1)
MOVE.L D3,D2 ;temporarily store our guess in D2
SUB.L D1,D2 ;D2 contains the difference between the last sqrt guess, in D1, and the new sqrt guess, in D3
MOVE.L D3,D1 ;get ready for the next iteration
CMP.L #1,D2 ; D2 minus 1 -> set condition codes
BHI loopStart ;if D2 > #1, we are outside our tolerance so we loop again
;when finished, D3 and D1 both contain the square root
This code is quite daft since it uses too many registers for such a simple operation. It could even be outright wrong. Let me know what you think. I have not practiced M68k code yet. I'm only answering so that you don't get complete silence.