This is an example of the custom button url that would be used to redirect a user to the Lightning Edit Modal. We also need to include the backgroundContext URL parameter, so the record page is loaded underneath the window.
Here is an example that references a Case object. The beginning of the URL is standard up to the saveURL2 parameter.
/apex/lcrm_scp__scpClone?rid={!Case.Id}&cfg=MyConfig&saveURL2=%2Flightning%2Fr%2FCase%2F%5Bnewid%5D%2Fedit%3FbackgroundContext%3D%252Flightning%252Fr%252FCase%252F%5Bnewid%5D%252Fview
The saveURL2 parameter is an encoded URL that contains [newid] in two spots. These are replaced with the new cloned record id. The backgroundContext URL parameter is double encoded, and this contains what should display behind the Edit modal.
Below is the parameter decoded one time. It’s easier to see the URL of the edit modal page, and the backgroundContext parameter with its encoded value.
/lightning/r/Case/[newid]/edit?backgroundContext=%2Flightning%2Fr%2FCase%2F[newid]%2Fview
Below is the backgroundContext decoded, and we can see the View page url.
/lightning/r/Case/[newid]/view
To customize this for your environment, you will need to replace a few things in the code.