The RIGOL MSO series have a different encryption than the DSOs. Thus you need a JTAG cable to dump the memory and extract the keys.
TS;JR
Basically you can just follow the steps in https://www.youtube.com/watch?v=OvcGn_ScG5w
However, at the step when you need to generate the keys with ./rigup license [keyfile] 0x1C0FF, do not just run the command. Run cat [keyfile], and compare with the device serial you obtained by SCPI IDN command. You will discover that the serial number is wrong.
rigup scan - Version 0.4.1
Hacked up for MSO1000Z(-S) rmd79, 0ff eevblog.com
RC5KEY1: 9BBBBBBBBBBBBBBBBBBBBBBBBBFAD3E6
RC5KEY2: 8BBBBBBBBBBBBBBBBBBBBBBBBB0A699C
XXTEAKEY: 8BBBBBBBBBBBBBBBBBBBBBBBBB24850B
PUBKEY: 00XXXXXXXXXXXXXX
PRIVKEY: 00XXXXXXXXXXXXXX
SERIAL: DS1ZA203000000 [ This is WRONG ]
Replace the serial number with the one you got using SCPI *IDN?.
Now you can generate the license with ./rigup [keyfile] mso1100z.txt 0x1C0FF.
And install with SCPI :SYSTem:OPTion:INSTall Y4HHHHW66SW9HHHHMU75BBBBBBBB.
Voilà! Now you have a scope with all options unlocked.