November 28, 2020

Since Apple's new macOS Big Sur dropped a little over 2 weeks ago, some of you may want to tinker around with the new OS in your homelab.

Below I'll list out the steps I used to get this up and running in my lab environment.

More...

This guide is primarily written for an Intel based machine running a Type 1 bare metal hypervisor. 

For AMD machines. Please refer to the comments at the end of this guide for any workarounds. One of my readers named Vince Whiston found a solution to his AMD issues using VMware Workstation here.

Prerequisites

  1. macOS 11 Big Sur ISO
  2. VMware ESXi 7 Update 1
  3. macOS Unlocker

We'll need these items above to complete this entire process. Don't worry, I'll walk you through all the steps in obtaining these items.

macOS 11 Big Sur ISO

Ok first off we'll need the macOS 11 Big Sur iso. Throughout the vast interwebs, you can search and find the Big Sur ISO all over the place to download.

So go get it! But...

...if you do go this route, make sure you look up how to validate the SHA1 and MD5 hash values of the iso before using it. 

I, for one, like to create it myself using another macOS native machine like a Macbook. I know some of you don't have access to a dedicated macOS environment and it's probably the reason why you are following this guide. 🙂 

I just like to know that the iso is clean and not injected with anything malicious. 

With this...Can I use iCloud, iMessage, etc...?

Download macOS 11 Big Sur ISO

  • On your macOS dedicated device, navigate to download the Big Sur official release here
  • Once you've clicked the link, the App Store App will open prompting you to Get Big Sur. 

Once you've clicked on the Get button. Continue downloading Big Sur which sits at roughly 12.18 GB. Depending on your network connection the download time may vary. 

DO NOT CLICK CONTINUE!

Once the download completes, you'll be prompted with the install window to Continue. Instead of continuing, simply click on Command + Q to close the window. 

You'll now have the Big Sur App listed under your application folder. This is what we'll use to convert it into an iso in the next step. 

Mount Installer then Convert to ISO

Next we'll need to run a series of commands to mount the installer to ultimately convert to an iso to be used by ESXi. 

Launch terminal and run the following command. 

sudo -i

This command will then prompt you for the administrator password for your specific device. Once you enter it in, you'll be logged in as the root user which is the account we'll use to execute the rest of the commands. 

From the root# prompt, enter the following command. 

 hdiutil create -o /tmp/bigsur -size 12700.1m -volname bigsur -layout SPUD -fs HFS+J

This will create the directory /tmp/bigsur.dmg

Next we are going to mount the dmg using the following command. 

hdiutil attach /tmp/BigSur.dmg -noverify -mountpoint /Volumes/BigSur

This should result in the following output below. 

Then we'll create the install media from our mount with this command. 

sudo /Applications/Install\ macOS\ Big\ Sur.app/Contents/Resources/createinstallmedia --volume /Volumes/bigsur --nointeraction

The output below is what you should see while making the disk bootable. 

Afterwards, eject the attached disk using the command below

hdiutil eject -force /volumes/Install\ macOS\ Big\ Sur

This should result in the disk being ejected. 

Then finally we need to convert the image to a .cdr with the following command. A .cdr is essentially a macOS version of an .iso. Not to be confused with a Compact Disc Recordable. 

hdiutil convert /tmp/bigsur.dmg -format UDTO -o /Users/erickdimalanta/Desktop/BigSur.cdr

Make sure you change the /Users/erickdimalanta/Desktop/BigSur.cdr to your own username for the path you are working in. Once you run the command below you should see a BigSur.cdr file on your desktop which sits at about 13.32 GB. 

Now all we need to do is simply rename the file to .iso. I like running the mv(move) command in place below to essentially do a rename of the file. 

mv /Users/erickdimalanta/Desktop/BigSur.cdr /Users/erickdimalanta/Desktop/BigSur.iso

Don't forget to change the username path to your own username. Or you can simply rename the actual file by:

  1. Select file
  2. Press Return/Enter
  3. Rename .cdr to .iso
  4. Confirm the change to Use .iso

Clean Up

I hate leaving trails of data all over the place. So let's cleanup after ourselves. If you remember initially we created the bigsur.dmg in the following path /tmp/bigsur.dmg. Let's clean up this directory with the following command. 

rm -rf /tmp/bigsur*

This command force removes everything in the directory specified without prompting for confirmation. So we are removing everything in the /tmp folder with anything that begins with bigsur.

Transfer ISO to Datastore

Next we need to navigate to the VMware homelab environment and transfer the newly created BigSur.iso to either a datastore or Content Library if you are running a vCenter. For purposes of this guide, I'll simply upload it to a datastore. 

  1. Login to your ESXi host
  2. Navigate to Storage and select the Datastore you want to place the .iso in. 
  3. Click on Datastore Browser
  4. Select the Datastore if not selected already
  5. Select the folder you'll place the .iso in. I'll use the ISO folder
  6. Then click on Upload

Navigate to the directory of the BigSur.iso we created. Select the BigSur.iso and click on Open

Review the Recent Tasks then monitor to completion. 

macOS Unlocker Install

Now that the .iso is in the datastore, it's time to prep the host to be able to run macOS in a VM. There is an unlocker written in python that modifies the vmware-vmx file to allow macOS to boot. Without this unlocker, it simply doesn't work and just does a boot loop. It'll show the Apple Logo loading screen and then ultimately displays an error. 

Your computer restarted because of a problem. Press a key or wait
a few seconds to continue starting up.

Disclaimer: This unlocker should not to be run in a production environment. This workaround is only to personally evaluate and have a macOS environment for purposes of testing. 

Download the Unlocker

You'll first need to download the unlocker zip file here if you didn't do a git clone. 

I ran the git clone to obtain the esxi-unlocker-301.tgz file.

However, if you simply downloaded the unlocker zip file, your downloaded file should be named esxi-unlocker-master.zip

Upload to Datastore

Once you've obtained the esxi-unlocker-301.tgz or esxi-unlocker-master.zip. Upload this to the same location we uploaded the BigSur.iso in the steps above. 

esxi-unlocker-301.tgz

esxi-unlocker-master.zip

SSH into ESXi Host to Install Unlocker

SSH into your host and navigate to the unlocker location. Once there, unzip the unlocker using the following command. 

For esxi-unlocker-301.tgz

tar xzvf esxi-unlocker-301.tgz

Uhh...the heck is a TAR or .tgz file? 

For esxi-unlocker-master.zip (skip if you've done the above step)

unzip esxi-unlocker-master.zip

Install Unlocker Patch

Now that we've extracted the archive, we can run the install patch. There is actually a validation checker to make sure the patch is successful when installed. Let's do that first. Run the following command to validate. 

./esxi-smctest.sh

We see the result obviously shows false since we've not yet installed the patch. Now let's do that. Run the following command to install the patch. 

./esxi-install.sh

For the esxi-unlocker-master.zip version, you'll need to navigate down to the extracted folder using the below command. From here you'll be able to view all the above files to run the patch.  

cd esxi-unlocker-301/

Before running the patch you may receive a "Permission denied" error. Simply navigate back one level using the command "cd .." (without the quotes) and run the following command against the root folder to grant permissions to the folder and all the files recursively within. 

chmod 775 -R esxi-unlocker-301/

After running the permissions command, we can navigate inside the folder again using "cd esxi-unlocker-301"

Once we've run the install "./esxi-install.sh", we'll need to now reboot the server for the configuration changes to take effect. In the same command window, you can simply type reboot then hit Enter. 

reboot

Validate Successful Patch Install

Once the ESXi host is back up, ssh into the host and navigate to the same location where we unzipped the Unlocker file. We are going to want to run the validation command again to see the result. 

./esxi-smctest.sh

Now we see the result is quite different. It now shows that smcPresent = true. We should now be able to install macOS without running into that boot loop. 

macOS 11 Big Sur VM Install

Yey we're almost there.  

Now that the unlocker has been installed and verified, let's now continue on to install Big Sur. 

Create New Virtual Machine

Let's navigate to the Virtual Machine menu item on the left then click on Create / Register VM. 

Then select Create a new virtual machine and click on Next

Next type out a name for your new VM. Select the following options from the drop down menu for the rest of the categories. When finished, click on Next. 

From here, you'll want to select your storage on which datastore to place your machine. 

The next option is to configure the hardware specs of the VM.

So set your CPU, Memory, Storage size, type of provisioning, Network, and most importantly configure your CD/DVD Drive to boot from the BigSur.iso we've created. Remember, this is located on the datastore we've uploaded it to in an earlier step. Once done, click on Next

Then finally validate your settings, then click on Finish when complete. 

Power On VM & Format Hard Disk

Finally, let's power on our newly created VM to see if it bypasses the boot loop since applying the Unlocker patch. 

Woot! Success! No more boot loop. 😀

Ok so let's continue on with the install. Choose your language of choice and click on the bottom right arrow

We'll need to format the newly provisioned disk so that macOS can recognize the disk to install itself to. 

Select the Disk Utility and click on Continue

Highlight the VMware Virtual SATA Hard Drive Media and click on the Erase button. 

Enter the Name for your Disk and keep the defaults for Format - APFS & Scheme - GUID Partition Map

Once done, click on Erase

The formatting process will begin. Once complete click Done

Now quit Disk Utility by navigating to the top menu and selecting Disk Utility > Quit Disk Utility

Install macOS 11 Big Sur

From here click on Install macOS Big Sur then click on Continue

Click Continue here

Click on Agree twice to Accept the Software License Agreement.

Select the newly formatted HDD then click on Continue

Big Sur will now start to install...

Don't be fooled by the initial time it tells you the install will be done by. In reality it took over an hour or so. 

Anyway, install complete. I won't go through the rest of the self-explanatory setup.

You now have your macOS 11 environment to play around in. Enjoy! 🙂

**Credits to shanyungyang's github and everyone else involved that helped me put this simple guide together and making this work.**

RELATED POSTS

  • Hi – Weird one I can’t work out, my root user on ssh can’t run the script

    [root@localhost:/vmfs/volumes/609a6ea6-117a2926-0468-1c697a634ab7/mactool] ./esxi-install.sh
    -sh: ./esxi-install.sh: Operation not permitted

    It has +x and even tried 777 perms on the file, wont run

  • Anyone with working Nvidia card GPU passthrough? Passhtrough works but drivers installation seems to be issue. I was able to load drivers by some miracle but it dones not work still.

    • Sorry for the late reply Robert. I have an Nvidia Grid K2 but I haven’t tried passthrough just yet. I know vSphere 7 changed some of the behavior regarding GPU passthrough. I’ll have to do some testing when I can.

  • doesn’t work for me, i tried it on pre 7.0.1 and 7.0.1a, after using the unlocker and reboot, my esxi complain about unlocker.gz and refuse to boot.

    currently i’m running on 7.0.1d, can you do an update to the guide please? thanks!

  • Erick, thanks for the invaluable article. One issue I’m running into is that sometimes the keyboard doesn’t work in special situations. For example, on the initial macOS login screen I can type in my password and log in just fine but when the Mac is locked (e.g. Lock Screen command is given) I can no longer enter my password. I end up having to force reboot the VM to get back into the OS. I can’t figure out why. This happens in some other situations as well. I’m using the standard US Keyboard settings but I am using a Macbook Pro to login to the VM. I’m doing this through the ESXi Embedded Host Client.

  • Most-Excellent Tutorial – Thank You! I successfully install Big Sur 11.2.3. A couple of things that may help someone else.

    – had to disable “Secure Boot” on the ESXi server

    – With this command: hdiutil create -o /tmp/bigsur -size 12700.1m -volname bigsur ……
    I had to bump the value up to 13100.1

    Again, Thank You!

    • Great. Thanks for sharing. 🙂

      That makes sense since this guide was written for the initial release of macOS 11. Bumping the size up will account for the updates at 11.2.3 when creating a volume image from that version.

  • Great guide! However, 75% into to install, the VM reboots and I return to the “Recovery” screen. Any idea why this could happen?

        • The unlocker should actually work on 6.5, 6.7 & 7. You running Intel or AMD? Can you try running the “./esxi-smctest.sh” and see if it outputs correctly according to the guide.

          I would try an uninstall/reinstall of the patch as well. Also to clarify on you first message. Big Sur goes 75% into the install then resets? I would try completely wiping the disk and formatting again as APFS.

        • I had exactly the same issue as Vincent. Esxi 6.5 in combination with Big Sur 11.3.2. Tried reinstalling unlocker and with Big Sur image 11.0.1, with no luck.
          I did a fresh install on a new usb drive of esxi 7.1 and imported the already created VM with Big Sur 11.0.1 on it, installation continued and no more boot loop. Will try following days to create a VM with Big Sur 11.3.2 on Esxi 7.1…

  • Hello, the unlock worked perfectly but after I try power on VM (first for install), apple logo appears, then boot loop with error message. What I messed up? 🙁

    • Hi Adam. Sounds like the error that happens when the unlocker isn’t installed. Can you ssh into your host and run “./esxi-smctest.sh” which should show smcPresent = True. Example below.

      unlocker-validated

        • Yikes. Hmm..first thing I would do, would be to attempt a reinstall.

          1. Run ./esxi-uninstall.sh
          2. Reboot
          3. After reboot run ./esxi-install.sh
          4. Reboot
          5. After reboot run ./esxi-smctest.sh

          • Hi Erick, done but same result 🙁 Can be that my config (Asus Z10PE-D8 WS with 2 x E5-2680 v3) is not supported somehow?

          • Ah bummer! Few quick q’s:

            1. Are you running ESXi as a Type 1 on bare metal or Type 2 using VMware Workstation?
            2. Did you create the big sur iso yourself or download it?
            3. Can you screenshot and share any errors?

          • 1. Type 1 on bare metal (and EXSI 7.0 Update 1)
            2. Yes I made it, based on your tutorial (my original downloaded installer version is 11.2.1, did you tried with this version or just older one? Maybe is this the problem? My is newer?)
            3. I just see boot loop few times (apple logo with loader bar then “Your computer restarted…” then again logo, etc) after shows only a link (support.apple.com/mac/startup)

          • Since it lets you see the apple logo, is it possible to get into the macOS recovery? Otherwise I would check the vmware.log located within the virtual machine directory on the datastore.

          • Adam. Yea I’m not entirely sure just yet. I did see this in your log which looks to be the cause of the boot loop.

            2021-03-04T16:00:03.266Z| vcpu-0| W003: DarwinPanic: panic(cpu 0 caller 0xffffff8001200940): “root image authentication failed (err = 22)n”@/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-7195.81.3/bsd/kern/imageboot.c:1015
            2021-03-04T16:00:03.271Z| vcpu-0| I005: Chipset: The guest has requested that the virtual machine be hard reset.

            This guide was written for version 11 and I have yet to try to install 11.2.1. The version difference may not be the exact reason for your boot loop but I’m just thinking out loud here. Perhaps I’ll give it a shot this weekend.

          • Thank You Erick 🙂 Till then I’ll try install 11 today (if I find this version on the web). By the way did you updated your 11 after install? Thank You for help

          • I haven’t updated it since installing and writing this guide. I have been doing a bit of maintenance on my home lab server so it’s been down for a bit. I’ll see if I can find the time to work on it this weekend. 🙂

          • Hi Erick, I downloaded the first public version (11.0.1) and it’s working. I wanted to check what happens after I upgrade to current version, but the updater says this (11.0.1) is the newest, so I leave as is. I hope there will be a solution for newer versions to work with ESXI.

          • Adam that’s great. Glad to hear that version works. I haven’t gotten around to testing the upgrade on my setup just yet.

            If the update eventually becomes available on your Big Sur install, make sure you take a snapshot before you upgrade in case it stops working so you can simply revert the snap. 🙂

          • Hi Erick, this uprade thing is very strange because when I installed the Big Sur (11.0.1) on my mac, the update become available right away, only the Big Sur on ESXI acts strange, so I don’t know will the upgrade available ever 🙁

  • Thanks Erick Dimalanta for the wall through.

    Im having a bit of trouble though, can you confirm if your using Intel or AMD CPU in the host also is the ESXi nested in VMware Workstation, or another, or a Baremetal install.

    Trying to figure out if its because im using a Ryzen or becasue ive nested the ESXi while for testing.

    I can confirm that version 3.0.2 installs with the notes from Michael and the unlocker shows in “cd /etc/rc.local.d”

    Install just sits on a black screen with apple logo.

      • Hi Erick, So I’m certain that the problem is related to the AMD CPU or CPUID specifically.

        I’m VMware Workstation i need to add the following to get MacOS to progress past black screen

        cpuid.0.eax = “0000:0000:0000:0000:0000:0000:0000:1011”
        cpuid.0.ebx = “0111:0101:0110:1110:0110:0101:0100:0111”
        cpuid.0.ecx = “0110:1100:0110:0101:0111:0100:0110:1110”
        cpuid.0.edx = “0100:1001:0110:0101:0110:1110:0110:1001″
        cpuid.1.eax= ” 0000:0000:0000:0001:0000:0110:0111:0001″
        cpuid.1.ebx = “0000:0010:0000:0001:0000:1000:0000:0000”
        cpuid.1.ecx = “1000:0010:1001:1000:0010:0010:0000:0011”
        cpuid.1.edx = “0000:1111:1010:1011:1111:1011:1111:1111”
        featureCompat.enable = “FALSE”

        I can add the CPUID to the VMX file on ESXi and the VM will turn on but again not progress to install.

        But when i add the featureCompat.enable = “FALSE” to the VM in ESXi the VM fails to load at all.

        Two questions, what does featureCompat.enable = “FALSE” actually do and is there an ESXi equivalent?

          • After nearly giving up I found the below settings for AMD systems running ESXI 7.0.1

            smc.version = “0”
            cpuid.0.eax = “0000:0000:0000:0000:0000:0000:0000:1011”
            cpuid.0.ebx = “0111:0101:0110:1110:0110:0101:0100:0111”
            cpuid.0.ecx = “0110:1100:0110:0101:0111:0100:0110:1110”
            cpuid.0.edx = “0100:1001:0110:0101:0110:1110:0110:1001”
            cpuid.1.eax = “0000:0000:0000:0001:0000:0110:0111:0001”
            cpuid.1.ebx = “0000:0010:0000:0001:0000:1000:0000:0000”
            cpuid.1.ecx = “1000:0010:1001:1000:0010:0010:0000:0011”
            cpuid.1.edx = “0000:0111:1000:1011:1111:1011:1111:1111”
            smbios.reflectHost = “TRUE”
            hw.model = “MacBookPro14,3”
            board-id = “Mac-551B86E5744E2388”

            Also th epost installation tweaks are healfull in Getting the Mac App Store working

            https://amd-vm.hackintosh-guides.ml/post-installation-tweaks

            Erick, i know you can test the AMD settings but did you want to include in walk through? I can send screen shots if needed.

          • Awesomesauce Vince! This is great and I’m glad to see that worked out for you. I’ve updated my guide to refer to this comment thread for any AMD based setups. 🙂

          • Hi Vince,

            thank you for sharing this information.

            I started having the same ‘stuck on apple logo’ issue after I upgraded my esxi box from an intel xeon to an AMD Ryzen 9 3950X.

            While this information was not sufficient to ‘unstuck’ my macOS installation, it did point me in the right direction.

            In my case, I had to change this line:
            cpuid.1.edx = “0000:1111:1010:1011:1111:1011:1111:1111”
            to this
            cpuid.1.edx = “0000:0111:1000:1011:1111:1011:1111:1111”

            After that, my big sur installation booted up just fine 🙂

          • Hi Michele. Thanks for stopping by and sharing your findings to the community. I’m sure it’ll be a huge help to anyone that has a similar issue.

            awesome

  • Hi, I install Big Sur on my PC, after installation it work fine, install an update 11.2 and still work fine, but 11.3 is not working it it crash back to the login screen also the 11.3 beta 2 and same thing. is there a fix? using vmware workstation 16.0. thanks

    • Hi Michael. This guide was written specifically for ESXi running on bare metal as a type 1 hypervisor and patching ESXi. This does not apply to type 2 hypervisors such as VMware Workstation. Although, I do recall seeing guides for that floating around the interwebs. 😉

  • Hey.. I try this HowTo on my AMD Epyc Server with ESXi 7.0u1, but nothing happened. Starting the VM, the apple is showing and nothing happen then. Is it because I have an AMD host?

    • The patch was written to modify esxi which loads the patch at esxi boot. Can you ssh into your host and run the ./esxi-smctest.sh and see the following result below?

      smctest

      I have not personally tested this on AMD based procs.

  • Erick,

    Excellent guide! Thanks for all of the screenshots and great directions! However, it seems like people are getting caught up with the ESXi Unlocker.

    Using the ZIP is easiest since it can be downloaded right from Git. However, the directions say to upload the file to the datastore, unzip, and run the install script. But that produces the error about unlocker.tgz not existing.

    Looks like there are 2 other files in there that could build this. esxi-build.py appears to only run on a Mac, and looks like it will create the necessary file. Sounds like this is the correct step.

    There is also unlocker.py which runs on ESXi and enables the SMC, but does not survive the reboot.

    Any chance to include the missing steps about creating the unlocker.tgz file on the Mac?

    Thanks for helping everyone with this great article!

    Michael

      • Unfortunately, I’m getting an error of “Copied archive /bootbank/unlocker.tgz is corrupt, deleting: 1”. Maybe I just need to use the esxi-build.sh to create one of these files on a Mac? Any tips? Thank you!

        • Hi Michael. You’re right. There is definitely something wrong with this 302 package. I’ve updated the hyperlink again with the 301 version which includes all the files you’ll need.

          I also updated the guide under the Install Unlocker Patch with a few more quick steps to ensure everything works correctly. I’ve just tested this myself and it all seems to work as intended.

          Thanks again for taking the time to comment and let me know about this incorrect step. Hope the new updates help. 😉

          • Erick,

            I actually just got version 3.02 to work! It took a bit of testing, but I finally figured it out.

            After unzipping the .zip file in the datastore, I needed to make the .sh scripts executable by running “chmod +x esxi*”. I found that I also needed to run “chmod 555 etc/rc.local.d/unlocker.py”. This sets up the file with the correct permissions for the system.

            Then your original .zip file you had (the one without the unlocker.tgz) was correct, but needed an extra command to create the .tgz file. Running “tar zcf unlocker.tgz etc” would build the unlocker.tgz file using the file in the etc/rc.local.d/ directory.

            If after running the esxi-install.sh file, the smctest reports “false” then it is most likely a permissions issue. You can check this by running “cd /etc/rc.local.d” and then running “ls -lah”. You should see the unlocker.py file in there, and it needs to have read and execute permissions for all 3 groups. Unfortunately, you can’t change the permissions here, as a reboot will reset them. You need to run the uninstall script, make sure the permissions are correct with the unlocker.py file that was from the zip file.

            Thank you for helping us get to the bottom of these issues! I hope this helps everyone who is running into issues!

          • Awesome work Michael. Glad you dug a little deeper than I did, as I simply grabbed the v301 files that I knew worked when I wrote this guide.

            Thanks again for your persistence.

            high_five

  • Eric,

    Great guide. All the steps worked perfectly. Used the git-clone method as you describe and built the unlocker.tgz on my mac. However after I successfully install the patch I run “esxi-smctest.sh” after the reboot and it still comes up as false. I am running ESXI 7.01 and use esxi-unlocker-302 which is supposed to support this version. Any ideas?

    Todd

    • Hi Todd. That’s odd. I would try the following:
      1. Run ./esxi-uninstall.sh
      2. Reboot
      3. After reboot run ./esxi-install.sh
      4. Reboot
      5. After reboot run ./esxi-smctest.sh

      Hopefully that works out. 🙂

  • Hello and thank you for the post. When running the VMware unlocker I receive an error stating “File unlocker.tgz does not exist”. Where do I find this file as I do not see it in the GitHub link either? Thank you in advance.

  • {"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}
    >
    Malcare WordPress Security