To “netboot” a mac, you need to setup a netboot server AND create a netboot image that is capable of running on the macs that you plan to netboot.

For a NetBoot server see:
https://github.com/jamf/NetSUS/tree/master/docs

To create a netboot image:

Start with a MacOS installer from the App Store like “Install MacOS High Sierra”.

Use the application AutoDMG to turn that installer into a DMG – disk image.
For AutoDMG see: https://www.jamf.com/jamf-nation/third-party-products/324/autodmg?view=info

Next use the application AutoCasperNBI to turn that DMG disk image into an Netboot Image or “.nbi” file.
For AutoCasperNBI see: https://www.jamf.com/jamf-nation/third-party-products/363/autocaspernbi?view=info

After that, copy the .nbi folder to your netboot server. 

Lately I have been running a NetBoot server on a NetSus server on ubuntu 16.04 (see above). On that server, you connect to the web portal and select netboot. There “upload a netboot image”. This should open an SMB connection to the netboot smb share and you copy the “.nbi” file to that location. Copy the “.nbi” file not the folder that contains it. By the way – I had to change the smbuser password in the web interface for the server before I could mount this share correctly.

Once the .nbi file is copied up you should be able to enable that netboot image in the web interface. Then it becomes available to macs on the vlan to boot to.

For sanity sake:
On MacOS netboot servers which are going away the netboot images live in: /Library/NetBoot/NetBootSP0/
On Ubuntu servers the netboot images live in: /srv/NetBoot/NetBootSP0/