You’re right that the size of a library (file size, not just file count) is an important factor in the download time from iCloud, but it also depends on the speed of your connection. In my experience, the sync speed if Apple apps with iCloud is much slower than my available bandwidth, and the progress bar for sync is notoriously inaccurate. Just the other day I was helping a friend and the download estimate was something like four days, and then a few minutes later it said 11 hours.
Summary: syncing content from iCloud is not something you should watch and wait for. It always works for me, but it’s slow and the process indicators aren’t super useful. The one single variable that will make the most difference is to connect your computer to your internet sorter via an ethernet cable and disable wifi.
Regarding the download you thought was complete and ejecting the drive, that sound fine, but I’d have to see it for myself.