Below are two setups to ensure that there will be no unwanted guests joining your Zoom Meeting.
1) Check the box to add a password to the meeting. Without the meeting URL and password, unwanted guests will not be able to access the meeting.
2) On the calendar invitation in Google Calendar towards the middle you should see a briefcase icon. Next to this, you will want to change the visibility of this event to "Private". Doing this will ensure that only invited guests can see this calendar event's details such as the meeting ID and password. Even those that have access to view your calendar will not be able to see the meeting information unless they are directly added to the event.