Yeah, if there is no sign, there is no place for a new house to go. You'll need to put a sign back somewhere in town where you want the new neighbor to live.