Justin Bieber and Hailey Baldwin got engaged just one month after rekindling their romance, and now it’s full steam ahead with the wedding.

“As of now, they want a small ceremony with their families,” a source close to Bieber tells PEOPLE exclusively. “They are not planning a huge, celebrity wedding. They are getting married for love and don’t want a flashy wedding.”

After the “Sorry” singer, 24, popped the question in the Bahamas on Saturday, the couple jetted back to New York City where Baldwin, 21, showed off her massive oval-cut ring (which he picked out because he could “see Hailey’s face in it”) while out and about on Wednesday and Thursday.

On Thursday, the two enjoyed a stroll around Brooklyn, where a casual Bieber offered his athleisure-clad fiancée a piece of gum. And on Wednesday, the stars were spotted boarding a helicopter in Manhattan to spend quality time upstate with Baldwin’s actor father Stephen. “Justin is still the happiest ever,” says the source. “They visited Hailey’s family.”

Bieber confirmed their engagement on Monday in a heartfelt post on Instagram, writing: “Was gonna wait a while to say anything but word travels fast, listen plain and simple Hailey I am soooo in love with everything about you! So committed to spending my life getting to know every single part of you loving you patiently and kindLY.”

Shortly after, Baldwin tweeted: “Not sure what I did in life to deserve such happiness but I am so utterly grateful to God for giving me such an incredible person to share my life with! No words could ever express my gratitude.”