Hello and welcome to talk parrots.
You've only had your new bird for 4 weeks so you have a long way to go and I had to laugh at your description when you said "I don't want to sit by the cage all day."
Because that's exactly what you have to do.
You need to teach this bird that you can be trusted because the behavior its showing is it wants to bond with you but its new and frightened of you still. Slow and steady motions are needed, set the cage some where you can sit comfortably in front of it. Have the door open and place your hand in the cage and set it on a perch not near the bird, talk to the bird softly, read a book out loud to the bird to keep yourself entertained and look up at the bird while blinking your eyes so you don't spook him. People who stare at birds frighten them because only a predator will stare at a bird before attacking.
Do not move your hand, let the bird see you are not threatening it with any motions at all. Just talk and read and get the bird use to your voice.
You have to do this for an hour or more. DAILY. Every day! See if the bird comes closer to you during the time you are reading out loud. They do respond to voices so you will be luring the bird closer to you by talk.
Keep up the chatter. If the bird starts to feel comfortable it will start approaching you in a few days or so. Have some millet or treats to offer it in your fingers but do not chase the bird around the cage with your hand. Your hand is what it fears right now.
Leave your hand resting on the perch with the treat, get the bird to come to you and munch. Give it lots of praise as it comes closer and eats and keep doing this for days and weeks at a time. Soon you ill be able to gently stroke the birds tummy and press your finger to its belly and get the bird to step up.
It's a lot of work and takes a lot of time to get a bird use to you. You have to put in the work.
The bird is already excited about being your friend but its scared about your approach. Take your time daily and it will work out.