If you aren't using it much, Im not sure what your usage is, it sounds like something may be running in the background because there is no reason for the battery to drain without usage so quickly. Or the battery could just be shit.
I don't know exactly how these batteries work, but our servers at work all have a battery (to protect data being written to a hard drive in case the power just cuts) and there are two things in terms of charge of a battery. One is absolute state and relative state. The absolute state is how much charge the battery can hold compared to the rating of the battery. This goes down over time and when the battery goes through charge cycles, it'll slowly eat into the absolute charge. The relative charge is the actual charge of the battery compared to the absolute charge. I am believing, and I have no source just my own knowledge from working with server batteries (which may or may not be the same/similar to cell phone batteries) that the % you see on your phone is actually the relative charge. It's very possible your absolute state of charge never actually gets to 100%, maybe it is operating at 50% due to age/usage/other factors. I have no idea if that's where you are at, but it's possible your battery is just shit at this point.
BUT I would download one of these power saver apps to see what it has to say about applications running in the background because even if your charge isn't good, having apps run in the background will still cause the battery to drain faster. Samsung is also known for "bloatware" where they have a lot of apps installed that you can't remove, but you may be able to disable them through a power saver app or through your application manager on the phone.
Can I change the batteries on a Galaxy 4 myself (without having to buy a shit ton of tools I'll never use again)?
Very easy to change on your own (and meant to, meaning its not meant to go to an apple store for a genius, but meant for anyone to replace),
https://www.youtube.com/watch?v=xBIEkbjs8Ms video tutorial. Sadly they took the removable battery out of these phones which is a huge negative IMO. Loved that feature.