Is there a way to make the user's back button on their browser, call a javascript function instead of going back a page?
You can't override the behaviour that if a user follows a link to your page, clicking Back will take them off it again.
But you can make JavaScript actions on your page add entries into the history as though they were clicks to new pages, and control what happens with Back and Forward in the context of those clicks.
There are JavaScript libraries to help with this, with Really Simple History being a popular example.
