Great questions. Let me start by saying I'm sorry for your loss. Whether it was a person or a pet each life serves a purpose and it is always sad when another light goes out on the earth. Why is a tricky question and it could be for a few reasons. It is really so individual that I can't generalize an answer. I am sorry. I can answer what happens though...
From what I've been told the soul is very clever. It actually knows when there is the threat of death and can/ does literally jump out of the physical body before passing. This is how we get near death experiences & astral travel (although this is not dying). The soul is attached to the body through an umbilical like cord. When it leaves it still remains attached, until such a time when the physical body ceases to function. In an 'out of body' or 'near death' experience, you'll hear of people saying they were hovering above the body, still able to see & hear all that was going on in the room. When this happens the person has no physical connection and therefore will not feel any pain or any comforts unfortunately. Once the physical body stops functioning the soul is then disconnected. They are then considered 'earth bound'. A good depiction of this is actually in the movie Ghost with Patrick Swayze and Whoopie Goldberg. Although the depiction of the medium is a bit cheesy. The spirit will remain this way until they become accustomed to their new metaphysical body, see that their funeral and other arrangements have been made and help their family to cope with their transition. Quite often shortly after the funeral, a family member will see this person in a dream. This would be the easiest way for them to let their family know that they made out ok and that they are fine.
At this point they would move into the state of limbo. Limbo is a neutral state. It's literally the energy or space between heaven and earth. When we are in limbo we attach ourselves to a person, place or thing. Everyone passes through limbo and it is not the purgatory other religions would have you believe. Sometimes people will choose to stay here because they aren't ready to move forward. They usually attach themselves to family or loved ones who are still struggling with their passing.
When they are ready they will move into heaven. They will do something called 'life review' or the 'judgement'. This is when we process all that took place in our lifetime. When this is completed they have their free will back to come and go as they please. At this time they will have the opportunity to come back as a new life. Usually not remembering anything from the previous life. In my experience it isn't unusual for a spirit to take 20 years or so to decide to come back!
Please remember friends we have free will on this side and it also extends to the other side. How quickly or slowly one moves through the different stages is entirely up to them.
I do hope that helps a bit. Know that whatever stage your loved one is in that they can and do hear you. So talk to them. They will be comforted by your words and prayers. This will help them to process faster and hopefully more to a better place of communication.
In any event, I wish you all love, and peace at this time.
Love & Light, Truly