Hyperbolic vs sigmoid - How are these different?

Hyperbolic vs Sigmoid

Sigmoid > Hyperbolic tangent: As you mentioned, the application of Sigmoid might be more convenient than hyperbolic tangent in the cases that we need a probability value at the output (as @matthew-graves says, we can fix this with a simple mapping/calibration step). In other layers, this makes no sense.

Hyperbolic tangent > Sigmoid: Hyperbolic tangent has a property called "approximates identity near the origin" which means tanh(0)=0, tanh′(0)=1, and tanh′(z) is continuous around z=0 (as opposed to σ(0)=0.5 and σ′(0)=0.25). This feature (which also exists in many other activation functions such as identity, arctan, and sinusoid) lets the network learn efficiently even when its weights are initialised with small values. In other cases (e.g. Sigmoid and ReLU) these small initial values can be problematic. Further Reading: Random Walk Initialization for Training Very Deep Feedforward Networks