In the string class apex, what does the .substring function do?

From the String class apex documentation-

substring(startIndex)

Returns a new String that begins with the character at the specified zero-based startIndex and extends to the end of the String. So given that String.valueOf(results.get('capital_social')) evaluates to BRL 20.000,00 (Twenty thousand reais)... following that with substring(3) was previously how you were removing the "BRL" (but not the space between that and the numeric portion) from the beginning of your string. substring(3) cuts out the first 3 characters (the returned string starts with character 4 (i.e. index 3) of the original string)

 B  R  L     2  0  .  0  0  0  ,   0   0  ...
[0][1][2][3][4][5][6][7][8][9][10][11][12]...
[discard][             keep              ]
When you changed the code to use .replaceAll('[^0-9,.]', '') though, that was taking care of removing the "BRL " (with the space) as well as the "(Twenty thousand reais)", leaving you with 20,000.00
By keeping the substring(3), you were now discarding part of your desired data
 2  0  .  0  0  0  ,  0  0
[0][1][2][3][4][5][6][7][8]
[discard][     keep       ]
Now your string is 000,00, and you remove "." (which doesn't exist in your string at this point) and replace(',', '.').
So the final step in that statement is effectively Decimal.valueOf('000.00').

Leading zeros are removed, except for the first digit, leaving you with your observed value of 0.00.

By removing the substring(3), the rest of your steps were working on the value 20.000,00. With no data discarded, the rest of your processing steps ended up as Decimal.valueOf('20000.00');