How to make upsert in salesforce work instead of inserting new duplicate records?

From documentation:

The upsert in salesforce statement matches the sObjects with existing records by comparing values of one field. If you don’t specify a field when calling this statement, the upsert statement uses the sObject’s ID to match the sObject with existing records in Salesforce. Alternatively, you can specify a field to use for matching. For custom objects, specify a custom field marked as external ID. For standard objects, you can specify any field that has the idLookup property set to true. For example, the Email field of Contact or User has the idLookup property set.

Couple of points to remember:
If the key is not matched, a new object record is created.
If the key is matched once, the existing object record is updated.
If the key is matched multiple times, an error is generated and the object record is neither inserted or updated.
So it's quite clear how upsert statements work and what are configurable parameters. So in your case it's likely to create two records since there is no indexed field i.e. id or externalId to detect duplicate records.