How does malicious javascript work? How can it be mitigated?

JavaScript gives web pages authors, good and bad, the ability to run any code they want when your browser visits or is steered to their page. Although the various JavaScript implementations have some security functions to try and keep JS code from doing anything overtly hostile to your computer, two problems emerge: that code has bugs, such as discussed in the earlier answers, which allow for attack or exploitation and many things that aren't outright hostile can lead to bad ends (eg a popup from a FakeAV which asks for your payment information).

Example current malicious Javascript-related attack techniques that are quite effective use hidden iframes to load JS malware from other compromised sites which then tries to execute in the browser. This is seen in advertisements included into big popular sites as well as in less well-trafficked ones. If successful it may then continue on to exploit local system software. In this manner the various versions of the Black Hole Exploit Kit attack vulnerable versions of PDF and Flash software to infect the host machine with botnet clients.

It's been difficult for browsers and system makers to make their legitimate messages hard to counterfeit. Windows User Account Control is one of the best techniques because of how it interrupts every other program when it needs privileges to complete a task. Most browser and software pop-up messages are easily faked and you should be wary of them.

To avoid much of this use browser add-ons, plugins, and configuration to only allow JavaScript from sites you trust or sites you authorize. NoScript for Firefox is quite effective at reducing these risks.