The issues with Mtgox right now seems to be two fold. Firstly, there doesn't seem to be a clear separation between the web interface and the backend trading engine, those two things seem very tightly coupled. Secondly there's only really one way to access the trading engine, if that goes offline trades can't get through and malicious users can use that fact to their advantage.
The diagram above outlines what you might want an exchange to look like. The colored boxes are trader facing, the white boxes are internal systems and hosts.
The trading engine
The core trading engine should be able to deal with 1000s of transactions per second, this should be easy to achieve, even with a single highend server. A cluster of servers, hosted at multiple sites, with transactions kept is sync would be non-trivial, but obviously desirable.
As well as maintaining a database of accounts and open trades the trading engine also writes all trades to a transaction log, the interface only allows new transactions to be created in the log, transactions can not be amended or deleted.
In order to execute trades the trading engine has access to a second host which is connected to the Bitcoin network, this host is used for transferring funds in and out of the Exchanges wallets. All Bitcoin transfer requests the mediated by a "transaction validation" server. The trading engine itself will of course already be enforcing trading limits, however the validation server will filter out abnormally large transactions, which will then require manual verification.
The server containing the wallets itself uses a separate Internet connection to communicate with the Bitcoin network. Funds are stored in multiple, encrypted wallets of at most 1000BTC. Some degree of manual intervention maybe required to bring wallets online.User facing Services
Our current issues are mostly related to decoupling the trading engine from the trading interface, and that only a single trading interface exists. I'd like to see multiple trading interfaces, which would limit the likelihood of a malicious entity blocking all trades.
I'd like to see the ability for traders to install local, possibly automated trading system in the same hosting facility as the trading engine. Some might see automated trading as a bad thing, but it may well help maintain stability.
I'd like to see automated telephone trading, this may well have lower trading limits, but it would be a back-channel which almost everyone can access if the web interface was offline. It might well require a challenge/response token for access.
Another option is to allow trading via a distributed network of IRC servers, again sitting on a separate Internet connection, and requiring challenge/response for trading, but being part of a distributed network, much harder to DDOS.
Conclusion
These are just my notes on what an Exchange might look like. It would be great to see a modern, robust, open source exchange that looked something like this.
If you want to chat email me: [email protected]