I worked on these systems and I can categorically say this is the best summary of HFT I've ever come across.
Many of the systems, fiber networks, data centers, and software that I touched are mentioned here. I've never heard the term "bit fucking" before, but the explanation of having to understand what happens under the metal is very true. You have to have a good understanding of software, compilers, assembler, CPU micro architecture, network protocols, and operating systems to be able to operate in this space.
I don't really know what else to add, except that I and many others I worked with in this business think that batch auctions are the solution. Although the work is technically challenging and interesting, I certainly don't like the arms race aspect of it, and I wish the resources (natural and human) used by this sector could be better allocated. One of my big concerns was the added CO2 emissions of constantly spinning programs doing no useful work - all in the name of minimizing latency. I did some research into reducing the carbon footprint of our code by reducing idle spin cycles. Alas, I never really had the time
to get it working well before I decided to pack up and move on.
Why would the batch auction need to be FIFO? What about something like this:
1) When opening a "new tick", release data about trades in last tick, and start accepting new orders. Information about orders submitted to this tick is kept secret. Allow setting order price to practically arbitrary accuracy.
2) After tick closes, match orders in _random order_. That is, it should not matter at all on at which point in time inside this tick order arrived
3) repeat
Allowing arbitrary accuracy would help turn the competition from speed or stuffing the order book with huge number of orders to "spoof" the random matching, towards competition of just providing lower spread.
I understand that it would still advantageous to hold submitting trades until very end of the tick, to take into account as much outside information (other markets, etc.) as possible. To counter this, maybe the exact length of the tick could be randomized also.
What do you think?
So the biggest argument against random matching (whether in batch auction or open market book orders) is that the easiest way to game it is to figure out your expected fill rate. Then you just raise your order volume such that your expected fill rate matches your risk profile.
For the advantage of removing the speed advantage, you get the downside of providing advantage to people that have the loosest risk profiles and you further complicate the audit burden such that it is impossible to follow.
Think of it from the perspective of a value investor that set a market order 6 months ago. Under the current system once their price becomes top of the book they have a virtual certainty of getting filled. Under a (fair) random match they have the exact same chance of getting filled as someone who literaly just put in their order.
Batch auctions already exist in electronic markets. They still allow for huge advantages for speed.
If you want to get rid of the speed element (I'm not sure it's a problem) you should explicitly allow people to pay for priority. The easiest way to do this would be to allow for extremely small fractions of price. So instead of having to pay for faster lasers I could just change my pricing structure to tack on .00001 on to the price.
This proposal, while economically sound and something I have advocated for for years, fails to allow politicians and pundits to express their strong moral disapproval of geeks making money.
Also been involved in this line of work. How would you see the specifics of a batch auction working? I'd see it as not doing anything if the matches happened in a FIFO manner (it'd still pay to be first). Would a pro-rata batch auction fix the problem? Or would we have to introduce a random component to match all crossing orders in each batch?
Batch auctions also have the opposite problem, whereby it makes sense to submit your bids as late as possible, to let you utilise the very latest information prior to the auction close. This might actually balance out some of the FIFO advantages?
In a lot of FIFO markets, you will see price levels that have a large number of orders on them. Being in the front of this line is very valuable to a market maker. Many will have those orders there weeks in advance specifically for the purpose of maintaining a good place in line. I wouldn't see this changing in a batch FIFO auction.
But I've never seen such a match before, so who knows.
Ah, I was thinking that the concept of these batch auctions was to start with a completely clean order book every round. But as you say, that wouldn't be any good for people wanting long-lasting orders.
From what I understand batch auctions would match every participant's first order with everyone else's first order. Then all second orders would be matched together, all third, and so on. I think this is how one of the FX exchanges is doing it now (or will be soon).
Caveat: I'm not a trader but was more on the engineering side.
But what do you do when there are more participants on one side of the same price than the other? If it is FIFO you haven't eliminated any of the incentives for speed.
I don't understand why HFT is still around, if it's considered a problem.
Naively, couldn't all the craziness about HFT be resolved by enforcing a 1-second tick on the markets?
Like, right now it appears that trades occur instantly. Instead, queue requests and have the trades occur only once a second. Or 5 seconds.
Would this work? I'm assuming that either there's no motivation to solve the problem, or there are difficulties that I'm unaware of.
---
I've been wondering this about HFT for a while and this seemed like a good place to ask. Then I went back and finished reading the article, and this solution is proposed at the end. Curious about whether it would work on not.
HFT gives me a minor amount of hope for our regulatory environment: the forces and popular opinion of Big Finance all oppose it, but regulators continue to allow it.
The short answer is that HFT is like Wal-Mart: it's trendy and cool to complain about it, but in terms of providing better and faster inventory to ordinary customers, it works better than everything that came before.
Except Wal-Mart is hated for being (allegedly) bad for poor workers, electronic trading's victims have primarily been the Wall Street old boys' club. Why the public is convinced this is a bad thing is beyond me.
This is what really sucks about that Michael Lewis book. He could have told a great story to the public about how everything they had been told was wrong and that the HFT nerds (just like the baseball nerds) had used technology to make the world a better place. He has the talent to do this so well.
But he fucked up and bought the story the buy side scammers were selling him hook, line, and sinker.
>>Naively, couldn't all the craziness about HFT be resolved by enforcing a 1-second tick on the markets?
There are several arguments against it:
1)liquidity is good(1), more liquidity is better; why artificially limit liquidity, just let people provide as much as needed at as low price as market dictates;
2)Options are good(2), limiting liquidity increases prize of certain options (as issuer needs liquidity to offer options and the less liquidity there is the more expensive issuing those becomes, the effect might be small between 1s and 0s but it's still there)
3)Certain types of arbitrage would be less efficient. Arbitrage is good(3).
(1)-the concept of liquidity is that when you have some goods you want to sell, you don't need to invest time/effort/money to sell it; that in turns frees your time to do productive stuff; the effect is obvious in situations like retail stores buying from manufacturer (so manufacturer don't need to do advertising, building stores etc. themselves so they have more resources to actually produce goods); the effect doesn't disappear magically at 1 second scale, more liquidity (faster, cheaper) is always better for overall productivity.
(2)-options allow certain ventures which normally would be too high variance (forgive the gambling term) to pursue; they are form of insurance; reducing risk is almost always good but it costs to provide; the lower the prize, the better;
(3)-arbitrage is good because the prices of goods/stocks become close to true value; we want that, we also want to pay for that as little as possible; the more effective the arbitrage the better, why stop at 1 second, when you can have cheaper/better ?
This is not the whole story but those points are clear benefits of HFT. Liquidity is cheaper today than it ever was. Spreads are the lowest as well. Some people who used to make bank charging ridiculous spreads are now not making bank - this again is great.
The arms race argument is that resources go into something counter-productive (like stockpiling nukes) instead into something productive. In case of HFT resources (at least some of the time) go into something desirable. It is a race but it is a race to improve desirable things (liquidity, arbitrage). You rarely hear argument about "arm race" in situations like that. It's usually called progress. Usually we applaud progress in situations where service provides needs to compete to offer the best/cheapest product. In case of HFT this point is often somehow overlooked.
I think none of these arguments are very good against the idea of one-auction-a-second.
1) Liquidity is good, but remember who it is for. Uninformed traders, who want possession of the asset itself, will not care about a one-second delay because one second is too small an amount of time in human terms. Being able to trade a fraction of a second earlier is of no benefit. Being able to trade quickly on human timescales is the great benefit of algorithmic trading.
Informed traders, such as market makers, even market-making algorithms, also shouldn't care about a one-second delay. Their contribution to the market is the price. If you are a very good market maker, and you can quote prices better than anyone else in the market, your prices will be the better ones anyway, regardless of how the market is structured. And the profits are the market's rewards for quoting good prices, not trading quickly.
I would say the general idea is that liquidity providers and liquidity consumers compete, above all, on the price, and a one-second delay does nothing to hurt their incentives.
2) Again, I think it's quite obvious that on human timescales, on which actual economic decisions are made, there is no benefit to information arriving just a fraction earlier than it would otherwise. If you think of a market's purpose as revealing the price of an asset, it really shouldn't matter if the price is revealed 1ms earlier. What matters is that it is revealed at all, and revealed correctly with as many market participants as possible.
3) Because arbitrage becomes better when it contributes information to the market, not when it contributes information to the market 1ms earlier than another arbitrageur.
I think your points are completely valid arguments for algorithmic trading. Algorithms providing market participants with better prices than humans are indeed better for everyone. But for high-frequency trading, with the emphasis on high-frequency, I don't think this is very convincing.
>>. Being able to trade a fraction of a second earlier is of no benefit. Being able to trade quickly on human timescales is the great benefit of algorithmic trading.
Informed traders, such as market makers, even market-making algorithms, also shouldn't care about a one-second delay.
I think the other poster countered it well:
>>The impact is that by being fast market makers can reduce some of their risk, which means that they can price more aggressively. If they can't reduce that risk via speed, they will reduce it in other ways, namely pricing less aggressively. This will increase price spreads and decrease liquidity.
Human investors may not care about getting stuff 1ms faster but they should care about getting the stuff cheaper (or selling it for better price).
>>Again, I think it's quite obvious that on human timescales, on which actual economic decisions are made, there is no benefit to information arriving just a fraction earlier than it would otherwise.
Well, yeah, but again more stuff happens in one second than in one milisecond. It means there is more risk of more bad things happening. If there is possibility of more bad things happening the liquidity providers need to price liquidity higher to mitigate that new risk.
Well, it might not be much. Maybe even at some point it really is a race to the bottom kind of situation but why make judgement about it ?
If ISPs compete offering bigger and bigger bandwidth and faster and faster ping then it becomes pointless at some point again as only thing new competitor achieves is taking place of the incumbent without contributing significantly to what customers get. 100GB/sec or 1TB/sec, who cares. 1 second tick is similar to 100GB cap - supposedly there is no point at making the technology more efficient any more.
Your argument comes down to: "The cost of providing new technology is higher than gains customer get and that is achieved by putting previous provider of the services out of business - hence waste".
It might be true or it might not. We just usually don't care much because that is one of the situations where free market/competition does well in practice.
Those technological arm races have other benefits as well, you never know what other uses the technology is going to have or what kinds of scientific advancement it stimulates.
One day you may have 35% faster ping from Tokyo to New York because instead of optic fibers the connection will use some of technology mentioned in the article.
Ah cool. It's much more important to focus on these reduced costs than to focus on the timescales involved in HFT. You are right to say that humans don't care about millisecond timescales but they do very much care about price.
Only the computers care about those timescales but it's by fighting it out there that they can give you the best price.
"How does 1 second ticks stop liquidity and arbitrage? It just stops the faster connections wins."
No it doesn't. Let's suppose in your 1 second time tick you have more people attempting to buy at a price than sell at the same price? Who gets to buy and who doesn't? If it is FIFO then the speed advantage is still there. If it isn't FIFO it can be applied to non 1 second ticks as well.
"HFT aren't really adding liquidity since they only buy when they can sell straight away."
This is a completely untrue statement. Lots and lots of HFT hold inventory. Their inventory periods can range from seconds to minutes to weeks.
The impact is that by being fast market makers can reduce some of their risk, which means that they can price more aggressively. If they can't reduce that risk via speed, they will reduce it in other ways, namely pricing less aggressively. This will increase price spreads and decrease liquidity.
>I don't understand why HFT is still around, if it's considered a problem.
Primarily because regulatory capture has, until now, overruled all attempts to reign it in. The banks and hedge funds engaged in this activity wield vast amounts of political power.
>Naively, couldn't all the craziness about HFT be resolved by enforcing a 1-second tick on the markets?
Not really. That just changes the game from "who can trade a picosecond before you?" to "who can trade a picosecond before the tick?".
An almost microscopic tax on transactions would prevent it, though while having a minimal impact on all other trading activity.
This is silly. HFT is a $10-50B/year industry, depending on who's numbers you go with. (You can ballpark the numbers by multiplying trade volume by the spread.)
For comparison, Morgan Stanley alone made $8B in profit in the last quarter [1]. HFT is tiny, a drop in the bucket compared to the rest of the financial industry. Further, many big players in the industry are against HFT since HFT reduces their information advantage over smaller players [2].
The danger in a 1-second tick is that a) a massive change like this is hard to do [3] and b) it would likely increase volatility. A stylized fact of dynamical systems is that continuous feedback reduces volatility, whereas discrete step sizes increase it - witness how many numerical ODE solvers can blow up even while the underlying ODE is stable.
[3] Bloomberg, Excel and other calculators contain mathematical errors (e.g., "weekends are sometimes treated as a business day for the purpose of interest accrual") which cannot be fixed because the change would be too disruptive.
>This is silly. HFT is a $10-50B/year industry, depending on who's numbers you go with. For comparison, Morgan Stanley alone made $8B in profit in the last quarter [1].
I'm not sure what your point is here? There's more to finance than HFT? There are other, bigger rip offs out there?
His point is that small industries (especially ones with very low margins like HFT) cannot wield the political muscle required to invoke "regulatory capture". For instance BlackRock, a large hedge fund that has been very vocal in it's anti HFT stance made more profits in 1 quarter last year than the entire HFT industry. If regulatory capture was going to happen, don't you think they would be in a better position to accomplish this?
Bill Ackman is another giant player who is highly vocal against HFT and is known to spend donation money to attempt to make trades go his way.
HFT is the little guy in this story, the little guy doesn't get to create regulatory environments.
The few larger players who came out against it have done so only relatively recently, and in Goldman's case, only subsequent to being under investigation by the AG for its own HFT practices.
Found in a 2 minute google search. Your position is false, that no one was against HFT until recently. Large participants have been actively against it for a very long time, further pointing out how incorrect your position is on regulatory capture.
You have an ax to grind and an incomplete understanding of how the market and HFT work. It makes you look foolish.
>Found in a 2 minute google search. Your position is false, that no one was against HFT until recently.
My position was that no large banks or hedge funds or other concentrations of political power were against it until relatively recently. Which is completely true.
An individual senator is not capable of making vast changes to the law by themselves. Your belief that this IS possible is completely absurd.
>You have an ax to grind and an incomplete understanding of how the market and HFT work. It makes you look foolish.
Frankly I don't know how you're still defending barclays for selling an service promising freedom from HFT predators and then introducing HFT traders to it so they can collect additional fees.
Your belief that these traders were dumb and did not know what they were buying and therefore deserved to be lied to by Barclays is beyond retarded. They knew exactly what they were doing. You do not.
"My position was that no large banks or hedge funds or other concentrations of political power were against it until relatively recently. Which is completely true."
I mentioned several giant hedge funds (orders of magnitude bigger than the entire HFT industry) that have been arguing against HFT for a very long time. Further, the first article I found on a 2 minute google search, mentions a large fund that was speaking out against HFT in 2009. This is as simple of proof as I can provide that your position that HFT is not opposed by large finance groups as untrue.
"Frankly I don't know how you're still defending barclays for selling an service promising freedom from HFT "
Not once in this whole thread have I defended Barclay's and trying to paint me with that brush is either a sign of extreme obtuseness or terrible rhetoric. What I argue is that the reason Barclay's needed to defraud their customers is that their claim (and all other dark pool's) claim that they could provide a better trading environment by excluding HFTs has been historically proven as false. Traders are not dumb, they move to the venues that provide them with the best trading environment and that has continued to be venues that allow market makers EVEN IN THE FACE OF FRAUD.
No matter how many times you paint HFT as "predators" or "sharks" or as "spearing whales" you've yet to describe a single behavior attributed to HFTs that is in any way unethical or illegal.
[edit: having to work around HN reply rules, I apologize]
To the reply below about BlackRock. Do you really think it is a realistic comparison to compare BlackRocks recent profit (2013) to one of HFTs best year (2009)? At the very least you should have mentioned that it was different years.
"Were this true, dark pools would not exist in the first place"
Except that there are all manner of product offerings that don't make sense for their clients. The easiest analogy in this case is extended warranties. "If extended warranties were so bad why do so many people buy them?" Of course this is a ridiculous argument. People buy extended warranties to hedge risk that is over rated due to warranty salespersons. The exact same way dark pools are.
" There is a link I posted below in response to you that describes such a strategy"
It does no such thing. It hand waves a lot of semi-technical non specific jargon around an idea that is easy for people to get upset about. I could just as easily write an article about how multicast networks were dangerous. It wouldn't make it so.
"TABB Group estimates that annual aggregate profits of low-latency arbitrage strategies exceed $21 billion, spread out among the few hundred firms that deploy them."
>What I argue is that the reason Barclay's needed to defraud their customers is that their claim (and all other dark pool's) claim that they could provide a better trading environment by excluding HFTs has been historically proven as false.
Were this true, dark pools would not exist in the first place. Traders would have stuck to the public exchanges and paid their (lower) fees.
>No matter how many times you paint HFT as "predators" or "sharks" or as "spearing whales" you've yet to describe a single behavior attributed to HFTs that is in any way unethical or illegal.
False. There is a link I posted below in response to you that describes such a strategy (spearing the whale) in precise detail.
"You mentioned Blackrock, whose net income last year was $3 billion.
http://www.wallstreetandtech.com/trading-technology/the-real...?
"TABB Group estimates that annual aggregate profits of low-latency arbitrage strategies exceed $21 billion, spread out among the few hundred firms that deploy them.""
Thanks for using 1 number from now and another number from 5 years ago. Are you willing to concede the rest of your rhetorical high ground?
Even if we discount the market impact numbers that you have to juke, I found a senator in 2009 speaking out against HFT. Can you find one defending it?
"Were this true, dark pools would not exist in the first place. Traders would have stuck to the public exchanges and paid their (lower) fees."
Maybe if the buy side firms didn't engage in fraud to keep their clients out of public exchanges that would be true.
That's the "profit" number which is useful and much lower than people expect, but not what people normally use when describing the size of an industry.
> Primarily because regulatory capture has, until now, overruled all attempts to reign it in. The banks and hedge funds engaged in this activity wield vast amounts of political power.
Your invocation of the "regulatory capture" trope doesn't even make sense in this context. The big banks and hedge funds aren't the major HFT players, and many of them are opposed to HFT.
Remind me which of those banks are calling for it to be strictly regulated or prohibited?
I've heard that some banks are now selling "protection" against HFT sharks in the form of dark pools (which is often not really protection at all - see Barclays). Bearing that in mind I can't imagine that it would hurt for them to put out a marketing message that is disdainful of the practice while still quietly engaging in it.
You might like to read Matt Levine's article on Barclays. Choice quote:
Barclays, like other dark pool operators, knew this: that you can run a pristine dark pool without "predatory" high frequency traders, and without much trading, or you can run a useful dark pool with high-frequency traders.
tl;dr; No HFT, no liquidity, and people who want to trade take their business elsewhere.
[edit: If Barclays didn't need to bring HFTs in to provide liquidity, why did they subsidize HFTs? Matt Levine crunches the numbers - the biggest HFTs paid Barclays about $3M assuming a wild overestimate of the amount of liquidity they take, compared to institutional investors paying $4B.]
Exchanges subsidise market-making, because that is the unquestionably beneficial activity.
But the high-frequency aspect of it is separate from the algorithmic-trading aspect of it, so one can criticise high-frequency trading for its uselessness, while accepting that algorithmic trading is hugely beneficial.
You are acting like algorithmic/hft/market making terms are clearly defined.
It turns out that the vast majority of HFT are algorithmic market makers. Why is that? Because if they can trade more they are more profitable (if they are good at market making). This means that they can make less on each trade driving down the bid/ask spread which is good for everyone. Further, if they can reduce their risk by trading fast, then they don't have to price that risk into the spread once again making the price better for everyone.
You can't separate hft from algorithmic market making because it is a central driving premise of the industry.
Algorithmic traders compete on price: they aim to offer better prices, and narrower spreads, to liquidity consumers.
High-frequency traders compete on the speed, with which orders are placed. The parasitic traders are the ones who profit not by providing you with a service, but by taking advantage of your inability to place and remove orders as quickly as they can.
You can do both of these at the same time, but the two competitive advantages are different and separate.
Market making is what all this activity is supposed to do, because it is a service provided by the liquidity providers, through the exchanges, to the liquidity consumers.
Market makers compete with each other on speed. If market maker A and B both want to provide you liquidity at $10.00/10.05, whichever one is fastest will be the one to get your business.
If market maker A wants to compete on price and provide liquidity at $10.003/10.049, too bad. That's illegal. He must compete on speed instead.
I wrote some tutorials on mechanics of this a while back, you might find them useful:
By being very fast at taking into account new information as fast as possible market makers reduce their risk, thus allowing them to offer better prices.
The vast majority of HFT trades are market making - with razor thin margins. The vast majority of profit making trades do not involve market making, however - they usually involve trading on an information advantages, or spearing a whale (large institutional investor trying to buy or sell large amounts of stock).
It's good to mix in a large dose of almost-zero-profit market making trades in with your more ethically and legally dubious trades. It helps cover your tracks - both from regulators and other market participants.
Do you have a citation for this? What does "spearing a whale" even mean? What is ethically or legally dubious about electronic market making?
Please just come out and say what you think HFT is, how it works and why it is ethically or legally dubious, because your claims in this thread border on the hysterical.
There was a rather famous incident with a trader nicknamed "London Whale", who accumulated a massive position in the CDS market, which then led JPMorgan Chase to lose immense amounts of money on the trade. The money went to those people that correctly figured out what was going on in the market.
In general, if you know important information about someone else's positions, you can sometimes use that information to profitably trade against them.
The London Whale did not involve HFT. In fact, it is a rather good example of how little HFT matters in the world of finance. One division at Chase (and if you believe them 1 man) cost the company more in 1 year than HFT makes in profits. The other sides of that trade were not HFT firms, but other human driven trading funds. So of course information advantages allow you to make money, that's the point of the market and it is no more unethical or illegal when an HFT makes money than it is when those human traders were on the other side of the London Whale.
This blog article has huge technical problems and is not a financial community technical journal. It is in fact a political blog with it's own agenda to push. Even without these giant warning signs what they describe in this article is not illegal or unethical.
After reading this article your assertion is that it is unethical and/or illegal for market makers to use standard market orders that are filled in legal and standard ways as price information signals, but it is not illegal and/or unethical for large market participants to hide their order flow with specialized orders and/or venues?
>It is in fact a political blog with it's own agenda to push.
It is describing a commonly used HFT trading strategy in objective terms. Your ad hominem attack on their credentials and agenda does not change this.
>After reading this article your assertion is that it is unethical and/or illegal for market makers to use standard market orders that are filled in legal and standard ways as price information signals, but it is not illegal and/or unethical for large market participants to hide their order flow with specialized orders and/or venues?
What was just described is detecting a large institutional order through the use of 'pinging' trades and then effectively taking advantage of that knowledge by trading ahead of it. That is NOT market making.
It is not strictly speaking insider trading or front running either, but it combines features of both practices.
"Your ad hominem attack on their credentials and agenda does not change this."
An ad hominem attack assumes that my attack is based on an irrelevant fact about the arguer. That the arguer has no experience in, and cannot articulate a common description of the issue in question is not an ad hominem attack. Just like saying that I have no basis in describing rabbinical law and should therefore not be determining what is and is not kosher, me saying that your source doesn't know what they are talking about when it comes to electronic trading is what is important.
"What was just described is detecting a large institutional order through the use of 'pinging' trades and then effectively taking advantage of that knowledge by trading ahead of it."
What is described in that article is 1 market participant, using a well known and perfectly legal market means to send in orders that they want to trade and then extrapolating from the completion of those orders, information about another participant that is using highly exclusionary, extra-market venues (dark pools) to hide their desire to change the price of an instrument.
That you think the market participant that is most disadvantaged in this relationship should be penalized more says more about your biases than anything else in the market dynamic.
"You do not need to sell privileged access to HFTs to your dark pool of institutional investors in order to have liquidity."
Except that is untrue historically. The reason dark pools sell that privileged access is that they can't provide liquidity any other way and their clients find it better to trade in HFT allowed environments than in ones without liquidity.
IEX is trying something new, driving liquidity via scare tactics and publicity but we still don't know how that will work out.
"Barclays were ripping off their clients. It wasn't even a complicated scam."
Barclay's lied about allowing HFT firms into their dark pool. That is fraud but it has nothing to do with bad behavior on the part of the HFT. If you are suggesting that dark pools have inherent conflicts of interest that might be a more valid argument than "HFT IS THE EVILS"
Historically there was no HFT. Historically we have managed to have liquidity without it. Historically, the thing that brought us the greatest liquidity (something that is useful but vastly overrated) in the equities markets is electronic trading.
>The reason dark pools sell that privileged access is that they can't provide liquidity
I think it's cute the way that you're pretending that they were doing their investors a favor. They were selling dark pools to investors because investors WANTED to get away from HFT. If they wanted HFT infested waters they could have traded on the public markets!
What the fuck do you think they were DOING there in the first place? They happily sacrificed a sliver of extra (and largely unnecessary) liquidity for the chance to not get ripped off.
They thought they wanted to get away from HFT. But once they were away from HFT they realized that the liquidity wasn't there and started trading on other venues. Barclay's dark pool was failing as it had no participation and like almost all the other dark pools in existence to get around this they started letting HFTs in.
The difference is that in Barclay's case they lied about it and thus committed fraud.
>They thought they wanted to get away from HFT. But once they were away from HFT they realized
THEY WERE PAYING TO BE IN A DARK POOL IN THE FIRST PLACE BECAUSE THEY WANTED TO BE AWAY FROM HFT. There was no "they realized".
Once they got away, Barclays tried to figure out if there was any BARCLAYS could extract more profits from these dark pools than the fees they were already paid. There was: allow HFT sharks to take advantage of their customers and collect fees for letting them do so.
If what you said were true (and frankly I cannot possibly see how you could believe it) they would have ZERO reason to commit fraud and lie about selling privileged access to HFT traders.
"Once they got away they did NOT start whinging that there was not enough liquidity."
Of course they didn't. Professional market participants don't whinge. They go to venues that provide the services they want, namely cheap liquidity. Those venues were ones that allowed HFT market makers.
When you say "they got away", you do know that those market participants moved to open exchanges where HFTs "run rampant" and stopped paying Barclay's for their service.
"allow HFT sharks to take advantage of their customers and collect payment for letting them do so."
A) what do you mean when you say HFT shark? You keep using that term, in what I assume is a pejorative but it doesn't mean anything.
B) If the point was extracting more fees why did Barclay's subsidize HFT trading? They charged HFTs an order of magnitude less than their clients, that was a central part of the complain. If it was an example of allowing advantage taking, wouldn't the fee structure be reversed? That is, if I have a chance to fleece someone, wouldn't I pay up for that instead of require payment for it?
"If what you said were true they would have no reason to lie"
Of course they had a reason to lie. They claimed that they could provide a high liquidity environment without letting market makers trade in their dark pool, despite the overwhelming historical evidence to the contrary. Admitting that they were wrong brings their whole value proposition to buy side investors into question and delegitimizes their dark pool.
>Of course they didn't. Professional market participants don't whinge. They go to venues that provide the services they want, namely cheap liquidity.
If they prioritized liquidity above all else, as you appear to think they should, they wouldn't be using a dark pool in the first place. They would have been using a public exchange.
>When you say "they got away", you do know that those market participants moved to open exchanges where HFTs "run rampant" and stopped paying Barclay's for their service.
Yes, I'm sure they all left after it was revealed that Barclays was lying to them and allowing HFT predation on their trading activity.
>what do you mean when you say HFT shark?
An HFT trader that parasitically extracts value from a non-HFT traders - often institutional investors (e.g. pension funds) putting through large orders. The link I put below describing a 'whale' to you explains just one parasitic strategy.
>If the point was extracting more fees why did Barclay's subsidize HFT trading? They charged HFTs an order of magnitude less than their clients
A) Subsidizing != Charging lower fees
B) Citation, please (that OVERALL fees were higher, not just fees per trade - HFT will obviously get bulk discounts).
>that was a central part of the complain.
The central part of the complaint was that the service was advertised as HFT free when, in fact, it was not. The marketing literature is all out there saying this.
"If they prioritized liquidity above all else, as you appear to think they should"
No, I think they should prioritize execution costs above all else. It turns out that fees then liquidity costs are the biggest variables in determining execution cost. Public exchanges with low fees and high liquidity provide the lowest execution cost, no matter what the dark pool salesmen claim (fraudulently or not).
"Yes, I'm sure they all left after it was revealed that Barclays was lying to them"
Actually the complaint shows they were leaving before the lying started. This is what you are missing. The lying was a response to the natural market condition that execution costs are lowered in environments with HFT participants. That Barclay's had staked their reputation on that not being true is what led to the fraud.
"The central part of the complaint was that the service was advertised as HFT free when, in fact, it was not."
And that is fraud. The book should be thrown at Barcaly's but that doesn't add any insight into the arguments surrounding HFT, other than in real world conditions, even buy side clients prefer venues with HFT provided liquidity, even in the face of fraudulent marketing claims.
The market stops rewarding those who do the best research and make the best value judgments about stocks and simply rewards those with the fastest exchange connections and best hardware (who can parasite upon the research and risklessly make money by covertly front running it).
If value investors stop being rewarded for doing good research and correctly assessing the true value of a stock they will usually withdraw from the market (c.f. market for lemons: akerlof).
It should be pretty obvious why a stock market whose values are disconnected from reality is a bad thing, and why having a pool of investors who do good real world research and get rewarded for it is a good thing.
When you say HFT sharks, what do you mean? What practice are you describing. The reason that this is important is that the range and scope of electronic/algorithmic trading is highly varied. Saying you are against HFT can mean that you are against a wide variety of things.
I don't see your point. It's entirely consistent to say that restrictions on HFT would be in Goldman's interests, but until those are in place, they make some money engaging in HFT. In other words, the advantages to Goldman of restricting HFT (which hurts its major business line of trading on information advantages) would be worth more than losing whatever little HFT revenue they make (they're not a major player).
Not really. That just changes the game from "who can trade a picosecond before you?" to "who can trade a picosecond before the tick?"
The lack of live market book data means that you won't need the ultra fast connections, so at least one part of the networking arms race becomes de-weaponised.
You still have the problem of it being advantageous to bid at the end of the auction, but this could be solved by pricing. Just like the exchanges give incentives to offer prices rather than to take them, they could make bidding at the start of an auction cheaper than at the end of it.
Been trying to figure this out myself. What if we have 3 offers and 2 bids crossing at the same price after the 1 second tick is up. Which two offers get filled? If the answer is, the first two, then you've solved nothing. Maybe you randomize it? That could bring up some interesting issues.
Ya, if you randomize it than I have in incentive to submit more bids than I actually want filled because I've estimated what my fill rate is going to be. Of course everyone else has that incentive too which is going to negatively effect my fill rate which seems like it would lead to a pretty unstable system.
I believe the exchanges have no interest in "solving" HFT since they make a good amount of dinero around it. IEX has made a business out of 1 second delay (700ms actually).
HFTs nearly all provide liquidity rather than take it, which means that the exchange pays them to trade. You are simply wrong - the exchange loses money on most HFTs.
HFTs provide far more liquidity than is ever needed during normal periods and during periods of market distress (when liquidity is sorely needed) they suddenly and sharply withdraw that liquidity from the market.
Just to be clear, in market circles removing liquidity means a specific thing. It means, crossing a spread to buy or sell existing orders. HFTs do not do this in times of market distress.
You can make an argument that HFT market makers stop providing liquidity during strange market conditions, but they do not "remove" it.
IEX has made a business out of being a dark pool explicitly designed for buy side hedge funds to be able to avoid market forces. The 700ms delay is the least of those features & is more of a marketing gimic.
IMO this is one of the best articles I have read on HFT. It gives a good, balanced overview of the technology and the fundamental principles behind how automated trading works, without any particular viewpoint or opinion directing the text.
The section near the end on FPGAs and programming styles was an eye-opener to me. I wonder how many other firms have taken risk checks out of their code to save some nanoseconds...
> without any particular viewpoint or opinion directing the text
You say that like that's a good thing.
I know that a lot of rubbish has been written about HFT and algorithmic trading, but it's sad that no one seems to be able to write about its merits without starting a flame war.
It's an outgrowth of poorly designed matchmaking software at exchanges. The could completly distroy HFT without costing society anything or adding transaction fees.
Do you have any specific issues in mind when you say that?
A lot of the rules that markets operate by come from standards and regulations, not software design. The software design reflects the kind of markets exchanges want to set up, so it's not a software problem. The people who wrote the standards had specific markets and specific outcomes in mind when they wrote them.
I work in this industry. I've come to the conclusion that I will be leaving very soon. There are a number of reasons, but the primary one is because it's become clear to me that I spend the vast majority of my time on trying to figure out how to take advantage of flaws in exchange hardware/software. Sorry, I can't go into specifics.
That being said, I don't know if it is possible to create an ungame-able matching engine.
I can't help but think of that article yesterday about how we're avoiding the Big Problems (https://news.ycombinator.com/item?id=8261098). It's crazy how much money and brain power people have invested in HFT which is essentially just turning time into money with no benefit to society. What could the human race have achieved if the same amount of effort was focused in other areas?
There are a lot of misconceptions in your comment that we need to unpack.
1) there isn't that much money in HFT. It has an ever decreasing number of players working for an ever decreasing amount of profit. Other areas of finance such as hedge funds, traditional investment banks, and derivatives creation are much bigger and have just as many "smart" people working in them. Don't get me started on advertising...
2) HFT doesn't turn time into money. It buys & sells risk in the form of liquidity. One way that it does this more efficiently is by being able to get in and out of positions fast.
HN, predictably, has a huge bias towards the capital raising/allocation side of the financial markets, but this is not why most participants are involved. Hedging risk is another huge (probably bigger) reason the markets exist. HFT allow that risk to be hedged more cheaply and more predictably so they provide value to the rest of the market.
It could be that our current economic system implements a pretty terrible resource allocation algorithm, where many workers' jobs consist in undoing the efforts of employees from concurrent companies.
In trading, a quant's job is to outsmart or outspeed the other bank's team. They invest considerable brain power, effort and money into producing incremental improvements that are quickly rendered meaningless by the work of the team in the building across the street.
Another prime example is advertising, a zero-sum game where competing brands fight to undo other brands' brainwashing efforts and replace it with their own brainwashing. A waste of the time and energy of million of humans.
Advertizing and HFT are only zero sum for competing products in an established market. For new products it can be a net benifit ex: "talk to your doctor about ED."
Don't be mad that some other people's brains are bent on stuff of no particular benefit to you. Their brains, their choice.
Just be glad that some people's brains are bent on stuff of benefit to you, and concentrate, if you like, on what of general benefit you can do with your own brain.
The economy is geared towards playing such zero sum games. This is pretty much by design - policies such as austerity, ZIRP and financial deregulation shift the allocation of capital to speculation (gambling) and away from productive investment.
Can someone help me understand the following statement (specifically the don't branch part) ?
"Don’t touch the kernel. Don’t touch main memory … Don’t branch.’ (That third commandment means don’t fill your program with ‘if’ statements – ones with the generic form ‘if A then do B else do C’ – because they get in the way of a modern microprocessor’s capacity to do several things in parallel.)"
It is probably referring to the fact that code that branches allot will make the prefetcher/branch predictor fail more frequently which causes slower memory loads and less ops. Not sure why it mentions parallelism though.
HFT is a two sided coin. For every person who complains that it's screwed them out, they're forgetting all the times they've saved money on narrower spreads due to extra liquidity. Possible?
Many of the systems, fiber networks, data centers, and software that I touched are mentioned here. I've never heard the term "bit fucking" before, but the explanation of having to understand what happens under the metal is very true. You have to have a good understanding of software, compilers, assembler, CPU micro architecture, network protocols, and operating systems to be able to operate in this space.
I don't really know what else to add, except that I and many others I worked with in this business think that batch auctions are the solution. Although the work is technically challenging and interesting, I certainly don't like the arms race aspect of it, and I wish the resources (natural and human) used by this sector could be better allocated. One of my big concerns was the added CO2 emissions of constantly spinning programs doing no useful work - all in the name of minimizing latency. I did some research into reducing the carbon footprint of our code by reducing idle spin cycles. Alas, I never really had the time to get it working well before I decided to pack up and move on.
Happy to answer any questions anyone may have.