Rodrick’s Web Log !!

- UNIX domain sockets use the file system as the address name space.  This
means you can use UNIX file permissions to control access to communicate
with them.  I.e., you can limit what other processes can connect to the
daemon — maybe one user can, but the web server can’t, or the like.
With IP sockets, the ability to connect to your daemon is exposed off
the current system, so additional steps may have to be taken for
security.  On the other hand, you get network transparency.  With UNIX
domain sockets, you can actually retrieve the credential of the process
that created the remote socket, and use that for access control also,
which can be quite convenient on multi-user systems.

- IP sockets over localhost are basically looped back network on-the-wire
IP.  There is intentionally “no special knowledge” of the fact that the
connection is to the same system, so no effort is made to bypass the
normal IP stack mechanisms for performance reasons.  For example,
transmission over TCP will always involve two context switches to get to
the remote socket, as you have to switch through the netisr, which
occurs following the “loopback” of the packet through the synthetic
loopback interface.  Likewise, you get all the overhead of ACKs, TCP
flow control, encapsulation/decapsulation, etc.  Routing will be
performed in order to decide if the packets go to the localhost.
Large sends will have to be broken down into MTU-size datagrams, which
also adds overhead for large writes.  It’s really TCP, it just goes over
a loopback interface by virtue of a special address, or discovering that
the address requested is served locally rather than over an ethernet
(etc).

- UNIX domain sockets have explicit knowledge that they’re executing on
the same system.  They avoid the extra context switch through the
netisr, and a sending thread will write the stream or datagrams directly
into the receiving socket buffer.  No checksums are calculated, no
headers are inserted, no routing is performed, etc.  Because they have
access to the remote socket buffer, they can also directly provide
feedback to the sender when it is filling, or more importantly,
emptying, rather than having the added overhead of explicit
acknowledgement and window changes.  The one piece of functionality that
UNIX domain sockets don’t provide that TCP does is out-of-band data.  In
practice, this is an issue for almost no one.

In general, the argument for implementing over TCP is that it gives you
location independence and immediate portability — you can move the client
or the daemon, update an address, and it will “just work”.  The sockets
layer provides a reasonable abstraction of communications services, so
it’s not hard to write an application so that the connection/binding
portion knows about TCP and UNIX domain sockets, and all the rest just
uses the socket it’s given.  So if you’re looking for performance locally,
I think UNIX domain sockets probably best meet your need.  Many people
will code to TCP anyway because performance is often less critical, and
the network portability benefit is substantial.

Right now, the UNIX domain socket code is covered by a subsystem lock; I
have a version that used more fine-grain locking, but have not yet
evaluated the performance impact of those changes.  I’ve you’re running in
an SMP environment with four processors, it could be that those changes
might positively impact performance, so if you’d like the patches, let me
know.  Right now they’re on my schedule to start testing, but not on the
path for inclusion in FreeBSD 5.4.  The primary benefit of greater
granularity would be if you had many pairs of threads/processes
communicating across processors using UNIX domain sockets, and as a result
there was substantial contention on the UNIX domain socket subsystem lock.
The patches don’t increase the cost of normal send/receive operations, but
due add extra mutex operations in the listen/accept/connect/bind paths.

Robert N M Watson

Five Ways to Avoid Data Latency

By Jim Middlemiss
March 01, 2005

Investment management firms can spend millions of dollars on fancy trading software and complex algorithms, but, if the system managing their market data is slow, then they’re wasting their time and money. Data latency, the lag experienced when data is moved from one place to another, robs firms of their ability to leverage those investments and react quickly to market changes.While it might be only a millisecond - one-thousandth of a second (think: a blink of an eye) - data latency can be the difference between getting a fill or being shut out of the market.

Danny Moore, COO of Incline Village, Nev.-based Wombat Financial Software, which makes software that allows for direct exchange connections, estimates that 60 percent to 70 percent of high-performance brokerage firms don’t even know what latency is. “If you don’t know what it is or where it’s coming from, you can’t address it,” he says.

But there are things firms can do to reduce data latency. Following are five ways to tackle the problem.

1. Build a direct feed to an exchange. By connecting directly to an exchange, as opposed to a data consolidator, investment firms can gain between 150 milliseconds to 500 milliseconds in transmission times. Direct-exchange feeds eliminate the data hops and thus reduce latency, Moore explains.

A direct-exchange feed, however, is more complex and more expensive. Firms need to build and maintain a system that cleans, formats and processes data so that it can be distributed throughout the organization.

2. Colocate servers near an exchange. This is an option for firms whose main office is located far away from the exchange. Five miles is one thing; 50 miles might be something else. Market data can’t travel faster than the speed of light, which is 186 miles a millisecond. So, if servers are 3,000 miles away, a firm may be sacrificing 16 milliseconds or more, notes Moore.

However, Guy Tagliavia, president and CEO of InfoDyne Corp., a Chicago-based firm that makes technology that connects exchanges to firms, disputes this assertion. He says that based on his firm’s testing with brokerages, it “doesn’t really matter where the server is located.”

3. Make IT physically part of the trading floor. By placing IT professionals and traders side by side, IT experts can monitor workflow among traders, applications and systems in real time.

For example, firms need to be prudent in the way they manage data, says Mike Alsip, head of product management at Reuters Market Data Systems in Chicago. “Limit the amount of data elements and value-added processing to what is absolutely needed to support pre-trade analysis and execution.”

4. Re-evaluate your middleware structure. Internally, ticker plant software handles the incoming data, which flows through a firm’s middleware to downstream applications. Depending on that technology, firms could be sacrificing as much as three seconds, according to Wombat Financial’s Moore. “There are still a lot of firms out there with internal systems that are doing much more damage to latency than the aggregate data feed,” he says.

“It’s not just about latency; it’s about latency at high through-put rates,” adds InfoDyne’s Tagliavia. For example, if you can send 1,000 messages in a millisecond but it takes one second to send 40,000 messages, that’s a problem. “You need to be able to support high through-put and still maintain flat, low-latency,” Tagliavia points out.

5. Get applications closer to the ticker plant. Locating critical applications closer to the ticker plant also reduces data latency. “It’s all about eliminating hops,” says Tagliavia. “Processing hops is expensive in terms of latency.”

Reuters’ Alsip adds that firms need to look at “flat client-site distribution architecture. You must be able to distribute from the direct feed directly to applications, minimizing the local area network hops in between.” Such architecture, he says, “needs to be able to scale to handle applications subscribing to the low-latency direct data feed.”

Q. Why do options ECN/Brokers need to send orders matched internally out to the exchanges if you have a buyer and seller who have prenegociated on prices? what is the reason behind this? I’m talking about listed options that are traded on CME, ISE etc..

A1. This is the age old question of the options space. Every time someone enters this business from the equities market they are amazed at the exchange execution clause. To put it bluntly, all orders need to be executed on the floor of an exchange. This is ostensibly to improve transparency, but it is really to preserve the dominance of the exchanges and prevent the OTC firms from encroaching even more on their turf. If you got rid of the exchange execution regulation, about half of the listed exchange volume would go OTC overnight. The sad fact that is about half of the volume in this business, and just about 100% of the institutional volume, is arranged off-the-floor. The floor is just the place for the firms to print their orders and for the exchanges to collect their pound of flesh. The exchanges don’t really provide any meaningful price discovery or liquidity anymore.

A2. This is an age-old example of protectionism in the options business, phusnikn. I think your question answers itself when you think about it. If brokers didn’t have to send their internalized orders to the exchanges, the exchanges would lose about half of their volume. That would put just about all of them out of business.

At the same time, it would also destroy transparency in the options market. If every firm could simply print its own orders, it would make it far more difficult to piece together the true flow of options volume. I’m not a fan of the exchanges, but I also don’t want to fragment the market any more than it already is. For better and for worse, we’re stuck with the system that we have.

Here is a little script I through together which some may find useful. I’n my environment I have implemented automount/autofs for user directories and local directories (none auto mounted home directories) for application accounts. The biggest reason for this is to have less dependency on external services such as LDAP, NIS, NFS etc.. These services are just much more prone to network failures.I use this script in conjunction with kickstart an application created by Redhat to automate the Linux installations over a network. The core concept is pretty simple the script will generate a rsa key for the service account user if one does not exist then copy the users public file to a centralized location on each of my hosts I have scripts that run daily that will pull down this authorized_keys file. This process has greatly increased productivity for my users as it now allows them to easily write scripts that can run across many servers with out being prompted for a password.

#!/usr/bin/perl -w
# 03/15/2008
# This script is used to keep authorizedKeys file in sync

use strict;
use warnings;
use IO::File;

my $homedir             = $ENV{”HOME”};
my $sshPrvKey           = $homedir . “/.ssh/id_rsa”;
my $sshPubKey           = $sshPrvKey . “.pub”;
my $authorizedFile      = ‘/net/10.0.0.1/export/homedirs/svcprd/authorized_keys’;
my $sshKeyGen           = ‘/usr/bin/ssh-keygen’;
my $found               = 1;

unless ( -e $sshPrvKey )
{
  system(”$sshKeyGen -t rsa -N ”” -f $sshPrvKey”);
  writePublicKeyToAuthKeyFile();
}
else
{
  writePublicKeyToAuthKeyFile();
}
sub writePublicKeyToAuthKeyFile {
  my $pubKey = getKeyString($sshPubKey);
  my $rsakeystring;

  sysopen ( OUT, $authorizedFile, O_RDWR|O_APPEND|O_CREAT, 0644);
  open ( IN, “$authorizedFile” ) or die (”Unable to read file $authorizedFile: $!”);
  while (  )
  {
    chomp;
    $rsakeystring = (split(/s/,$_))[1];
    if($rsakeystring)
    {
      if( $pubKey eq $rsakeystring )
      {
        $found = 0;
      }
    }
  }
  close ( IN );

  if( $found == 0 )
  {
    print “key found in $authorizedFilen”;
  }
  else {
    print ( OUT `cat $sshPubKey`);
    close ( OUT );
  }
}

sub getKeyString
{
  my $file = $_[0];
  my $rsaString;
  open ( FH, $file ) or die (”Unable to open file $file: $!”);
  while (  )
  {
    chomp;
    $rsaString = (split(/s/,$_))[1];
  }
  close ( FH );
  return ( $rsaString );
}

Types of Orders

C# has a pretty slick feature called delegates, delegates are variables that reference a function. Delegates work just like call backs in C++. Call backs in C++ uses pointers to functions to pass them as parameters to other functions. Delegates are similar in feature but it is more type safe, which stands as a stark contrast with C++ function pointers. A delegate can hold reference/s to one more more functions and invoke them as and when needed. It’s like a function pointer, but it can hold references to static and instance methods. For instance methods, the delegate stores a reference to the function’s entry point, as well as to the object. A delegate defines what the user’s event handler should return, and what its parameter list should be.To define a delegate in C#, use the following syntax:

The C# keyword delegate
The event handler’s return type
The delegate identifier
The event handler’s parameter list, enclosed in parentheses

The cloest feature I can think of in Java to C# delegates are anonymous classes. The code below shows just how powerful and simple delegates are there’s probably tons of inefficienty in the code it was only an example to teach myself how to use delegates

— Delegate2.cs —

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;namespace DelegateEvent2

{

    class Inventory

    {

        public int ProductID;

        public int Quantity;

        public string ProductName;

        private bool autoReOrder = false;

public Inventory(bool autoReOrderStock)

        {

            this.autoReOrder = autoReOrderStock;

        }

public Inventory()

        {

            // default constructor

        }

// create event

        public delegate void lowStockEventHandler(int ProductID,

 	Inventory obj);

        public event lowStockEventHandler onLowStock;

public void order(int number)

        {

            if(Quantity < number)

            {

                onLowStock(ProductID, this);

            }

        }

public bool orderSupplies(int count)

        {

            return true;

        }

public int GetQuantity

        {

            get

            {

                return this.Quantity;

            }

        }

public bool GetAutoReOrderStatus

        {

            get

            {

                return this.autoReOrder;

            }

        }

    }

}

— EOF —
— Program.cs —

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;namespace DelegateEvent2

{    class Program

    {

        static void Main(string[] args)

        {

            Inventory Item = new Inventory(false);

            // register two methods

            Item.onLowStock +=new Inventory.lowStockEventHandler(Item_onLowStock);

            Item.onLowStock +=new Inventory.lowStockEventHandler(Item_reOrderSupplies);

            Item.ProductID = 1;

            Item.ProductName = "Dell 21\" LCD Flat Panel";

            Item.Quantity = 5;

            Item.order(10);

            Console.ReadLine();

        }

static void Item_onLowStock(int ProductID, Inventory Item)

        {

            Console.WriteLine("The stock is too low for ProductID = {0}

 	Products now on Back Ordered {1}", ProductID, Item.GetQuantity);

        }

static void Item_reOrderSupplies(int ProductID, Inventory Item)

        {

            if (Item.GetAutoReOrderStatus.Equals(true))

            {

                Console.WriteLine("ProductID: {0} - {2} has autoReOrder set to true

 		ordering supplies now to fill backfill qty {1}",

                                      ProductID, Item.GetQuantity, Item.ProductName);

            }

            else

            {

                // nothing to do

            }

        }

    }

}

— EOF —

I don’t know much about Arastra other than they make some kick-ass 10GE switching gear and its rumored that Google may be using some of their hardware. Anyway I came across this list of challenging interview questions kudos to Arastra for providing this very useful link.
http://www.arastra.com/recruit/quiz.py

Description

JOP is the implementation of the Java Virtual Machine (JVM) as concrete machine in hardware. The design has been sucessfully implemented in low cost FPGA devices from Altera (ACEX 1K50, Cyclone) and Xilinx (Spartan II and Spartan-3).

JOP is open-source under the GNU General Public License, version 3.

Features

• Very small core:
  • about 2000 LCs - 3000 LCs (configurable)
  • fmax is 100 MHz in a Cyclone EP1C6
• Real-time features:
  • architecture designed to simplify WCET analysis
  • cycle accurate time interrupt (not tick based)
  • real-time enhanced thread model
• WISHBONE master

Status

• VHDL files and and supporting tools for the design are available from OpenCores CVS.
• Further information can be found at http://www.jopdesign.com
• A wiki is available at http://www.jopwiki.com/

Linux systems should have capability to transmit packets at the maximum rate of the connected network link. Many default systems are not optomized for high speed networks that are capable of processing millions of messages a second here are some general sysctl tunables I would recommend for anyone looking to get the most out of the Linux TCP stack.

The optimal socket buffer size is twice the size of the bandwidth * delay product (BDP) of the link. However, the default max TCP buffer size is too small on long fat networks. For example, to fill a 125MB/s (i.e. 1Gbps) pipe with 100ms one-way latency requires 125 * 0.2 = 25MB of data.

net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.core.rmem_default = 65536
net.core.wmem_default = 65536

netdev_max_backlog specifies the maximum length of the input queues for the processors. The default value is 300 (packets). Linux has to wait up to scheduling time to flush buffers (due to bottom half mechanism). Therefore, this value can limit the network bandwidth when receiving packets.

net.core.netdev_max_backlog = 3000

Increase Linux autotuning TCP buffer limits min, default, and max number of bytes to use
net.ipv4.tcp_rmem = 4096 87380 8388608

net.ipv4.tcp_wmem = 4096 65536 8388608

Increase number of pages, not bytes

net.ipv4.tcp_mem = 8388608 8388608 8388608

Allows tcp window size to not stay reduced for new connections

net.ipv4.route.flush = 1

The txqueuelen is another buffer in the kernel stack that can affect performance; 
especially of tcp transfers. When the system is sending out too much data from the ip
layer to the ethernet device driver layer, this buffer, txqueuelen, may overflow. In
tcp, this has the effect of a congestion event causing the cwnd to halve.

txqueuelen : 5000

 increase netdev_max_backlog for high throughput

netdev_max_backlog : 20000 # For 10 Gig-E, increase to 30000

Don’t cache ssthresh from previous connection

tcp_no_metrics_save = 1
tcp_moderate_rcvbuf = 1

TCP can be used for latency-sensitive applications, but several factors must be considered when choosing it over other protocols that may be better suited to latency-sensitive applications.

TCP provides low-latency delivery only if:

• All receivers can always keep up with the sender
• The network is never congested

This essentially boils down to “TCP is OK for latency-sensitive applications if nothing ever goes wrong.”

TCP Latency Behavior

Broadly speaking, the way that TCP does rate control makes it unsuitable for latency-sensitive applications. A TCP receiver will add latency whenever packet loss or network routing causes packets to arrive out of order. A TCP sender will add latency when going faster would cause network congestion or when it would be sending faster than the receiver can process the incoming data.

TCP Receiver-Side Latency

TCP only supports one delivery model: in-order delivery. This means that a TCP receiver must add latency whenever data arrives out of order so as to put the data back in order. TCP also often unnecessarily retransmits data that was already successfully received after out-of-order data is received.

There are two main causes of out-of-order data reception. The most frequent cause is packet loss, either at the physical or network-layer. Another, less frequent cause of out-of-order data reception is that packets can take different paths through the network; one path have more latency than another. In either case, TCP inserts latency to put the packets back in order.

When TCP cannot be avoided, it’s probably best to:

• Use clever buffering and non-blocking sockets with TCP to drop data for congested TCP connections when data loss is preferable over latency.
• Consider disabling Nagle’s algorithm with the TCP_NODELAY option to setsockopt(). Although this may produce decreased latency, it often does so at the expense of efficiency.
• Route latency-sensitive TCP traffic over a dedicated network where congestion is minimized or eliminated.

Today I attended a conference sponsored by Tibco about low latency messaging and their Tibco Rendezvous product. The conference covered many new features in Tibco/RV 8.0 such as the new IMP module “In Memory Processing” this module allows RVD to be linked directly into your application and bypassing the overhead of establishing a TCP socket to a local or remote RVD daemon you can actually have RVD running inside your application yeah!!. Another nifty feature in TibcoRV 8 was the client side retransmission request this addresses the “Crybaby Receiver Problem” that were apparent in older versions of RVD prior to 8.

Other topics discussed in brief were:

C/C++ vs Java for Real-time Systems
===============================
- The general consensus was that GC is unavoidable especially when working in the sub microsecond levels. Real-time Java is still in its infancy when ran on anything but Solaris. The demand for a real-time JVM will pick extensively for Linux as more demand for low latency application emerge current vendors in this space are Sun, IBM, BEA. Using a real-time JVM on Solaris requires application code changes. The current offerings from IBM, and BEA requires custom kernel patches for Linux systems.

TCP Offload Engines
===============================
- Because of the high interrupt request when running Gig-E and 10 Gig-E ToE cards allow you to take the server CPU out of I/O processing by shifting the TCP/IP processing tasks to this dedicated i/o adaptor.

Infiniband DMA
===============================
- The current king when it comes to low latency connectivity, some disadvantages with Infiniband are high cost associated with deploying infiniband switches and cards, also there are no standard API/specification between vendors this can often lock you into a particular vendor Cisco or Voltaire. Using DMA “Direct Memory Access” with infiniband requires application code changes.Tibco showed us some numbers of 20 ~ 30 microseconds when using infinband with “In memory processing RVD”.

10 Gig-Ethernet
===============================
- No significant performance gain over traditional Gig-E today in terms of (latency) . Do to the nature of x86 hardware pushing 10 Gig-E can be very difficult because of limited bandwidth on the FSB “Front Side Bus”.

Using a custom Linux kernel’s
===============================
- I’m still a bit skeptical about using a custom kernels because of the added work and overhead for the minimal gain. Since the release of the 2.4 kernel (~5 years now) Linux has migrated away from statically linking 3rd party drivers and subsystems into the core kernel. Most devices drivers and subsystems are “LKM’s” loadable kernel modules so I don’t believe you will see huge performance gains by explicitly removing various components from the stock kernel provide by vendors like Redhat and Suse.
Many commercial applications today assume you’re using the stock kernel from either Redhat or Suse so deviating away from the vanilla product can be a pain from a support standpoint. Redhat and Suse both provide a Real-time distro for very latency sensitive applications I believe this is a better route than rolling your own RT distro.

FPGA, and ASIC hardware processors
===============================
- AzulSystems is a hardware appliance vendor that sells a hardware processor that can execute native Java byte code in hardware their system can scale up to 48 cores. Up to 8 cores can be dedicated to real-time garbage collection.

- Netezza is another hardware appliance vendor for database query/analytics. All queries are done in hardware with FPGA cards through a term they coin as Asymmetric Massively Parallel Processing(tm)

Spread is an open source toolkit that provides a high performance messaging service that is resilient to faults across local and wide area networks. Spread functions as a unified message bus for distributed applications, and provides highly tuned application-level multicast, group communication, and point to point support. Spread services range from reliable messaging to fully ordered messages with delivery guarantees.

The API seems pretty straight forward for the most part I was able to startup the spread daemon and send a few test messages in a few minutes.

./configure –prefix=/opt/spread && make && make install

The default configuration is good enough for basic testing

[rbrown@macbookpro:/opt/spread/etc] $ egrep '^[^#]' spread.conf
Spread_Segment 127.0.0.255:4803 {
localhost 127.0.0.1
}
I hacked up a quick client to send test messages to the spread daemon on port 4803

1. Startup the spread daemon /opt/spread/sbin/spread &
2. Startup the spuser utility this is a handy utility to send/recieve messages from the daemon from clients.
a. Startup spuser utility : ./spuser
b. Join a group that will receive messages from clients : j group1

Using the following code you should be able to compile and send a message to the spread daemon and see this message register in the spuser utility.

javac -cp /path/to/spread.jar SpreadExample.java

--- SpreadExample.java ---
import spread.*;
import java.net.InetAddress;
import java.io.*;

/* Send the msg data */
// System.out.println("Sent the following msg to " + sp_host.getHostAddress() + " \"" + msg + "\" " + '\n');
connection.multicast(msgbox);
connection.disconnect();
}
}


I actually just stumbled upon this, but very cool way to go GOOG!!

http://finance.google.com/finance/historical?q=NYSE:LEH&histperiod=daily&start=25&num=25

Reuters has announced the launch of Reuters Data Feed Direct (RDFD) for the Options Price Reporting Authority’s (OPRA) FAST enabled data feed. Reuters Data Feed Direct OPRA is designed to process over one million updates per second using industry leading technology to leverage OPRA’s FAST implementation, allowing clients to reduce bandwidth cost.

Kanye might have his own views but mine would be — The job, the family, the fucking big television, the washing machine, the car, the compact disc and electrical tin opener, good health, low cholesterol, dental insurance, mortgage, starter home, leisure wear, luggage, three piece suite, DIY, game shows, junk food, children, walks in the park, 9 to 5, good at golf, washing the car, choice of sweaters, family Christmas, index pension, tax exemption, clearing gutters, getting by, looking ahead, til the day you die.