The first version of mongrel was a well known, very quick and well respected web server.

The author of mongrel1, Zed Shaw, has been working on version 2 for some time with the intention of addressing the lessons he learnt developing mongrel1.

mongrel2 has some novel and compelling features that make it an ideal and flexible centrepiece of a web environment.

mongrel2 Features

  • excellent and quick http parsing

    The mongrels have correct and fast http parsers written in C generated by Ragel.

  • http message processors (‘handlers’) are decoupled from mongrel2

    mongrel2 handlers do not execute in the mongrel2 address space and can be hosted / run on the same or a different host to mongrel2 (anywhere network-reachable).

    Multiple instances of a handler can be configured and run for performance and / or reliability.

  • use of ZeroMQ for the middleware layer

    One of mongrel2 most novel and compelling features is its use of ZeroMQ for the middleware messaging layer for mongrel2 to handler communication. ZeroMQ is a lightweight asynchronous message passing protocol implementation that presents the familiar socket interface.

  • handlers can be written in most popular languages

    Using ZeroMQ enables mongrel2 to be language-agnostic - as long as the handler can “talk” ZeroMQ, it can communicate with mongrel2 successfully. ZeroMQ bindings already exist for most (all?) of the common scripting languages.

  • easy to build reliable environments

    It would be straightforward to create a global lightweight mongrel environment to serve current and future needs for intra and inter data centre administration.

  • frugal memory footprint

  • source is available

    Usual suspect: Github

Setting up mongrel2

The mongrel2 manual is very good and worth reading being both comprehensive and well-written.

Prereqs

ZeroMQ must be installed on the mongrel server and on any server running a handler. Some guidance on installing ZeroMQ can be found in another post.

The mongrel2 server needs some dependent packages; these work for me:

1
sudo apt-get install libxml2-dev libxslt-dev

Download

Download the mongrel2 tarball from the link on Github.

At the time of writing the latest version was 1.8.0. (Note the link in the banner of the main website may be old.)

Build and Install

I recommend you read the installation chapter of the manual but, briefly, its a standard, well-behaved make:

1
2
3
4
5
wget https://github.com/zedshaw/mongrel2/tarball/v1.8.0
tar -xvf v1.8.0
cd ./zedshaw-mongrel2-bc721eb
make
sudo make install

mongrel2 installs into /usr/local

Configuration

Configuring mongrel2 for basic operation isn’t hard but again I would strongly encourage you to read the relevant chapter of the manual - there is a wealth of information there.

Section 3.3 (‘Source 8’) give a simple example (given below) of a text-based configuration and section 3.4 begins the explanation of what the elements mean and do.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
main = Server( 
    uuid="f400bf85-4538-4f7a-8908-67e313d515c2", 
    access_log="/logs/access.log", 
    error_log="/logs/error.log", 
    chroot="./", 
    default_host="localhost", 
    name="test", 
    pid_file="/run/mongrel2.pid", 
    port=6767, 
    hosts = [ 
        Host(name="localhost", routes={ 
            ’/tests/’: Dir(base=’tests/’, index_file=’index.html’, 
                             default_ctype=’text/plain’) 
        }) 
    ] 
) 
 
servers = [main]

mongrel2 has a utility command m2sh to load a text-based configuration into its internal (sqlite) database format. For example, if the above configuration were to be stored in ./config/test1.conf, the following command would create the sqlite database in ./db/test1.sqlite.

1
m2sh load -config ./config/test1.conf --db ./db/test1.sqlite

Starting and stopping mongrel2

m2sh does the honours again, for example using the database created above, mongrel2 could be started by this command:

1
m2sh start --db ./db/test1.sqlite -host localhost -sudo

m2sh to stop as well:

1
m2sh stop --db ./db/test1.sqlite -host localhost

Example - A Ruby data acquisition handler

I was interested in handlers written in Ruby and found some useful links out there especially one by Carson McDonald that gave me a great heads up on writing a mongrel2 handler in Ruby. Thanks Carson!

The context for my example is data acquisition: accepting HTTP requests with useful payloads from many sources, extracting the useful data and loading it into rows in a relational database (MySQL) table.

In Carson’s code mongrel2 and the handler were collocated on the same server, in mine they aren’t. In my scenario, curl acts as the http client sending requests to mongrel2 running on one server (192.168.16.200; ports 9994 & 9995) which contacts a handler running on another server (ports 9994 & 9995) which in turn write to table in a mysql database running on another server.

This topology shows the flexibility of running the various components wherever is most appropriate; in fact, it may make more sense in this case to run the handler on the database server. You decide.

I don’t have any rigorous or quantative performance figures but even on my desktop running all three servers as VirtualBox guests. the solution processed a very respectable number of requests per second.

Note I wouldn’t claim this code or setup is anywhere near production quality - proof of concept only (as witness the debug statements I’ve left in).

Configure mongrel2 for the data acquisition handler

Note 1: the configuration uses ports 9994 & 9995 for the mongrel2 to handler communication.

Note 2: mongrel2 does not need to know the dns name or ip address of the handler’s server: mongrel2 does not need to know beforehand where the handler is running - the handler connects to mongrel2, not the other way around.

Note 3: mongrel2 listens for client requests on port 6767 in this configuration; change to e.g. 80 for the usual, conventional operation.

Note 4: The send_ident would usually be a UUID but it can be any string (but not very secure).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# mongrel2 config for data acquisition  handler storing into MySQL

# Lots of ideas and code from Carson McDonald (http://www.ioncannon.net/programming/1384/example-mongrel2-handler-in-ruby/)

# the connection to the handler

# N.B.  use the IP address of the mongrel server 
#       becuase (remote) handler will bind to that IP

handler_mysql = Handler(send_spec='tcp://192.168.16.200:9995', 
                        send_ident='HANDLER-MYSQL-9995-9994', 
                        recv_spec='tcp://192.168.16.200:9994', recv_ident='') 

# the host 

mongrel2 = Host(name="daq_mongrel2", routes={
  '/mysql': handler_mysql 
}) 

# the server to run the host

main = Server( 
  uuid="2f62bd5-9e59-49cd-993c-3b6013c28f05", 
  access_log="/logs/access.log", 
  error_log="/logs/error.log", 
  chroot="./", 
  pid_file="/run/mongrel2.pid", 
  default_host="daq_mongrel2", 
  name="main", 
  port=6767, 
  hosts=[mongrel2] 
) 

settings = {"zeromq.threads": 1} 

servers = [main]

Start mongrel2

On the mongrel server, cd to (e.g.) the chroot folder and save the above in ./config/ruby_data_capture1.conf and load into into the sqlite database:

1
m2sh load -config ./config/ruby_data_capture1.conf --db ./db/ruby_data_capture1.sqlite

Note the chroot statement in the configuration, when mongrel2 starts it will chroot to that directory.

Start mongrel2

1
m2sh start --db ./db/ruby_data_capture1.sqlite -host daq_mongrel2 -sudo

You will be prompted for your password.

One mongrel2 has started ok, you will be able to see it listening on the two ports

1
2
3
4
5
6
7
netstat -antp

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 192.168.16.200:9994     0.0.0.0:*               LISTEN      -               
tcp        0      0 192.168.16.200:9995     0.0.0.0:*               LISTEN      -               

etc

Either mongrel2 or the handler can be started first. Until mongrel2 has a client’s request for the handler, it will not attempt to contact it.

Starting the Ruby data capture handler

On the handler server, save the following ruby script into e.g. ruby_data_capture1.rb.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
# mongrel2 data acquisition handler capturing data into MySQL

# Lots of ideas and code from Carson McDonald (http://www.ioncannon.net/programming/1384/example-mongrel2-handler-in-ruby/)

require 'zmq'
require 'json'
require 'mysql2'

eye = 'DAQ MySQL 1'

mongrel2Server = '192.168.16.200'

recvPort = '9995'
respPort = '9994'

recvURL = "tcp://#{mongrel2Server}:#{recvPort}"
respURL = "tcp://#{mongrel2Server}:#{respPort}"

mysqlHost = 'mysqlvm1'
mysqlUser = 'root'
mysqlPass = 'sa'

mysqlDBName = 'samples' # mysql database name
mysqlTBName = 'sampledata' # mysql table name

puts("#{eye} STARTED")

#useMySQL = false  # dont drive MySQL
useMySQL = true   # drive MySQL

dbConn = case useMySQL
         when TrueClass then
           puts("#{eye} CONNECTNG TO MYSQL db >#{mysqlDBName}< table >#{mysqlTBName}<")
           r = begin

                 Mysql2::Client.new(:host => mysqlHost, :database => mysqlDBName, :username => mysqlUser, :password => mysqlPass)
               rescue Exception => e
                 puts("Exception on MySQL connection establishment e >#{e}<")
                 raise
               ensure
                 #con.close if con
               end
           puts("#{eye} CONNECTED TO MYSQL db >#{mysqlDBName}< table >#{mysqlTBName}< con >#{r.class}< >#{r}<")
           r
         else
           nil
         end

handler_thread = Thread.new do
  
  handler_ctx = ZMQ::Context.new(1)
  
  receive_queue = handler_ctx.socket(ZMQ::PULL)
  receive_queue.connect(recvURL)
  
  puts("#{eye} RECV X1 recvURL >#{recvURL}<")
  
  response_publisher = handler_ctx.socket(ZMQ::PUB)
  response_publisher.connect(respURL)

  puts("#{eye} RESP X1 respURL >#{respURL}<")
  
  response_publisher.setsockopt(ZMQ::IDENTITY, "82209006-86FF-4982-B5EA-D1E29E55D481")
  
  stop_queue = handler_ctx.socket(ZMQ::PULL)
  stop_queue.connect("ipc://shutdown_queue")
  
  puts("#{eye} STOP X1")
  
  stopped = false
  until stopped do
    selected_queue = ZMQ.select([receive_queue, stop_queue])
    if selected_queue[0][0] == stop_queue # Anything on the stop_queue ends processing
      stop_queue.close
      receive_queue.close
      response_publisher.close
      handler_ctx.close
      stopped = true
    else
      # Request comes in as "UUID ID PATH SIZE:HEADERS,SIZE:BODY,"
      sender_uuid, client_id, request_path, request_message = receive_queue.recv(0).split(' ', 4)
      len, rest = request_message.split(':', 2)
      headers = JSON.parse(rest[0...len.to_i])
      len, rest = rest[(len.to_i+1)..-1].split(':', 2)
      body = rest[0...len.to_i]

#=begin
      begin
        client_id_size = sprintf("%d", client_id).size
        request_query = request_message['QUERY']
        post_uri = headers['URI']
        post_pattern = headers['PATTERN']
      end
#=end
      post_values = Hash[*body.gsub('&','=').split('=')]
      
      puts("#{eye} RECV X3 post_values >#{post_values.class}< >#{post_values}<") 
      
      useMySQL && begin
                    
                    keyNames = ['(', "#{r = post_values.keys.map{|k| [k, ',']}.flatten; r.pop; r.join}", ')'].join 
                    
                    # can quote ints - MySQL will cast      

                    keyValues = ['(', "#{r = post_values.values.map{|k| ['"', k, '"', ',']}.flatten; r.pop; r.join}", ')'].join                    

                    insertCommand = "INSERT INTO #{mysqlTBName} #{keyNames} VALUES #{keyValues};"
                    
                    puts("#{eye} INSERT keyNames >#{keyNames}< keyValues >#{keyValues} insertCommand >#{insertCommand}")      
                    
                    true && begin
                      dbConn.query("#{insertCommand}")                
                    rescue Exception => e
                       puts("#{eye} MySQL INSERT EXCEPTION e >#{e}<")
                      raise
                    end
                    
                  end
      
      #puts("#{eye} RECV X2 sender_uuid >#{sender_uuid}< client_id >#{client_id_size}< >#{client_id}< request_path >#{request_path}< request_query >#{request_query}< request_message >#{request_message.class}< >#{request_message}<")
      
      if headers['METHOD'] == 'JSON' and JSON.parse(body)['type'] == 'disconnect'
        next # A client has disconnected, might want to do something here...
      end
      
      # Response goes out as "UUID SIZE:ID ID ID, BODY"
      #content_body = "Hello world!"
      content_body = "#{eye} client_id >#{client_id}<\n"
      #response_value = "#{sender_uuid} 1:#{client_id}, HTTP/1.1 200 OK\r\nContent-Length: #{content_body.size}\r\n\r\n#{content_body}"
      response_value = "#{sender_uuid} #{client_id_size}:#{client_id}, HTTP/1.1 200 OK\r\nContent-Length: #{content_body.size}\r\n\r\n#{content_body}"
      response_publisher.send(response_value, 0)
    end
  end
end

ctx = ZMQ::Context.new(1)
stop_push_queue = ctx.socket(ZMQ::PUSH)
trap('INT') do # Send a message to shutdown on SIGINT
  stop_push_queue.bind("ipc://shutdown_queue")
  stop_push_queue.send("shutdown")
end

handler_thread.join

stop_push_queue.close

puts("#{eye} FINSIHED")

__END__

Start the handler as you’d expect:

1
ruby ./ruby_data_capture1.rb

You will see some initial diagnostics but then it will go quiet.

Create the MySQL database and table

Needs only be done once of course. I use a simple Ruby script e.g.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
#!/usr/bin/ruby

require 'mysql2'

mysqlHost = 'mysqlvm1'
mysqlUser = 'root'
mysqlPass = 'sa'

mysqlDBName = 'samples'
mysqlTBName = 'sampledata'

#puts("\n"*10)

createTBTextNom = <<-"EOH"
        (
         id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
         name VARCHAR(100),
         age INT,
         sex VARCHAR(1),
         dose INT,
         responsea INT,
         responseb INT,
         weight INT,
         colour VARCHAR(20),
         state VARCHAR(10)
       )
EOH

createTBText = createTBTextNom.gsub("\n",'')
createTBText = createTBTextNom
createTBText = createTBTextNom.split("\n").compact.map(&:strip).join

puts("CREATNG TABLE >#{createTBText}<")

begin
  dbConn = Mysql2::Client.new(:host => mysqlHost, :database => mysqlDBName, :username => mysqlUser, :password => mysqlPass)
  dbConn.query("DROP TABLE IF EXISTS #{mysqlTBName};")
  dbConn.query("CREATE TABLE IF NOT EXISTS #{mysqlTBName} #{createTBText};")
rescue Exception => e
  puts("e >#{e}<")
  #puts e.error
ensure
  #con.close if con
end

puts("CREATED TABLE >#{createTBText}<")

__END__

Fire some client requests at mongrel2

I have a simple script with a number of curls in e.g.

1
2
3
4
5
6
#!/bin/sh -e
curl http://192.168.16.200:6767/mysql/samples/1 -d 'age=41' -d 'name=joe' -d 'colour=red' -d 'state=ill' -d 'sex=F' -d 'weight=98' -d 'dose=82' -d 'responsea=20' -d 'responseb=19'
curl http://192.168.16.200:6767/mysql/samples/2 -d 'age=14' -d 'name=jane' -d 'colour=blue' -d 'state=nomore' -d 'sex=F' -d 'weight=138' -d 'dose=3' -d 'responsea=31' -d 'responseb=36'
curl http://192.168.16.200:6767/mysql/samples/3 -d 'age=98' -d 'name=fred' -d 'colour=red' -d 'state=ok' -d 'sex=M' -d 'weight=179' -d 'dose=9' -d 'responsea=13' -d 'responseb=19'
curl http://192.168.16.200:6767/mysql/samples/4 -d 'age=73' -d 'name=jane' -d 'colour=blue' -d 'state=ill' -d 'sex=M' -d 'weight=74' -d 'dose=72' -d 'responsea=6' -d 'responseb=56'
curl http://192.168.16.200:6767/mysql/samples/5 -d 'age=64' -d 'name=lucy' -d 'colour=yellow' -d 'state=find' -d 'sex=M' -d 'weight=13' -d 'dose=69' -d 'responsea=44' -d 'responseb=40'

On the handler server, you will see the diagnostic puts e.g.

1
2
3
4
5
6
7
8
9
DAQ MySQL 1 INSERT keyNames >(age,name,colour,state,sex,weight,dose,responsea,responseb)< keyValues >("41","joe","red","ill","F","98","82","20","19") insertCommand >INSERT INTO sampledata (age,name,colour,state,sex,weight,dose,responsea,responseb) VALUES ("41","joe","red","ill","F","98","82","20","19");
DAQ MySQL 1 RECV X3 post_values >Hash< >{"age"=>"14", "name"=>"jane", "colour"=>"blue", "state"=>"nomore", "sex"=>"F", "weight"=>"138", "dose"=>"3", "responsea"=>"31", "responseb"=>"36"}<
DAQ MySQL 1 INSERT keyNames >(age,name,colour,state,sex,weight,dose,responsea,responseb)< keyValues >("14","jane","blue","nomore","F","138","3","31","36") insertCommand >INSERT INTO sampledata (age,name,colour,state,sex,weight,dose,responsea,responseb) VALUES ("14","jane","blue","nomore","F","138","3","31","36");
DAQ MySQL 1 RECV X3 post_values >Hash< >{"age"=>"98", "name"=>"fred", "colour"=>"red", "state"=>"ok", "sex"=>"M", "weight"=>"179", "dose"=>"9", "responsea"=>"13", "responseb"=>"19"}<
DAQ MySQL 1 INSERT keyNames >(age,name,colour,state,sex,weight,dose,responsea,responseb)< keyValues >("98","fred","red","ok","M","179","9","13","19") insertCommand >INSERT INTO sampledata (age,name,colour,state,sex,weight,dose,responsea,responseb) VALUES ("98","fred","red","ok","M","179","9","13","19");
DAQ MySQL 1 RECV X3 post_values >Hash< >{"age"=>"73", "name"=>"jane", "colour"=>"blue", "state"=>"ill", "sex"=>"M", "weight"=>"74", "dose"=>"72", "responsea"=>"6", "responseb"=>"56"}<
DAQ MySQL 1 INSERT keyNames >(age,name,colour,state,sex,weight,dose,responsea,responseb)< keyValues >("73","jane","blue","ill","M","74","72","6","56") insertCommand >INSERT INTO sampledata (age,name,colour,state,sex,weight,dose,responsea,responseb) VALUES ("73","jane","blue","ill","M","74","72","6","56");
DAQ MySQL 1 RECV X3 post_values >Hash< >{"age"=>"64", "name"=>"lucy", "colour"=>"yellow", "state"=>"find", "sex"=>"M", "weight"=>"13", "dose"=>"69", "responsea"=>"44", "responseb"=>"40"}<
DAQ MySQL 1 INSERT keyNames >(age,name,colour,state,sex,weight,dose,responsea,responseb)< keyValues >("64","lucy","yellow","find","M","13","69","44","40") insertCommand >INSERT INTO sampledata (age,name,colour,state,sex,weight,dose,responsea,responseb) VALUES ("64","lucy","yellow","find","M","13","69","44","40");

On the MySQL server, in a mysql session, you will be able to see the row count increase and the contents of the rows e.g.

1
2
3
use samples;
select count(s) from sampledata;
select * from sampledata;

Final Words

This is / was a fairly trivial - albeit useful - example of how mongrel2 can be used.

But it does demonstrate clearly the flexability and utility of the mongrel2 architecture, especially Zed’s great idea of using ZeroMQ for the middleware, messaging layer allowing for language-agnostics handlers and whatever topology works best.