Bloggitation

Feedback

None. Oops, one comment from Shaun, thank you.

Updates

Since my previous post I found a little bug in the CEAN installation. Or maybe it’s just not very well documented thing:

After the installation the path to the control socket is pointing by default to /usr/local/var :

=ERROR REPORT==== 25-Jun-2007::10:37:24 ===
Failed to create/manipulate the ctlfile 
called /usr/local/var/run/yaws/ctl-default
Either problems with permissions or  earlier runs of yaws 
with the same id  <default> as this, check dir for perms
None of Yaws ctl functions will work

Seems during the compilation you can control where the control socket will be via VARDIR environment variable, but we installing from binaries, so…!? I couldn’t find a way to set this path during the run time.

For me I solved the problem by recompiling the ${CEAN}/erlang/lib/yaws-1.68/src/yaws_generated.erl and replace the original ebin/yaws_generated.beam file:

$ cd ${CEAN}/erlang/lib/yaws-1.68/src/
%% correct the paths inside the yaws_generated.erl
%% to point to the CEAN install directory
$ erlc yaws_generated.erl
$ mv yaws_generated.beam ../ebin

OK, let’s go back to the today’s topic

Erlang project directory structure

Like all we know:

“Convention over Configuration” means a developer only needs to specify unconventional aspects of their application.—Wikipedia, Ruby On Rails

What is the typical erlang project structure?

• ebin/ – directory with compiled .beam files – in general you need to deploy only them to the remote location
• src/ – directory with source .erl files
• include/ – directory with .hrl files, defining macros, records etc.
• priv/ – application specific configuration, docroot for the yaws-based projects

And because I know you are lazy here is an escript to automate the erlang project directory creation . Usage:

$ wget -O ~/bin/rails http://files.zhekov.net/erlday.txt && chmod 755 ~/bin/rails
$ rails ~/Work/erlang/erl_first

Oops, mistake. Better name it erlday , not rails, you still need your good old Rails, do you? ;)

How can you recompile all your sources? What about the Makefile? Or Rakefile? Good news for the Windows guys: You just don’t need all of them!

1. On the top of your project directory create a file with name Emakefile and content:

{"src/*", [debug_info, {outdir, "ebin"}, {i,"include"}]}.

2. Start the erlang shell and type make:all().

$ erl
...
1> make:all().

3. There is no step 3 ;)

Note: If you are using the erlday escript , it will create automatically the Emakefile for you.

How to find where are make module sources:

$ erl
...
1> code:which(make).
"/home/erl/cean/erlang/lib/tools-2.5.4/ebin/make.beam" 

so it will be .../src/make.erl .

Practice – comet chat with yaws

And because only reading is not good, here something to play with – Comet-based, yaws-powered chat.

WARNING! The Comet application sources belong to Mikage-san . I only changed the messages to be in English. And because there was nothing mentioned about the licensing, I put a link with the original sources in the beginning of the files.

Get the erl_comet tarball and uncompress it

$ mkdir ~/Work/erl_comet && cd ~/Work/erl_comet
$ wget http://files.zhekov.net/erl_comet.tgz && tar xvzf erl_comet.tgz

Adjust the paths to the installation directory

• src/chat.erl – path to the yaws.hrl file
• priv/yaws.conf – path to the log directory, address, port, and path to docroot.
• priv/docroot/index.yaws – the URL to your template file

Recompile everything and quit the erlang shell session (Ctrl+G)

$ erl
...
1> make:all().
Recompile: src/chat
up_to_date

Note: Specially for that project I included a Makefile, so you can use make and make run .

Start the application

$ make run
...
1> 
=INFO REPORT==== 25-Jun-2007::11:56:31 ===
Yaws: Using config file priv/yaws.conf
yaws:Add path "/home/erl/cean/erlang/lib/yaws-1.68/examples/ebin" 
yaws:Running with id="comet" 
Running with debug checks turned on (slower server) 
Logging to directory "/home/erl/cean/var/log" 

=INFO REPORT==== 25-Jun-2007::11:56:31 ===
Yaws: Listening to 127.0.0.1:4080 for servers
 - http://localhost:4080 under /home/erl/Work/erlang/erl_comet/priv/docroot

=INFO REPORT==== 25-Jun-2007::11:56:31 ===
sync call chat:start

Point your browser to URL http://localhost:4080/ and start chatting.

Next time – inets, pico and yaws deployment. Or something else, if there are requests…

I’ll be in vacation for about 10 days, so be patient… ;)

Common Crap

After the first article there was more then 1500 hits. And interesting, maybe about 80% of them coming from RubyCorner ! Feedback: near to zero – just several comments on the blog and on reddit . Nothing to improve? Nothing to add? Hm, so it’s real:

Web 2.0 is about voting. It make you stupid!

OK. Cut the crap and let’s go on the topic

Install yaws

Hopefully you already have the CEAN installed . Next good thing is to install yaws – HTTP high performance 1.1 webserver particularly well suited for dynamic-content webapplications (something like a mongrel for Erlang ;) )

1> cean:install(yaws).
+ ibrowse md5=<<212,78,39,29,222,40,222,24,91,65,162,253,27,109,180,42>>
+ compiler md5=<<19,177,55,189,37,11,164,23,75,115,129,152,51,24,167,234>>
+ inets md5=<<41,196,119,166,237,121,195,180,130,134,225,129,36,167,207,153>>
+ ssl md5=<<75,245,109,43,92,129,104,171,158,38,157,214,21,117,107,3>>
+ tools md5=<<15,253,151,211,133,14,99,68,114,47,193,244,20,251,111,76>>
+ xmerl md5=<<60,100,38,136,168,113,16,236,239,184,152,155,148,148,37,112>>
+ xmlrpc md5=<<158,3,240,16,89,215,193,84,48,248,191,58,98,30,54,53>>
+ yaws md5=<<88,53,90,204,22,201,30,118,182,116,117,36,67,251,39,191>>
ok

Create var/log , var/run and var/www INSIDE YOUR CEAN directory.

$ cd ~/cean
$ mkdir -p var/lib var/run var/www

Download the example config file and the yaws startup script (they are just modified versions from the Erlang sources, you can do this by yourself)

$ cd ~/cean
$ wget -O yaws.conf http://files.zhekov.net/yaws.conf.txt
$ cd ~/bin
$ wget -O yaws http://files.zhekov.net/yaws.txt && chmod 755 yaws

Correct the paths in the both files (##CEANDIR## , $CEANDIR, your $OSDIR – the default is linux-x86). Start yawsing

$ yaws -i -c ~/cean/yaws.conf 
Erlang (BEAM) emulator version 5.5.4 [source] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.5.4  (abort with ^G)
1> 
=INFO REPORT==== 14-Jun-2007::12:45:01 ===
Yaws: Using config file /home/erl/cean/yaws.conf
yaws:Add path "/home/erl/cean/erlang/lib/yaws-1.68/ebin" 
yaws:Running with id=default
Running with debug checks turned on (slower server) 
Logging to directory "/home/erl/cean/var/log" 

=INFO REPORT==== 14-Jun-2007::12:45:01 ===
Yaws: Listening to 127.0.0.1:8080 for servers
 - http://localhost:8080 under /home/erl/cean/var/www

Next time: Erlang project directory structure and “makefile” a.k.a “Convention over Configuration”.

After the last Osaka Erlang meeting , decided to start a column of articles for Erlang everyday use. Recently there is a big interest in Erlang, but the beginning is always difficult. Some push during the first steps usually helps. OK, on the topic:

Installation

I hear several time: “It’s take me 30 min to compile it…”, “The PLT compile takes … min…”, “On Windows it’s difficult…”. What are you doing guys?

CEAN 1.2 is online, with 200 packages. Binary packages are available for up to 17 platforms.

Just download the binaries for your platform and you are ready. There are two flavors:

• “Production” – only .beam files, small size, ready for production use
• “Developer” – .beam files AND sources (.erl) – go with this for your development machine

Plus you get package management (install, uninstall, dependencies tracking) for free:

[~/cean]$ ./start.sh 
Erlang (BEAM) emulator version 5.5.4 [source] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.5.4  (abort with ^G)
1> cean:installed().
["cean",
 "compiler",
 "debugger",
 ...
 "yaws"]
2> cean:install("gs").
+ gs md5=<<241,57,155,224,152,243,10,41,231,159,223,113,196,45,116,195>>
ok
3> cean:uninstall("snmp").
can not uninstall snmp. ["mnesia"] are dependent upon it.
ok

By default the installer and the archive both install everything in one directory. Usage is through the included start.sh shell script (not sure how is on Windows).

What to use it from other directories?

Easy, just a thin wrapper will fix this:

$ cat ~/bin/erl
#!/bin/sh

CEANDIR=/home/erl/cean
${CEANDIR}/start.sh -pa ${CEANDIR}/lib/*/ebin "$@" 

Missing erlc (compiler)?

Just add -compile to the arguments of the script above (some alias will do the job too):

$ erl -compile processes.erl

Need to_erl, run_erl

(for example for the yaws start/stop script)? Add the erts bin directory to your path:

$ cat .bash_profile
PATH=~/cean/erlang/erts-5.5.4/linux-x86/bin:"${PATH}" 

Happy erlanging. Hm, what to post about the next time? Some ideas:

• project directories and “makefile”
• deployment – inets, pico, yaws

Stay tuned…

Second approximation of the conversion between old roman and integer digits. Added some error checks and released the roma2int() functionality

%% convert between integer and old roman
%% usage:
%%   convert:int2roma(555) -> "DLV" 
%%   convert:int2roma(999) -> "DCCCCLXXXXVIIII" 
%%   convert:roma2int("DLV") -> 555
%%   convert:roma2int("DCCCCLXXXXVIIII") -> 999

-module(convert).
-export([int2roma/1,roma2int/1]).

i2r(1)    -> "I";
i2r(5)    -> "V";
i2r(10)   -> "X";
i2r(50)   -> "L";
i2r(100)  -> "C";
i2r(500)  -> "D";
i2r(1000) -> "M".

r2i($I) -> 1;
r2i($V) -> 5;
r2i($X) -> 10;
r2i($L) -> 50;
r2i($C) -> 100;
r2i($D) -> 500;
r2i($M) -> 1000;
r2i(X)  -> erlang:error("invalid input: " ++ [X]).

roma2int(L) -> catch(lists:sum([r2i(X) || X <- L])).

int2roma(X) when is_integer(X), X < 10000 ->
  lists:flatten(int2roma(X,[1000,500,100,50,10,5,1])).
int2roma(_,[]) -> []; %% done
int2roma(X, [H|T])->
  [lists:duplicate(X div H, i2r(H))|int2roma(X rem H, T)].

Still a lot to do, like checking the order of character occurences in the string ( roma2int() ), better error messages etc. But for an exercise maybe it’s enough.

• 05-06-2007 update: based on yhara-san’s code I simplified the roma2int() function to one line. It even reports which exactly is the bad symbol. if check in int2roma() replaced by clause with guard.
• 07-06-2007 update: some hints from the okkez-san’s code – no need to have a X>0 clause, cause in case of 0 it just will not execute the lists:duplicate part.
• 13-06-2007 update: List1++List2 construction seems depricated (slow), replaced with \[H|T\] in int2roma()

03-Jun-2007, Toyonaka, Hotarugaike

Osaka Erlang Meeting #1

RTFM

Most of the time was just reading the japanese translation of the “Programming Erlang” book. 53 pages full of japanese ;) Pretty difficult task. All people divided in 2 groups – have already started erlang at least once, still not. The beginners – needed to read all pages as fast as possible and merge with the first group – also difficult task ;)

fun function_name/args_num
%% for example
%% fun fizzbuzz/1. <=> fun(X)-> fizzbuzz(X) end. 

interesting: soap – ajax – comet relation :)

roman2integer or integer2roman

In the last 30-40 minutes we’ve got an interesting task to code – convert old roman to/from integer. I decided to go with the integer to roman part of the problem. There was not enough time on the meeting, so I continued in the nearest Starbucks, and in home…Hm, Erlang is catchy ;) OK. so about the initial approximation of the solution:

%% convert integer to old roman
%% usage:
%%   convert:to_roma(555) -> "DLV" 
%%   convert:to_roma(999) -> "DCCCCLXXXXVIIII" 
-module(convert).
-compile(export_all).

-define(R,[1000,500,100,50,10,5,1]).

i2r(1)    -> "I";
i2r(5)    -> "V";
i2r(10)   -> "X";
i2r(50)   -> "L";
i2r(100)  -> "C";
i2r(500)  -> "D";
i2r(1000) -> "M".

repeat(Times,X) ->
  lists:duplicate(Times,i2r(X)).

to_roma(X) ->
  lists:flatten(to_roma(X,?R)).
to_roma(_,[]) -> [];
to_roma(X,[H|T]) ->
  {Div, Rem} = {X div H, X rem H},
  if
    Div > 0 ->
      [repeat(Div,H)|to_roma(Rem,T)];
    true ->
      to_roma(Rem,T)
  end.

Still don’t like the first part. Need more efficient i2r() implementation. All advices, suggestions for improving are welcome.

Like I mentioned in one of my Jaiku posts it will be good to have a page with answers for the Erlang Course Exercises . Still nobody around with such a page, so I started to put my answers in the Mercurial repository . Some of the exercises seems useful even for everyday use. I’m pretty happy with my lists-related functions library . Until now I finished with the “Simple sequential programs” and “Simple recursive programs” sections. Stay tuned…

{active, Boolean}

...If the active option is true, which is the default, everything received from the socket will be sent as messages to the receiving process. If the active option is set to false (passive mode), the process must explicitly receive incoming data by calling gen_tcp:recv/N or gen_udp:recv/N (depending on the type of socket). If the active option is set to once (active once), one data message from the socket will be sent to the process. To receive one more message, setopts/2 must be called again with the {active,once} option…

Note: Active mode provides no flow control; a fast sender could easily overflow the receiver with incoming messages. Use active mode only if your high-level protocol provides its own flow control (for instance, acknowledging received messages) or the amount of data exchanged is small.

{keepalive, Boolean} – powered by© Comet ;)

(TCP/IP sockets) Enables periodic transmission on a connected socket, when no other data is being exchanged. If the other end does not respond, the connection is considered broken and an error message will be sent to the controlling process. Default disabled.

{reuseaddr, Boolean}

Allows or disallows local reuse of port numbers. By default, reuse is disallowed.

So the default listening socket options are (overwrite only the differences if needed):

[{active, true},
 {keepalive, false},
 {packet, 0},
 {reuseaddr, false}].

Like a part of my Erlang study, I decided to code a pure-man queues management service, similar to Amazon SQS – Even Simpler Queue Service (or Erlang SQS you decide ;) ). For now it’s just one file qserver.erl , but i hope to make it better. First I based my code on OTP ( gen_server ), but after this decided to write it from scratch. Lessons learned:

• access to the processes by name, not by Pid is more human friendly
• implement the whole API with simple cast (async) and call (sync) messages
• one function with guards (do_q) is maybe better than several functions with different names
• pg2: gave me better name service, then global:register/unreister

Usage:

Erlang (BEAM) emulator version 5.5.3 [source] [async-threads:0] [hipe] [kernel-poll:false]
1> c(qserver).
{ok,qserver}
2> qserver:start('aa').
{ok,<0.38.0>}
3> qserver:start('bb').
{ok,<0.40.0>}
4> qserver:listq().    
[bb,aa]
5> qserver:inq('aa',123).
{ok,123}
6> qserver:lenq('aa').    
1
7> qserver:inq('aa',[1,2,3]).
{ok,[1,2,3]}
8> qserver:revq('aa').       
ok
9> qserver:outq('aa').
[1,2,3]
10> qserver:outq('aa').
123
11> qserver:outq('aa').
** exited: empty **
12> qserver:stop('aa').
ok
13> qserver:outq('aa').
{oops,{no_such_group,aa}}

There is still a lot to be done – data exchange with the queues (i want to use UBF for data transport format), more robust name service (pg2 is based on gen_server, so if the node with the database die, all names will be lost) – gen_server_cluster based , ACL to the queues etc.

Other Activities:

• There is no enough time for real blogging, so I just pasting random stuff on ThouTr , my thumblog
• Finally made a ‘front page’ for zhekov.net – (c) KISS ;) – acts like a portal to all my web sites
• Started using Mercurial for my personal development – distributed, svn-like commands, coming with web frontend by default (but CGI only = pretty slow)

Luke Gorrie’s comment to ‘Erlang: parallel-map and parallel-foreach’ article

Parallel map brainfsck implementation in Erlang:

%% Map function F over list L in parallel.
parmap(F, L) ->
  Parent = self(),
  [ receive 
      {Pid, Result} -> Result  end  || 
          Pid <- [ spawn( fun() -> Parent ! {self(), F(X)}  end) || X <- L ]
  ].

The notation [ F(X) || X <- L] means “The list of F(X) such that X is taken from the list L” .

Something always to start with:

-module(ctemplate).
-compile(export_all).
start() ->
    spawn(fun() -> loop([]) end).

rpc(Pid, Query) ->
    Pid ! {self(), Query},
    receive
        {Pid, Reply} ->
            Reply
    end.

loop(X) ->
    receive
        Any ->
            io:format("Received:~p~n" ,[Any]),
            loop(X)
    end.

Programming Erlang Book

Learn how to write truly concurrent programs—programs that run on dozens or even hundreds of local and remote processors. See how to write high reliability applications—even in the face of network and hardware failure—using the Erlang programming language.

Pretty useful piece of code:

is_pid_alive(Pid) 
  when is_pid(Pid) ->
    rpc:call(node(Pid), erlang, is_process_alive, [Pid]).

Checking if some process is alive. But working only with Erlang processes. I cannot check the status of some general TCP server. Still pretty cool. Based on that function I added a status check to Reverl. Maybe will use it for the other proxy nodes checks. The result: gateway behaviour with RR backend choice and backends failover tracking . Usage:

$ erl
1> l(gproxy3).
{module,gproxy3}
2> S1=gproxy3:start_be(fun(X) -> 2*X end).
<0.33.0>
3> S2=gproxy3:start_be(fun(X) -> 3*X end).
<0.35.0>
4> S3=gproxy3:start_be(fun(X) -> 4*X end).
<0.37.0>
5> G=gproxy3:start_gw([S1,S2,S3]).
<0.39.0>
6> gproxy3:rpc(G,21).
42
7> gproxy3:rpc(G,21).
63
8> gproxy3:rpc(G,21).
84
9> S4=gproxy3:start_be(fun(X) -> 5*X end).
<0.44.0>
10> gproxy3:add_be(G,S4).
{be_add,<0.44.0>}
11> gproxy3:del_be(G,S2).
{be_del,<0.35.0>}
12> gproxy3:list_be(G).  
[<0.44.0>,<0.33.0>,<0.37.0>]
13> gproxy3:stop(S1).
stop
14> gproxy3:rpc(G,21).                     
105
15> gproxy3:rpc(G,21).

=ERROR REPORT==== 26-Feb-2007::17:55:38 ===
(<0.39.0> gproxy3:30) backend <0.33.0> is down
84
16> gproxy3:rpc(G,21).
105

Next step: to figure what parts can be handled by the already existing gen_server behaviour and rewrite my code to separate the differences.

Created the reverl project on Google Code .

Reverl is reverse proxy, load balancer and HTTPS front-end for Web server(s), implemented in Erlang.

Before to start the real coding, I needed a more general proxy(gateway) behaviour.

What exactly is proxy (or gateway)? It’s the middle-man between the client and a set of backend service providers, the guy who get the requests, resend them to some backend, get the response for that backend and return it back to the client. The interesting part is the choice of backend with possible load balancing and tracking backends failover. In general the proxy(gateway) give you High Availabity (HA) of the provided services.

So on the question. The first code approximation:

-module(gproxy0).
-export([start_be/1,start_gw/1,change_be/2,rpc/2]).

%% start the server (backend)
start_be(Fun) -> spawn(fun() -> loop(Fun) end).

%% start the proxy (frontend)
start_gw(Backend) -> spawn(fun() -> gloop(Backend) end).

%% Pid can be gateway or server 
rpc(Pid, Q) ->
  Pid ! {self(), Q},
  receive {Pid, Reply} -> Reply
  end.

%% change the backend , send to the GATEWAY
change_be(Gw,Backend) -> Gw ! {be_change,Backend}.

%% server (backend) loop
loop(Fun) ->
  receive
    %% direct request
    {From, X} -> From ! {self(), Fun(X)}, loop(Fun);
    %% request via gateway
    {Gw, From, X} -> Gw ! {From, ok, Fun(X)}, loop(Fun)
  end.

%% gateway (frontend) loop
gloop(Backend) ->
  receive
    {be_change,Backend1} -> gloop(Backend1);
    {Client, X} -> Backend ! {self(), Client, X}, gloop(Backend);
    {Client, ok, X} -> Client ! {self(), X}, gloop(Backend)
  end.

Usage

$ erl
1> c(gproxy0).
{ok,gproxy0}
2> F1=fun(X) -> 2*X end.      % provided service
#Fun<erl_eval.6.56006484>
3> B1 = gproxy0:start_be(F1). % start the backend
<0.38.0>
4> gproxy0:rpc(B1,21).        % service request to the backend
42
5> G=gproxy0:start_gw(B1).    % create the gateway with one backend
<0.41.0>
6> gproxy0:rpc(G,21).         % service request to the gateway
42
7> B2=gproxy0:start_be(fun(X) -> 3*X end).  % new backend
<0.44.0>
8> gproxy0:change_be(G,B2).   % change the backend
{be_change,<0.44.0>}
9> gproxy0:rpc(G,21).                     
63

More sophisticated code versions (changing service code on the fly, handling errors etc.) are available for download :

• gproxy.erl—General gateway behaviour
• gproxy1.erl—General gateway behaviour with some errors protection
• gproxy2.erl—Gateway behaviour with simple round robin between backends

Stay tuned…

* Cast (send msg)

A ! B

* Event (get msg)

receive A -> A end

* Call (RPC)

A ! {self(), B},
receive 
  {A, Reply} -> 
    Reply
  end

* Callback

receive
  {From, A} ->
    From ! F(A)
  end

Some trickery follows ;). Here something cool – Parallel RPC:

par_rpc([A, B, C], M)

par_rpc(Ps, M) ->
  Self = self(),
  Tags = map(fun(I) ->
               Tag = make_ref(),
               spawn(fun() ->
                       Val = rpc(I, M),
                       Self ! {Tag, Val}
                     end),
               Tag
               end, Ps),
   yield(Tags).

yield([]) -> [];
yield([H|T]) ->
  Val1 = receive {H, Val} -> Val end,
  [Val1|yield(T)].

Origin (PDF): Making Reliable Distributed Systems in the Presence of Software Errors

• Everything is a process.
• Processes are strongly isolated.
• Process creation and destruction is a lightweight operation.
• Message passing is the only way for processes to interact.
• Processes have unique names.
• If you know the name of a process you can send it a message.
• Processes share no resources.
• Error handling is non-local.
• Processes do what they are supposed to do or fail.

...The statement about error handling is perhaps less obvious. When we make a fault-tolerant system we need at least two physically separated computers. Using a single computer will not work, if it crashes, all is lost. The simplest fault-tolerant system we can imagine has exactly two computers, if one computer crashes, then the other computer should take over what the first computer was doing. In this simple situation even the sodware for fault-recovery must be non-local; the error occurs on the first machine, but is corrected by sodware running on the second machine.