Applications

2 Applications
Applications are used for "packaging" system components. An application consists of a set of resources, such as modules, registered names, and processes.
By structuring the system into a well-defined set of applications, the system designer is forced to think about the system in terms of its sub-components and to decide which functionality each application should have.
2.1 Programming an Application
Several operations are possible with an application. In particular, we can load, unload, start and stop an application.
When an application is loaded, the system checks that all the resources the application will need are present. If loading is successful, the application can be started at a later time.
When an application is started, an application_master process is created. Application master is the group leader of all the processes in the application.
The application master is aware of every process in the application. Thus, if the application was stopped the application master would terminate all processes, for which this application is responsible, in a controlled fashion.
When an application is unloaded, all code relating to the application is removed.

Application Controller and Applications
All operations on applications are coordinated by a single process with the name application_controller. The shaded circles in the above diagram represent applications. Each individual application is controlled by an application master.
2.2 The Application Resource File
The resources required by each application is defined in an application resource file. These files have the extension .app, which specifies the resources required by the application and how the application should be started.
A resource file (ApplName.app) has the following syntax:
{application, ApplName,
  [{description,  String},
   {vsn,          String},
   {id,           String},
   {modules,      [{Mod1,Vsn1}, Mod2, {Mod3,Vsn3} .., {ModN,VsnN}]},
   {maxP,         Int | infinity},
   {maxT,         Seconds | infinity},
   {registered,   [Name1, Name2, ...]},
   {applications, [Appl1, Appl2, .., ApplN]},
   {included_applications, [Appl1, Appl2, .., ApplN]},
   {env,          [{Par1, Val1}, {Par2, Val2} .., {ParN, ValN}]},
   {mod,          {Mod, StartArgs}},
   {start_phases, [{Phase, PhaseArgs}]}]}.
    
The keys have the following meanings:

description
Textual description of the application.

vsn
Version of the application. It should not contain a directory separator.

id
Product identification of the application.

modules
List of all the modules and their versions introduced by this application. A module can be listed without a version, with only the module name stated. A module can only be defined in one application.

maxP
Maximum allowed number of simultaneous processes which this application can manage (or the atom infinity). The key maxP is optional and defaults to infinity.

maxT
Maximum time that the application can run (or the atom infinity). The key maxT is optional and defaults to infinity.

registered
Lists all the names of registered processes used in this application.

applications
List of applications which must be started before this application can be started. Most applications have dependencies to the kernel and stdlib applications.

included_applications
List of applications which are included by this application. An included application is loaded, but not started, by the application_controller. Processes implemented in an included application should be placed underneath a supervisor in the including application. This key is optional and defaults to an empty list.

env
List of the environment variables in the application. Each parameter ParX is an atom, and the associated ValX can be any term. The env key is optional and defaults to an empty list.

mod
Application call back module of the application behavior. The application master starts the application by evaluating the function Mod:start(Type, StartArgs), refer to the kernel reference manual. When the application has stopped, by command or because it terminates, the application master calls Mod:stop(State) to let the application clean up. If no State was returned from Mod:start/2, then Mod:stop([]) is called. The mod key should be omitted for applications which are code libraries, such as the stdlib application. These applications have no dynamic behavior of their own and should not have a start function.

start_phases
List of start phases and the attached start arguments for the application. The application master starts the application by evaluating the function Mod:start_phase(Phase, Type,PhaseArgs), for each defined start phase. (See also the Kernel reference manual). Each parameter Phase is an atom, and the associated PhaseArgs is a list of any terms. The key start_phases is optional, but the behavior of the system is dependent on the key being defined or not (see kernel reference manual and the Starting Applications chapter below).

2.3 The Application Directory
Each application should be placed in a separate directory. This directory should be divided into several sub-directories. The following sub-directories should exist in the application directory .../ApplName:

src
ebin
priv
include

This structure is necessary because tools such as systools rely on this structure to be able to generate boot scripts and release packages. The release handling procedure on target machines also depends on this structure to be able to upgrade to new releases.

In the target environment, the application directory name contains the application version.
The following description of the sub-directories listed also indicates if the sub-directory is mandatory or optional.

src.
This directory contains the source code. This directory is mandatory in the development environment, but optional in the target environment. If source code is written in several different languages, a sub-directory with the name e_src can be created below the src directory to store the Erlang source code.

ebin.
This directory contains the Erlang object code, for example jam files. The application resource file should also be placed here. This directory is mandatory.

priv.
This directory is used for application specific files. For example, C executables should be placed here, or in a bin directory below the priv sub-directory. The application designer can determine the directory structure below priv. The function code:priv_dir/1 should be used to access the priv sub-directory. This directory is optional. It can be omitted if the application does not include any application specific files.

include.
Included files must be placed in this directory. The application exports these files for inclusion in other application modules. The include_lib("ApplName/include/File.hrl") module attribute should be used to include a file from another application. This directory is optional.

2.4 Configuring an Application
Application specific configuration parameters should be specified in the .env key in the resource file. The application can then call application:get_env(ApplName, Parameter) to retrieve the values for the configuration parameters. You can also override configuration parameters by using the system configuration file. This file is specified with the command line parameter -config FileName. The file FileName.config contains a list of configuration parameters for the applications. For example:
[{sasl, [{sasl_error_logger, tty},
         {errlog_type, error},
         {disk_space_check_interval, 10}]},
 ...
 {ApplNameN, [{Par1, Val1},
              {Par2, Val2}]}].
    
The parameters over ride can also be executed from the command line:
erl -ApplName Par1 Val1 Par2 Val2 ...
    
Each term should be an Erlang term. However, in the Unix shell, the term must be enclosed in single quotation marks. For example: '{file, "a.log"}'.

2.5 The Application Master
Each individual application is controlled by an application master.

Example of an Application
The application master is responsible for all processes running in an application. The application master can kill all processes which belong to the application.
The application master assumes that the process started by the start function is responsible for the internal details of the application. This process is assigned the following special role:

If this process terminates, then the application master assumes that the application as a whole has terminated.
If the process started first terminates with a normal exit, then the application is assumed to have terminated correctly.
If the process started first terminates with an abnormal exit, then the application is assumed to have terminated with an error.

2.6 Included Applications
An application can include other applications. Processes implemented in an included application should be placed underneath a supervisor in the including application. This way, you include applications in order to group them together.

Primary Application and Included Applications
An application which is not included by any other application is called a primary application. An application can only be included by one other application. If you want several applications to include an application, it has to be designed as a library application without a start function (the mod key in the .app file).
When an application is loaded, the application controller ensures that all included applications are also loaded. When an application is started , the application controller will only start the primary application by default (see chapter, Starting Applications).

The information in the following paragraph only applies to the Erlang System/Open Telecom Platform (OTP) environment.

When building a release, the included_applications key, which is specified in the application resource file, can be overridden with the .rel file. Therefore, all applications have the implicit environment (env) variable included_applications in order to read the current configuration. This is useful when an application has multiple start parts (included applications) depending on customer requirements, and where there is no need install or load excluded parts in the system. Refer to the section Release Structure in the System Architecture Support Libraries Manual for more details.
2.7 Distributed Applications
In a distributed system with several cooperating Erlang nodes, there is a need to control certain applications in a distributed manner. Some applications may be specified to run on one of several nodes. The application controllers on the different nodes can be arranged to monitor each other. If one node goes down, the application controller on another node notices this and restart the application on its node. These applications are called distributed applications, as opposed to local applications, which are always started on the local node. An example of a standard local application is kernel; there is always one local instance of kernel running on an Erlang node.
With this definition, a local application may be distributed in the sense that it uses services on other nodes, or cooperates with applications on other nodes. For example, the application controller sees the Mnesia DBMS started as a local application. In this description, only the control of applications is discussed.
Because a distributed application may move between nodes, some addressing mechanism is required to ensure that it can be addressed by other applications, regardless on which node it currently executes. This issue is not addressed here, but the standard Erlang modules global or pg can be used for this purpose.
2.7.1 Specifying Distributed Applications
The configuration parameter distributed for the application kernel defines which applications are distributed, and on which nodes they may run. This parameter is of the form {ApplName,[NodeDesc]} or {ApplName, Time, [NodeDesc]}, where NodeDesc = Node |{Node, ..., Node}. This data structure specifies a list of nodes where the application ApplName may execute, and the order in which these nodes should be used to start the application. If the nodes are specified in a tuple, the order is undefined. If a node crashes and Time has been specified, then the application controller will wait for Time milliseconds before attempting to restart the application on another node. If Time is not specified, it defaults to 0, and if a node goes down the application is restarted immediately on another node.
For example, suppose that the application myapp is defined to be distributed as {myapp, 5000, [cp1@cave, {cp2@cave, cp3@cave}]}. Suppose further that all nodes are up and running when myapp is started. It is then started at cp1, as shown in the figure below.

Application myapp - Situation 1
If cp1 goes down, the system checks which one of the other nodes, cp2 or cp3, has the least number of running applications, but waits for 5 seconds for cp1 to restart. If cp1 does not restart and cp2 runs fewer applications than cp3, then myapp is restarted on cp2.

Application myapp - Situation 2
Suppose now that cp2 goes down as well and does not restart within 5 seconds. myapp is now restarted on cp3.

Application myapp - Situation 3
If cp2 now restarts, it will not restart myapp, because the order between nodes cp2 and cp3 is not defined.

Application myapp - Situation 4
However, if cp1 restarts as well, the function application:takeover/2 moves myapp to cp1, because cp1 has a higher priority than cp3 for this application. In this case, Mod:start({takeover, cp3@cave}, StartArgs) is executed at cp1.

Application myapp - Situation 5
For a distributed application, the Nodes list distribution specification must be the same at each involved node. The application controllers check this when they contact each other at start-up. If a mismatch is found, the application controller, and thus the entire Erlang node, terminates with reason {distribution_mismatch, ApplName, Node}. The reason for this behavior is that the application may not start at all, it may start at several nodes, or the application controller may be left hanging if there is a mismatch in the specification
2.7.2 Starting Distributed Applications
Each node which is included in the distribution specification for a distributed application must make an application:start call for the distributed application. The start-up sequence between local and distributed applications is also synchronized.
To illustrate this point, suppose that a local application named localapp uses the distributed application myapp, described in the previous example, with the distribution specification {myapp, 5000, [cp1@cave, {cp2@cave, cp3@cave}]}. It is assured that localapp will not be started on cp1, cp2 or cp3 until cp1 starts myapp. In this way, the start of distributed applications is a synchronization point for all nodes that may run the application.
When a local application is dependent on a distributed application but is configured to run on a different node than the distributed application a synchronization problem occurs, this is solved by also starting the distributed application on the node where the local application will run. This ensures that the applications are started in the correct order on the right nodes.
Please note when the above configuration applies the local application must be loaded before the distributed application is started. Normally this is not a problem because all applications are loaded before any applications are started.
When distributed applications are specified, it is advised that the sync_nodes functionality in the kernel application is used. If not used, and several nodes are started at the same time, the node that comes up first will start all applications, and the other nodes will take over the applications when they come up.
2.8 Starting Applications
A primary application can be started in one or two steps. The first step is mandatory and the purpose of it is to start the main supervisor of the application and possible permanent children. The second step is optional and its purpose is to synchronize processes within an application.
2.8.1 Start Supervisors
The start parameters for the first step are defined in the mod key in the resource file for the primary application. The application master will evaluate the function Mod:start(Type, StartArgs).
Mod and StartArgs parameters are defined in the mod key. The parameter Type states what type of start is running, ie. normal, takeover or fail-over.
Takeover signifies that the application is distributed and is to be moved from another node to this node, due to operator invention or due to this (newly started) node is defined as superior to the other node in the configuration parameter distributed.
Fail-over signifies that this node should start the application due to a crash of the node where the application was previously running.
All other starts will result in a normal start type.
Note: Fail-over is only valid if the start_phases key is defined, otherwise this start type is denoted as normal. If older application versions without start phases are being used, it is possible to set the start_phases key to an empty list.

2.8.2 Synchronize Processes
The second step is executed only if the start_phases key is defined in the resource file of the primary application.
If a primary application has a defined key, all its included applications must also have the key defined. There is, however, a possibility to include applications without start phases by wrapping them in an another application (how to do this is explained later).
In this step the application master evaluates the function Mod:start_phase(Phase, Type, PhaseArgs) for each phase in the specified order. Mod is defined in the mod key, Phase and PhaseArgs in the start_phases key, and Type is the same as in the first step. An example showing (a part of) an application's resource file and the evaluated functions:
myApp.app
:
{mod, {myApp, StartArgs},
{included_applications, []},
{start_phases, [{init, InitArgs}, {go, GoArgs}]},
:


evaluated functions
myApp:start(Type, StartArgs)                % start the main supervisor 
                                            % and permanent children
myApp:start_phase(init, Type, InitArgs)     % start the application 
                                            % in the init phase
myApp:start_phase(go, Type, GoArgs)         % start the application 
                                            % in the go phase

      
A primary application is responsible for starting included applications. The primary application can read the included applications start phases by calling application:get_key(Application, start_phases). There is, however, a possibility to automate the start of included applications, but first an example where the synchronization is taken care by the primary application:
primApp.app
:
{mod, {primApp, PrimAppStartArgs},
{included_applications, [inclOne, inclTwo]},
{start_phases, [{init, InitArgs}, {go, GoArgs}]},
:

inclOne.app
:
{mod, {inclOne, NotUsedArgs},
{start_phases, [{go, GoArgs1}]},
:

inclTwo.app
:
{mod, {inclTwo, NotUsedArgs},
{start_phases, [{init, InitArgs2}, {go, GoArgs2}]},
:


evaluated functions
primApp:start(Type, PrimAppStartArgs)       % start the main supervisor 
                                            % and permanent children

primApp:start_phase(init, Type, InitArgs)   % start the PrimApp and InclTwo 
                                            % applications in init phase
primApp:start_phase(go, Type, GoArgs)       % start all applications
                                            % in go phase
      
To automate the synchronization of the included applications a predefined module application_starter is to be used as the Module in the mod key. application_starter takes the Mod and StartArgs as parameters: {mod, {application_starter, [{Mod, StartArgs}]}.
It is possible to have unique start phases in different applications. The order of the phases is defined in the primary applications start_phases key for the whole tree. This aserts that all included applications must have their start phases defined in the including applications' resource files.
The application_starter will execute only one start phases at a time, from left to right, searching and executing (where indicated) start phases in the included applications.
The above example is continued below using an automatic start for included applications:
primApp.app
:
{mod, {application_starter, [primApp, PrimAppStartArgs]},
{included_applications, [inclOne, inclTwo]},
{start_phases, [{init, InitArgsPrim}, {go, GoArgsPrim}]},
:

inclOne.app
:
{mod, {inclOne, NotUsedArgs},
{start_phases, [{go, GoArgs1}]},
:

inclTwo.app
:
{mod, {inclTwo, NotUsedArgs},
{start_phases, [{init, InitArgs2}, {go, GoArgs2}]},
:


evaluated functions
primApp:start(Type, PrimAppStartArgs)         % start the main supervisor 
                                              % and permanent children
primApp:start_phase(init, Type, InitArgsPrim) % start the primApp application 
                                              % in the init phase
inclTwo:start_phase(init, Type, InitArgs2)    % start the inclTwo application 
                                              % in the init phase
primApp:start_phase(go, Type, GoArgsPrim)     % start the primApp application 
                                              % in the go phase
inclOne:start_phase(go, Type, GoArgs1)        % start the inclOne application 
                                              % in the go phase
inclTwo:start_phase(go, Type, GoArgs2)        % start the inclTwo application 
                                              % in the go phase

      
For the reason mentioned above, the subsequently included applications on the same branch must have their start phases defined in all the including applications resource files (start_phases key). The application_starter will then run the start phases in a sequentially correct start order.
Note:When starting the application tree the start phase does not descend level by level but follows a branch of the tree (starting applications as it descends) before moving to the next branch.
There is an example of the recursive use of the application starter below and a diagram of the go start phase flow.

Flow of the go start phase.
primApp.app
:
{mod, {application_starter, [primApp, PrimAppStartArgs]},
{included_applications, [inclOne, inclTwoPrim]},
{start_phases, [{prim, PrimArgs}, {init, InitArgs}, {some, SomeArgs}, 
                {spec, SpecArgs}, {go, GoArgs}]},
:

inclOne.app
:
{mod, {inclOne, NotUsedArgs},
{included_applications, []},
{start_phases, [{spec, SpecArgs}, {go, GoArgsOne}]},
:

inclTwoPrim.app
:
{mod, {application_starter, [inclTwoPrim, NotUsedArgs]},
{included_applications, [incl2A, incl2B]},
{start_phases, [{init, []}, {some, []}, {go, []}]},
:

incl2A.app
:
{mod, {incl2A, []},
{included_applications, []},
{start_phases, [{some, SomeArgs2A}, {go, GoArgs2A}]},
:

incl2B.app
:
{mod, {incl2B, []},
{included_applications, []},
{start_phases, [{init, InitArgs2B}]},
:


evaluated functions
primApp:start(Type, PrimAppStartArgs)        % start the main supervisor 
                                             % and permanent children
primApp:start_phase(prim, Type, PrimArgs)     
primApp:start_phase(init, Type, InitArgs)    
inclTwoPrim:start_phase(init, Type, [])        
incl2B:start_phase(init, Type, InitArgs2B)  
primApp:start_phase(some, Type, SomeArgs)    
inclTwoPrim:start_phase(some, Type, [])        
incl2A:start_phase(some, Type, SomeArgs2A) 
primApp:start_phase(spec, Type, SpecArgs)   
inclOne:start_phase(spec, Type, SpecArgs) 
primApp:start_phase(go, Type, GoArgs)      
inclOne:start_phase(go, Type, GoArgsOne)   
inclTwoPrim:start_phase(go, Type, [])          
incl2A:start_phase(go, Type, GoArgs2A)     

      
Mixing applications with start_phases keys together with applications which have no start_phases keys is not allowed. The wrapper application's only function is to start an application which has no start_phases key. Without a wrapper the application will not start.
primApp.app
:
{mod, {primApp, PrimAppStartArgs},
{included_applications, [with, wrapper]},
{start_phases, [{init, InitArgs}, {go, GoArgs}]},
:

with.app
:
{mod, {with, NotUsedArgs},
{included_applications, []},
{start_phases, [{init, InitArgsW}, {go, GoArgsW}]},
:

wrapper.app
:
{mod, {wrapper, NotUsedArgs},
{included_applications, []},
{start_phases, [{init, WrapArgs}]},
:


evaluated functions
primApp:start(Type, PrimAppStartArgs)       % start the main supervisor 
                                            % and permanent children
primApp:start_phase(init, Type, InitArgs)   % start the primary application 
                                            % in the init phase
with:start_phase(init, Type, InitArgsW)     % start the with application 
                                            % in init phase
wrapper:start_phase(init, Type, WrapArgs)   % start the without  
                                            % application 
primApp:start_phase(go, Type, GoArgs)       % start the primary application 
                                            % in the go phase
with:start_phase(go, Type, GoArgsW)         % start the with application 
                                            % in go phase
      
2.9 An Example
The following example assumes some familiarity with the Erlang boot script The example shows a simple boot script and config file for nodes cp1, cp2, cp3 and cp4, and applications myapp and localappl. These are the nodes and applications which are described in the previous section, with the exception that node cp4 has been added. This node only executes the local application localapp.
2.9.1 Boot Script
The following boot script is used at all four nodes. Each node makes an application:start call for myapp, although this distributed application is started only at one node.
{script,{"Dist Test","1.0"},
 [{preLoaded, [init,erl_prim_loader]},
  {progress, preloaded},
  <...>
  {progress,init_kernel_started},
  {apply,{application,load,
          [{application,
            myapp,
            [{description,"MYAPP"},
             {vsn,"1.0"},
             {modules, []},
             {applications,[kernel, stdlib]},
             {env,[]}]}]}},
  {apply,{application,load,
          [{application,
            localapp,
            [{description,"LOCALAPP"},
             {vsn,"1.0"},
             {modules, []},
             {applications,[kernel, stdlib, myapp]},
             {env,[]}]}]}},
  {progress,applications_loaded},
  {apply,{application,start,[kernel,permanent]}},
  {apply,{application,start,[stdlib,permanent]}},
  {apply,{application,start,[myapp,permanent]}},
  {apply,{application,start,[localapp,permanent]}},
  {progress,started}]}.
2.9.2 Config File
The following configuration file is used at all four nodes.
   [{kernel, [{sync_nodes_optional, [cp1@cave, cp2@cave, cp3@cave, cp4@cave]},
        {sync_nodes_timeout, 10000},
        {distributed, [{myapp, [cp1@cave, {cp2@cave, cp3@cave}]}]}]}].
      
The application controller notices that cp4 cannot start myapp and then ensures that cp4 cannot start localapp until another node has started myapp.