Knowledge Base on camlcity.org

Knowledge Base on camlcity.org http://knowledge.camlcity.org en Technical details about O'Caml KB 003: Disable setuid-root on Linux http://www.camlcity.org/knowledge/kb_003_disable_setuid.html http://www.camlcity.org/knowledge/kb_003_disable_setuid.html <div> <b>What I learned when programming the GODI autobuilder</b><br/>  </div> <div> In Linux books and the usual documentation you can read about two ways of securing chroot environments: by clearing the setuid-root bits file by file, or by disabling the setuid mechanism in the mount flags. Actually, there is a third way since kernel 2.6.26. </div> <div> Recently I secured the sandbox environment running the GODI autobuilder. It compiles and possibly runs untrusted code coming from the GODI repository. One open security hole was that this chroot environment did not systematically disable setuid. As you might know, it is possible on all Unix OS to set a flag for an executable so that this program runs as a different user (setuid bit). If this other user is root, the executable gains system privileges, and this can be potentially abused to run user code with such privileges. Well, not every setuid program is a problem, but it might be (generally or in special circumstances), and it is good practice to turn off all setuid execution in secured chroot environments. <p> My chroot directories are partially bind-mounted, and thus I could not easily disable setuid in the mount flags. But there is another way: Just clear the capability bounding set (bset), and you are done. </p><h2>Capabilities</h2> <p> What's that? Well, the root user gets under Linux its special privileges because it has <i>capabilities</i>. Every capability corresponds to a special right, e.g. root can run "chown" with arbitrary arguments because it has the CAP_CHOWN bit set. A non-root user has no capabilities. So our goal is to prevent that root (or any other user) can get capablities in the secured environment. When an executable is run setuid-root the kernel just adds the capabilities of root to the process (besides changing the uid to 0). </p><p> The trick is now that one can restrict the capabilities a process (and all its subprocesses) can ever get. There is a mask for capabilities called the bounding set. This mask controls which capabilities an executable can additionally get from the setuid bit (or from a file capability), and by emptying this set, this feature is effectively turned off. This mask does neither have an effect on the capabilities a process already has nor on the capabilities the process inherits to children. It just controls what can be gained by setting file attributes. </p><p> Unfortunately, there is no command to do this. We need to write a little C program <code>dropbset.c</code>: </p><pre> #include <unistd.h> #include <stdio.h> #include <stdlib.h> #include <sys/prctl.h> #include <linux/prctl.h> #include <linux/capability.h> #include <errno.h> int main (int argc, char *argv[]) { int code; unsigned long cap; for (cap=0; cap <= 63; cap++) { code = prctl(PR_CAPBSET_DROP, cap, 0, 0, 0); if (code == -1 && errno != EINVAL) { perror("prctl error"); exit(1); } } code = execv("/bin/bash", argv); if (code == -1) { perror("exec error"); exit(1); } exit(0); // hmmmm... } </pre> The program can be simply compiled with <pre> gcc -o dropbset dropbset.c </pre> Before running it, we need to ensure it gets the required privileges to do the PR_CAPBSET_DROP operation. As root, do <pre> setcap CAP_SETPCAP=pe dropbset </pre> This command sets a <i>file capability</i> - a mechanism very much like the setuid bit, but restricted to a single capability only. Here, we need CAP_SETPCAP in order to run the operation. If you did not set this file capability, you could run dropbset only as root. (NB: The file capability is stored together with the file in the extended attributes.) <p> Now let's try it (remember dropbset just execs bash, and hence is called in exactly the same way): As non-root do: </p><pre> ./dropbset -c "ping 127.0.0.1" </pre> You get an error "Operation not permitted". ping is normally setuid-installed so that unprivileged users get raw access to the network for pinging. <p> Once the bset is empty, there is no way a child process can add any capability to its effective set because of file attributes. <b>Not even root can do this:</b> </p><pre> # id uid=0(root) gid=0(root) groups=0(root) # dropbset -c 'ping 127.0.0.1' ping: icmp open socket: Operation not permitted </pre> </div> <div> Gerd Stolpmann works as O'Caml consultant </div> <div> </div> KB 002: OCaml Programs As Shared Libraries http://www.camlcity.org/knowledge/kb_002_shared_library.html http://www.camlcity.org/knowledge/kb_002_shared_library.html <div> <b>Compile programs into .so files</b><br/>  </div> <div> Since quite some time the OCaml code generator supports the creation of position-independent code (PIC). This is e.g. required to load OCaml modules dynamically. Another use is to output OCaml programs as shared libraries. </div> <div> <p> Let's assume we have this little program: </p><pre style="font-size:smaller"> prog.ml: print_endline "Hello World, OCaml";; </pre> The main program in C: <pre style="font-size:smaller"> main.c: #include <stddef.h> #include <stdio.h> #include <caml> int main(int argc, char *argv[]) { printf("Hello World, C\n"); caml_startup(argv); return 0; } </caml></stdio.h></stddef.h></pre> The idea here is that the main program is written in C, and initializes from there the OCaml program. <p>The following applies to OCaml-3.11 and higher. </p><h2>Statically-linked version as documented in the manual</h2> Let's first look at what you can also find in the manual: <p>Here, we create a single prog_impl.o file from prog.ml, and this object contains the whole code of the OCaml program, but omits the OCaml runtime: </p><pre style="font-size:smaller"> $ ocamlopt -c prog.ml $ ocamlopt -output-obj -o prog_impl.o prog.cmx </pre> For compiling the C program we use <code>ocamlc</code> as driver because it adds all required options when invoking the C compiler (alternatively you can grab the options from "ocamlc -config", variable <code>native_c_compiler</code>): <pre style="font-size:smaller"> $ ocamlc -c main.c </pre> Linking everything together: <pre style="font-size:smaller"> $ cc -o main main.o prog_impl.o -L`ocamlc -where` -lasmrun -lm -ldl </pre> <code>libasmrun.a</code> is available in the OCaml standard library directory, and we have to add this directory to the search path with <code>-L`ocamlc -where`</code>. <p><b>Creating libprog.a:</b> A possible simplification for the user is to put the whole OCaml code, including the runtime, into a static archive: </p><pre style="font-size:smaller"> $ rm -rf libprog.a $ ar rc libprog.a prog_impl.o $ asmrun_members=$(ar t `ocamlc -where`/libasmrun.a | grep -v '__.*') $ mkdir -p tmp_asmrun $ ( cd tmp_asmrun && ar x `ocamlc -where`/libasmrun.a ) $ for mem in $asmrun_members; do ar rc libprog.a tmp_asmrun/$mem; done $ ranlib libprog.a $ rm -rf tmp_asmrun </pre> The link command is then: <pre style="font-size:smaller"> $ cc -o main main.o -L. -lprog -lm -ldl </pre> Note that we need to add the contents of libasmrun.a to libprog.a. If we do not do this, and try to add -lasmrun instead, we run into a problem with circular references between static libraries (which is not supported by the linker). <p>From where did we get "-lm -ldl"? If you want to be 100% correct, take this list from the variable <code>native_c_libraries</code> printed by "ocamlc -config". </p><h2>Dynamically-linked version</h2> Unfortunately, this depends on your platform/CPU: <ul> <li>x86-32: A standard OCaml installation works. There is a downside, though: OCaml cannot create PIC for x86-32. This is not a blocker on x86-32, because the dynamic loader of the OS can relocate on the fly. Nevertheless, there is a price to pay, namely that the text segment cannot be put into sharable memory, i.e. the RAM consumption is higher. </li><li>x86-64: You need to build OCaml so that libasmrun.a is compiled as PIC - see below how to do it. Code produced by ocamlopt is normally PIC anyway unless you give -fno-PIC on the command-line. Just don't do this, and the generated code will be right. </li><li>ARM (since OCaml-4.00): Basically it can be made working, but there is a difficulty: The code generator for ARM disables PIC by default. Especially the standard library is not PIC, and any other 3rd-party code isn't, too. The simplest way to fix this is to change the file <code>asmcomp/arm/arch.ml</code> in the OCaml sources so that the variable <code>pic_code</code> is initialized with <code>true</code>. In addition to this, you need to compile libasmrun.a with PIC as for x86-64. </li><li>Other CPU's are not supported.</li> </ul> <p> <b>How to enable PIC for libasmrun.a:</b> When building OCaml, you need to configure it so that PIC is enabled: </p><ul> <li>Linux (and probably other platforms using the GNU toolchain): <pre style="font-size:smaller"> ./configure -cc "gcc -fPIC" -aspp "gcc -c -fPIC" </pre> </li><li>MacOS X: same </li><li>Other platforms: Unknown </li></ul> If you are using GODI, you can add configure options in godi.conf: <pre style="font-size:smaller"> OCAML_CONF_ARGS=-cc "gcc -fPIC" -aspp "gcc -c -fPIC" </pre> Note that you need to rebuild both godi-ocaml-src and godi-ocaml. <p> <b>How to create libasmrun.so:</b> Now do </p><pre style="font-size:smaller"> $ ocamlopt -output-obj -o libprog.so prog.cmx $ cc -o main main.o -Wl,-rpath,. -L. -lprog </pre> Test it (Linux - on OS X use "otool -L"): <pre style="font-size:smaller"> $ ldd ./main </pre> This should print libprog.so as dependent library. <p>Note that it is not necessary in this case to add "-lm -ldl" to the link command, as this has already been recorded in libprog.so. </p> </div> <div> Gerd Stolpmann works as O'Caml consultant </div> <div> </div> KB 001: OMake Recipes http://www.camlcity.org/knowledge/kb_001_omake_recipes.html http://www.camlcity.org/knowledge/kb_001_omake_recipes.html <div> <b>How to use the OMake build utility best</b><br/>  </div> <div> The <a href="http://omake.metaprl.org">OMake utility</a> is easy to use for simple projects: After "omake --install" it drops a template OMakefile into the current directory, and you simply need to fill it out to get started. But what if you need more than what is provided in the template? Here, I've collected a number of recipes for advanced configurations. </div> <div> <p> <b>Available recipes:</b> </p><ul> <li><a href="#multi-programs">Building multiple programs or libraries</a> </li><li><a href="#multi-dirs">Building in multiple directories</a> </li><li><a href="#camlp4">How to enable camlp4</a> </li><li><a href="#deviating-flags">Sections with deviating compiler flags</a> </li><li><a href="#deps">Handling dependencies between project libraries</a> </li><li><a href="#many-deps">Many dependencies between project libraries</a> </li><li><a href="#ocamldoc">How to create ocamldoc documentation</a> </li><li><a href="#cmdline">How to interpret command-line parameters</a> </li><li><a href="#objs-in-subdirs">Putting the build objects into subdirectories</a> </li><li><a href="#profile">Profile builds</a> </li><li><a href="#generate">Generating files</a> </li><li><a href="#stublibs">Compiling stub libraries</a> </li><li><a href="#dot-prob">The ":" problem</a> </li><li><a href="#clean">Cleaning completely</a> </li><li><a href="#expr-or-stmt">Expressions or statements?</a> </li></ul> <p> <b>Starting point:</b> A simple OMakefile for a program could look like: </p><pre style="font-size:smaller"> USE_OCAMLFIND = true OCAMLPACKS = lwt FILES[] = foo bar .DEFAULT: $(OCamlProgram myprog, $(FILES)) </pre> For a library, just use <pre style="font-size:smaller"> .DEFAULT: $(OCamlLibrary mylib, $(FILES)) </pre> instead of the <code>OCamlProgram</code> line. The following recipes are, in some sense, variations of these simple patterns. <p> Normally, you just only set variables to control the build. Some of the variables only affect the linker, some only the compiler, and a number both (here the important ones): </p><style type="text/css"> .omakevars { font-size: smaller } .omakevars TH { text-align: left } .var { font-family: monospace } </style> <p> </p><table class="omakevars"> <tr><th>Variable</th> <th>Compiler?</th> <th>Linker?</th> <th>Comment</th> </tr> <tr><td class="var">USE_OCAMLFIND</td> <td>X</td> <td>X</td> <td>true/false - whether to use findlib</td></tr> <tr><td class="var">OCAMLINCLUDES</td> <td>X</td> <td>X</td> <td>search path for compiled modules and archives, also used by the dependency scanner</td></tr> <tr><td class="var">OCAMLFINDFLAGS</td> <td>X</td> <td>X</td> <td>additional flags to pass to ocamlfind</td></tr> <tr><td class="var">OCAMLPACKS</td> <td>X</td> <td>X</td> <td>findlib packages to include in build</td></tr> <tr><td class="var">BYTE_ENABLED</td> <td>X</td> <td>X</td> <td>true/false - whether bytecode compiler is enabled</td></tr> <tr><td class="var">NATIVE_ENABLED</td> <td>X</td> <td>X</td> <td>true/false - whether nativecode compiler is enabled</td></tr> <tr><td class="var">OCAMLCFLAGS</td> <td>X</td> <td>X</td> <td>flags to pass to "ocamlc"</td></tr> <tr><td class="var">OCAMLOPTFLAGS</td> <td>X</td> <td>X</td> <td>flags to pass to "ocamlopt"</td></tr> <tr><td class="var">OCAMLFLAGS</td> <td>X</td> <td>X</td> <td>flags to pass to "ocamlc" and "ocamlopt"</td></tr> <tr><td class="var">OCAML_BYTE_LINK_FLAGS</td> <td></td> <td>X</td> <td>flags to pass to "ocamlc" when linking an executable</td></tr> <tr><td class="var">OCAML_NATIVE_LINK_FLAGS</td> <td></td> <td>X</td> <td>flags to pass to "ocamlopt" when linking an executable</td></tr> <tr><td class="var">OCAML_LINK_FLAGS</td> <td></td> <td>X</td> <td>flags to pass to both "ocamlc" and "ocamlopt" when linking executables</td></tr> <tr><td class="var">OCAML_LIBS</td> <td></td> <td>X</td> <td>library archives of the project to link into executables</td></tr> <tr><td class="var">OCAML_OTHER_LIBS</td> <td></td> <td>X</td> <td>library archives outside the project to link into executables</td></tr> <tr><td class="var">OCAML_LIB_FLAGS</td> <td></td> <td>X</td> <td>extra flags when linking libraries</td></tr> </table> <hr/> <a name="multi-programs"></a> <h2>Building multiple programs or libraries</h2> OMake allows it to split an OMakefile into sections. Each section can define its own set of variables, and its own build rules. A section defines a scope for variables: a section sees the variables of the surrounding section, but modifications to variables are local to the section, and not automatically propagated to the surrounding section (unless you "export" definitions - not covered here). <p>For example, this OMakefile builds two programs from a different set of files, and links different libraries into the programs: </p><pre style="font-size:smaller"> USE_OCAMLFIND = true OCAMLPACKS = lwt section FILES[] = foo bar .DEFAULT: $(OCamlProgram myprog1, $(FILES)) section FILES[] = bar baz .DEFAULT: $(OCamlProgram myprog2, $(FILES)) </pre> Basically, it makes sense to set linking-related variables that are interpreted by <code>OCamlProgram</code> and <code>OCamlLibrary</code>, like <code>OCAML_LIBS</code> (see "Starting point" for a list of variables interpreted at link time). <p> What does not work here is to set variables that (only or also) affect the compiler (i.e. "ocamlc -c" and "ocamlopt -c"). If you e.g. set <code>OCAMLINCLUDES</code> or <code>OCAMLPACKS</code> within a section, there will be no effect on compilation (only on linking). Below you find another recipe that shows how to do that, but this simple recipe here does not allow it. </p><hr/> <a name="multi-dirs"></a> <h2>Building in multiple directories</h2> Given you have a directory hierarchy <pre> root +--- dir1 +--- dir2 </pre> and you want to build in all three directories. How can we describe this? <p><b>Solution 1: Multiple OMakefiles</b> </p><p>Here, only root contains OMakeroot, and all three directories have their own OMakefile. Also, you need to "chain" the files: In root/OMakefile, there needs to be a <code>.SUBDIRS</code> directive, like: </p><pre style="font-size:smaller"> # Before .SUBDIRS: Put here definitions that are visible in the whole project: USE_OCAMLFIND = true OCAMLPACKS = lwt .SUBDIRS: dir1 dir2 # After .SUBDIRS: Add here definitions that are only applied in the root dir: FILES[] = mainprog OCAMLINCLUDES = dir1 dir2 OCAML_LIBS = dir1/lib1 dir2/lib2 .DEFAULT: $(OCamlProgram myprog, $(FILES)) </pre> Now, put into dir1/OMakefile and dir2/OMakefile the definitions that are local to these directories, e.g. <pre style="font-size:smaller"> FILES[] = file1 files OCAMLFLAGS = -g OCAMLPACKS += netstring .DEFAULT: $(OCamlLibrary lib1, $(FILES)) </pre> Basically, the definitions in the sub-OMakefiles have their own scoping environment, as if they were encapsulated in a <code>section</code>. The scoping rules for directories and for sections are not identical, though. In particular, you can also define in a sub-OMakefile variables that affect the compilation, like <code>OCAMLFLAGS</code>, <code>OCAMLPACKS</code>, or <code>OCAMLINCLUDES</code>. <p><b>Solution 2: .SUBDIRS with body</b> </p><p>Sometimes, you don't want to put OMakefiles into subdirectories, or the OMakefiles in several directories would be identical. To handle this case, it is allowed to put the definitions for the subdirectory into the body of .SUBDIRS (i.e. indented): </p><pre style="font-size:smaller"> # Before .SUBDIRS: Put here definitions that are visible in the whole project: USE_OCAMLFIND = true OCAMLPACKS = lwt .SUBDIRS: dir1 dir2 .DEFAULT: $(OCamlProgram myprog, main) </pre> This works exactly as if you had written the definitions in the body into separate OMakefiles in the subdirs. <hr/> <a name="camlp4"></a> <h2>How to enable camlp4</h2> If you use ocamlfind, this is very simple, just set <pre style="font-size:smaller"> OCAMLFINDFLAGS += -syntax camlp4o OCAMLPACKS += camlp4 </pre> (or use "-syntax camlp4r" if you prefer the revised syntax). In order to activate a custom preprocessor written for camlp4, like xstrp4, just add this findlib package to <code>OCAMLPACKS</code> <pre style="font-size:smaller"> OCAMLPACKS += xstrp4 </pre> Note that the variable <code>OCAMLFINDFLAGS</code> not only affects ocamlc and ocamlopt, but also ocamldep, i.e. the dependency scanning is also done with camlp4 and the activated syntax extensions. <p>Some custom preprocessors accept command-line arguments. For example, you can define macros on the command-line with the macro preprocessor: </p><pre style="font-size:smaller"> OCAMLPACKS += camlp4.macro # the macro preprocessor OCAMLFINDFLAGS += $(mapprefix -ppopt, -D NAME=value) </pre> The strange "mapprefix" expression just prepends <code>-ppopt</code> to every space-separated part of the second argument - well, in the end, camlp4 is called with <code>-D NAME=value</code> appended to the command-line. <p> If you don't trust in ocamlfind, you can also directly set the preprocessor command, e.g. </p><pre style="font-size:smaller"> OCAMLFLAGS += -pp camlp4of </pre> but custom extensions like <code>xstrp4</code> can then not be enabled by simply adding the package to <code>OCAMLPACKS</code>. Instead, you would have to find out the right options for camlp4of, and add them to the command passed with -pp. <p> As camlp4 reduces the compilation speed tremendously, you often want to enable it only for a few files, but not for the whole project. See the following recipe how to do this. </p><hr/> <a name="deviating-flags"></a> <h2>Sections with deviating compiler flags</h2> As already explained above, the <code>section</code> construct normally does not allow to modify compiler flags, but only linker flags. You may for example want to do this to invoke a preprocessor only for certain source files. <p> Note that you do not need this technique if you just set the compiler flags in a sub-OMakefile to different values than in the main OMakefile, because OMake already handles this case automatically. </p><p> The compiler flags have an effect on the standard build rules, which are defined by pattern rules like </p><pre style="font-size:smaller"> %.cmo: %.ml $(OCamlC) -c $< </pre> (the actual standard definitions are a lot more complicated), where <code>OCamlC</code> is a variable that also expands the compiler flags like <code>OCAMLINCLUDES</code>. Normally, the build rules just take the values these flags have at the end of the OMakefile. If you now want to force OMake to take the values set in a certain <code>section</code>, the trick is to recall the build rule within the section. Here is an example: <pre style="font-size:smaller"> section OCAMLINCLUDES = additional_include_dir foo.cmi: foo.cmo: foo.cmx: foo.o: </pre> This means that the module <code>foo</code> is compiled with the changed compiler flags. Note that we omit the bodies of the build rules - which OMake interprets to leave the body unchanged (i.e. take it from the pattern), and only to apply the current set of variables to the old body. <p> One application is to instruct OMake to compile certain modules with a preprocessor. The following defines a function doing this: </p><pre style="font-size:smaller"> Camlp4o(module) = section OCAMLPACKS += camlp4 OCAMLFINDFLAGS += -syntax camlp4o $(module).cmi: $(module).cmo: $(module).o: $(module).cmx: </pre> You can call this function for every module as in <pre style="font-size:smaller"> Camlp4o(mod1) Camlp4o(mod2) </pre> to enable camlp4 just for these modules. <p>Note that you need to name the modules explicitly that get a deviating set of compiler flags. You can, however, use an iteration to apply the deviating set to all modules you deal with in a section: </p><pre style="font-size:smaller"> section FILES[] = foo bar baz CAMLP4_FILES[] = bar baz OCAML_LIBS[] = mylib1 mylib2 foreach(f, $(CAMLP4_FILES)) Camlp4o($(f)) .DEFAULT: $(OCamlProgram myprog1, $(FILES)) </pre> <p><b>Not limited to camlp4:</b> This is, of course, not limited to requiring camlp4. Another example it to increase the inlining value just for a certain file: </p><pre style="font-size:smaller"> Inline(module,limit) = section OCAMLOPTFLAGS += -inline $(limit) $(module).cmi: $(module).cmo: $(module).o: $(module).cmx: Inline(mod1, 10) Inline(mod2, 15) </pre> <hr/> <a name="deps"></a> <h2>Handling dependencies between project libraries</h2> Imagine you build a number of libraries, either in separate OMakefiles or in separate sections. For example, let's assume this directory hierarchy: <pre style="font-size:smaller"> root +--- directory1 # containing lib1 +--- directory2 # containing lib2 </pre> It is very simple to just use a library while building another (or building a program). E.g. if you need lib1 to build lib2, just set in <code>directory2/OMakefile</code> <pre style="font-size:smaller"> OCAMLINCLUDES[] = ../directory1 </pre> to make the library available. <code>OCAMLINCLUDES</code> is a far-reaching variable: the directories enumerated in it are not only added as <code>-I</code> flags in the compiler commands to run, but also considered for dependency scanning. That means when a source file in directory1 is changed, the modules that depend on it in directory2 are automatically found out, and added to the list of things to recompile. <p>If you build a program using the libraries, you have to do a tiny additional step: In the OMakefile building the program also set <code>OCAML_LIBS</code> in addition to <code>OCAMLINCLUDES</code>. <code>OCAML_LIBS</code> needs to be set to the paths to the library archives (i.e. to the cma/cmxa files, but omit the suffix). For example, if there are programs to build in <code>root/OMakefile</code>, set </p><pre style="font-size:smaller"> OCAMLINCLUDES[] = directory1 directory2 OCAML_LIBS[] = directory1/archive1 directory2/archive2 </pre> to link these archives while building the programs. <hr/> <a name="many-deps"></a> <h2>Many dependencies between project libraries</h2> <p> OMake will automatically take care of the dependencies between the libraries, and even between individual files. However, there is a disadvantage of doing only this: OMake does not provide a means of defining indirect dependencies well. For example, if you have <code>lib1</code> using <code>lib2</code>, and the latter uses <code>lib3</code>, you need to list <code>lib3</code> as a prerequisite for <code>lib1</code>, even if there is no direct relationship. For large projects with dozens of libraries this turns out to be a big problem, because such "expanded" dependencies are unmaintainable. </p><p><b>Solution 1:</b> Define the dependencies only in the main OMakefile. </p><p>The idea is that there is a paragraph at the beginning of the main OMakefile describing the dependencies, e.g. if we have a directory hierarchy </p><pre style="font-size:smaller"> root +--- directory1 # contains lib1 +--- directory2 # contains lib2 +--- directory3 # contains lib3 +--- directory4 # build something else </pre> we can define a number of variables in <code>root/OMakefile</code> as <pre style="font-size:smaller"> LIB3_INCLUDES[] = $(dir directory3) LIB2_INCLUDES[] = $(dir directory2) $(LIB3_INCLUDES) LIB1_INCLUDES[] = $(dir directory1) $(LIB2_INCLUDES) </pre> and in the sub-OMakefile for a certain library we set only <code>OCAMLINCLUDES</code> to one of the prepared variables, e.g. if lib4 needs lib1, we set in <code>directory4/OMakefile</code> for lib4: <pre style="font-size:smaller"> OCAMLINCLUDES = $(LIB1_INCLUDES) </pre> <p>Why did we call the function <code>dir</code> while setting the <code>LIB*_INCLUDES</code> variables? Remember that we use the variable <code>LIB1_INCLUDES</code> in a subdirectory, so e.g. <code>directory2</code> is here actually <code>../directory2</code>. If we invoke <code>dir</code>, the variable magically rembers where it was defined, and if it is recalled from a different directory, OMake automatically adjusts the path. </p><p><small>(Detail: You might have noticed that <code>LIB1_INCLUDES</code> is actually an array of an array. This does not harm us here, because when OMake expands such structured variables to command-line arguments, it just flattens them to a list of arguments.)</small> </p><p>For building programs, we also need the library archives. We can use the same pattern as for the include path: </p><pre style="font-size:smaller"> LIB3_ARCHIVES[] = $(file directory3/archive3) LIB2_ARCHIVES[] = $(file directory2/archive2) $(LIB3_ARCHIVES) LIB1_ARCHIVES[] = $(file directory1/archive1) $(LIB2_ARCHIVES) </pre> (Note that we need here to call <code>file</code> instead of <code>dir</code>). <p>For creating an executable linking in lib1 and all predecessors, just use </p><pre style="font-size:smaller"> OCAMLINCLUDES = $(LIB1_INCLUDES) OCAML_LIBS = $(LIB1_ARCHIVES) </pre> If the library archives have all the same name, you can also do: <pre style="font-size:smaller"> OCAMLINCLUDES = $(LIB1_INCLUDES) OCAML_LIBS = $(addsuffix /archivename, $(LIB1_INCLUDES)) </pre> If the archives have the same name as the directories: <pre style="font-size:smaller"> mapname(name) = value $(name)/$(basename $(name)) OCAMLINCLUDES = $(LIB1_INCLUDES) OCAML_LIBS = $(foreach $(mapname), $(LIB1_INCLUDES)) </pre> <p><b>Solution 2 (advanced):</b> Extract the dependency data from META files. </p><p>Here we assume that we have already META files in the subdirectories. We want to use ocamlfind to look up paths and archive names: </p><pre style="font-size:smaller"> root +--- directory1 # contains lib1 | +---- META +--- directory2 # contains lib2 | +---- META +--- directory3 # contains lib3 | +---- META +--- directory4 # contains lib4 +---- META </pre> The first thing to do is to set <code>OCAMLPATH</code> so it contains the <code>root</code> directory: Into <code>root/OMakefile</code> put: <pre style="font-size:smaller"> setenv(OCAMLPATH, $(absname .):$(getenv OCAMLPATH,$(string))) </pre> <p><small>(Details: <code>absname</code> returns the absolute path name of the argument. <code>$(string)</code> is just the empty string. In OMake there is no good way of writing the empty string as a quoted literal.)</small> </p><p>Now we can refer to the libraries as packages. The package names are just the directory names, e.g. "directory3" would mean lib3. It is, however, not sufficient to just add these names to <code>OCAMLPACKS</code>, because OMake would then not consider these libraries as part of the project, but as packages outside the project. </p><p>The solution is to still set <code>OCAMLINCLUDES</code> in the sub-OMakefiles like in solution 1. This time, however, we call ocamlfind for getting the values. To do so, define in <code>root/OMakefile</code> before the <code>.SUBDIRS</code> line </p><pre style="font-size:smaller"> all_dirs = $(subdirs .) get_project_findlib_dirs(pkg) = dirs = $(shella ocamlfind query -r $(pkg)) fdirs = foreach (d, $(dirs)) if $(mem $(dir $(d)), $(all_dirs)) fdirs += $(d) export export value $(fdirs) get_project_findlib_archives(pkg,pred) = dirs = $(shella ocamlfind query -format %d/%a -predicates $(pred) -r $(pkg)) fdirs = foreach (d, $(dirs)) if $(mem $(dir $(d)), $(all_dirs)) fdirs += $(d) export export value $(fdirs) </pre> Note that these functions only output directories and archives inside the project. <p><small>(Detail: Note that the <code>mem</code> function is used to check whether a directory <code>d</code> occurs in a list of directories <code>all_dirs</code>. This works independently of the path by which the directories are referenced. Actually, <code>d</code> is normally a relative path, and <code>all_dirs</code> contains absolute paths. Cool, isn't it.)</small> </p><p>How to use this: In the sub-OMakefiles that create libraries: </p><pre style="font-size:smaller"> OCAMLPACKS += directory1 directory2 OCAMLINCLUDES = $(get_project_findlib_dirs $(OCAMLPACKS)) </pre> <p>With the <code>OCAMLPACKS</code> line the project-local findlib packages are added to the build. The <code>OCAMLINCLUDES</code> setting is only necessary to instruct the scanner that the dependencies to these packages are also needed. </p><p>In the sub-OMakefiles that create programs (here just <code>program</code>): </p><pre style="font-size:smaller"> OCAMLPACKS += directory1 directory2 program.run: $(get_project_findlib_archives $(OCAMLPACKS), byte) program.opt: $(get_project_findlib_archives $(OCAMLPACKS), native) </pre> You need the two "program.run"/"program.opt" lines for every program you create. The effect is that the library archives of the project-local packages are added as dependency to the build. <hr/> <a name="ocamldoc"></a> <h2>How to create ocamldoc documentation</h2> The following function works well - all interfaces with mli suffix are assumed to contain docs: <pre style="font-size:smaller"> public.InterfaceDoc(name, files) = protected.mlifiles = $(filter-exists $(addsuffix .mli, $(files))) protected.cmifiles = $(addsuffix .cmi, $(removesuffix $(mlifiles))) $(name).idoc: $(mlifiles) $(cmifiles) /.PHONY/OCamlGeneratedFilesTarget ocamlfind ocamldoc -dump $(name).idoc -stars \ $(PREFIXED_OCAMLINCLUDES) -package "$(OCAMLPACKS)" \ $(mlifiles) return $(name).idoc </pre> You call this function like this in every OMakefile compiling <code>$(FILES)</code>: <pre style="font-size:smaller"> InterfaceDoc(filename, $(FILES)) </pre> This creates a file <code>filename.idoc</code> (idoc for interface documentation) in the current directory. This file is a binary description of the documentation, and it can be used in a final ocamldoc step (modify this as needed): <pre style="font-size:smaller"> IDOCS[] = subdirectory1/filename1.idoc subdirectory2/filename2.idoc .PHONY: html-doc html-doc: $(IDOCS) mkdir -p html rm -rf html/* ocamlfind ocamldoc -d html -stars -t "My title" -html \ $(mapprefix -load, $(IDOCS)) </pre> <hr/> <a name="cmdline"></a> <h2>How to interpret command-line parameters</h2> If you try to modify a variable by setting it on the command-line, this often does not work. For example, if you try to override <code>BYTE_ENABLED</code> with <pre style="font-size:smaller"> $ omake BYTE_ENABLED=true </pre> this setting is often ignored. The reason for this is that omake interprets command-line assignments just as initial assignments, but not as overriding assignments like traditional "make". If your OMakefile contains already something like <pre style="font-size:smaller"> BYTE_ENABLED = $(not $(OCAMLOPT_EXISTS)) </pre> this will simply overwrite the initial setting given on the command-line. <p>Note that the standard build definitions, as in e.g. <code>OCaml.om</code> (part of the OMake deployment) also set many control variables. These settings are, however, overridable. Look at OMakeroot to see why: </p><pre style="font-size:smaller"> open build/C open build/OCaml open build/LaTeX DefineCommandVars() .SUBDIRS: . </pre> In English, this just means that the three standard build definition files are loaded, and first after that the command-line assignments are evaluated. Finally, the <code>.SUBDIRS: .</code> directive includes the OMakefile in the same directory. <p>So, what to do? If you have a block at the beginning of your OMakefile where you customize variables, like </p><pre style="font-size:smaller"> BYTE_ENABLED = true NATIVE_ENABLED = $(OCAMLOPT_EXISTS) OCAMLPACKS = p1 p2 p3 </pre> you just call <code>DefineCommandVars</code> again after this block: <pre style="font-size:smaller"> DefineCommandVars() </pre> This evaluates the assignments given on the command-line again, and the caller can override your customizations. <p>There is also the possibility to set variables only if they aren't already set. Unfortunately, OMake does not know the popular "make" syntax "variable ?= value", and we have to do it on our own: </p><pre style="font-size:smaller"> if $(not $(defined X)) X = default_value export </pre> There are other formulations: <pre style="font-size:smaller"> var_default(name, value) = if $(not $(defined $(name))) setvar($(name), $(value)) export export var_default(X, default_value) </pre> Another option, closer to a normal assignment: <pre style="font-size:smaller"> var_default(value, name) = # different order! if $(not $(defined $(name))) setvar($(name), $(value)) export export var_default(X) default_value </pre> <hr/> <a name="objs-in-subdirs"></a> <h2>Putting the build objects into subdirectories</h2> Normally, OMake puts the output files of a build into the same directories as the source files. It is possible to change this. Assume the directory hierarchy is <pre style="font-size:smaller"> root +---- directory1_src +---- directory1_out +---- directory2_src +---- directory2_out </pre> and we want that the output files of the sources in <code>*_src</code> appear in <code>*_out</code>. This is achieved by changing the <code>.SUBDIRS</code> line in <code>root/OMakefile</code> into <pre style="font-size:smaller"> vmount(-l, directory1_src, directory1_out) # "minus ell" as first arg vmount(-l, directory2_src, directory2_out) .SUBDIRS: directory1_out directory2_out </pre> The effect of <code>vmount</code> is that the files in the source directories are automatically linked into the output directories. After that, you do not refer to the source directories anymore, but just to the output directories. <p><b>Building several versions of the same:</b> You can use the vmount feature to build several output directories from the same source directory, and have different configurations for each output. Example: </p><pre style="font-size:smaller"> section vmount(-l, directory1_src, directory1_normal) ENABLE_DEBUG = false .SUBDIRS: directory1_normal section vmount(-l, directory1_src, directory1_debug) ENABLE_DEBUG = true .SUBDIRS: directory1_debug </pre> Of course, both versions share the same OMakefile (residing in <code>directory1_src</code>). Because of this, we set one variable <code>ENABLE_DEBUG</code> differently in both versions. You can now use conditionals like <pre style="font-size:smaller"> if $(ENABLE_DEBUG) OCAMLFINDFLAGS += $(mapprefix -ppopt, -D DEBUG) # set a camlp4 macro for IFDEF </pre> in <code>directory1_src/OMakefile</code> to enable the debug code if the OMakefile is interpreted in the "right" output directory. <p><b>Note that the OMake developers consider vmount still as experimental.</b> </p><hr/> <a name="profile"></a> <h2>Profile builds</h2> A gprof profile build is enabled by passing -p to ocamlopt. In an application project, we can do this like <pre style="font-size:smaller"> if $(defined ENABLE_PROF) OCAMLOPTFLAGS += -p export </pre> If the user calls omake on the command-line as <pre style="font-size:smaller"> omake ENABLE_PROF=anyvalue </pre> the feature is enabled. <p><b>Simultaneous profile and non-profile builds:</b> If you develop a library for the general public, you may want to provide both a non-profile and a profile build. Given the following directory layout </p><pre style="font-size:smaller"> root +---- directory1_src +---- directory1_normal +---- directory1_gprof </pre> you can use the vmount feature to enable the profile build only in directory1_gprof. (We assume here that you do not build anything in <code>root</code>.) In <code>root/OMakefile</code> replace the normal <code>.SUBDIRS</code> line with <pre style="font-size:smaller"> section vmount(-l, directory1_src, directory1_normal) ENABLE_PROF = false .SUBDIRS: directory1_normal section vmount(-l, directory1_src, directory1_gprof) ENABLE_PROF = true .SUBDIRS: directory1_gprof </pre> Put all your sources into <code>directory1_src</code>, and leave the other two directories empty. The <code>directory1_src/OMakefile</code> should contain <pre style="font-size:smaller"> if $(defined ENABLE_PROF) OCAMLOPTFLAGS += -p BYTE_ENABLED = false export </pre> at the beginning. If you now build your projects, you get everything twice: In <code>directory1_normal</code> profiling is disabled, whereas in <code>directory1_gprof</code> it is enabled. <p>When installing your project as findlib library, there is the possibility to put all archive and cmi files into the same findlib directory. The cmi files should be the same in both versions, so you can take them from either version. The native archives, however, are different (both the .cmxa and the .a files). An install target (in <code>root/OMakefile</code>) could e.g. do </p><pre style="font-size:smaller"> .PHONY: install install: ln-or-cp directory1_gprof/myarchive.cmxa directory1_gprof/myarchive.p.cmxa ln-or-cp directory1_gprof/myarchive.a directory1_gprof/myarchive.p.a ocamlfind install mylib META \ directory1_src/*.mli \ directory1_normal/*.cmi \ directory1_normal/myarchive.cma \ directory1_normal/myarchive.cmxa \ directory1_normal/myarchive.a \ directory1_gprof/myarchive.p.cmxa \ directory1_gprof/myarchive.p.a </pre> and the META file would refer to the archives like <pre style="font-size:smaller"> archive(byte) = "myarchive.cma" archive(native) = "myarchive.cmxa" archive(native,gprof) = "myarchive.p.cmxa" </pre> This way, the profile-enabled version of your library is automatically taken when <code>ocamlfind ocamlopt -p</code> runs. <p><b>Note that the OMake developers consider vmount still as experimental.</b> </p><hr/> <a name="generate"></a> <h2>Generating files</h2> Three standard cases are already handled by OMake: <ul> <li>ocamllex: OMake automatically finds files ending in .mll, and runs ocamllex to generate to corresponding .ml files </li><li>ocamlyacc: Similarly, files ending in .mly are processed by ocamlyacc </li><li>menhir: If you set <code>MENHIR_ENABLED=true</code>, files ending in .mly are instead processed by menhir </li></ul> If you use a different generator, you need to add rules for handling these. <p>Example: The <code>atdgen</code> utility can generate I/O serializers for given type definitions. If called like </p><pre style="font-size:smaller"> atdgen -t foo.atd </pre> it generates from the description <code>foo.atd</code> two files, namely <code>foo_t.ml</code> and <code>foo_t.mli</code>. We create this rule for it: <pre style="font-size:smaller"> %_t.ml %_t.mli: %.atd atdgen -t $< </pre> Note that you should describe all output files on the left side of the colon, and all input files on the right side. In the command, <code>$<</code> is replaced by the leftmost input file. <p>It most cases, adding such a rule is already fully sufficient to drive the generation, and to include the generated files into the dependency graph. OMake calls ocamldep with the <code>-modules</code> flag meaning that it emits the dependencies by module names and not by files. Because of that, ocamldep also prints dependencies on modules that do not yet exist as files, but first need to be generated. OMake then postprocesses these dependencies, and compares them with the left sides of the rules to check whether the files can be generated. </p><hr/> <a name="stublibs"></a> <h2>Compiling stub libraries</h2> OMake contains full support for C projects. Nevertheless, for stub libraries wrapping C functions for use in OCaml some special definitions should be made. <p>The problem here is that OMake has its own idea what the C compiler to call is, and with which flags. With a few configurations, though, one can tell OMake to call the right C compiler with the right flags: </p><pre style="font-size:smaller"> CC_CONFIG = $(shell $(OCAMLC) -config | grep bytecode_c_compiler:) CC = $(nth 1, $(CC_CONFIG)) CFLAGS = $(nth-tl 2, $(CC_CONFIG)) CFLAGS += -I$(shell $(OCAMLC) -where) </pre> These are essentially the same flags that are added when you compile a C file and use ocamlc as driver (e.g. <code>ocamlc -c file.c</code>). <p>In a native-only project you can replace "bytecode_c_compiler" with "native_c_compiler" (this avoids the -fPIC flag giving a minimal boost). </p><p>The built-in link function <code>OCamlLibrary</code> does not support stub libraries. Because of this we need to define our own rules. Given the following source files </p><pre style="font-size:smaller"> foo.ml foo.mli foo_stubs.c </pre> we would like to generate these outputs: <pre style="font-size:smaller"> libfoo.a # containing foo_stubs.o libfoo.so foo.cma # the compiled foo.ml foo.cmxa </pre> The rules doing this: <pre style="font-size:smaller"> OCAML_MKLIB_FLAGS = ... # set e.g. to: -lbar if $(BYTE_ENABLED) foo.cma libfoo$(EXT_LIB): foo.cmo foo_stubs$(EXT_OBJ) ocamlmklib -o foo foo.cmo foo_stubs$(EXT_OBJ) \ $(OCAML_BYTE_LINK_FLAGS) $(OCAML_LINK_FLAGS) $(OCAML_MKLIB_FLAGS) if $(NATIVE_ENABLED) foo.cmxa libfoo$(EXT_LIB): foo.cmx foo_stubs$(EXT_OBJ) ocamlmklib -custom -o foo foo.cmx foo_stubs$(EXT_OBJ) \ $(OCAML_NATIVE_LINK_FLAGS) $(OCAML_LINK_FLAGS) $(OCAML_MKLIB_FLAGS) </pre> Note that we do not specify that libfoo.so (or libfoo.dll on Windows) is also created, at least on many platforms. <hr/> <a name="dot-prob"></a> <h2>The ":" problem</h2> Beware that the OMake grammar has some ambiguities. For example <pre style="font-size:smaller"> echo A : B </pre> does not output "A : B", but rather defines a dependency that "echo" and "A" depend on "B". <b>How to circumvent:</b> <ul> <li>Put : into quotes (":") if meaningful (note that such quotes are passed down to invoked commands) </li><li>Put : into OMake quotes ($":"). This should always work </li><li>Quote : with a backslash (\:) </li><li>It is also no problem if the colon occurs inside <code>$(...)</code> </li></ul> Note that the problem does not occur when you define a variable (e.g. <code>X = A : B</code>). <hr/> <a name="clean"></a> <h2>Cleaning completely</h2> I often switch between different ocaml versions (e.g. currently between ocaml-3.12 and ocaml-4.00), and I have of a separate omake instance for each version. So far everything works great. <p>Sometimes, however, I compile a project with a different ocaml version than the last time. Now I get errors, even for "omake clean", e.g.: </p><pre style="font-size:smaller"> *** omake: reading OMakefiles *** omake error: File /opt/godi-4.00/lib/omake/build/C.om: line 4, characters 2-32 unbound variable: public.OMakeVersion </pre> The problem is that OMake caches some values between invocations, namely in the files ending in <code>.omc</code>. <p>The solutions: </p><ul> <li>Remove all <code>.omc</code> files, and <code>.omakedb</code> in the topmost directory </li><li>Call omake with flag <code>--flush-includes</code> </li></ul> <hr/> <a name="expr-or-stmt"></a> <h2>Expressions or statements?</h2> You may already have wondered by we write <pre style="font-size:smaller"> .DEFAULT: $(OCamlProgram ...) </pre> if the program is to built by default, but only <pre style="font-size:smaller"> OCamlProgram(...) </pre> if not. Essentially, OMake distinguishes between contexts where you invoke functions as expressions returning values, and contexts where functions are only seen as statements, returning nothing. <p>On the outermost level, everything is a statement, e.g. </p><ul> <li>A statement defining a variable: <pre>VAR = expr</pre> </li><li>A statement defining a build rule: <pre> file.html: file.txt transform-txt-to-html file.txt </pre> </li><li>A conditional statement: <pre> if $(condition) statement1 else statement2 </pre> </li></ul> Within statements, you can substitute arguments by using expressions. In an expression, the executable code is within <code>$(...)</code> parentheses: <ul> <li>Substitute the value of a variable: <pre>$(VAR)</pre> </li><li>Call a function: <pre>$(getenv PATH)</pre> </li><li>Conditional: <pre>$(if $(condition),$(getenv V1),$(getenv V2))</pre> </li></ul> Actually, the distinction between expressions and statements is only syntactical, because statement blocks are allowed to return values: <pre> value $(expression...) </pre> (in function definitions you can also use <code>return</code> if you want to leave the function immediately). And, of course, expressions may have side effects, e.g. <pre> $(setvar VAR, $(...)) </pre> sets <code>VAR</code> in the same way as the statement <code>VAR=$(...)</code> would have. In some sense, the duality between "statement syntax" and "expression syntax" exists to keep some compatibility with traditional "make". <p>Now back to our initial question. <code>OCamlProgram</code> is a function that defines a number of build rules, and finally returns the main targets as an array of strings. If you execute <code>OCamlProgram</code> as a statement, the returned array is silently dropped. If you invoke it as an expression, as in </p><pre style="font-size:smaller"> .DEFAULT: $(OCamlProgram prog, $(FILES)) </pre> the function returns the array [| "prog"; "prog.opt"; "prog.run" |] after definining the build rules as side effect, and this is the same as if you had written <pre style="font-size:smaller"> OCamlProgram(prog, $(FILES)) .DEFAULT: prog prog.opt prog.run </pre> effectively declaring these targets as default targets. <br/> <br/> </div> <div> Gerd Stolpmann works as O'Caml consultant </div> <div> </div>