None: ode
1. A lyric poem with complex stanza forms

I have these chronic problems with missing or invalid header files. Today I’m going to deal with one of them.

Headers in question are X11/Xos.h and X11/Xfuncproto.h from my attempt to build firefox for mingw32. (Note that I have no idea what X11 is. And I suspect that X11 is something very obvious.). They are the missing dependency in several different files: cppsetup.c, include.c, parse.c, main.c, def.h.

It obviously looks like a group of source files that include the same header file def.h, that includes the missing headers X11/Xos.h and X11/Xfuncproto.h. So, I will go straight to def.h

def.h:
#ifndef NO_X11
#include <X11/Xos.h>
#include <X11/Xfuncproto.h>

Ok, this suggest that these are actually external headers, that I supposedly have under /usr/include/X11 or /usr/X11/include. So what is X11? X11 core headers? I feel like I’m missing something very obvious. However, after some time “poking” around, I think, I need xorg-x11-proto-devel package.

After installing it, I got my X11 headers under /usr/include/X11. And now I can continue with the firefox build.

First of all, I’m going to get a firefox build that passed unit tests. To do that, I need to go to tinderbox and look for “WINNT 5.2 mozilla-central build”. Next step is to find the latest successful source code revision.

Right now (March 21st, 5:35 pm GMT) the latest “green” revision is 1633e6077d77.

Since I already have mozilla-central source on my machine, I’m just going to update to this revision. Under mozilla-central folder:
hg pull
hg update -r 1633e6077d77

After that I need to configure .mozconfig. I’ll take a sample mozconfig from MDC. It looks like this:
#Specify the cross compile
CROSS_COMPILE=1
ac_add_options --enable-application=browser
ac_add_options --host=i686-linux
ac_add_options --target=i686-mingw32
ac_add_options --enable-default-toolkit=cairo-windows
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../mozilla-mingw
# Mozilla trunk uses many Vista only features on Windows, so we should disable some components to make it buildable with mingw32.
ac_add_options --enable-debug
ac_add_options --disable-optimize
ac_add_options --disable-tests
ac_add_options --disable-embedding-tests
ac_add_options --disable-installer
ac_add_options --disable-accessibility

#ac_add_options –disable-vista-sdk-requirements
ac_add_options –with-windows-version=502
ac_add_options –disable-updater

#change this to where your libIDL-config file locate.
#I have libIDL itself, but I don’t have libIDL-config
#Installing libIDL-devel package seemed to fix it
HOST_LIBIDL_CONFIG=/usr/bin/libIDL-config-2
#Config your moztools position
#No need to change this
GLIB_PREFIX=$HOME/moztools
LIBIDL_PREFIX=$HOME/moztools

#disable xpcom stdcall calling convention because of gcc 4.3.0 bug
CPPFLAGS=”-DMOZ_DISABLE_XPCOM_STDCALL”

#Parallel compilation
mk_add_options MOZ_MAKE_FLAGS=”-j5″

Now, when I have a good mozconfig file, I need to apply some patches to make mingw happy.

Under mozilla-central:

mkdir ../mingw-patches
wget -O ../mingw-patches/disable-xpcom-stdcall.patch https://bugzilla.mozilla.org/attachment.cgi?id=332690
wget -O ../mingw-patches/gcc_stdcall.patch https://bugzilla.mozilla.org/attachment.cgi?id=332691
wget -O ../mingw-patches/crypto.patch https://bugzilla.mozilla.org/attachment.cgi?id=334847
wget -O ../mingw-patches/trivial.patch https://bugzilla.mozilla.org/attachment.cgi?id=334848
patch -p1 < ../mingw-patches/disable-xpcom-stdcall.patch
patch -p1 < ../mingw-patches/gcc_stdcall.patch
patch -p1 < ../mingw-patches/crypto.patch
patch -p1 < ../mingw-patches/trivial.patch

Some of the patches failed. I applied them manually. Now, I’m going to try and built it.  Let’s see what happens.

make -f client.mk build

Problem #1: *** Couldn't find autoconf 2.13.
Solution: yum install autoconf213

Problem #2:
checking for i686-mingw32-gcc... no
checking for i686-pc-mingw32-gcc... no
checking whether the C compiler (: ) works... no
configure: error: installation or configuration problem: C compiler cannot create executables.

Solution:
Add PATH=$PATH:$INSTALL/bin to your .bash_profile.
Where $INSTALL is your mingw32 folder.

Problem #3:
In file included from .../mozilla-central/config/mkdepend/cppsetup.c:29:
In file included from .../mozilla-central/config/mkdepend/include.c:30:
In file included from .../mozilla-central/config/mkdepend/parse.c:29:
In file included from .../mozilla-central/config/mkdepend/pr.c:29:
In file included from .../mozilla-central/config/mkdepend/main.c:29:
In file included from .../mozilla-central/config/mkdepend/def.h:30:21:
In file included from .../mozilla-central/config/mkdepend/def.h:31:28:
error: X11/Xos.h: No such file or directory
error: X11/Xfuncproto.h: No such file or directory
Solution:
I knew it was too easy.

Problem #4:
This time an error is half a mile long.
Solution:
I’m working on it.

Some reading material:
Cross-compiling Mozilla
The GNU configure and build system – Cross Compilations Tools
How Mozilla’s build system works

This is what I’m going to need:

• GNU Binutils – This is linker, assembler, and various other tools.
• GCC 4.3 – I have GCC 4.4.3. I might have to install GCC 4.3, if GCC 4.4.3 doesn’t play well with Dehydra.
• mingw-runtim, w32api, and Cygwin winsup – It has something to do with GCC cross compiler.
• Netscape wintools – This is something that is need for some xpidl. A quick search reveals that it (might) stands for [X]Cross Platform Interface Definition Language.
• And, off course, configuring Firefox build.

1. binutils-2.18 surprisingly installed with no problems.
2. Install mingw-runtime-3.14
3. Same for w32api-3.11.
4. Get GCC 4.3 source and winsup-src-20100315.

I had to get mpfr to install GCC, but other than that everything went without a problem.

Time to check the compiler:
[root@xxxxxx ~]# osd/mingw32/bin/i686-mingw32-gcc -v
Using built-in specs.
Target: i686-mingw32
Configured with: ../gcc-4.3.0/configure --prefix=/root/osd/mingw32/ --target=i686-mingw32 --with-gnu-ld --with-gnu-as --enable-__cxa_atexit --enable-languages=c,c++
Thread model: win32
gcc version 4.3.0 (GCC)

[root@xxxxxx ~]# osd/mingw32/bin/i686-mingw32-gcc test.c -o test.exe
[root@xxxxxx ~]# ls test*
test.c test.exe

On Windows:
C:\>test.exe
hello world!

I’m almost ready to build Firefox.

It’s going to be a nice distraction from my Mac adventure

Here is the plan for bullet point lovers:

1. Register on landfill and file the bug
2. Get Thunderbird source code and build a debug build
3. Find where and whan needs to be fixed
4. Submit a diff to my bug report
5. Get someone to review it

I had no problems with registration and creation of the bug report. I’m not going to go deep into details on how to build TB, since it’s very similar to the simple firefox build I’ve done earlier.

(Edit: I should probably take my words back. It took about 20+ builds to get a correct one. In half of the attempts to compile TB, it froze in the middle and had to be shut down. Considering each build takes 2-3 hours on my machine, it’s quite some time wasted. And when it wasn’t freezing, there were build errors 2 hours into compilation and linking.)

Now, when it is finally done, I end up with the following:

I wonder what platform it’s referring to…

Anyways, I’m going to forget that I do not have a working TB debug build and try to find the place to fix this bug without it.

I haven’t used TB for years, so I’m just going to guess. Mailto is essentially a URL. Some programs highlight plain text URLs and put them in special tags, usually html. With that in mind, I’m going to search comm-central for some kind of URL parser.

It looks like nsURLParsers.cpp that has an interesting method  ParseURL(…). However, after looking at ParseURL I do not think this is what I need.

It figures, I’m looking at this from a wrong perspective. Mailto tag has to be created first. And only then (when it’s clicked, for example) it’s parsed by the program. So, to create an html tag out of plain text, I would need some kind of text to html converter, that would parse anything that looks like an email address.

It looks like mozTXTToHTMLConv.cpp is the file I need.

Functions to look at:

CompleteAbbreviatedURL(…) – This function will add “mailto:” to a string that looks like “mail@host.domain” or some “wr@ngurl.here”. The only thing it checks is “@” sign followed  by “.” sign. This would probably also match “a@.”, and convert it to “mailto:a@.”. But this doesn’t seem to be the place, since this function serves a single purpose of fixing incomplete URLs.

CheckURLAndCreateHTML(…) – This function looks more promising. I understand what this function is supposed to do, but I have no idea what it is exactly doing. It’s clear that it add “href” tag to the txtURL, but what about those modes: RFC1738, RFC2396E, freetext, and abbreviated. I would assume that it’s used somewhere else to make a correct HTML for this URL. That is something to keep in mind.

And the last one – FindURL(…), which is called by ScanTXT(…) upon finding “@”, “.”, and “:” signs.  “enum:modetypes” comments in FindURL answer my previous question about RFCs. From looking at those comments, RFC2396E and abbreviated are the ones I should interested in, since those match mailto URLs.

From looking at the FindURL code, it checks for “@” sign and marks mode as RFC2396E and then checks for “.” sign and marks mode as abbreviated.

Then it looks for the start of URL based on the mode. For abbreviated, it will look from the position of “@” sign in the string for a first defined special character, or space, or non-digit. Makes sense so far.

If previous step is successful it will look for the end of URL based on the mode. And again I’m more interested in abbreviated and freetext mode parsing here. In a similar to FindURLStart(…) fashion,  it looks for the first defined special character or space and marks it as the end of URL. For emails, however, it looks for “(“, ” ‘ “, and any non-ASCII character, and marks the end of URL. It will also eliminate any trailing “.,;!?-\”, since these signs are not allowed at the end of an email address. Now, I’m certain that I need to make a change to mozTXTToHTMLConv::FindURLEnd for the case of email address.

Let’s take a look at the requirements:

Valid Addresses

1. a@a.com
2. a.a@a.a
3. a.a@a.a.com

Invalid Addresses

1. a@.
2. a@..
3. a@..f
4. a@….f
5. @.
6. @.com
7. @a.com
8. @a
9. @a.
10. @a..
11. @a…com
12. a@a.a..a

It looks like a lof of work.

I’m 100% sure that valid addresses 1, 2, and 3 are going to be matched, since FindURLStart and FindURLEnd will terminate when then find a space. Invalid addresses is a different story. It is hard to come up with a single rule for all of them without breaking existing ones. Cases 5 to 11 do not require a FindURLStart change, since the program already checks if recipient’s mail address is present.

The change is going to be made in mozTXTToHTMLConv::FindURLEnd(…):

The main requiremens for this one are:
a) there must be something between “@”, “.”, and any consecutive “.” as in “@doe.somesite.somewhere.domain…”
b) there must be a domain name after “.” as in “@doe.domain?..”

I’m going to start with a Mac build and follow instructions in this blog post in my attempt to build it.

I need to get toolwhip source code and compile cctools and ld64. These are the parts of a custom toolchain. To spice things up I will also need Apple’s gcc.

I will start with cctools following instructions in cctools.README.

To compile cctools:
CC="gcc -m32 -std=gnu99" CFLAGS="-O2" make -f Makefile.linux

And it fails:
/usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory

Google tells me that I need to install glibc-devel package:
yum -y install glibc-devel

Second try:
/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/4.3.2/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: cannot find -lgcc_s

That is great. Linker doesn’t like libgcc_s.so for some reason.

A-smarter-than-me-person-over-on-fedora-forums notes that this problem has something to do with compiling 32-bit app on a 64-bit system and suggests that installing GCC compatibility packages compat-libstdc++-33 and compat-libstdc++-296 should help.

I do not know if this is the case. It looks like I would need to do a lot of extra work on a 64-bit box.

“If the mountain won’t come to Muhammad, Muhammad must go to the mountain.”

I’m going to move to 32-bit box.

I’m back to compiling cctools; this time on a 32-bit box. And… it compiles just fine. Now I need to install it. I created some folders, symbolic links, and a script as described in cctools.README under installation.

I have no idea if it’s working or not. Or if I did everything right. But I’m just going to live with it by now.

Next step is to compile and install ld64. I’m going to follow instructions in ld64.README.

To compile ld64:
CC="gcc -m32 -std=gnu99" CXX="g++ -m32" CFLAGS="-O2" CXXFLAGS="${CFLAGS}" make -f Makefile.linux

And it fails. It cannot find declaration of functions like fopen() which is a part of stdio.h.

This does not make any sense. (Edit: It might be related to gcc 4.4.x bug)

Ok, nevermind. I added the following line to /toolwhip-read-only/ld64/src/Options.cpp:

#include <stdio.h>

I feel as if I’m fixing my first assignment in IPC144.

This time around Options.cpp compiled fine, but they might have a 1000 + 1 file with similar problems. Did I mention that I like inflicting pain on myself?

I did the same to:
MachOReaderDylib.hpp
MachOWriterExecutable.hpp
MachOReaderRelocatable.hpp

Ok, the linker compiled. And I created some folders and links mentioned in ld64.README under installation.

By the way, I have revision 176 of toolwhip. And I compile it with gcc 4.4.2.

The next and hopefully the last step to get this toolchain done is Apple’s gcc. I will work on it later.

I’m too exhausted
To write any “Haiku” right now
Do not even start

Donwload Apple’s gcc to toolwip directory and patch it:
wget http://www.opensource.apple.com/tarballs/gcc_42/gcc_42-5574.tar.gz
tar -xvzf gcc_42-5574.tar.gz
patch -p0 --verbose -i gcc_42-5574.patch

To install it, I’m going to follow gcc_42-5574.README.

The first thing I need is “appropriate” Mac SDKs. I’m not a Mac user. And I don’t have Mac to copy SDKs from. So, I wonder which one is appropriate?

It seems that there are two ways to do it: the easy way and the hard way. The easy way consists of bribing a friendly Mac user to get you a copy of /Developer/SDKs/MacOSX10.5.sdk folder. The hard way consists of downloading XCode and getting necessary packages from there.

Since I said earlier that I like inflicting pain on myself, I will go with the hard way.

There are a couple of way to open .dmg files.

1. To open “f****** stupid useless .dmg file” in Linux
2. To open .dmg file in Windows
3. HFSExplorer for both

Once that’s done, all packages can be found under… package folder. I’m going to make a wild guess and say that MacOSX10.5.pkg is the package I need. However, now I’m stuck with “f****** stupid useless .pkg file”. And after long search I found nothing (that actually works) on how to unpack .pkg files outside Mac. (I’m almost waiting for someone to point out, that it’s something simple and obvious).

That leaves me no choice, but to go the easy way:
beer -h friendlyMacUser -o MacOSX10.5.sdk.gz

And time to build it.

First problem:
P: lipo: can't open input file: /usr/lib/libSystem.B.dylib (No such file or directory)

It’s looking for this library in the standard folder.

S: ln -s /Developer/SDKs/MacOSX10.5.sdk/usr/lib/libSystem.B.dylib /usr/lib/libSystem.B.dylib

Second problem:
make[3]: execvp: i686-apple-darwin9-ranlib: Permission denied

I’m doing everything as root. However,i686-apple-darwin9-ranlib is a script I created earlier (see cctools.README). And the error has everything to do with a permission of the script.

chmod +x /Developer/usr/bin/i686-apple-darwin9-ranlib

Third problem:
ld warning: in /Developer/SDKs/MacOSX10.5.sdk/usr/lib/libc.dylib, file is not of required architecture
A long list of missing references...


This is 9/10 bad on my scale of Bad, because it has everything to do with the MacOSX10.5.sdk files I got from a random Mac.

Below is the content of libc.dylib:
XSym
0015
8ddc754cd3d0fce1cdf43a87f3e54bca
libSystem.dylib

It looks like it points to libSystem.dylib, which, in turn, points to libSystem.B.dylib. And libSystem.B.dylib is compiled library.

I think I will have to review build log, find any reference of libSystem.B.dylib, see what objects libSystem.B.dylib is compiled from, and check their source code. Or I can just use nm, but this is a Mac library, so it might not work. So, by doing this, I might find out what “architercute” it supports, and what I’m doing wrong.

It figures that nm doesn’t work after all. It cannot recognize “dylib” files. So, the only way is to check the build log.

First of all, I’m going to get Mozilla build tools for windows. I’m running windows 7 and VS2008, so I do not need to worry about .NET Framework. But I will most likely need some of the windows SDKs.

Mozilla build tools is going to install lots of stuff to “drive:\mozilla-build” (by default). Next thing I’m going to do is to set environment to use VS2008. In cmd under mozilla-build folder execute start-msvc9.bat.

C:\mozilla-build>start-msvc9.bat
"Mozilla tools directory: C:\mozilla-build\"
Windows SDK directory: C:\Program Files\Microsoft SDKs\Windows\v6.0A\
Windows SDK version: 6.0A
Setting environment for using Microsoft Visual Studio 2008 x86 tools.
Mozilla build environment: MSVC version 9.

I guess, it worked.

Next step is to get the source code. Mozilla uses HG to manage source code (you can get HG here). I’m going to get the code from the main tree:

hg clone http://hg.mozilla.org/mozilla-central/

I’m doing it over my college wireless. It’s probably not the best idea, because there is 1GB of source code data.

C:\Program Files (x86)\Mercurial>hg clone http://hg.mozilla.org/mozilla-central/
destination directory: mozilla-central
requesting all changes
adding changesets
adding manifests
transaction abort!
rollback completed
abort: connection ended unexpectedly

Thought so… So, first step should have been “look for a lan cable”.

Here is a “haiku”:

Found a LAN cable
Broadband connection is nice
Very Cold outside

Before I build I have to make sure that there is no spaces in the path to “make”. And I have to have a mozconfig file in the build directory.

mozconfig should containt the following lines:
. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-ff-release
mk_add_options MOZ_MAKE_FLAGS="-j4"

So now when I have the source code all I have to do is “make -f client.mk”. Spirits of my ancestors tell me that “make” is a unix program. Here is where MozillaBuild comes into play. Under “drive:\mozilla-build”, I’m going to execute “start-msvc9.bat” to get a build console. Inside the console I’m going to execute “make -f client.mk” to start compiling firefox.

And SDK version check fails:

checking for Windows SDK being recent enough... no
configure: error: You are targeting Windows version 0x06010000, but your SDK only supports up to version 0x06000000.

It basically requires windows 7 SDK, that can be downloaded here. 3 ISO files, 1500 mb each. I will reduce the world carbon dioxide emissions by not downloading those files. Instead, I’m going to add “ac_add_options –with-windows-version=600″ to mozconfig and see if it works.

I will see you after the brea… after it builds.

It failed with the following error:

cl : Command line error D8027 : cannot execute ‘c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\c2.dll’

I have another “haiku”:

Building Firefox
Installing Visual Studio SP1
Not good at all

I let VS2008 SP1 install first and re-started “make”. Now, I just need to wait for it to finish.

An hour later the build is finished. Time to test.

drive:\mozilla-build\mozilla-central\objdir-ff-release\dist\bin\firefox.exe

Welcome to Minefield

Minefield installed
It has not crashed on me
Broken Promises

Overall, it wasn’t too difficult to do “simple firefox build”. The main requirements are keyboard, monitor, and patience… and a lan cable.

Now, let’s say there is a solution out there, that can be copied and will work right away. A honest student wouldn’t copy it without understanding, but rather review existing solutions, apply them, and get something out of them. Another student would just copy the solution. That student hasn’t done any work and learnt nothing.

There is middle ground here. Without openly sharing complete solutions, students can discuss and share solutions to specific problems. ‘This is how I handled user sessions on my JSP website’ or ‘This is how I ported this solution to Windows’. Students can review each others work and collaborate. So that everybody learns from each others experience in a way that still allows to evaluate all students based on the work they have done in the course.