Firebird Trace and Audit Services

Posted by Vlad Horsun to the Firebird Development List.

I am glad to introduce new Firebird facility, based on TraceAPI initially developed by Nickolay Samofatov and maintained and given to us by RedSoft (thanks!).

Below is brief overview of what was done. It will be included into HEAD soon, I hope.

Trace session - set of engine-generated events to trace and placement of trace output. Trace session also have its unique ID, optional name, state, flags, creator user and creation timestamp.

List of available trace events is fixed and determined by the Firebird engine. List of events to trace, which data items to trace and placement of trace output is specified by trace configuration when trace session created.

There are two kinds of trace sessions: system audit and user trace. System audit session is started by the engine itself and obtains configuration from the text file. This file name is set via new setting in firebird.conf and by default is "none", i.e. no system audit is configured. There may be only one system audit trace session, obviously. Configuration file contains list of traced events and placement of trace log(s). It is very flexible and allows to log different set of events
for different databases into different log files. Example configuration file is attached.

User trace session is managed by user via Services API. There are five new services for this purposes - to start, stop, suspend, resume trace session and to enlist all known trace sessions.

When user application starts trace session it passed optional name and (mandatory) session configuration. Session configuration is a text. Rules and syntax the same as for configuration file mentioned above except of log placement. Engine stores output of user session in set of temporary files and deleted it automatically. After application starts user trace session it must read session's output from service (using isc_service_query). When application decide to stop its trace session it just do detach from service. Also user may manage (suspend\resume\stop) any own trace session. Administrator allowed to manage any trace session.

If user trace session was created by ordinary user it will trace only connections established by this user.

User trace sessions is not preserved when all Firebird processes is stopped. I.e. if user started trace session and Firebird SS or SC process is shutted down, this session is stopped and will not be restarted with new Firebird process. For CS this is not the case as user trace session can't live without connection with service manager and dedicated CS process.

There are three main use cases:

1. Constant audit of engine.
This is served by system audit trace.

2. On demand interactive trace of some (or all) activity in some (or all) databases.
Application start user trace session and show traced events to user in real time on the screen. User may pause\continue trace and at last stop it.

3. Aggregate of some activity for a relatively long period of time (few hours or even whole day) to analyse it later.
Application start user trace session and save trace log in file (or set of files). Session must be stopped manually by the same application or by another one.

Firebird's service manager utility (fbsvcmgr.exe) may be used to manage trace sessions. Later specialized console utility will be introduced (at least i plan to develop it soon after beta). Of course third party vendors are welcome to implement own GUI based monitoring and trace applications.

InterBase 2009 comes with multicore support

I have just read the following news article on SDTimes:
InterBase 2009 comes with multicore support
A couple of aspects to this story seeme a little peculiar, while some comments require some clarification.

"Thus the update for InterBase 2009 is relatively subdued"

correct, I do not see any major new features in this new version, however a couple of new features highlighted are

"such as over-the-wire encryption and AES as an encryption scheme".

If you want over the wire encryption for Firebird you could use Zebedee. There have been a number of papers re. this in the past. However this is something that could be implemented in Firebird fairly easily if we so wished.

In terms of database encryption there is code to allow it now, however the original implementation was not reliable. We did put together a more detailed proposal, but after some lengthy discussions it was decided that such functionality was going to be too complicated for the majority of users, and most of them didn't want it any way.

I do have trouble working out what

"The biggest change, perhaps, is the removal of any non-symmetric multiprocessing (SMP) compatible versions of the database. All versions of InterBase will now be able to use multiple processors and cores."

means.. CodeGear have been stating that InterBase was SMP enabled since InterBase 7.0. So what does this mean? Was InterBase non SMP capable/SMP friendly on some platforms? Was there code that wasn't SMP enabled? Has the code been cleaned up, so that all the old platforms have been removed. Or is it just a licensing issue where the new SMP license now supports up to 8 cores/CPUs, while previously you had to buy extra CPU licenses?

One particular quote was very confusing

"For the existing open-source Firebird database, which is based on InterBase, these changes won't trickle down; that free database does not include SMP, disaster recovery or high availability features."

Firebird was originally based on the open sourced version of InterBase 6.0, but the code bases are now quite different after 8 years of independent development work. So to state that Firebird is based on InterBase, although this is technically correct, it is a little disingenuous.

Stating that changes in InterBase will not trickle down to Firebird is correct. InterBase as per Inprise/Borland/CodeGear/Embarcadero is a closed source database. After the code was originally open sourced on SourceForge by Borland, it was forked and Firebird was created. InterBase was then "closed sourced" by Borland, and development work has continued on InterBase in a closed source manner. No code or feature at any time has come from InterBase to Firebird. Although it is possible that Firebird code may have made it into InterBase. However we have no way of verifying that.

In terms of SMP, by re-introducing the process based Classic Arhitecture, Firebird does support SMP. However a process based architecture isn't multi-threaded. That's an issue that's being addressed in Firebird V2.5 (SuperClassic) and Firebird 3.0.

I assume by "Disaster Recovery" this refers to "Journaling with Write Ahead Logging (WAL)" something that has not been implemented in Firebird. I am unsure what the other "High Availability" features are.

Finally

"In October, he said that InterBase ToGo will be made available. Normally, InterBase has a memory footprint of around 100MB. ToGo will be slimmed to around 3MB"

I am lost on this one. I am sure by this he means a disk footprint, not memory. And most of the size of the normal disk footprint is taken up by a stupid installer (imho) and docs. My assumption is that InterBase ToGo is effectively a variant on Firebird Embedded, but since ToGo will not be available until October it is difficult to comment.

Solaris 10, Firebird SuperServer and the Service Management Facility

I recently released Firebird V2.04 SuperServer for Solaris 10. This new build uses the Solaris 10 Service Management Facility to stop/start Firebird. Here are some notes on how it works.
I created the following manifest file (it can be found in /opt/firebird/ as gds_db-tcp.xml):

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<!--
Service manifest for firebird
-->

<service_bundle type='manifest' name='firebird'>
<service
name='application/firebird'
type='service'
version='1'>
<create_default_instance enabled='true' />

<exec_method
type='method'
name='start'
exec='/opt/firebird/bin/fbmgr.bin -start'
timeout_seconds='10'>
<method_context>
<method_credential user='firebird' group='firebird' />
</method_context>
</exec_method>

<exec_method
type='method'
name='stop'
exec='/opt/firebird/bin/fbmgr.bin -stop'
timeout_seconds='10'>
<method_context>
<method_credential user='firebird' group='firebird' />
</method_context>
</exec_method>

<template>
<common_name>
<loctext xml:lang='C'>
Firebird
</loctext>
</common_name>
</template>
</service>
</service_bundle>

This manifest file is automatically installed by the pkg installer.
You can manually stop and start firebird using the following commands, instead of using fbmgr.bin

svcadm enable svc:/application/firebird:default
svcadm disable svc:/application/firebird:default

You can verify whether the Firebird "service" is running by using the following:

svcs -a | grep firebird

Supposedly the new Service Management Facility provides automatic recovery from software and hardware errors, so theoretically SuperServer on Solaris 10 no longer has need of the Firebird Guardian, since SMF should do the same job that the Guardian was designed to do. This is going to need some testing ....

Solaris 10 and iscsi

Seems that applying all of the patches discussed previously has broken my version of Solaris, as it now seems to have problems booting.

It seems to hang during the boot cycle, so I had to do some hunting to try and find out the cause.
Boooting with a -v (verbose) shows that the system is hanging in iscsi.
After some googling it seems the only way to fix this, so you can get in is to do the following..

1. Boot using -m milestone=none
2. Login as root
3. svcadm enable -r filesystem/minimal
4. svcadm enable -r filesystem/local
5. cd /etc/iscsi
6. mv iscsi_v1.dbc iscsi_v1.dbc.bad
7. reboot

Now you can get in, I suggest you go and install the latest iscsi path from sunsolve:
Patch number 119091 for x86 Solaris.

reboot and hopefully all will be well.

Solaris and the 256 File Descriptor Problem

Some of you reading this may remember that back when InterBase 6.0 was released on Solaris we ran into a fundamental problem on Solaris. It seems that since 1992 there has been an open bug documenting Solaris' file descriptor problem: "32-bit stdio routines should support file descriptors >255."

Bug id:
1085341

When this issue was discovered, we chose to use SFIO, a stdio variant from AT&T
Comments on this issue can be found in jrd/common.h which forces a superserver build to use our own internal stdio.h found in extern/SfIO/include.

Well it looks like in Solaris 10 Sun have finally fixed the issue.
More details can be found in the article entitled:
Solaris OS Solutions to 32-Bit stdio's 256 File-Descriptors Limitation by Giri Mandalika dated May 2007

Suffice it say - the bug has been fixed, but in a fairly typical Sun way.... the fix is more complicated than it should be... why they didn't just change the char to a short and be done with it I really don't know.

Anyway it should be possible on Solaris 10 to use this fix instead of our "own" implementation of stdio.h, just by adding a couple of lines to gds.cpp
e.g.

#ifdef SOLARIS
#include <stdio_ext.h>
#endif
after including

and then adding the following at around about line 3535 (Firebird 2.0.4)
#ifdef SOLARIS
enable_extended_FILE_stdio (-1, -1);
#endif

Hmm that seems to be a nice simple solution, except that its always more complicated than that.
It seems that this fix is only available from later releases of Solaris 10, so if you have a version of Solaris from Solaris 10 3/05 through Solaris 10 11/06 you will need to install just 3 patches...
Well no - thats not the case. I had to do the following to get this to work...

Download and install the following patches:

1. 118855-36
2. 118919-21
3. 119255-53
4. 120273-21
5. 123840-04
6. 127756-01
7. 125504-02
8. 122661-08
9. 125548-02
10. 126420-01
11. 126424-03

And then finally in single user mode, followed by a reconfigure reboot:
To boot Solaris x86 in single user mode, requires some fun and games too...

At the GRUB menu, type 'e' before it times out and takes the default boot option.
Scroll to the default multiuser boot line, then type 'e' egain.
This will bring you in to edit mode. Add a ' -s' at the end of the line.
Then type 'b' to boot.

12. 120012-14

Install this patch - then reconfigure

At the GRUB menu, type 'e' before it times out and takes the default boot option.
Scroll to the default multiuser boot line, then type 'e' egain.
This will bring you in to edit mode. Add a ' -r' at the end of the line.
Then type 'b' to boot.

I am not sure that we can ask Solaris users of Firebird if they do not have the fix already installed to go through this, so we will have a look at keeping our own SfIO. However currently that will not compile using gcc 4.2.1 and thats something else we need to try and fix.

Addendum:
sftp is now broken - so you need to add two more patches to the list above

13. 119253-26
14. 124629-07

MacOSX and stdin,stdout,stderr

I am in the process of porting a friends software to MacOSX. Don't ask what it does yet, all will be revealed soon. The code compiles quite happily on Solaris and Linux, but fails with the following error on MacOSX...

Why?

dbFile.c: In function ‘_setLogfile’:
dbFile.c:243: error: invalid lvalue in assignment
dbFile.c:244: error: invalid lvalue in assignment
dbFile.c: In function ‘fieldFillGen’:

I know what the problem is -
Seems on MacOSX 10.4 and below:
stdin, stdout, and stderr are addresses of a FILE rather than a FILE*
On MacOSX 10.5 they are FILE*

So depending on the OS version you are targeting will determine whether the code compiles or not. The issue goes back to the way FreeBSD originally implemented this in stdio.h which is the base for MacOSX, and it looks like it finally became "correct" i.e. portable and adhering to other implementations in MacOSX 10.5
A quick look at /usr/include/stdio.h confirms this...
the FILE* appears within the ifdef for _DARWIN_UNIX_03

__BEGIN_DECLS
#if __DARWIN_UNIX03
extern FILE *__stdinp;
extern FILE *__stdoutp;
extern FILE *__stderrp;
#else /* !__DARWIN_UNIX03 */
extern FILE __sF[];
#endif /* __DARWIN_UNIX03 */
__END_DECLS


So I need to come up with a simple workaround for what was originally valid code on other operating systems:

stderr = wk;
stdout = wk;

the following seems to work:

dup2(wk->_file ,STDERR_FILENO);
dup2(wk->_file ,STDOUT_FILENO);

Standard delete v's Firebird delete

Alex and I have just committed a slew of changes to the code in Firebird 2.1.
This all started when we did the 32 bit builds for Firebird on MacOSX.

We found that in remote we would get asserts trying to create a database. On deeper and further investigation over a period of a couple of weeks we realised that in some cases the standard delete was being called instead of the redefined version in Firebird i.e. it invoked free() instead of
MemoryPool::globalFree(). After some extensive googling we found other people have also run into similar problems.

As a solution it was suggested to use:
1. Use switches -fvisibility=hidden and -fvisibility-inlines-hidden together.
2. Use switch -fvisibility=hidden and __attribute__((always_inline)).

Trying with second combination we found that we sometimes got (depending on the version of gcc) an error:
sorry, unimplemented: inlining failed in call to 'void operator delete(void*) throw ()'.

It seems that without -fvisibility=hidden and __attribute__((always_inline)) gcc is
using non-inline body for operator delete. And is randomly using the wrong body, the one from the standard operator delete.

The first combination helps to avoid this bug - but any entrypoint, which is to be exported, now has to be marked with __attribute__((visibility="default")).