Paul Beach's Blog

Firebird 2.5 SuperServer and MacOSX 10.7 (Lion)

2011-07-26T02:04:00.000-07:00

There is a small problem with Firebird 2.5 SuperServer and MacOSX 10.7 (Lion)...
SuperServer will not start, and produces a crash report header similar to this

Update 26th August 2011
Its not just 2.5 SuperServer, reports have come in that SuperServer for Firebird 2.1.4 will not start, also the same problem exists for Firebird 2.5 SuperClassic.

Process: fbserver [706]
Path: /Library/Frameworks/firebird.framework/Versions/A/Resources/bin/fbserver
Identifier: fbserver
Version: ??? (???)
Code Type: X86 (Native)
Parent Process: fbguard [166]

Date/Time: 2011-06-10 07:59:42.374 -0400
OS Version: Mac OS X 10.7 (11A480b)
Report Version: 9

Anonymous UUID: B7EDF790-CE72-4B21-A982-B9EA4F4E2088

Crashed Thread: 1 Dispatch queue: com.apple.libdispatch-manager

Exception Type: EXC_BAD_INSTRUCTION (SIGILL)
Exception Codes: 0x0000000000000001, 0x0000000000000000

Application Specific Information:
BUG IN CLIENT OF LIBDISPATCH: Do not close random Unix descriptors

The bug is fixed in Firebird 2.5.1 but Firebird 2.5.1 is not available yet. So I have fixed the problem also in the Firebird 2.5 code and produced 32bit and 64bit builds that work properly.

I have had new builds uploaded to Sourceforge to fix the problem with Firebird
2.1.4 SuperServer, you can distinguish the new builds from the old by the build no.
Old = 18393, New = 18393. Because we have some issues with 2.5.1 that are currently being investigated, I plan to update the 2.5 builds on Sourceforge soon.

Update 28th Sep 2011
The Firebird 2.5 builds have also been replaced on Sourceforge, so you just need to download the latest build. However Firebird 2.5 will be replaced by Firebird 2.5.1 very shortly, builds are now taking place and will be QA'd and released shortly.

Any problems - please let me know.

Firebird V2.1 Error: value exceeds the range for valid dates

2011-07-04T06:51:00.000-07:00

You are restoring a backup of a Firebird 1.5 or 2.0 database to Firebird 2.1 and you see an error similar to this:

gbak: writing data for table xyz
gbak:20000 records written
gbak: Error: value exceeds the range for valid dates
gbak: Error: gds_$receive failed
gbak:Exiting before completion due to errors

How do you go about solving the problem?
Well first of all a quick visit to the Firebird bug tracker reveals
http://tracker.firebirdsql.org/browse/CORE-1714 and a couple of comments from Dimitry Yemanov.

“The error means to say that some column has an invalid date value (outside the supported range). Prior to V2.1, it was possible to store such invalid values in the database, but now it's prohibited. A verbose output should point you to a problematic table.” Also “The current behavior is intended and is unlikely to be changed.”

1. Firstly use the –v (verbose option of gbak) to find out the table that is causing the problem.
2. Check the table xyz for date columns
3. Perform the following SQL operation on all the date columns you found in 2.

isql> update XYZ set MY_DATE_COLUMN = MY_DATE_COLUMN;

At some point during the process you will see the same error that gbak produced, but now you will know which column is causing the problem.

4. Install Firebird 1.5 or 2.0, by going back to the version of Firebird that allows for invalid dates, you will be at least able to correct the invalid date to something more appropriate.

5. Now lets check for dates that are outside of their proper range (01 Jan 0001 - 31 Dec 9999)

Below date zero:

isql> select PRIMARY_KEY from XYZ where MY_DATE_COLUMN < '0001-01-01'

if an error occurs correct the date to something more appropriate and meaningful

Maximum date

isql> select PRIMARY_KEY from XYZ where MY_DATE_COLUMN > '9999-12-31'

if an error occurs correct the date to something more appropriate and meaningful

6. You can now backup the database using Firebird 1.5 or 2.1 and successfully restore under 2.1

Firebird 3.0 - Shared Page Cache

2011-05-10T00:47:00.000-07:00

Posted by Vlad on the Firebird Development List.

After more than a year of development, testing, switching on another tasks and returning back I'm ready to commit shared page cache implementation into trunk.

I consider it stable enough to be committed into SVN. It runs some stability tests more than 10 hours for longest run and many shorter runs at different configurations.

Here I want to do overview of changes, what was tested, what was not, and what is work in progress.

Page cache was re-implemented looking at both Vulcan code (thanks to Jim) and our code base. It is not a blind copy of Vulcan code, every decision was thinking on and adjusted when needed with my understanding and expirience with existing code.

The cache is syncronized at different levels. There is a few BufferControl (BCB) level sync's used to syncronize access to the common structures such as LRU queue, Precedence graph, Dirty page list and so one. These common sync's should be locked at as short as possible period of time as this is kind of global locks. Also there is a two syncs for every BufferDesc (BDB) - one for guard contents of page buffer tself and second one is to prevent partial writes of page wich edition is not completed and to prevent simultaneous write of the same page by different threads.

I don't explain cache algoritms here, if someone have a question I'll be happy to answer it. And I not going to said that page cache implementation is frozen - if necessary we will change it of course (there is still space for impovements and bugs fixing).

You know I already committed SyncObject class (and related classes) ported from Vulcan. I removed some unused code and restored fairness of locking : initially it was fair in Vulcan too (i.e. all lock requests was put in waiting queue and granted in first-in-first-out, or fair, order), but later Jim found some (unknown to me) ssue with this and made preference for SHARED locks. I found no performance issues with fair locking so I decided to revert that and restore original behavior. Of course it could be changed when necessary.

Shared page cache is not an isolated change. It affects many parts of engine and our syncronization model changed significantly again. There was agreement that we will not implement shared metadata cache in v3 as this is risky and we could not deliver V3 in reasonable time frame.

In shared cache mode we have single Database instance and many attachment instances linked to it (this is not new).

All metadata objects moved into Attachment. Metadata syncronization is guarded by attachment's mutex now. Database::SyncGuard and company are replaced by corresponding Attachment::XXX classes.

To make AST's work we need to release attachment mutex sometimes. This is very important change after v2.5 : in v2.5 attachment mutex is locked during whole duration of API call and no other API call (except of asyncronous fb_cancel_operation) could work with "busy" attacment. In V3 this rule is not worked anymore. So, now we could run more that one API call on the same attachment (of course not really simultaneously). I'm not sure it is safe but not disabled it so far.

To make asyncronous detach safe I introduced att_use_count counter which is incremented each time when API call is entered and decremented on exit. detach now marks attachment as shutdown and waits for att_use_count == 0 before processing.

Parallel access to the attachment could be easy disabled making every API call wait for att_use_count == 0 on enter or even introducing one more mutex to avoid spin wait.

Also it seems this counter make obsolete att_in_use member as detach call should wait for att_use_count == 0 and drop call should return "object is in use" if att_use_count != 0.

All AST's related to attachment-level objects should take attachment mutex before access attachment internals. This is implemented but not tested!

Transaction inventory pages cache (TIP cache) was reworked and is shared by all attachments.

To avoid contention on common dbb_pool its usage was replaced by att_pool when possible. To make this task slightly easy there was introduced jrd_rel::rel_pool which points currently to the attachment's pool. All relation's "sub-objects" (such as formats, fields, etc) is allocated using rel_pool (it was dbb_pool before). When we'll return metadata objects back to the Database it will be easy to redirect rel_pool to dbb_pool at one place in code instead of makeing tens of small changes again.

About stability testing of different parts of the engine:
- page cache - tested and worked
- nbackup - tested and worked
- monitoring - tested (less) and seems worked
- server stop\database shutdown - somewhat tested, no crash observed, client reaction is not perfect (mostly network write errors)
- shadow - not tested
- metadata changes in concurrent environment - not tested
- garbage collection thread - not tested, needs review and rework of some

implementation details
- cache writer thread - not tested, needs review and rework
- external connections - not tested
- external engines - not tested

In configuration file there are two new settings :
a) SharedCache - boolean value which rules the page cache mode
b) SharedDatabase - boolean value which which rules the database file open mode.

Currently they are common (as whole configuration) but soon it will be per-database settings. Below is few examples of how it could be used :

- SharedCache = true, SharedDatabase = false (default mode)
this is traditional SuperServer mode when all attachments share page cache and database file is opens in exclusive mode (only one server process could work with database)

- SharedCache = false, SharedDatabase = true
this is Classic mode when each attachment have its own page cache and many server processes could work with the same database.

To run SuperClassic you should use switch -m in command line of firebird.exe (on Windows) or run fb_smp_server (on Posix, here i'm not sure and Alex will correct me)

Else ClassicServer will run.

- SharedCache = true, SharedDatabase = true
this is completely new mode in which database file could be opened by many server processes and each process could handle many attachments which will share page cache (i.e. per-process page cache).

It could be used to run few SS processes, for example, or to run "main" SS process and have ability to attach using embedded to make some special tasks.

Must note that our lock manager is not ready to work in this mode, so we can't use it right now.

Also there is unknown how performance will be affected when few SS with big cache will work with the same database.

- SharedCache = false, SharedDatabase = false
Looks like single process with single attachment will be allowed to work with database with such settings. Probably you can find an applications for it ;)

One more change in configuration is that CpuAffinityMask setting changed its default value and it is 0 now. It allows new SS to use all available CPU's\cores by default.

OAT and MON$

2011-03-31T07:19:00.000-07:00

Just as a "for your information", should anybody be interested.
The following query on Firebird V2.x gets the attachment that holds the oldest active transaction via the monitoring tables.

SELECT *
FROM MON$ATTACHMENTS
where MON$ATTACHMENT_ID =
(select MON$ATTACHMENT_ID FROM MON$TRANSACTIONS
where MON$TRANSACTION_ID =
(SELECT min(MON$TRANSACTION_ID) FROM MON$TRANSACTIONS))

HPUX Oracle and Itanium

2011-03-31T02:12:00.000-07:00

The announcement by Oracle last week that it will stop developing software for the Itanium processor will affect users of Oracle on HPUX Integrity servers. No surprise then that we see the following blog post - Oracle is abandoning HP-UX customers…Fight back! from Ed Boyajian the CEO of EnterpriseDB suggesting that users should consider migrating to their version of PostgreSQL, even though as he admits PostgreSQL has not supported the HP-UX platform very well, but that is now going to change.

For those who are curious, Firebird V2.5 builds just fine on Itanium HPUX (IA-64). See prefix.hpux_aCC and prefix.hpux_ia64 in the builds/posix directory. The only issue we have in providing such builds, is access to the relevant hardware/operating system to do a build, and bringing upto date the HPUX installer.

Forced Writes?

2011-02-09T02:57:00.000-08:00

Forced Writes are turned ON by default on Windows since the release of Firebird V1.0.

See the following item on the tracker: CORE-591

But Forced Writes were OFF on Linux even if they were turned ON, until an issue with Linux (Posix) was resolved in Firebird V2.1 Beta 2 and this fix has now also ben backported to Firebird V2.0.4.

See the following item on the tracker: CORE-1476

There was some discussion amongst the Firebird developers about the change (either to OFF on Win or to ON on Linux so we would have consistent defaults), but so far there has not been a final resolution since Windows and Linux handle I/O differently, and its perfectly valid to have different default settings. Before the fix in Firebird V2.1 Linux never did forced writes even when asked to, and pretty much everybody got used to the performance with Forced Writes off, when we turned them back on correctly the performance decrement was definitely going to be a problem.

Basically, Linux writes pages out of the page cache as often as possible, so the file on disk stays pretty much current, whilst Windows writes to disk only when it must, so the difference between the disk and cache pre Firebird V1.5 could be enormous - hours or days of changes if the database file was not closed correctly. So, although it may be preferable that all systems should run with forced writes on, for Firebird on Windows it was pretty critical.

Within Firebird V1.5 we attempted to make sure that the issue of Windows cache, the Firebird cache and what was on disk could not be vastly different even if Forced Writes were turn off by introducing the following two parameters in the Firebird configuration file. However the new parameters do not guarentee complete consistency between the cache and disk, for that you do need Forced Writes turned on.

The parameters:

# ----------------------------
#
# How often the pages are flushed on disk
# (for databases with ForcedWrites=Off only)
#
# Number of unflushed writes which will accumulate before they are
# flushed, at the next transaction commit. For non-Win32 ports,
# the default value is -1 (Disabled)
#
# Type: integer
#
#MaxUnflushedWrites = 100

#
# Number of seconds during which unflushed writes will accumulate
# before they are flushed, at the next transaction commit. For non-Win32
# ports, the default value is -1 (Disabled)
#
# Type: integer
#
#MaxUnflushedWriteTime = 5

Colissimo - What a joke.

2011-02-09T02:06:00.000-08:00

If you want to send a package to anywhere in France, La Poste insist that you send it by Colissimo. Colissimo is more "expensive" than normal post, but it has insurance just in case its delivered late, goes missing or whatever you are sending gets broken.

Recently I sent a couple of Firebird mugs via Colissimo to a customer/Firebird user in France. After receipt and opening of the carefully packed box, the customer found that one of the mugs was cracked. After a brief email exchange he agreed to send me the damaged mug and I would send him a new one.

Once I had the broken mug, I asked the local post office for the basic repayment for the damaged mug from Colissimo. Their response was to hand me a form to fill in, and suggest I post the completed form direct to Colissimo. OK - job done - now I sit back and wait for them to send me the money for something they had managed to break.

After a couple of weeks a letter arrives telling me that they are looking into it, two weeks later another letter arrives, this one apologises, and says we cannot pay you the insurance because the reciever of the damaged goods should have filled in the complaint form and not the sender.

WTF? When you receive damaged goods from someone, what is the first thing you do?
Well, I would have thought, you contact the sender, ask if they want it back and ask for a replacement.... Its then the senders problem to deal with the people he used to send it.

Not in France it seems. By now we have paid for three lots of Colissimo (send original, send it back, send another) and wasted a whole lot of time. The result - huge profits for Colissimo, zero service for the customer. I now give up and throw the whole lot in the bin. Life is just too short to start aguing with a "fonctionnaire".

DEC founder Ken Olsen is dead - Silence at the Mill

2011-02-09T00:58:00.001-08:00

The Register today contained an obituary to Ken Olsen, the man who created DEC. Reading the article brings back many memories for me. The first computer I worked with in the mid 1980s was a VAX 11/750, along with Dec Professionals. We implemented a distributed accounting system (MAS-M from Hoskyns) for BRS Northern. I then moved to BRS Group where we started writing a more advanced system using the latest DEC software at the time. ACMS, TDMS, RDB etc, I was the DBA responsible for designing and implementing the corporate database. Great times, great memories. I will raise a glass in your memory tonight.

A Quick Start Guide to Index Optimisation

2011-01-21T02:53:00.000-08:00

Below a Quick Start Guide to index optimisation, courtesy of Pavel Cisar.

1. Always define ALL integrity constraints: primary, foreign and unique constraints. As this will automatically cover join optimisations.

2. Define a separate index on EACH column you will use for additional constraints in your queries (eg. WHERE clause filter conditions) that are not covered by point 1.

3. Update your index statistics regularly as you add/change/delete data in your database. A good general rule is to update statistics as part of your regular database maintenance routine (when you do a backup, sweep etc.), and/or whenever you add/change a quarter of all the rows, once the table is bigger than 10,000 rows.

4. Once you have a representative sample of real world data in your database, you can then evaluate the usability of the indices to eliminate those that will not help your queries (a worthless index will only slow down your inserts and updates!) and add new composite or expression indices that speed up specific queries.

5. Useless indices are typically those with low selectivity (few distinct values). Run some queries with filter condition on bad cardinality columns and check whether the optimizer uses the index or not. If the low selectivity index is always/often used with other conditions, you can improve it's selectivity and thus usability by
creating a composite index on columns that are used together for filter conditions instead of independent indices on these columns.

6. If you always/frequently use a group of columns for filter conditions, a composite index on these columns can boost the query performance, but do it only if you're not satisfied with performance that individual column indices provide.

Promenade - Chezelles - 2010

2010-09-06T04:54:00.000-07:00

Every first sunday afternoon at 2.00pm in September (except for the year it was cancelled due to a clash with an agricultural show), there is a "promenade gastronomique" (a walk where you taste local produce) at Chezelles. The walk this Sunday was 9km (thankfully because the thermometer was showing 29 degress centigrade in the shade).

On the walk, we tasted local wines, goats cheese, jams, honey products and beer, and had a great time.

I have never seen this "walk" advertised, but its very popular. If you are in the area next year at the same time, then it might be worth having a go.

Cost this year was 4 Euros per person.

Stirling EH924 25th/26th July 1943 Essen

2010-09-02T06:35:00.000-07:00

Its been a while since I posted anything to blog. There are no major reasons as to why I haven't posted, just haven't had anything major or interesting to say.

However while doing research about my Grandfather's RAF career I came across this comment in the 51 Squadron Operations Record book for the night of the 25th/26th July 1943.

Details of Sortie on Flight
.... "An aircraft flying immediately behind this Halifax was shot down in the Goch area. Halifax itself however returned to base without incident."

I was curious as to which aircraft was shot down, and a web search revealed the following on lostbombers.co.uk. The data on that site being taken from William Chorley's excellent books about RAF Bomber Command's losses during the Second World War. Note the information on lostbomers.co.uk is out of date and was compiled from William Chorley's books without permission from William or his publishers.

Stirling EH924 Information
Type Stirling
Serial Number EH924
Squadron 620
X1D QS-B
Operation Essen
Date 1 25th July 1943
Date 2 26th July 1943

Further Information

"Serial Range EH921 - EH961. 41 Stirling Mk.111. Part of a batch of 120 Short S.29 Stirling Mk.111. EH875-EH909; EH921-EH961;

EH977-EH996; EJ104-EJ127 Mk.111, of which EH897, EH950 and EJ106 were converted to Mk.1V. Delivered by Austin Motors Ltd between Jun43 and Sep43.
Contract No.B982939/39. Delivered to No.32MU 30Jun43, to No.620 Sqdn 11Jul43.
EH924 was one of three No.620 Sqdn Stirlings lost on this operation. See: BF511; EE906.
Airborne 2226 25Jul43 from Chedburgh. Cause of loss not established. Crashed at
Asperden, 3 km from Goch. Those killed were taken for burial in the Stadtfriedhof at Monchengladbach. They have been subsequently re-interred in the Rheinberg War Cemetery. Sgt J.D.Rathbone KIA Sgt J.H.Wallace KIA Sgt R.Wild KIA F/O J.F.Shepherd KIA Sgt J.F.Wells KIA Sgt A.Simons KIA Sgt D.H.Castling RCAF Inj Sgt D.H.Castling was confined to Hospital due injuries. No PoW No.

No other bomber lost that night crashed anywhere near Goch. So it looks as if EH924 was shot down by a German night-fighter. I hope somebody out there finds this information useful.

Dell Latitude D800

2010-04-29T03:48:00.001-07:00

I have an "oldish" laptop, a Dell Latitude D800 to be precise which I have had for a few years. I use it when on the road, and my wife uses it at home. Recently I have had problems turning it on. This morning it refused to start compeletely, some extensive googling for a possible solution, turned up this nugget of information posted by PaxRiverWino on TechRepublic.

I have copied the information below - just in case it ever goes missing.

THIS FIXES IT. Dell Latitude 800 no power, no light, does not boot

Take out the battery and unplug from the AC adapter. Open your computer so you are facing the screen. There is a grey strip behind the keyboard that is about 1 inch on the left and a half inch on the right. It will say LATITUDE|D800 on it. Pry it open on both ends. On the right you will see a plug in device. Remove it from being seated and then reseat it. Replace the strip. I'll bet you that is all it takes.

PaxRiverWino - too right - This did indeed fix it!

More detailed query plans, part 2 (textual output)

2010-02-12T00:45:00.000-08:00

Posted by Dimitry to the Firebird Developer List

Getting back to this subject, I'd like to discuss possible textual formats for the structured query plans.

Below is a sample query used to demonstrate the idea (taken from the TPC-R suite):

select first 10
l_orderkey,
sum(l_extendedprice * (1 - l_discount)) as revenue,
o_orderdate,
o_shippriority
from
customer,
orders,
lineitem
where
c_mktsegment = 'BUILDING'
and c_custkey = o_custkey
and l_orderkey = o_orderkey
and o_orderdate < date '1995-03-15'
and l_shipdate > date '1995-03-15'
group by
l_orderkey,
o_orderdate,
o_shippriority
order by
2 desc,
o_orderdate;

This is a detailed plan output that could be shown for this query. It's based on the current binary access path format:

SELECT
-> FIRST N
-> SORT
-> AGGREGATE
-> SORT
-> LOOP JOIN (INNER)
-> FILTER
-> TABLE [ORDERS] ACCESS BY DBKEY
-> BITMAP
-> INDEX [ORDERS_ORDERDATE] SCAN
-> FILTER
-> TABLE [CUSTOMER] ACCESS BY DBKEY
-> BITMAP
-> INDEX [CUSTOMER_PK] SCAN
-> FILTER
-> TABLE [LINEITEM] ACCESS BY DBKEY
-> BITMAP
-> INDEX [LINEITEM_PK] SCAN

In fact, it's the real output which works in my private tree for a couple of months already, but I don't insist on the representation (inspired by Oracle and PGSQL), so feel free to criticize.

As soon as we have the new binary access path format (discussed in part 1) implemented, the output could look like this (just an example):

SELECT
[cost: 360000, rows: 10]
-> FIRST N (100)
[cost: 360000, rows: 10]
-> SORT ( DESC, o_orderdate ASC)
[cost: 360000, rows: 100]
-> AGGREGATE (SUM)
[cost: 350000, rows: 100]
-> SORT (l_orderkey ASC, o_orderdate ASC, o_shippriority ASC)
[cost: 300000, rows: 75000]
-> LOOP JOIN (INNER)
[cost: 150100, rows: 75000]
-> FILTER (o_orderdate < date '1995-03-15')
[cost: 75050, rows: 75000]
-> TABLE [ORDERS] ACCESS BY DBKEY
[cost: 75050, rows: 75000]
-> BITMAP
[cost: 50]
-> INDEX [ORDERS_ORDERDATE] RANGE SCAN
[cost: 50, used segments: 1]
etc

Another question I'd like to raise is the API to get the textual plan representation. fb_info_sql_access_path is expected to return the binary access path. We could have another tag e.g. fb_info_sql_access_path_as_text which works similar the current isc_info_sql_plan, i.e. performs the transformation on the server.

Another option could be to follow the fb_interpret() way and offer a client-side (actually, Y-valve) API call which would perform the binary-to-text conversion. The latter approach may look unreliable in the case of client/server version mismatch, but the worst possible thing for the client would be to get a reduced plan with unknown items printed as e.g. .

Comments?

RFC: Concept of the stored table/index statistics

2010-02-12T00:38:00.000-08:00

Dimitry posted the following for discussion to the Firebird developers list.

Currently, the only stored statistical information is the index selectivity (per segment since ODS11). Number of records (aka cardinality) in tables is estimated using the number of data pages and table format size (aka table width). This information is far not enough to allow the optimizer making good decisions.

From another side, we have GSTAT which returns much more information which is very useful by itself (to DBA or developer) but which could also be used by the optimizer. And v3.0 is already offering even more details in the GSTAT output.

I was thinking about combining these two approaches together.

In the proposed new world, statistics would be stored inside the database in a binary form (read: as a blob) along with its header which includes: format version number, timestamp of its collection and probably some other fields. We could offer a built-in blob filter which translates the binary data into the textual form (e.g. looking like the GSTAT output).

It would contain all the data that GSTAT is currently able to report and even more, surely extensible in the future. It would consist of two parts: table statistics (complete -- including fields statistics, or reduced -- without fields) and index statistics (perhaps also in complete and reduced form, where complete one would contain e.g. value distribution histograms). I'd store them in RDB$RELATIONS.RDB$STATISTICS and RDB$INDICES.RDB$STATISTICS but the latter name is already in use. RDB$STAT_INFO? A separate table?

The optimizer would use that stored statistics to find better execution plans. If the statistics is considered being invalid/outdated, it could default to some simpler calculations, like the ones used currently, or it could still use the outdated statistics. There may be different rules for such a consideration, e.g. number of records in the stats vs the quick estimation based on data pages, or too old timestamp, or too big mismatch between the estimated cost and the real one calculated at runtime, etc. Threshold values could be configurable per database. An
invalid/outdated statistics would also trigger its re-calculation in the
background.

GSTAT gets new switches that would be used to:

(a) re-collect the statistics from disk and show it (as it works now)
(b) re-collect the statistics from disk and store it in the database
(c) show the statistics stored currently
(d) invalidate the stored statistics thus forcing a delayed re-scan

I'm not sure whether the default behaviour should be legacy (a) or (c).

It also gets sub-options that could control the level of details we need: only table level, including columns, including histograms, etc.

We could also add appropriate SQL commands to the engine, e.g.:

SET STATISTICS [FOR] {TABLE | INDEX} [options]
DROP STATISTICS [FOR] {TABLE | INDEX} [options]

or:

ALTER TABLE {SET | DROP} STATISTICS [options]
ALTER INDEX {SET | DROP} STATISTICS [options]
ALTER DATABASE {SET | DROP} STATISTICS [options]

or whatever. The current SET STATISTICS INDEX could be kept intact for backward compatibility or adapted to the new semantics.

Only table owners and DBA would be allowed to update/reset the stored statistics.

As you can see, there are many details that deserve discussions. I've intentionally omitted kinds of statistical values that might be stored and how they could be used.

But before going into the details, I'd like to have a basic feedback whether it's considered being a good concept or not.

I don't pretend to have the entire work completed any time soon, but I'd do my best in setting up the core infrastructure (which could later evolve into something wider) in v3.0.

Comments please.

Type 7 database pages and Firebird 2.x

2010-02-08T00:39:00.000-08:00

Norman Dunbar posted the following on the Firebird Development list:

"I'm documenting the internal page formats of a database for the Doc project. I've checked in (to cvs) the document so far, but obviously a half finished document is no good in real life. To this end, I have a couple of questions on the internal formats of the type 7 database pages for Firebird 2.x, if you don't mind:

As before I have read Ann's (1.5?) documentation over at IBPhoenix/R&D, however, Firebird 2 seems to have changed things (slightly).

What compression is used in the page/btree_nod entries? I've got a database with a couple of known entries and I cannot decode the btree_nods sensibly. Happy to be pointed at the appropriate code file in the source.

Is there a difference in the page layout at all if the page I'm examining is not a leaf page? (btr_level != 0)"

Dimitry Yemanov replied.

"This isn't going to be easy to answer, as the page layout is much different between ODS10 and ODS11.

In all ODS incarnations, prefix compression is used for index keys. It means that the first key is stored "as is" and subsequent keys are stored as "deltas" represented by three values:
(1) length of the data that should be taken from the prior key (aka prefix),
(2) length of the data that is stored in our key (aka suffix), and
(3) the suffix itself which length is described by (2).

In ODS11, key internals (prefix length, suffix length, page number, record number) are also kinda compressed to store only the significant part of an appropriate integer value.

In ODS10, you'll see them of the fixed size:

struct btree_nod
{
UCHAR btn_prefix; --> prefix length
UCHAR btn_length; --> suffix length
UCHAR btn_number[4]; --> page or record number
UCHAR btn_data[1]; --> suffix data of btn_length bytes
};

Also, ODS11 has some special flags in the first byte of the index entry which allows to avoid storing prefix/suffix values at all in some cases.

Relevant source code: jrd/btr.cpp and jrd/btn.cpp.

> Is there a difference in the page layout at all if the page I'm examining is not a > leaf page? (btr_level != 0)

They're nearly identical. IIRC keys on non-leaf pages contain both page numbers and record numbers.

Also beware about the jump nodes introduced in ODS11. It's a sparse lookup table (key -> offset) which is stored on the page along with the keys themselves."

Ann Harrison also replied.

> They're nearly identical. IIRC keys on non-leaf pages contain both page numbers
> and record numbers.

That's right. Leaf pages contain nodes that have a header, data - possibly compressed, and the record number of the record that corresponds to the data. Upper level pages in ODS11 contain nodes that have a header, data, record number that contains that data, and the page number of the lower level page that starts with that
data value. The reason for "promoting" the record number to the upper level is not to give faster access to the record during a normal search, it's to avoid a garbage collection problem with indexes with lots of duplicates.

In ODS-10 and earlier, the algorithm for storing duplicates put the newest record first. So if you stored 100,000 records with the same key value and a generated primary key, the records would be stored in primary key order (more or less) and the index entries would be stored in inverse primary key order. When you delete
those records, you delete the lowest primary key value first. Unfortunately, the index entry for the key with duplicates will be at the end of the chain of duplicates, so you have to read 99,999 entries before you find the one you want.

Then you delete the second record and read 99,998 entries, etc. It's called
thrashing the cache.

In ODS-11 and higher, the record number becomes part of the key. Duplicates are stored in record number order. In effect, every key is unique.

> Also beware about the jump nodes introduced in ODS11. It's a sparse lookup table
> (key -> offset) which is stored on the page along with the keys themselves.

Jump nodes are like an index into an index page. Doesn't seem to make sense, but with pages larger than about 4K, Firebird was spending an inordinate amount of time reading across index pages - reading on average 1/2 the page size at each level. Jump nodes reduce the average read to about 500 bytes. As it turns out, the optimal size for an index page is different from the optimal size for data and jump nodes are a way of getting the best performance for both.

Note that systems that don't compress keys can use a binary search on an index page, so the size of an index page doesn't matter as much. On the other hand, they pay for that binary search in I/O.

Solaris, Firebird and Robust Mutexes

2010-02-05T01:46:00.000-08:00

We have a large Firebird user on Solaris who noticed the following problem with the cuurent Solaris build (pre 2.1.4)

"If there are a bunch of fb_inet_servers running (or any other app like isql, Gpre type apps etc), then it is possible to kill one or more of these processes and hang up all the rest.

I suspect (hunch only) that some mutex or other has been created, and the killed processes can't release it...

The easiest way to get the problem to appear is to create 100 or so busy processes, and to start killing them until the problem appears.

Be nice if you had an idea of how to sort this.."

Cue conversation with Alex about the issue.

"This is known issue, though we have never been able to reproduce it, except using a debugger to stop in particular place and then kill the process. If some process locks a global mutex in the lock (or event) manager, and for some reason (e.g kill) the process dies when the mutex is still locked, then the mutex remains locked
forever. Non SolarisMT ports (like Linux or HPUX) do not have this problem.

The problem is solved in Firebird V2.5 and I think we can backport it to older versions, because it's well localized (related to mutex initialization), and it also seems it requires Solaris 10, but I am not sure whether the required system calls are present in the base release or whether an upgrade is required."

For reference - this is the code in Firebird 2.5, that fixes the issue:

#ifdef HAVE_PTHREAD_MUTEXATTR_SETPROTOCOL
int protocolRc = pthread_mutexattr_setprotocol(&mattr,
PTHREAD_PRIO_INHERIT);
if (protocolRc && (protocolRc != ENOTSUP))
{
iscLogStatus("Pthread Error", (Arg::Gds(isc_sys_request) <<
"pthread_mutexattr_setprotocol" <<
Arg::Unix(protocolRc)).value());
}
#endif
#ifdef USE_ROBUST_MUTEX
LOG_PTHREAD_ERROR(pthread_mutexattr_setrobust_np(&mattr,
PTHREAD_MUTEX_ROBUST_NP));
#endif
(this is mutex init code) and

#ifdef USE_ROBUST_MUTEX
if (state == EOWNERDEAD)
{
// We always perform check for dead process
// Therefore may safely mark mutex as recovered
LOG_PTHREAD_ERROR(pthread_mutex_consistent_np(mutex->mtx_mutex));
state = 0;
}
#endif

(this is checked if the mutex lock returns an error)

To make sure we can use this code Solaris must support the PTHREAD_MUTEX_ROBUST_NP attribute.

The answer to this is yes - Solaris does support it.

So we backported the relevant code and started the build only to find the following compile error

../src/jrd/isc_sync.cpp: In function 'int ISC_mutex_init(mtx*, SLONG)':
../src/jrd/isc_sync.cpp:3026: error: 'LOCK_ROBUST' was not declared in this
scope
../src/jrd/isc_sync.cpp: In function 'int ISC_mutex_lock(mtx*)':
../src/jrd/isc_sync.cpp:3049: error: 'mutex_consistent' was not declared in
this scope

To fix this you need to upgrade to libc version SUNW_1.23 as this was implemented in 2008 sometime.. see this link.

Stoves

2010-01-19T02:04:00.001-08:00

Thats both a cooker and a maker of them. Follow this link.

I bought a Stoves 700DFD0 cooker, gas hob, grill, double oven in December 2003, and paid quite a chunk of money for it. seven years on, I would like my money back. Never will I purchase anything from Stoves again.

The top oven/grill of my Stoves freestanding double oven has now broken down / stopped working for the third time in four years, each time the same problem and the replacement of the same part, the thermostat. On previous occasions we have a, got a repair company to fix it, b ordered the part ourself, and got a friend to fix it, and finally this time after it failed after nine monthes we got our electrician/repair man, who is a good friend, to speak directly to them and try and find what was going on. It seems that this part has now been replaced by a new design, not because it keeps breaking down, just because ...it has.

So we ordered the new part directly from Stoves and he came round to repair the cooker. The new part we had been sent is also faulty. As is the replacement part (new thermostat) for the main oven (which currently has the endearing, if somewhat dangerous habit, of turning itself on when it feels like it).

Other issues, besides the fact it breaks down on a regular basis. The stainless steel hob gets stained by any burnt food and is uncleanable. Gas Burners which don't sit properly and hence won't light. Wire cooking racks which no longer fit into
their supports. An oven temperature 20 degrees higher than it should be AFTER allowing for the fan and which is hotter at the back than the front.

So if you want a new cooker, think hard about buying a "Stoves".

Physical ODS Changes for ODS 12

2009-10-26T01:57:00.000-07:00

Posted to the Firebird Development list by Dimitry Yeamanov:

Below is the initial proposal (to be corrected/extended by others) that
describes the physical ODS changes we'd like to see in ODS 12.

Header page

* rework implementation ID to simplify the porting split into endianness and swap_double flags?)

* deprecate page buffers and sweep interval (in favor of the per-database values defined in .conf)

Pointer page / data page

* add "swept" flag
(allows to skip pages that have no dead or backversions, thus improving the GC / sweep performance)

Page inventory page

* use its own field instead of common "pag::reserved" (related to the misuse by PAG_allocate() happened in ODS 11.1)

New page type (bitmap?) to link SCNs to page numbers.
This may noticeably improve the incremental NBackup performance, avoiding reading pages not changed since the last backup.

Switch from SLONG to ULONG for transaction numbers and page numbers.

Extend blob length field from ULONG to UINT64 (and maybe allow blob levels higher than 2?).

Consider better alternative (instead of SQZ) for record compression.
Tests are required.
If not found, extend SQZ for sequences longer than 127 bytes.

Consider better packing of index key data (record numbers) and compound index keys (described by Jim in fb-architect).
Tests are required.
Extend the ACL format to cover new permission types and objects.

Additions are welcome, as well as questions/comments re. the
aforementioned items.

System Catalog Additions for ODS 12

2009-10-26T01:25:00.001-07:00

The following appeared in the Firebird Development List from Dimitry Yemanov:

Here I'd like to have the possible additions to the system tables
discussed and agreed on. Physical ODS changes will be discussed in a
separate thread. As you understand, the sooner we have it resolved, the
easier it would be to set up the time frame for the v3.0 Alpha release.

I'll start with my own proposals, expecting others to follow up with
both comments/critique and their own suggestions. The changes already
done by Adriano are not included here.

RDB$FIELDS
+ RDB$SECURITY_CLASS (standard USAGE privilege for domains)
+ RDB$OWNER_NAME (prohibits altering/dropping domains by non-owners)

RDB$RELATIONS
+ RDB$STATISTICS (blob with binary statistics for the optimizer)

RDB$FUNCTIONS
+ RDB$FUNCTION_ID (allows metadata versioning for PSQL functions)
+ RDB$FUNCTION_BLR (BLR of a PSQL function)
+ RDB$VALID_BLR (BLR validation flag)
+ RDB$DEBUG_INFO (debug information)
+ RDB$LEGACY_FLAG (flag indicating whether this function is legacy
UDF style or new style)
+ RDB$SECURITY_CLASS (standard EXECUTE permission for functions)
+ RDB$OWNER_NAME (prohibits altering/dropping functions by non-owners)

RDB$FUNCTION_ARGUMENTS
+ RDB$ARGUMENT_NAME (input argument name)
+ RDB$FIELD_SOURCE (reference to a domain describing argument)
+ RDB$DEFAULT_SOURCE (default value for argument)
+ RDB$DEFAULT_BLR (default value BLR for argument)
+ RDB$COLLATION_ID (collation ID override for argument)
+ RDB$NULL_FLAG (null flag override for argument)
+ RDB$ARGUMENT_MECHANISM (full domain or TYPE OF)
+ RDB$FIELD_NAME (field name for TYPE OF)
+ RDB$RELATION_NAME (relation name for TYPE OF)

RDB$FILTERS
+ RDB$SECURITY_CLASS (USAGE privilege -- non standard, but probably
useful?)
+ RDB$OWNER_NAME (prohibits altering/dropping filters by non-owners)

RDB$GENERATORS
+ RDB$START_VALUE (start value per SQL standard)
+ RDB$INCREMENT_VALUE (increment value per SQL standard)
+ RDB$MIN_VALUE (minimum value per SQL standard)
+ RDB$MAX_VALUE (maximum value per SQL standard)
+ RDB$CYCLE_FLAG (cycle option per SQL standard)
+ RDB$SECURITY_CLASS (allows to have the standard USAGE privilege for
sequences)
+ RDB$OWNER_NAME (prohibits altering/dropping sequences by non-owners)

RDB$EXCEPTIONS
+ RDB$SECURITY_CLASS (USAGE privilege for exceptions -- non standard,
but probably useful?)
+ RDB$OWNER_NAME (prohibits altering/dropping by non-owners)

RDB$CHARACTER_SETS
+ RDB$SECURITY_CLASS (standard USAGE privilege)
+ RDB$OWNER_NAME (prohibits altering/dropping by non-owners)

RDB$COLLATIONS
+ RDB$SECURITY_CLASS (standard USAGE privilege)
+ RDB$OWNER_NAME (prohibits altering/dropping by non-owners)

As you can see, almost everything shown above is related to either
enhanced metadata security or implementation of PSQL functions.

I don't pretend to add all the aforementioned functionality in v3.0,
these fields can be reserved until actually used later.

I have intentionally avoided the "schemas / longer identifiers" topic
here, as IMO it deserves a separate discussion. Is anybody willing to
start it? :-)

New monitoring tables/fields will also be discussed separately.

Opinions?

Adriano, if you intend to switch from UNICODE_FSS to UTF8 in ODS12
system tables, let's please discuss it as well (here or in a different
thread).

USE_BLOCKING_THREAD

2009-08-24T02:53:00.000-07:00

Another interesting email conversation between a developer and Dimitry Yemanov (Firebird core developer) produced this information:

Q.
What is the intent for the USE_BLOCKING_THREAD? I see it is enabled for all configurations except superserver. What is the rationale for that?

A.
The blocking thread (sometimes referred to as the AST delivery thread) listens for the blocking event notification posted by other processes and calls an appropriate BAST (blocking AST) routine inside the local process. This is necessary to asynchronously release or downgrade locks acquired by the current process but requested by other processes and is a vital part of the lock manager.

SuperServer doesn't need this feature as it performs direct AST calls from the lock requester to the lock holder (as everybody lives inside a single process).

The v2.5 embedded engine allows both in-process and cross-process interaction, thus it has the blocking thread enabled. For a single process embedded engine, USE_BLOCKING_THREAD may be undefined.

Cache flushes to disk and Firebird

2009-08-14T03:30:00.001-07:00

The following fragment of a message appeared on the Firebird development list. As the question and answer are interesting, I thought I would put it on the blog.

"If I want to make cache flushes to disk very rare, do I set MaxUnflushedWrites & MaxUnflushedWriteTime to -1 or to some large value?
What other settings adjustments can make table flushes to disk more rare?
Is there any combination of settings that will keep the entire db in cache and never flush it to disk?"

Dimitry Yemanov answered as follows:

Forced writes = OFF (at the database level)
MaxUnflushedWrites = -1,
MaxUnflushedWriteTime = -1 (in firebird.conf)

With these settings, Firebird will never explicitly flush the cache to disk. However, flushes may still be performed by the operating system in the background. I believe you understand that these settings are somewhat risky from the database consistency point of view. There is no way to completely avoid flushes at all.

Firebird /Java Plugin Milestone One

2009-07-06T00:13:00.000-07:00

Posted by Adriano in Firebird Development:

"I'm pleased to announce availability [1] of the first test version of
FB/Java plugin.

FB/Java is a plugin for FB external engines feature, to be presented in
v3.0, that allows execution of Java code at the server side. The plugin
allows classes/resources in the file system or stored in the database.
File system classes/resources are global and the code run with all
permissions. Database classes/resources are per-database and code runs
accordingly with per-engine configured security policy.

The plugin has been made with the assumption that anything could be done
using existing FB infrastructure. You don't need to setup FTP server or
shared directory in the server to update your code or edit security police.

Installation/update of database JARs works from client command line
utility or from SQLJ package procedures. Actually, the same Java class
that run in the client is the one that runs in the server. You can put
JARs in the database using a Java URL, so lookup of them is not
restricted to client or server.

Security handling is configured in a central database, with a single
PERMISSION table. Permissions are granted to users that executes the
Java code. This way to configure permissions is not adequate for every
(or most) cases, and roles are not supported at the moment. I suppose
and desire we can have a better design in the future, before a final
release.

It's welcome any attempt to bypass the security system. But please,
report it. :-)

This version also lacks some notification mechanism for when the code
are going to be unloaded. You may note the problem in the example, right
after disconnect all attachments from the database the log file remains
locked, as it wasn't been garbaged collected. And no kind of "hot
redeployment" support had been implemented or tested. However, I have
ideas of how these problems can be solved in very good ways, based on
existing models. For unloading (or any other type of notification),
JPA-like listeners. For maintain state after redeployments, serializable
sessions.

In the file plugins\java-engine\examples\fbjava\README, you found
instructions for a "quick start" with it. In
plugins\java-engine\examples\tests there are functional tests for many
of its feature (i.e., how type mappings works, etc).

The plugin is based on Jaybird 2.2 (HEAD). I've put it in a maven
repository at [2], with its version defined as 2.2-jrt-SNAPSHOT.

The server includes my v3.0 changes, supporting packages, DDL triggers,
over clause and some internal changes. It may have others instabilities.

If you have a JAVA_HOME environment variable defined, it should just
works without any other configuration. The quick start assumes you had
defined FIREBIRD, ISC_USER and ISC_PASSWORD. In [1], I've zipped plugin
and engine together, but plugin installation consists in just unpacking
it in FB directory.

Documentation has room for improvements, and is at [3]. I've done it
with FB docbook tools.

Source code is in [4].

Happy testing and thanks.
Adriano"

[1] http://www.firebirdsql.org/downloads/rabbits/asfernandes/fbjava/fbjava-m1.zip
[win32]
[2] http://www.firebirdsql.org/downloads/rabbits/asfernandes/maven2
[3] http://www.firebirdsql.org/downloads/rabbits/asfernandes/fbjava/fbjava.pdf
(included in [1])
[4] https://firebird.svn.sourceforge.net/svnroot/firebird/java-plugin/tags/2.2-jrt-M1

IBPhoenix Web Site

2009-06-30T00:35:00.001-07:00

I need to post a note on my blog about the IBPhoenix Web site, it seems we are having problems with the T1 line. It looks like there is a problem in the central phone office, where our T1 line originates. It's been a problem now for several days. The issue has been escalated, so hopefully things will improve shortly.

../src/include/types_pub.h:88: error: conflicting declaration 'typedef int ISC_LONG'

2009-06-25T04:16:00.000-07:00

Building Firebird 64bit on Solaris (sparc or intel) can sometimes produce the following error:
-c ../src/jrd/dsc.cpp -o ../temp/boot/jrd/dsc.o
../src/include/types_pub.h:88: error: conflicting declaration 'typedef int ISC_LONG'
../src/include/fb_types.h:69: error: 'ISC_LONG' has a previous declaration as 'typedef SLONG ISC_LONG'
../src/include/types_pub.h:89: error: conflicting declaration 'typedef unsigned int ISC_ULONG'
../src/include/fb_types.h:70: error: 'ISC_ULONG' has a previous declaration as 'typedef ULONG ISC_ULONG

If you see this - then you made a mistake in the build process. Configure.in when you invoked it, assumed that the size of long was 4, and you are attempting to build Firebird for a 32bit environment. As per the prefix.solaris files, you must set the following before running configure:

export CXXFLAGS='-m64'
export CFLAGS='-m64'
export LDFLAGS='-m64'

Configure will now automtically realise that you want to build in a 64 bit environment and set the size of long to 8.

Some Positive Feedback for Firebird

2009-06-24T23:33:00.001-07:00

This message appeared on the Firebird Development list, from Jens Zurawski

"I think from time to time, one should really send some positive feedback
to the people who are involved into the development of the firebird
DBMS. Thank you very much for your efforts to make this software this good!

Just before a week or so we eventually upgraded our production server
from linux kernel 2.4 to 2.6.x and the firebird engine from 2.0.x to
2.1.2. As some of you might remember, long time ago we had some big
troubles with kernel 2.6.x and firebird, because our system is under
very heavy contention and we stumbled over many freezes, so we were
stuck to the 2.4.x kernel. (In average there are 75-150 simultaneous
connections to one database, in peak times up to 300. The database is
more than 20GBs in size and is serving for a pool of approx. 7 different
applications, starting from web server (php/java) over some internal
logistics processings (java) up to the CRM front end (Win32 native), and
additionally there are frequent update tasks with long running
transactions (about 10-30 minutes) and massive changes to the database
records in them).

Now our old database server broke down (hardware problem) and we had to
buy a new one and this time we needed to go for a 2.6.x kernel because
the new hardware was not supported from the old kernel. To my pleasant
surprise, the actual FB 2.1.2 is working very well with a 2.6 kernel and
the heavy contention we still have on our database. Only thing one has
to do is to dramatically increase the semaphores. In the first place we
increased the LockSemCount to 250 (which was the limit of the default
kernel setup) but that wasn't enough. After half of the day firebird
crashed down with a "Fatal lock manager error: semop failed (acquire),
errno: 22", accepted no more new connections and also didn't serve the
existing connections any more. Now after we increased the kernel limit
and set the LockSemCount up to 1024, everything is working very well. No
more freezes, no more crashes. (BTW: ironically this hint I was getting
from a guy who just has collected some experiences with an Oracle
installation on Linux ;-)

So, what I want to say: Thanks a lot for all the good work! firebird is
really my most favourite open source project.

Ah, and the new monitoring tables in FB 2.1 are a gift from heaven! Now
I'm able to track down problems from applications which are not written
by me, because now I can simply monitor hanging transactions or long
running "sub-optimal" queries and send the developer a hint ;) Also
thanks a lot for this feature!"

64 bit Firebird 2.5, and Solaris Sparc

2009-05-07T00:18:00.000-07:00

Looks like the job is already done... I read the following post on Firebird Development from Bill Oliver (SAS)

"I confirm that the issues are fixed in HEAD. I confirmed this on 64-bit
Solaris Sparc, built with the Sun Studio native compiler - using pthreads!!!

"How cool is that! Thanks very much guys for the quick turn-around.

Also, big thanks to Alex for his assitance with this port!!"

Excellent work!

GCC 4.01 MacOSX & Weak Symbols

2009-04-22T03:01:00.000-07:00

During the QA process for Firebird V2.1.2 we found that there was a problem with creating a database remotely. In effect the inet server would crash.

On further analysis we found that we have a problem where the function free()(standard C dynamic memory deallocator) was being invoked with the incorrect parameter, an address which had not been allocated using malloc().

After setting up the code to make the inet server call abort we got the following stack trace

(fragment)

#0 0x90005efc in free ()
#1 0x010440b8 in Firebird::AbstractString::~AbstractString (this=0xbfffcb04)
at ../src/include/../common/classes/fb_string.h:375
#2 0x010440f8 Firebird::StringBase::~StringBase
(this=0xbfffcb04) at ../src/include/../common/classes/fb_string.h:391
#3 0x0101593c in LD_setup_attributes (textTypeName=0xbfffcd60 "ES_ES",
charSetName=0xbfffccdc "ISO8859_1",
configInfo=0x6a1204 "filename=$(this)/fbintl;icu_versions=default",
srcLen=39, src=0x69a898 "DISABLE-COMPRESSIONS=1;SPECIALS-FIRST=1",
dstLen=512, dst=0xbfffcdb8 "DISABLE-COMPRESSIONS=1;SPECIALS-FIRST=1,")
at ../src/intl/ld.cpp:457
#4 0x3011b1b4 in Jrd::IntlManager::setupCollationAttributes
(collationName=@0xbfffd0e4, charSetName=@0xbfffd0b8,
specificAttributes=@0xbfffd14c, newSpecificAttributes=@0xbfffd178)
at ../src/jrd/IntlManager.cpp:317

From this back trace you can see that free() is called from the operator delete[], where we would have expected MemoryPool::deallocate() to be used. For some
reason the default operator delete[] is used.

Strange, how did this code arrive at address 0x010440b8?

After an analysis of nm output for fbintl and the linker's map of symbols Alex found that at this address is in fact located the redefined operator delete[], which was compiled in via ld.cpp. However in the assembly version of the file (ld.cpp), it was correctly redefined (i.e. it was calling MemoryPool::deallocate()) operator delete[].

Houston it looks as if we have a buggy toolchain.

But we also need to take into account that AbstractString dtor() and operator delete[] are also WEAK symbols. Weak meaning that such symbols may be present in many object files, and the linker at link-time will take it from the relevant one. But in this case it looks as if it was taken from another file...

But after we compiled into assembler all of the files that are used in the fbintl
library we found that they all contained the correctly redefined operator
delete[]. As such there is no use of operator delete[], calling std free() function in our code, but anyway we get such use in the resulting binary.

In effect it looks like weak symbols are broken on MacOSX GCC 4.01 (Apple build 5747). And the only way to fix a problem is to avoid them.

We are currently compiling Firebird on PPC 10.4 (Tiger) using a new switch -fno-weak

fno-weak:
"Do not use weak symbol support, even if it is provided by the linker. By default, G++ will use weak symbols if they are available. This option exists only for testing, and should not be used by end-users; it will result in inferior code and has no benefits."

MallocLogFile, MacOSX & LaunchDaemons

2009-04-16T08:07:00.001-07:00

I have been debugging the fb_inet_server trying to find a problem in Firebird 2.1.2 Classic for PowerPC. That takes a little bit of setting up, but I will try and explain that in another blog post. Part of one of the things that I wanted to check was what malloc was doing. However by default any "error" messages get sent to standard out, however you can set the environment variable MallocLogFile to capture messages to a specified file. But setting that in my normal environment is going to work because the fb_inet_server is actually launched by the LaunchDaemon and is then also forked. So we need to set the environment variable in the LaunchDaemon script directly.

The solution:

You need to edit the xml script that "launches" the fb_inet_server. This file can be found in /Library/LaunchDaemons and is called org.firebird.gds.plist

First of all unload the current launch definition

launchctl unload /Library/LaunchDaemons/org.firebird.gds.plist

Then edit the file as follows
After

<string>org.firebird.gds</string>

Add

<Key>EnvironmentVariables</Key>
<dict>
<Key>MallocLogFile</Key>
<String>/Library/Frameworks/Firebird.framework/Malloc.log</String>
</dict>

Save and

launchctl load /Library/LaunchDaemons/org.firebird.gds.plist

Modern Music

2009-03-04T10:23:00.000-08:00

The Brits - Hmmm - Not my list of recommended music....
Try listening to:

1. The Killers (excellent)
2. Editors (good)
3. The View (acquired taste)

Enjoy

Bosmere Products Ltd

2009-02-23T03:20:00.000-08:00

I don't normally do this, but about a month ago I sent the following letter to a company in the UK. I am still waiting for a reply. I am not expecting to receive one, nor do I expect a refund.

So if anybody out there is googling for Patio Set Covers, and happens to bump into this post...

We had bought a patio set cover for our patio tables and chairs in November 2008, and covered them up using this cover. By February 2009, the cover was falling apart.
Letter follows.

Customer Services
Bosmere Products Ltd,
Unit 5 Mitchell Way,
Airport Service Road,
Portsmouth,
PO3 5PR,
England.

Dear Sir,

I am writing to express my disappointment with the quality of your 6 seater circular patio set cover.

I purchased this cover from Chinnor Garden Centre, Chinnor, Oxfordshire for £39.99 on 4th November 2008 while on a short trip to the UK from my home in France.

Within a week the first of the supposedly heavy duty ties had broken – followed by the other 3 within the next month. Attempts to re-tie the broken cords were useless (they simply snapped again) and now the metal eyelets have ripped away from the plastic cover, now making it impossible to secure the cover even with heavy duty polypropylene rope which I purchased.

On removing the cover, it is also apparent that large areas of the plastic coating on the inside of the cover have completely worn away and it is no longer waterproof.
We do not live in the Outer Hebrides but south of the Loire valley in France, therefore it has most certainly not been exposed to unusually severe weather here!
It is clear that this product is not fit for purpose – it is marketed as being top quality PVC backed polyester, secure in windy conditions and giving full winter protection – it is obvious that after just two and a half months use that it is none of these.

As we live in France and very seldom travel to the UK we are unable to return the cover to the place where we purchased it. I would ask you therefore to refund the cost of the cover.

Yours faithfully
Mr Paul Beach.

Daily Telegraph Cryptic No 25851

2009-02-13T03:41:00.000-08:00

I am a great fan of the Daily Telegraph Cryptic crossword, and have been doing, or attempting to do this crossword, just about every day since I was a student at University.

I was introduced to it by my Grandmother many years ago, when stayed with her in Banbury.

"We" (and I use We, advisedly) used to do the crossword when we had breakfast, usually my Grandmother doing the crossword and explaining how things worked, whilst every now and then I would have a eureka moment. Anyway enough digression.

I found the following web site that specialises in helping people through the Telegraph Cryptic crossword in a way that is much more friendly than normal "specialist" cryptic sites.

Big Dave's Telegraph Crossword Blog

For those of you who don't know Libellule, is french for Dragonfly.

Memory Utilisation by the Firebird Classic Server

2009-01-30T04:04:00.000-08:00

The following was posted by Dimitry Yemanov on the Firebird Development list in response to some questions about Memory Usage within Firebird (Classic)

"Need some help understanding how memory allocation is done per process on Linux 1.5.5 CS. It's understandable that there is some amount of working memory needed to support fb process, plus per connection amount, plus cache depending on database page size and buffer count."

Per process (independent) memory usage:
- page buffers (40MB in your case)
- metadata cache (cannot be predicted/controlled, depends on the schema complexity, especially number of interdependent PSQL objects)
- sort buffers (turned off by default on Classic, see SortMemUpperLimit in firebird.conf)
- undo log (cannot be predicted/controlled, wastes memory mostly for multiple batch updates in the same transaction)

Shared memory usage (depends on other connections):
- lock table (starts with LockMemSize and auto-growing)
- event table (starts with EventMemSize and auto-growing)

"What makes me wander is how concurrent user connection count affects memory utilization per process. Somehow the more users we have on the server - the bigger overhead is added per connection. When comparing mem utilization on same database having single connection and 55 for example - utilized memory delta increases by ~33MB. On server having 160 connections it's almost 2-3 times more. Is this lock manager related data? 30MB just for that?"

The lock manager data is shared, but it's mapped into the address space of the every Classic process. I dunno whether the Linux tools include the mmap'ed memory into the counters you check or not. If so, it gets easy to explain: yes, this increase is caused by the lock table. You can draftly calculate the lock manager memory requirements by this formula:

number of connections * number of page buffers * 100 bytes.

The exact size could be found in the output of fb_lock_print (header section) at runtime. With your quite large (for Classic) page cache settings, the numbers should be nearly the same as you experience.

Firebird Trace and Audit Services

2009-01-30T03:58:00.000-08:00

Posted by Vlad Horsun to the Firebird Development List.

I am glad to introduce new Firebird facility, based on TraceAPI initially developed by Nickolay Samofatov and maintained and given to us by RedSoft (thanks!).

Below is brief overview of what was done. It will be included into HEAD soon, I hope.

Trace session - set of engine-generated events to trace and placement of trace output. Trace session also have its unique ID, optional name, state, flags, creator user and creation timestamp.

List of available trace events is fixed and determined by the Firebird engine. List of events to trace, which data items to trace and placement of trace output is specified by trace configuration when trace session created.

There are two kinds of trace sessions: system audit and user trace. System audit session is started by the engine itself and obtains configuration from the text file. This file name is set via new setting in firebird.conf and by default is "none", i.e. no system audit is configured. There may be only one system audit trace session, obviously. Configuration file contains list of traced events and placement of trace log(s). It is very flexible and allows to log different set of events
for different databases into different log files. Example configuration file is attached.

User trace session is managed by user via Services API. There are five new services for this purposes - to start, stop, suspend, resume trace session and to enlist all known trace sessions.

When user application starts trace session it passed optional name and (mandatory) session configuration. Session configuration is a text. Rules and syntax the same as for configuration file mentioned above except of log placement. Engine stores output of user session in set of temporary files and deleted it automatically. After application starts user trace session it must read session's output from service (using isc_service_query). When application decide to stop its trace session it just do detach from service. Also user may manage (suspend\resume\stop) any own trace session. Administrator allowed to manage any trace session.

If user trace session was created by ordinary user it will trace only connections established by this user.

User trace sessions is not preserved when all Firebird processes is stopped. I.e. if user started trace session and Firebird SS or SC process is shutted down, this session is stopped and will not be restarted with new Firebird process. For CS this is not the case as user trace session can't live without connection with service manager and dedicated CS process.

There are three main use cases:

1. Constant audit of engine.
This is served by system audit trace.

2. On demand interactive trace of some (or all) activity in some (or all) databases.
Application start user trace session and show traced events to user in real time on the screen. User may pause\continue trace and at last stop it.

3. Aggregate of some activity for a relatively long period of time (few hours or even whole day) to analyse it later.
Application start user trace session and save trace log in file (or set of files). Session must be stopped manually by the same application or by another one.

Firebird's service manager utility (fbsvcmgr.exe) may be used to manage trace sessions. Later specialized console utility will be introduced (at least i plan to develop it soon after beta). Of course third party vendors are welcome to implement own GUI based monitoring and trace applications.

InterBase 2009 comes with multicore support

2008-09-09T00:56:00.000-07:00

I have just read the following news article on SDTimes:
InterBase 2009 comes with multicore support
A couple of aspects to this story seeme a little peculiar, while some comments require some clarification.

"Thus the update for InterBase 2009 is relatively subdued"

correct, I do not see any major new features in this new version, however a couple of new features highlighted are

"such as over-the-wire encryption and AES as an encryption scheme".

If you want over the wire encryption for Firebird you could use Zebedee. There have been a number of papers re. this in the past. However this is something that could be implemented in Firebird fairly easily if we so wished.

In terms of database encryption there is code to allow it now, however the original implementation was not reliable. We did put together a more detailed proposal, but after some lengthy discussions it was decided that such functionality was going to be too complicated for the majority of users, and most of them didn't want it any way.

I do have trouble working out what

"The biggest change, perhaps, is the removal of any non-symmetric multiprocessing (SMP) compatible versions of the database. All versions of InterBase will now be able to use multiple processors and cores."

means.. CodeGear have been stating that InterBase was SMP enabled since InterBase 7.0. So what does this mean? Was InterBase non SMP capable/SMP friendly on some platforms? Was there code that wasn't SMP enabled? Has the code been cleaned up, so that all the old platforms have been removed. Or is it just a licensing issue where the new SMP license now supports up to 8 cores/CPUs, while previously you had to buy extra CPU licenses?

One particular quote was very confusing

"For the existing open-source Firebird database, which is based on InterBase, these changes won't trickle down; that free database does not include SMP, disaster recovery or high availability features."

Firebird was originally based on the open sourced version of InterBase 6.0, but the code bases are now quite different after 8 years of independent development work. So to state that Firebird is based on InterBase, although this is technically correct, it is a little disingenuous.

Stating that changes in InterBase will not trickle down to Firebird is correct. InterBase as per Inprise/Borland/CodeGear/Embarcadero is a closed source database. After the code was originally open sourced on SourceForge by Borland, it was forked and Firebird was created. InterBase was then "closed sourced" by Borland, and development work has continued on InterBase in a closed source manner. No code or feature at any time has come from InterBase to Firebird. Although it is possible that Firebird code may have made it into InterBase. However we have no way of verifying that.

In terms of SMP, by re-introducing the process based Classic Arhitecture, Firebird does support SMP. However a process based architecture isn't multi-threaded. That's an issue that's being addressed in Firebird V2.5 (SuperClassic) and Firebird 3.0.

I assume by "Disaster Recovery" this refers to "Journaling with Write Ahead Logging (WAL)" something that has not been implemented in Firebird. I am unsure what the other "High Availability" features are.

Finally

"In October, he said that InterBase ToGo will be made available. Normally, InterBase has a memory footprint of around 100MB. ToGo will be slimmed to around 3MB"

I am lost on this one. I am sure by this he means a disk footprint, not memory. And most of the size of the normal disk footprint is taken up by a stupid installer (imho) and docs. My assumption is that InterBase ToGo is effectively a variant on Firebird Embedded, but since ToGo will not be available until October it is difficult to comment.

Solaris 10, Firebird SuperServer and the Service Management Facility

2008-09-04T02:03:00.001-07:00

I recently released Firebird V2.04 SuperServer for Solaris 10. This new build uses the Solaris 10 Service Management Facility to stop/start Firebird. Here are some notes on how it works.
I created the following manifest file (it can be found in /opt/firebird/ as gds_db-tcp.xml):

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>


<service_bundle type='manifest' name='firebird'>
<service
name='application/firebird'
type='service'
version='1'>
<create_default_instance enabled='true' />

<exec_method
type='method'
name='start'
exec='/opt/firebird/bin/fbmgr.bin -start'
timeout_seconds='10'>
<method_context>
<method_credential user='firebird' group='firebird' />
</method_context>
</exec_method>

<exec_method
type='method'
name='stop'
exec='/opt/firebird/bin/fbmgr.bin -stop'
timeout_seconds='10'>
<method_context>
<method_credential user='firebird' group='firebird' />
</method_context>
</exec_method>

<template>
<common_name>
<loctext xml:lang='C'>
Firebird
</loctext>
</common_name>
</template>
</service>
</service_bundle>

This manifest file is automatically installed by the pkg installer.
You can manually stop and start firebird using the following commands, instead of using fbmgr.bin

svcadm enable svc:/application/firebird:default
svcadm disable svc:/application/firebird:default

You can verify whether the Firebird "service" is running by using the following:

svcs -a | grep firebird

Supposedly the new Service Management Facility provides automatic recovery from software and hardware errors, so theoretically SuperServer on Solaris 10 no longer has need of the Firebird Guardian, since SMF should do the same job that the Guardian was designed to do. This is going to need some testing ....

Solaris 10 and iscsi

2008-07-21T03:46:00.000-07:00

Seems that applying all of the patches discussed previously has broken my version of Solaris, as it now seems to have problems booting.

It seems to hang during the boot cycle, so I had to do some hunting to try and find out the cause.
Boooting with a -v (verbose) shows that the system is hanging in iscsi.
After some googling it seems the only way to fix this, so you can get in is to do the following..

1. Boot using -m milestone=none
2. Login as root
3. svcadm enable -r filesystem/minimal
4. svcadm enable -r filesystem/local
5. cd /etc/iscsi
6. mv iscsi_v1.dbc iscsi_v1.dbc.bad
7. reboot

Now you can get in, I suggest you go and install the latest iscsi path from sunsolve:
Patch number 119091 for x86 Solaris.

reboot and hopefully all will be well.

Solaris and the 256 File Descriptor Problem

2008-07-02T05:25:00.000-07:00

Some of you reading this may remember that back when InterBase 6.0 was released on Solaris we ran into a fundamental problem on Solaris. It seems that since 1992 there has been an open bug documenting Solaris' file descriptor problem: "32-bit stdio routines should support file descriptors >255."

Bug id:
1085341

When this issue was discovered, we chose to use SFIO, a stdio variant from AT&T
Comments on this issue can be found in jrd/common.h which forces a superserver build to use our own internal stdio.h found in extern/SfIO/include.

Well it looks like in Solaris 10 Sun have finally fixed the issue.
More details can be found in the article entitled:
Solaris OS Solutions to 32-Bit stdio's 256 File-Descriptors Limitation by Giri Mandalika dated May 2007

Suffice it say - the bug has been fixed, but in a fairly typical Sun way.... the fix is more complicated than it should be... why they didn't just change the char to a short and be done with it I really don't know.

Anyway it should be possible on Solaris 10 to use this fix instead of our "own" implementation of stdio.h, just by adding a couple of lines to gds.cpp
e.g.

#ifdef SOLARIS
#include <stdio_ext.h>
#endif
after including

and then adding the following at around about line 3535 (Firebird 2.0.4)
#ifdef SOLARIS
enable_extended_FILE_stdio (-1, -1);
#endif

Hmm that seems to be a nice simple solution, except that its always more complicated than that.
It seems that this fix is only available from later releases of Solaris 10, so if you have a version of Solaris from Solaris 10 3/05 through Solaris 10 11/06 you will need to install just 3 patches...
Well no - thats not the case. I had to do the following to get this to work...

Download and install the following patches:

1. 118855-36
2. 118919-21
3. 119255-53
4. 120273-21
5. 123840-04
6. 127756-01
7. 125504-02
8. 122661-08
9. 125548-02
10. 126420-01
11. 126424-03

And then finally in single user mode, followed by a reconfigure reboot:
To boot Solaris x86 in single user mode, requires some fun and games too...

At the GRUB menu, type 'e' before it times out and takes the default boot option.
Scroll to the default multiuser boot line, then type 'e' egain.
This will bring you in to edit mode. Add a ' -s' at the end of the line.
Then type 'b' to boot.

12. 120012-14

Install this patch - then reconfigure

At the GRUB menu, type 'e' before it times out and takes the default boot option.
Scroll to the default multiuser boot line, then type 'e' egain.
This will bring you in to edit mode. Add a ' -r' at the end of the line.
Then type 'b' to boot.

I am not sure that we can ask Solaris users of Firebird if they do not have the fix already installed to go through this, so we will have a look at keeping our own SfIO. However currently that will not compile using gcc 4.2.1 and thats something else we need to try and fix.

Addendum:
sftp is now broken - so you need to add two more patches to the list above

13. 119253-26
14. 124629-07

MacOSX and stdin,stdout,stderr

2008-06-05T03:00:00.000-07:00

I am in the process of porting a friends software to MacOSX. Don't ask what it does yet, all will be revealed soon. The code compiles quite happily on Solaris and Linux, but fails with the following error on MacOSX...

Why?

dbFile.c: In function ‘_setLogfile’:
dbFile.c:243: error: invalid lvalue in assignment
dbFile.c:244: error: invalid lvalue in assignment
dbFile.c: In function ‘fieldFillGen’:

I know what the problem is -
Seems on MacOSX 10.4 and below:
stdin, stdout, and stderr are addresses of a FILE rather than a FILE*
On MacOSX 10.5 they are FILE*

So depending on the OS version you are targeting will determine whether the code compiles or not. The issue goes back to the way FreeBSD originally implemented this in stdio.h which is the base for MacOSX, and it looks like it finally became "correct" i.e. portable and adhering to other implementations in MacOSX 10.5
A quick look at /usr/include/stdio.h confirms this...
the FILE* appears within the ifdef for _DARWIN_UNIX_03

__BEGIN_DECLS
#if __DARWIN_UNIX03
extern FILE *__stdinp;
extern FILE *__stdoutp;
extern FILE *__stderrp;
#else /* !__DARWIN_UNIX03 */
extern FILE __sF[];
#endif /* __DARWIN_UNIX03 */
__END_DECLS

So I need to come up with a simple workaround for what was originally valid code on other operating systems:

stderr = wk;
stdout = wk;

the following seems to work:

dup2(wk->_file ,STDERR_FILENO);
dup2(wk->_file ,STDOUT_FILENO);

Standard delete v's Firebird delete

2008-06-04T05:04:00.000-07:00

Alex and I have just committed a slew of changes to the code in Firebird 2.1.
This all started when we did the 32 bit builds for Firebird on MacOSX.

We found that in remote we would get asserts trying to create a database. On deeper and further investigation over a period of a couple of weeks we realised that in some cases the standard delete was being called instead of the redefined version in Firebird i.e. it invoked free() instead of
MemoryPool::globalFree(). After some extensive googling we found other people have also run into similar problems.

As a solution it was suggested to use:
1. Use switches -fvisibility=hidden and -fvisibility-inlines-hidden together.
2. Use switch -fvisibility=hidden and __attribute__((always_inline)).

Trying with second combination we found that we sometimes got (depending on the version of gcc) an error:
sorry, unimplemented: inlining failed in call to 'void operator delete(void*) throw ()'.

It seems that without -fvisibility=hidden and __attribute__((always_inline)) gcc is
using non-inline body for operator delete. And is randomly using the wrong body, the one from the standard operator delete.

The first combination helps to avoid this bug - but any entrypoint, which is to be exported, now has to be marked with __attribute__((visibility="default")).

64bit PPC build for MacOSX

2008-04-21T03:00:00.000-07:00

Someone asked me why there wasn't a 64bit version of Firebird for MacOS powerpc... so I tried a build on the powerpc MacOSX box we have...
sw_vers -productVersion = 10.4.11
Build fails when we try to build config_root.cpp.

/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:272: error: typedef 'ExceptionHandlerProcPtr' is initialized (use __typeof__ instead)
/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:272: error: 'ExceptionInformation' was not declared in this scope
/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:272: error: 'theException' was not declared in this scope
/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:306: error: 'ExceptionInformation' was not declared in this scope
/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:306: error: 'theException' was not declared in this scope
/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:307: error: expected primary-expression before 'userUPP'
/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/MachineExceptions.h:307: error: initializer expression list treated as compound expression
/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/fp.h:1338: error: 'SIGDIGLEN' was not declared in this scope

Some googling explains what the problem is. Seems MacOSX is 10.4 is not 100% 64bit friendly and the carbon framework will not support a full 64bit build until MacOSX 10.5.

We use the carbon framework in a few, but very specific places, mainly to load UDF libraries as bundles and also the fbintl library.

So....

I either need to upgrade our powerpc Mac to MacOSX 10.5 to do the build...
or
Anybody out there got a powerpc Mac running MacOSX 10.5 that I could use to check the build?

Editline, Curses and Solaris AMD64

2008-04-08T07:23:00.000-07:00

We finally solved the last major problem in the Firebird 2.0x build for Solaris 10 AMD 64bit. Whenever we tried to include editline in the build isql_static would coredump. We tracked the problem down to the following:

Seems there is a problem in the Solaris' native implementation of libcurses, the function tgetstr() returns a bad pointer (non-NULL, but out of the process space). This means that until Sun fix it we can't use libcurses to link with editline on Solaris 64bit AMD.

So we came up with a solution...
We downloaded and built ncurses... (5.5) then copied libncurses.a to gen/firebird/lib along with the normal editline.a
Then adjusted make.defaults to remove references to -lcurses for LINK_LIBS and STATICLINK_LIBS and added -lncurses.

We now have a succesful build that seems to work properly. I hope to put the 64bit Solaris AMD builds up in the firebird prerelease area for testing purposes, then commit the changes made to the code base. Then wen 2.04 is ready to be released will create proper versions for full release.

Hope that somebody out there finds this useful, or at least the builds anyway.

Wrongly Named libfbclient on Solaris

2008-04-07T05:51:00.000-07:00

There is a small problem with the currently shipping versions of Firebird 2.0.3 for Solaris.
The problem doesn't affect the general workings of Firebird, but if you try to link with the libfbclient library using your own program, you will run into a problem where you will consistently get the following error:

ld: fatal: library -lfbclient: not found

The problem is due to a misnamed libfbclient.so in /opt/firebird/lib

To correct:
cd /opt/firebird/lib
rm libfbclient.so
ln -s libfbclient.so.2 libfbclient.so

This has already been corrected in CVS

64bit Firebird for Solaris 10 AMD64

2008-04-01T02:33:00.000-07:00

Looks like having a 64bit build of Firebird on Solaris 10 AMD64 is going to be a non-starter.
Alex and I have done all the hard work, and we have a build (currently without editline, since thats a problem in itself), but when we try and run isql created by the build we get the following:

ld.so.1: ./isql: fatal: relocation error: R_AMD^$_32S:
file /usr/lib/64/libfbembed.so.2: symbol (unknown): vakue 0xfffffd7ffee00000
does not fit

or in debug mode trying to run isql from the build directly:
ld.so.1: isql: fatal: relocation error: R_AMD64_PC32
file /home/pbeach/firebird64/gen/firebird/lib/libfbembed.so.2:
symbol main: value 0x2800155d494 does not fit.

A bit of googling finds this:

Bug ID 5102797
Synopsis AMD64: static relocation in dynamic library fails

If you read the details, seems the bug has been around since 15th Sept 2004
and because of its nature, there are no plans to fix it until Solaris 11.

Addendum: 2nd April 2007
Looks like we have found the problem.
LIB_LINK_OPTIONS in the prefix build file was -G
-G on Solaris should tell the linker to create a shared library
However because we are using gcc, something somewhere was getting confused.
If we pass -fPIC and -shared instead the problem is resolved.

From the gcc manaual:

-shared

Produce a shared object which can then be linked with other objects to form an executable. Not all systems support this option. For predictable results, you must also specify the same set of options that were used to generate code (-fpic, -fPIC, or model suboptions) when you specify this option

On some systems, `gcc -shared' needs to build supplementary stub code for constructors to work. On multi-libbed systems, `gcc -shared' must select the correct support libraries to link against. Failing to supply the correct flags may lead to subtle defects. Supplying them in cases where they are not necessary is innocuous.

FreeAdhocUDF for MacOSX

2008-03-28T04:37:00.000-07:00

I have just finished porting FreeAdhocUDF to MacOSX and sent the Makefiles, code, plus 32bit UDF libraries for Firebird 1.5.5 and Firebird 2.x to the FreeAdhocUDF types to be included in the distribution.

Issues that you might need to be aware of.
1. In Firebird 1.5.5 we don't ship libib_util.dylib, and even if we did you wouldn't be able to link to it. It seems that Firebird 1.5.5's installer doesn't copy the library to the Framework. This has now been fixed for the next release. Secondly the library was being built as a bundle. All Firebird UDF's need to be created as bundles to work on MacOSX. You can't link a bundle to a bundle. This has also been fixed.
2. In Firebird 1.5.5 you must remove any extension from the UDF library (e.g dylib) to get it to work. So for example FreeAdhocUDF.dylib needs to be renamed to FreeAdhocUDF. This bug has been fixed in flu.cpp for the next version.

Firebird 2.x does not have any of these problems :-)

In compiling FreeAdhocUDF for MacOSX, it has to be noted that MacOSX has no idea of sysinfo.h, so I had to add a chunk of code to handle the getting of a MAC address, and to get the random_info/randomness for GUIDs. This does seem to work nicely now.

The FreeAdhocUDF libraries I have provided are 32bit intel for Firebird 1.5.5 and Firebird 2.x, if you want to build ppc libraries, or 64bit libraries enjoy. If you have problems, let me know.

On ppc you "might" have to add the following to the Makefile
i.e. change
LDFLAGS=$(CF) $(LIB) -bundle -flat_namespace -undefined suppress to
LDFLAGS=$(CF) $(LIB) -bundle -flat_namespace -undefined suppress /usr/sys/bundle.o

Exceptions, gcc and Solaris 10 AMD 64bit

2008-03-21T08:06:00.000-07:00

For a little while now Alex and I have been battling a problem on Solaris 10 64bit AMD. We have been trying to port Firebird 2.0x to 64bit Solaris. However after fixing a few issues, e.g. defining the platform etc, we found that isql_static would core dump on trying to create a database. This is where the fun begins...

The default debugger for gcc shipped with Solaris 10 can't handle 64bit applications. Solution - build your own from a later version. At least this way we can now step through the Firebird code and try to see whats happening..

After some careful debugging sessions we find that Firebird seems to be OK, when you try to create a database, Firebird first trys to open it as an existing database, if it can't find an existing database we throw an internal exception and go on to create it.

The problem is isql_static is occurring when the exception is thrown...

Program received signal SIGSEGV, Segmentation fault.
0x0000000000488385 in __EH_FRAME_BEGIN__ ()

(gdb) bt
#0 0x0000000000488385 in __EH_FRAME_BEGIN__ ()
#1 0xfffffd7ffedacf3c in _Unwind_RaiseException_Body () from /lib/64/libc.so.1
#2 0xfffffd7ffedad129 in _Unwind_RaiseException () from /lib/64/libc.so.1
#3 0xfffffd7ffef3c71e in __cxa_throw (obj=0x1, tinfo=0x1, dest=0x474e5543432b2b00)
at /builds/sfw10-gate/usr/src/cmd/gcc/gcc-3.4.3/libstdc++-v3/libsupc++/eh_throw.cc:75
#4 0x000000000056510b in Firebird::status_exception::raise (status_vector=0x8d19e0)
at ../src/common/fb_exception.cpp:197
#5 0x0000000000694137 in ERR_punt () at ../src/jrd/err.cpp:562
#6 0x0000000000693ced in ERR_post (status=335544344) at ../src/jrd/err.cpp:441
#7 0x00000000005c66d4 in PIO_open (dbb=0xfffffd7ffec29050, string=@0xfffffd7fffdec9e0,
trace_flag=false, connection=0x0, file_name=@0xfffffd7fffdeca20, share_delete=false)
at ../src/jrd/os/posix/unix.cpp:646

So Firebird looks OK, i.e. its doing what it should do, however Solaris is causing a crash dump on the exception.

Some more detective work uncovered the following:

The problem is caused by the use of the -lc flag which was explicitly added to linker command line by autoconf.

Because we always use g++/gcc as linker, there is no need for us to need add in libc explicitly - it's done by compiler itself when it invokes ld. But autoconf seems to be doing it anyway.

When it is added to the command line, libc (Solaris native library) then happens to be added to ld's command line before libgcc_s where we would find gcc's own exception support library. If this happens any call to functions that have the same name, that happen to be present in both libraries, will go to libc, not libgcc_s.

There is at least one such function - Unwind_RaiseException(), which is called by g++ generated
code when exception is thrown. But instead of libgcc_s::Unwind_RaiseException() libc::Unwind_RaiseException() is called, but not with the parameters the Solaris native function expects - cue core dump.

Solution:

Don't use -lc

Where is libfbclient.dylib on MacOSX?

2008-03-13T02:13:00.000-07:00

There was a post to the Firebird Support list, asking how you link Qt to the libfbclient.dylib on MacOSX Firebird SuperServer. The problem being, that libfbclient.dylib is not "shipped" as part of the MacOSX framework... heres the detailed answer on how it works.

Firebird on MacOSX is delivered as a framework,

For more information on a frameworks look on : http://developer.apple.com

In a nutshell - "A framework is a hierarchical directory that encapsulates shared resources, such as a dynamic shared library, nib files, image files, localized strings, header files, and reference documentation in a single package. Multiple applications can use all of these resources simultaneously. The system loads them into memory as needed and shares the one copy of the resource among all applications whenever possible."

This is the "best" way to deliver an application on MacOSX.

However to do this the main dylibs (shared libraries) that run Firebird become part of the framework itself.

The main dylib that drives Classic is libfbembed, for SuperServer its libfbclient.

With Classic we ship both libfbembed and libfbclient, libfbclient as a separate dylib can be found in the libraries directory.

for SuperServer we ship libfbclient

However libfbembed (Classic) and libfbclient (SuperServer) are not directly obvious as shared libraries, i.e. if you look for them by that name with a dylib extension you are not going to find them. However if you took a close look at postfix.darwin in the Firebird source builds/posix (this is the code that sets up the framework) you will find that they become Firebird.framework/Firebird which is a link to Firebird.framework/Versions/Current/Firebird.

Where Firebird.framework/Firebird is libfbembed.dylib for Classic and
Firebird.framework/Firebird is libfbclient.dylib for SuperServer

Building GCC 4.2.1 On Solaris

2008-03-11T05:45:00.000-07:00

Alex and I are currently trying to implement a 64bit version of Firebird 2.x for Solaris 10, and we are running into a number of problems. However they shouldn't be insurmountable. Every port has its problems one way or another, but this one is proving to be particularly annoying....

The default version of gcc shipped with Solaris 10 is 3.4.3 and is found in /usr/sfw/bin
The current available version of gcc (released is) 4.3
So the version Sun makes available dates from 2005...
One assumes that Sun has no interest in letting Solaris users use more modern versions of gcc and would rather we use their own compiler.
Hmm so much for Sun and open source...

But to properly compile on Solaris 10 with 64bit see notes from gcc.gnu.org

"i?86-*-solaris2.10
Use this for Solaris 10 or later on x86 and x86-64 systems. This configuration is supported by GCC 4.0 and later versions only.

It is recommended that you configure GCC to use the GNU assembler in /usr/sfw/bin/gas but the Sun linker, using the options --with-gnu-as --with-as=/usr/sfw/bin/gas --without-gnu-ld --with-ld=/usr/ccs/bin/ld"

So gcc 4.0 is the lowest version you can use for cross-compiling 32bit and 64bit applications on Solaris 10...

So we have to build our own gcc to do the firebird compilation, here's how you do it....
1. Download a copy of the gcc source (e.g. gcc-4.1.2.tar.gz) from gcc.gnu.org
2. Install gnu tar, solaris tar has problems extracting the source from the gcc tar file.
3. You will probably need to install libintl too (you can get a pre-built version of this froim www.sunfreeware.com)
4. gunzip gcc-4.1.2.tar.gz
5. /usr/local/bin/tar -xvf gcc-4.1.2.tar
6. cd gcc-4.1.2
7. mkdir objdir
8. cd objdir
9. ../configure --with-gnu-as --with-as=/usr/sfw/bin/gas --without-gnu-ld --with-ld=/usr/ccs/bin/ld --enable-shared --enable-languages=c,c++
10. gmake
11. as root - make install

Your new version of gcc will be installed in /usr/local/bin

Symbol Variants - MacOSX

2008-02-29T06:57:00.000-08:00

Up until now it has been impossible to run kinterbasdb against SuperServer builds of Firebird for MacOSX on Tiger (10.4). When you try and import kinterbasdb into Python the following error occurs:
Symbol not found fputs$UNIX2003
This is a bit of a problem.
Reading around the issue indicates that symbol usage between MacOSX 10.4 and 10.5 has changed because of the introduction of UNIX cconformance.
This article on developer.apple.com explains why we might see UNIX2003 symbols.
Basically anything compiled on 10.5 will use the new conforming UNIX2003 sysmbols where appropriate.
However we shouldn't be seeing these symbols if we compile everything with
--mmacosx-version-min=10.4 and/or set MACOSX_DEPLOYMENT_TARGET=10.4
So why when we try and run kinterbasdb on 10.4 we have a problem?
A nm -a libfbclient.dylib shows that there are 4 UNIX2003 symbols in the library...
and I had no idea where they were coming from.
It seems that we have a link problem, and the only way to make sure that they do not get pulled in to the libfbclient library is to insist that the linker links against the 10.4 sdk.
A small change to prefix.darwin_i386 seems to solve the problem
Add LD_FLAGS+=-isysroot /Developer/SDKs/MacOSX10.4u.sdk
Why the linker ignores the flag/environment variable is a mystery to solve another day.

Firebird and MacOSX Uninstalls

2008-02-21T03:33:00.000-08:00

MacOSX doesn't have an uninstall utility.
I use the following script to clean up Firebird installs on Leopard.
It should work on Tiger as well.
Would it make sense to include this in the Firebird package, along with a similar script for earlier versions of MacOSX (pre 10.4 for the powerpc builds)

#!/bin/sh
echo "Clean Services"
echo "Clean User"
dscl localhost -delete /Local/Default/Users/firebird
echo "Clean Group"
dscl localhost -delete /Local/Default/Groups/firebird
if [ -f "/Library/StartupItems/Firebird" ]; then
echo "Remove SuperServer StartupItem"
rm -fr /Library/StartupItems/Firebird
fi
if [ -f "/Library/LaunchDaemons/org.firebird.gds.plist" ]; then
echo "Remove Launchd"
launchctl unload /Library/LaunchDaemons/org.firebird.gds.plist
rm /Library/LaunchDaemons/org.firebird.gds.plist
fi
echo "Remove Framework"
rm -fr /Library/Frameworks/Firebird.framework
echo "Remove Receipt"
rm -fr /Library/Receipts/Firebird*.pkg

#Add the following for Firebird V2.5
echo "Remove /tmp/firebird"
rm -fr /tmp/firebird

Python, kinterbasdb and 64bit MacOSX 10.5

2008-02-19T04:35:00.000-08:00

Now we have 64bit builds of Firebird 2.1 for MacOSX 10.5, we have issues trying to test them properly using the Firebird QMTest QA suite. This QA suite relies on Python and kinterbasdb. Currently Philippe Makowski has been able to run the tests from a 64bit Linux client. But is unable to run the tests locally. Well currently the python framework that comes with MacOSX 10.5 is only 32bit and this forces a 32bit build of kinterbasdb, since the build environment of kinterbasdb uses distutils to use the same compile switches as python. So I thought I would have a look at the issue and see if I could build a 64bit version of kinterbasdb for MacOSX 10.5.

This is how I did it...

1. Download the source distribution of Python
2. Edit configure.in and change line 817, so it looks like this
BASECFLAGS="$BASECFLAGS -Wno-long-double -no-cpp-precomp -mno-fused-madd -arch x86_64"
3. Make sure that (just in case)
a, CFLAGS, CXXFLAGS and LDFLAGS are exported with -arch x86_64
e.g. export CFLAGS='-arch x86_64'
b, export MACOSX_DEPLOYMENT_TARGET=10.5
4. autoconf
5. ./configure --disable-toolbox-glue
This disables Carbon as its only 32bit and you will get compile errors if you don't
We are not building a framework, or a universal binary just a simple standalone version of 64bit python. So do not enable those options.
6. make
You may get a warning/errors possibly in some of the extensions e.g.
ld warning in /System/Library/Frameworks/TK.framework/Tk missing required architecture x86_64 in file.
I guess you would have to download the relevant code and rebuild these as 64bit if you need them. But since we are only interested in kinterbasdb I guess I can afford to ignore it.
A check on the built python.exe shows that it is Mach-O 64 bit executable.

Now we need to build kinterbas db using this python.exe
1. Download kinterbasdb from sourceforge
-> 2. Replace in the kinterbasdb source the __init__.py with this one :
https://firebird.svn.sourceforge.net/svnroot/firebird/qa/trunk/kinterbasdb/__init__.py
Note I forgot to add this step when I wrote the original instructions.
3. Make sure you have the 64bit Firebird 2.1 for MacOSX installed
4. Make sure you are running as root
su and log in
export MACOSX_DEPLOYMENT_TARGET=10.5
export CFLAGS, CXXFLAGS and LDFLAGS ='-arch x86_64'
The install option will want to put the kinterbasdb in /usr/local/lib/python2.5
Although I suspect you can change this by using a different --prefix when you build python
5. pathtojustbuilt64bitpython/python.exe setup.py install
or
pathtojustbuilt64bitpython/python.exe setup.py build
if you just want to build it.
6. You can check what you have installed by
file /usr/local/lib/python2.5/site-packages/kinterbasdb/_kinterbasdb.so
Mach-O 64bit bundle x86_64

Now we can test it...
create a database using isql
pathtojustbuilt64bitpython/python.exe
>>> import kinterbasdb;kinterbasdb.init(type_conv=200)
Don't use mx.DateTime use the default python DateTime
>>> con = kinterbasdb.connect(dsn='localhost:/pathtodd/test.fdb', user='sysdba', password='masterkey')

It works ....

Testing and Debugging the Service Manager on MacOSX

2008-02-12T03:20:00.000-08:00

For Classic - SuperServer is easy.

PATH=$PATH:/Library/Frameworks/Firebird.framework/Resources/bin
export PATH

In one terminal session:
gsec -user xxx -password -xxx -database localhost:/Library/Frameworks/Firebird.framework/Resources/English.lproj/var/security2.fdb
This will start a remote connection to gsec via the service manager...
ps -eaf | grep fb
and look for the running fb_inet_server

In the other
cd /Library/Frameworks/Firebird.framework/Resources/bin
gdb ./fb_inet_server --pid=
Set the pid to the running fb_inet_server
gdb> break svc.cpp:2461
set an appropriate break point - in this case when vfork starts
gdb> continue

In the original session
gsec> display

In the gdb session the debugger should break when vfork is called in svc.cpp

Now do what you need to - to walk through the code.

If all is working properly, try doing a backup using the service manager...
gbak -v -b -user xxxx -password xxxx /Users/pbeach/databases/test.fdb /Users/pbeach/databases/test.fbk -se localhost:service_mgr

Adding a User to a Group using dscl

2008-02-08T00:31:00.000-08:00

How to add a user to the Firebird group on MacOS using dscl

dscl localhost append /Local/Default/Groups/firebird GroupMembership pbeach
or
dscl . append Groups/firebird GroupMembership pbeach

Nine Below Zero

2008-02-03T02:29:00.000-08:00

Non Technical for a change:

2nd Feb 2008 + Cafe Francaise@Chinon + Nine Below Zero * (Rhythm + Blues) + 150 people = Superb gig.

64bit Firebird for MacOS 10.5 (Leopard)

2008-01-18T03:42:00.000-08:00

Today I put the finishing touches to a native 64bit port of Firebird for MacOS 10.5 (Leopard).
This required the following changes to the Firebird 2.1 code base (B2_1_Release). For anybody else considering adding a new platform to Firebird then you may find this useful.

Changes:
1. src/jrd/common.h (line 289)
#ifdef __x86_64__
#define DARWIN64
#define IMPLEMENTATION isc_info_db_impl_darwin_x64 /* 73 */
#endif

2. src/jrd/inf_pub.h (line 203)
add
isc_info_db_impl_darwin_x64 = 73,

3. src/jrd/license.h (line 135)
amend
#ifdef DARWIN
#if defined(i386)
#define FB_PLATFORM "UI" /* Darwin/Intel */
to
#if defined(i386) || defined (__x86_64__)

4. src/jrd/pag.cpp (line 155)
insert (last)
static const int CLASS_DARWIN_X64 = 31; // Darwin x64
amend (line 158)
static const int CLASS_MAX = CLASS_DARWIN_X64;
since this is the last one added

5. src/jrd/pag.cpp (line 245)
insert (last)
archLittleEndian // CLASS_DARWIN_X64
make sure you add a , to the class above...

6. src/jrd/pag.cpp (line 322)
Pick up the right class for the right architecture..
add
#ifdef DARWIN64
const SSHORT CLASS = CLASS_DARWIN_X64
#endif

7. src/remote/protocol.h (line 150)
add a new architecture
arch_darwin_64 = 41,
and update arch_max
arch_max = 42

8. src/remote/remote_def.h (line 72)
add a new architecture
#elif defined(DARWIN64)
const P_ARCH ARCHITECTURE = arch_darwin_64;

9. src/remote/xdr.cpp
Make sure SWOP DOUBLE is defined correctly
by appending
|| defined(DARWIN64)
to the intel crowd.

10. src/jrd/utl.cpp (line 218)
add
"Firebird/Darwin/Intel64" /* 73 */

11. src/common/classes/fb_atomic.h (line 82)
add || defined (__x86_64__)

12. configure.in (line 62)
In the target darwin section add
dnl CPU_TYPE=x86_64
This can be uncommented when you do a 64bit build

Now read the notes in builds/posix/prefix.darwin_x86_64

1. edit configure.in so that MAKEFILE_PREFIX=darwin_x86_64
2. Uncomment the CPU_TYPE
3. edit extern/icu/source/config/mh-darwin to set the right 64bit flags (-arch x86_64) for LD_FLAGS
for CFLAGS, CXXFLAGS
4. export CFLAGS='-arch x86_64'
5. export CXXFLAGS='-arch x86_64'

Then check and make sure that
MACOSX_DEPLOYMENT_TARGET=10.5
and export that for good measure.

Build should be good to go. Welcome to native 64bit Firebird on MacOS

depot Files and HP-UX

2008-01-09T07:42:00.000-08:00

I have just finished writing a depot install for Firebird Classic 2.0x on HPUX . A depot install is the HP equivalent of a Linux rpm or Solaris package. The operating system install uninstall utilities take command of looking after the files. Although in more complex areas you still have to have the relevant postinstall and postremove scripts to handle things like links etc.
To allow a user other than root to use SD-UX you need to give yourself access to it e.g.
swacl -m user:pbeach:crwit -l host
To create a depot install you need to write a .psf file (the Firebird specific one will be found in builds/install/arch-specific/hpux/classic/firebirdCS.psf), you then

test it using:
swpackage -vv -p ./firebirdCS.psf

create it using:
swpackage -d "| gzip -c > /firebird2/gen/firebird.depot.gz" -x media_type=tape -vv -s firebirdCS.psf

install it using:
swinstall -s /firebird.depot.gz

list installed packages using
swlist

remove it using
swremove Firebird

Open Source Code Contains Security Holes

2008-01-09T00:24:00.000-08:00

The following was sent to Charles Babcock at Information week in reply to an article entitled:

Open Source Code Contains Security Holes

As a developer and administrator of the Firebird Project I completely reject the statement you made in the above article.

"The somewhat moribund Firebird project, for example, is listed with 195 identified defects, of which it has verified zero and fixed zero. The active Firefox browser project, on the other hand,
has fixed 370 bugs, verified 56 and faces another 246 to verify and fix."

The Firebird project is in fact incredibly active - perhaps a look at this chart on our bug tracker might give you a clue.

http://tinyurl.com/yt5pgl

Firstly the Firebird project reviewed the Coverity results almost immediately they were published and found that the report isn't actually related to the Firebird engine. This URL shows our appropriate comments from the 7th March 2006:
http://www.firebirdnews.org/?p=180

Also more comments from Claudio on the 26th March 2006:
http://www.firebirdnews.org/?p=243

Secondly in a more detailed reply to the actual "PR" issue raised by David Maxwell, open source strategist for Coverity. If you had asked about this before printing the article you could have put some facts straight.

Nearly all of the 195 identified defects are in fact actually within an external piece of code we use for character sets and collation sequences ICU

http://www-306.ibm.com/software/globalization/icu/index.jsp

"The International Component for Unicode (ICU) is a mature, portable set of C/C++ and Java libraries for Unicode support, software internationalization (I18N) and globalization (G11N),
giving applications the same results on all platforms."

A open source project maintained by IBM. I will admit that we are using an older version of ICU (3.0) than is currently available and we will be upgrading to a newer version in the near future.
But this is not something that is a trivial exercise, as it means that any database using a different version of ICU would be incompatible with the version we ship. We plan to upgrade ICU
in Firebird version 2.5

Other defects reported are one in
usr/include/c++/4.0.2/i386-redhat-linux/bits/gthr-default.h
Not our problem either....

And there are four defects in firebird2/src/gpre/pretty.cpp a piece of old code used with a pre-compiler (gpre) to make BLR look good. BLR (Binary Language Representation),
Firebird's internal compiled language. This doesn't affect the Firebird server at all.

I would like you to print a correction or at least acknowledge the innacuracy of the article as regards Firebird.

Regards
Paul Beach

Firebird 2.03 on HPUX 11

2008-01-07T05:45:00.000-08:00

I have just released a classic version of Firebird V2.03 for HPUX 11. This is a 64bit version for PA-RISC 2.0. The current install mechanism as per Firebird 1.5.x uses a tar file and scripts to install firebird into /opt/firebird etc. I think the time has now come to try and write a depot file so that we can use the HPUX installer to install firebird. Initial analysis indicates that I need to write a .psf file that contains the relevant instructions, and that I can test it using the following command:
swpackage -p -vv -s ./firebirdCS.psf -d /sd/firebird
Where:
-p = preview mode (don't do anything - just test the psf file)
-vv = very verbose
-s = source psf file
-d = destination

Leopard 64bit OS and Firebird

2007-12-21T08:54:00.000-08:00

For anybody who may even be remotely interested MacOS 10.5 (Leopard) is a 64 bit OS and allows you to build and run 64 bit binaries. The current Firebird build for MacOS 10.4 and MacOS 10.5 is 32bit. I now have a 64bit build for Leopard. When I get the chance I will explain exactly what was done. I think we will start offering 64bit builds for MacOS with Firebird 2.1 for those who want them.

O_DIRECT and MacOS

2007-12-20T04:52:00.000-08:00

Mac's don't support the O_DIRECT flag for files but F_NOCACHE should do the trick....
For more details look at jrd/os/posix/unix.cpp and forced writes in the 2.1 cvs tree.

Preprocesser Macros

2007-12-18T02:07:00.001-08:00

The following are the C preprocessor macros used for making various compile-time checks against MacOSX architectures - Firebird currently checks for the 32bit macros... I hope to try a 64bit intel build on Leopard in the not too distant future. This will mean defining yet another "new" platform.

__ppc__
__ppc64__
__i386__
__x86_64__

MACOSX_DEPLOYMENT_TARGET

2007-12-18T02:02:00.001-08:00

I have read in a number of places that I should be using

gcc -mmacosx-version-min=10.4

rather than the

MACOSX_DEPLOYMENT_TARGET=10.4

environment variable because the latter has now been deprecated.

NO! Any attempt to compile Firebird on MacOS 10.5 and run it on MacOS 10.4 will create bus errors. You must set the MACOSX_DEPLOYMENT_TARGET variable!

(The Firebird build environment now uses both - just to make 100% certain)

MacOS Architectures Compile Flags

2007-12-13T05:04:00.000-08:00

It is possible to define what architecture you wish to compile against on MacOS using the following C/C++ flag(s):

-arch ppc (ppc 32bit)
-arch ppc64 (ppc 64bit)
-arch i386 (intel 32bit)
-arch x86_64 (intel 64bit)

Its even possible to combine them to create "Universal Binaries" however the endian-ness of Firebird on ppc v's intel and the way that Firebird is currently built makes this somewhat impossible. Something for the future I think.

Trying To Debug Semaphores

2007-12-12T06:30:00.001-08:00

We are trying to debug semaphore creation and usage on MacOS to get Signal Safe Semaphores implemented in SuperServer. But we are having some issues... (as per previous posts)
We have a gds__log print in semaphore.h to see whether the semaphore is created.

If we try and start the server automatically via
the StartupItem (Mac's method for starting Services)
we see:
sem=0x0
If we try and run ./fbserver from gdb or the command
line we see
sem=0x3
In the firebird.log file.

It looks like for darwin semaphore's handle is really a file descriptor - nothing more. When statring firebird as a service, there are no opened files passed to it, because it's supposed to be a daemon. Therefore the first opened file (or semaphore) is 0. When starting from the command line, stdin(0), stdout(1) and stderr(2) are already opened. Therefore - we get 3 which is the next free handle.

DTrace

2007-12-12T05:38:00.000-08:00

MacOS 10.5 (Leopard) has replaced the kernel debugging tool KTrace with a new tool called DTrace.
KTrace was relatively simple to use, and although not probably the best tool around for looking at what the kernel is doing - it did give you information that you needed simply and easily.
For those of you who might even be marginally interested in DTrace.
Here's a link on MacTech.
And thats just the tip of the iceberg.

Great - something completely new to try and understand. Why couldn't Apple have left KTrace and its friends on the 10.5 system, so we could use it, while we got to grips with DTrace?

Looks like DTruss will do the trick....

Whats That Number?

2007-12-12T03:55:00.000-08:00

For those who might not know, the number that you see after the table name in gstat for example (130) is the relation id. For those of you who might not have 100+ tables, you might ask why do you always see 3 digits. Well the answer is as follows:

Numbers 0-127 are reserved for system table relation ids. So 128, 129+, etc., in creation order are used for normal table/relation ids.

Analyzing database pages ...
ACC_CHARGES (130)
Primary pointer page: 264, Index root page: 265
Data pages: 1, data page slots: 1, average fill: 16%
Fill distribution:
0 - 19% = 1
20 - 39% = 0
40 - 59% = 0
60 - 79% = 0
80 - 99% = 0

Leopard V's Tiger

2007-12-10T03:46:00.000-08:00

Looks like we are going to need separate download binaries for Firebird on Leopard and pre-Leopard (Tiger). Firebird 2.0.3 compiled on Leopard works perfectly on Leopard, but when you try to run and install Firebird that was built on Leopard on Tiger, all the utilities give a bus error, and you get the following type of debug trace from gdb:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00000000
0x302188f0 in Firebird::MemoryPool::allocate_nothrow ()
(gdb) bt
#0 0x302188f0 in Firebird::MemoryPool::allocate_nothrow ()
#1 0x3000e5d6 in gds__alloc_debug ()
#2 0x00012c8f in parse_arg ()
#3 0x00018d51 in ISQL_main ()
#4 0x000190bf in main ()

Supposedly a EXC_BAD_ACCESS/KERN_PROTECTION_FAILURE — is caused by the thread trying to write to read-only memory. This is always caused by a data access.

So why does Firebird 2.03 compiled on Tiger work perfectly? For some strange reason we smell a rat with either the Operating System or the compiler. But what the problem is - is a very good question. I am open to suggestions.

Using the link command:

-Wl,-macosx_version_min -Wl,10.4 has no effect.

Building Firebird 2.0x MacOS - semaphore.h

2007-12-07T03:32:00.001-08:00

If you download the Firebird Source Code and want to build your own version of Firebird 2.0x on MacOS 10.4 (Tiger) you will probably have to make a minor change to the code. Seems there are changes in /usr/include/sys/semaphore.h between 10.4 and 10.5 to make semaphore.h more compliant.
We use (for Mac purposes) the following in src/common/classes/semaphore.h at line 106

if (sem == SEM_FAILED) {

Which seems obvious right?

Well on 10.4 the build will fail with the following error:

make -f ../gen/Makefile.libfbclient libfbclient
g++ -DSUPERCLIENT -D_THREAD_SAFE -I../src/include/gen -I../src/include -I../src/vulcan -I../extern/icu/source/common -I../extern/icu/source/i18n -DNAMESPACE=Vulcan -ggdb -DDARWIN -pipe -p -MMD -fPIC -fno-common -Wall -arch i386 -DDEV_BUILD -c ../src/jrd/alt.cpp -o ../temp/superclient/jrd/alt.o
../src/include/../jrd/../common/classes/semaphore.h: In constructor 'Firebird::SignalSafeSemaphore::SignalSafeSemaphore()':
../src/include/../jrd/../common/classes/semaphore.h:106: error: ISO C++ forbids comparison between pointer and integer
../src/jrd/../jrd/common.h: At global scope:

A quick check of /usr/include/sys/semaphore.h on 10.4 and 10.5 shows:

10.4 #define SEM_FAILED -1
10.5 #define SEM_FAILED ((sem_t *)-1)

Hmm - 10.5 looks more like what we would expect (same as linux for example)
A problem with types? Looks like it.

Change line 106 in src/common/classes/semaphore.h

from:
if (sem == SEM_FAILED) {
to:
if (sem == (sem_t*)SEM_FAILED) {

Or upgrade your /usr/include/sys/semaphore.h to something thats correct!

Semaphores With Timeouts

2007-12-07T03:14:00.000-08:00

Firebird normally makes use of semaphores with timeouts, and I read on an Apple mailing list the following question posted in May 2004:

"I have a project that depends on semaphores with timeouts (sometimes
referred to as realtime extensions to semaphores). I use autoconf to
select either SysV semaphores or POSIX semaphores, but MacOS X does not
have a complete set of either one. Neither sem_timedwait nor semtimedop
are implemented. On top of that, the POSIX semaphore implementation
doesn't even have sem_init (though I did see the recent discussion about
it on this list, so I'm moderately hopeful for 10.4).
Are semtimedop or sem_timedwait implementations planned anytime soon?"

Well he answer is no. Leopard MacOS 10.5 doesn't solve the problem either. Hence a number of changes we have had to make to the Firebird code and build process to handle this. Look at the latest (being committed) configure.in and semaphore.h to see how we have had to try and work around what should not be a problem by using a poor man's semaphore based on a mutex and conditional variable.

Hint to the Apple developers - You really should fix this soon.

32bit or 64bit binaries on MacOS

2007-12-06T01:47:00.001-08:00

I have just built a version of Firebird 2.03 Classic for MacOS using a new MacOS 10.5 (Leopard) system I got recently. My basic tests show everything working OK. So a copy of the build was sent to Philippe Makowski for QA. However after install he gets a bus error trying to run isql. He is running MacOS 10.4 (Tiger).

First thought as to what the problem might be was that Leopard is a 64bit OS whilst Tiger is 32bit and that gcc/ld might have built incompatible binaries for the two platforms by default.

Well - Hmm - heres the problem, how do you find out whether a binary is 64bit or not?
MacOS has no ldd. But it does have something called otool
file ./isql gives the following answer: ./isql: Mach-O executable i386
Without going into the ins and outs of how MacOS loads binaries etc there is no simple and easy way to find out for sure - even using otool...

The solution:
otool -h ./isql
./isql
mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
0xfeedface 7 3 0x00 2 16 1868 0x00000085

The key is the magic number, looking at /usr/include/mach-o/loader.h will give you the magic number for 32bit and 64bit binaries...
0xfeedface = 32bit
0xfeedfacf = 64bit

So we can see from the information above that we do have a 32 bit build. So thats not the problem.