Based on a document written by Fulvio, a while ago, I finally spent time writing a make file that will automatically create an embedded version of Firebird that will run on MacOSX as a bundle. All you do is make a normal build of Firebird Classic and then use this make file to create the Firebird.app folder which is set up in a way that allows you to access a database via isql for example without the need for a full framework. The make file detailed below was written for Firebird 2.5.x
To make a Classic version of Firebird for MacOSX:
./configure
make
cd gen
make -B -f Makefile.install
Then you can use the following make file to create an embedded version.
FBE=../gen/firebird/Firebird.app
BINLOC=../gen/firebird/Firebird.app/Contents/MacOS/firebird/bin
LIBLOC=../gen/firebird/Firebird.app/Contents/MacOS/firebird
OLDPATH=/Library/Frameworks/Firebird.framework/Versions/A/Libraries
all:
-$(RM) -rf $(FBE) ../gen/firebird/Firebird.app
mkdir -p $(FBE)/Contents
mkdir -p $(FBE)/Contents/MacOS
mkdir -p $(FBE)/Contents/Resources
mkdir -p $(FBE)/Contents/Frameworks
mkdir -p $(FBE)/Contents/Plugins
mkdir -p $(FBE)/Contents/SharedSupport
mkdir -p $(FBE)/Contents/MacOS/firebird
mkdir -p $(FBE)/Contents/MacOS/firebird/bin
mkdir -p $(FBE)/Contents/MacOS/firebird/intl
cp ../builds/install/misc/firebird.conf.in $(FBE)/Contents/MacOS/firebird.edit
cp ../gen/firebird/firebird.msg $(FBE)/Contents/MacOS/firebird/firebird.msg
cp ../gen/firebird/lib/libfbembed.dylib $(FBE)/Contents/MacOS/firebird/libfbembed.dylib
cp ../gen/firebird/lib/libicudata.dylib $(FBE)/Contents/MacOS/firebird/libicudata.dylib
cp ../gen/firebird/lib/libicui18n.dylib $(FBE)/Contents/MacOS/firebird/libicui18n.dylib
cp ../gen/firebird/lib/libicuuc.dylib $(FBE)/Contents/MacOS/firebird/libicuuc.dylib
cp ../gen/firebird/security2.fdb $(FBE)/Contents/MacOS/firebird/security2.fdb
cp ../gen/firebird/bin/gbak $(FBE)/Contents/MacOS/firebird/bin/gbak
cp ../gen/firebird/bin/isql $(FBE)/Contents/MacOS/firebird/bin/isql
cp ../builds/install/misc/fbintl.conf $(FBE)/Contents/MacOS/firebird/intl/fbintl.conf
cp ../gen/firebird/intl/libfbintl.dylib $(FBE)/Contents/MacOS/firebird/intl/libfbintl.dylib
cp ../builds/install/arch-specific/darwin/embed.Info.plist $(FBE)/Contents/Info.plist
sed s'/#RootDirectory =/RootDirectory = .\/firebird/' <$(FBE)/Contents/MacOS/firebird.edit >$(FBE)/Contents/MacOS/firebird.conf
rm $(FBE)/Contents/MacOS/firebird.edit
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Firebird \
../libfbembed.dylib $(BINLOC)/isql
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicuuc.dylib \
../libicuuc.dylib $(BINLOC)/isql
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicudata.dylib \
../libicudata.dylib $(BINLOC)/isql
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicui18n.dylib \
../libicui18n.dylib $(BINLOC)/isql
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Firebird \
../libfbembed.dylib $(BINLOC)/gbak
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicuuc.dylib \
../libicuuc.dylib $(BINLOC)/gbak
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicudata.dylib \
../libicudata.dylib $(BINLOC)/gbak
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicui18n.dylib \
../libicui18n.dylib $(BINLOC)/gbak
install_name_tool -change $(OLDPATH)/libicuuc.dylib @loader_path/libicuuc.dylib \
$(LIBLOC)/libfbembed.dylib
install_name_tool -change $(OLDPATH)/libicudata.dylib @loader_path/libicudata.dylib \
$(LIBLOC)/libfbembed.dylib
install_name_tool -change $(OLDPATH)/libicui18n.dylib @loader_path/libicui18n.dylib \
$(LIBLOC)/libfbembed.dylib
install_name_tool -change $(OLDPATH)/libicudata.dylib @loader_path/libicudata.dylib \
$(LIBLOC)/libicuuc.dylib
install_name_tool -change $(OLDPATH)/libicuuc.dylib @loader_path/libicuuc.dylib \
$(LIBLOC)/libicui18n.dylib
install_name_tool -change $(OLDPATH)/libicudata.dylib @loader_path/libicudata.dylib \
$(LIBLOC)/libicui18n.dylib
You can now tar a copy of the Firebird.app directory (and underlying), place anywhere on your system,
and set the RootDirectory in firebird.conf, or set the FIREBIRD environment variable to the relevant location of the firebird directory in Firebird.app, and your application should run if placed in the same directory.
I am hoping that this will generate some comments/feedback, as it is still under test at the moment.
Would Mac users like this committed to SVN? Should we produce a dedicated MacOSX embedded build along with the others (32bit/64bit/lipo)?
Thursday, September 20, 2012
Friday, September 7, 2012
CORE-3740 and Firebird V2.5.2
CORE-3740 - SELECT using IN list with 153 or more elements causes crash.
This problem only occurs on MacOS, and cannot be reproduced on Linux or Windows. Initial analysis showed that adjusting the optimisation level of the code as it links (O1 instead of O3) would increase the number of INS that could be supported but you would still get a crash eventually if the number of INS was large enough.
The consensus of opinion was that this was a stack issue, so we increased the stack from the default 8mb in launchctl to 64mb, our test still crashed.
We then tried embedded (local) firebird and increased the cache there using ulimit, strangely enough this worked. So it definitely was a stack issue but where? Some debugging code and some careful googling showed the problem. Even Classic launches a forked inet_server via a new thread.
From an Apple Technical Q&A
"Each Mac OS X process is launched with a default stack size of 8 Megabytes. This allocation is used exclusively for the main thread's stack needs. Each subsequent thread created is allocated its own default stack, the size of which differs depending on the threading API used. For example, the Mac OS X implementation of Pthreads defines a default stack size of 512 Kilobytes, while Carbon MPTasks are created with a 4 Kilobyte stack."
Patch applied to src/jrd/ThreadStart.cpp
-#ifdef _AIX
-// adjust stack size for AIX
+#if defined(_AIX) || defined(DARWIN)
+// adjust stack size
// For AIX 32-bit compiled applications, the default stacksize is 96 KB,
// see. For 64-bit compiled applications, the default stacksize
// is 192 KB. This is too small - see HP-UX note above
+// For MacOS default stack is 512 KB, which is also too small in 2012.
+
size_t stack_size;
state = pthread_attr_getstacksize(&pattr, &stack_size);
if (state)
Firebird::system_call_failed::raise("pthread_attr_getstacksize");
- if (stack_size < 0x40000L)
+ if (stack_size < 0x400000L)
{
- state = pthread_attr_setstacksize(&pattr, 0x40000L);
+ state = pthread_attr_setstacksize(&pattr, 0x400000L);
if (state)
Firebird::system_call_failed::raise("pthread_attr_setstacksize", state);
This problem only occurs on MacOS, and cannot be reproduced on Linux or Windows. Initial analysis showed that adjusting the optimisation level of the code as it links (O1 instead of O3) would increase the number of INS that could be supported but you would still get a crash eventually if the number of INS was large enough.
The consensus of opinion was that this was a stack issue, so we increased the stack from the default 8mb in launchctl to 64mb, our test still crashed.
We then tried embedded (local) firebird and increased the cache there using ulimit, strangely enough this worked. So it definitely was a stack issue but where? Some debugging code and some careful googling showed the problem. Even Classic launches a forked inet_server via a new thread.
From an Apple Technical Q&A
"Each Mac OS X process is launched with a default stack size of 8 Megabytes. This allocation is used exclusively for the main thread's stack needs. Each subsequent thread created is allocated its own default stack, the size of which differs depending on the threading API used. For example, the Mac OS X implementation of Pthreads defines a default stack size of 512 Kilobytes, while Carbon MPTasks are created with a 4 Kilobyte stack."
Patch applied to src/jrd/ThreadStart.cpp
-#ifdef _AIX
-// adjust stack size for AIX
+#if defined(_AIX) || defined(DARWIN)
+// adjust stack size
// For AIX 32-bit compiled applications, the default stacksize is 96 KB,
// see
// is 192 KB. This is too small - see HP-UX note above
+// For MacOS default stack is 512 KB, which is also too small in 2012.
+
size_t stack_size;
state = pthread_attr_getstacksize(&pattr, &stack_size);
if (state)
Firebird::system_call_failed::raise("pthread_attr_getstacksize");
- if (stack_size < 0x40000L)
+ if (stack_size < 0x400000L)
{
- state = pthread_attr_setstacksize(&pattr, 0x40000L);
+ state = pthread_attr_setstacksize(&pattr, 0x400000L);
if (state)
Firebird::system_call_failed::raise("pthread_attr_setstacksize", state);
Monday, June 18, 2012
An ODBC driver for Firebird on MacOSX
I was recently contacted by a Firebird user on MacOSX who was trying to get the Firebird ODBC driver to build on MacOSX. I thought I would try and step in to help.
For those of you who follow the CVS checkins for OdbcJdbc you might have noticed the addition of a number of a new directory in the Builds directory called Gcc.darwin containing a makefile, a readme and a .sh file to create a lipo'ed dylib. The driver sucessfully builds and passes some simple tests. Feel free to contact me for a copy of the dylib to test further.
The readme contains the following information:
These instructions should allow a user to get a working user DSN for ODBC to connect
to Firebird on MacOS. Comments please to Paul Beach (pbeach at ibphoenix.com.
To build the library, edit makefile.darwin, and select the ARCH (i386 or X86_64).
then make -B -f makefile.darwin all
The Firebird ODBC library can be built in both 64bit or 32bit format.
lipo.sh creates a fat libary that can be used on either version of Firebird.
1. Download the MacOSX ODBC Administrator from support.apple.com
http://support.apple.com/downloads/ODBC_Administrator_Tool_for_Mac_OS_X
and install.
2. Once installed it can be accessed via Applications/Utilities/ODBC Administrator
3. Place the libOdbcFB.dylib in $HOME/odbc for example then add the Firebird driver
(dylib) name and location to the Drivers tab in the Administrator.
e.g
Description: Firebird ODBC Driver
Driver file: /Users/username/odbc/libOdbcFB.dylib
Define as: User
This will create an odbcinst.ini file in $HOME/Library/ODBC
4. Now you need to create a User DSN via an odbc.ini file.
Use the text below as an example, copy and paste into an odbc.ini
file placed in $HOME/Library/ODBC
Make sure that you modify the text so it points to your database
and uses your username and password.
[ODBC Data Sources]
Test = Firebird
[Test]
Driver = /Users/username/odbc/libOdbcFb.dylib
Description = Test Firebird ODBC
Dbname = localhost:/Users/databases/test.fdb
Client =
User = SYSDBA
Password = masterkey
Role =
CharacterSet = NONE
ReadOnly = No
NoWait = No
Dialect = 3
QuotedIdentifier = Yes
SensitiveIdentifier = No
AutoQuotedIdentifier = No
[ODBC]
Trace = 0
TraceAutoStop = 0
TraceFile =
TraceLibrary =
This User DSN should appear in the User DSN tab the next time you load the
ODBC Administrator.
5. You can test whether it works using iodbctest and then using the dsn
dsn=Test, if all is well it should connect and you can issue SQL statements.
To create a System wide version of the ODBC driver, copy the libOdbcFB.dylib
to /usr/lib make sure the Administrators Drivers tab now points to this file.
Copy the DSN above to /Library/ODBC and modify.
Note: (13th June 2012)
The ODBC library is linked to libfbclient.dylib found in the Firebird framework
Libraries directory. Not all SuperServer builds of Firebird have this library installed
by default. If this is the case get a copy of the Firebird Classic build and extract
the libfbclient library and place it in
/Library/Frameworks/Firebird.framework/Versions/A/Libraries
For those of you who follow the CVS checkins for OdbcJdbc you might have noticed the addition of a number of a new directory in the Builds directory called Gcc.darwin containing a makefile, a readme and a .sh file to create a lipo'ed dylib. The driver sucessfully builds and passes some simple tests. Feel free to contact me for a copy of the dylib to test further.
The readme contains the following information:
These instructions should allow a user to get a working user DSN for ODBC to connect
to Firebird on MacOS. Comments please to Paul Beach (pbeach at ibphoenix.com.
To build the library, edit makefile.darwin, and select the ARCH (i386 or X86_64).
then make -B -f makefile.darwin all
The Firebird ODBC library can be built in both 64bit or 32bit format.
lipo.sh creates a fat libary that can be used on either version of Firebird.
1. Download the MacOSX ODBC Administrator from support.apple.com
http://support.apple.com/downloads/ODBC_Administrator_Tool_for_Mac_OS_X
and install.
2. Once installed it can be accessed via Applications/Utilities/ODBC Administrator
3. Place the libOdbcFB.dylib in $HOME/odbc for example then add the Firebird driver
(dylib) name and location to the Drivers tab in the Administrator.
e.g
Description: Firebird ODBC Driver
Driver file: /Users/username/odbc/libOdbcFB.dylib
Define as: User
This will create an odbcinst.ini file in $HOME/Library/ODBC
4. Now you need to create a User DSN via an odbc.ini file.
Use the text below as an example, copy and paste into an odbc.ini
file placed in $HOME/Library/ODBC
Make sure that you modify the text so it points to your database
and uses your username and password.
[ODBC Data Sources]
Test = Firebird
[Test]
Driver = /Users/username/odbc/libOdbcFb.dylib
Description = Test Firebird ODBC
Dbname = localhost:/Users/databases/test.fdb
Client =
User = SYSDBA
Password = masterkey
Role =
CharacterSet = NONE
ReadOnly = No
NoWait = No
Dialect = 3
QuotedIdentifier = Yes
SensitiveIdentifier = No
AutoQuotedIdentifier = No
[ODBC]
Trace = 0
TraceAutoStop = 0
TraceFile =
TraceLibrary =
This User DSN should appear in the User DSN tab the next time you load the
ODBC Administrator.
5. You can test whether it works using iodbctest and then using the dsn
dsn=Test, if all is well it should connect and you can issue SQL statements.
To create a System wide version of the ODBC driver, copy the libOdbcFB.dylib
to /usr/lib make sure the Administrators Drivers tab now points to this file.
Copy the DSN above to /Library/ODBC and modify.
Note: (13th June 2012)
The ODBC library is linked to libfbclient.dylib found in the Firebird framework
Libraries directory. Not all SuperServer builds of Firebird have this library installed
by default. If this is the case get a copy of the Firebird Classic build and extract
the libfbclient library and place it in
/Library/Frameworks/Firebird.framework/Versions/A/Libraries
Wednesday, February 22, 2012
Firebird V2.0.7
We are currently preparing to release Firebird 2.0.7, since I take responsibility for the Mac builds, I did a 2.0.7 build on MacOSX 10.7 using the development tools installed by XCode 4.1 (gcc 4.2.1 etc). I set up the relevant environment variables for this older 32bit only build CFLAGS, CXXFLAGS, LD_FLAGS and also set the MACOSX_DEPLOYMENT_TARGET=10.4. The builds completed without any problems, some simple tests on MacOSX 10.7 showed no problems.
Now - Imagine my surprise when Philippe told me that when he tried to QA the builds on MacOSX 10.5 we got this error on SuperServer startup.
Process: fbserver [623]
Path:
/Library/Frameworks/Firebird.framework/Resources/English.lproj/var/bin/fbserver
Identifier: fbserver
Version: ??? (???)
Code Type: X86 (Native)
Parent Process: fbguard [310]
Date/Time: 2012-02-09 09:42:13.939 +0100
OS Version: Mac OS X 10.5.8 (9L31a)
Report Version: 6
Anonymous UUID: FA7F8C0C-581B-4153-ADBE-2BCB59C5F823
Exception Type: EXC_BREAKPOINT (SIGTRAP)
Exception Codes: 0x0000000000000002, 0x0000000000000000
Crashed Thread: 0
Dyld Error Message:
Symbol not found: ___moddi3
Referenced from:
/Library/Frameworks/Firebird.framework/Resources/English.lproj/var/bin/fbserver
Expected in: /usr/lib/libSystem.B.dylib
A simple nm and grep shows the following for the 10.5 libSystem.B.dylib
nm /usr/lib/libSystem.B.dylib | grep moddi3
00083490 t ___moddi3
0003df70 t ___umoddi3
On 10.5 ___moddi3 is defined in libgcc_s.1.dylib which links to libsystem.B.dylib
From what I can gather these *moddi3 symbols are routines for doing 64bit maths on 32bit systems.
However the same on 10.7 returns nothing, however the symbols can be found in libgcc_s.10.5.dylib which does not link to libgcc_s.1.dylib hence the missing symbols message.
Supposedly this can be fixed by linking explicity to to libgcc_s.10.5.dylib
using -lgcc_s.10.5. I got round it by booting up my copy of Snow Leopard (10.6) and
building Firebird 2.0.7 there instead of on MacOSX 10.7
So basically if you want to buld an application that is deployable on 10.5 or less, you can't - unless you know how to get around the above problem. Its as if although you can build 32bit applications for 32bit versions of MacOSX, they won't run, because they are incompatible with any version of the OS less than 10.6.
Now - Imagine my surprise when Philippe told me that when he tried to QA the builds on MacOSX 10.5 we got this error on SuperServer startup.
Process: fbserver [623]
Path:
/Library/Frameworks/Firebird.framework/Resources/English.lproj/var/bin/fbserver
Identifier: fbserver
Version: ??? (???)
Code Type: X86 (Native)
Parent Process: fbguard [310]
Date/Time: 2012-02-09 09:42:13.939 +0100
OS Version: Mac OS X 10.5.8 (9L31a)
Report Version: 6
Anonymous UUID: FA7F8C0C-581B-4153-ADBE-2BCB59C5F823
Exception Type: EXC_BREAKPOINT (SIGTRAP)
Exception Codes: 0x0000000000000002, 0x0000000000000000
Crashed Thread: 0
Dyld Error Message:
Symbol not found: ___moddi3
Referenced from:
/Library/Frameworks/Firebird.framework/Resources/English.lproj/var/bin/fbserver
Expected in: /usr/lib/libSystem.B.dylib
A simple nm and grep shows the following for the 10.5 libSystem.B.dylib
nm /usr/lib/libSystem.B.dylib | grep moddi3
00083490 t ___moddi3
0003df70 t ___umoddi3
On 10.5 ___moddi3 is defined in libgcc_s.1.dylib which links to libsystem.B.dylib
From what I can gather these *moddi3 symbols are routines for doing 64bit maths on 32bit systems.
However the same on 10.7 returns nothing, however the symbols can be found in libgcc_s.10.5.dylib which does not link to libgcc_s.1.dylib hence the missing symbols message.
Supposedly this can be fixed by linking explicity to to libgcc_s.10.5.dylib
using -lgcc_s.10.5. I got round it by booting up my copy of Snow Leopard (10.6) and
building Firebird 2.0.7 there instead of on MacOSX 10.7
So basically if you want to buld an application that is deployable on 10.5 or less, you can't - unless you know how to get around the above problem. Its as if although you can build 32bit applications for 32bit versions of MacOSX, they won't run, because they are incompatible with any version of the OS less than 10.6.
Tuesday, July 26, 2011
Firebird 2.5 SuperServer and MacOSX 10.7 (Lion)
There is a small problem with Firebird 2.5 SuperServer and MacOSX 10.7 (Lion)...
SuperServer will not start, and produces a crash report header similar to this
Update 26th August 2011
Its not just 2.5 SuperServer, reports have come in that SuperServer for Firebird 2.1.4 will not start, also the same problem exists for Firebird 2.5 SuperClassic.
Process: fbserver [706]
Path: /Library/Frameworks/firebird.framework/Versions/A/Resources/bin/fbserver
Identifier: fbserver
Version: ??? (???)
Code Type: X86 (Native)
Parent Process: fbguard [166]
Date/Time: 2011-06-10 07:59:42.374 -0400
OS Version: Mac OS X 10.7 (11A480b)
Report Version: 9
Anonymous UUID: B7EDF790-CE72-4B21-A982-B9EA4F4E2088
Crashed Thread: 1 Dispatch queue: com.apple.libdispatch-manager
Exception Type: EXC_BAD_INSTRUCTION (SIGILL)
Exception Codes: 0x0000000000000001, 0x0000000000000000
Application Specific Information:
BUG IN CLIENT OF LIBDISPATCH: Do not close random Unix descriptors
The bug is fixed in Firebird 2.5.1 but Firebird 2.5.1 is not available yet. So I have fixed the problem also in the Firebird 2.5 code and produced 32bit and 64bit builds that work properly.
I have had new builds uploaded to Sourceforge to fix the problem with Firebird
2.1.4 SuperServer, you can distinguish the new builds from the old by the build no.
Old = 18393, New = 18393. Because we have some issues with 2.5.1 that are currently being investigated, I plan to update the 2.5 builds on Sourceforge soon.
Update 28th Sep 2011
The Firebird 2.5 builds have also been replaced on Sourceforge, so you just need to download the latest build. However Firebird 2.5 will be replaced by Firebird 2.5.1 very shortly, builds are now taking place and will be QA'd and released shortly.
Any problems - please let me know.
SuperServer will not start, and produces a crash report header similar to this
Update 26th August 2011
Its not just 2.5 SuperServer, reports have come in that SuperServer for Firebird 2.1.4 will not start, also the same problem exists for Firebird 2.5 SuperClassic.
Process: fbserver [706]
Path: /Library/Frameworks/firebird.framework/Versions/A/Resources/bin/fbserver
Identifier: fbserver
Version: ??? (???)
Code Type: X86 (Native)
Parent Process: fbguard [166]
Date/Time: 2011-06-10 07:59:42.374 -0400
OS Version: Mac OS X 10.7 (11A480b)
Report Version: 9
Anonymous UUID: B7EDF790-CE72-4B21-A982-B9EA4F4E2088
Crashed Thread: 1 Dispatch queue: com.apple.libdispatch-manager
Exception Type: EXC_BAD_INSTRUCTION (SIGILL)
Exception Codes: 0x0000000000000001, 0x0000000000000000
Application Specific Information:
BUG IN CLIENT OF LIBDISPATCH: Do not close random Unix descriptors
The bug is fixed in Firebird 2.5.1 but Firebird 2.5.1 is not available yet. So I have fixed the problem also in the Firebird 2.5 code and produced 32bit and 64bit builds that work properly.
I have had new builds uploaded to Sourceforge to fix the problem with Firebird
2.1.4 SuperServer, you can distinguish the new builds from the old by the build no.
Old = 18393, New = 18393. Because we have some issues with 2.5.1 that are currently being investigated, I plan to update the 2.5 builds on Sourceforge soon.
Update 28th Sep 2011
The Firebird 2.5 builds have also been replaced on Sourceforge, so you just need to download the latest build. However Firebird 2.5 will be replaced by Firebird 2.5.1 very shortly, builds are now taking place and will be QA'd and released shortly.
Any problems - please let me know.
Monday, July 4, 2011
Firebird V2.1 Error: value exceeds the range for valid dates
You are restoring a backup of a Firebird 1.5 or 2.0 database to Firebird 2.1 and you see an error similar to this:
gbak: writing data for table xyz
gbak:20000 records written
gbak: Error: value exceeds the range for valid dates
gbak: Error: gds_$receive failed
gbak:Exiting before completion due to errors
How do you go about solving the problem?
Well first of all a quick visit to the Firebird bug tracker reveals
http://tracker.firebirdsql.org/browse/CORE-1714 and a couple of comments from Dimitry Yemanov.
“The error means to say that some column has an invalid date value (outside the supported range). Prior to V2.1, it was possible to store such invalid values in the database, but now it's prohibited. A verbose output should point you to a problematic table.” Also “The current behavior is intended and is unlikely to be changed.”
1. Firstly use the –v (verbose option of gbak) to find out the table that is causing the problem.
2. Check the table xyz for date columns
3. Perform the following SQL operation on all the date columns you found in 2.
isql> update XYZ set MY_DATE_COLUMN = MY_DATE_COLUMN;
At some point during the process you will see the same error that gbak produced, but now you will know which column is causing the problem.
4. Install Firebird 1.5 or 2.0, by going back to the version of Firebird that allows for invalid dates, you will be at least able to correct the invalid date to something more appropriate.
5. Now lets check for dates that are outside of their proper range (01 Jan 0001 - 31 Dec 9999)
Below date zero:
isql> select PRIMARY_KEY from XYZ where MY_DATE_COLUMN < '0001-01-01'
if an error occurs correct the date to something more appropriate and meaningful
Maximum date
isql> select PRIMARY_KEY from XYZ where MY_DATE_COLUMN > '9999-12-31'
if an error occurs correct the date to something more appropriate and meaningful
6. You can now backup the database using Firebird 1.5 or 2.1 and successfully restore under 2.1
gbak: writing data for table xyz
gbak:20000 records written
gbak: Error: value exceeds the range for valid dates
gbak: Error: gds_$receive failed
gbak:Exiting before completion due to errors
How do you go about solving the problem?
Well first of all a quick visit to the Firebird bug tracker reveals
http://tracker.firebirdsql.org/browse/CORE-1714 and a couple of comments from Dimitry Yemanov.
“The error means to say that some column has an invalid date value (outside the supported range). Prior to V2.1, it was possible to store such invalid values in the database, but now it's prohibited. A verbose output should point you to a problematic table.” Also “The current behavior is intended and is unlikely to be changed.”
1. Firstly use the –v (verbose option of gbak) to find out the table that is causing the problem.
2. Check the table xyz for date columns
3. Perform the following SQL operation on all the date columns you found in 2.
isql> update XYZ set MY_DATE_COLUMN = MY_DATE_COLUMN;
At some point during the process you will see the same error that gbak produced, but now you will know which column is causing the problem.
4. Install Firebird 1.5 or 2.0, by going back to the version of Firebird that allows for invalid dates, you will be at least able to correct the invalid date to something more appropriate.
5. Now lets check for dates that are outside of their proper range (01 Jan 0001 - 31 Dec 9999)
Below date zero:
isql> select PRIMARY_KEY from XYZ where MY_DATE_COLUMN < '0001-01-01'
if an error occurs correct the date to something more appropriate and meaningful
Maximum date
isql> select PRIMARY_KEY from XYZ where MY_DATE_COLUMN > '9999-12-31'
if an error occurs correct the date to something more appropriate and meaningful
6. You can now backup the database using Firebird 1.5 or 2.1 and successfully restore under 2.1
Tuesday, May 10, 2011
Firebird 3.0 - Shared Page Cache
Posted by Vlad on the Firebird Development List.
After more than a year of development, testing, switching on another tasks and returning back I'm ready to commit shared page cache implementation into trunk.
I consider it stable enough to be committed into SVN. It runs some stability tests more than 10 hours for longest run and many shorter runs at different configurations.
Here I want to do overview of changes, what was tested, what was not, and what is work in progress.
Page cache was re-implemented looking at both Vulcan code (thanks to Jim) and our code base. It is not a blind copy of Vulcan code, every decision was thinking on and adjusted when needed with my understanding and expirience with existing code.
The cache is syncronized at different levels. There is a few BufferControl (BCB) level sync's used to syncronize access to the common structures such as LRU queue, Precedence graph, Dirty page list and so one. These common sync's should be locked at as short as possible period of time as this is kind of global locks. Also there is a two syncs for every BufferDesc (BDB) - one for guard contents of page buffer tself and second one is to prevent partial writes of page wich edition is not completed and to prevent simultaneous write of the same page by different threads.
I don't explain cache algoritms here, if someone have a question I'll be happy to answer it. And I not going to said that page cache implementation is frozen - if necessary we will change it of course (there is still space for impovements and bugs fixing).
You know I already committed SyncObject class (and related classes) ported from Vulcan. I removed some unused code and restored fairness of locking : initially it was fair in Vulcan too (i.e. all lock requests was put in waiting queue and granted in first-in-first-out, or fair, order), but later Jim found some (unknown to me) ssue with this and made preference for SHARED locks. I found no performance issues with fair locking so I decided to revert that and restore original behavior. Of course it could be changed when necessary.
Shared page cache is not an isolated change. It affects many parts of engine and our syncronization model changed significantly again. There was agreement that we will not implement shared metadata cache in v3 as this is risky and we could not deliver V3 in reasonable time frame.
In shared cache mode we have single Database instance and many attachment instances linked to it (this is not new).
All metadata objects moved into Attachment. Metadata syncronization is guarded by attachment's mutex now. Database::SyncGuard and company are replaced by corresponding Attachment::XXX classes.
To make AST's work we need to release attachment mutex sometimes. This is very important change after v2.5 : in v2.5 attachment mutex is locked during whole duration of API call and no other API call (except of asyncronous fb_cancel_operation) could work with "busy" attacment. In V3 this rule is not worked anymore. So, now we could run more that one API call on the same attachment (of course not really simultaneously). I'm not sure it is safe but not disabled it so far.
To make asyncronous detach safe I introduced att_use_count counter which is incremented each time when API call is entered and decremented on exit. detach now marks attachment as shutdown and waits for att_use_count == 0 before processing.
Parallel access to the attachment could be easy disabled making every API call wait for att_use_count == 0 on enter or even introducing one more mutex to avoid spin wait.
Also it seems this counter make obsolete att_in_use member as detach call should wait for att_use_count == 0 and drop call should return "object is in use" if att_use_count != 0.
All AST's related to attachment-level objects should take attachment mutex before access attachment internals. This is implemented but not tested!
Transaction inventory pages cache (TIP cache) was reworked and is shared by all attachments.
To avoid contention on common dbb_pool its usage was replaced by att_pool when possible. To make this task slightly easy there was introduced jrd_rel::rel_pool which points currently to the attachment's pool. All relation's "sub-objects" (such as formats, fields, etc) is allocated using rel_pool (it was dbb_pool before). When we'll return metadata objects back to the Database it will be easy to redirect rel_pool to dbb_pool at one place in code instead of makeing tens of small changes again.
About stability testing of different parts of the engine:
- page cache - tested and worked
- nbackup - tested and worked
- monitoring - tested (less) and seems worked
- server stop\database shutdown - somewhat tested, no crash observed, client reaction is not perfect (mostly network write errors)
- shadow - not tested
- metadata changes in concurrent environment - not tested
- garbage collection thread - not tested, needs review and rework of some
implementation details
- cache writer thread - not tested, needs review and rework
- external connections - not tested
- external engines - not tested
In configuration file there are two new settings :
a) SharedCache - boolean value which rules the page cache mode
b) SharedDatabase - boolean value which which rules the database file open mode.
Currently they are common (as whole configuration) but soon it will be per-database settings. Below is few examples of how it could be used :
- SharedCache = true, SharedDatabase = false (default mode)
this is traditional SuperServer mode when all attachments share page cache and database file is opens in exclusive mode (only one server process could work with database)
- SharedCache = false, SharedDatabase = true
this is Classic mode when each attachment have its own page cache and many server processes could work with the same database.
To run SuperClassic you should use switch -m in command line of firebird.exe (on Windows) or run fb_smp_server (on Posix, here i'm not sure and Alex will correct me)
Else ClassicServer will run.
- SharedCache = true, SharedDatabase = true
this is completely new mode in which database file could be opened by many server processes and each process could handle many attachments which will share page cache (i.e. per-process page cache).
It could be used to run few SS processes, for example, or to run "main" SS process and have ability to attach using embedded to make some special tasks.
Must note that our lock manager is not ready to work in this mode, so we can't use it right now.
Also there is unknown how performance will be affected when few SS with big cache will work with the same database.
- SharedCache = false, SharedDatabase = false
Looks like single process with single attachment will be allowed to work with database with such settings. Probably you can find an applications for it ;)
One more change in configuration is that CpuAffinityMask setting changed its default value and it is 0 now. It allows new SS to use all available CPU's\cores by default.
After more than a year of development, testing, switching on another tasks and returning back I'm ready to commit shared page cache implementation into trunk.
I consider it stable enough to be committed into SVN. It runs some stability tests more than 10 hours for longest run and many shorter runs at different configurations.
Here I want to do overview of changes, what was tested, what was not, and what is work in progress.
Page cache was re-implemented looking at both Vulcan code (thanks to Jim) and our code base. It is not a blind copy of Vulcan code, every decision was thinking on and adjusted when needed with my understanding and expirience with existing code.
The cache is syncronized at different levels. There is a few BufferControl (BCB) level sync's used to syncronize access to the common structures such as LRU queue, Precedence graph, Dirty page list and so one. These common sync's should be locked at as short as possible period of time as this is kind of global locks. Also there is a two syncs for every BufferDesc (BDB) - one for guard contents of page buffer tself and second one is to prevent partial writes of page wich edition is not completed and to prevent simultaneous write of the same page by different threads.
I don't explain cache algoritms here, if someone have a question I'll be happy to answer it. And I not going to said that page cache implementation is frozen - if necessary we will change it of course (there is still space for impovements and bugs fixing).
You know I already committed SyncObject class (and related classes) ported from Vulcan. I removed some unused code and restored fairness of locking : initially it was fair in Vulcan too (i.e. all lock requests was put in waiting queue and granted in first-in-first-out, or fair, order), but later Jim found some (unknown to me) ssue with this and made preference for SHARED locks. I found no performance issues with fair locking so I decided to revert that and restore original behavior. Of course it could be changed when necessary.
Shared page cache is not an isolated change. It affects many parts of engine and our syncronization model changed significantly again. There was agreement that we will not implement shared metadata cache in v3 as this is risky and we could not deliver V3 in reasonable time frame.
In shared cache mode we have single Database instance and many attachment instances linked to it (this is not new).
All metadata objects moved into Attachment. Metadata syncronization is guarded by attachment's mutex now. Database::SyncGuard and company are replaced by corresponding Attachment::XXX classes.
To make AST's work we need to release attachment mutex sometimes. This is very important change after v2.5 : in v2.5 attachment mutex is locked during whole duration of API call and no other API call (except of asyncronous fb_cancel_operation) could work with "busy" attacment. In V3 this rule is not worked anymore. So, now we could run more that one API call on the same attachment (of course not really simultaneously). I'm not sure it is safe but not disabled it so far.
To make asyncronous detach safe I introduced att_use_count counter which is incremented each time when API call is entered and decremented on exit. detach now marks attachment as shutdown and waits for att_use_count == 0 before processing.
Parallel access to the attachment could be easy disabled making every API call wait for att_use_count == 0 on enter or even introducing one more mutex to avoid spin wait.
Also it seems this counter make obsolete att_in_use member as detach call should wait for att_use_count == 0 and drop call should return "object is in use" if att_use_count != 0.
All AST's related to attachment-level objects should take attachment mutex before access attachment internals. This is implemented but not tested!
Transaction inventory pages cache (TIP cache) was reworked and is shared by all attachments.
To avoid contention on common dbb_pool its usage was replaced by att_pool when possible. To make this task slightly easy there was introduced jrd_rel::rel_pool which points currently to the attachment's pool. All relation's "sub-objects" (such as formats, fields, etc) is allocated using rel_pool (it was dbb_pool before). When we'll return metadata objects back to the Database it will be easy to redirect rel_pool to dbb_pool at one place in code instead of makeing tens of small changes again.
About stability testing of different parts of the engine:
- page cache - tested and worked
- nbackup - tested and worked
- monitoring - tested (less) and seems worked
- server stop\database shutdown - somewhat tested, no crash observed, client reaction is not perfect (mostly network write errors)
- shadow - not tested
- metadata changes in concurrent environment - not tested
- garbage collection thread - not tested, needs review and rework of some
implementation details
- cache writer thread - not tested, needs review and rework
- external connections - not tested
- external engines - not tested
In configuration file there are two new settings :
a) SharedCache - boolean value which rules the page cache mode
b) SharedDatabase - boolean value which which rules the database file open mode.
Currently they are common (as whole configuration) but soon it will be per-database settings. Below is few examples of how it could be used :
- SharedCache = true, SharedDatabase = false (default mode)
this is traditional SuperServer mode when all attachments share page cache and database file is opens in exclusive mode (only one server process could work with database)
- SharedCache = false, SharedDatabase = true
this is Classic mode when each attachment have its own page cache and many server processes could work with the same database.
To run SuperClassic you should use switch -m in command line of firebird.exe (on Windows) or run fb_smp_server (on Posix, here i'm not sure and Alex will correct me)
Else ClassicServer will run.
- SharedCache = true, SharedDatabase = true
this is completely new mode in which database file could be opened by many server processes and each process could handle many attachments which will share page cache (i.e. per-process page cache).
It could be used to run few SS processes, for example, or to run "main" SS process and have ability to attach using embedded to make some special tasks.
Must note that our lock manager is not ready to work in this mode, so we can't use it right now.
Also there is unknown how performance will be affected when few SS with big cache will work with the same database.
- SharedCache = false, SharedDatabase = false
Looks like single process with single attachment will be allowed to work with database with such settings. Probably you can find an applications for it ;)
One more change in configuration is that CpuAffinityMask setting changed its default value and it is 0 now. It allows new SS to use all available CPU's\cores by default.
Subscribe to:
Posts (Atom)
