tag:blogger.com,1999:blog-89867825578793162382024-02-20T11:49:57.248-08:00Paul Beach's BlogPaul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.comBlogger87125tag:blogger.com,1999:blog-8986782557879316238.post-49796416893225598402021-01-08T06:41:00.008-08:002021-01-11T01:11:46.597-08:00Firebird Embedded in a sandboxed MacOS App<p>For those who might not be aware, Firebird on MacOS is now relocatable, in that you don't necessarily have to install it as a Framework, this also means that you can create an embedded version out of the current installer.<br /><br />A typical structure would look like this<br /><br />|-- firebird.conf<br />|-- firebird.msg<br />|-- intl<br />| |-- fbintl.conf<br />| `-- libfbintl.dylib<br />|-- lib<br />| |-- libfbclient.dylib<br />| |-- libib_util.dylib<br />| |-- libicudata.dylib<br />| |-- libicui18n.dylib<br />| |-- libicuuc.dylib<br />|`-- plugins<br /> `-- libEngine12.dylib <br /><br />And the firebird.conf file would then typically be amended for the following<br /><br />Providers = Engine12<br />ServerMode = Classic <br /> <br />For the last few weeks Alex and I (along with a Firebird user) have been working on getting Firebird embedded to work properly in a sandboxed app that can then be deployed on the App Store...<br /> <br />To do this we had to solve issues with temp files, the use of inter process communications by the<br />Firebird lock manager and the location of the Firebird log file.<br /><br />Note: adding LSEnvironment to the plist file defining new locations for FIREBIRD_TEMP and FIREBIRD_LOCK does not work.<br /><br /><key>LSEnvironment</key><br /><dict><br /> <key>ENV_VAR</key><br /> <string>value</string><br /></dict><br /> <br />So the first issue to be addressed was the following<br /><br />macosx Wed Dec 2 15:33:57 2020<br /> ConfigStorage: Cannot initialize the shared memory region<br /> Can not access lock files directory /tmp/firebird.tmp.YDzrhQ<br /><br />It seems thats sandboxed Apps cannot to write into /tmp at all, they have to use their own /tmp-folder. We can detect whether we are running in a sandboxed environment using the following calls to the Mac security subsystem<br /><br />task = SecTaskCreateFromSelf(nil), <br />value = SecTaskCopyValueForEntitlement(task, "com.apple.security.app-sandbox" as CFString, nil),<br /><br />Now if we now know if we are sandboxed we can use NSTemporaryDirectory() for the location of the relevant Firebird temporary files. <br /><br />Having fixed that issue, it was time to move onto the lock manager.<br /> <br />[FireDAC][Phys][FB]lock manager error.)<br /> <br />macosx Thu Dec 17 17:37:00 2020<br /> event_init()<br /> operating system directive semctl failed<br /> Operation not permitted<br /> <br />According to the Apple sandbox guide restricting IPC (Inter Process Communication) is also part of MacOS sandbox implementation. After some serious head scratching we tried the following test<br /> <br />#include <stdio.h><br />#include <pthread.h><br /> <br />#define ER(x) { int tmpState = (x); if (tmpState) { printf("Failed %s error %d\n", #x, tmpState); } }<br /> <br />int main() {<br /> <br /> pthread_mutex_t mutex;<br /> pthread_mutexattr_t mutexattr;<br /> <br /> ER(pthread_mutexattr_init(&mutexattr));<br /> ER(pthread_mutexattr_setpshared(&mutexattr, PTHREAD_PROCESS_SHARED));<br /> ER(pthread_mutex_init(&mutex, &mutexattr));<br /> ER(pthread_mutex_lock(&mutex));<br /> ER(pthread_mutex_unlock(&mutex));<br /> ER(pthread_mutex_destroy(&mutex));<br /> <br />c++ test.cpp -pthread<br /> <br />In case of failure it will print error messages, on success we should get nothing. We got nothing. <br /><br />For many years Firebird worked with system V IPC on MacOS which is different to many other *nixes where mutexes and conditional variables in shared memory were used instead. The use of system V IPC was caused by a lack of for shared mutexes on MacOS when Firebird was originally ported to it. Currently (as was proved by test above) MacOS now supports such mutexes. Based on this it we decided to stop using system V IPC in Firebird and a perform a cleanup of the lock manager code as part of the effort to provide sandbox support. As a side effect Firebird should now run faster using shared mutexes.<br /> <br />At the same time as we worked on the lock manager issue we also had an issue with the firebird log file, we were not allowed to write to its normal default location by the sandbox. Because we now know we are running in a sandbox we can relocate the placement of the log file within utils.cpp<br /> <br />Supposedly using something like ~/Library/Application Support/Firebird/ should work<br /> <br />case Firebird::IConfigManager::DIR_LOG:<br /> s = "~/Library/Application Support/Firebird/";<br /> s += name;<br /> return s;<br /> <br />Unfortunately it seems this workaround was removed relatively recently forcing us to resort to putting the log file within the App itself ~/LibraryContainers/Company.App/Data/Library/Application Support/Company/<br /> <br />Changing the location of the log file is relatively easy but although this allowed the App to run and resolved the issue, it was not an ideal solution since the location of the log file is now hard coded within Firebird and is App dependent and this would mean that every time somebody wanted an embedded Firebird for a sandboxed App they would have to change the path to the log file and recompile Firebird.<br /> <br />The solution was to do away with the Firebird log file and use the OSLog framework instead, and use this to capture any messages from the database engine, so if you are running embedded Firebird in a sandboxed App you can access the Firebird log messages using the Console or a terminal command like the following<br /> <br />log show --predicate 'eventMessage contains "macpro.home"' --start '2021-01-06 14:00:00' --end '2021-01-06 14:30:00' --info<br /> <br />where macpro.home is the name of the computer. You don't need the start and end, but it does help reduce the number of messages. <br /> <br />There is an added plus from using this. The OSLog framework is also supported on IOS 10+ and now that the App Store will accept dynamic libraries, this means that we should be able to compile an embedded version of Firebird for IOS that can also run sandboxed Apps. If anyone is interested in sponsoring this work please contact me. </p><p>This work was sponsored by kiC Gesellschaft für Softwareentwicklung mbH.<br /></p>Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com5tag:blogger.com,1999:blog-8986782557879316238.post-68305806581615455022019-09-11T04:51:00.002-07:002019-10-01T02:17:29.600-07:00Firebird 3 Embedded on MacOSX<br />
I have finally managed to get around to preparing a mechanism for creating an embedded Firebird Framework on MacOSX (many thanks to the customer who sponsored the work). I will be committing the updated makefile embed.darwin shortly to B3_0_Release.<br />
You can use the makefile to create a 32bit or 64 bit framework, depending on the build of Firebird you want.<br />
Once you have a build of Firebird, you need to run make.platform.postfix, and then embed.darwin. The embeded framework is created in gen/Release/frameworks.<br />
<br />
If you can't build Firebird from scratch you can download a copy of the embedded framework (currently Firebird 3.0.4) from IBPhoenix. (Approx 15mb)<br />
<a href="http://www.ibphoenix.com/downloads/MacEmbedded3x32.tar.gz"><br /></a>
<a href="http://www.ibphoenix.com/downloads/MacEmbedded3x32.tar.gz">32bit Embedded Framework</a><br />
<a href="http://www.ibphoenix.com/downloads/MacEmbedded3x64.tar.gz">64bit Embedded Framework </a><br />
<br />
Below is the embed.darwin makefile (updated 01-Oct-2019)<br />
<br />
# Makefile script to generate an embedded Firebird Framework from a sucessful Firebird build <br /># To be run from the gen directory of a Firebird Release build<br /># Your application needs to be placed in the Resources/app directory.<br /><br /> FBE=../gen/Release/frameworks/FirebirdEmbedded.framework<br /> BINLOC=../gen/Release/frameworks/FirebirdEmbedded.framework/Versions/A/Resources/bin<br /> LIBLOC=../gen/Release/frameworks/FirebirdEmbedded.framework/Versions/A<br /> INTLOC=../gen/Release/frameworks/FirebirdEmbedded.framework/Versions/A/Resources/intl<br /> PLULOC=../gen/Release/frameworks/FirebirdEmbedded.framework/Versions/A/Resources/plugins<br /> UTILOC=../gen/Release/frameworks/FirebirdEmbedded.framework/Versions/A/Libraries<br /> OLDPATH=/Library/Frameworks/Firebird.framework/Versions/A/Libraries<br /><br />all:<br /><br /> -$(RM) -rf $(FBE)<br /> mkdir -p $(FBE)/Versions/A/Resources<br /> mkdir -p $(FBE)/Versions/A/Resources/intl<br /> mkdir -p $(FBE)/Versions/A/Resources/plugins<br /> mkdir -p $(FBE)/Versions/A/Resources/bin<br /> mkdir -p $(FBE)/Versions/A/Resources/app<br /> mkdir -p $(FBE)/Versions/A/Headers<br /> mkdir -p $(FBE)/Versions/A/Libraries<br /> ln -s Versions/Current/Headers $(FBE)/Headers<br /> ln -s Versions/Current/Resources $(FBE)/Resources<br /> ln -s Versions/Current/Libraries $(FBE)/Libraries<br /> ln -s A $(FBE)/Versions/Current<br /><br /> cp ../gen/Release/firebird/firebird.conf $(FBE)/Versions/A/Resources/firebird.conf<br /> cp ../gen/Release/firebird/plugins.conf $(FBE)/Versions/A/Resources/plugins.conf<br /> cp ../gen/Release/firebird/firebird.msg $(FBE)/Versions/A/Resources/firebird.msg<br /> cp ../gen/Release/firebird/lib/libfbclient.dylib $(FBE)/Versions/A/libfbclient.dylib<br /> cp ../gen/Release/firebird/plugins/libEngine12.dylib $(FBE)/Versions/A/Resources/plugins/libEngine12.dylib<br /> cp ../gen/Release/firebird/lib/libicudata.dylib $(FBE)/Versions/A/libicudata.dylib<br /> cp ../gen/Release/firebird/lib/libicui18n.dylib $(FBE)/Versions/A/libicui18n.dylib<br /> cp ../gen/Release/firebird/lib/libicuuc.dylib $(FBE)/Versions/A/libicuuc.dylib<br /> cp ../gen/Release/firebird/bin/gbak $(FBE)/Versions/A/Resources/bin/gbak<br /> cp ../gen/Release//firebird/bin/isql $(FBE)/Versions/A/Resources/bin/isql<br /> cp ../gen/Release/firebird/intl/fbintl.conf $(FBE)/Versions/A/Resources/intl/fbintl.conf<br /> cp ../gen/Release/firebird/intl/libfbintl.dylib $(FBE)/Versions/A/Resources/intl/libfbintl.dylib<br /> cp ../gen/Release/firebird/lib/libib_util.dylib $(FBE)/Versions/A/Libraries/libib_util.dylib<br /><br /><br /> install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Firebird \<br /> @loader_path/../../libfbclient.dylib $(BINLOC)/isql<br /> install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Firebird \<br /> @loader_path/../../libfbclient.dylib $(PLULOC)/libEngine12.dylib<br /> install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicuuc.dylib \<br /> @loader_path/../../libicuuc.dylib $(BINLOC)/isql<br /> install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicudata.dylib \<br /> @loader_path/../../libicudata.dylib $(BINLOC)/isql<br /> install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicui18n.dylib \<br /> @loader_path/../../libicui18n.dylib $(BINLOC)/isql<br /> <br /> install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Firebird \<br /> @loader_path/../../libfbclient.dylib $(BINLOC)/gbak<br /> install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicuuc.dylib \<br /> @loader_path/../../libicuuc.dylib $(BINLOC)/gbak<br /> install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicudata.dylib \<br /> @loader_path/../../libicudata.dylib $(BINLOC)/gbak<br /> install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicui18n.dylib \<br /> @loader_path/../../libicui18n.dylib $(BINLOC)/gbak<br /> <br /> install_name_tool -change $(OLDPATH)/libicuuc.dylib @loader_path/libicuuc.dylib \<br /> $(LIBLOC)/libfbclient.dylib<br /> install_name_tool -change $(OLDPATH)/libicudata.dylib @loader_path/libicudata.dylib \<br /> $(LIBLOC)/libfbclient.dylib<br /> install_name_tool -change $(OLDPATH)/libicui18n.dylib @loader_path/libicui18n.dylib \<br /> $(LIBLOC)/libfbclient.dylib<br /> install_name_tool -change $(OLDPATH)/libicudata.dylib @loader_path/libicudata.dylib \<br /> $(LIBLOC)/libicuuc.dylib<br /> install_name_tool -change $(OLDPATH)/libicuuc.dylib @loader_path/libicuuc.dylib \<br /> $(LIBLOC)/libicui18n.dylib<br /> install_name_tool -change $(OLDPATH)/libicudata.dylib @loader_path/libicudata.dylib \<br /> $(LIBLOC)/libicui18n.dylib<br /><br /> install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Firebird \<br /> @loader_path/../../libfbclient.dylib $(INTLOC)/libfbintl.dylib<br /><br /> install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicuuc.dylib \<br /> @loader_path/../libicuuc.dylib $(INTLOC)/libfbintl.dylib<br /> install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicudata.dylib \<br /> @loader_path/../libicudata.dylib $(INTLOC)/libfbintl.dylib<br /> install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicui18n.dylib \<br /> @loader_path/../libicui18n.dylib $(INTLOC)/libfbintl.dylib<br /><br /> install_name_tool -id @rpath/libfbclient.dylib $(LIBLOC)/libfbclient.dylib<br /> install_name_tool -id @rpath/libicudata.dylib $(LIBLOC)/libicudata.dylib <br /> install_name_tool -id @rpath/libicui18n.dylib $(LIBLOC)/libicui18n.dylib<br /> install_name_tool -id @rpath/libicudata.dylib $(LIBLOC)/libicudata.dylib<br /> install_name_tool -id @rpath/libicuuc.dylib $(LIBLOC)/libicuuc.dylib<br /> install_name_tool -id @rpath/libicudata.dylib $(LIBLOC)/libicudata.dylib<br /> install_name_tool -id @rpath/libib_util.dylib $(UTILOC)/libib_util.dylib<br /> install_name_tool -id @rpath/libfbintl.dylib $(INTLOC)/libfbintl.dylib<br /> install_name_tool -id @rpath/libEngine12.dylib $(PLULOC)/libEngine12.dylib<br />Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com0tag:blogger.com,1999:blog-8986782557879316238.post-20191575626840528672017-12-20T08:57:00.002-08:002017-12-20T08:57:39.797-08:00Firebird and Windows 10 Fall Creators Update<br />
<br />
Windows 10 is well known for encouraging a love/hate relationship
with its users. From the beginning Microsoft implemented a policy of
uninstalling users' software without their consent during installation
or update of Windows 10. Many articles on the net discuss this, <a class="reference external" href="https://www.howtogeek.com/243581/windows-10-may-delete-your-programs-without-asking/">for example</a> however, until now, Firebird seemed to be unaffected.<br />
<br />
Not any more. Windows 10 Creators Edition (Build 1709), released in
September 2017 seems to have added Firebird to its hit list of dangerous
applications. This problem was first reported <a class="reference external" href="https://www.tenforums.com/software-apps/95782-windows-10-creators-uninstall-program-automatically-block-install.html">here</a>.<br />
Because the report mentioned Firebird 1.5 it seemed unfortunate but,
hey, Firebird 1.5 has been out of support for eight years. It was first
released back in 2003 in the heyday of Windows XP and fourteen years is a
long time in the software industry. But not only had Firebird been
uninstalled, it was impossible to re-install it. This was likely to
affect one of our clients so we spent a long time trying to work around
that problem.<br />
<br />
Since then we have heard reports from other clients whose customers
use Firebird 2.5. After updating Windows 10 to build 1709 Firebird was
nowhere to be seen. We have also heard that Windows 10 Enterprise is
affected so this is not isolated to the so-called Creators edition. And,
at the time of writing we haven't had any feedback about the status of
Firebird 3.0.<br />
<br />
Luckily, Windows 10 does not block installation of any version of
Firebird from V2.0 onwards. But having to re-install is still very
annoying, and who knows when Microsoft might decide to change the rules
on that?<br />
We spent a long time with our client trying to work out how to
re-install Firebird 1.5. Manually setting the configuration using the
Properties | Compatibility page and choosing an older version of Windows
didn't work. However, running the compatibility troubleshooter did,
even though the same setting were being chosen.<br />
<br />
Unfortunately, there is so much clicking required that no ordinary
user could be relied upon to achieve the desired goal. And trying to
talk someone through that over the phone or by email would be extremely
frustrating for all involved.<br />
<br />
Rebuilding the installer with a newer, Windows 10 aware version of
InnoSetup didn't work. But weirdly, a rapidly hacked version using an
InterBase template for InstallShield did. Hmmm. And hacking an InnoSetup
based installer together with the Firebird 1.5 zip package also worked.
So what was going on there?<br />
<br />
What criteria does Windows 10 use to decide which programs to ban
from installation? There is a known bug in the cpl applet that ships
with Firebird 1.5 that can crash Windows Explorer so we tried building
an installer without that applet. But no, that wasn't it either.<br />
<br />
Finally our client had a brainwave. What happens if
Firebird-1.5.6.5026-0-Win32.exe was renamed to, say, setup.exe? Well,
what do you know? It works. So simple that we really wanted to kick
ourselves for not thinking of it first. And as for the poor Brazilian
guy in the forum above, I hope he is sitting down when/if he finds out
the solution. It is enough to make anyone want to run screaming into the
woods.Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com2tag:blogger.com,1999:blog-8986782557879316238.post-29533521093609798222016-03-10T00:45:00.000-08:002016-04-21T06:29:26.799-07:00DYLD_LIBRARY_PATH and El Capitan<br />
With the release of Firebird 3 RC2 it is time to concentrate properly on the port of Firebird on MacOSX, Firebird 3 was buildable but we don't have a proper installer, ICU is no longer directly included and there<br />
are a number of other bits and pieces that need to be addressed.<br />
<br />
The last time I built Firebird 3 was before I upgraded to El Capitan. I did the upgrade to address some issues with the Firebird installer.<br />
<br />
Lets just say when I started ny first build I wasn't expecting too many problems.... <br />
<br />
When we build Firebird on OSX we arbitrarily point to the ICU location (build_environment/firebird/lib), after copying over the files from the initial ICU build, the Firebird build then finds the ICU libraries dynamically (when needed) because DYLD_LIBRARY_PATH is sey to point to the location via prefix.darwin (make.platform).<br />
<br />
While the ICU libraries are being built we set the final actual framework install names and locations of the ICU libraries (using -install_name) and they then placed in the appropriate directory of the framework <br />
when the installer is packaged and built.<br />
<br />
Every time I tried to build Firebird it would fall over when trying to build msg.fdb with ISQL.<br />
The error was:<br />
"Could not find acceptable ICU library"<br />
which seemed very strange as I know it was placed exactly where it should be and where DYLD_LIBRARY_PATH should be picking it up.<br />
<br />
Setting DYLD_LIBRARY_PATH in my default terminal worked successfully, checking DYLD_PRINT_ENV seeemed to show the path being set etc., after much head scratching I eventually put the following line of code into unicode_util.cpp at the function: <br />
UnicodeUtil::ConversionICU& UnicodeUtil::getConversionICU()<br />
<br />
printf("DYLD_LIBRARY_PATH=%s\n", getenv("DYLD_LIBRARY_PATH"));<br />
<br />
When I next ran the build, this is what was returned..<br />
<br />
DYLD_LIBRARY_PATH=(null)<br />
<br />
Something very strange was definitely going on. Some further research eventually turned up this thread on the<br />
PostgreSQL forums<br />
<br />
http://www.postgresql.org/message-id/561E73AB.8060800@gmx.net<br />
<br />
"Apparently the behaviour is that DYLD_LIBRARY_PATH is prevented from being inherited into any system binaries. E.g. a shell. But specifying it inside a shell script will allow it to be inherited to children of<br />
that shell, unless the child is a protected process in turn."<br />
<br />
My reaction was the same as Tom Lane's but expressed in much stronger terms.<br />
<br />
The simplest solution to the problem is to<a href="https://dberzano.github.io/alice/install-aliroot/prereq-osx/#system_integrity_protection"> turn off System Integrity Protection</a><br />
<br />
via Recovery Mode (Command-R), and then set csrutil disable in a terminal and reboot.<br />
<br />
Note that this is a development only issue and will not affect anybody using the built<br />
binaries when the port is complete. Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com2tag:blogger.com,1999:blog-8986782557879316238.post-62008913703018185852015-10-07T04:11:00.002-07:002015-10-08T00:11:05.289-07:00How to install/upgrade Firebird manually on El Capitan <br />
Background: The current Firebird automatic installer on El Capitan is no longer working, its format has been<br />
finally deprecated. The installer is currently undergoing a complete re-write to comply with the Flat Package Format. Until this work is completed, you can manually install Firebird using the following method.<br />
<br />
<b>Classic</b><br />
<br />
Download FirebirdCS-2.5.4-26856-x86_64.pkg.zip<br />
Unzip it<br />
cd FirebirdCS-2.5.4-26856-x86_64.pkg<br />
cd Contents<br />
gunzip Archive.pax <br />
pax -r -f Archive.pax (or unpack the file using Finder)<br />
as sudo<br />
cp -r Firebird.framework /Library/Frameworks/Firebird.framework<br />
cd Resources<br />
./postinstall<br />
<br />
If you are upgrading from a previous version use:<br />
./preupgrade<br />
and then<br />
./postupgrade<br />
<br />
<b>SuperServer</b><br />
<br />
Download FirebirdSS-2.5.4-26856-x86_64.pkg.zip<br />
Then follow the same steps as above, and ignore any errror messages that reference<br />
a, /Resources/doc/doc: No such file or directory<br />
b, chmod /Library/StartupItems/Firebird/Firebird: No such file or directory<br />
c, /Library/LaunchDemons/org.firebird.gds.plist: service is already loaded.<br />
<br />
a ps -eaf should show both fbserver and fbguard running.Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com3tag:blogger.com,1999:blog-8986782557879316238.post-33880745168130659622015-07-27T01:01:00.002-07:002015-07-27T01:01:17.548-07:00gbak -statFrom the Bug Tracker:<br />
<br />
Vlad Khorsun commented on CORE-1999:<br />------------------------------------<br /><br />gbak now has a new command-line switch<br /><br /> -ST(ATISTICS) TDRW show statistics:<br /> T time from start<br /> D delta time<br /> R page reads<br /> W page writes<br /><br />Sample output:<br /><br />firebird>gbak -v -stat tdrw -r o a.fbk a.fdb<br />gbak:opened file a.fbk<br />gbak: time delta reads writes<br />gbak: 0.173 0.173 0 0 transportable backup -- data in XDR format<br />gbak: 0.175 0.002 0 0 backup file is compressed<br />gbak: 0.177 0.001 0 0 backup version is 10<br />gbak: 0.270 0.092 0 650 created database s:\Temp\A+.FDB, page_size 8192 bytes<br />gbak: 0.273 0.002 0 2 started transaction<br />gbak: 0.274 0.001 0 0 restoring domain RDB$29<br />gbak: 0.275 0.001 0 0 restoring domain RDB$12<br />...<br />gbak: 18.661 0.002 0 0 restoring data for table TEST1<br />gbak: 18.698 0.036 0 0 10000 records restored<br />gbak: 18.735 0.036 0 0 20000 records restored<br />...<br />gbak: 25.177 0.036 0 0 1770000 records restored<br />gbak: 25.220 0.042 0 1633 1780000 records restored<br />gbak: 25.256 0.036 0 0 1790000 records restored<br />...<br />gbak: 38.702 0.002 0 0 restoring privilege for user SYSDBA<br />gbak: 38.707 0.004 22 0 creating indexes<br />gbak: 45.015 6.308 82 38394 activating and creating deferred index T2_VAL<br />gbak: 45.132 0.116 3 9 activating and creating deferred index TEST_S_UNQ<br />gbak: 46.486 1.354 0 10775 activating and creating deferred index RDB$PRIMARY1<br />gbak: 46.566 0.079 0 9 activating and creating deferred index T1_IDX<br />gbak: 46.661 0.095 5 15 committing metadata<br />gbak: 46.665 0.003 8 10 fixing views dbkey length<br />gbak: 46.666 0.001 1 18 updating ownership of packages, procedures and tables<br />gbak: 46.671 0.005 0 0 adding missing privileges<br />gbak: 46.673 0.001 0 0 fixing system generators<br />gbak: 46.682 0.008 4 13 finishing, closing, and going home<br />gbak: 46.684 0.002 171 82442 total statistics<br />gbak:adjusting the ONLINE and FORCED WRITES flags<br />Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com0tag:blogger.com,1999:blog-8986782557879316238.post-69297314392362348342015-06-01T05:34:00.005-07:002015-06-01T05:37:21.700-07:00The semantics of isc_tpb_autocommitFrom Vlad Horsun:<br />
<br />
A simplified overview of the autocommit code:<br />
<br />
When a transaction, marked as TRA_autocommit performs any of following actions, it is marked also as TRA_perform_autocommit<br />
<br />
<ul>
<li>insert</li>
<li>update</li>
<li>delete</li>
<li>select with lock</li>
<li>post event</li>
</ul>
The TRA_perform_autocommit flag is checked when<br />
<ul>
<li>the engine receives input message</li>
<li>the engine sends an output message</li>
<li>the engine starts to execute a request</li>
<li>the engine finishes executing a DDL request</li>
</ul>
When the TRA_perform_autocommit flag is detected, the engine runs on-commit triggers (not for DDL, that looks like a bug) and performs commit retaining. A new transaction will have TRA_autocommit flag set and TRA_perform_autocommit not set.Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com0tag:blogger.com,1999:blog-8986782557879316238.post-86176814800674978322015-02-18T07:31:00.001-08:002015-07-02T01:44:47.156-07:00Yosemite, SuperServer, and StartupItems<br />
There seems to be a small problem with Firebird on Yosemite (OSX 10.10), it looks like Apple have finally deprecated StartupItems, and currently SuperServer uses this to start itself on reboot.<br />
<br />
Apple state:<br />
"Startup Items<br />
Deprecation Note: Startup items are a deprecated
technology. Launching of daemons through this process may be removed or
eliminated in a future release of OS X."<br />
<br />
Which is obviously what has happened.<br />
<br />
This is followed by <br />
<br />
"Unless your software requires compatibility with OS X v10.3 or earlier, use the launchd facility instead."<br />
<br />
Which is currently what Classic does.<br />
<br />
<br />
If you install Firebird after upgrading to Yosemite, you can start the server manually by simply using<br />
<br />
<i>/Library/StartupItems/Firebird/Firebird start</i><br />
<br />
However if Firebird was already installed, it looks as if the StartupItems folder is removed during the upgrade, but you should be able to start the server using the following....<br />
<br />
<i>/Library/Frameworks/Firebird.framework/Resources/bin/fbguard -daemon -forever</i><br />
<br />
But if you want Firebird SuperServer to start automatically on reboot you could use the following org.firebird.gds.plist file in /Library/LaunchDaemons<br />
<br />
<textarea cols="80" rows="15"><?xml version="1.0" encoding="UTF-8"?>
<plist version="1.0">
<dict>
<key>Label</key>
<string>org.firebird.gds</string>
<key>ProgramArguments</key>
<array>
<string>/Library/Frameworks/Firebird.framework/Resources/bin/fbguard</string>
<string>-daemon</string>
<string>-forever</string>
</array>
<key>RunAtLoad</key>
<true>
</true></dict>
</plist>
</textarea>
<br />
<br />
and the command<br />
<br />
<i>launchctl load org.firebird.gds.plist</i><br />
<br />
should now start Firebird immediately and also on reboot.<br />
<br />
<u><b>Update 2nd July 2015 </b></u><br />
<br />
I got an email from David Nock suggesting that the following plist would do a better job at managing SuperServer on Yosemite (10.10). Some brief tests suggest that he is right. I will be committing the following for 2.5.5, but I am documenting this here in case anybody wants to use this.<br />
<br />
<textarea cols="80" rows="15"><?xml version="1.0" encoding="UTF-8"?>
<plist version="1.0">
<dict>
<key>Label</key>
<string>org.firebird.gds</string>
<key>ProgramArguments</key>
<array>
<string>/Library/Frameworks/Firebird.framework/Resources/bin/fbguard</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>KeepAlive</key>
<true/>
</dict>
</plist>
</textarea>
<br />
<br />Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com5tag:blogger.com,1999:blog-8986782557879316238.post-80966272057639517732014-11-12T02:44:00.003-08:002014-11-12T02:44:24.487-08:00Services API extension - Nbackup SupportFrom docs/README.services_extension.<br />
<br />Services API extension - Nbackup support.<br />(Alex Peshkov, peshkoff@mail.ru, 2008)<br /><br />Nbackup performs two logical groups of operations - locking and unlocking a database and backup and restoring it. It doesn't make much sense duplicating locking and unlocking in using services, because that functionality is present remotely in via the SQL language interface (ALTER DATABASE). But backup and restore must be run on localhost and the only way to access them is via nbackup utility.<br /><br />
Therefore expanding the services API to support this functionalty is useful.<br /><br />The following actions were added:<br />
<br />isc_action_svc_nbak - incremental nbackup,<br />isc_action_svc_nrest - incremental database restore.<br /><br />
The following parameters were added:<br />isc_spb_nbk_level - backup level (integer),<br />isc_spb_nbk_file - backup file name (string),<br />isc_spb_nbk_no_triggers - do not run DB triggers (option).<br /><br />Samples of use of new parameters in fbsvcmgr utility (supposing login and<br />password have been set using some other method):<br /><br />
Create backup level 0:<br /> fbsvcmgr service_mgr action_nbak dbname employee nbk_file e.nb0 nbk_level 0<br /> Create backup level 1:<br /> fbsvcmgr service_mgr action_nbak dbname employee nbk_file e.nb1 nbk_level 1<br /> Restore database from this files:<br /> fbsvcmgr service_mgr action_nrest dbname e.fdb nbk_file e.nb0 nbk_file e.nb1Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com0tag:blogger.com,1999:blog-8986782557879316238.post-82643500580494518122013-09-26T01:26:00.001-07:002013-12-18T06:09:12.069-08:00An update about Firebird on iOS<br />
Based on a comment I made in response to someone asking about the availability of Firebird on IOS (after someone mentioned that the recent release of Delphi XE5 has support for an embedded version of InterBase for iOS and Android), I thought I would write a few more details of the hows and wherefores here.<br />
<br />
1. I can confirm that yes, I have managed to cross-compile Firebird for iOS on my Mac. But note, cross-compiling in Firebird 2.5 is difficult and not for the faint hearted (especially ICU). The ability to cross-compile is only really available in the Firebird 3.0 tree.<br />
<br />
2. The current build I have created is a normal Mac framework, and it needs to be finessed into an embedded bundle using this script.<br />
<br />
<a href="http://paulbeachsblog.blogspot.fr/2012/09/firebird-embedded-on-macosx.html">Firebird embedded on MacOSX</a><br />
<br />
The script is available in svn in the B2_5_Release tree, located in the builds/install/arch-specific/darwin directory. This will take an existing new build of Firebird 2.5 and convert it into an appropriate bundle format. However the current code still has a few issues. The current (2.5.2) code expects Firebird to be created as a framework, so the config_root module for darwin doesn't have the necessary "smarts" to be able to work out where things are automatically depending on whether the install is a framework, or if its embedded. However new code committed recently to B2_5_Release does. The build I currently have doesn't use this new code and as such requires that the environment variable FIREBIRD must me set. <br />
<br />
3. I had planned to rebuild the code using the new embedded support for locating various important files that embedded needs, when my Mac died. The Mac is now currently being repaired and having a new motherboard fitted. Cost 600.00 Euros. It seems this is common fault on Macs of this age. So any further work will have to wait.<br />
<br />
4. To test the build currently, it seems you will need an iOS device (iPhone or iPad) running iOS 6.1+ as the code was compiled with the following switch -miphonos-version-min=6.1. You also need to be need to be registered as a developer at Apple, and you will need to get the relevant certificate ids from them for the devices that you will test or use. Firebird like InterBase will never be availble via the App Store as it relies on dynamically linked libraries. Only statically linked applications are allowed in the App Store (security issues). <br />
<br />
5. I now have an iPad I can use to start testing (thanks to a very generous donation), but haven't yet signed up as an official Apple developer and logged my devices. <br />
<br />
If anyone wants to help/assist in anyway, please feel free to contact me.<br />
<br />
<b>18th December 2013</b><br />
<br />
<br />
An updated build of embedded Firebird for IOS can now be downloaded from <br />
<br />
<a href="http://www.ibphoenix.com/downloads/FirebirdIOSembed.tar">http://www.ibphoenix.com/downloads/FirebirdIOSembed.tar</a><br />
<br />
There were some issues with loading dylibs, these have been corrected using the install_name_tool -change<br />
command. The default embedded script I used to build the embedded package forgets that I had to cross compile ICU 5.1 rather than build ICU with Firebird.<br />
<br />
isql/gbak (or any other firebired utility) can now find the relevant libs they need by searching the directory above them i.e. MacOS. There was however an issue if you used<br />
<br />
<tt>symbolic links, symbolic links, extra ``/'' characters, and references to <u>/./</u> and
<u>/../</u> in the <u>file_name</u>.</tt><br />
<br />
This has now been corrected using the realpath function.<br />
I haven't tested this myself yet on IOS, so once again feedback would be appreciated.<br />
<br />
<br />
<br />
<br />
<br />Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com4tag:blogger.com,1999:blog-8986782557879316238.post-14682152246499672342013-08-01T05:09:00.000-07:002013-08-01T09:45:30.282-07:00Some Analytics<br />
Known new Firebird installations (direct via the /afterinstall page) between 4th June 2011 and 31th July 2013 (since the transition to new Firebird website)<br />
<br />
Total: 1,584,731<br />
<br />
The following numbers don't just count only the above direct visits (as there are about 1%-2% of installs coming from other sources)<br />
<br />
By Country:<br />
<br />
1. Brazil 555,483<br />
2. Russia 160,982<br />
3. Turkey 104,448<br />
4. Poland 75,960<br />
5. Germany 65,597<br />
6. Ukraine 57,509<br />
7. South Korea 46,202<br />
8. China 44,076<br />
9. Italy 38,526<br />
10.USA 33,064<br />
11.Mexico 26,688<br />
12.Spain 25,654<br />
13.Czech Rep. 25,035<br />
14.Indonesia 22,732<br />
15.Colombia 21,964<br />
16.France 21,758<br />
17.Bulgaria 19,602<br />
18.(not set) 16,827<br />
19.Belarus 16,414<br />
20.South Africa 14,466<br />
21.Japan 14,289<br />
22.UK 13,314<br />
23.Hungary 11,189<br />
24.India 10,451<br />
25.Malaysia 10,421<br />
<br />
Others are below 10,000<br />
<br />
By Sub Continent Region:<br />
<br />
1. South America 604,253<br />
2. Eastern Europe 381,068<br />
3. Eastern Asia 114,604<br />
4. Western Asia 113,098<br />
5. Western Europe 109,585<br />
6. Southern Europe 84,432<br />
7. South-Eastern Asia 55,412<br />
8. Northern America 37,434<br />
9. Central America 31,323<br />
10.Northern Europe 24,592<br />
<br />
By Continent:<br />
<br />
1. Americas 676,227<br />
2. Europe 599,677<br />
3. Asia 308,592<br />
4. Africa 37,020<br />
5. (not set) 16,827<br />
6. Oceania 10,008<br />
<br />
We know that Brazil is the world leader in Firebird installations (one installation of Firebird per 362 citizens), but it still quite amazing that one single South American country has more Firebird installations than Europe in total. <br />
<br />
Third place for Turkey is very interesting. I wonder what they are using Firebird for? It must be something quite large as 100k installations is not a small amount (one Firebird installation per 725 citizens, where for example the Russian Federation has one installation per 881).<br />
<br />
Fourth for Poland is not so surprising, but perhaps it does indicate that it would be a good place for some kind of regional Firebird event.<br />
<br />
South Korea and China taking 7th and 8th place is also interesting. Although total numbers are "small" for China, almost one hundred thousand installations combined is an interesting foothold. (It's about 1/3 of the approximately 300,000 installations in Asia). We previously assumed that the strong regional leader was Japan (installations per capita), but it's actually South Korea with one Firebird installation per 1086 citizens)<br />
<br />
So we took these 1,584,731 servers and calculated an equivalent cost of buying InterBase for a single user at (200.00 Euros + 38.00 VAT), then thats a saving to Firebird users of 377 million Euros, or 2,262 million Euros if these installations were all translated into 10 user servers (1,200.00 Euros + 228.00 Euros VAT), or<br />
4,714 million Euros if they were all 25 user servers (2,500 Euros + 475.00 VAT).<br />
<br />
Very little of those savings make their way back to the Firebird project (less than 0.1%), assuming that we took the lowest figure (377 Million Euros) indicated by single user installs. <br />
<br />
(cuurent prices taken from Embarcadero online shop)Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com3tag:blogger.com,1999:blog-8986782557879316238.post-46218624208766095512013-08-01T02:28:00.004-07:002013-08-01T06:22:33.100-07:00IBConsole<br />
I had to do some research on behalf of a customer, to see how viable IBConsole (released with InterBase 6.0) would be with Firebird 2.5.2 Classic. (remember it was developed for SuperServer via the Services API only). The results were a little surprising.<br />
<br />
1. Creating and adding users works<br />
2. Sweep works<br />
3. Backup works<br />
4. Restore works (Although it does generate an access violation in the console on completion)<br />
5. Transaction Recovery (limbo's) generates an unavailable database error.<br />
6. Shutdown - Database shutdown completed successfully, database has been shut down<br />
and is currently in single user mode. A gstat -h indicates that the database is in multi-user maintenace mode, so the database has been closed for maintenance and multiple connections are allowed only for SYSDBA or the database owner only. <br />
7. Restart completes sucessfully, and gstat -h indicates that the database is now in normal mode.<br />
8. View Metadata generates a RDB$CHARACTER_SET not found error.<br />
9. The SQL tool seems to work ok.<br />
<br />
So in general - its still usable with some restrictions<br />
<br />
Personally if you want a sraightforward tool, I would recommend FlameRobin. Its free and it supports interactive SQL, backup/restore etc and user management. Its also cross platform so you can run it locally on Linux and MacOSX if you want.Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com2tag:blogger.com,1999:blog-8986782557879316238.post-80196420530894305782013-05-31T01:54:00.002-07:002013-08-01T06:22:44.387-07:00How to backup a working Firebird database using a third party backup tool.<br />
One of the basic problems of trying to use a third party backup tool, or simple file copy of a Firebird database whilst a database is active and online, is that the tool or utility has no concept of transactions, so all you get is a copy of what is in the O/S buffer or disk of the database at the time you make the backup. However this "copy" of the database might be changing as new or active transactions are committed to the database while the copy is taking place. This is likely to produce at best, an inconsistent database, or at worst something that is corrupt and can't be used. Prior to Firebird 2.0 (other than using gbak) the only way you could do a backup or copy like this was to shut the database down, and make sure that no users were accessing the database before you invoked the third party backup tool or copy.<br />
<br />
However it is possible to use Nbackup to achieve the functional equivalent of a gbak and use a third party backup tool.<br />
<br />
The first thing you need to do is start a "freeze" on your database using the following syntax.<br />
<br />
nbackup -U username -P password -L database.fdb<br />
<br />
This will effectively lock the database, a flag is placed on the database header page, and it is set to "Locked" to let the engine know that all amended database pages that are written to the database are now being redirected to a delta file. <br />
<br />
Changes are flushed from the internal (Firebird) database cache to the O/S cache when a transaction is committed, if forced writes are on then these changes are flushed directly to disk, the final task on commit is to mark the transaction as committed in the Transaction Inventory Page. Once the database is locked, all commits are written to the delta file rather than the database, thus ensuring that the database is kept in a consistent state. <br />
<br />
Once the lock is applied, a simple gstat -h on the database wil show the "LOCKED" status as an optional database attribute. Once the -L command has done its work, a dela file will now be capable of receiving any committed changed database pages.<br />
<br />
You can now use your an alternative backup tool whilst database users continue to work. When your backup tool has finished doing what it needs to do to take a copy of the database, you can "unfreeze" the database using the nbackup -N (unlock) command.<br />
<br />
nbackup -U username -P password -N database.fdb <br />
<br />
The unfreeze causes nabckup to merge the changed pages from the delta file back into the main database, when completed the delta file is removed and the database header is changed back to its normal state.<br />
The backup you made of the database in its frozen state will still be in a "LOCKED" state, so if you need to restore it users will be unable to attach to it until you perform a "fixup". The fixup will reset the locked flag on the database header page back to normal, even though there isn't a delta file associated with the database.<br />
<br />
<b>Note</b>: If you are going to use this capability, please make sure that you are using the latest version of Firebird, as a number of bugs in nbackup have been fixed since its original release. Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com0tag:blogger.com,1999:blog-8986782557879316238.post-87100243053908069092013-05-23T05:52:00.001-07:002013-08-01T06:22:56.457-07:00CORE-4100<br />
An attempt to explain the rationale behind Vlads fix for CORE-4100.<br />
<br />
When a sweep has successfully finished its work it advances the Oldest Interesting Transaction (OIT) up to the value of the Oldest Snapshot Transaction (OST) that was recorded when the sweep started. The OIT transaction is the first transaction in a state other than committed in the database’s Transaction Inventory Pages (TIP), while the OST is the oldest transaction that was started in Snapshot mode.<br />
<br />
If while the sweep is running, there are more new transactions started than is the sweep interval (by default, when the OIT is 20,001 transactions less then the Oldest Snapshot Transaction), it is possible that the new OIT value could again be more than the sweep interval less the OST value, thus ensuring that a new sweep could start immediately.<br />
<br />
After a sweep has completed the first new transaction will pick up the updated OIT value from the saved OST on the database header page that was recorded when the sweep began, it will also read the actual OST from the header page, as well as the Oldest Active Transaction (OAT), the first transaction marked as active in the TIP pages. If the sweep condition is then met, a sweep begins. <br />
<br />
Ideally what should happen is that the transaction should pick up the recalculated OIT from (in transaction) rather than the OIT from the header page in order to determine whether a sweep should start or not.<br />
<br />
An example:<br />
<br />
1. Transaction 1000 was rolled back, therefore the next transaction when it calculates the OIT will use 1000 and is now considered a stuck or “interesting” value.<br />
<br />
2. By the time transaction 21001 occurs we have the following numbers:<br />
OIT 1000 <br />
OST 21000, <br />
Next 21001<br />
<br />
3. An automatic sweep is started, and it makes a note that the OST is 21000<br />
<br />
4. While the sweep is running 30000 new transactions get started and committed.<br />
<br />
5. When the sweep has finished doing its garbage collection and is about to advance the OIT, the numbers on the database header page are in effect<br />
OIT 1000 <br />
OST 51000 <br />
Next 51001<br />
<br />
6. The sweep then advances the OIT up to previously noted OST (21000)<br />
<br />
7. A new transaction is started and it then obtains the following numbers from the database header page:<br />
tra_oldest (OIT) 21000<br />
tra_oldest_active (OAT) 51002<br />
tra_number (OST) 51002<br />
<br />
However within the transaction it has also recalculated the new oldest interesting transaction number as 51001 which will be written to the database header page at the end of the transaction.<br />
<br />
8. However based on the OIT read from the database header page the following condition below is true<br />
<br />
tra_oldest_active (OAT) - tra_oldest (OIT) > sweep_interval<br />
51002 – 21000 > 20000 therefore a sweep will be started again.<br />
<br />
9. However when sweep starts the database header page will have been updated to contain an OIT of 51001, so instead of doing the above, we really should check the local OIT that is going to be written out to the header page rather than the header page itself, before deciding on whether to do a sweep or not.<br />
<br />
<br />
<br />Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com0tag:blogger.com,1999:blog-8986782557879316238.post-72773572924851095312012-09-20T07:22:00.000-07:002014-05-15T05:15:16.450-07:00Firebird Embedded on MacOSX<br />
Based on a document written by Fulvio, a while ago, I finally spent time writing a make file that will automatically create an embedded version of Firebird that will run on MacOSX as a bundle. All you do is make a normal build of Firebird Classic and then use this make file to create the Firebird.app folder which is set up in a way that allows you to access a database via isql for example without the need for a full framework. The make file detailed below was written for Firebird 2.5.x<br />
<br />
(updated 15th April 2014)<br />
<br />
file embed.darwin - committed to svn B2_5_Release/builds/install/arch-specific/darwin and also to trunk <br />
Code was added to config_root.cpp to simulate binreloc (posix) on MacOSX. Find the name of the excutable in the directory below firebird, strip the executable name, strip the lowest directory, now use the remainder to find the configuration file, firebird.conf, now read the conf file to get the actual RootDirectory for Firebird.<br />
<br />
embed.darwin updated below, to fix a couple of other issues.<br />
<br />
To make a Classic version of Firebird for MacOSX:<br />
./configure<br />
make<br />
cd gen<br />
make -B -f Makefile.install<br />
<br />
Then you can use the following make file to create an embedded version.<br />
copy embed.darwin from builds/install/arch-specific/darwin<br />
<br />
make -B -f embed.darwin <br />
<br />
# Makefile script to generate an embedded Firebird bundle from an existing Framework<br />
<br />
FBE=../gen/firebird/Firebird.app<br />
BINLOC=../gen/firebird/Firebird.app/Contents/MacOS/firebird/bin<br />
LIBLOC=../gen/firebird/Firebird.app/Contents/MacOS/firebird<br />
INTLOC=../gen/firebird/Firebird.app/Contents/MacOS/firebird/intl<br />
OLDPATH=/Library/Frameworks/Firebird.framework/Versions/A/Libraries<br />
<br />
all:<br />
-$(RM) -rf $(FBE) ../gen/firebird/Firebird.app<br />
mkdir -p $(FBE)/Contents<br />
mkdir -p $(FBE)/Contents/MacOS<br />
mkdir -p $(FBE)/Contents/MacOS/firebird<br />
mkdir -p $(FBE)/Contents/Resources<br />
mkdir -p $(FBE)/Contents/Frameworks<br />
mkdir -p $(FBE)/Contents/Plugins<br />
mkdir -p $(FBE)/Contents/SharedSupport<br />
mkdir -p $(FBE)/Contents/MacOS/firebird/bin<br />
mkdir -p $(FBE)/Contents/MacOS/firebird/intl<br />
<br />
cp ../gen/install/misc/firebird.conf $(FBE)/Contents/MacOS/firebird/firebird.conf<br />
cp ../gen/firebird/firebird.msg $(FBE)/Contents/MacOS/firebird/firebird.msg<br />
cp ../gen/firebird/lib/libfbembed.dylib $(FBE)/Contents/MacOS/firebird/libfbembed.dylib<br />
cp ../gen/firebird/lib/libicudata.dylib $(FBE)/Contents/MacOS/firebird/libicudata.dylib<br />
cp ../gen/firebird/lib/libicui18n.dylib $(FBE)/Contents/MacOS/firebird/libicui18n.dylib<br />
cp ../gen/firebird/lib/libicuuc.dylib $(FBE)/Contents/MacOS/firebird/libicuuc.dylib<br />
cp ../gen/firebird/lib/libib_util.dylib $(FBE)/Contents/MacOS/firebird/libib_util.dylib<br />
cp ../gen/firebird/security2.fdb $(FBE)/Contents/MacOS/firebird/security2.fdb<br />
cp ../gen/firebird/bin/gbak $(FBE)/Contents/MacOS/firebird/bin/gbak<br />
cp ../gen/firebird/bin/isql $(FBE)/Contents/MacOS/firebird/bin/isql<br />
cp ../builds/install/misc/fbintl.conf $(FBE)/Contents/MacOS/firebird/intl/fbintl.conf<br />
cp ../gen/firebird/intl/libfbintl.dylib $(FBE)/Contents/MacOS/firebird/intl/fbintl.dylib<br />
cp ../builds/install/arch-specific/darwin/embed.Info.plist $(FBE)/Contents/Info.plist<br />
<br />
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Firebird \<br />
../libfbembed.dylib $(BINLOC)/isql<br />
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicuuc.dylib \<br />
../libicuuc.dylib $(BINLOC)/isql<br />
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicudata.dylib \<br />
../libicudata.dylib $(BINLOC)/isql<br />
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicui18n.dylib \<br />
../libicui18n.dylib $(BINLOC)/isql<br />
<br />
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Firebird \<br />
../libfbembed.dylib $(BINLOC)/gbak<br />
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicuuc.dylib \<br />
../libicuuc.dylib $(BINLOC)/gbak<br />
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicudata.dylib \<br />
../libicudata.dylib $(BINLOC)/gbak<br />
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicui18n.dylib \<br />
../libicui18n.dylib $(BINLOC)/gbak<br />
<br />
install_name_tool -change $(OLDPATH)/libicuuc.dylib @loader_path/libicuuc.dylib \<br />
$(LIBLOC)/libfbembed.dylib<br />
install_name_tool -change $(OLDPATH)/libicudata.dylib @loader_path/libicudata.dylib \<br />
$(LIBLOC)/libfbembed.dylib<br />
install_name_tool -change $(OLDPATH)/libicui18n.dylib @loader_path/libicui18n.dylib \<br />
$(LIBLOC)/libfbembed.dylib<br />
install_name_tool -change $(OLDPATH)/libicudata.dylib @loader_path/libicudata.dylib \<br />
$(LIBLOC)/libicuuc.dylib<br />
install_name_tool -change $(OLDPATH)/libicuuc.dylib @loader_path/libicuuc.dylib \<br />
$(LIBLOC)/libicui18n.dylib<br />
install_name_tool -change $(OLDPATH)/libicudata.dylib @loader_path/libicudata.dylib \<br />
$(LIBLOC)/libicui18n.dylib<br />
<br />
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicuuc.dylib \<br />
@loader_path/../libicuuc.dylib $(INTLOC)/fbintl.dylib<br />
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicudata.dylib \<br />
@loader_path/../libicudata.dylib $(INTLOC)/fbintl.dylib<br />
install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicui18n.dylib \<br />
@loader_path/../libicui18n.dylib $(INTLOC)/fbintl.dylib<br />
<br />
install_name_tool -id @rpath/libfbembed.dylib $(LIBLOC)/libfbembed.dylib<br />
install_name_tool -id @rpath/libicudata.dylib $(LIBLOC)/libicudata.dylib <br />
install_name_tool -id @rpath/libicui18n.dylib $(LIBLOC)/libicui18n.dylib<br />
install_name_tool -id @rpath/libicudata.dylib $(LIBLOC)/libicudata.dylib<br />
install_name_tool -id @rpath/libicuuc.dylib $(LIBLOC)/libicuuc.dylib<br />
install_name_tool -id @rpath/libicudata.dylib $(LIBLOC)/libicudata.dylib<br />
install_name_tool -id @rpath/libib_util.dylib $(LIBLOC)/libib_util.dylib<br />
<br />
<br />
You can now tar a copy of the Firebird.app directory (and underlying), place anywhere on your system,<br />
and set the RootDirectory in the firebird.conf (17th July 2013 - this now works, previously it didn't), or set the FIREBIRD environment variable to the relevant location of the firebird directory in Firebird.app, If your application is under launchctl, you can set the FIREBIRD variable in the Info.plist file provided and your application should run if placed in the same directory. Unfortunately OS X Mavericks does not use the environment.plist, so you need to set the environment variable using launchctl a simple script like the following should do the trick.<br />
<br />
(setup.command)<br />DIR=$(cd $(dirname "$0"); pwd)<br />launchctl setenv FIREBIRD $DIR/Firebird.app/Contents/MacOS/firebird<br />echo setenv FIREBIRD $DIR/Firebird.app/Contents/MacOS/firebird | sudo tee<br />/etc/launchd.conf<br />
<br />
Should we produce a dedicated MacOSX embedded build along with the others (32bit/64bit/lipo)?<br />
<br />
The (Firebird.app) can be downloaded from <a href="http://www.ibphoenix.com/downloads/firebirdApp.zip">www.ibphoenix.com/downloads/firebirdApp.zip</a> should anybody want it.Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com24tag:blogger.com,1999:blog-8986782557879316238.post-20524682067934684062012-09-07T06:05:00.003-07:002013-08-01T06:23:22.062-07:00CORE-3740 and Firebird V2.5.2<br />
<a href="http://tracker.firebirdsql.org/browse/CORE-3740">CORE-3740</a> - SELECT using IN list with 153 or more elements causes crash.<br />
This problem only occurs on MacOS, and cannot be reproduced on Linux or Windows. Initial analysis showed that adjusting the optimisation level of the code as it links (O1 instead of O3) would increase the number of INS that could be supported but you would still get a crash eventually if the number of INS was large enough. <br />
The consensus of opinion was that this was a stack issue, so we increased the stack from the default 8mb in launchctl to 64mb, our test still crashed.<br />
<br />
We then tried embedded (local) firebird and increased the cache there using ulimit, strangely enough this worked. So it definitely was a stack issue but where? Some debugging code and some careful googling showed the problem. Even Classic launches a forked inet_server via a new thread.<br />
<br />
From an <a href="http://developer.apple.com/library/mac/#qa/qa1419/_index.html">Apple Technical Q&A</a><br />
"Each Mac OS X process is launched with a default stack size of 8 Megabytes. This allocation is used exclusively for the main thread's stack needs. Each subsequent thread created is allocated its own default stack, the size of which differs depending on the threading API used. For example, the Mac OS X implementation of Pthreads defines a default stack size of 512 Kilobytes, while Carbon MPTasks are created with a 4 Kilobyte stack."<br />
<br />
Patch applied to src/jrd/ThreadStart.cpp<br />
<br />
-#ifdef _AIX<br />
-// adjust stack size for AIX<br />
+#if defined(_AIX) || defined(DARWIN)<br />
+// adjust stack size<br />
<br />
// For AIX 32-bit compiled applications, the default stacksize is 96 KB,<br />
// see <pthread .h=".h">. For 64-bit compiled applications, the default stacksize<br /> // is 192 KB. This is too small - see HP-UX note above<br /><br />+// For MacOS default stack is 512 KB, which is also too small in 2012.<br />+<br /> size_t stack_size;<br /> state = pthread_attr_getstacksize(&pattr, &stack_size);<br /> if (state)<br /> Firebird::system_call_failed::raise("pthread_attr_getstacksize");<br /><br />- if (stack_size < 0x40000L)<br />+ if (stack_size < 0x400000L)<br /> {<br />- state = pthread_attr_setstacksize(&pattr, 0x40000L);<br />+ state = pthread_attr_setstacksize(&pattr, 0x400000L);<br /> if (state)<br /> Firebird::system_call_failed::raise("pthread_attr_setstacksize", state);</pthread>Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com0tag:blogger.com,1999:blog-8986782557879316238.post-70975496467047367232012-06-18T05:53:00.002-07:002015-12-01T00:53:43.466-08:00An ODBC driver for Firebird on MacOSX<br />
I was recently contacted by a Firebird user on MacOSX who was trying to get the Firebird ODBC driver to build on MacOSX. I thought I would try and step in to help.<br />
<br />
For those of you who follow the CVS checkins for OdbcJdbc you might have noticed the addition of a number of a new directory in the Builds directory called Gcc.darwin containing a makefile, a readme and a .sh file to create a lipo'ed dylib. The driver sucessfully builds and passes some simple tests. Feel free to contact me for a copy of the dylib to test further.<br />
<br />
The readme contains the following information:<br />
<br />
These instructions should allow a user to get a working user DSN for ODBC to connect <br />
to Firebird on MacOS. Comments please to Paul Beach (pbeach at ibphoenix.com.<br />
<br />
To build the library, edit makefile.darwin, and select the ARCH (i386 or X86_64).<br />
then make -B -f makefile.darwin all<br />
<br />
The Firebird ODBC library can be built in both 64bit or 32bit format. <br />
lipo.sh creates a fat libary that can be used on either version of Firebird.<br />
<br />
1. Download the MacOSX ODBC Administrator from support.apple.com<br />
http://support.apple.com/downloads/ODBC_Administrator_Tool_for_Mac_OS_X<br />
and install.<br />
<br />
2. Once installed it can be accessed via Applications/Utilities/ODBC Administrator<br />
<br />
3. Place the libOdbcFB.dylib in $HOME/odbc for example then add the Firebird driver <br />
(dylib) name and location to the Drivers tab in the Administrator.<br />
e.g <br />
Description: Firebird ODBC Driver<br />
Driver file: /Users/username/odbc/libOdbcFB.dylib<br />
Define as: User<br />
This will create an odbcinst.ini file in $HOME/Library/ODBC<br />
<br />
4. Now you need to create a User DSN via an odbc.ini file.<br />
Use the text below as an example, copy and paste into an odbc.ini<br />
file placed in $HOME/Library/ODBC<br />
Make sure that you modify the text so it points to your database <br />
and uses your username and password.<br />
<br />
<br />
[ODBC Data Sources]<br />
Test = Firebird<br />
<br />
[Test]<br />
Driver = /Users/username/odbc/libOdbcFb.dylib<br />
Description = Test Firebird ODBC<br />
Dbname = localhost:/Users/databases/test.fdb<br />
Client = <br />
User = SYSDBA<br />
Password = masterkey<br />
Role = <br />
CharacterSet = NONE<br />
ReadOnly = No<br />
NoWait = No<br />
Dialect = 3<br />
QuotedIdentifier = Yes<br />
SensitiveIdentifier = No<br />
AutoQuotedIdentifier = No<br />
<br />
[ODBC]<br />
Trace = 0<br />
TraceAutoStop = 0<br />
TraceFile = <br />
TraceLibrary = <br />
<br />
This User DSN should appear in the User DSN tab the next time you load the <br />
ODBC Administrator.<br />
<br />
5. You can test whether it works using iodbctest and then using the dsn<br />
dsn=Test, if all is well it should connect and you can issue SQL statements.<br />
<br />
To create a System wide version of the ODBC driver, copy the libOdbcFB.dylib<br />
to /usr/lib make sure the Administrators Drivers tab now points to this file.<br />
<br />
Copy the DSN above to /Library/ODBC and modify.<br />
<br />
Note: (13th June 2012)<br />
The ODBC library is linked to libfbclient.dylib found in the Firebird framework<br />
Libraries directory. Not all SuperServer builds of Firebird have this library installed<br />
by default. If this is the case get a copy of the Firebird Classic build and extract<br />
the libfbclient library and place it in <br />
/Library/Frameworks/Firebird.framework/Versions/A/Libraries<br />
<br />
Note: (30th Nov 2015)<br />
Since Mavericks (10.9) Apple no longer ship the default odbc header files as
part of their SDK, so you need to get the files from an older SDK e.g. /Developer/SDKs/MacOSX10.7.sdk/usr/include or download the files you need from <a href="http://www.iodbc.org/" rel="noreferrer" target="_blank">http://www.iodbc.org/</a><br /> Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com2tag:blogger.com,1999:blog-8986782557879316238.post-77033833957178301172012-02-22T05:28:00.002-08:002013-08-01T06:23:50.519-07:00Firebird V2.0.7<br />
We are currently preparing to release Firebird 2.0.7, since I take responsibility for the Mac builds, I did a 2.0.7 build on MacOSX 10.7 using the development tools installed by XCode 4.1 (gcc 4.2.1 etc). I set up the relevant environment variables for this older 32bit only build CFLAGS, CXXFLAGS, LD_FLAGS and also set the MACOSX_DEPLOYMENT_TARGET=10.4. The builds completed without any problems, some simple tests on MacOSX 10.7 showed no problems. <br />
<br />
Now - Imagine my surprise when Philippe told me that when he tried to QA the builds on MacOSX 10.5 we got this error on SuperServer startup.<br />
<br />
Process: fbserver [623]<br />
Path:<br />
/Library/Frameworks/Firebird.framework/Resources/English.lproj/var/bin/fbserver<br />
Identifier: fbserver<br />
Version: ??? (???)<br />
Code Type: X86 (Native)<br />
Parent Process: fbguard [310]<br />
<br />
Date/Time: 2012-02-09 09:42:13.939 +0100<br />
OS Version: Mac OS X 10.5.8 (9L31a)<br />
Report Version: 6<br />
Anonymous UUID: FA7F8C0C-581B-4153-ADBE-2BCB59C5F823<br />
<br />
Exception Type: EXC_BREAKPOINT (SIGTRAP)<br />
Exception Codes: 0x0000000000000002, 0x0000000000000000<br />
Crashed Thread: 0<br />
<br />
Dyld Error Message:<br />
Symbol not found: ___moddi3<br />
Referenced from:<br />
/Library/Frameworks/Firebird.framework/Resources/English.lproj/var/bin/fbserver<br />
Expected in: /usr/lib/libSystem.B.dylib<br />
<br />
A simple nm and grep shows the following for the 10.5 libSystem.B.dylib<br />
<br />
nm /usr/lib/libSystem.B.dylib | grep moddi3<br />
00083490 t ___moddi3<br />
0003df70 t ___umoddi3<br />
<br />
On 10.5 ___moddi3 is defined in libgcc_s.1.dylib which links to libsystem.B.dylib<br />
<br />
From what I can gather these *moddi3 symbols are routines for doing 64bit maths on 32bit systems. <br />
<br />
However the same on 10.7 returns nothing, however the symbols can be found in libgcc_s.10.5.dylib which does not link to libgcc_s.1.dylib hence the missing symbols message.<br />
<br />
Supposedly this can be fixed by linking explicity to to libgcc_s.10.5.dylib<br />
using -lgcc_s.10.5. I got round it by booting up my copy of Snow Leopard (10.6) and <br />
building Firebird 2.0.7 there instead of on MacOSX 10.7<br />
<br />
So basically if you want to buld an application that is deployable on 10.5 or less, you can't - unless you know how to get around the above problem. Its as if although you can build 32bit applications for 32bit versions of MacOSX, they won't run, because they are incompatible with any version of the OS less than 10.6.Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com4tag:blogger.com,1999:blog-8986782557879316238.post-4643777491679401792011-07-26T02:04:00.000-07:002013-08-01T06:24:02.464-07:00Firebird 2.5 SuperServer and MacOSX 10.7 (Lion)<br />
There is a small problem with Firebird 2.5 SuperServer and MacOSX 10.7 (Lion)...<br />
SuperServer will not start, and produces a crash report header similar to this<br />
<br />
<span style="font-weight: bold;">Update </span>26th August 2011 <br />
Its not just 2.5 SuperServer, reports have come in that SuperServer for Firebird 2.1.4 will not start, also the same problem exists for Firebird 2.5 SuperClassic.<br />
<br />
Process: fbserver [706]<br />
Path: /Library/Frameworks/firebird.framework/Versions/A/Resources/bin/fbserver<br />
Identifier: fbserver<br />
Version: ??? (???)<br />
Code Type: X86 (Native)<br />
Parent Process: fbguard [166]<br />
<br />
Date/Time: 2011-06-10 07:59:42.374 -0400<br />
OS Version: Mac OS X 10.7 (11A480b)<br />
Report Version: 9<br />
<br />
Anonymous UUID: B7EDF790-CE72-4B21-A982-B9EA4F4E2088<br />
<br />
Crashed Thread: 1 Dispatch queue: com.apple.libdispatch-manager<br />
<br />
Exception Type: EXC_BAD_INSTRUCTION (SIGILL)<br />
Exception Codes: 0x0000000000000001, 0x0000000000000000<br />
<br />
Application Specific Information:<br />
BUG IN CLIENT OF LIBDISPATCH: Do not close random Unix descriptors<br />
<br />
The bug is fixed in Firebird 2.5.1 but Firebird 2.5.1 is not available yet. So I have fixed the problem also in the Firebird 2.5 code and produced 32bit and 64bit builds that work properly. <br />
<br />
I have had new builds uploaded to Sourceforge to fix the problem with Firebird <br />
2.1.4 SuperServer, you can distinguish the new builds from the old by the build no. <br />
Old = 18393, New = 18393. Because we have some issues with 2.5.1 that are currently being investigated, I plan to update the 2.5 builds on Sourceforge soon.<br />
<br />
<span style="font-weight: bold;">Update </span>28th Sep 2011<br />
The Firebird 2.5 builds have also been replaced on Sourceforge, so you just need to download the latest build. However Firebird 2.5 will be replaced by Firebird 2.5.1 very shortly, builds are now taking place and will be QA'd and released shortly. <br />
<br />
Any problems - please let me know.Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com3tag:blogger.com,1999:blog-8986782557879316238.post-83748433476362129512011-07-04T06:51:00.000-07:002013-08-01T06:24:16.317-07:00Firebird V2.1 Error: value exceeds the range for valid dates<br />
You are restoring a backup of a Firebird 1.5 or 2.0 database to Firebird 2.1 and you see an error similar to this:<br />
<br />
gbak: writing data for table xyz<br />
gbak:20000 records written<br />
gbak: Error: value exceeds the range for valid dates<br />
gbak: Error: gds_$receive failed<br />
gbak:Exiting before completion due to errors<br />
<br />
How do you go about solving the problem?<br />
Well first of all a quick visit to the Firebird bug tracker reveals <br />
http://tracker.firebirdsql.org/browse/CORE-1714 and a couple of comments from Dimitry Yemanov.<br />
<br />
“The error means to say that some column has an invalid date value (outside the supported range). Prior to V2.1, it was possible to store such invalid values in the database, but now it's prohibited. A verbose output should point you to a problematic table.” Also “The current behavior is intended and is unlikely to be changed.”<br />
<br />
1. Firstly use the –v (verbose option of gbak) to find out the table that is causing the problem.<br />
2. Check the table xyz for date columns<br />
3. Perform the following SQL operation on all the date columns you found in 2. <br />
<br />
isql> update XYZ set MY_DATE_COLUMN = MY_DATE_COLUMN;<br />
<br />
At some point during the process you will see the same error that gbak produced, but now you will know which column is causing the problem.<br />
<br />
4. Install Firebird 1.5 or 2.0, by going back to the version of Firebird that allows for invalid dates, you will be at least able to correct the invalid date to something more appropriate.<br />
<br />
5. Now lets check for dates that are outside of their proper range (01 Jan 0001 - 31 Dec 9999)<br />
<br />
Below date zero:<br />
<br />
isql> select PRIMARY_KEY from XYZ where MY_DATE_COLUMN < '0001-01-01'<br />
<br />
if an error occurs correct the date to something more appropriate and meaningful<br />
<br />
Maximum date<br />
<br />
isql> select PRIMARY_KEY from XYZ where MY_DATE_COLUMN > '9999-12-31'<br />
<br />
if an error occurs correct the date to something more appropriate and meaningful<br />
<br />
6. You can now backup the database using Firebird 1.5 or 2.1 and successfully restore under 2.1<br />
<br />
For anyone interested, the following is a shell script provided by a friend that can be used on Linux to correct the above errors automagically across multiple databases.<br />
<br />
#!/bin/bash -u<br />
# correct OLD invalid dates in a DB<br />
#<br />
# Ray Holme, July 2013 - 207-583-6613<br />
# Rainbow Applications Inc., Waterford, ME 04088<br />
#<br />
ARGCNT=$#<br />
MYNAME=`/bin/basename $0`<br />
DATEL="'1/1/0001'"<br />
DATEH="'12/31/9999'"<br />
DRYRUN=0<br />
ERRCNT=0<br />
FLDS=/tmp/$$.flds<br />
ISQL=/opt/firebird/bin/isql<br />
ListDBS=""<br />
LOG=/tmp/$$.out<br />
NOFILE="could not find"<br />
NOTNULL="is not null"<br />
RANGE="not between $DATEL and $DATEH"<br />
RM=/bin/rm<br />
SQL=/tmp/$$.sql<br />
USAGE="usage: $MYNAME <-d> DBname1 <dbname2> ...>"<br />WORK=/tmp/$$.wrk<br /><br />ABORT() { $RM -f $FLDS $SQL $WORK; echo $MYNAME aborted; exit $1; }<br />INCR_ERRCNT() { ERRCNT=`echo $ERRCNT + 1 | /bin/bc`; }<br /><br />ANSWER_ME() {<br /> OK=2; until [ $OK -ne 2 ];<br /> do<br /> echo "Please type '$2' or '$3'."; echo -n "$1 [$2]: "; read ANS<br /> if [ "$ANS" = "$2" -o "$ANS" = "" ]; then OK=1<br /> elif [ "$ANS" = "$2" ]; then OK=0<br /> fi<br /> done<br />}<br /><br />#get list of dbs to do<br />until [ $# -eq 0 ];<br /> do case "$1" in<br /> -d) DRYRUN=1; ARGCNT=`echo $ARGCNT - 1 | /bin/bc` ;;<br /> *) if [ ! -f $1 ]; then echo $NOFILE $1; INCR_ERRCNT;<br /> else ListDBS="$ListDBS $1" ;<br /> fi ;;<br /> esac<br /> shift<br /> done<br />if [ "$ListDBS" = "" ]; then echo "$USAGE"; exit 1; fi<br />trap "ABORT 1" 1 2 3<br /><br />if [ ! -w /etc/passwd ]; then echo you must be root to execute $MYNAME; exit 1; fi<br />$RM -f $LOG # get what we need to do (worst case)<br />for DB in $ListDBS<br /> do<br /> $RM -f $FLDS<br /> if [ $ARGCNT -gt 1 ]; then<br /> ANSWER_ME "about to check $DB - c to continue, s to skip" c s<br /> if [ $OK -eq 0 ]; then continue; fi # skipped<br /> fi <br /> $ISQL $DB -pag 9999 > /dev/null 2> $WORK <<eof br="">output $FLDS;<br />select r.rdb\$relation_name, rf.rdb\$field_name<br /> from rdb\$relations r<br /> inner join rdb\$relation_fields rf on (r.rdb\$relation_name = rf.rdb\$relation_name)<br /> inner join rdb\$fields f on (f.rdb\$field_name = rf.rdb\$field_source)<br /> where r.rdb\$view_source is null and r.rdb\$system_flag = 0<br /> and f.rdb\$field_type = 35<br /> order by 1, 2;<br />EOF<br /> if [ -f $WORK ]; then XXX=`cat $WORK`; else XXX=""; fi<br /> if [ "$XXX" != "" ]; then echo problems working $DB; INCR_ERRCNT; continue; fi<br /> /bin/egrep -v "RELATION_NAME|==========" < $FLDS | grep -v '^$' \<br /> | /bin/awk '{printf("%-32.32s %s\n", $1, $2)}'> $WORK<br /> /bin/mv $WORK $FLDS<br /> if [ -f $FLDS ]; then XXX=`cat $FLDS`; else XXX=""; fi<br /> if [ "$XXX" = "" ]; then echo "no dates in $DB - curious (skip)"; continue; fi<br /># pass 1 - we show the records with errors embedded<br /> echo "Listing effected records in DB: $DB" >> $LOG<br /> echo "output $LOG;" > $SQL<br /> LastTable="" <br /> while read TABLE FIELD<br /> do<br /> if [ "$TABLE" != "$LastTable" ]; then<br /> if [ "$LastTable" != "" ]; then echo ";" >> $SQL; fi<br /> LastTable="$TABLE"<br /> echo "select distinct 'table: $TABLE' from rdb\$database;" >> $SQL<br /> echo "select * from $TABLE where " >> $SQL<br /> else echo -n " or " >> $SQL<br /> fi<br /> echo "( $FIELD $NOTNULL and ($FIELD $RANGE ))" >> $SQL<br /> done < $FLDS<br /> if [ "$LastTable" != "" ]; then echo ";" >> $SQL; fi<br /> $ISQL $DB -pag 9999 -i $SQL<br /># pass 2 - do clean up<br /> $RM $SQL<br /> while read TABLE FIELD<br /> do<br /> echo "update $TABLE set $FIELD = $DATEL where $FIELD $NOTNULL and $FIELD < $DATEL;" >> $SQL<br /> echo "update $TABLE set $FIELD = $DATEH where $FIELD $NOTNULL and $FIELD > $DATEH;" >> $SQL<br /> echo "commit;" >> $SQL<br /> done < $FLDS<br /> echo "exit;" >> $SQL<br /> if [ $DRYRUN -eq 0 ]; then $ISQL $DB -pag 9999 -i $SQL<br /> else echo "Would run this sql on $DB"; /bin/more $SQL<br /> fi<br /> done # end of outer (DB) loop<br /><br />$RM -f $FLDS $SQL $WORK<br />echo file $LOG contains a list of all effected records<br />exit $ERRCNT<br /> </eof></dbname2><!---d--><!---d--></-d>Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com1tag:blogger.com,1999:blog-8986782557879316238.post-40379323516148901382011-05-10T00:47:00.000-07:002013-08-01T06:24:30.809-07:00Firebird 3.0 - Shared Page Cache<br />
Posted by Vlad on the Firebird Development List.<br />
<br />
After more than a year of development, testing, switching on another tasks and returning back I'm ready to commit shared page cache implementation into trunk.<br />
<br />
I consider it stable enough to be committed into SVN. It runs some stability tests more than 10 hours for longest run and many shorter runs at different configurations.<br />
<br />
Here I want to do overview of changes, what was tested, what was not, and what is work in progress.<br />
<br />
Page cache was re-implemented looking at both Vulcan code (thanks to Jim) and our code base. It is not a blind copy of Vulcan code, every decision was thinking on and adjusted when needed with my understanding and expirience with existing code.<br />
<br />
The cache is syncronized at different levels. There is a few BufferControl (BCB) level sync's used to syncronize access to the common structures such as LRU queue, Precedence graph, Dirty page list and so one. These common sync's should be locked at as short as possible period of time as this is kind of global locks. Also there is a two syncs for every BufferDesc (BDB) - one for guard contents of page buffer tself and second one is to prevent partial writes of page wich edition is not completed and to prevent simultaneous write of the same page by different threads.<br />
<br />
I don't explain cache algoritms here, if someone have a question I'll be happy to answer it. And I not going to said that page cache implementation is frozen - if necessary we will change it of course (there is still space for impovements and bugs fixing).<br />
<br />
You know I already committed SyncObject class (and related classes) ported from Vulcan. I removed some unused code and restored fairness of locking : initially it was fair in Vulcan too (i.e. all lock requests was put in waiting queue and granted in first-in-first-out, or fair, order), but later Jim found some (unknown to me) ssue with this and made preference for SHARED locks. I found no performance issues with fair locking so I decided to revert that and restore original behavior. Of course it could be changed when necessary.<br />
<br />
Shared page cache is not an isolated change. It affects many parts of engine and our syncronization model changed significantly again. There was agreement that we will not implement shared metadata cache in v3 as this is risky and we could not deliver V3 in reasonable time frame. <br />
<br />
In shared cache mode we have single Database instance and many attachment instances linked to it (this is not new). <br />
<br />
All metadata objects moved into Attachment. Metadata syncronization is guarded by attachment's mutex now. Database::SyncGuard and company are replaced by corresponding Attachment::XXX classes. <br />
<br />
To make AST's work we need to release attachment mutex sometimes. This is very important change after v2.5 : in v2.5 attachment mutex is locked during whole duration of API call and no other API call (except of asyncronous fb_cancel_operation) could work with "busy" attacment. In V3 this rule is not worked anymore. So, now we could run more that one API call on the same attachment (of course not really simultaneously). I'm not sure it is safe but not disabled it so far. <br />
<br />
To make asyncronous detach safe I introduced att_use_count counter which is incremented each time when API call is entered and decremented on exit. detach now marks attachment as shutdown and waits for att_use_count == 0 before processing.<br />
<br />
Parallel access to the attachment could be easy disabled making every API call wait for att_use_count == 0 on enter or even introducing one more mutex to avoid spin wait. <br />
<br />
Also it seems this counter make obsolete att_in_use member as detach call should wait for att_use_count == 0 and drop call should return "object is in use" if att_use_count != 0.<br />
<br />
All AST's related to attachment-level objects should take attachment mutex before access attachment internals. This is implemented but not tested!<br />
<br />
Transaction inventory pages cache (TIP cache) was reworked and is shared by all attachments.<br />
<br />
To avoid contention on common dbb_pool its usage was replaced by att_pool when possible. To make this task slightly easy there was introduced jrd_rel::rel_pool which points currently to the attachment's pool. All relation's "sub-objects" (such as formats, fields, etc) is allocated using rel_pool (it was dbb_pool before). When we'll return metadata objects back to the Database it will be easy to redirect rel_pool to dbb_pool at one place in code instead of makeing tens of small changes again.<br />
<br />
About stability testing of different parts of the engine:<br />
- page cache - tested and worked<br />
- nbackup - tested and worked<br />
- monitoring - tested (less) and seems worked<br />
- server stop\database shutdown - somewhat tested, no crash observed, client reaction is not perfect (mostly network write errors)<br />
- shadow - not tested<br />
- metadata changes in concurrent environment - not tested<br />
- garbage collection thread - not tested, needs review and rework of some <br />
<br />
implementation details<br />
- cache writer thread - not tested, needs review and rework <br />
- external connections - not tested<br />
- external engines - not tested<br />
<br />
In configuration file there are two new settings :<br />
a) SharedCache - boolean value which rules the page cache mode<br />
b) SharedDatabase - boolean value which which rules the database file open mode.<br />
<br />
Currently they are common (as whole configuration) but soon it will be per-database settings. Below is few examples of how it could be used :<br />
<br />
- SharedCache = true, SharedDatabase = false (default mode)<br />
this is traditional SuperServer mode when all attachments share page cache and database file is opens in exclusive mode (only one server process could work with database)<br />
<br />
- SharedCache = false, SharedDatabase = true<br />
this is Classic mode when each attachment have its own page cache and many server processes could work with the same database.<br />
<br />
To run SuperClassic you should use switch -m in command line of firebird.exe (on Windows) or run fb_smp_server (on Posix, here i'm not sure and Alex will correct me)<br />
<br />
Else ClassicServer will run.<br />
<br />
- SharedCache = true, SharedDatabase = true<br />
this is completely new mode in which database file could be opened by many server processes and each process could handle many attachments which will share page cache (i.e. per-process page cache).<br />
<br />
It could be used to run few SS processes, for example, or to run "main" SS process and have ability to attach using embedded to make some special tasks.<br />
<br />
Must note that our lock manager is not ready to work in this mode, so we can't use it right now.<br />
<br />
Also there is unknown how performance will be affected when few SS with big cache will work with the same database.<br />
<br />
- SharedCache = false, SharedDatabase = false<br />
Looks like single process with single attachment will be allowed to work with database with such settings. Probably you can find an applications for it ;)<br />
<br />
One more change in configuration is that CpuAffinityMask setting changed its default value and it is 0 now. It allows new SS to use all available CPU's\cores by default.Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com1tag:blogger.com,1999:blog-8986782557879316238.post-88516966613331134842011-03-31T07:19:00.000-07:002013-08-01T06:24:45.795-07:00OAT and MON$<br />
Just as a "for your information", should anybody be interested. <br />
The following query on Firebird V2.x gets the attachment that holds the oldest active transaction via the monitoring tables. <br />
<br />
SELECT *<br />
FROM MON$ATTACHMENTS<br />
where MON$ATTACHMENT_ID =<br />
(select MON$ATTACHMENT_ID FROM MON$TRANSACTIONS<br />
where MON$TRANSACTION_ID =<br />
(SELECT min(MON$TRANSACTION_ID) FROM MON$TRANSACTIONS))Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com0tag:blogger.com,1999:blog-8986782557879316238.post-37949527772329470112011-03-31T02:12:00.000-07:002013-08-01T06:25:01.194-07:00HPUX Oracle and Itanium<br />
The <a href="http://www.oracle.com/us/corporate/press/346696">announcement</a> by Oracle last week that it will stop developing software for the Itanium processor will affect users of Oracle on HPUX Integrity servers. No surprise then that we see the following blog post - <a href="http://blogs.enterprisedb.com/2011/03/28/oracle-abandoning-hp-ux/">Oracle is abandoning HP-UX customers…Fight back!</a> from Ed Boyajian the CEO of EnterpriseDB suggesting that users should consider migrating to their version of PostgreSQL, even though as he admits PostgreSQL has not supported the HP-UX platform very well, but that is now going to change.<br />
<br />
For those who are curious, Firebird V2.5 builds just fine on Itanium HPUX (IA-64). See prefix.hpux_aCC and prefix.hpux_ia64 in the builds/posix directory. The only issue we have in providing such builds, is access to the relevant hardware/operating system to do a build, and bringing upto date the HPUX installer.Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com0tag:blogger.com,1999:blog-8986782557879316238.post-42785649875496484262011-02-09T02:57:00.000-08:002013-08-01T06:25:16.106-07:00Forced Writes?<br />
Forced Writes are turned ON by default on Windows since the release of Firebird V1.0.<br />
<br />
See the following item on the tracker: <a href="http://tracker.firebirdsql.org/browse/CORE-591">CORE-591</a><br />
<br />
But Forced Writes were OFF on Linux even if they were turned ON, until an issue with Linux (Posix) was resolved in Firebird V2.1 Beta 2 and this fix has now also ben backported to Firebird V2.0.4.<br />
<br />
See the following item on the tracker: <a href="http://tracker.firebirdsql.org/browse/CORE-1476">CORE-1476</a> <br />
<br />
There was some discussion amongst the Firebird developers about the change (either to OFF on Win or to ON on Linux so we would have consistent defaults), but so far there has not been a final resolution since Windows and Linux handle I/O differently, and its perfectly valid to have different default settings. Before the fix in Firebird V2.1 Linux never did forced writes even when asked to, and pretty much everybody got used to the performance with Forced Writes off, when we turned them back on correctly the performance decrement was definitely going to be a problem. <br />
<br />
Basically, Linux writes pages out of the page cache as often as possible, so the file on disk stays pretty much current, whilst Windows writes to disk only when it must, so the difference between the disk and cache pre Firebird V1.5 could be enormous - hours or days of changes if the database file was not closed correctly. So, although it may be preferable that all systems should run with forced writes on, for Firebird on Windows it was pretty critical. <br />
<br />
Within Firebird V1.5 we attempted to make sure that the issue of Windows cache, the Firebird cache and what was on disk could not be vastly different even if Forced Writes were turn off by introducing the following two parameters in the Firebird configuration file. However the new parameters do not guarentee complete consistency between the cache and disk, for that you do need Forced Writes turned on.<br />
<br />
The parameters: <br />
<br />
# ----------------------------<br />
#<br />
# How often the pages are flushed on disk<br />
# (for databases with ForcedWrites=Off only)<br />
#<br />
# Number of unflushed writes which will accumulate before they are<br />
# flushed, at the next transaction commit. For non-Win32 ports,<br />
# the default value is -1 (Disabled)<br />
#<br />
# Type: integer<br />
#<br />
#MaxUnflushedWrites = 100<br />
<br />
#<br />
# Number of seconds during which unflushed writes will accumulate<br />
# before they are flushed, at the next transaction commit. For non-Win32<br />
# ports, the default value is -1 (Disabled)<br />
#<br />
# Type: integer<br />
#<br />
#MaxUnflushedWriteTime = 5Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com2tag:blogger.com,1999:blog-8986782557879316238.post-76933176433639263032011-02-09T02:06:00.000-08:002013-08-01T06:26:13.575-07:00Colissimo - What a joke.<br />
If you want to send a package to anywhere in France, La Poste insist that you send it by Colissimo. Colissimo is more "expensive" than normal post, but it has insurance just in case its delivered late, goes missing or whatever you are sending gets broken.<br />
<br />
Recently I sent a couple of Firebird mugs via Colissimo to a customer/Firebird user in France. After receipt and opening of the carefully packed box, the customer found that one of the mugs was cracked. After a brief email exchange he agreed to send me the damaged mug and I would send him a new one. <br />
<br />
Once I had the broken mug, I asked the local post office for the basic repayment for the damaged mug from Colissimo. Their response was to hand me a form to fill in, and suggest I post the completed form direct to Colissimo. OK - job done - now I sit back and wait for them to send me the money for something they had managed to break.<br />
<br />
After a couple of weeks a letter arrives telling me that they are looking into it, two weeks later another letter arrives, this one apologises, and says we cannot pay you the insurance because the reciever of the damaged goods should have filled in the complaint form and not the sender. <br />
<br />
WTF? When you receive damaged goods from someone, what is the first thing you do? <br />
Well, I would have thought, you contact the sender, ask if they want it back and ask for a replacement.... Its then the senders problem to deal with the people he used to send it. <br />
<br />
Not in France it seems. By now we have paid for three lots of Colissimo (send original, send it back, send another) and wasted a whole lot of time. The result - huge profits for Colissimo, zero service for the customer. I now give up and throw the whole lot in the bin. Life is just too short to start aguing with a "fonctionnaire".Paul Beachhttp://www.blogger.com/profile/01887777727779537695noreply@blogger.com12