An update about Firebird on iOS

Based on a comment I made in response to someone asking about the availability of Firebird on IOS (after someone mentioned that the recent release of Delphi XE5 has support for an embedded version of InterBase for iOS and Android), I thought I would write a few more details of the hows and wherefores here.

1. I can confirm that yes, I have managed to cross-compile Firebird for iOS on my Mac. But note, cross-compiling in Firebird 2.5 is difficult and not for the faint hearted (especially ICU). The ability to cross-compile is only really available in the Firebird 3.0 tree.

2. The current build I have created is a normal Mac framework, and it needs to be finessed into an embedded bundle using this script.

Firebird embedded on MacOSX

The script is available in svn in the B2_5_Release tree, located in the builds/install/arch-specific/darwin directory. This will take an existing new build of Firebird 2.5 and convert it into an appropriate bundle format. However the current code still has a few issues. The current (2.5.2) code expects Firebird to be created as a framework, so the config_root module for darwin doesn't have the necessary "smarts" to be able to work out where things are automatically depending on whether the install is a framework, or if its embedded. However new code committed recently to B2_5_Release does. The build I currently have doesn't use this new code and as such requires that the environment variable FIREBIRD must me set.

3. I had planned to rebuild the code using the new embedded support for locating various important files that embedded needs, when my Mac died. The Mac is now currently being repaired and having a new motherboard fitted. Cost 600.00 Euros. It seems this is common fault on Macs of this age. So any further work will have to wait.

4. To test the build currently, it seems you will need an iOS device (iPhone or iPad) running iOS 6.1+ as the code was compiled with the following switch -miphonos-version-min=6.1. You also need to be need to be registered as a developer at Apple, and you will need to get the relevant certificate ids from them for the devices that you will test or use. Firebird like InterBase will never be availble via the App Store as it relies on dynamically linked libraries. Only statically linked applications are allowed in the App Store (security issues).

5. I now have an iPad I can use to start testing (thanks to a very generous donation), but haven't yet signed up as an official Apple developer and logged my devices.

If anyone wants to help/assist in anyway, please feel free to contact me.

18th December 2013


An updated build of embedded Firebird for IOS can now be downloaded from

http://www.ibphoenix.com/downloads/FirebirdIOSembed.tar

There were some issues with loading dylibs, these have been corrected using the install_name_tool -change
command. The default embedded script I used to build the embedded package forgets that I had to cross compile ICU 5.1 rather than build ICU with Firebird.

isql/gbak (or any other firebired utility) can now find the relevant libs they need by searching the directory above them i.e. MacOS. There was however an issue if you used

symbolic links, symbolic links, extra ``/'' characters, and references to /./ and /../ in the file_name.

This has now been corrected using the realpath function.
I haven't tested this myself yet on IOS, so once again feedback would be appreciated.



  

Some Analytics

Known new Firebird installations (direct via the /afterinstall page) between 4th June 2011 and 31th July 2013 (since the transition to new Firebird website)

Total: 1,584,731

The following numbers don't just count only the above direct visits (as there are about 1%-2% of installs coming from other sources)

By Country:

1. Brazil    555,483
2. Russia    160,982
3. Turkey    104,448
4. Poland    75,960
5. Germany    65,597
6. Ukraine    57,509
7. South Korea    46,202
8. China    44,076
9. Italy    38,526
10.USA        33,064
11.Mexico    26,688
12.Spain    25,654
13.Czech Rep.    25,035
14.Indonesia    22,732
15.Colombia    21,964
16.France    21,758
17.Bulgaria    19,602
18.(not set)    16,827
19.Belarus    16,414
20.South Africa    14,466
21.Japan    14,289
22.UK        13,314
23.Hungary    11,189
24.India    10,451
25.Malaysia    10,421

Others are below 10,000

By Sub Continent Region:

1. South America    604,253
2. Eastern Europe    381,068
3. Eastern Asia        114,604
4. Western Asia        113,098
5. Western Europe    109,585
6. Southern Europe    84,432
7. South-Eastern Asia    55,412
8. Northern America    37,434
9. Central America    31,323
10.Northern Europe    24,592

By Continent:

1. Americas    676,227
2. Europe    599,677
3. Asia        308,592
4. Africa    37,020
5. (not set)    16,827
6. Oceania    10,008

We know that Brazil is the world leader in Firebird installations (one installation of Firebird per 362 citizens), but it still quite amazing that one single South American country has more Firebird installations than Europe in total.

Third place for Turkey is very interesting. I wonder what they are using Firebird for? It must be something quite large as 100k installations is not a small amount (one Firebird installation per 725 citizens, where for example the Russian Federation has one installation per 881).

Fourth for Poland is not so surprising, but perhaps it does indicate that it would be a good place for some kind of regional Firebird event.

South Korea and China taking 7th and 8th place is also interesting. Although total numbers are "small" for China, almost one hundred thousand installations combined is an interesting foothold. (It's about 1/3 of the approximately 300,000 installations in Asia). We previously assumed that the strong regional leader was Japan (installations per capita), but it's actually South Korea with one Firebird installation per 1086 citizens)

So we took these 1,584,731 servers and calculated an equivalent cost of buying InterBase for a single user at (200.00 Euros + 38.00 VAT), then thats a saving to Firebird users of  377 million Euros, or 2,262 million Euros if these installations were all translated into 10 user servers (1,200.00 Euros + 228.00 Euros VAT), or
4,714 million Euros if they were all 25 user servers (2,500 Euros + 475.00 VAT).

Very little of those savings make their way back to the Firebird project (less than 0.1%), assuming that we took the lowest figure (377 Million Euros) indicated by single user installs.

(cuurent prices taken from Embarcadero online shop)

IBConsole

I had to do some research on behalf of a customer, to see how viable IBConsole (released with InterBase 6.0) would be with Firebird 2.5.2 Classic. (remember it was developed for SuperServer via the Services API only). The results were a little surprising.

1. Creating and adding users works
2. Sweep works
3. Backup works
4. Restore works (Although it does generate an access violation in the console on completion)
5. Transaction Recovery (limbo's) generates an unavailable database error.
6. Shutdown - Database shutdown completed successfully, database has been shut down
and is currently in single user mode. A gstat -h indicates that the database is in multi-user maintenace mode, so the database has been closed for maintenance and multiple connections are allowed only for SYSDBA or the database owner only.
7. Restart completes sucessfully, and gstat -h indicates that  the database is now in normal mode.
8. View Metadata generates a RDB$CHARACTER_SET not found error.
9. The SQL tool seems to work ok.

So in general - its still usable with some restrictions

Personally if you want a sraightforward tool,  I would recommend FlameRobin. Its free and it supports interactive SQL, backup/restore etc and user management. Its also cross platform so you can run it locally on Linux and MacOSX if you want.

How to backup a working Firebird database using a third party backup tool.

One of the basic problems of trying to use a third party backup tool, or simple file copy of a Firebird database whilst a database is active and online, is that the tool or utility has no concept of transactions, so all you get is a copy of what is in the O/S buffer or disk of the database at the time you make the backup.  However this "copy" of the database might be changing as new or active transactions are committed to the database while the copy is taking place. This is likely to produce at best, an inconsistent database, or at worst something that is corrupt and can't be used. Prior to Firebird 2.0 (other than using gbak) the only way you could do a backup or copy like this was to shut the database down, and make sure that no users were accessing the database before you invoked the third party backup tool or copy.

However it is possible to use Nbackup to achieve the functional equivalent of a gbak and use a third party backup tool.

The first thing you need to do is start a "freeze" on your database using the following syntax.

nbackup -U username -P password -L database.fdb

This will effectively lock the database, a flag is placed on the database header page, and it is set to "Locked" to let the engine know that all amended database pages that are written to the database are now being redirected to a delta file.

Changes are flushed from the internal (Firebird) database cache to the O/S cache when a transaction is committed, if forced writes are on then these changes are flushed directly to disk, the final task on commit is to mark the transaction as committed in the Transaction Inventory Page. Once the database is locked, all commits are written to the delta file rather than the database, thus ensuring that the database is kept in a consistent state.

Once the lock is applied, a simple gstat -h on the database wil show the "LOCKED" status as an optional database attribute. Once the -L command has done its work, a dela file will now be capable of receiving any committed changed database pages.

You can now use your an alternative backup tool whilst database users continue to work. When your backup tool has finished doing what it needs to do to take a copy of the database, you can "unfreeze" the database using the nbackup -N (unlock) command.

nbackup -U username -P password -N database.fdb

The unfreeze causes nabckup to merge the changed pages from the delta file back into the main database, when completed the delta file is removed and the database header is changed back to its normal state.
The backup you made of the database in its frozen state will still be in a "LOCKED" state, so if you need to restore it users will be unable to attach to it until you perform a "fixup". The fixup will reset the locked flag on the database header page back to normal, even though there isn't a delta file associated with the database.

Note: If you are going to use this capability, please make sure that you are using the latest version of Firebird, as a number of bugs in nbackup have been fixed since its original release.

CORE-4100

An attempt to explain the rationale behind Vlads fix for CORE-4100.

When a sweep has successfully finished its work it advances the Oldest Interesting Transaction (OIT) up to the value of the Oldest Snapshot Transaction (OST) that was recorded when the sweep started. The OIT transaction is the first transaction in a state other than committed in the database’s Transaction Inventory Pages (TIP), while the OST is the oldest transaction that was started in Snapshot mode.

If while the sweep is running, there are more new transactions started than is the sweep interval (by default, when the OIT is 20,001 transactions less then the Oldest Snapshot Transaction), it is possible that the new OIT value could again be more than the sweep interval less the OST value, thus ensuring that a new sweep could start immediately.

After a sweep has completed the first new transaction will pick up the updated OIT value from the saved OST on the database header page that was recorded when the sweep began, it will also read the actual OST from the header page, as well as the Oldest Active Transaction (OAT), the first transaction marked as active in the TIP pages. If the sweep condition is then met, a sweep begins.

Ideally what should happen is that the transaction should pick up the recalculated OIT from (in transaction) rather than the OIT from the header page in order to determine whether a sweep should start or not.

An example:

1. Transaction 1000 was rolled back, therefore the next transaction when it calculates the OIT will use 1000 and is now considered a stuck or “interesting” value.

2. By the time transaction 21001 occurs we have the following numbers:
OIT     1000
OST     21000,
Next     21001

3. An automatic  sweep is started, and it makes a note that the OST is 21000

4. While the sweep is running 30000  new transactions get started and committed.

5. When the sweep has finished doing its garbage collection and is about to advance the OIT, the numbers on the database header page are in effect
OIT     1000
OST     51000
Next    51001

6. The sweep then advances the OIT up to previously noted OST (21000)

7. A new transaction is started and it then obtains the following numbers from the database header page:
tra_oldest (OIT)            21000
tra_oldest_active (OAT)        51002
tra_number (OST)        51002
 
However within the transaction it has also recalculated the new oldest interesting transaction number as 51001 which will be written to the database header page at the end of the transaction.

8. However based on the OIT read from the database header page the following condition below is true

tra_oldest_active (OAT) - tra_oldest (OIT) > sweep_interval
51002 – 21000 > 20000 therefore a sweep will be started again.

9. However when sweep starts the database header page will have been updated to contain an OIT of 51001, so instead of doing the above, we really should check the local OIT that is going to be written out to the header page rather than the header page itself, before deciding on whether to do a sweep or not.