Siaqodb 4.0 stable

Siaqodb 4.0 stable version released, about features it brings take a look on previous blog posts: here and here.


Azure Mobile Services and Siaqodb

In our latest BETA we introduced SiaqodbSyncMobile which allows you to work with your data even there is no connection available and synchronize later on when connection is available with a cloud database.

Why?

With Azure Mobile Services you can store data in the cloud using Windows Azure SQL database, blob storage, table storage or third party data services like MongoDB, but it requires a permanent and stable connection for your app. There are a lot of situations when connection for mobile devices is poor/unstable so it leads to poor user experience.
SiaqodbSyncMobile solves this problem by offering a fully offline working mode for your app; basically you can manipulate data without connection at all, by storing it on local Siaqodb database and, later on, when connection is stable, you may synchronize those changes with cloud database. Even more you can work in parallel offline/online, so if a Save to server fails, you can fall back and store that entity offline in Siaqodb database. It also brings costs benefits since your traffic with cloud will decrease.

The synchronization process it’s complex internally but very powerful, offering possibility and flexibility to detect and resolve conflicts if two or more clients write changes to same item. Azure Mobile Services offers Optimistic Concurrency Control but only for Update operation, it does not catch and handle Delete operations. With SiaqodbSyncMobile all conflicts may be detected and resolved as app needs; those conflicts are caught using server-side scripts where you can define your custom business logic.

How it works?

Supposing you have your entity defined as:

 public class TodoItem
    {
        [JsonProperty(PropertyName = "id")]
        public string Id { get; set; }

        [JsonProperty(PropertyName = "text")]
        public string Text { get; set; }

        [JsonProperty(PropertyName = "complete")]
        public bool Complete { get; set; }

        [JsonProperty(PropertyName = "__version")]
        public string Version { get; set; }
    }

Notice that besides business logic properties:Id,Text and Complete, there is also added Version property which is “system property”.

“__version” field is a TimeStamp which is updated every time a record is modified(on server). To be able to handle concurrency you’ll have to add this system property to your entities, but you don’t have to assign value to it or use it, it will be handled automatically by SiaqodbSyncMobile and on server side by Azure system.

You can now manipulate objects online/offline or mix them.
If your app is online you may work directly with Azure Mobile Services:

using Microsoft.WindowsAzure.MobileServices;
.....
 
   MobileServiceClient mobileService = new MobileServiceClient("https://yourAMS.azure-mobile.net/","yourAppKey");
   IMobileServiceTable<TodoItem> todoTable = mobileService.GetTable<TodoItem>();
   await todoTable.InsertAsync(new TodoItem(){Id=Guid.NewGuid().ToString()});

If your app is offline you can store the object on local Siaqodb database and later on Sync:

using Sqo;
using SiaqodbSyncMobile;
.....
    SiaqodbMobile localDatabase = new SiaqodbMobile("https://yourAMS.azure-mobile.net/","yourAppKey", "mylocaldb");
    await localDatabase.StoreObjectAsync(todoItem);

later on when connection is back:

//here all local changes will be uploaded to server and new changes(done by other clients) will be downloaded
    await localDatabase.SyncProvider.Synchronize();
 

That’s it, now your application works regardless connection.

How synchronization process works?

There are 2 big steps in the Sync process:
A.Upload changes to server
B.Download changes to server

A1.Inserts are packaged and uploaded to server in batch(so all inserted records are uploaded in one package/one request)
A2.Updates are packaged and uploaded to server in batch(so all updated records are uploaded in one package/one request)
A3.Deletes are packaged and uploaded to server one by one (Delete script does not allow batch upload since it gets only ID as argument and not the entity)

B.Download all entities at once(inserted, updated and deleted-yes even deleted, but only IDs and tableNames)
B1.Store inserted and updated items on local db.
B2.Apply deletes on local db.

This is the simplified flow of the Sync process.

How concurrency conflicts are handled and resolved?

Two or more clients may write changes/or delete the same item, at the same time. Without any conflict detection, the last action would overwrite any previous updates/delete which is not desired.
SiaqodbSyncMobile detects those conflicts and allows you to resolve them or it will resolve them automatically by our customization of server-side scripts. We offer a tool to generate those scripts and also instructions how to upload them all at once.

Let’s imagine a typical case where we have 2 clients(C1 and C1) and a record R1; in the following cases a conflict may occur:

1.C1 update R1, C2 update R1
2.C1 update R1, C2 delete R1
3.C1 delete R1, C2 update R1
4.C1 delete R1, C2 delete R1

When C1 and C2 synchronize those changes to server, it depends who will Sync first.
Let’s suppose C1 Sync first and no other clients updated/deleted R1, then all C1 changes will be applied with no conflict.
When C2 will try to Sync, based on __version column and tombstone table, server side scripts will detect a conflict and will handle as:
1.a.If conflictWinner=”server”-> means C2 update will be rejected since C1 update was applied already
b.If conflictWinner=”client”->means C2 updated will be applied over C1 update
c.You can customize who is “winner” based on your business needs.

2.a.If conflictWinner=”server”-> means C2 delete will be rejected since C1 update was applied already
b.If conflictWinner=”client”->means C2 delete will be applied over C1 update
c.You can customize who is “winner” based on your business needs.

3.a.If conflictWinner=”server”-> means C2 update will be rejected since C1 delete was applied already
b.If conflictWinner=”client”->means C2 update will be applied over C1 delete(so record will be reinserted into db with C2 changes)
c.You can customize who is “winner” based on your business needs.

4.a.If conflictWinner=”server”-> nothing happen since C1 delete was applied already
b.If conflictWinner=”client”->nothing happen since C1 delete was applied already
c.You can customize who is “winner” based on your business needs.

Inside Siaqodb4.0Beta package you will find also a tutorial how to start using Siaqodb and Azure Mobile Services and also full examples.

We wait for your feedback!

Siaqodb 4.0 is here

We just launched Siaqodb 4.0 as BETA version. This new version brings again some cool features:

  • Document database features
  • Portable Class Library(PCL) support and unify assemblies( no more sync/async assemblies, no more trial assemblies)
  • Nuget packages for all platforms
  • Azure Mobile Services – synchronization with local Siaqodb database

DocumentDB support…yes Siaqodb has now features from document databases; this feature was demanded by our customers and also we consider will bring a lot of value to Siaqodb. Now a certain property/field may be stored as a Document(concept from Document Databases) – full object graph is stored together as a unit/snapshot. To use this feature you have to mark a property/field with [Sqo.Attributes.Document] attribute or to add via SiaqodbConfigurator class.
Example:

class Customer
{
....
[Document]
public Invoice TheInvoice{get;set;}
}

Important to notice is that serialization engine of a Document is plug-able, so you can use Protocol Buffers, BSON, MsgPack or any other serialization engines. We stronlgy recommend to use Protocol Buffers due to availability of Protobuf-NET project which runs on almost all platforms Siaqodb runs; from our tests is fastest serializer for .NET and also have a compact size.
To set your desired serialization plugin you have to implement IDocumentSerializer interface and set it via SiaqodbConfigurator.SetDocumentSerializer(…)
Examples of plugins:

//serialization plugin using protobuf-net

        public class ProtoBufSerializer : IDocumentSerializer
        {

            #region IDocumentSerializer Members

            public object Deserialize(Type type, byte[] objectBytes)
            {
                using (MemoryStream ms = new MemoryStream(objectBytes))
                {
                    return ProtoBuf.Serializer.NonGeneric.Deserialize(type, ms);
                }
            }

            public byte[] Serialize(object obj)
            {
                using (System.IO.MemoryStream ms = new System.IO.MemoryStream())
                {
                    ProtoBuf.Serializer.NonGeneric.Serialize(ms, obj);
                    return ms.ToArray();
                }
            }

            #endregion
        }

        //serialization plugin using BSON
        public class BSONSerializer : IDocumentSerializer
        {
            #region IDocumentSerializer Members
            readonly Newtonsoft.Json.JsonSerializer serializer = new Newtonsoft.Json.JsonSerializer();
            public object Deserialize(Type type, byte[] objectBytes)
            {
                using (MemoryStream ms = new MemoryStream(objectBytes))
                {
                    var jsonTextReader = new Newtonsoft.Json.Bson.BsonReader(ms);
                    return serializer.Deserialize(jsonTextReader, type);
                }
            }

            public byte[] Serialize(object obj)
            {
                using (System.IO.MemoryStream ms = new System.IO.MemoryStream())
                {
                    var writer = new Newtonsoft.Json.Bson.BsonWriter(ms);
                    serializer.Serialize(writer, obj);

                    return ms.ToArray();
                }
            }

            #endregion
        }

Example to set the document serialization plugin:

SiaqodbConfigurator.SetDocumentSerializer(new ProtoBufSerializer());

See how it works in our example project.

About PCL support: you can build now only one PCL DataLayer assembly and use it everywhere without recompilation, this brings a lot of benefits when developing cross platform solutions.
Take a look on our examples project.

Nuget, yes we published Nuget packages so it’s very easy to integrate it in your projects with VisualStudio or XamarinStudio.

Azure Mobile Services Sync: We enable now to use Azure Mobile Services even when the device is offline by giving possibility to Synchronize a Siaqodb local database with a server side database using Azure Mobile Services. We will dedicate another blog post for this. If you are eager to test take a look on our examples projects and tutorial to setup it.

PS:This version is in BETA for now so we do not recommend to use in production yet.

Would be great to hear feedback about new features!

Use async everywhere

We just release Siaqodb version 3.7 which bring fantastic new features, first of all ASYNC support on mostly all platforms:
WinRT, .NET 4.0, .NET 4.5, Silverlight 4.0 and 5.0, WindowsPhone 7.1 and 8.0, Xamarin.iOS and Xamarin.Android and at the same time we provide same API on all platforms, even WinRT version now has SYNC and ASYNC API so your Data Layer may be shared accross platforms without any change.
For ASYNC API we have new assemblies called Siaqodb*Async.dll which contains SYNC + ASYNC API, but we also provide SYNC-only API like in previous versions.
Let’s see some code:

public async Task DoSomeDBOperations()
{
 Siaqodb siaqodb = new Siaqodb(); 
 await sq.OpenAsync(dbFolder);

 await sq.StoreObjectAsync(new Employee());

 IList<Employee> list = await sq.LoadAllAsync<Employee>();

 var query = await (from Employee employee in siaqodb
                    where employee.Age > 50
                    select employee).ToListAsync();

}
 

So basically we have pairs of methods in Siaqodb class: one normal (SYNC) and one that has suffix “Async”. About queries: a query will run Async if ToListAsync() method is called.

Important to say is that Siaqodb fully implements Task-based Asynchronous Pattern (TAP) so ultimately any Query or Store operation may end-up calling newly added async methods of IO streams like:
FileStream.ReadAsync/FileStream.WriteAsync.

Siaqodb*Async assemblies on platforms like .NET 4.0, Silverlight, WindowsPhone 7.1 depends on Microsoft.Bcl.Async package which can be found as NuGet package from http://nuget.org/

This new version also brings fixes and improvements for example by new buffering feature, as result queries that require full scan are 3 till 5 times faster now. See more info about fixes and complete release notes on our download page.

Enjoy!

Siaqodb 3.6 released

We just released a new version which brings the most requested feature in last period: OID is not mandatory anymore in a persistent Type.
So instances of following class may be stored by Siaqodb:

public class Person
    {
        public DateTime BirthDate { get; set; }
        public string Name { get; set; }
        public string FirstName { get; set; }
        private int age;
    }

There are required more details how we implement it and how the engine will detect what objects need to be updated and what are seen as new objects since OID may miss from class definition.
Important to say first is that the new version is fully backward compatible, also important is that if you add OID property to
a persistent type, it will be still managed by the engine, so autoincremented; this can be still very useful in many scenarios and by having OID along with object you can update them directly or compare with existing objects in DB.
However OID still exists internally, the engine still identify the uniqueness of the object by OID.

For the types that not have defined OID property,when objects are loaded from database we keep them in a “weak cache structure”, so when a object it is stored, the engine check first if its reference exists in cache, get the OID if exists and then update the object, if the reference is not found in the cache then the engine will consider object as new and will insert it in the database. By a “weak cache structure” we mean that we cache only WeakReferences of objects, so if an object is not referenced anymore by your app, it can be collected by the garbage collector.
Also the cache automatically clean-up empty weak references, you don’t have to worry or configure anything about it.

Another thing is that if later on, in project life-cycle,you would need OID along with object, you can add the property at any time to the class definition and will be filled by the engine automatically with no extra overhead.

Enjoy!

Siaqodb 3.5 released

We just released Siaqodb 3.5 which includes Siaqodb for WinRT and also Siaqodb for WindowsPhone 8.

The most exciting is the release for WinRT, see our previous post for more info. In the current version we added indexes and all other features of version 3.5, so Siaqodb has now a fully async API. We are also happy that the approach we took with async API and “async LINQ” queries is the same with the approach Microsoft took with EntityFramework 6.0, so .ToListAsync(), .CountAsync(),.FirstAsync() etc.

Also important release is the Siaqodb for WindowsPhone 8 release, the SDK for WindowsPhone 8 has some important changes, for example one that affected Siaqodb and did not compiled was the fact that now TypeInfo is a type in System.Reflection and we had same type defined inside Siaqodb so it was a namespace conflict, luckily it was not public, so we could refactor easy. Having Windows 8 kernel, WindowsPhone 8 SDK brings all async stuff, so it opens door to bring our async version also to WindowsPhone 8, stay tuned.

As new features in this version most important is the new Index storage, now all indexes resides in a BTree structured files and not in a AVLTree anymore. With this change performance is much better in most of the cases and we also expect much more stability.

We bring also a new utility class: SiaqodbUtil through you can now Shrink your database files and also Repair persisted object if they got corrupted, see the documentation how to use it.

Download it now and enjoy new features!

Siaqodb – local database for Metro style apps

We are very proud to announce that we finally ported Siaqodb to .NET profile for Metro style apps (NETFX_CORE) and fully async API. We have ported previously on several platforms but now it took longer because about 80% of the methods had to be rewritten. Until now we have used in our code preprocessor directives like #IF SILVERLIGHT, #IF MonoTouch etc, because it was easier to maintain in this way Siaqodb sources for multiple platform, but NETFX_CORE could not be used in a similar way because the sources are changed too much as result of a few things, most important:
1.”async methods”
2. Reflection changes

We know portable class library but shortly: it doesn’t work for Siaqodb.

To pass validation for Windows 8 App Store, your Metro style app should call all the I/O operations async, which means a lot for the Siaqodb library. The methods that may fall into calling a I/O operation must be async and they have to be called with “await” in case you need the result returned for further processing; and await can be used only in a async method, so all methods from the lowest layer to the uppermost layer must be async.

Another important aspect for Metro style apps:
“A Metro style app runs in a security container with limited access to the file system, network resources, and hardware. Whenever a user installs an app from the Windows Store, Windows looks at the metadata in the Package.appxmanifest file to figure out what capabilities the app needs to function. For example, an app might need to access data from the Internet, documents from the user’s Document Library, or the user’s webcam and microphone. When the app is installed, it displays to the user the capabilities it needs, and the user must grant permission for it to access those resources. If the app doesn’t request and receive access to a resource it needs, it will not be allowed access to that resource when the user runs it.”
(source MSDN)

Just as you can access in Silverlight the IsolatedStorage, in a similar way you will have full and unrestricted access to app’s local folder found in C:\Users\\AppData\Local\Packages\. Here you can create, modify and delete files and folders without asking user’s permission. Local data access is provided using Windows.Storage.ApplicationData.
So, for opening a Siaqodb database you have to pass as database name the folder where your data will be stored like:


      Siaqodb siaqodb = new Siaqodb();
      await siaqodb.Open(ApplicationData.Current.LocalFolder);

As you know the only query engine for Siaqodb is LINQ and Siaqodb LINQ provider is implemented by extending IEnumerable interface; bad news is that currently there is not shipped any IAsyncEnumerable interface that would support LINQ and also having a method signature like
public async Task> GetEnumerator()
or/and IAsyncEnumerator where MoveNext() method is async, and -very important- Microsoft says that such a interface/s will not be shipped for now.

So we had to find a way to use LINQ but also to load objects from database async as result of that LINQ statement. We’ve made our IAsyncEnumerator which is used by most methods within our LINQ provider. The ISqoQuery interface still implements IEnumerable but when it’s called its GetEnumerator() method, an exception is thrown and client API must use our X_Async() methods instead. Basically we parse lambda expression in the same way as we did before, but when objects need to get materialized, we should call an async method since it may fall in a I/O operation.

For the standard .NET version, Siaqodb LINQ provider can optimize following LINQ operations by re-defining new methods for:

Any(...)
Count(...)
First(...)
FirstOrDefault(...)
Join(...)
Last(...)
LastOrDefault(...)
OrderBy(...)
OrderByDescending(...)
Select(...)
Single(...)
SingleOrDefault(...)
Take(...)
ThenBy(...)
ThenByDescending(...)
Skip(...)
Where(...)

Most of the methods above will throw exceptions now, but the following methods must be used for Siaqodb for Metro Style apps:

AnyAsync(...)
CountAsync(...)
FirstAsync(...)
FirstOrDefaultAsync(...)
LastAsync(...)
LastOrDefaultAsync(...)
OrderBy(...)
OrderByDescending(...)
Select(...)
SingleAsync(...)
SingleOrDefaultAsync(...)
TakeAsync(...)
ThenBy(...)
ThenByDescending(...)
SkipAsync(...)
Where(...)

And to get result of a LINQ query that returns ISqoQuery you always have to call ToListAsync() method, otherwise, fetching the IEnumerable will throw exception.

So let’s see some code:

Siaqodb siaqodb = new Siaqodb();
await siaqodb.Open(ApplicationData.Current.LocalFolder);
//store the first person
await siaqodb.StoreObject(new Person() { FirstName = "Michael", LastName = "Smith" };);

//load all persons stored in database
var allPersons=await siaqodb.LoadAll<Person>();

//make a LINQ query
var persons=await(from Person pers in siaqodb
                        where pers.FirstName=="Michael"
                        select pers).ToListAsync();
                

The current version is in BETA stage, you can download it and use it as trial for 90 days
The Indexes are the only missing feature in this version( but still read performance is very good), all the rest of features of Siaqodb 3.2.0.5 are included in the Siaqodb for Metro style apps.

Very important, we ported also all UnitTests and all passed!

Since this is a BETA version, we wait for your feedback, so if you have questions/remarks shout us at support at siaqodb dot com.

So go to our download page and grab it now, you have a lot of examples that can be found in MetroExample project included in the setup!

SiaqodbManager now on Mac OSX

Because we saw high interest for Siaqodb from Unity3D developers, especially this being reflected in our sales for this platform, we ported
SiaqodbManager now on Mac OSX but also on any OS that support Mono, you can start it from terminal by: mono /User/…/SiaqodbManagerMono.exe

SiaqodbManagerMono.exe is WindowsForms based app so it does not look native Mac OSX app, but it is working fine and can be very helpful for developers, you can run ad-hoc LINQ queries and it has all features SiaqodbManager has on Windows. Only one limitation exists: on LINQ editor, the LINQ statements are not highlighted like on Windows version.
Enjoy!
siaqodbManager on Mac OSX

Siaqodb 3.2 released

We just released version 3.2 which brings great new features:

  • support for Dictionarytypes
  • add support for unlimited size jagged arrays types
  • add support unlimited nested Lists ( ex: List<List<int>>)
  • add support for IList items properties in LINQ Include() method (ex: .Include(“Employees.Nationality”); where Employees is IList)
  • performance improved on insert with 30% in same cases
  • new methods for fast insert: siaqodb.StartBulkInsert(…) and siaqodb.EndBulkInsert(…)
  • better information about exceptions
  • other improvements

So now objects of following class may be stored in Siaqodb database:

public class MyClass
{
public int OID{get;set;}

public List<List<int>> JaggedList {get;set;}

public int[][][] Matrix;

public Dictionary<int,AnotherClass> Dictionary{get;set;}
}

In LINQ queries over new supported types Siaqodb may optimize following methods:
-ContainsKey(…) of a Dictionary
-ContainsValue(…) of a Dictionary
but also it can optimize and compare a list with sub-list by its elements not by reference.
Examples:

var query=from MyClass m in siaqodb
where m.Dictionary.ContainsKey(10)
select m;

//or:
AnotherClass ac=new AnotherClass();
ac.Id=10;
var query=from MyClass m in siaqodb
where m.Dictionary.ContainsValue(ac)
select m;

Both runs optimized and at second example it is compared values of properties/fields of ‘ac’ object instead of comparing by its reference so you are able to pull from database data you want.

Next example:

List list=new List();
list.Add(1);
list.Add(2);

var query=from MyClass m in siaqodb
where m.JaggedList.Contains(list)
select m;

In this case it runs optimized and again it is compared each element of inner list from database with elements of ‘list’ object provided to query.

Other new methods that can be used when a lot of data need to be stored, are StartBulkInsert(…) and StartBulkInsert(…), example:

siaqodb.StartBulkInsert(typeof(Order));
try
{
for (int i = 0; i < n; i++)
{
....
Order order=new Order();
s_db.StoreObject(order);
}

}
finally
{
s_db.EndBulkInsert(typeof(Order));
}

We hope you’ll enjoy new features,if so, go to download page and grab the latest version.

Siaqodb 3.1 released

We just released version 3.1 of Siaqodb, as new features we added:

  • new method: StoreObjectPartially(…) added to Siaqodb class. Through this method you can save only a certain properties of an object and not a full object. This method arise as a feature request after partial object loading and eager loading feature was added, so it is possible that you did not load the object fully from database(complex nested objects was not loaded) and when this object would be saved, all complex properties values will be set to null. To avoid that and also in other scenarios you can use now new method :
    StoreObjectPartially(object obj,params string[] properties);
    
  • optimize LINQ methods:OrderBy(…), OrderByDescending(…), ThenBy(…), ThenByDescending(…). All these methods are optimized now by our LINQ provider implementation.
  • add support for storing directly generics types. Now the following type is storable:
    
    class MyClass<T>
    {
        public List<T> myList;
        public int OID{get;set;}
    }
    
    
  • added Replace parameter on LoadingObjectEventArgs class. There exists cases when you want to implement a caching system and, when object that has nested objects is loaded from database, instead of letting Siaqodb engine to load that object from database you can check by OID and Type if object exists in your cache and if yes, assign it to Replace parameter, so the engine assign your provided object and return to the caller.
  • new method EnableOptmisticConcurrency(…) added to SiaqodbConfigurator class useful when you work with SiaqodbOffline (in Sync Framework Provider context).
  • Enjoy new version and a Happy New Year from Siaqodb team!