Michel Krämer

With the grow­ing num­ber of devices that can col­lect spa­ti­otem­poral in­form­a­tion, as well as the im­prov­ing qual­ity of sensors, the geo­spa­tial data volume in­creases con­stantly. Be­fore the raw col­lec­ted data can be used, it has to be pro­cessed. Cur­rently, ex­pert users are still re­ly­ing on desktop-based Geo­graphic In­form­a­tion Sys­tems to per­form pro­cessing work­flows. However, the volume of geo­spa­tial data and the com­plex­ity of pro­cessing al­gorithms ex­ceeds the ca­pa­cit­ies of their work­sta­tions. There is a paradigm shift from desktop solu­tions to­wards the Cloud, which of­fers vir­tu­ally un­lim­ited stor­age space and com­pu­ta­tional power, but de­velopers of pro­cessing al­gorithms of­ten have no back­ground in com­puter sci­ence and hence no ex­pert­ise in Cloud Com­put­ing.

Our re­search hy­po­thesis is that a mi­croservice ar­chi­tec­ture and Do­main-Spe­cific Lan­guages can be used to or­ches­trate ex­ist­ing geo­spa­tial pro­cessing al­gorithms, and to com­pose and ex­ecute geo­spa­tial work­flows in a Cloud en­vir­on­ment for ef­fi­cient ap­plic­a­tion de­vel­op­ment and en­hanced stake­holder ex­per­i­ence. We present a soft­ware ar­chi­tec­ture that con­tains ex­ten­sion points for pro­cessing al­gorithms (or mi­croservices), a work­flow man­age­ment com­pon­ent for dis­trib­uted ser­vice or­ches­tra­tion, and a work­flow ed­itor based on a Do­main-Spe­cific Lan­guage. The main aim is to provide both users and de­velopers with the means to lever­age the pos­sib­il­it­ies of the Cloud, without re­quir­ing them to have a deep know­ledge of dis­trib­uted com­put­ing. In or­der to con­duct our re­search, we fol­low the Design Sci­ence Re­search Meth­od­o­logy. We per­form an ana­lysis of the prob­lem do­main and col­lect re­quire­ments as well as qual­ity at­trib­utes for our ar­chi­tec­ture. To meet our re­search ob­ject­ives, we design the ar­chi­tec­ture and de­velop ap­proaches to work­flow man­age­ment and work­flow mod­el­ling. We demon­strate the util­ity of our solu­tion by ap­ply­ing it to two real-world use cases and eval­u­ate the qual­ity of our ar­chi­tec­ture based on defined scen­arios. Fi­nally, we crit­ic­ally dis­cuss our res­ults.

Our con­tri­bu­tions to the sci­entific com­munity can be clas­si­fied into three pil­lars. We present a scal­able and modi­fi­able mi­croservice ar­chi­tec­ture for geo­spa­tial pro­cessing that sup­ports dis­trib­uted de­vel­op­ment and has a high avail­ab­il­ity. Fur­ther, we present novel ap­proaches to ser­vice in­teg­ra­tion and or­ches­tra­tion in the Cloud as well as rule-based and dy­namic work­flow man­age­ment without a pri­ori design-time know­ledge. For the work­flow mod­el­ling we cre­ate a Do­main-Spe­cific Lan­guage that is based on a novel lan­guage design method.

Cur­rent 3D data cap­tur­ing as im­ple­men­ted on for ex­ample air­borne or mo­bile laser scan­ning sys­tems is able to ef­fi­ciently sample the sur­face of a city by bil­lions of un­se­lect­ive points dur­ing one work­ing day. What is still dif­fi­cult is to ex­tract and visu­al­ize mean­ing­ful in­form­a­tion hid­den in these point clouds with the same ef­fi­ciency. This is where the FP7 IQmu­lus pro­ject enters the scene. IQmu­lus is an in­ter­act­ive fa­cil­ity for pro­cessing and visu­al­iz­ing big spa­tial data. In this study the po­ten­tial of IQmu­lus is demon­strated on a laser mo­bile map­ping point cloud of 1 bil­lion points sampling ~ 10 km of street en­vir­on­ment in Toulouse, France. After the data is up­loaded to the IQmu­lus Ha­doop Dis­trib­uted File Sys­tem, a work­flow is defined by the user con­sist­ing of re­til­ing the data fol­lowed by a PCA driven local di­men­sion­al­ity ana­lysis, which runs ef­fi­ciently on the IQmu­lus cloud fa­cil­ity us­ing a Spark im­ple­ment­a­tion. Points scat­ter­ing in 3 dir­ec­tions are clustered in the tree class, and are sep­ar­ated next into in­di­vidual trees. Five hours of pro­cessing at the 12 node com­put­ing cluster res­ults in the auto­matic iden­ti­fic­a­tion of 4000+ urban trees. Visu­al­iz­a­tion of the res­ults in the IQmu­lus fat cli­ent helps users to ap­pre­ci­ate the res­ults, and de­velopers to identify re­main­ing flaws in the pro­cessing work­flow.

In this pa­per we pro­pose a soft­ware ar­chi­tec­ture that al­lows for pro­cessing of large geo­spa­tial data sets in the cloud. Our sys­tem is mod­u­lar and flex­ible and sup­ports mul­tiple al­gorithm design paradigms such as MapRe­duce, in-memory com­put­ing or agent-based pro­gram­ming. It con­tains a web-based user in­ter­face where do­main ex­perts (e.g. GIS ana­lysts or urban plan­ners) can define high-level pro­cessing work­flows us­ing a do­main-spe­cific lan­guage (DSL). The work­flows are passed through a num­ber of com­pon­ents in­clud­ing a parser, in­ter­preter, and a ser­vice called job man­ager. These com­pon­ents use de­clar­at­ive and pro­ced­ural know­ledge en­coded in rules to gen­er­ate a pro­cessing chain spe­cify­ing the ex­e­cu­tion of the work­flows on a given cloud in­fra­struc­ture ac­cord­ing to the con­straints defined by the user. The job man­ager eval­u­ates this chain, spawns pro­cessing ser­vices in the cloud and mon­it­ors them. The ser­vices com­mu­nic­ate with each other through a dis­trib­uted file sys­tem that is scal­able and fault-tol­er­ant. Com­pared to pre­vi­ous work de­scrib­ing cloud in­fra­struc­tures and ar­chi­tec­tures we fo­cus on the pro­cessing of big het­ero­gen­eous geo­spa­tial data. In ad­di­tion to that, we do not rely on only one spe­cific pro­gram­ming model or a cer­tain cloud in­fra­struc­ture but sup­port sev­eral ones. Com­bined with the pos­sib­il­ity to con­trol the pro­cessing through DSL-based work­flows, this makes our ar­chi­tec­ture very flex­ible and con­fig­ur­able. We do not only see the cloud as a means to store and dis­trib­ute large data sets but also as a way to har­ness the pro­cessing power of dis­trib­uted com­put­ing en­vir­on­ments for large-volume geo­spa­tial data sets. The pro­posed ar­chi­tec­ture design has been de­veloped for the IQmu­lus re­search pro­ject fun­ded by the European Com­mis­sion. The pa­per con­cludes with the eval­u­ation res­ults from ap­ply­ing our solu­tion to two ex­ample work­flows from this pro­ject.

Today’s politi­cians are con­fron­ted with new in­form­a­tion tech­no­lo­gies to tackle com­plex de­cision-mak­ing prob­lems. In or­der to make sus­tain­able de­cisions, a pro­found ana­lysis of so­ci­etal prob­lems and pos­sible solu­tions (policy op­tions) needs to be per­formed. In this policy-ana­lysis pro­cess, dif­fer­ent stake­hold­ers are in­volved. Be­sides in­ternal dir­ect ad­visors of the policy makers (policy ana­lysts), ex­ternal ex­perts from dif­fer­ent sci­entific dis­cip­lines can sup­port evid­ence-based de­cision mak­ing. Des­pite the al­leged im­port­ance of sci­entific ad­vice in the policy-mak­ing pro­cess, it is ob­served that sci­entific res­ults are of­ten not used. In this work, a concept is de­scribed that sup­ports the col­lab­or­a­tion between sci­ent­ists and politi­cians. We pro­pose a sci­ence-policy in­ter­face that is real­ized by in­clud­ing in­form­a­tion visu­al­iz­a­tion in the policy-ana­lysis pro­cess. There­fore, we identify syn­ergy ef­fects between both fields and in­tro­duce a meth­od­o­logy for ad­dress­ing the cur­rent chal­lenges of sci­ence-policy in­ter­faces with visu­al­iz­a­tion. Fi­nally, we de­scribe three ex­em­plary case stud­ies car­ried out in European re­search pro­jects that in­stan­ti­ate the concept of this ap­proach.

We eval­u­ate three Open-Source WebGL frame­works (X3­DOM, three.js and Cesium) and in­vest­ig­ate their use for geo­spa­tial ap­plic­a­tions in the Web. Over the course of one year we car­ried out five case stud­ies and de­veloped dif­fer­ent soft­ware pro­to­types us­ing these frame­works. We fo­cus on geo­spa­tial ap­plic­a­tions since they re­quire unique fea­tures that bring the eval­u­ated WebGL frame­works to their lim­its. Geo­spa­tial data is typ­ic­ally het­ero­gen­eous and very large. Geo­spa­tial ap­plic­a­tions re­quire spe­cial data man­age­ment and in­ter­ac­tion tech­niques. We de­scribe the cri­teria that we defined to test the WebGL solu­tions and present our ex­per­i­ence from work­ing with them. We give a qual­it­at­ive com­par­ison and fin­ish the pa­per with con­clu­sions and an out­look on fu­ture re­search per­spect­ives.