Applications
Computational problems
What type of computational problems is solvable by application of newly developing infrastructure? The simplest situation is large amount of small, mutually independent jobs. Typical example can be parametric studies in which the same calculation is repeated for many times with slightly modified parameters. Results of such calculations are combined and used for evaluation of impact of parametr change (e.g. rigidity of a used glue) for the properties of the whole system. Grid is the optimal environment for such types of calculations. A user prepares large batch - that can contain even million of independent jobs - and grid environment will take care of their scheduling, running, results collecting and potetially also final processing. Grid will also handle the crashed jobs that can be automatically submitted again.
Similar type of jobs is treatment of extensive data files "piece after piece". Simple example is digital smoothing of satellite images. Each image can be treated individually, it is possible to split large images into smaller ones, individual parts can be handled separately and the result can be merged into a single image again. This (parallel) processing is the most interesting type of jobs for which Grids have been developed.
These types of jobs are extensive jobs whose processing using one CPU would take incredibly long time in general especially if their memory requirements are too big. Classical approach for solving these jobs is to use a special - and very expensive - parallel computers. The job is simultaneously processed by large number of CPUs that have access to very large memory (if we have one thousand of CPUs each with one GB memory, we have together considerable 1 TB of internal memory accessible for one job). Parallel computers are basically nothing more than a set of processors and memories interconnected by very fast network especially with special properties. A computer cluster (group of computers interconnected by "standard", still very fast, network and located in a small space) is the same at a specific level of abstraction and on a higher abstraction level we talk about a Grid (distance among the computers is higher in several orders of magnitude compared to a cluster).
Programming for parallel computers is unfortunately very hard because the job has to be split in a such a way that each processor has to be busy and the processors do not wait for each other one. The speed of data transfer inside a parallel computer is extremely high. This enables exchange of large volumes of data among processors without the loss of total performance. The situation in grid environment is much more complicated. The distance among processors and used network - high-speed Internet - cause much higher price (means time) of data transfer. Algorithms and procedures that are used in parallel computers are not sufficiently effective in grid environment. Therefore they have to be substituted by new ones or some of jobs types can not be currently effectively solved using Grid.
The basic term concerning parallelization of a job is the granularity of the problem. If the problem is highly dependent on the result of other sub-problems, with fine-grained parallel calculations. This is the case, for example, in a calculation of the weather, which can be split into many smaller calculations of the weather in small volumes of the atmosphere. Each of these calculations is strongly affected by what is happening in neighbouring volumes. In fact, even changes in the weather very far away can have an impact. In practice, the transformation of such job types for grid environment is highly difficult and fine-grained parallel calculations require very clever programming to make the most of their parallelism, so that the right information is available to processors at the right time.
At the opposite end of the granularity scale are coarse-grained or embarrassingly parallel calculations, where each sub-problem is independent of all others. This is the case, for example for so-called Monte Carlo simulations, where you vary the parameters in a complex model of a real system and study the results using statistical techniques - a sort of computer experiment used for example in computational chemistry. Each calculation can be done independently of the others in this case. Also the mentioned example of digital images processing belongs to the same category.
As a rule of thumb, fine-grained calculations are better suited to big, monolithic parallel supercomputers, or at least very tightly coupled clusters of computers, which have lots of identical processors with an extremely fast, reliable network between the processors, to ensure there are no bottlenecks in communications. This type of computing is often referred to as high-performance computing.
On the other hand, embarrassingly parallel calculations are ideal for a more loosely-coupled network of computers, since delays in getting results from one processors will not affect the work of the others. These types of calculations are often referred to as high-throughput computing.
Naturally, the grid environment prefers coarse-grained parallel calculations. But in fact, many of the interesting problems in science require a combination of fine- and coarse-grained approaches as homogeneous coarse-grained algorithms are very rare. And this is where the Grid can be particularly powerful due to availability of suitable computational resources for fine-grained sub-jobs whose results can be combined by the manner of coarse-grained sub-jobs.
Real example of described approach is doing complex climate modeling of the Earth, when researchers want to see how the calculations depend on different parameters in their models. So they want to launch many similar calculations. Each one is a fine-grained parallel calculation, because predicting climate is like predicting the weather on a longer time scale. So each calculation needs to be run on a single cluster or supercomputer. However, the many independent calculations could be distributed over many different clusters on the Grid, thus adding coarse-grained parallelism and saving a lot of time.
Suitable decomposition of a computational job followed by assignment of the sub-jobs to corresponding parts of the Grid is the area currently changing from an art - controled by small group of masters - through (a lot of human work) to enginnering discipline whose tools are available for growing group of users.
Applications
It is expected that the major part of the forming infrastructure will be exploited, especially in the first phase of the project, by applications from physics of elementary particles (HEP, High Energy Physics) that is in the Czech republic represented by the Institute of Physics of Czech Academy of Sciences. For HEP community is Grid necessary and in essence the only solution how to store and process data in PB (10^12) ordes of magnitude from particle physics experiments (Atlas, CMS, Alice, LHCb and other). Concurrently Grid enables cooperation in really worldwide scale - after all worlwide web has its roots also in CERN, the centre of HEP research.
One of the key aim of the EGEE project is at the same time broadening of the users foundation through wide spectrum of potential aplications. The portfolio of problems solvable using grid environment is relatively broad but their usage in grid environment is usually hindered by insufficient readiness on both sides - Grid specialists and users themselves. Taking into account described types of computational problems, the characteristic applications include computational chemistry simulations of biologically significant systems, processing of bioinformatic and medical data, material simulations and tests, growing interest is from monitoring and processing Earth observation data, distribution and analysis of astronomical measurements and particle physics experiments, and especially recently increased interest from real-time video and image processing. All applications using EGEE infrastructure will come under one of three following categories.
"Pilot" applications serve for the first testing of EGEE middleware implementation and verification of functionality and performance of the grid environment. The computational grid for Large Hadron Collider (LCG, Large Hadron Collider Computational Grid) - used as a model for the environment allowing processing petabytes of data from the particle physics CERN experiments - was the first pilot application in the first phase of the project. The second pilot application are Biomedical Grids where several medical communities are facing equally daunting challenges to cope with the flood of bioinformatics and healthcare data - data-mining from genomic databases or indexing medical databases in hospitals containing many terabytes of data for one hospital per a year.
The next type of applications are so-called "internal" that contain projecs solved by people involved in EGEE project as well but simultaneously cooperating with institutes outside of EGEE. In fact, these are in most cases national projects with considerable grid computing experience.
Last type of are so-called "external" applications requiring explicit support of selected user groups so that their applications would be adopted to the grid environment built by EGEE and simultaneously the users would obtain sufficient knowledge for their effective usage. Therefore, within the EGEE project there is an extensive dissemination activity including seminars and courses focused on specific user groups.