SpagoBI - Comprehensive Open Source Business Intelligence and Big Data analytics suite.
The suite operates in a portal which is managed by admin who grants access to the reports and functionalities.
The suite offers the following tools:
- Reporting;
- Multidimensional analysis (OLAP);
- Charts;
- KPIs;
- Interactive cockpits;
- Ad-hoc reporting;
- Location Intelligence;
- Free enquiry;
- Data mining;
- Network Analysis;
- Extract, Transform and Load;
- Collaboration;
- Office automation;
- Masterdata management;
- On-line process management.
Server-Side of the tool implemented in Java+Tomcat server making it cross platform. Most of SpagoBI libraries licensed under MPLv2 license although libraries with GPL also exist.
Pentaho - Robust BI set of tools. Named a visionary in Gartner 2016 Magic Quadrant for Business Intelligence and Analytics Platforms report. Powerful enterprise ready BI environment which includes but not limited to the following features:
- Agile platform for data integration, transformations and analysis;
- Native connectivity to Big Data sources such as Hadoop distributions, NoSQL stores, and analytical databases;
- Machine learning functions;
- Internet of Things Analytics;
- Extensive data integration.
Server-side implemented in Java running on Tomcat application server. There are two products offered by Pentaho company- Enterprise (paid) edition of the application and Community Edition- Open Sourced set of packages licensed under Apache license and different variations of GPL.
Hadoop - is a framework library which helps implementation of distributed data processing systems. Distributed systems compute and store data in many machines to achieve higher processing speed and reliability. Hadoop is highly scalable, it can work on a single machine as well as on thousands. Hadoop is based on Java. Incensed under Apache Licence V2.0.
Spagic - Open Source library based universal middle-ware for SOA/BPM solutions. By its name the middle-ware operates between two or more applications and services. The purpose of the middle-ware is to set reliable communication between applications to automate certain business processes or to create data exchange with legacy applications. As an simple example may be an ERP (Enterprise Resource planning) and CRM (Customer Resource Management) integration in the means of keeping customer information and orders aligned among the systems. An ERP example could be SAP and CRM - Salesforce CRM. The middle-ware design tools have designers to assemble actions in a business process graphically.
Spagic features the following:
- Services definition in graphical tool as well as by setting configuration files;
- Process Modelling;
- Deployment and execution complex processes with human interaction tasks;
- User interaction with editable PDF files.
Spagic is developed on Java and runs on Tomcat application server. The product is licensed under GNU Lesser General Public Licence (GNU LGPL). Spagic is part of SpagoWorld initiative.
Apache Camel is a complete framework for the integration following EIP (Enterprise Integration Patterns). It provides structured well manageable way in developing complex integrations in an Enterprise. The framework is message oriented where components communicate with each others through messages, nothing more. In other words Apache Camel is a messaging technology with routing. It supports most of communication protocols such as FTP, JSON, HTTP etc. and provides:
- Concrete implementation of all widely used Enterprise Integration Patterns (EIP);
- Connectivity to great variety of transports and APIs;
- Easy to use Domain Specific Languages (DSLs) to wire EIPs and transport together.
Apache Camel is Java based, Open Source products licensed under Apache License.
Add hoc - single communication set-up among small number of applications. Most of the time it may not be as cost effective to deploy middle-ware tools to connect few applications in a small organization. In such cases in-place communication interfaces are configured or developed using available application integration features. The technology used may vary completely depending on the components which to be connected together, on what technology the components themselves are made on. Add hoc integrations are considered as short term solutions and should be replaced as the organization develops.
.NET - probably the most famous of Microsoft's technologies. First released by Microsoft over decade ago, it became popular due to its sufficiency. The technology itself is language interoperable and cross platform. Meaning that it can be deployed on Windows, Linux based client and servers as well as on a platform made by Apple. It allows creation of WEB, workstation and server based applications. Good developer's tools and advanced developers support network attract many developers making it wide spread and recognizable. Over time it became reliable and well proven to use, especially on Windows-based platforms.
The technology is Open Sourced.
MS SQL - the acronym means MS SQL Server - advanced relational database management system developed by Microsoft. SQL means Structured Query Language. The system main function is to store, retrieve and process data. The system acts as a server meaning that it may work on a shared environment among many users simultaneously. It robustness allows the usage of the server for different purposes such as WEB-site data storage management and processing, Workstation client, BI client or vast variety of other applications. The server also includes powerful Integration and Analysis services which can be used for advanced data integration and analysis objectives.
The server may be deployed on Windows-based host and has dozen licensing models depending but not limited to number of processor core quantity which can be used by the server, maximum database size and quantity of users. Microsoft offers limited free editions of the server which can well be used for evaluation purposes or relatively small databases and operations.
Automation - basically means automation of Microsoft Documents processing and generation or other custom task related to MS Windows such as one or few processes periodic execution etc. Obviously, when it comes to the document processing, MS Office licensing applies.
Other technologies may entail JavaScript for WEB client-side development including SVG (Scalar Vector Graphics) dynamic objects running on browser-side, WCF (Windows Communication Foundation), WPF (Windows Presentation Foundation), WF (Windows Workflow Foundation). The technologies by themselves are free to use but the set of developer's tools as well as server-side libraries may be subject to paid licenses.
PHP - server-side scripting language providing necessary tools suite for developing static and dynamic web-pages, web-based APIs. The language compiler communicates directly to the API of the web-server favouring fast code execution. Most of CMSes (Content Management System) on the Internet are based on PHP as WordPress, Drupal, Joomla etc. As well as on-line shopping platforms such as Prestashop, Magento, osCommerce etc. PHP is Open Sourced technology distributed under the PHP License by means free to use and distribute. Paid license development tools exist offering own advantages among free Open Sourced.
MySQL, meaning as Oracle MySql database server- is an Open Sourced RDBMS (Relational Data Base Management System). Its reliability, security, high performance, scalability and up-time made It probably the most popular Open Sourced database system, especially for WEB applications. In many cases MySql used for mission critical applications. It is also used for storing and processing BigData, BI (Business Intelligence) systems and in embedded (limited purpose computing devices) applications. Its Enterprise-scale manageability brings it to the level of paid license products such as Microsoft SQL Server. Variety of free developer's tools made it wide-spread globally. Licensed under GPL v2 (General Public License) which permits free commercial usage.
JavaScript - most popular WEB scripting language. Most of the cases the scripts are used to run on WEB-pages making dynamic content possible. Does not require any plugins installed on the browser. Partial WEB-page content refresh is possible because of JavaScript. To simplify WEB client development, many frameworks were developed over time such as JQuery, ExtJs (Sencha) etc. All of them are based on JavaScript. Some of the frameworks are free for commercial use such as JQuery and paid - ExtJs although still Open Sourced. Although the scripts running on web-page have security restrictions, they can not list or manipulate files on the computer where browser runs the script, apart from page or domain specific cookies. Full page manipulations and background communications to the server are permitted. JavaScript or certain JavaScript libraries also can be used to manipulate complex graphics on the page. An example of graphics technology is SVG (Scalar Vector Graphics) which becomes a standard also consuming hardware accelerated browser API. The usage of JavaScript is obviously free for commercial purposes. There are powerful free developer's tools as well as paid with many features making developer more effective.
Pyphon - widely used general-purpose, high-level, interpreted (execution on the fly, no compilation) programming language. It has underling design philosophy which emphasizes code readability and allows developer to use fewer lines of code. This greatly impacts programming time and then allows to streamline support and changes implementation in the future reducing TCO (Total Cost of Ownership). For WEB programming in particular Django framework developed. The language can be used for Server and Workstation applications development running on Linux and Windows platforms. The usage of it is free for commercial purposes.
Other technologies include but not limited to SOAP, XML, C++ compiled binaries, bash scripting to automate certain tasks.
Nagios - for over 15 years it became an industry standard for IT infrastructure monitoring. The monitoring processes identify and alert about problems before they affect critical business process. Alerts allow administrators or system monitoring staff get into action to resolve the problem before it has disturbed the business. It greatly reduces system downtime, increases network and server health making applications highly available. Nagios is an Enterprise ready scalable, reliable and flexible monitoring platform. Apart from built-in infrastructure monitoring tools, it also offers possibility to customize and create new tools with custom logic, which can be used for checking application specific processes. For example, web-page or web-service return time or certain process logic correct execution.
The tools bring the following capabilities:
- It allows to plan for upgrades before an outdated system causes problem and interrupts the business;
- Gives possibilities to respond to issues at first sign of a problem;
- Automatic problem fixing where possible;
- Technical team responses coordination;
- Ensures an organization SLAs (Service Level Agreement) are being met;
- Ensures IT infrastructure outages have minimal effect on your organization bottom line;
- Allows monitoring of entire infrastructure and business processes.
It has the following features
- Comprehensive monitoring - possibility to monitor variety of network protocols, infrastructure components, system metrics, operation systems etc. in a single application;
- Visibility & awareness - centralized view of IT infrastructure in a web interface with detailed exhaustive information on a problem;
- Problem remediation - event handlers may execute application or service restart in case of a problem;
- Proactive planning - trending and capacity planning add-ons ensure awareness on ageing infrastructure;
- Reporting - historical and availability reports to ensure SLAs are being met;
- Multi-user access - role security framework to ensure the users see the status of components dedicated to them;
- Extendable architecture - hundreds of community developed add-ons and integration with in-house and third-party applications and services;
- Fail-over capabilities - to ensure continuous monitoring of critical components;
- Vibrant community - over 1+ million users world-wide;
- Customizable code – Open Sourced released under GPL (General Public License).
Alerts can be delivered to technical staff by email, SMS or custom script execution. The tool may monitor networks, Windows or Linux based servers and workstations, also applications running on those. It has two versions: paid one and community edition free.
Remote access - the team supporting your IT infrastructure or applications does not have to be on premises. Secure remote access tools can be used helping to solve problems when they arise - Microsoft RDP, VNC over ssh, NoMachine etc.
System Development Life Cycle refers to process which consists of the following stages: planning, creating, testing, deploying, maintaining and decommission of a system. Two types of the process usually used in regard to software development: Waterfall and Agile. Each of them have its own advantages and disadvantages.
Waterfall - mainly used in large projects when it is not possible to cut the scope in small chunks and move gradually till the completion or in small projects when there is no ambiguity in requirements and all stages of the project are prepared and clearly understood. An example of a large project can be the Space exploration when there is only one opportunity to fire. The whole project is planned in advance from start till the end with no coming back route. In other words, there is sequential, not iterative way of moving to project outcomes. Waterfall model was used as primary in early days of IT technologies. The process consists of the following stages, each stage follows the next one in sequence:
- Requirements - project requirements collection, formalization, systematization and examination;
- Design - solution design, which meets the requirements list produced in requirements stage. Technical documentation is the outcome of this stage;
- Implementation - at this stage realization comes due following the design documentation;
- Verification - the outcomes of Implementation stage get thoroughly verified and moved to Production (Live) usage. Verification stage also continues certain timespan when the outcomes of verification run in Production mode;
- Maintenance - the project outcome are fully operational and technically supported to achieve stability.
Any stage in Waterfall can not be omitted.
Advantages of Waterfall model:
- ease of use and understand
- the stages are processed one at a time
- Works well in small projects
Disadvantages:
- Difficulty to go back to preceding stage when something fails. Due to ambiguous requirements for example;
- Not working product may be produced at late project stages;
- High level of risk and uncertainty;
- Not suitable for projects when requirements may change over project lifespan.
Agile - opposite to waterfall model, it is iterative. Whole project scope is divided in small chunks which execute incrementally. Agile promotes adaptive planning, evolutionary development, early delivery, continues improvement and encourages rapid and flexible response to change. Single project chunk has the following stages: Planning, Development, Testing and release. The chunk timespan is small enough to cover one or few simple requirements and fit into maximum two weeks till release.
Agile manifesto is based on twelve principles:
- Customer satisfaction by early and continuous delivery of valuable software;
- Welcome changing requirements, even in late development;
- Working software is delivered frequently (weeks rather than months);
- Close, daily cooperation between business people and developers;
- Projects are built around motivated individuals, who should be trusted;
- Face-to-face conversation is the best form of communication (co-location);
- Working software is the principal measure of progress;
- Sustainable development, able to maintain a constant pace;
- Continuous attention to technical excellence and good design;
- Simplicity;
- Best architectures, requirements, and designs emerge from self-organizing teams;
- Regularly, the team reflects on how to become more effective, and adjusts accordingly.
Advantages:
- Higher customer satisfaction;
- Working software delivered frequently;
- Close cooperation among teams involved;
- Regular adaptation;
- Late changes welcomed.
Disadvantages:
- Difficulty to assert effort required to achieve overall project objectives;
- Lack of necessity on design and documentation which then makes maintenance difficult and costly;
- The project can get easily off track when there is lack of customer involvement;
- Only senior programmers can be involved in the project.
In our company best practise is to use mixture of both methodologies to achieve manageability of the project and low maintenance efforts. The mixture at the end results in reduced TOC (Total Cost of Ownership).