Data Generation of test data

Writing your own test data generator

There are cases where it makes sense to generate data in your SQL Server database. If you are dealing with one table this can be a pretty straight forward task and you may consider writing it yourself as a simple app or code that runs before your tests run.

The hard part to rolling your own generator for a single table is the generation of data. There are some nuget packages that will help with this.

NBuilder - this is great for creating object graphs. It should be able to play with Entity Framework and be saved to database. In theory in situ updates could be done.

Bogus - you can save changes using Entity Framework to the database. The relationships would be handled by entity framework automatically. If the data is loaded from the data and then updated using Bogus generated data the data could be anonymized in situ. It does have really nice options for rules the data must follow.

REX - a command line tool to generate data that follows a regular expression. Very cool.

Free-ish products for generating data for single table

If you don't want to roll your own there are other options as well for simple one table type options. Below are some you may want to check out:

Yan DATA - It is a web based tool that requires you to enter your table definition into a web form and then generate the data from there. This one is nice in that it support many formats as output like SQL, JSON, Excel, XML, CSV, and HTML. It does up to 10,000 rows and is free / donation.

Mockaroo - It is similar to Yan DATA, but is limited to 1000 rows or $50 or $500 / year and its focus is on realistic data types. It even has datatypes for specific industries. It has some additional options for outputting the data including some that are database specific, etc. Interestingly they do have a REST url you can use to get data and use in your automation. It does also allow you to define your tables based on table create SQL statements or Excel column headers. It also lets you specify a formula for a column and specify how many empty values should be used.

Products for generating data for multiple related tables (they have a price)

RedGate SQL Data Generator - It is recommended by Troye Hunt.

DevArt dbForge Data Generator - appears to be a very similar tool.

FeatureToggle Review

FeatureToggle
FeatureToggle is a popular feature toggle package that has a good architecture, regular updates, a training video, doesn't use magic strings, and is extensible. The documentation can be found here.

FeatureToggle Review
Multiple platforms: .NET Desktop/Server, Windows phone, Windows Store
No magic strings
No default fallback values - throws exception
Flexible provider model to allow for swapping out of parts.
Straight forward to use
Extensible via Providers and Custom Toggles.

Downside
I don't like that I have to specify the connection string that I want to use for each toggle. It would be nice if it could use a default connection string instead. It seems a bit clunky to specify a key for the feature toggle AND one to point to the database connection string. Perhaps that can be changed.

Support Configurations

• Compiled
• Local configuration (app.config / web.config or App.xaml)
• Centralized SQL Server

Architecture

Built-in Toggle classes use providers to get configurations out specific sources such as database, configuration file, etc.

Strongly typed objects => Compile time checks to make sure it is completely removed.

Built-in Toggles

• AlwaysOnFeatureToggle
• AlwaysOffFeatureToggle
• SimpleFeatureToggle
• EnableOnOrAfterDateFeatureToggle 
• EnableOnOrBeforeDateFeatureToggle
• EnableBetweenDatesFeatuerToggle
• EnableOnDaysOfWeekFeatureToggle
• RandomFeatureToggle - could use for random a/b testing
• SqlFeatureToggle - toggle from value in SQL Server database

Installation

Nuget: FeatureToggle

NOTE: FeatureToggle.Core is installed when FeatureToggle is installed.

Overview of Usage

For each feature that a feature toggle is required, a new class that inherits from one of the built-in Toggle classes is required.

Compiled Toggle Usage

public class MyFeature : AlwaysOnFeatureToggle {}

In place where want to use it, create a new instance of the MyFeature class.

i.e.

public MyFeature MyFeature1 = new MyFeature();

Can add to ViewModel to use on Razor page.

i.e. 

@if (Model.MyFeature1.FeatureEnabled)

{

....html here

}

Config File Usage

In this example it would be the same as above, except we change the class we inherit from so that we can get the value from the config file. Note, the ViewModel and Razor page did not need to change even though we are changing what the logic is for hiding the feature.

i.e. 

public class MyFeature : SimpleFeatureToggle{}

We do need to add it to the configuration file. We just need to add a new key to the appSetting. It must follow the convention such that the key starts with "FeatureToggle." and concatenated to the name of our class (MyFeature in this case).

i.e.

<appSetting>

<add key="FeatureToggle.MyFeature" value="true"/>

</appSettings>

SqlFeature Toggle Usage

We will continue our example as before, except we need to also tell the FeatureToggle where the database is.

In a database create a table called Toggles

ToggleName nvarchar(100) not null (primary key)

Value bit not null

NOTE: The table and column names or types are not important since we will write the SQL to access it later.

We need to add the connection string to the list of connection strings in the web.config.

<connectionString>

<add name="MyDB" connectionString="typical connection string here" />

</connectionString>

Insert a record with the ToggleName being the name of our class (MyFeature in this case) and value = True.

We need to specify the connection string in the web.config. We put it in as an appSetting key-value as

<appSettings>

<add key="FeatureToggle.MyFeature.ConnectionStringName" value="MyDB" />

<add key="FeatureToggle.MyFeature.SqlStatement" value="select value from Toggles where ToggleName = 'MyFeature'" />

</appSettings>

Removing a Toggle

Delete the class. Rebuild project. Review each compiler error. Remove from web.config. 

Creating a Custom Toggle

Custom Toggles are good when the value is based on some business logic.

Continuing on the examples...

public class MyBusinessLogicToggle : IFeatureToggle 

{

public bool FeatureEnabled { get { return businessLogicHere;}}

}

Change our Feature Toggle so that it inherits from the custom feature toggle. This can be reused with multiple feature toggles;

public class MyFeature : MyBusinessLogicToggle {}

Creating a Custom Provider

The base class for Feature Toggles has a property called ToggleValueProvider. Setting this value to a custom Provider allows us to change the default provider to a custom one.

We can create a custom provider by creating a subclass of a existing provider. 

i.e. 

public MyProvider : IBooleanToggleValueProvider

{

public bool EvaluateBooleanToggleValue(IFeature toggle)

{

return logicHere;

}

}

Some Alternatives

FeatureSwitch

FeatureSwitcher

References

Most of the data is sourced from the Implementing Feature Toggles in .NET with FeatureToggle on PluralSight.

Quick review of nice Open Source JavaScript Grids / Data Tables

DataTables

Website: http://datatables.net/
This is a very powerful and mature Grid and my top choice. My only complaint is that it is almost overwhelming because it is so feature rich and if you need editing, the coolest features you have to pay for. It supports just about any scenario out there and the docmentation is GREAT. It does a good job making simple cases easy to do though. In fact, they have a generator that generates all the code you need to get started after you specify how you want to use it, but that unfortunately is with the paid Editor. They do have for free a tool to help you figure out exactly what files you need for downloading based on your needs. It does allow cells to have input elements (see example), but there isn't any special support for them unless you buy the Editor. If you go that route, it has inline editing, popup screen editing, and single field editing just like the Editablegrid product (below) does AND is supported specifically with ASP.NET (and PHP), but again only for the paid Editor. It does support having all the rows editable at once though. It is very easy to use, but has lots of ways to do very advanced things. It can get data from the server or client. Works well with large amounts of data. The documentation is very well done. There are lots of examples and they include how it was done. It also has extensions for exporting data to Excel, etc.

Bootstrap Table

Website: http://bootstrap-table.wenzhixin.net.cn/
This one is good and has examples in GitHub, but the documentation / demo on the site is quite light. It does have additional documentation and examples on GitHub though. It is nice that users can export data to multiple formats, choose just the columns they want, search, etc. If you click on a cell it will bring up an editor for that field and can have associated validation with it.

Editablegrid

Website: http://www.editablegrid.net
This one is uniquely designed for inline editing. To edit a cell, just click it and it becomes editable. It supports PHP binding, but doesn't seem to support ASP.NET directly, but may not matter unless you need mass amounts of data editing. This grid is a bit light on the features, but is still pretty nice.

Capriza Review

Overview

Capriza is designed  specifically for non-developers. Capriza rapidly converts, transforms, and optimizes existing web-based desktop applications into secure, lightweight mobile apps (zapps), complete with modern mobile capabilities such as GPS, camera, barcode scanning, click-to-call, etc

Key Features

• Zero coding required, but can do own coding also

• Zero APIs

• Citrix or live-screenscraping like concept
• Very fast app creation

• Produces HTML5-based mobile apps  (Capriza call them zapps) from standard web-based applications that were targeted to run on the desktop browser.
• Mobile experience is different from desktop experience; it is generally simplified
• Runs on any modern device and platform
• Mobile extension kit to add custom widgets, native functionality, etc
• SSO

• Optimized for SAP and SalesForce.com, but can work on any web application
• Mobile services such as
• GPS
• Camera
• Bar code scanner
• Click-to-call
• Location services

Distribution (MDM)

• Capriza Native App: available free through either the Apple App Store or Google Play; users login to see apps
• Capriza HTML App: For enterprises that don't want a native app use any modern browser to access the non-native app.

• Custom Enterprise URL: Centrally distribute and manage apps through a custom, corporate branded internet or intranet URL/Domain.

• Enterprise App Store (EAS)for distributing apps that is corporate branded.

• Homegrown EAS

Monitoring

• Cloud-based, real-time management dashboard
• Users
• Infrastructure
• Zapp Health monitoring

• Analytics
• Usage
• Adoption
• Feedback

Security

• Can run their software behind the corporate firewall in 8 minutes to access apps on the intranet
• If it is public the cloud can be leveraged

Platforms

• iOS
• Android
• BlackBerry

• Windows Phone

• Samsung BADA

• Firefox OS

Devices

• Smartphones

• Tablets
• Desktops

Browser Support

• Safari

• Chrome

• Android stock browser

• Others

Professional Services are available

How it Works

Zapps work in a manner similar to Citrix. The Zapps are actually thin clients. These clients don't store anything locally on devices which lowers security concerns. All communication between components use HTTPS. Here is how a typical interaction would look:

1. User launches a zapp on their mobile device it

2. The zapp connects to the Capriza Relay Server (on premise or cloud) via HTTPS. It manages the communication between the zapp and the Runtime Agents.

3. The Capriza Relay Server starts a Runtime Agent which is a headless browser.

4. The Runtime Agent connects to the legacy web app and is rendered (not visible to anyone) in the Runtime Agent.

5. The Runtime Agent (or maybe the Runtime Agent not sure) translates the legacy web app to a mobile friendly format and sends it to the Zapp (mobile app) to be displayed to the user.

Developer Experience

• Capriza Designer (Firefox Plug-in)

• Walk through web application and drag parts that you are interested in to the mobile app screen to create the screens.

• Screens are customizable
• No programming needed

• Can change layouts, etc.

Changes

• When the legacy web application changes zapp will need to change also and should be part of change management.

Cost Model

Depends on how want to license

• Buy Platform for unlimited use
• Per User per month for smaller uses

• Prices vary based on specific details of how to deploy, etc.

Conclusions

I have investigated the product.  Below is a summary of what I found. In general please understand that this tool does NOT allow you to add functionality to a mobile application that is not already on the legacy web application. So, it is not a tool for doing new mobile development if there is not an existing web application that it will interact with. The presentation of the user interface is changed to be for a mobile device. This is typically much simplified to be more task specific. It could be very good for bring existing web applications to the mobile device when source code level access to the existing web application is not available. It could also be useful for prototyping changes to web applications that we do have source code level access to as well.

Pros:

• Very fast application creation when it plays nicely with the Capriza tooling

• Works on any mobile device

• Low cost of development due to time savings

• Distribution simplified

• Minor changes such as layout and cosmetics do not require the mobile application to be modified typically

Cons:

• Cannot be any faster than the existing web site and there is some overhead

• Some web apps will be work better with this tool than others

• Major changes to legacy web application affect the mobile application

Oracle APEX Review

Overview

Oracle APEX = Oracle Application Express

Been around since 2004 under various names

Browser Based development and deployment

Think of it as a replacement for MS Access, but on the web and multi-user with an Oracle backend.

RAD tool for the Oracle Database; think Forms over data

Declaratively build web 2.0 applications

Leverage SQL and PL/SQL skills.

Multi-tenant Hosting

Departmental Solutions is focus

The Oracle store is built with APEX

Lots of options for Authentication or even custom

Built-in support to prevent URL tampering

20 canned themes or can create your own, including one for starting mobile.

Wizards to create forms including master detail, etc, but has basic layout, functionality, etc.

Create web services with wizard. A web service can be created from a region also.

Can add regions to a page and add most anything to the region including forms, etc.

Cost

Fully supported by Oracle

Free with Oracle Database (include Oracle XE)

Skill Set Required

PL/SQL

SQL

Skill Set Required for Customization

HTML

JavaScript

CSS

Advantages

Robust migration path for Oracle Forms application to APEX.

Very easy to generate data-driven CRUD data entry style applications that include simple reporting.

Migrating MS Access, or Excel files to a multi-user web environment

Similar to SharePoint lists such that you can create tables, and UI based on an Excel spreadsheet.

Free with Oracle database license

RAD or demo or POC

Built-in themes

Customization done using standards such as JavaScript, HTML, and CSS.

Disadvantages

Debugging can be painful. No breakpoints,

Business Logic written in PL/SQL. No layers such as a business layer and data access layer. SQL or stored procedures are referenced directly.

Web based development environment  feels a bit clumsy and slow.

Tightly bound to PL/SQL and Oracle.

You must work within the paradigm that is defined by APEX. If the application doesn't fit the paradigm the effort drastically increases. For instance javascript and html/css become the way of working.  This is a much different skillset. Alternatively, plug-ins can be created to extend APEX.

Migration path from MS Access to APEX is limited and works on with simple

The time you save in development time can be quickly lost in debugging, support, and customization.

No version control 

Architecture

• Web Browser
• Apache with mod_plsql/EPG web listener
• Application Express
• Meta Data

KEY FEATURES according to a Oracle APEX Specialist

FRAMEWORK

The APEX framework uses SQL and PL/SQL on the back-end, and HTML, CSS, and JavaScript for the user interface. SQL and PL/SQL are solid and proven languages and they allow APEX developers to leverage the features of the Oracle database. HTML, CSS, and JavaScript are industry standard components for building web applications.

SUPPORT FOR MULTIPLE AUTHENTICATION SCHEMES

APEX supports various authentication schemes such as LDAP, database, Single Sign on, Oracle Access Manager, custom, etc.  This makes it convenient to integrate with any existing applications and authentication systems.

DECLARATIVE MOBILE APPLICATION DEVELOPMENT

APEX is bundled with the popular jQuery mobile libraries and provides declarative support for building mobile applications. A special mobile theme provides APEX mobile applications with typical mobile features such as page transitions and gestures including swipe, pinch, and tap. Custom mobile themes can easily be created with the jQuery mobile theme roller.

RESPONSIVE DESIGN FOR OPTIMAL VIEWING ON DESKTOP, TABLET AND SMARTPHONES

APEX applications can be rendered on desktops, tablets, and smartphones by choosing a theme based on a responsive design. APEX includes a set of modern themes that are based on CSS3 and HTML5, and supports HTML5 charts, and HTML5 item types such as sliders and toggles.

PACKAGED APPLICATIONS – A FULL DEVELOPMENT SUITE

APEX is bundled with a suite of business productivity applications that can be freely used to assist with the management and control of projects. The suite contains applications for bug tracking, issue logs, checklist management, meeting minutes, group calendar, decision management, document management, and project management.

RESTFUL WEB SERVICES

APEX has built-in support for RESTful web services and allows applications to access data and services over the internet or intranet using standard web APIs. Database web services that implement SQL or PL/SQL can also be created. RESTful web services in APEX requires the APEX Listener, a J2EE based alternative for the Oracle HTTP server and mod_plsql.

DATABASE INTEGRATION

APEX is a component of the Oracle database and applications build with APEX can utilize or benefit from any feature of the database such as advanced security, RAC, Spatial, Analytics, Multimedia, XML DB, Job Scheduler, utility packages, etc.  APEX uses SQL and PLSQL to interact with the Oracle host database.

GLOBALIZATION SUPPORT

Applications build in APEX can run concurrently in different languages. Applications are developed in a primary language and can be mapped to a supported target language. Strings in the primary application are exported to a XML Localization Interchange File (XLIFF) where they are translated and imported and automatically used by the translated application.

TEAM DEVELOPMENT

Team development is a built-in feature that allows a group of developers, working on a single application, manage new features, to dos, bugs, and milestones. Users of an application can provide instant feedback which can then be classified as feature, to do, or bug.

EXTENSIBLE

The APEX development framework supports plugins that allow developers to extend the functionality of their applications with reusable custom or third-party extensions. APEX applications can also be extended with custom HTML, CSS, or JavaScript.

OTHER FEATURES

APEX is also stacked with other utilities and features that greatly improve developer productivity throughout the life of a project.

Version Control Support – Applications can be automatically exported into SQL script files where they can be included in a version control system. APEX provides an application exporter utility as well as an application splitter utility for splitting application into individual page scripts.

Error Handling – Developers can create a single error handling function that handles exception consistently across all pages in an application.

Oracle Forms Migration Tool – If Java/ADF is not for you, APEX is a viable option for converting Oracle Forms applications. APEX includes a forms migration tool to assist with the migration of Oracle Forms applications.

Accessibility – APEX applications have automatic built-in accessibility support. Applications can be rendered in high contrast mode or screen reader mode in order to meet accessibility requirements.       

Utility Reports – APEX has a comprehensive list of reports that provide real-time information on applications. Some of the more useful reports are:

• Change History – List of changes made by developers
• Advisor – Quality control review of an application
• Database Object Dependencies – List of database objects used by an application
• Debug Messages – List of debug messages generated by an application
• Recently Updated Pages – List of pages that were recently updated

Features

Reports

Forms

Charts

Calendar

Templates

Navigation

Validations

Processes

Computations

Branches

Web Services

Email Services

Translation Services

Conditional Processing

Authentication

Authorization

Session State Management

Logging & Monitoring

Interactive Reports

Integrating with Other Tools / Services

SQL

PL/SQL

RAC

Spatial

OLAP

Flashback

Web Services

Text

Multimedia

Analytic Functions

Globalization

XML DB

eBusiness Suite

Migration to APEX

Direct Excel Conversion

MS Access Conversion Support

Oracle Forms Conversion Support

Security

Popular authentication supported as well as custom

Can hide columns based on user access level

Screenshots of the development environment

Built-in User Management

List of applications in the workspace

List of all pages in an application

Design a page

Options for creating a new application

Add a page

Create a Form

Sample User Interface

of the application created from a Spreadsheet

This is an interactive report. We can add charts, group by, add aggregate columns, filter, highlight, export to CSV, re-order columns, hide columns, save report, etc.

Conclusion

Oracle APEX may be a good choice depending on your needs. Here are some key criteria for this to be a good choice:

• Oracle database is your database of choice
• You know SQL and even better you know PL/SQL
• Your application is essentially a CRUD or forms over data applications or data-centric application
• You want to do RAD
• You don't want to write lots of code
• For customization you will need JavaScript and HTML/CSS knowledge
• For writing Business logic and customizing the application you will need PL/SQL knowledge
• To get the largest savings in time you will want to use the predefined forms and wizards, but this is not required.

It also means you are willing to give up some of the more common development practices

• Having separate tiers for business logic and data access. Instead you must want your business logic to be in the database in packages.
• IDE that runs on your pc
• Robust debugger
• Unit tests
• Version control (Yes you can export files and them to version control, but it isn't quite the same thing in my opinion)
• If you want to do OOP or MVC this is
• Drag n drop files from your desktop into the IDE
• Less control over files in your project

If the above criteria is okay with you then Oracle APEX is worth looking at. You get a lot of functionality. Similar to what you would get with SharePoint, MS Access, or meta-data tools, but with different strengths and weaknesses as noted above.

References:

Introduction to Oracle Application Express (APEX) Video

Oracle APEX - A Feature-Rich Development Platform you should be using

GEOTAB review

I recently had the pleasure of reviewing GEOTAB GPS System. They have given a lot of thought into making their platform accessible via their API and their Architecture is very nice. 

Recommendation / Summary

After reviewing their website in detail and attending a demo for GEOTAB, I would technically recommend this product.  The ability to integrate with their product is strong, scalable, and well designed. Their proprietary hardware has one of the lowest failure rates in the industry (or so they say) and provides additional information that adds value to the data. To get the best features, you should use their proprietary hardware. The system is extensible from a hardware and software perspective. You can readily integrate with their GEOTAB. 

Hardware

GEOTAB has proprietary hardware, but also works with third party hardware. Their proprietary hardware appears to be superior to the competition because it uses accelerometers as well as GPS. This allows it to accurately be able to identify drivers that hard brake at low speeds, identify drivers that are distracted, etc. It has a high compatibility with most every vehicle. Installation is very easy (3 minutes they say) to install the device by plugging it into the data port of the vehicle. The hardware can be extended to interface with additional sensors, driver identification via RFID, and Garmin devices. With the Garmin devices two communication be done.

Reports/Monitoring/Dashboards

They have 30 reports, dashboards and pride themselves on productivity monitoring / reporting. Most vendors appear to record data at a periodic interval, but they do not. The reason is that some key data can be missed. This is actually a big benefit in recording the accuracy of the data.

API

The API is available in C# and JavaScript, but can also be called by any language that can make a HTTP GET or POST request via HTTPS (and most any language can). The API appears to be robust and they have a web farm so it should be scalable as well. The API is well documented and has many examples. There is a data integration Windows application they provide the source code for that extracts data from GEOTAB. It could easily be extended to import the data into your databases. You should be able to use this API to integrate with GateHouse GPS Portal if you want to or store the data in a database or file system. You can programmatically add users, devices, etc via the API as well.

GEOTAB Software Extensibility

Their web application UI can be extended using JavaScript to add calls to your own systems, or tweak their pages. The pages are unique urls so linking to their pages should be relatively easy.

Email notifications can easily be customized by users, but notifications can also be sent as texts. Their UI can also be embedded in your systems as well.

MADP changes from 2013 to 2014 according to Gartner

In the world of MADP (Mobile Application Develop) things change very fast. In fact, in April 2012 Gartner said, "Designers have to make complex trade-offs between native, hybrid and Web-oriented mobile architectures. From 2003 through 2009, Gartner observed that the majority of high-value mobile applications were written as native, but that began to change around 2009 as techniques for wrapping Web technologies emerged. These techniques create hybrid mobile applications (wrapped applications, where the container is native code, but the experience leverages the WebView capability of the OS) or as mobile Web applications. In addition, some Web-oriented applications began to offer HTML5 features, such as advanced rendering and local storage, that previously were only available within the native style.

This caused a rapid shift of focus for many enterprises toward Web-oriented techniques — so much so that, based on our surveys conducted in 2011, 40% of enterprise application developers were still targeting native first for a variety of reasons (such as performance and disconnected mode). This migration will continue for two to three more years, and we predict that, by 2015, 80% of all mobile applications developed will be hybrid or mobile-Web-oriented."

After reading about MADPs, I decided it might be useful to visually see how the players have changed from April 2012 to August 2013. So, I took the Magic Quadrant charts from 2012 and 2013 and superimposed them into one chart, added arrows from 2012 to 2013 and consolidated the labels to produce the diagram below.

NOTE: The ones in red are new in 2013. The ones in gray disappeared in 2013 from the rankings.
Some observations:

Antenna - didn't see much change in score, but is a pretty strong Leader.
Kony and Adobe - are neck and neck and both moved from a strong Visionary to Leader.
Appcelerator - They have the strongest vision out of all the companies. They need a little work on executing before they will be a Leader though.
Salesforce.com - was Niche and has moved into a strong Challenger; nearly a leader with a tad bit more vision.
jQuery - moved from fairly weak Visionary to a Leader because of their improved Execution. They still seem to be a bit light on the vision though.
IBM - moved from Niche player to a strong Leader. They lead in second on both execution and vision. They could be a good choice overall.
SAP - stayed in the Leader circle, but has gained some execution, but at the expense of losing some vision.
Google - stayed in the Niche player square, but has significantly increased their vision.
Blackberry - stayed in Niche player square, and really just lost its ability to execute. Not much hope on this one I think.
Microsoft - stayed in the Niche player square, and gained a little vision, but at the expense of execution.
Apple - stayed in the Niche player square, and gained noticeable amount of vision, but again at the cost of execution.

I am very curious what the 2014 Gartner report will bring.