Async in C#

Tips

A thread is a relatively heavy weight structure. 1 thread takes 1M of Stack Space. A task is NOT a thread. You can have multiple tasks running on a single thread (such as UI thread).

Task.Delay() is like Thread.Sleep(), but it is a Task that won't complete for a specified amount of time.

The keyword async is not part of the method signature. For example, it doesn't show up in interfaces. It is convention to add Async to the end of a method name though. This tell the reader it is async.

Return Task, not void.

await

Frees thread to do other things such as update UI, etc. Before await is called the stack trace is in the method you called it from which makes sense. The potentially surprising thing is that the line after the await does NOT have the same stack trace. It is now in the same thread, but clearly not in the same calling method anymore. This is because the compiler creates a state machine when the code is complied. It's building what would have been done by hand before we had the keyword await in C#.

• await frees a thread while we wait for an operation to complete. For example, handle another request in web scenario, update UI thread in desktop scenario. 
• It does NOT block the thread, it actually frees the thread to do something else. 
• The execution is stopped until the task is complete. 
• When the task is complete the thread gets focus and execution continues. This is similar in concept to a callback after something is complete.

Exception Handling

If there is an exception in an async method it will be thrown and can be caught as if it was not an async method. For example, a simple try-catch.

Free resources for Parallel Programming

Below is a list of free resources for parallel programming with .NET.

Microsoft's main site for all things parallel

• http://msdn.microsoft.com/concurrency

Parallel Programming in the .NET Framework

• https://docs.microsoft.com/en-us/dotnet/standard/parallel-programming/index

Parallel Programming with Microsoft .NET: Design Patterns for Decomposition and Coordination on Multicore Architectures.

• https://msdn.microsoft.com/en-us/library/ff963553.aspx
• Highly recommended, easy to read FREE ebook.

Passing data to a Task using parameter to avoid race condition

Requirements

Consider the scenario where you want to pass the current value of i to a new task that gets created in a for loop. We want each task to have a unique value of i (0...9).

The wrong approach (closures)

In general, it is a bad idea do use closures to pass data to a task when the value will change before the task starts. Below is a classic race condition such that the value of i is not read until the task starts, but before most of the tasks start all the tasks have been created. This will result in gotParamNow being equal to 10 (or any value really, but not what we intended).

for (int i=0; i<10; i++)

{

Task.Factory.StartNew( () =>

{

int gotParamNow = i;

}

);

}

The right approach (parameter)

If you want to pass data when the Task is CREATED (not when started as it would do when using closures) then you can pass the data to the StartNew method as shown below.

for (int i=0; i<10; i++)
{
Task.Factory.StartNew( (arg) =>
{
int gotParamNow = (int)arg;
},
i
);
}

In the code above the value i changes over time, so we want to pass it to the StartNew method so that when the task starts the data has the value that was passed to it and will result in each task having a unique value of i passed to it and thus gotParamNow will be unique as well.

Task Cancellation

Intro

When doing speculative work (starting multiple thread and only caring about the first result that comes back) these thread can take up valuable resources such as cpu or are long running, etc. We can just let these threads run to completion, but that won't free up the resources until they are done. Ideally once we have the result we want to cancel the threads once we don't need them to run anymore.

Cooperative Model

Creator passes a cancellation token, starts the task, later signals cancel, etc
Task monitors the token, if it is cancelled then it performs the cleanup and throws an exception.

Status on Task is set to "Canceled"


var cts = new CancellationTokenSource();
var token = ctx.Token;

Task t = Task.Factory.StartNew( () =>
{
try{
while (...)
{
// check for cancellation at start of each iteration
if (token.IsCancellationRequested)
{
// clean up
// NB: Make sure this exception bubbles up (as shown here) to the caller.
token.ThrowIfCancellationRequested();
}

// do the stuff would normally do here
...
}
}
catch (OperationCancelledException)
{
throw;
}
catch (Exception ex)
{
// handle exception
}
}, 
token // allows .NET to set status appropriately
);

// this may be invoked by a menu, keystroke, or other reason.
// This will trigger the canelling of the task.
// This would typically not be in the same location as the rest of the code here.
if (some condition occurs) {
cts.Cancel();
cts = new CancellationTokenSource(); // If need to re-run
}

try 
{ 
t.Wait(); // throws exception if cancelled
}
catch (AggregateException ae) {
ae = ae.Flatten();
foreach(var ex in ae.InnerExceptions)
{
if (ex is OperationCancelledException)
// do something (ignoring it now)
else
// handle exception (ex.Message will get error message)
}

}

Note: By passing the same token to several tasks will cause all the tasks to be cancelled when cts.Cancel() is called.

Caveat: Once Cancel() has been called on a task, the token passed to the Task cannot be used again once its status has been changed to Canceled. If you re-run the task you will need to create a new token as shown above. A good place is right after Cancel() is called.

Alternative approach

Use a global variable and check it to see if the task has been canceled. The trick in either implementation is to get the right level of polling. Too often will have performance implications and too little and cancelling won't be quick enough and waste resources.

Reference

Content is based on Pluralsight video called Introduction to Async and Parallel Programming in .NET 4 by Dr. Joe Hummel. 

Writing Better User Stories - Part 3 (The right amount of detail)

Ideally a user story should be more than a quarter of a sprints velocity. If this is the case then user stories should be able to be finished during a sprint in most cases. If the team is having trouble completing user stories it may be that they have too much or too little detail. Below are some techniques for determining what the right level of detail is for a story.

Too Little Detail

If a story does not have enough detail you may find that there are too many unknowns and result in there being a lot of questions to product owner during the sprint. The delay here is during the sprint and development and is very visible. The answers can delay the development or increase the complexity of the story as well.

Too Much Detail

If a story has too much detail you may find that the team has tried to remove all uncertainty from the user story and that new functionality is taking longer to deliver from the point the request for new functionality was made. The delay in deliver here is hidden by spending lots of time refining the story BEFORE it is brought into the sprint. This shows in areas like acceptance criteria with too much detail, or all screen designed completed, or not allowing a story into a sprint if it has an uncertainty. If all creativity has been removed from the developer there is probably too much detail.

The Sweet Spot

Ask the team "Did you get just enough detail to complete this iteration's user stories and was that information provided just in time?" in the retrospective. This will help in future stories. Strive to get the right level of detail "on average", not every story.

Detail should be provided "Just enough and Just in time" because too early and the information will be out of date by the time the team works on the user story. The will likely cause delays in the sprint or waste time reworking a story.

Analysts job with Agile is to provide the information Just in Time since there is not large chunk of analysis done at the start of the project. This is in contrast to traditional waterfall methods where there is an initial stage where the analyst figures out all the details upfront. This means that the chances that the analyst will get the level of details or get something wrong increases. This means some (a small number) stories will be affected, but not likely all of them. However, this is better than building in delays, padding, etc into every story.

The idea is striving for perfection when moving this fast will not be effective. Consider something like Fedex that has 2 day delivery. They expect that a low percentage of packages will not be delivered as expected. The alternative is to not have 2 day delivery and guarantee 5 day delivery instead, or increase the number of staff, planes, vehicles, etc. These would affect the price of the product as well as reduce the overall benefit that most people experience just so we get 100%. The same can be said for Agile projects.

Writing Better User Stories - Part 2 (Splitting Stories using SPIDR)

Benefits of taking time to split stories

• Deeper understanding of the feature
• Can be delivered in a sprint which keeps velocity more consistent which makes it easier to pull in the right amount of work into a sprint
• Shows progress
• Smaller chunks are easier to understand
• teams are motivated to finish a small story and feel a sense of accomplishment when they do.

Bad Ideas

• Doing development in one sprint and testing in another sprint.
• Having the entire team present when splitting every story.

Using SPIDR to Split Any Story (5 ways)

The goal is not to split stories into the smallest story possible and produce lots of small stories. Instead the goal is to split the story as few times as possible such that it can comfortably be completed in a sprint.

Spike (S) 

a research activity intended to build knowledge (not deliver functionality). Use cases are times when a team cannot decide between different ways of implementing something, or a new technology is being used. Use it only as a last resort (after the other 4 approaches have been tried).

Paths (P)

A path becomes a story

Consider the paths through a story and split each path into its own story. Draw a simple flow chart and look for the conditional or places user has to decide a path to go down (Credit Card or paypal for example). Looking for the word OR may also aid in identifying different paths. Look for paths that will deliver value for the users.

A complex step becomes a story

Sometimes the complexity is not in the different paths, it is actually in just one step in the feature. In this case it may make sense to combine several paths into one path if such that they comfortably fit into a sprint. Then take the one step that is complex make it a separate story. For example, maybe the step that is complex in a shopping cart checkout is Confirming the order in which this case maybe a story for just confirming the order makes sense or even may need to be broken down into multiple stories if it doesn't fit into a sprint.

Interfaces

Split a story across multiple user interfaces (mobile OS, browser) or data interfaces. The could mean separate stories to support iOS, Android, Windows phone, etc. Or it could mean separate stories for each type of browser such as a story for Edge, another for Chrome, and another for Firefox, etc. The reality of browser differences is that usually one browser will have an issue. So, the user stories can be split into Troublesome browser and non-troublesome browser stories.

Complex User Interface

If the user interface is complex is complex it can be broken down into two stories. In the first story the goal would be to just get the data on the screen and not worry about design. The focus is to make sure it is wired up to the backend. The second story would be going back and refining the user interface to be fancy and work in a certain way.

File Formats

If you have a service that needs to import data in XML, JSON, and CSV then you can break out each format into a story. The first one would have the largest estimate because the extra work needs to be done for the importing, while the other stories would just really be about parsing different formats.

Data (D)

Develop an initial story that supports only a subset of the data. Consider how different aspects of data can be used for a particular story. Think of this is how to make AI smarter by adding in more scenarios or data sources. If making recommendations, one user story could be I want the feature to make recommendations that consider x. Another story could be I want the feature to make  recommendations that consider y. Each user story may not be enough to release or be really useful to user, but it is a step towards the releasable product that has lots of value.

Rules (R)

Relax business rules or technology standards or industry standards in an initial user story. Subsequent stories can tight up the rules.Other functional stories can be added and completed before the rules are tightened to allow more feedback on the larger solution. Sometimes rules show up as non-functional requirements such as performance.

Context

You don't need to try all 5 ways to split stories. Just start with the way that you think will likely work and if that works then you are done. If it doesn't then try another way and if that works then you are done. Continue until you have successfully split a story into smaller stories that fit comfortably in a sprint.

Caveat

Some stories cannot be split, but they are fundamentally large and rare. Using SPIDR will make it possible to split nearly all stories

Writing Better User Stories - Part 1 (Using Story Mapping)

Stories are not simply a replacement for a requirements specification. It is meant to be an interactive, regular, shorter process.

Focus on One significant thing

Focus the meeting on a single significant objective (that will take a few months to do). This allows us to plan at a quarterly focus which also typically works well for management.

Don't do story writing workshop every sprint. It is a bad idea to do so.

Minimum Viable Product (MVP) is a good thing to focus on because it promotes feedback early.

Minimum Marketable Feature (MMF) is a subset of functionality that users need, but it is enough to be valuable to users when released.

Who should be in story writing workshops

Product Owner presents the significant objective

Scrum Master or Coach to facilitate the meeting and keep it run smoothly and effectively

Any other person that will be involved in the development such as Developers, Tests, Technical Writers, DBAs, Designer, Architects. Don't leave people out this is the time where everyone learns about the story. This will allow questions to be answered, new perspective brought in, and more new and innovative solutions will likely be produced.

An exception to this is when multiple teams are involved. In this case a subset of each team may be useful to keep the size of the meeting smaller and more manageable. An alternative would be to break the story down further so that each team can have their own sub-objective and meeting and the full team would be included in these smaller story writing workshops.

Users and Stakeholders can be invited and probably should be. However if it becomes too distracting or side-tracks the workshop because they are arguing over priority of stories then it may be better to leave them out. Alternatively, reassure them that the workshop is about understanding the user story workshop, and discussing it does not prioritize it over another story.

Visualize relationship between stories

A story map is quite useful for visualizing how stories relate. This is typically describing a sequence of stories connected with THEN.

For example:

Then add Alternatives below sequence at the position where it is in the sequence. Or keyword is used to connect it in the sequence. The higher priority items should be at the top, and low at the bottom.

This is helpful for identifying missing stories. To do so, ask the following questions while reviewing the story map. Ask these questions for each type of user in the user story.

1. What will a user most likely want to do next?
2. What mistakes could a user make here?
3. What could confuse a user at this point?
4. What additional information could a user need.

Subsequently you can then break story map into releases such that each horizontal line is a new version. It can be useful to label each version as well.

Reference:

Better User Stories - an excellent series on writing better user stories.

Exception handling of Tasks in .NET

Exception Handling rules

Task Parallel Library (TPL) handles exceptions well.

If a task throws an exception E that goes unhandled:

• task is terminated
• E is caught, saved as part of an AggregateException AE, and stored in task object's Exception property.
• AE is re-thrown when one of the following is called: .Wait, .Result, or .WaitAll

Example with no exception handling:

Task<int> t = Task.Factory.StartNew(code);

int r = t.Result;

Simple example with exception handling:

Task<int> t = Task.Factory.StartNew(code);

try { int r = t.Result; }

catch (AggregateException ae) 

{

Console.WriteLine(ae.InnerException.Message);

}

Working with AggregateException graph (preferred way)

The above handling is not really sufficient. When catching the AggregateException it can be an graph of Exceptions connected via the InnerExceptions property. For example, AggregateException.InnerExceptions can contain another AggregateExceptions or Exceptions. We need to look at the leaves of the graph to get the real exceptions of interest. An easy way to do this is to use the Flatten() method;

Task<int> t = Task.Factory.StartNew(code);

try { int r = t.Result; }

catch (AggregateException ae) 

{

ae = ae.Flatten();

foreach (Exception ex in ae.InnerExceptions)

Console.WriteLine(ex.Message);

}

Example of what happens when an exception is NOT "observed"

Task t = Task.Factory.StartNew(() =>
{
int d = 0;
int answer = 100 / d;
}
);

This will cause the application to crash when garbage collection is executed.

Exception handling Design and the need for observing

It is highly recommended that you "observe" all unhandled exceptions so that when the task is garbage-collected the exception will be re-thrown then. Unfortunately, this is not the ideal place to handle the exception.

To observe you can do it by doing one of the following:

• call .Wait or touch .Result - the exception is re-thrown at this point
• call Task.WaitAll - the exception(s) are re-thrown when all have finished
• touch the task's Exception property after the task has completed
• subscribe to TaskScheduler.UnobservedTaskException which is particularly useful if a third party block of code throws the exception.

NOTE: Task.WaitAny() does NOT observe the exception.

Wait Example

try { t.Wait(); }

catch (AggregateException ae) { ... }

Exception accessed Example

if (t.Exception != null)

{ ... }

Result accessed Example

try { var r = t.Result; }

catch (AggregateException ae) { ... }

Last resort exception handling

If you don't "observe" an exception you can still handle it by subscribing to the TaskScheduler.UnobservedTaskException. 

Use cases for using this method:

• speculative tasks that you don't cancel and don't really care about the result once you get one result. If one of those tasks throws an exception you don't want it to be thrown when garbage collection takes place.
• Using a third party library that you don't trust and don't know if it will throw any exceptions.

You only want to subscribe once to the event. Good places to do this are in the application startup code, a static constructor, etc.

How to subscribe:

TaskScheduler.UnobservedTaskException += new EventHandler<UnobservedTaskExceptionEventArgs>(MyErrorHandler);

Example handling the error

static void MyErrorHandler(object sender, UnobservedTaskExcpetionEventArgs e)

{

Console.WriteLine($"Unobserved error:{e.Exception.Message}");

e.SetObserved();

}

Note the call to e.SetObserved(). This is what tells .NET that we have observed the Exception and now it will not be thrown at garbage collection. 

Reference

Content is based on Pluralsight video called Introduction to Async and Parallel Programming in .NET 4 by Dr. Joe Hummel. 

Waiting for a task to finish and harvesting Result in .NET 4

The below techniques work for code tasks and facade tasks.

Explicitly Waiting for a single task

Sometime when working with task you need to wait for a computation to complete before you can do something with the result (such as write it to the screen) as shown below.

Task t = Task.Factory.StartNew( /* code */ );

t.Wait();

Console.WriteLine(t.Status);

The result would be one of the following:

• RanToCompletion
• Canceled
• Faulted

Explicitly Waiting for Multiple tasks (Ordered)

decimal min = 0;

decimal max = 0;

Task t_min = Task.Factory.StartNew(() => {min = data.Price.Min();});

Task t_max = Task.Factory.StartNew(() => {max = data.Price.Max();});

t_min.Wait();

t_max.Wait();

Console.WriteLine(min);

Console.WriteLine(max);

Note: You can use Wait() any task as well if they are dependent on each other. This reduces the benefits of parallelism.

Harvesting Result (Implicitly Waiting for Multiple tasks Sequentially)

While there is nothing wrong with explicitly waiting there is a cleaner way to do it and with less code code.

Task<decimal> t_min = Task.Factory.StartNew(() => {return data.Price.Min();});

Task<decimal> t_max = Task.Factory.StartNew(() => {return data.Price.Max();});

Console.WriteLine(t_min.Result);

Console.WriteLine(t_max.Result);

Notice no variables that could have a race condition if more than one task was accessing one of them. Notice no calls to Wait() because Result implicitly calls Wait() before it returns the value.

Explicitly Waiting for All tasks to complete

The above waiting scenario implies that the order that we wait for the tasks to finish matters. If it does not matter and we want to wait for all tasks to complete before continuing then semantically it is better to use WaitAll().

Here is the same code as above, but instead using a WaitAll();

Task<decimal> t_min = Task.Factory.StartNew(() => {return data.Price.Min();});

Task<decimal> t_max = Task.Factory.StartNew(() => {return data.Price.Max();});

Task.WaitAll(new Task[] {t_min, t_max});

Console.WriteLine(t_min.Result);

Console.WriteLine(t_max.Result);

Explicitly Waiting for ANY task to complete

There are scenarios searches that are returning the first response and displaying it and ignoring the other responses that make sense. In this scenario, we don't care which one is first, but we do want to wait for one to complete.

Here is a new example using WaitAny(). It takes a list of tasks as input and returns the index in that array that finished first. That index can be used to get the results of the task that completed first as shown below.

Task<string> t_msn = Task.Factory.StartNew(() => {return SearchMsn;});

Task<string> t_google = Task.Factory.StartNew(() => {return SearchGoogle();});

Task[] tasks = new Task[] {t_msn, t_google};

int firstIndex = Task.WaitAny(tasks);

Task firstTask = tasks[firstIndex];

Console.WriteLine(firstTask.Result);

WaitAllOneByOne pattern

There are scenarios where you want to wait for all tasks to finish, but process results as each one completes. Another way to think of this is, imagine we want to start several tasks and we don't know in what order they will complete, but we want to process them as they are finished. This assumes that we don't need to wait for them all before we start processing the results.

This pattern is useful when:

• Some tasks may fail - discard / retry
• Overlap computations with result processing - aka hide latency

There is no built in feature for this and this is more of a pattern that you can implement if this is your scenario. Here is one conceptual implementation.

while (tasks.Count > 0)

{

int taskIndex = Task.WaitAny(tasks.ToArray());

// check and observe exception and ignore it
if (tasks[taskIndex].Exception == null)
{

// process tasks using tasks[taskIndex].Result;

}

tasks.RemoveAt(taskIndex);

}

// if we get here all tasks failed. Act accordingly

Notice this is NOT a tight while loop that will spin while the tasks are processing. The Task.WaitAny() blocks execution of the while loop until a task completes then in loops again and starts another Task.WaitAny() and continues this until all tasks have been processed.

Task Composition (One-to-One)

Sometimes we want the completion of one task to trigger the start of another task. This can be done as noted before using simple Wait() in the second task. This is the TPL approach to Wait().

That might look like this

Task t1 = Task.Factory.StartNew(() => { /* code here */ });

Task t2 = Task.Factory.StartNew(() => { t1.Wait(); /* more code here */ });

The above implementation will work, however t2 actually starts BEFORE t1 is finished. This is probably reasonable if we remember to do a t1.Wait() first thing in the code of t2. From an orchestration perspective and responsibility point of view t2 should not be concerned with waiting for t1. This should be at a higher level. This will make testing the code for t2 much easier.

To fill the deficiencies noted above we can use the ContinueWith() method wait to start t2 until after t1 has finished. One added benefit is this allows .NET to optimize the scheduling such that both t1 and t2 would be on the same thread and could actually optimize away the wait.

That code might look like this.

Task t1 = Task.Factory.StartNew(() => { /* code here */ });

Task t2 = T1.ContinueWith((antecedent) => { /* more code here */ });

Note, the parameter antecedent references the task that just finished. This can be used to check the status of the task (t1 in this case), get the result, etc.

For example to get the result from t1 the above code would be changed to look like this:

Task t1 = Task.Factory.StartNew(() => { /* code here */ });

Task t2 = T1.ContinueWith((antecedent) => { var result = antecedent.Result; /* more code here */ });

Task Composition (Many-to-one)

TPL has alternatives for WaitAll() and WaitAny() also. Instead of using WaitAll() and WaitAny(), you can use ContinueWhenAll() and ContinueWhenAny respectvely.

ContinueWhenAll

Using our example for WaitAll() we had:

Task<decimal> t_min = Task.Factory.StartNew(() => {return data.Price.Min();});

Task<decimal> t_max = Task.Factory.StartNew(() => {return data.Price.Max();});

Task.WaitAll(new Task[] {t_min, t_max});

Console.WriteLine(t_min.Result);

Console.WriteLine(t_max.Result);

We could convert that to use ContinueWhenAll() as follows:

Task<decimal> t_min = Task.Factory.StartNew(() => {return data.Price.Min();});

Task<decimal> t_max = Task.Factory.StartNew(() => {return data.Price.Max();});

Task[] tasks = new Task[] {t_min, t_max};

Task.Factory.ContinueWhenAll(tasks, (setOfTasks) => 

{

Console.WriteLine(t_min.Result);

Console.WriteLine(t_max.Result);

});

ContinueWhenAny

Using our example for WaitAny() we had:

Task<string> t_msn = Task.Factory.StartNew(() => {return SearchMsn;});

Task<string> t_google = Task.Factory.StartNew(() => {return SearchGoogle();});

Task[] tasks = new Task[] {t_msn, t_google};

int firstIndex = Task.WaitAny(tasks);

Task firstTask = tasks[firstIndex];

Console.WriteLine(firstTask.Result);

We can convert that to use ContinueWhenAny() as follows:

Task<string> t_msn = Task.Factory.StartNew(() => {return SearchMsn;});

Task<string> t_google = Task.Factory.StartNew(() => {return SearchGoogle();});

Task[] tasks = new Task[] {t_msn, t_google};

Task.Factory.ContinueWhenAny(tasks, (firstTask) =>

{

Console.WriteLine(firstTask.Result);

}

The code is similar to using WaitAll() and WaitAny(), but again a bit easier to test because the responsibility of the orchestration is at a higher level. It also reduces a few lines of code.

Reference

Content is based on Pluralsight video called Introduction to Async and Parallel Programming in .NET 4 by Dr. Joe Hummel. 

Closures and Race Conditions

Closure = code + supporting data environment

int x = 123;

Task. Factory.StartNew( () =>
{
    x = x + 1;
}
);
...
Console.WriteLine(x);

In the example above x is called a closure variable. The compiler has to figure out how to pass the value x to the lambda expression. The compiler does this By Reference, not by value. Think of By Reference as pointers to the same memory space for the shared variable x.

So, everywhere you see x in the above example it all references the same x. This means if x is changed anywhere by any of the code including the one if the separate thread we have a race condition (since the variables are read and written).

The program could either print out 123 or 124 depending on which thread finishes first. The result is not consistent.

This means we need to take steps to make sure only one thread is read or writing the variable at a time.

Behind the scenes

The compiler will generate the following code for the code above

private sealed class c__DisplayClass2

{

public int x;

public void b__0()

{

this.x = this.x + 1;

}

}

cgobj = new c__DisplayClass2();
cgobj.x = 123;
delegate = new Action(cgobj.b__0);
Task.Factory.StartNew(delegate);
...

Console.WriteLine(cgobj.x);

Reference

Content is based on Pluralsight video called Introduction to Async and Parallel Programming in .NET 4 by Dr. Joe Hummel. 

Lambda Expression

With Parameters

A method is a named block of code

public void DoSomething(parameters)
{
    code...
}

Lambda expression = unnamed block of code

They are easy to identify because they have a => in the statement to identify the lambda expression.

(parameters) => 
{
     code...
}

No Parameters

A method with NO parameters would look like this

public void DoSomething()
{
    code...
}

The lambda expression would look like

() => 
{
     code...
}

Behind the scenes

Lambda express = custom class and delegate

The compiler will create a class with an arbitrary name and a method with an arbitrary name and implemented with the same signature as the lambda expression.

Actions

You can think of Actions as pointers to methods.

An Action is an instance of a delegate.

When you see a signature for a method that requires a parameter that is of type Action what it is asking for is a lambda expression.

NOTE: To create an Action you can also use a method as shown below.
Action action = new Action(DoSomething());

Reference

Content is based on Pluralsight video called Introduction to Async and Parallel Programming in .NET 4 by Dr. Joe Hummel. 

Basic overview of Tasks and Task-based Programming in .NET 4+

What and Why

Async Programming - hide latency of potentially long-running or blocking operations (i.e. I/O) by starting them in the background.

Parallel Programming - reduce time of CPU-bound computations by dividing workload & executing simultaneously.

Here we are talking about Tasks and Task Parallel Library (TPL) which gives us:

• Cancelling
• Easier Exception Handling
• Higher-level constructs
• ...

NOTE: Before this we had

• Threads
• Async Programming Model (i.e. async delegate invocation)
• Event-based Async Pattern (i.e. BrackgroundWorker class)
• QueueUserWorkItem

Async / Parallel Components of .NET 4

Task Parallel Library (TPL) - library of functions for tasks and the notion of a task

Task Scheduler - responsible for mapping tasks to available worker threads

Resource Manager - manages pool of worker threads

Parallel LINQ (PLINQ) - like link, but runs in Parallel

Concurrent Data Structures - queue, bag, dictionary

Use Cases

Interactive UI

In the UI if there is a non-async chunk of code executing in the UI thread then while the computation is running the UI will be unresponsive. For example you can't drag the window properly or click any other buttons, etc.

Processing requests in parallel such as a website or doing independent computations.

Creating a task

using System.Threading.Tasks;

Task t = new Task( code );

t.Start();

code = computation to perform

Start() = tells .NET that task *can* start, then returns immediately. The program "forks" and now there are two code streams that are executing concurrently (original and T).

Task

A task = object representing an ongoing computation
Provides:

• Check status
• wait
• harvest results
• store exceptions

Think of a task as an object having the following properties

• Status
• Result
• Exception

Types of Tasks

Code Tasks

Executes given operation.

Example: 

Task t1 = Task.Factory.StartNew(() => { /* code */});

Facade Tasks

Task over existing operation. You use a facade task to provide a common API to the different task technologies that have been used over the years. So, instead of rewriting existing async code you can still benefit from the higher level constructs of a Task based api.

Example: 

var op = new TaskCompletionSource<T>();

Task t2 = op.Task;

Execution model

• Code-based tasks are executed by a thread on some processor
• Thread is dedicated to task until task completes.
• If there is only one core then the threads share the core. This has extra overhead, but allows UI and tasks to be running concurrently. UI isn't intensive so not a problem sharing on thread and switching between the two of them. 
• Ideally there are multiple cores and each task runs on a different core such that they are running concurrently and in parallel with less overhead.

Actions

When you create a new Task object one of the signatures accepts an Action as the parameter. An easy way to pass your chunk of code as an Action is with a Lambda expression as shown below.

Task t = new Task( () => { many lines of code here} );

t.Start();

Equivalent, but slightly more efficient way of creating the task and starting it in one line:

Task t = Task.Factory.StartNew( () => { many lines of code here } );

The multiple lines of of code will now execute in a background thread.

Reference

Content is based on Pluralsight video called Introduction to Async and Parallel Programming in .NET 4 by Dr. Joe Hummel. 

How to Mock Entity Framework when using Repository pattern and async methods

What we are testing

Imagine you are using the repository pattern or something similar and it is implemented something like this. I like this as the DbContext (InvoiceModel is a subclass of it) is not held for just the duration of this method. This is a very simple example with no parameters or corresponding Where() being used.

public class InvoiceRepository
    {
        public async Task<IEnumerable<Invoice>> GetInvoicesAync()
        {
            using (var ctx = new InvoiceModel())
            {
                return await ctx.Invoices.ToListAsync();
            }
        }
    }

The first challenge and solution

The problem with this that it is not really possible to mock out the Entity Framework because we can't get a reference to the DbContext (InvoiceModel) since it is created here. That is no problem, we can use the Factory pattern to allow us to have the advantage of holding it just for the duration of this method, but also allowing us to mock it. Here is how we might amend the above InvoiceRepository class to use the InvoiceModelFactory might be used.


public class InvoiceRepository
    {
        private IInvoiceModelFactory _invoiceModelFactory;

        public InvoiceRepository(IInvoiceModelFactory invoiceModelFactory)
        {
            _invoiceModelFactory = invoiceModelFactory;
        }


        public async Task<IEnumerable<Invoice>> GetInvoiceAsync()
        {
            using (var ctx = _invoiceModelFactory.Create())
            {
                return await ctx.Invoices.ToListAsync();
            }
        }
    }

What's the factory pattern?

The factory pattern is useful delaying the creation of an object (our InvoiceModel in this case). If we also create an interface for the factory (IInvoiceModelFactory) and also gives us the ability to change the factory in testing to create whatever kind of implementation of IInvoiceModel that we want to.

public interface IInvoiceModelFactory

    {

        InvoiceModel Create();

    }

public class InvoiceModelFactory : IInvoiceModelFactory

    {

        public InvoiceModel Create()

        {

            return new InvoiceModel();

        }

    }

Mock the Entity Framework 

When I say mock the Entity Framework, it really ends up being DbContext which is InvoiceModel in our case. I'm using NSubstitute, but any mocking framework should be able to be used. To help with mocking the entity framework I recommend using EntityFramework.NSubstitute. There are version of it for most mocking frameworks. It provides the implementation of SetupData() below.

NB. If you are using a newer version of NSubstitute than EntityFramework.NSubstitute requires you can get the source and build it yourself. It is really only 6 files.

Helpers methods for improved readability of tests

There are some methods I created to wire up the required mocks and make the tests easier to read.

private IInvoiceModelFactory GetSubstituteInvoiceModelFactory(List<Invoice> data)

{

var context = GetSubstituteContext(data);

var factory = Substitute.For<IInvoiceModelFactory>();

factory.Create().Returns(context);

return factory;

}

private InvoiceModel GetSubstituteContext(List<Invoice> data)

{

var set = GetSubstituteDbSet<Invoice>().SetupData(data);

var context = Substitute.For<InvoiceModel>();

context.Invoices.Returns(set);

return context;

}

private DbSet<TEntity> GetSubstituteDbSet<TEntity>() where TEntity : class

{

return Substitute.For<DbSet<TEntity>, IQueryable<TEntity>, IDbAsyncEnumerable<TEntity>>();

}

Writing the Test

Now it is pretty straight forward to write the actual test. We create the invoice data that would be in the database as a List of Invoices. 

[TestMethod]

public async Task GetInvoicesAsync_ReturnsInvoices()

{

//Arrange

var invoices = new List<Invoice>

{

new Invoice(),

new Invoice()

};

var factory = GetSubstituteInvoiceModelFactory(invoices);

        var invoiceRepo = new InvoiceRepository(factory);

//Act

var result = await invoiceRepo.GetInvoicesAsync();

//Assert

Assert.IsTrue(result.Any());

}

That is it. :)

Testing Private Methods

There are times where it is much easier to test a private method instead of the public method. I could agrue this is a not a good idea, but let's assume this is what we want to do.

Option 1: Reflection

You can access anything via Reflection, but it can be a bit difficult to read, particularly for a test. This is probably the hardest to figure out and I would not recommend it.

Option 2: PrivateObject

You can use PrivateObject to invoke the private method in your unit test. 

The syntax would be something like this:

var calculator = new Calculator();
var privateLogic = new PrivateObject(calculator);
privateLogic.Invoke("Add", 1, 1);

To make this easier and more user friendly you could wrap this logic up into a method. A step further you could create an extension method for Calculator in your test project that has an extension method called Add with the same parameters as the private Add method. Then from the test perspective it would act like the private method is public.

Option 3: ReflectionMagic

Another more intuitive option is ReflectionMagic available on Nuget that uses dynamic objects to expose private bits. It can be used as a syntactically easy way to access most anything you can using Reflection. The upside of this is that you don't need to do anything special to access the private method. Unfortunately there is no compiler or Intellisense to help you with the parameters to pass it, but no special coding is needed.

You could use it something like this:

var calculator = new Calculator();
calculator.AsDynamic().Add(1,1);

This is so easy, and requires no special code it probably is not worth writing the wrapper method as I talked about the the PrivateObject option.

NOTE: I had mixed results with ReflectionMagic, but the same raw source code did work.

Data Generation of test data

Writing your own test data generator

There are cases where it makes sense to generate data in your SQL Server database. If you are dealing with one table this can be a pretty straight forward task and you may consider writing it yourself as a simple app or code that runs before your tests run.

The hard part to rolling your own generator for a single table is the generation of data. There are some nuget packages that will help with this.

NBuilder - this is great for creating object graphs. It should be able to play with Entity Framework and be saved to database. In theory in situ updates could be done.

Bogus - you can save changes using Entity Framework to the database. The relationships would be handled by entity framework automatically. If the data is loaded from the data and then updated using Bogus generated data the data could be anonymized in situ. It does have really nice options for rules the data must follow.

REX - a command line tool to generate data that follows a regular expression. Very cool.

Free-ish products for generating data for single table

If you don't want to roll your own there are other options as well for simple one table type options. Below are some you may want to check out:

Yan DATA - It is a web based tool that requires you to enter your table definition into a web form and then generate the data from there. This one is nice in that it support many formats as output like SQL, JSON, Excel, XML, CSV, and HTML. It does up to 10,000 rows and is free / donation.

Mockaroo - It is similar to Yan DATA, but is limited to 1000 rows or $50 or $500 / year and its focus is on realistic data types. It even has datatypes for specific industries. It has some additional options for outputting the data including some that are database specific, etc. Interestingly they do have a REST url you can use to get data and use in your automation. It does also allow you to define your tables based on table create SQL statements or Excel column headers. It also lets you specify a formula for a column and specify how many empty values should be used.

Products for generating data for multiple related tables (they have a price)

RedGate SQL Data Generator - It is recommended by Troye Hunt.

DevArt dbForge Data Generator - appears to be a very similar tool.

Unit Testing internal methods in c#

Sometimes you want to unit test private or protected methods on a class in one of your assemblies (MyAssembly). The problem is the unit test assembly (MyAssembly.Tests) is just another assembly and has to respect the access modifiers such as private, protected, public, internal, etc.

I generally subscribe to the notion that if you have a private method there is a good chance you should be moving that functionality to another class and using dependency injection to make that logic public and testable.

There are however exceptions to this such as when overriding a protected or protected method for example. While the same principle could be applied, it may or may not be the best choice. For cases where you deem it to not the best choice there is a solution.

In your assembly (of the class you are testing) you can add the following to the AssemblyInfo.cs file to grant special rights to any class (our unit test assembly in this case) such that it has access to internal (sorry private, protected, etc still can't) methods and thus allowing you to test them.

[assembly: InternalsVisibleTo("MyAssembly.Tests")]

If you really need to test private methods you can use PrivateObject to invoke the private method in your unit test. This feels a bit dirty though.

WordPress Intro Notes

Tips
Change Permalinks in Settings to use non date format unless just blog. Can change per page.


Design:
Custom Pages
Page Templates and Template Partial (part of a page)
Plug-in Page
Error Page


WordPress Theme Structure
Style.css should be in the location of theme .html files
functions.php
Custom Files

These are default templates that are used when a more specific template can't be found

Page Templates
index.php is the only required template file... most generic...should be flexible
page.php is used to display pages
home.php is used for the front page and shows latest posts
front-page.php is used for static front pages

Posts Templates
single.php is for individual post
archive.php is for post archives
category.php is for posts of a specific category
tag.php is for posts of a specific tags

Template Partials
a section or part of a web page that is encapsulated in a template that can be pulled in to any page.
comments.php
sidebar.php
footer.php
header.php
search.php
attachment.php
date.php
image.php
404.php
author.php
rtl.php
taxonomy.php

Can combine them
i.e.
Archive-cat.php
Page-contact.php
Single-lady.php

Template Hierarchy
Determines how WordPress chooses a template file to display a page.
There is a flowchart of wordpress.org



Development
wp-config.php
Define('WP_DEBUG', true); // for development only
everything outside of wp-content folder is overwritten on update
themes are at wp-content/themes/mythemehere
create a directory here as shown above.
create the two required files for a theme.
style.css
index.php
In order for WordPress to recognize our theme we have to add the following to the style.css file.
The basic is:
/*
Theme Name: Some user friendly name for my theme here
Author: me
Description: Custom theme created for my company
textdomain: mytheme
*/

At this point it will show under Themes in WordPress.

Can also copy it from an existing theme such as Twenty Sixteen. It is just a comment at the top of the style.css file
that says the name of the theme, etc.

Notice there is no image showing a preview of our theme. Can add a Screenshot.png in the mythemehere directory. It can't be larger than 1200 x 900.

To move over my css, Copy my css into the style.css file. If using something like bootstrap that has its own css files use wp_register_style and wp_enqueue_style and add_action. Or more simply: <link rel="stylesheet" type="text/css" href="<?php echo get_template_directory_uri(); ?>/style1.css" />
This should also work: <link href="<?php echo get_stylesheet_directory_uri().?>'/style2.css'; ?>" rel="stylesheet" /> and appears to be built for purpose.

A template should not include the stuff in the header or footer. That would be in the header and footer templates.

A simple example for index.php would be

<?php get_header(); ?>
<div>some html</div>
<?php get_footer(); ?>

In the header.php include all starting with <!DOCTYPE html><html>. Add <?php wp_head(); ?> just before the end of the </head> tag so php can inject their own stuff. Delete title tag so WordPress can do it.

In the footer.php include all starting with footer text and the </body><html>. Add <?php wp_footer(); ?> just before the end of the </body> tag so php can inject their own stuff.

When referencing an image from the front-page.php file (for example) use.

If we want want to create a template for a specific page (not the front page) then use name the file something like page-contact.php such that page-<slug>.php. The template will show under the Page Attributes when adding the page. In this scenario all the page specific html and content can be in this file since it will only ever be used by this page. Effectively, the header and footer need to be removed and replace with the php calls to pull in the footer.php and header.php files we defined. Also any paths to images, etc will need to be prefixed with <?php bloginfo('template_url')?>. The downside of putting the content in it will be that it is not editable from WordPress and the theme files itself would need to be updated. BTW, to find what the slug, edit the page and go to Screen Options and check Slug. The slug will now show on the page.
<img src="<?php bloginfo('template_url') ?>/images/myimage.jpg">
If in style.css it would just be images/myimage.jpg assuming images/myimage.jpg and the style.css are in the same directory in my theme directory.

We can also create one for all pages (global). In this case name the file page_full-width.php for example where full-width is not a slug for any page.

To add menus to the header we can use WordPress's menu that can be edited using WordPress's tooling. Just add <?php wp_nav_menu( $args = array ('menu_class' => 'my-nav-class', 'theme_location' => 'primary')); ?> where the menu is to be used

To have the navigation also show in the footer.php add
<?php wp_nav_menu( $args = array ('menu_class' => 'my-footer-nav-class', 'theme_location' => 'footer')); ?>

Notice the Menus or Widgets isn't showing under Appearance menu in WordPress. We need to add the functions.php to file to our theme.

NOTE: there are no namespaces so function names can conflict with other vendor's functions. Best practice is to include the name of theme in function name.

Add functions.php to theme.

A basic file would look like:

<?php

if (! function_exists('mytheme_setup')) :

function mytheme_setup() {
add_theme_support('title-tag');
}
endif;
add_action('after_setup_theme', 'mytheme_setup()');

/* Register Menus */
function register_mytheme_menus() {
register_nav_menus(
array(
'primary' => __('Primary Menu'),
'footer' => __('Footer Menu')
)
);
}

add_action('init', 'register_mytheme_menus');

/* Add Stylesheets, fonts, etc */
function mytheme_scripts() {
wp_enqueue_style('mytheme_styles', get_stylesheet_uri());
wp_enqueue_style('mythemem_google_fonts', 'http://fonts.googleapis.com/css...');
}
add_action('wp_enqueue_scripts', 'mytheme_scripts');

Now we can remove those stylesheets and fonts from the header.php.

Now Menus is showing under Appearance in WordPress. Now we can use the Menu editor in WordPress to create our menus. Notice WordPress knows about Primary Menu adn Footer Menu as theme locations.

Widget Areas allow the content to be changed in WordPress by changing a widget. In functions.php we need to register it.

/* Add Widget Areas */
function mytheme_widget_init() {
register_sidebar( array ('name' => __('Main Sidebar', 'mytheme'),
'id' => 'main-sidebar',
'description' => __('Some descr', 'mytheme'),
'before_widget' => '<section id="%1$s" class="%2$s">',
'after_widget' => '</section>',
'before_title' => '<h2 class="widget-title">',
'after_title' => '</h2>'
));
}
add_action('widgets_init', 'mytheme_widget_init');


Add a file called sidebar.php

In the file add something like

<?php if (is_active_sidebar('main-sidebar')) : ?>
<aside class="sidebar widget-area">
<?php dynamic_sidebar('main-sidebar'); ?>
</aside>
<?php endif; ?>

Now Widgets will show up under Appearance in WordPress. Any of WordPress's widgets can be added to the area now. This includes the Text/HTML editor to allow for custom content each time we use the page template

Can use Custom Post type to allow users to use WordPress to edit content using form. Like a data driven template. These custom types show up in menu and edited like a database.


Security Concerns:
Brute force login
Don't use admin as username
Use secure password
Enable two-factor authentication
.htaccess to blog certain ip addresses

Comment Spam
Keep an eye on your comments
Tweak Discussion settings to limit links and blacklist keyowrds or ip addresses
Require users to register in order to make comment



Plug-ins
Security Plugins
All in One WordPress Security and Firewall Plugin
iThemes Security
Securi Security

Top Backup Plugins
BackUpWordPress
BackupBuddy

Top SEO Plugins
WordPress SEO by Yoast

Other
Broken Link Checker
Google Analytics
W3 Total Cache
Sync - for managing multiple WordPress sites

Securing your ASP.NET MVC website Checklist

First, let me start by saying this is not a comprehensive list, but it is a good start.

Add headers for all requests

Add this to your web.config

<system.webServer>

    <httpProtocol>

      <customHeaders>

        <clear />

<remove name="X-AspNet-Version" />
<remove name="X-AspNetMvc-Version" />
<remove name="X-Powered-By" />
<remove name="Server" />

        <add name="X-XSS-Protection" value="1; mode=block"/>

        <add name="X-Content-Type-Options" value="nosniff"/>

        <add name="Strict-Transport-Security" value="max-age=31536000"/>
<add name="X-Frame-Options" value="DENY" />
<add name="Referrer-Policy" value="no-referrer" />

      </customHeaders>

    </httpProtocol>

  </system.webServer>

This does a good job of explaining what some of the header options are

Require Strong Passwords

Go to your AccountController and find the code that creates the PasswordValidator and change it to something like this. Length is the most important thing to consider from a cryptographic complexity. 

NOTE: 12 is the minimum required, but 16 is better to make it sufficiently time consuming to hack.

manager.PasswordValidator = new PasswordValidator

            {

                RequiredLength = 12,

                RequireNonLetterOrDigit = true,

                RequireDigit = true,

                RequireLowercase = true,

                RequireUppercase = true

            };

Remove ASP.NET Technology Headers

In Global.asax add the following to the Application_Start() event.

MvcHandler.DisableMvcResponseHeader = true;

You will also need to add the following to the web.config

<system.web>
<httpRuntime targetFramework="4.5.2" enableVersionHeader="false" />
</system.web>

Remove Server Info from headers

Add the following to Global.asax.cs

protected void Application_PreSendRequestHeaders()

        {

            if (HttpContext.Current != null)

            {

                HttpContext.Current.Response.Headers.Remove("Server");

            }

        }

Also read through security issues that require reviewing your code and maybe some knowledge of how your application is written.

Restrict origin of anything loaded

To be extra safe look at adding creating a white list of what stylesheets, scripts, etc can be loaded. This will take some digging on your site, but is probably worth the effort.

There is a nuget package that does some of this. This looks to be a better choice as it is per controller, etc and explains how to use it.

X-Frame-Options

MVC5 will add in the X-Frame-Option by default. If you want to remove it and make it DENY you will need to add the following line to your Application_Start() method in the Global.asax.cs.

System.Web.Helpers.AntiForgeryConfig.SuppressXFrameOptionsHeader = true;