Add 'Test With > .NET' and 'Test With > .NET Core' commands #21

jcansdale · 2016-07-26T10:49:38Z

Allows tests to be executed with a specific framework in .xproj style projects.

jcansdale · 2016-07-26T13:26:27Z

Getting there:

mwhelan · 2016-07-26T13:59:50Z

Wow, that looks great! It will be really useful to be able to just run .Net Core or .Net Desktop for multi-targeted tests.

Is .Net Desktop an established moniker for .Net 4x? As opposed to "classic" or something? Or .Net 1-4x?

I'm not familiar with In-Proc. What does that do?

jcansdale · 2016-07-26T14:19:22Z

Is .Net Desktop an established moniker for .Net 4x?

Yes, kind of. I don't particularly like it and expect it will become outdated. I'm very open to suggestions for alternatives. Just .NET maybe or even .net?

I'm not familiar with In-Proc. What does that do?

It runs tests in an app-domain in the VS process. It smuggles in a link to the VS extensibility object (called DTE). This lets me easily run tests on the running instance of VS. Very useful when developing extensions. ;)

mwhelan · 2016-07-26T14:39:43Z

My vote would be for plain ".Net" rather than ".Net Desktop." I think .Net/.Net Core would be clear and less ambiguous.

jcansdale · 2016-07-26T15:56:12Z

How does this look? (be fussy!)

jcansdale · 2016-07-26T17:41:03Z

Here is another possibility, which is maybe a bit less heavy:

mwhelan · 2016-07-26T17:43:12Z

Personally, I preferred the first one. Core is maybe slightly ambiguous. Not sure about the capitals in .NET though - perhaps that is what makes it a bit heavy? How about
.Net Core
.Net
without the capitals?

jcansdale · 2016-07-26T18:05:38Z

It's tricky!

.NET Core is pretty much always written with .NET in caps (see https://www.microsoft.com/net/core).

.Net is very often written as '.net' (with a larger font), often as '.Net' and I think less often as '.NET'. I've plumped for '.Net' when I write TestDriven.Net.

dotnet.exe seems to expand them as .NETCoreApp and .NETFramework. This would be another possibility:

mwhelan · 2016-07-26T18:20:01Z

I think I like this one best... :-)

jcansdale · 2016-07-26T19:07:56Z

Calling the command '.NET Framework' makes it a lot easier to search for when adding a keyboard shortcut (the 'Test With' menu is mostly for discoverability). Also, I like the way that (T)est, (F)ramework and (C)ore line up on a QWERTY keyboard. It makes for some potentially nice keyboard combinations. 😃

The next question is icons. Maybe the plain old triangles are okay? I guess using the same icon groups the related commands together.

mwhelan · 2016-07-26T20:03:58Z

I see what you mean. They do work well for keyboard combos. Not sure about the icons though. The old TD.Net icons were distinct from each other...? Is this a new icon for .Net Core?
http://ardalis.com/wp-content/uploads/2016/05/aspnetcore-1.png

jcansdale · 2016-07-26T22:17:29Z

Is this a new icon for .Net Core?

Good question. That logo looked familiar, but I didn't know its background. Here is a little info:
https://blogs.msdn.microsoft.com/asehmi/2008/10/24/new-net-logo/

It seems Microsoft have moved from a lower case '.net' to an upper case '.NET'. That helps a bit with the choice of menu labels.

jedidja · 2016-07-28T12:00:26Z

I don't see any icons for .NET Core on the website - although they do have a few on https://www.microsoft.com/net/core/platform you could use .... the penguin for instance ;) Maybe the "Unified" icon?

jcansdale · 2016-07-28T12:10:12Z

@jedidja Yes, let's go with the penguin. ;)

I'm thinking the sober triangles are maybe okay. Using the same icon helps group the related commands together.

I'm just putting the finishing touches touches to a new upload (that includes this functionality). It's amazing low long things like adding helpful error messages can take...

jcansdale · 2016-07-28T16:31:25Z

@mwhelan, @jedidja,

I've finally got a build to try with this functionality. With any luck it will work on all of your current .xproj based projects (with the issues @mwhelan came across resolved).

The 'Test With > .NET Framework' and 'Test With > .NET Core' command will execute the first of that framework type defined in your project.json. If there isn't a suitable framework defined, it should give a friendly message showing how to fix it.

Please give it a try and see if you can break it:
TestDriven.NET-4.0.3312_Extraterrestrial_Alpha.zip

Thanks!

mwhelan · 2016-07-28T16:56:13Z

OK, that has fixed the issue with running non-test projects... however, it is also now not running a test project that it was previously running correctly. There are 3 test projects in this repo.
https://github.com/mwhelan/Specify
It is running Tests/IntegrationTests, but should also run Specify.Examples.UnitSpecs

jcansdale · 2016-07-28T17:02:41Z

Does it show an exception or other message when you target an individual test?

jcansdale · 2016-07-28T17:08:32Z

Do you use a special NuGet repo? I can't resolve some packages...

mwhelan · 2016-07-28T17:40:33Z

That's odd. That package is publicly available on NuGet.
https://www.nuget.org/packages/TestStack.BDDfy/

I downloaded Specify from GitHub, deleted my local NuGet cache, and then built it and everything built OK and it found TestStack.BDDfy 4.3.1.

The only "odd" package is an alpha for FakeItEasy, but I have included its NuGet source in the nuget.config file.

mwhelan · 2016-07-28T17:41:16Z

I don't see an error. I just get the 0 tests run message.

jcansdale · 2016-07-28T17:47:22Z

I was using a daily build of .NET Core. That may have confused it. Sorry about the false alarm! Swapped out and it's building now.

Is this not working on yours?

Assembly: Specify.Tests.dll (expand me)

------ Test started: Assembly: Specify.Tests.dll ------

NUnit .NET Core Runner 3.4.0 
Copyright (C) 2016 Charlie Poole

Runtime Environment
    OS Platform: Windows
     OS Version: 10.0.10586
        Runtime: win10-x64

Test Files
    C:\Source\GitHub\Specify\src\tests\Specify.Tests\bin\Debug\netcoreapp1.0\Specify.Tests.dll



Scenario: Stub User Story Scenario For With Examples
    Given step with <first example> passed as parameter
    Given step with <second example> accessed via property

Examples: 
| First Example | Second Example |
| 1             | foo            |
| 2             | bar            |



Scenario: 


Scenario: should handle specification without examples


Specifications For: ConcreteObjectWithMultipleConstructors

Scenario: Create b d dfy story

Specifications For: ConcreteObjectWithNoConstructor

Scenario: Create b d dfy story

Specifications For: ConcreteObjectWithNoConstructor

Scenario: Create b d dfy story
    Given some precondition
      And some other precondition
    When action
      And another action
    Then an expectation
      And another expectation


Specifications For: ConcreteObjectWithNoConstructor

Scenario: specification step order should follow standard BDDfy conventions
    Given some precondition
      And some other precondition
    When action
      And another action
    Then an expectation
      And another expectation


100 passed, 0 failed, 0 skipped, took 3.33 seconds (netcoreapp1.0).

------ Test started: Assembly: Specify.IntegrationTests.dll ------

NUnit .NET Core Runner 3.4.0 
Copyright (C) 2016 Charlie Poole

Runtime Environment
    OS Platform: Windows
     OS Version: 10.0.10586
        Runtime: win10-x64

Test Files
    C:\Source\GitHub\Specify\src\tests\Specify.IntegrationTests\bin\Debug\netcoreapp1.0\Specify.IntegrationTests.dll


226 passed, 0 failed, 0 skipped, took 3.62 seconds (netcoreapp1.0).

========== Total Tests: 326 passed, 0 failed, 0 skipped, took 7.02 seconds ==========

mwhelan · 2016-07-28T18:11:36Z

Yes, mine run those two test projects the same as you are getting, but there is a third test project - Specify.Examples.UnitSpecs - where the tests are not running....

jcansdale · 2016-07-28T18:12:40Z

Ah, I see what happened. It's related to the "known test framework" issue you mentioned. I was doing some last minute tidying up and removed something that made all .NET Core projects look like they contain tests. It now doesn't realize that your project contains tests. :(

With any luck this build will work:
TestDriven.NET-4.0.0_Extraterrestrial_Alpha.zip

(I need to work on spotting .NET Core projects with tests, probably by looking inside the project.json)

jcansdale · 2016-07-28T18:23:01Z

------ Test started: Assembly: Specify.Examples.UnitSpecs.dll ------

xUnit.net .NET CLI test runner (64-bit .NET Core win10-x64)

6 passed, 0 failed, 0 skipped, took 3.21 seconds (netcoreapp1.0).

I've noticed that targeting xUnit tests inside generic types is problematic at the moment. I need to work on getting the Type.FullName -> xUnit test path working properly. Current workaround is to target a namespace.

namespace Specify.Examples.UnitSpecs
{
    <---- i.e. here

mwhelan · 2016-07-28T18:26:37Z

Do you have an example of what you mean by targeting xUnit tests inside generic types? I don't follow...

That build is running all the test projects again, but it is trying to run the non-test project again:

------ Test started: Assembly: Specify.Examples.dll ------

Couldn't find an appropriate testRunner to execute your tests.
Please ensure your project is a Console Application and has one of the following as a dependency:
"dotnet-test-nunit"
"dotnet-test-xunit"

Alternatively, any method or property without a test attribute can be executed as an "Ad hoc" test.

jcansdale · 2016-07-28T18:32:15Z

Yes, say you targeted this fact:

    public abstract class ScenarioFor<TSut> : Specify.ScenarioFor<TSut>
            where TSut : class
    {
        [Fact]
        public override void Specify()
        {
            base.Specify();
        }
    }

mwhelan · 2016-07-28T18:37:35Z

that's my whole framework... :-)

jcansdale · 2016-07-28T19:24:04Z

Ah okay, so you would never actually target the Specify method? In this project you would only target the_item_is_available or processing_an_order directly?

mwhelan · 2016-07-28T19:30:11Z

Actually, you would always target the Specify method. That is the only method in the test project that has a Fact/Test attribute, so the test runner picks it up and executes it, then BDDfy takes over and uses reflection to scan the class and run all the Given/When/Then methods.

jcansdale · 2016-07-28T19:40:26Z

Currently there are a few things you can target and have tests successfully run:

class: processing_an_order
class: orders_with_a_negative_quantity
namespace: Specify.Examples.UnitSpecs.OrderProcessing
assembly: Specify.Examples.UnitSpecs

If a future version you would also be able to target:

class: the_item_is_available
class: ScenarioFor<TSut>
method: ScenarioFor<TSut>.Specify

Which of these would ordinarily be useful? Do you tend to always run all tests in the project? Is console output important for the tests?

mwhelan · 2016-07-28T20:15:14Z

Yes, console output is really important. Previous versions of xunit and nunit used to display output of each BDDfy test in console, but the latest versions do not (Specify just sits on top of BDDfy). I think it is to do with the architectural changes they made to support parallelism. So, it would be great to get it through TD.Net.

I'm not sure what scenarios it would be useful to target abstract classes like the_item_is_available. Perhaps to run all the tests that inherited from that class?

The typical use case is to run a full class that inherits ScenarioFor<T>. These examples are actually quite atypical and are just there to show some of the flexibility that is possible - in particular to demonstrate that some styles possible in another framework can be achieved with Specify.

The normal use case looks like this. Unfortunately, test runners often just display the method name "Specify" for each test. Ideally the test runner would display the name of the class, DetailsForExistingStudent, as the test name, and the console would display the Given When Then sentences.....?

public class DetailsForExistingStudent : ScenarioFor<StudentController>
{
    ViewResult _result;
    private Student _student = new Student { ID = 1 };

    public void Given_an_existing_student()
    {
        Container.Get<ISchoolRepository>()
            .FindStudentById(_student.ID)
            .Returns(_student);
    }

    public void When_the_details_are_requested_for_that_Student()
    {
        _result = SUT.Details(_student.ID) as ViewResult;
    }

    public void Then_the_details_view_is_displayed()
    {
        _result.ViewName.Should().Be(string.Empty);
    }

    public void AndThen_the_details_are_of_the_requested_student()
    {
        _result.Model.Should().Be(_student);

``` }
}

jcansdale · 2016-07-28T20:38:08Z

Here's a build with console output from xUnit:
TestDriven.NET-4.0.0_Extraterrestrial_Alpha.zip

Unfortunately it can get messy, which is why I disabled it in previous builds.

mwhelan · 2016-07-29T08:36:59Z

Oh wow, that is brilliant! It is great to see the Given/When/Then steps output to the console, as happened with earlier versions of xUnit and NUnit. I'd love to know more about how you did that as it might provide clues to how we can resolve it in BDDfy. Thing is, it's probably xUnit-specific code is it? BDDfy is agnostic to all tools and just uses this reporter to print to the console.

jcansdale · 2016-07-29T10:12:11Z

The dotnet-test-xunit runner doesn't currently collect console output (and error). At the moment it leaks out and I'm able to scoop it up. Unfortunately it isn't synchronized with the tests (see below). There was also an issue where failed tests were being reported both to the console and as a test event (that TD.Net displays). All of this combined made the output of failed tests into a train-wreck on the output window!

Luckily, @bradwilson gave me a tip that will make it easier to filter out the duplicate fail message:
https://twitter.com/bradwilson/status/758827481834983424

Here is a simplified example of what I'm dealing with:

    public class xTests
    {
        [Fact]
        public void Test1()
        {
            Console.WriteLine("fu");
        }

        [Fact]
        public void Test2()
        {
            Console.WriteLine("bar");
        }
    }

...outputs the following text and events (some removed for clarity)...

------ Test started: Assembly: dotnet-test-xunit.Tests.dll ------

xUnit.net .NET CLI test runner (64-bit Desktop .NET win10-x64)

fu
bar
{"message":"testStarting","flowId":"378bc3d7f31e4e8a958a5d03fd3c5561","testName":"Test1"}

{"message":"testPassed","flowId":"378bc3d7f31e4e8a958a5d03fd3c5561","executionTime":0.0116612,"output":""}
{"message":"testStarting","flowId":"378bc3d7f31e4e8a958a5d03fd3c5561","testName":"Test2"}

{"message":"testPassed","flowId":"378bc3d7f31e4e8a958a5d03fd3c5561","executionTime":0.0002949,"output":""}

2 passed, 0 failed, 0 skipped, took 2.98 seconds (net451).

Notice the output appears before the tests and the "output" properties are empty. I need to find a way to make this look okay.

mwhelan · 2016-07-29T10:16:26Z

Yes, I had extraneous info for my Specify tests as well. Would be good to exclude xunit logging and confine it to the test/class name and test output.

jcansdale · 2016-07-29T14:17:49Z

Here's a more polished release for you to try:

TestDriven.NET-4.0.3316_Extraterrestrial_Alpha.zip

I've done the following:

Added support for targeting one or more projects using Test With > .NET Core/Framework.
Cleaned up console output from xUnit tests.
It will attempt to execute tests in any .xproj project (warning if it doesn't find any tests).

bradwilson · 2016-07-29T14:46:40Z

Yeah, the decision not to capture Console (and Debug and Trace) was an explicit one in v2. We have an injectable test output mechanism (ITestOutputHelper via ctor), and that is what will show up in the output of a test.

mwhelan · 2016-07-31T09:05:00Z

I uninstalled TD.Net and then installed build 3316 attached here, but not getting that same menu. I've restarted Visual Studio. Is there anything else I need to do?

jcansdale · 2016-07-31T10:00:01Z

You should only see it on the Code and Project context menus for .xproj
type projects. Is that what you're doing?

Could you see if anything appears in DebugView:
https://technet.microsoft.com/en-us/sysinternals/debugview.aspx

On 31 July 2016 at 10:05, Michael Whelan [email protected] wrote:

I uninstalled TD.Net and then installed build 3316 attached here, but not
getting that same menu. I've restarted Visual Studio. Is there anything
else I need to do?
[image: image]
https://cloud.githubusercontent.com/assets/187802/17275618/32ad51f2-5706-11e6-98cd-dbae53c1bd53.png

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#21 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALLR-EIrAK6JTkfQPr6Fc4IGd5CT8Fhjks5qbGU8gaJpZM4JVBRc
.

mwhelan · 2016-07-31T10:18:42Z

OK... I was right clicking on a file.

The icons look great. And I really like the feature of being able to choose
with target to run.

The output is excellent too, showing exactly the right BDDfy output for
both .Net Core and .Net Framework.
[image: Inline image 1]

Very jealous. Would love to replicate that in BDDfy... though without
taking a dependency on NUnit or xUnit. Are you getting this output from
xUnit's ITestOutputHelper? If so, does that mean this won't work with
NUnit?

On Sun, Jul 31, 2016 at 11:00 AM, Jamie Cansdale [email protected]
wrote:

You should only see it on the Code and Project context menus for .xproj
type projects. Is that what you're doing?

On 31 July 2016 at 10:05, Michael Whelan [email protected] wrote:

I uninstalled TD.Net and then installed build 3316 attached here, but not
getting that same menu. I've restarted Visual Studio. Is there anything
else I need to do?
[image: image]
<
https://cloud.githubusercontent.com/assets/187802/17275618/32ad51f2-5706-11e6-98cd-dbae53c1bd53.png

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<
#21 (comment)
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/ALLR-EIrAK6JTkfQPr6Fc4IGd5CT8Fhjks5qbGU8gaJpZM4JVBRc

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#21 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AALdmmxdshTcgVT3vl8LwO9s-p_Mz6y_ks5qbHIigaJpZM4JVBRc
.

jcansdale · 2016-08-09T12:41:32Z

Missed this comment before.

Very jealous. Would love to replicate that in BDDfy... though without
taking a dependency on NUnit or xUnit.

This might be possible at some point. I'm thinking of writing a dotnet-test-tdnet runner that can execute TestDriven.Net's ITestRunner based adapters directly.

Are you getting this output fro xUnit's ITestOutputHelper? If so, does that mean this won't work with
NUnit?

I'm scooping it up from the test process stdout. xUnit and NUnit aren't capturing it at the moment for this type of test runner.

jcansdale · 2016-08-09T12:42:12Z

'Test With > .NET' and 'Test With > .NET Core' seem to be working nicely. Closing this issue. 😄

jcansdale mentioned this issue Jul 29, 2016

Stack trace has no source links nunit/dotnet-test-nunit#60

Open

jcansdale added this to the v4.0-beta milestone Aug 1, 2016

jcansdale closed this as completed Aug 9, 2016

Add 'Test With > .NET' and 'Test With > .NET Core' commands #21

Add 'Test With > .NET' and 'Test With > .NET Core' commands #21

Comments

jcansdale commented Jul 26, 2016

jcansdale commented Jul 26, 2016

mwhelan commented Jul 26, 2016

jcansdale commented Jul 26, 2016 • edited Loading

mwhelan commented Jul 26, 2016

jcansdale commented Jul 26, 2016

jcansdale commented Jul 26, 2016

mwhelan commented Jul 26, 2016

jcansdale commented Jul 26, 2016

mwhelan commented Jul 26, 2016

jcansdale commented Jul 26, 2016

mwhelan commented Jul 26, 2016

jcansdale commented Jul 26, 2016

jedidja commented Jul 28, 2016

jcansdale commented Jul 28, 2016

jcansdale commented Jul 28, 2016

mwhelan commented Jul 28, 2016

jcansdale commented Jul 28, 2016

jcansdale commented Jul 28, 2016

mwhelan commented Jul 28, 2016

mwhelan commented Jul 28, 2016

jcansdale commented Jul 28, 2016 • edited Loading

mwhelan commented Jul 28, 2016

jcansdale commented Jul 28, 2016

jcansdale commented Jul 28, 2016 • edited Loading

mwhelan commented Jul 28, 2016

jcansdale commented Jul 28, 2016

mwhelan commented Jul 28, 2016

jcansdale commented Jul 28, 2016

mwhelan commented Jul 28, 2016

jcansdale commented Jul 28, 2016 • edited Loading

mwhelan commented Jul 28, 2016

jcansdale commented Jul 28, 2016

mwhelan commented Jul 29, 2016

jcansdale commented Jul 29, 2016 • edited Loading

mwhelan commented Jul 29, 2016

jcansdale commented Jul 29, 2016

bradwilson commented Jul 29, 2016

mwhelan commented Jul 31, 2016

jcansdale commented Jul 31, 2016 • edited Loading

mwhelan commented Jul 31, 2016

jcansdale commented Aug 9, 2016

jcansdale commented Aug 9, 2016

jcansdale commented Jul 26, 2016 •

edited

Loading

jcansdale commented Jul 28, 2016 •

edited

Loading

jcansdale commented Jul 28, 2016 •

edited

Loading

jcansdale commented Jul 28, 2016 •

edited

Loading

jcansdale commented Jul 29, 2016 •

edited

Loading

jcansdale commented Jul 31, 2016 •

edited

Loading